Ora-12520 12516 Ora-12520 12516 Received When Listener on Node 1 is Down in RAC Setup

Applies to:

Oracle Net Services - Version: 9.2.0.7.0 to 11.2.0.3 - Release: 9.2 to 11.2
Information in this document applies to any platform.
RAC instances configured with Primary and Secondary option.
Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production
Sun Solaris

Symptoms

When using a TAF connect string in a primary/secondary RAC environment. SQL*Plus throws an ORA-12520 when the listener  is down on  node1.

ORA-12520: TNS:listener could not find available handler for requested type of server

Attempting to connect directly to node 2 via its own listener fails with TNS-12516

The LSNRCTL Services output shows the following:    
    
    LSNRCTL for Solaris: Version 9.2.0.7.0 - Production on 09-JAN-2006 15:34:00
   
    Copyright (c) 1991, 2002, Oracle Corporation.  All rights reserved.
   
    Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC)))
    Service "xtcac.danet" has 2 instance(s).
      Instance "xtcac1", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=rotxtca01)(PORT=1521)))
      Instance "xtcac2", status READY/SECONDARY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:29 refused:0 state:ready
             LOCAL SERVER

Cause

ACTIVE_INSTANCE_COUNT set to 1 and primary instance is still running at the time of the test.
You have shutdown only the Listener so it leads to Ora-12516.

ACTIVE_INSTANCE_COUNT enables you to designate one instance in a two-instance cluster as the
primary instance and the other instance as the secondary instance. This parameter has no
functionality in a cluster with more than two instances.

When you set this parameter to 1, the first instance you start up becomes the primary instance and
accepts client connections. The second instance starts up as a secondary instance and can accept
client connections only if the first instance fails. In such an event, the secondary instance
becomes the primary instance.

When the failed instance can once again be started up, it starts up as the secondary instance, and
will not accept client connections unless the current primary instance fails.

Solution

You have Active(Primary) and Passive(Secondary) instance configuration in RAC. In that case the Failover wouldn't happen if just the Listener on(Primary) node1 is down. Still the Primary Instance is Up and no connections on secondary can be made.
So the Local connection on node2 also failed with tns-12516 errors.
Failover only happens if the Primary instance is down.
    
This type of connect-time failover happens properly on active-active RAC instances.