rac监听状态为"Not All Endpoints Registered"

1,现象描述，应用反应不能正常
[grid@sdcmpdb1 ~]$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 01-JUL-2013 16:39:49

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                01-JUL-2013 15:47:57
Uptime                    0 days 0 hr. 51 min. 52 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /oracle/11.2.0/grid/network/admin/listener.ora
Listener Log File         /oracle/app/oracle/diag/tnslsnr/sdcmpdb1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
The listener supports no services
The command completed successfully

[grid@sdcmpdb1 ~]$ crsctl status res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCHDG.dg
               ONLINE  ONLINE       sdcmpdb1                                     
               ONLINE  ONLINE       sdcmpdb2                                     
ora.DATA.dg
               ONLINE  ONLINE       sdcmpdb1                                     
               ONLINE  ONLINE       sdcmpdb2                                     
ora.LISTENER.lsnr
               ONLINE  INTERMEDIATE sdcmpdb1                 Not All Endpoints R 
                                                             egistered           
               ONLINE  ONLINE       sdcmpdb2                               
               
2,经过查询，原因如下，数据库起了两个listener，一个oracle用户起的，一个grid用户启动；正常情况下，应该只有grid用户起的
[grid@sdcmpdb1 ~]$  ps -ef|grep LISTENER
oracle    8317     1  0 15:41 ?        00:00:01 /oracle/app/oracle/product/11.2.0/db_1/bin/tnslsnr LISTENER -inherit
grid     14518     1  0 15:47 ?        00:00:00 /oracle/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid     21614 21346  0 16:45 pts/0    00:00:00 grep LISTENER
3，停掉一个后，应该恢复正常了
[grid@sdcmpdb1 ~]$ ps -ef|grep LISTENER
grid     14518     1  0 15:47 ?        00:00:00 /oracle/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid     21895 21724  0 16:49 pts/0    00:00:00 grep LISTENER
4，metailink上也有此报错的文章，主要是listener端口被占用导致不能正确启动导致
Cause

The problem is caused by another listener defined statically in listener.ora, using the same port and IP is running from the RDBMS ORACLE_HOME, started manually causing the default listener starting from GRID_HOME can not register its endpoint. Hence the error reported in dbca.
ps -ef | grep tns:

grid   7222 1 0 Apr26 ? 00:00:13 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
grid   7237 1 0 Apr26 ? 00:00:13 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit
oracle 7354 1 0 Apr26 ? 00:00:01 /u02/app/oracle/product/11.2.0/db/bin/tnslsnr LISTENER -inherit


  Another possible cause is the listener or scan listener being defined manually in listener.ora, for example:
LISTENER_SCAN3 =                          
 (DESCRIPTION =
   (ADDRESS = (PROTOCOL = TCP)(HOST = racnode1)(PORT = 1523))
 )

LISTENER_SCAN1 =                            
 (DESCRIPTION =
   (ADDRESS = (PROTOCOL = TCP)(HOST = racnode1)(PORT = 1521))

 

 
Solution

From 11.2 onwards, all listeners should be runing from GRID_HOME, listener and listener_scan<n> entry should be added automatically into listener.ora, no manual editing is required for TCP definition.

1. Stop the listener running from RDBMS ORACLE_HOME

$<RDBMS ORACLE_HOME>/bin/lsnrctl stop LISTENER

2. stop the listener from GRID_HOME

$<GRID_HOME>/bin/srvctl stop listener -n <node name>
$<GRID_HOME>/bin/srvctl stop scan_listener -i <scan#>

eg:

$<GRID_HOME>/bin/srvctl stop listener -n racnode1
$<GRID_HOME>/bin/srvctl stop scan_listener -i 1

If above command fails to stop the tnslsnr process, please use "kill -9 <pid of tnslsnr>" to stop the LISTENER and LISTENER_SCAN1 process.

3. remove any manually added LISTENER definition from listener.ora if it exists

4. restart the LISTENER and LISTENER_SCAN1  from GRID_HOME

$<GRID_HOME>/bin/srvctl start listener -n <node name>
$<GRID_HOME>/bin/srvctl start scan_listener -i <scan#>

5. check crsctl stat res -t output, they both should show ONLINE status now.

参考资料：Listener in INTERMEDIATE status with "Not All Endpoints Registered" [ID 1454439.1]