一次关于scan IP 的故障
来源：互联网 发布：经济发展数据库 编辑：程序博客网 时间：2024/05/21 17:36
环境：HP-UNIX , oracle 11.2.0.3 版本的三 node RAC
问题描述：问题发生前几天公司机房网络升级万兆网络，整个数据库服务器停机。网络升级完成后，
RAC环境开启正常。三天后应用的同事反映部分应用不能连接数据库，应用后台报：Listener refused
the connection with the following error:ORA-12514, TNS:listener does not currently know of
service requested in connect descriptor
尝试用本地的plsqldev 工具连接，也提示 ora-12505。
注：应用与本地plsqldev 的连接字符串配置都使用的是 scan ip
思路：
1、检查数据库实例是否正常；
2、检查service是否正常；
3、检查监听是否正常；
4、是否超过最大连接数
5、网络连接是否正常
6、查alert，listener ，集群，OS 日志
线索：
所有节点实例都正常，服务正常(共三个service，srv1,srv2,启在node1 上，srv3 在node3 上).
监听都是有启动，连接数足够充分。服务器端及客户端 ping scan IP 都能ping 通。
ping scan IP
# ping 10.1.194.208
PING 10.1.194.208: 64 byte packets
64 bytes from 10.1.194.208: icmp_seq=0. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=1. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=2. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=3. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=4. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=5. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=6. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=7. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=8. time=0. ms
----10.1.194.208 PING Statistics----
9 packets transmitted, 9 packets received, 0% packet loss
round-trip (ms) min/avg/max = 0/0/0
telnet scan ip :
# telnet 10.1.194.208 1521
Trying...
telnet: Unable to connect to remote host: Connection refused
发现不通
数据库 alert 日志没有任何错误。
监听log ：
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
08-FEB-2014 13:20:41 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=wasadmin))(SERVER=DEDICATED)(SERVICE_NAME=srv2)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.1.194.69)(PORT=44171)) * establish * srv2 * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
08-FEB-2014 13:20:42 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=wasadmin))(SERVER=DEDICATED)(SERVICE_NAME=srv2)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.1.194.69)(PORT=44174)) * establish * srv2 * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
08-FEB-2014 13:20:42 * (CONNECT_DATA=(SERVICE_NAME=querydb)(CID=(PROGRAM=python)(HOST=monitorcenter.yto56.com.cn)(USER=zabbix))) * (
ADDRESS=(PROTOCOL=tcp)(HOST=10.1.203.87)(PORT=60204)) * establish * querydb * 0
08-FEB-2014 13:20:42 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=wasadmin))(SERVER=DEDICATED)(SERVICE_NAME=srv1)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.1.194.82)(PORT=46107)) * establish * srv1 * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
08-FEB-2014 13:20:42 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=wasadmin))(SERVER=DEDICATED)(SERVICE_NAME=srv2)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.1.194.81)(PORT=45333)) * establish * srv2 * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
Sat Feb 08 13:20:43 2014
08-FEB-2014 13:20:43 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=wasadmin))(SERVER=DEDICATED)(SERVICE_NAME=srv2)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.1.194.81)(PORT=45335)) * establish * srv2 * 12514
有大量这类错误。
OS日志 无错误。
CRS日志正常。
crs_stat -t -v 发现所有RAC资源都正常。
鉴于以上原因难以确定，尝试重启监听，未能解决问题，步骤就不贴了。
于是想到重启 scan IP 。
# . /home/grid/.profile
# crsctl stop res ora.scan1.vip -f
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'expquery03'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'expquery03' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'expquery03'
CRS-2677: Stop of 'ora.scan1.vip' on 'expquery03' succeeded
# crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE expquery01
ora.DATA_DG.dg ora....up.type 0/5 0/ ONLINE ONLINE expquery01
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE expquery01
ora....N1.lsnr ora....er.type 0/5 0/0 OFFLINE OFFLINE
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE expquery01
ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE expquery01
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE expquery01
ora....01.lsnr application 0/5 0/0 ONLINE ONLINE expquery01
ora....y01.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y01.ons application 0/3 0/0 ONLINE ONLINE expquery01
ora....y01.vip ora....t1.type 0/0 0/0 ONLINE ONLINE expquery01
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE expquery02
ora....02.lsnr application 0/5 0/0 ONLINE ONLINE expquery02
ora....y02.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y02.ons application 0/3 0/0 ONLINE ONLINE expquery02
ora....y02.vip ora....t1.type 0/0 1/0 ONLINE ONLINE expquery02
ora....SM3.asm application 0/5 0/0 ONLINE ONLINE expquery03
ora....03.lsnr application 0/5 0/0 ONLINE ONLINE expquery03
ora....y03.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y03.ons application 0/3 0/0 ONLINE ONLINE expquery03
ora....y03.vip ora....t1.type 0/0 0/0 ONLINE ONLINE expquery03
ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE expquery01
ora.oc4j ora.oc4j.type 0/1 0/2 OFFLINE OFFLINE
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE expquery01
ora.querydb.db ora....se.type 0/2 0/1 ONLINE ONLINE expquery02
ora....rc1.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery01
ora....rc2.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery01
ora....rc3.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery03
ora.scan1.vip ora....ip.type 0/0 0/0 OFFLINE OFFLINE
# crsctl start res ora.scan1.vip
CRS-2672: Attempting to start 'ora.scan1.vip' on 'expquery02'
CRS-2676: Start of 'ora.scan1.vip' on 'expquery02' succeeded
# crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE expquery01
ora.DATA_DG.dg ora....up.type 0/5 0/ ONLINE ONLINE expquery01
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE expquery01
ora....N1.lsnr ora....er.type 0/5 0/0 OFFLINE OFFLINE
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE expquery01
ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE expquery01
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE expquery01
ora....01.lsnr application 0/5 0/0 ONLINE ONLINE expquery01
ora....y01.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y01.ons application 0/3 0/0 ONLINE ONLINE expquery01
ora....y01.vip ora....t1.type 0/0 0/0 ONLINE ONLINE expquery01
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE expquery02
ora....02.lsnr application 0/5 0/0 ONLINE ONLINE expquery02
ora....y02.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y02.ons application 0/3 0/0 ONLINE ONLINE expquery02
ora....y02.vip ora....t1.type 0/0 1/0 ONLINE ONLINE expquery02
ora....SM3.asm application 0/5 0/0 ONLINE ONLINE expquery03
ora....03.lsnr application 0/5 0/0 ONLINE ONLINE expquery03
ora....y03.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y03.ons application 0/3 0/0 ONLINE ONLINE expquery03
ora....y03.vip ora....t1.type 0/0 0/0 ONLINE ONLINE expquery03
ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE expquery01
ora.oc4j ora.oc4j.type 0/1 0/2 OFFLINE OFFLINE
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE expquery01
ora.querydb.db ora....se.type 0/2 0/1 ONLINE ONLINE expquery02
ora....rc1.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery01
ora....rc2.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery01
ora....rc3.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery03
ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE expquery02
但发现还是不能解决问题。
于是，想到网管必修的终极杀招————重启。
于是就把服务切到其它节点，先重启node1 ，咦，神奇的发现，重启之后，竟然可以连接了。
---------
废话这么多，其实还是没有找到真正的原因。关于这个问题，本人怀疑是因为网络的异常的缘故，导致scan ip 资源异常。
问题描述：问题发生前几天公司机房网络升级万兆网络，整个数据库服务器停机。网络升级完成后，
RAC环境开启正常。三天后应用的同事反映部分应用不能连接数据库，应用后台报：Listener refused
the connection with the following error:ORA-12514, TNS:listener does not currently know of
service requested in connect descriptor
尝试用本地的plsqldev 工具连接，也提示 ora-12505。
注：应用与本地plsqldev 的连接字符串配置都使用的是 scan ip
思路：
1、检查数据库实例是否正常；
2、检查service是否正常；
3、检查监听是否正常；
4、是否超过最大连接数
5、网络连接是否正常
6、查alert，listener ，集群，OS 日志
线索：
所有节点实例都正常，服务正常(共三个service，srv1,srv2,启在node1 上，srv3 在node3 上).
监听都是有启动，连接数足够充分。服务器端及客户端 ping scan IP 都能ping 通。
ping scan IP
# ping 10.1.194.208
PING 10.1.194.208: 64 byte packets
64 bytes from 10.1.194.208: icmp_seq=0. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=1. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=2. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=3. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=4. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=5. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=6. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=7. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=8. time=0. ms
----10.1.194.208 PING Statistics----
9 packets transmitted, 9 packets received, 0% packet loss
round-trip (ms) min/avg/max = 0/0/0
telnet scan ip :
# telnet 10.1.194.208 1521
Trying...
telnet: Unable to connect to remote host: Connection refused
发现不通
数据库 alert 日志没有任何错误。
监听log ：
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
08-FEB-2014 13:20:41 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=wasadmin))(SERVER=DEDICATED)(SERVICE_NAME=srv2)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.1.194.69)(PORT=44171)) * establish * srv2 * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
08-FEB-2014 13:20:42 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=wasadmin))(SERVER=DEDICATED)(SERVICE_NAME=srv2)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.1.194.69)(PORT=44174)) * establish * srv2 * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
08-FEB-2014 13:20:42 * (CONNECT_DATA=(SERVICE_NAME=querydb)(CID=(PROGRAM=python)(HOST=monitorcenter.yto56.com.cn)(USER=zabbix))) * (
ADDRESS=(PROTOCOL=tcp)(HOST=10.1.203.87)(PORT=60204)) * establish * querydb * 0
08-FEB-2014 13:20:42 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=wasadmin))(SERVER=DEDICATED)(SERVICE_NAME=srv1)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.1.194.82)(PORT=46107)) * establish * srv1 * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
08-FEB-2014 13:20:42 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=wasadmin))(SERVER=DEDICATED)(SERVICE_NAME=srv2)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.1.194.81)(PORT=45333)) * establish * srv2 * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
Sat Feb 08 13:20:43 2014
08-FEB-2014 13:20:43 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=wasadmin))(SERVER=DEDICATED)(SERVICE_NAME=srv2)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.1.194.81)(PORT=45335)) * establish * srv2 * 12514
有大量这类错误。
OS日志 无错误。
CRS日志正常。
crs_stat -t -v 发现所有RAC资源都正常。
鉴于以上原因难以确定，尝试重启监听，未能解决问题，步骤就不贴了。
于是想到重启 scan IP 。
# . /home/grid/.profile
# crsctl stop res ora.scan1.vip -f
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'expquery03'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'expquery03' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'expquery03'
CRS-2677: Stop of 'ora.scan1.vip' on 'expquery03' succeeded
# crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE expquery01
ora.DATA_DG.dg ora....up.type 0/5 0/ ONLINE ONLINE expquery01
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE expquery01
ora....N1.lsnr ora....er.type 0/5 0/0 OFFLINE OFFLINE
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE expquery01
ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE expquery01
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE expquery01
ora....01.lsnr application 0/5 0/0 ONLINE ONLINE expquery01
ora....y01.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y01.ons application 0/3 0/0 ONLINE ONLINE expquery01
ora....y01.vip ora....t1.type 0/0 0/0 ONLINE ONLINE expquery01
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE expquery02
ora....02.lsnr application 0/5 0/0 ONLINE ONLINE expquery02
ora....y02.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y02.ons application 0/3 0/0 ONLINE ONLINE expquery02
ora....y02.vip ora....t1.type 0/0 1/0 ONLINE ONLINE expquery02
ora....SM3.asm application 0/5 0/0 ONLINE ONLINE expquery03
ora....03.lsnr application 0/5 0/0 ONLINE ONLINE expquery03
ora....y03.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y03.ons application 0/3 0/0 ONLINE ONLINE expquery03
ora....y03.vip ora....t1.type 0/0 0/0 ONLINE ONLINE expquery03
ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE expquery01
ora.oc4j ora.oc4j.type 0/1 0/2 OFFLINE OFFLINE
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE expquery01
ora.querydb.db ora....se.type 0/2 0/1 ONLINE ONLINE expquery02
ora....rc1.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery01
ora....rc2.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery01
ora....rc3.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery03
ora.scan1.vip ora....ip.type 0/0 0/0 OFFLINE OFFLINE
# crsctl start res ora.scan1.vip
CRS-2672: Attempting to start 'ora.scan1.vip' on 'expquery02'
CRS-2676: Start of 'ora.scan1.vip' on 'expquery02' succeeded
# crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE expquery01
ora.DATA_DG.dg ora....up.type 0/5 0/ ONLINE ONLINE expquery01
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE expquery01
ora....N1.lsnr ora....er.type 0/5 0/0 OFFLINE OFFLINE
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE expquery01
ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE expquery01
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE expquery01
ora....01.lsnr application 0/5 0/0 ONLINE ONLINE expquery01
ora....y01.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y01.ons application 0/3 0/0 ONLINE ONLINE expquery01
ora....y01.vip ora....t1.type 0/0 0/0 ONLINE ONLINE expquery01
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE expquery02
ora....02.lsnr application 0/5 0/0 ONLINE ONLINE expquery02
ora....y02.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y02.ons application 0/3 0/0 ONLINE ONLINE expquery02
ora....y02.vip ora....t1.type 0/0 1/0 ONLINE ONLINE expquery02
ora....SM3.asm application 0/5 0/0 ONLINE ONLINE expquery03
ora....03.lsnr application 0/5 0/0 ONLINE ONLINE expquery03
ora....y03.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y03.ons application 0/3 0/0 ONLINE ONLINE expquery03
ora....y03.vip ora....t1.type 0/0 0/0 ONLINE ONLINE expquery03
ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE expquery01
ora.oc4j ora.oc4j.type 0/1 0/2 OFFLINE OFFLINE
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE expquery01
ora.querydb.db ora....se.type 0/2 0/1 ONLINE ONLINE expquery02
ora....rc1.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery01
ora....rc2.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery01
ora....rc3.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery03
ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE expquery02
但发现还是不能解决问题。
于是，想到网管必修的终极杀招————重启。
于是就把服务切到其它节点，先重启node1 ，咦，神奇的发现，重启之后，竟然可以连接了。
---------
废话这么多，其实还是没有找到真正的原因。关于这个问题，本人怀疑是因为网络的异常的缘故，导致scan ip 资源异常。
0 0
- 一次关于scan IP 的故障
- 关于DEL网卡驱动的一次故障
- scan ip的理解
- 一次关于DNS服务器的故障排错记录——RNDC故障
- 对scan ip的理解
- Zookeeper的一次迁移故障
- eth0 的一次意外故障
- 一次svn的故障处理
- 关于cell smart table scan一次问题处理
- 关于在网页中使用IP的一次技术讨论
- 关于等待事件enq: TX - allocate ITL entry的一次故障处理
- RAC8——scan ip的理解
- Oracle RAC的VIP和SCAN IP
- Oracle Study之--Oracle 11g RAC之scan ip访问故障
- 一次故障记录keepalived配置疏忽导致的故障
- 蓝的成长记——追逐DBA（18）：小机上WAS集群故障，由一次更换IP引起
- 一次ORA-60死锁故障的处理
- 一次由脚本升级引发的故障
- 关于ContextLoaderListener那点事
- 云计算（二十八）-【HBase】Rowkey设计
- android 如何让appwidget 里面设置TextView 滚动,android widget marquee text
- LeetCode - LRU Cache
- Java中的序列化Serialable高级详解
- 一次关于scan IP 的故障
- 如何查找论文是否被SCI,EI检索
- 堆和栈的区别
- PHP程序员突破成长瓶颈 – 附学习建议
- 了解下PHP的Reflection反射机制
- 算法对应计算机的书籍
- VS2010 修改模板文件,增加默认注释
- 京东IPO的几个细节：费用率已无处可降
- iOS XCode4.6.3 SVN 配置
