ERROR (notification:44) Unable to send notification: (111, 'Connection refused')



问题描述：

DWH 部门的192.168.147.69 用kettle 从192.168.147.91的ORACLE 抽取数据的时候不定期出现下述错误：
2015/05/14 12:43:58 - ECALLUSER.0 - ERROR (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : An error occurred, processing will be stopped:
2015/05/14 12:43:58 - ECALLUSER.0 - Error occurred while trying to connect to the database
2015/05/14 12:43:58 - ECALLUSER.0 -
2015/05/14 12:43:58 - ECALLUSER.0 - Error connecting to database: (using class oracle.jdbc.driver.OracleDriver)
2015/05/14 12:43:58 - ECALLUSER.0 - Io exception: The Network Adapter could not establish the connection

192.168.147.69 在vmware的虚拟环境中，192.168.147.91在ORACLE VM 环境中。

但是在192.168.147.69 用kettle 去访问另一个不同网段的数据库（这个库也在vmware环境），则没有问题。

1.首先检查了数据库的日志，没有发现问题。在192.168.147.69上用命令行访问192.168.147.91一直正常。

只是用kettle访问时会不规则的出现The Network Adapter could not establish the connection。

2. 两个机器之间用大包互相ping，也没发现有大的延迟。

3.检查了物理机与交换机之间的连通情况，也未发现问题。

排除了数据库的问题各种网络问题后，最终将问题定位到 ORACLE所在的OVM上

检查ORACLE VM 主机的日志，发现如下错误：

# tail -f /var/log/ovs-agent.log

[2015-05-15 10:17:406420] ERROR (notificationserver:244) Error initializing notification server:'Invalid URL Request (send) None&c

=5&s=1431656184447&lb=p&t=2&p=Model+Manager'

[2015-05-15 10:17:576427] ERROR (notification:44) Unable to send notification: (111, 'Connectionrefused')

[2015-05-15 10:18:106420] DEBUG (notificationserver:237) Trying to connect to manager.

[2015-05-15 10:18:116420] DEBUG (notificationserver:239) Connected to manager.

[2015-05-15 10:18:176427] ERROR (notification:44) Unable to send notification: (111, 'Connectionrefused')

[2015-05-15 10:18:376427] ERROR (notification:44) Unable to send notification: (111, 'Connectionrefused')

[2015-05-15 10:18:576427] ERROR (notification:44) Unable to send notification: (111, 'Connectionrefused')

[2015-05-15 10:19:176427] ERROR (notification:44) Unable to send notification: (111, 'Connectionrefused')

[2015-05-15 10:19:376427] ERROR (notification:44) Unable to send notification: (111, 'Connectionrefused')

[2015-05-15 10:19:576427] ERROR (notification:44) Unable to send notification: (111, 'Connectionrefused')

[2015-05-15 10:20:176427] ERROR (notification:44) Unable to send notification: (111, 'Connectionrefused')

[2015-05-15 10:20:376427] ERROR (notification:44) Unable to send notification: (111, 'Connectionrefused')

[2015-05-15 10:20:576427] ERROR (notification:44) Unable to send notification: (111, 'Connectionrefused')

[2015-05-15 10:21:176427] ERROR (notification:44) Unable to send notification: (111, 'Connectionrefused')

[2015-05-15 10:21:206420] ERROR (notificationserver:278) Error in NotificationServer process: 'InvalidURL Request (send) None&c=8&

s=1431656403866&lb=p&t=2&p=%3Ccom.oracle.odof.OdofIdentifier%3E%3Clong%3E9%3C%2Flong%3E%3C%2Fcom.oracle.odof.OdofIdentifier%3E%2Ccom

pareTo%2Cjava.lang.Object%2CNone%2C5'

Traceback (mostrecent call last):

  File "/usr/lib64/python2.4/site-packages/agent/daemon/notificationserver.py",line 253, in serve_forever

   foundry = cm.getFoundryContext()

  File"/usr/lib/python2.4/site-packages/com/oracle/ovm/mgr/api/manager/OvmManager.py",line 38, in getFoundryContext

   self.foundry = self.getModelManager().getFoundryContext()

  File"/usr/lib/python2.4/site-packages/com/oracle/ovm/mgr/api/manager/OvmManager.py",line 31, in getModelManager

    ifself.modelMgr == None:

  File"/usr/lib/python2.4/site-packages/com/oracle/ovm/mgr/api/manager/ModelManager.py",line 400, in __cmp__

   return self.compareTo(obj)

  File"/usr/lib/python2.4/site-packages/com/oracle/ovm/mgr/api/manager/ModelManager.py",line 290, in compareTo

   return self.exchange.invokeMethodByName(self.identifier,"compareTo","java.lang.Object",args,5,False)

  File"/usr/lib/python2.4/site-packages/com/oracle/odof/OdofExchange.py",line 68, in invokeMethodByName

   return self._send_(InvokeMethodByNameCommand(identifier, method, params, args,access))

  File"/usr/lib/python2.4/site-packages/com/oracle/odof/OdofExchange.py",line 164, in _send_

   return self._sendGivenConnection_(connection, command, timeout)

  File"/usr/lib/python2.4/site-packages/com/oracle/odof/OdofExchange.py",line 169, in _sendGivenConnection_

   connection.send(command, timeout)

  File"/usr/lib/python2.4/site-packages/com/oracle/odof/io/ServletConnection.py",line 67, in send

   raise OdofException("Invalid URL Request (send) %s&%s" %(self.url, request), sys.exc_info()[1])

OdofException:'Invalid URL Request (send)None&c=8&s=1431656403866&lb=p&t=2&p=%3Ccom.oracle.odof.OdofIdentifier%3E%3Clong%3E9%3C%2F

long%3E%3C%2Fcom.oracle.odof.OdofIdentifier%3E%2CcompareTo%2Cjava.lang.Object%2CNone%2C5'

 临时解决方案：

由于没人来维护这个ORACLE VM , 也没有续ORACLE VM的服务，只有重启ORACLE VM 和ORACLE VM Manager。

重启后这个问题消失。