redis 连接报错 GET_LIKE_ERROR 处理过程

1，redis报警

运营同事说app登录不了，运维工程师说卡在redis这里，然后去后台看了app工程日志，有如下报警信息：

 

2016-06-06 16:26:03,838 [http-6600-2:ERROR] com.plocc.ugc.redis.service.impl.RedisServiceImpl - GET_LIKE_ERROR

org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

                at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:97)

                at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:143)

                at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:41)

                at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:128)

                at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:91)

                at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:78)

                at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:177)

                at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:152)

                at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:84)

                at org.springframework.data.redis.core.DefaultHashOperations.get(DefaultHashOperations.java:44)

                at com.plocc.ugc.dao.impl.UgcRedisDaoImpl.getIsLike(UgcRedisDaoImpl.java:77)

                at com.plocc.ugc.redis.service.impl.RedisServiceImpl.getIsFave(RedisServiceImpl.java:126)

                at com.plocc.ugc.service.impl.RecommendationH5ServiceImpl.queryRecommendationList(RecommendationH5ServiceImpl.java:119)

                at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                at java.lang.reflect.Method.invoke(Method.java:606)

                at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)

                at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

                at org.springframework.remoting.support.RemoteInvocationTraceInterceptor.invoke(RemoteInvocationTraceInterceptor.java:77)

                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

                at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

                at com.sun.proxy.$Proxy188.queryRecommendationList(Unknown Source)

                at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)

 

               

               

2，在应用服务器上连接下当前连接的redis

在应用服务器上telnet端口，发现close了：

[tomcat@azure_ipower_app_3_29 logs]$ telnet192.168.3.40 6379

Trying 192.168.3.40...

Connected to 192.168.3.40.

Escape character is '^]'.

-ERR max number of clients reached

Connection closed by foreign host.

[tomcat@azure_ipower_app_3_29 logs]$

 

3，去redis服务器查看redis服务

但是在redis服务器上，redis是起着的：

[root@azure_ipower_cache1_3_40 ~]# ps-eaf|grep redis

root    43660     1  7 16:54 ?        00:02:06/usr/local/redis/bin/redis-server 127.0.0.1:6379                

root    43874 43409  0 17:24 pts/2    00:00:00 grep redis

[root@azure_ipower_cache1_3_40 ~]#

 

4，问题分析解决

重启了应用tomcat服务器以及重启了redis服务器还是一样的报错信息，后来想到了，最近做了redis服务器迁移，可能是旧的redis服务器的配置引起的，然后尝试停止了旧的redis的服务器，去掉修改了/etc/hosts里面的kafka、zookpeer指定：

 

注释掉kafka、zookpeer的映射：

vim /etc/hosts

#10.250.1.11     k1

#10.250.1.21     k2

#10.250.1.22     k3

# （这里使用redis是经过了zookpeer、kafka的转发的，所以对/etc/hosts里面配置的ip和别名的映射）。

 

 

然后在应用服务器上，telnetredis 端口，ok，可以用了：

[tomcat@azure_ipower_app_3_29ugc_service]$  telnet 192.168.3.40 6379

Trying 192.168.3.40...

Connected to 192.168.3.40.

Escape character is '^]'.

 

然后再启动应用服务器就ok了，应用能正常对redis进行操作了，公司业务系统也可以正常运转了。

 

 

网上搜索看到别人经历过的出错原因有：

（1）      linux防火墙iptables没有关闭

（2）      redis假死重启redis

（3）      连接的redis是从库不是主库导致