Jenkins Centos部署过程中异常No route to host问题解决

Jenkins在centos6.9（Final）下遇到如题所示没法路由主机问题，此问题是在部署阶段报出。我用Oracle VirtualBox通过Boot2Docker.iso实现的时候直接用的宿主IP 这里在centos直接自己安装的docker，所以出现差异导致找不到主机路由。

没法路由主机问题详细日志

Started by user cvnaviBuilding in workspace /root/.jenkins/workspace/webtest-centosFetching changes from the remote Git repositoryChecking out Revision 957ce0e60c3abcd970c025b6b92bc2ef356a8bf9 (refs/remotes/origin/master)Commit message: "Update Hello.java"[webtest-centos] $ /root/.jenkins/tools/hudson.tasks.Maven_MavenInstallation/apache-maven-3.3.9/bin/mvn -f pom.xml clean install[INFO] Scanning for projects...[INFO]                                                                         [INFO] ------------------------------------------------------------------------[INFO] Building webtest 0.0.1-SNAPSHOT[INFO] ------------------------------------------------------------------------[INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ webtest ---[INFO] Deleting /root/.jenkins/workspace/webtest-centos/target[INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ webtest ---[WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy filtered resources, i.e. build is platform dependent![INFO] skip non existing resourceDirectory /root/.jenkins/workspace/webtest-centos/src/main/resources[INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ webtest ---[INFO] Changes detected - recompiling the module![WARNING] File encoding has not been set, using platform encoding ANSI_X3.4-1968, i.e. build is platform dependent![INFO] Compiling 1 source file to /root/.jenkins/workspace/webtest-centos/target/classes[INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ webtest ---[WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy filtered resources, i.e. build is platform dependent![INFO] skip non existing resourceDirectory /root/.jenkins/workspace/webtest-centos/src/test/resources[INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ webtest ---[INFO] No sources to compile[INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ webtest ---[INFO] No tests to run.[INFO] [INFO] --- maven-war-plugin:2.2:war (default-war) @ webtest ---[INFO] Packaging webapp[INFO] Assembling webapp [webtest] in [/root/.jenkins/workspace/webtest-centos/target/webtest][INFO] Processing war project[INFO] Copying webapp resources [/root/.jenkins/workspace/webtest-centos/src/main/webapp][INFO] Webapp assembled in [25 msecs][INFO] Building war: /root/.jenkins/workspace/webtest-centos/target/webtest.war[INFO] WEB-INF/web.xml already added, skipping[INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ webtest ---[INFO] Installing /root/.jenkins/workspace/webtest-centos/target/webtest.war to /root/.m2/repository/com/boonya/app/webtest/0.0.1-SNAPSHOT/webtest-0.0.1-SNAPSHOT.war[INFO] Installing /root/.jenkins/workspace/webtest-centos/pom.xml to /root/.m2/repository/com/boonya/app/webtest/0.0.1-SNAPSHOT/webtest-0.0.1-SNAPSHOT.pom[INFO] ------------------------------------------------------------------------[INFO] BUILD SUCCESS[INFO] ------------------------------------------------------------------------[INFO] Total time: 1.870 s[INFO] Finished at: 2017-08-14T12:12:33+00:00[INFO] Final Memory: 20M/476M[INFO] ------------------------------------------------------------------------Deploying /root/.jenkins/workspace/webtest-centos/target/webtest.war to container Tomcat 8.x Remote with context webtestERROR: Build step failed with exceptionorg.codehaus.cargo.container.ContainerException: Failed to redeploy [/root/.jenkins/workspace/webtest-centos/target/webtest.war]at org.codehaus.cargo.container.tomcat.internal.AbstractTomcatManagerDeployer.redeploy(AbstractTomcatManagerDeployer.java:192)at hudson.plugins.deploy.CargoContainerAdapter.deploy(CargoContainerAdapter.java:77)at hudson.plugins.deploy.CargoContainerAdapter$DeployCallable.invoke(CargoContainerAdapter.java:147)at hudson.plugins.deploy.CargoContainerAdapter$DeployCallable.invoke(CargoContainerAdapter.java:117)at hudson.FilePath.act(FilePath.java:998)at hudson.FilePath.act(FilePath.java:976)at hudson.plugins.deploy.CargoContainerAdapter.redeploy(CargoContainerAdapter.java:114)at hudson.plugins.deploy.PasswordProtectedAdapterCargo.redeploy(PasswordProtectedAdapterCargo.java:93)at hudson.plugins.deploy.DeployPublisher.perform(DeployPublisher.java:64)at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:45)at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:730)at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:676)at hudson.model.Build$BuildExecution.post2(Build.java:186)at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:621)at hudson.model.Run.execute(Run.java:1760)at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)at hudson.model.ResourceController.execute(ResourceController.java:97)at hudson.model.Executor.run(Executor.java:415)Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)at java.net.PlainSocketImpl.socketConnect(Native Method)at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)at java.net.Socket.connect(Socket.java:589)at java.net.Socket.connect(Socket.java:538)at sun.net.NetworkClient.doConnect(NetworkClient.java:180)at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)at sun.net.www.http.HttpClient.New(HttpClient.java:308)at sun.net.www.http.HttpClient.New(HttpClient.java:326)at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202)at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966)at org.codehaus.cargo.container.tomcat.internal.TomcatManager.invoke(TomcatManager.java:561)at org.codehaus.cargo.container.tomcat.internal.TomcatManager.list(TomcatManager.java:876)at org.codehaus.cargo.container.tomcat.internal.TomcatManager.getStatus(TomcatManager.java:889)at org.codehaus.cargo.container.tomcat.internal.AbstractTomcatManagerDeployer.redeploy(AbstractTomcatManagerDeployer.java:173)... 17 morejava.net.NoRouteToHostException: No route to host (Host unreachable)at java.net.PlainSocketImpl.socketConnect(Native Method)at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)at java.net.Socket.connect(Socket.java:589)at java.net.Socket.connect(Socket.java:538)at sun.net.NetworkClient.doConnect(NetworkClient.java:180)at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)at sun.net.www.http.HttpClient.New(HttpClient.java:308)at sun.net.www.http.HttpClient.New(HttpClient.java:326)at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202)at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966)at org.codehaus.cargo.container.tomcat.internal.TomcatManager.invoke(TomcatManager.java:561)at org.codehaus.cargo.container.tomcat.internal.TomcatManager.list(TomcatManager.java:876)at org.codehaus.cargo.container.tomcat.internal.TomcatManager.getStatus(TomcatManager.java:889)at org.codehaus.cargo.container.tomcat.internal.AbstractTomcatManagerDeployer.redeploy(AbstractTomcatManagerDeployer.java:173)at hudson.plugins.deploy.CargoContainerAdapter.deploy(CargoContainerAdapter.java:77)at hudson.plugins.deploy.CargoContainerAdapter$DeployCallable.invoke(CargoContainerAdapter.java:147)at hudson.plugins.deploy.CargoContainerAdapter$DeployCallable.invoke(CargoContainerAdapter.java:117)at hudson.FilePath.act(FilePath.java:998)at hudson.FilePath.act(FilePath.java:976)at hudson.plugins.deploy.CargoContainerAdapter.redeploy(CargoContainerAdapter.java:114)at hudson.plugins.deploy.PasswordProtectedAdapterCargo.redeploy(PasswordProtectedAdapterCargo.java:93)at hudson.plugins.deploy.DeployPublisher.perform(DeployPublisher.java:64)at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:45)at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:730)at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:676)at hudson.model.Build$BuildExecution.post2(Build.java:186)at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:621)at hudson.model.Run.execute(Run.java:1760)at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)at hudson.model.ResourceController.execute(ResourceController.java:97)at hudson.model.Executor.run(Executor.java:415)Build step 'Deploy war/ear to a container' marked build as failureFinished: FAILURE

注：此问题与镜像的核心系统是centos还是ubuntu无关，两种镜像都拉取测试过都是此问题。

查看iptables状态

[root@mysqlserver ~]# service iptables statusTable: natChain PREROUTING (policy ACCEPT)num  target     prot opt source               destination         1    DOCKER     all  --  0.0.0.0/0            0.0.0.0/0           ADDRTYPE match dst-type LOCAL Chain POSTROUTING (policy ACCEPT)num  target     prot opt source               destination         1    MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0           2    MASQUERADE  tcp  --  172.17.0.1           172.17.0.1          tcp dpt:8080 3    MASQUERADE  tcp  --  172.17.0.2           172.17.0.2          tcp dpt:8080 Chain OUTPUT (policy ACCEPT)num  target     prot opt source               destination         1    DOCKER     all  --  0.0.0.0/0           !127.0.0.0/8         ADDRTYPE match dst-type LOCAL Chain DOCKER (2 references)num  target     prot opt source               destination         1    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:8091 to:172.17.0.1:8080 2    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:18091 to:172.17.0.2:8080 Table: filterChain INPUT (policy ACCEPT)num  target     prot opt source               destination         1    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:10050 state NEW,ESTABLISHED 2    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:21 state NEW,ESTABLISHED 3    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:3306 state NEW,ESTABLISHED 4    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:3036 5    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED 6    ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0           7    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           8    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22 9    REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited Chain FORWARD (policy ACCEPT)num  target     prot opt source               destination         1    DOCKER     all  --  0.0.0.0/0            0.0.0.0/0           2    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           ctstate RELATED,ESTABLISHED 3    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           4    REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited 5    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           Chain OUTPUT (policy ACCEPT)num  target     prot opt source               destination         1    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp spt:10050 state ESTABLISHED 2    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp spt:3306 state ESTABLISHED Chain DOCKER (1 references)num  target     prot opt source               destination         1    ACCEPT     tcp  --  0.0.0.0/0            172.17.0.1          tcp dpt:8080 2    ACCEPT     tcp  --  0.0.0.0/0            172.17.0.2          tcp dpt:8080 [root@mysqlserver ~]# 

查看Tomcat管理运行IP地址

Tomcat 管理地址：http://10.10.11.218:18091/manager/html

Server Information Tomcat Version JVM Version JVM Vendor OS Name OS Version OS Architecture Hostname IP Address Apache Tomcat/8.5.16 1.8.0_121-b13 Oracle Corporation Linux 2.6.32-504.el6.x86_64 amd64 a74e9c3227c0 172.17.0.2

所有映射地址都对上了，但是确仍然报无法路由到主机。

推测错误原因

 docker -d INFO[0000] Listening for HTTP on unix (/var/run/docker.sock) WARN[0000] You are running linux kernel version 2.6.32-504.el6.x86_64, which might be unstable running docker. Please upgrade your kernel to 3.10.0. INFO[0000] [graphdriver] using prior storage driver "devicemapper" WARN[0000] Running modprobe bridge nf_nat failed with message: , error: exit status 1 INFO[0000] Loading containers: start.                   ............INFO[0000] Loading containers: done.                    INFO[0000] Daemon has completed initialization          INFO[0000] Docker daemon                                 commit=786b29d/1.7.1 execdriver=native-0.2 graphdriver=devicemapper version=1.7.1INFO[0470] GET /v1.19/containers/json?all=1             INFO[0484] POST /v1.19/containers/cvnavi-centos-jenkins-20170814/start INFO[0492] POST /v1.19/containers/cvnavi-centos-tomcat-20170814/start 

注：很有可能是Linux内核版本过低导致的奇葩问题。

路由问题解决

原因：此路由问题是因为部署设置的tomcat 主机没有设置为Docker tomcat的容器IP。

设置之后部署成功：

Docker容器查看：

注意：我用Oracle VM通过Boot2Docker.iso实现的时候直接用的宿主IP 这里在centos直接自己安装的docker，所以出现差异导致找不到主机路由。