Hbase、elasticsearch整合中jar包冲突

来源：互联网发布：mac 推出编辑：程序博客网时间：2024/06/06 10:21

问题背景

再数据平台中，项目搭建需要使用es和HBASE搭建数据查询接口，整合的过程中出现jar包冲突的bug :com.google.common.base.Stopwatch.()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator

org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator        at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:239)        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:150)        ...        at java.lang.Thread.run(Thread.java:745)Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntil        ...        at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:75)        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)        ... 45 more

解决办法

经排查，确认是因为 com.google.guava 这个包引起的冲突。es依赖于18及以上版本，而HBASE只支持16及以上版本。而guava这个包从17开始内部发生变化，方法发生改变，所以18并不会向下兼容16版本。在项目运行的过程中，如果同时引入16、18版本，es及hbase的调用过程会发生混乱。那么接下来就好办了，我们可以重新打包，将guava18打入es内部，再在pom文件中显示引用guava16版本的包。这样，es调用包内部打入的guava18，而hbase调用外部的guava16。

重新打包

新建maven工程，在pom文件中进行如下配置：

<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0"         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">    <modelVersion>4.0.0</modelVersion>    <!--        this maven project is aim to fix the jar comflic in hbase & elasticsearch.        in es, required guava 18+ or up. but in hbase you should use guava 16- or blow.        cd the project directory and run 'sh ./cleanbuild.sh',you will get a new-self jar.        then, adjust your project pom.xml whth:        <dependency>            <groupId>douguo.shaded.elasticsearch</groupId>            <artifactId>douguo_shaded_elasticsearch</artifactId>            <version>1.0-SNAPSHOT</version>            <exclusions>                <exclusion>                    <groupId>org.elasticsearch</groupId>                    <artifactId>elasticsearch</artifactId>                </exclusion>            </exclusions>        </dependency>        finally, your jar comflic will be fixed.        @date:2017-11-30        @author:zhangjianfei        @since:douguo.shaded.elasticsearch-1.0.0    -->    <groupId>douguo.shaded.elasticsearch</groupId>    <artifactId>douguo_shaded_elasticsearch</artifactId>    <version>1.0-SNAPSHOT</version>    <properties>        <elasticsearch.version>2.4.1</elasticsearch.version>    </properties>    <dependencies>        <dependency>            <groupId>org.elasticsearch</groupId>            <artifactId>elasticsearch</artifactId>            <version>${elasticsearch.version}</version>        </dependency>        <dependency>            <groupId>org.elasticsearch.plugin</groupId>            <artifactId>shield</artifactId>            <version>${elasticsearch.version}</version>        </dependency>    </dependencies>    <build>        <plugins>            <plugin>                <groupId>org.apache.maven.plugins</groupId>                <artifactId>maven-shade-plugin</artifactId>                <version>2.4.1</version>                <configuration>                    <createDependencyReducedPom>false</createDependencyReducedPom>                </configuration>                <executions>                    <execution>                        <phase>package</phase>                        <goals>                            <goal>shade</goal>                        </goals>                        <configuration>                            <relocations>                                <relocation>                                    <pattern>com.google.guava</pattern>                                    <shadedPattern>douguo.shaded.elasticsearch.guava</shadedPattern>                                </relocation>                                <relocation>                                    <pattern>org.joda</pattern>                                    <shadedPattern>douguo.shaded.elasticsearch.joda</shadedPattern>                                </relocation>                                <relocation>                                    <pattern>com.google.common</pattern>                                    <shadedPattern>douguo.shaded.elasticsearch.common</shadedPattern>                                </relocation>                                <relocation>                                    <pattern>com.google.thirdparty</pattern>                                    <shadedPattern>douguo.shaded.elasticsearch.thirdparty</shadedPattern>                                </relocation>                            </relocations>                            <transformers>                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer" />                            </transformers>                        </configuration>                    </execution>                </executions>            </plugin>        </plugins>    </build>    <repositories>        <repository>            <id>elasticsearch-releases</id>            <url>http://maven.elasticsearch.org/releases</url>            <releases>                <enabled>true</enabled>                <updatePolicy>daily</updatePolicy>            </releases>            <snapshots>                <enabled>false</enabled>            </snapshots>        </repository>    </repositories></project>

将org.joda等4个可能有冲突的jar包通过maven-shade-plugin插件迁移后重新打个jar包从而使得在引入这个jar包时能够使用该jar包自己的依赖而不是使用外部依赖。这里需要注意的是，需要将com.google.common等4个包全部重新迁移，否则会出现java.lang.IllegalAccessError: tried to access method com.google.common.base的错误

项目打包

mvn clean install

新的依赖包会在.m2 maven仓库中，如果公司搭建了仓库的话，需要上传jar包。如果直接运行jar包的话，记得重新编译项目，并替换lib目录

项目载入新包

只需要在pom文件中配置：

<!-- douguo.shaded.elasticsearch --><dependency>    <groupId>douguo.shaded.elasticsearch</groupId>    <artifactId>douguo_shaded_elasticsearch</artifactId>    <version>1.0-SNAPSHOT</version>    <exclusions>        <exclusion>            <groupId>org.elasticsearch</groupId>            <artifactId>elasticsearch</artifactId>        </exclusion>    </exclusions></dependency><!-- https://mvnrepository.com/artifact/com.google.guava/guava --><!-- this guava is only used in habse     in es, 18.0+ is required, but hbase only supported 16.0 or blow.     clean as install douguo.shaded.elasticsearch --><dependency>    <groupId>com.google.guava</groupId>    <artifactId>guava</artifactId>    <version>16.0</version></dependency>

这样，guava18包就在douguo.shaded.elasticsearch下，es会优先调用。而外部配置的guava16就会被HBASE调用。2个版本的jar包互相独立存在！

到此，问题就解决了！

参考文章：

http://blog.csdn.net/sunshine920103/article/details/51659936
https://www.elastic.co/blog/to-shade-or-not-to-shade

阅读全文

1 0