Hbase、elasticsearch整合中jar包冲突的问题解决

问题背景

再数据平台中，项目搭建需要使用es和HBASE搭建数据查询接口，整合的过程中出现jar包冲突的bug :com.google.common.base.Stopwatch.()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator

org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator

at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:239)

at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:150)

...

at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator

at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntil

...

at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:75)

at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)

... 45 more

解决办法

经排查，确认是因为 com.google.guava 这个包引起的冲突。es依赖于18及以上版本，而HBASE只支持16及以上版本。而guava这个包从17开始内部发生变化，方法发生改变，所以18并不会向下兼容16版本。在项目运行的过程中，如果同时引入16、18版本，es及hbase的调用过程会发生混乱。那么接下来就好办了，我们可以重新打包，将guava18打入es内部，再在pom文件中显示引用guava16版本的包。这样，es调用包内部打入的guava18，而hbase调用外部的guava16。

重新打包

新建maven工程，在pom文件中进行如下配置：

100

101

102

103

104

105

106

107

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

<modelVersion>4.0.0</modelVersion>

<!--

this maven project is aim to fix the jar comflic in hbase & elasticsearch.

in es, required guava 18+ or up. but in hbase you should use guava 16- or blow.

cd the project directory and run 'sh ./cleanbuild.sh',you will get a new-self jar.

then, adjust your project pom.xml whth:

<dependency>

<groupId>douguo.shaded.elasticsearch</groupId>

<artifactId>douguo_shaded_elasticsearch</artifactId>

<version>1.0-SNAPSHOT</version>

<exclusions>

<exclusion>

<groupId>org.elasticsearch</groupId>

<artifactId>elasticsearch</artifactId>

</exclusion>

</exclusions>

</dependency>

finally, your jar comflic will be fixed.

@date:2017-11-30

@author:zhangjianfei

@since:douguo.shaded.elasticsearch-1.0.0

-->

<groupId>douguo.shaded.elasticsearch</groupId>

<artifactId>douguo_shaded_elasticsearch</artifactId>

<version>1.0-SNAPSHOT</version>

<properties>

<elasticsearch.version>2.4.1</elasticsearch.version>

</properties>

<dependencies>

<dependency>

<groupId>org.elasticsearch</groupId>

<artifactId>elasticsearch</artifactId>

<version>${elasticsearch.version}</version>

</dependency>

<dependency>

<groupId>org.elasticsearch.plugin</groupId>

<artifactId>shield</artifactId>

<version>${elasticsearch.version}</version>

</dependency>

</dependencies>

<build>

<plugins>

<plugin>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-shade-plugin</artifactId>

<version>2.4.1</version>

<configuration>

<createDependencyReducedPom>false</createDependencyReducedPom>

</configuration>

<executions>

<execution>

<phase>package</phase>

<goals>

<goal>shade</goal>

</goals>

<configuration>

<relocations>

<relocation>

<pattern>com.google.guava</pattern>

<shadedPattern>douguo.shaded.elasticsearch.guava</shadedPattern>

</relocation>

<relocation>

<pattern>org.joda</pattern>

<shadedPattern>douguo.shaded.elasticsearch.joda</shadedPattern>

</relocation>

<relocation>

<pattern>com.google.common</pattern>

<shadedPattern>douguo.shaded.elasticsearch.common</shadedPattern>

</relocation>

<relocation>

<pattern>com.google.thirdparty</pattern>

<shadedPattern>douguo.shaded.elasticsearch.thirdparty</shadedPattern>

</relocation>

</relocations>

<transformers>

<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer" />

</transformers>

</configuration>

</execution>

</executions>

</plugin>

</plugins>

</build>

<repositories>

<repository>

<id>elasticsearch-releases</id>

<url>http://maven.elasticsearch.org/releases</url>

<releases>

<enabled>true</enabled>

<updatePolicy>daily</updatePolicy>

</releases>

<snapshots>

<enabled>false</enabled>

</snapshots>

</repository>

</repositories>

</project>

将org.joda等4个可能有冲突的jar包通过maven-shade-plugin插件迁移后重新打个jar包从而使得在引入这个jar包时能够使用该jar包自己的依赖而不是使用外部依赖。这里需要注意的是，需要将com.google.common等4个包全部重新迁移，否则会出现java.lang.IllegalAccessError: tried to access method com.google.common.base的错误

项目打包