java.lang.NoClassDefFoundError: org/apache/commons/httpclient/URIException

I can’t get Hive to run an INSERT query into an external table stored in Elasticsearch. I’m using build elasticsearch-hadoop-2.2.0.BUILD-20151030.024731-132 with Elasticsearch 2.0 and Hive 1.2.1. The query is something like this:

add jar hdfs:///tmp/elasticsearch-hadoop.jar;
INSERT OVERWRITE TABLE [external table stored in Elasticsearch]
    SELECT
      [some columns]
FROM [some table]

When I perform the INSERT query, I get the following exception:

Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:247) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147) ... 14 more Caused by: java.lang.NoClassDefFoundError: org/apache/commons/httpclient/URIException at org.elasticsearch.hadoop.hive.HiveUtils.structObjectInspector(HiveUtils.java:57) at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:82) at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:97) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:356) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:363) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:482) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:439) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:482) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:439) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:482) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:439) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:232) ... 15 more

Is there a compatibility problem with this version of Hive, or do I need to include a different JAR?

Thanks

[ArcTheMaster]

Hi @costin and @michaelbironneau, I met the same error when using Hive on tez last week. I’m using Hive 2.0.1 and ES-Hadoop 2.3.

I think I found how to solve it. Before working with the connector, add the following jars in Hive: hive (shfs3453)> add jar /opt/application/Hive/current/lib/elasticsearch-hadoop.jar; add jar /opt/application/Hive/current/lib/elasticsearch-hadoop.jar Added [/opt/application/Hive/current/lib/elasticsearch-hadoop.jar] to class path Added resources: [/opt/application/Hive/current/lib/elasticsearch-hadoop.jar] hive (shfs3453)> add jar /opt/application/Hadoop/current/share/hadoop/common/lib/commons-httpclient-3.1.jar; add jar /opt/application/Hadoop/current/share/hadoop/common/lib/commons-httpclient-3.1.jar Added [/opt/application/Hadoop/current/share/hadoop/common/lib/commons-httpclient-3.1.jar] to class path Added resources: [/opt/application/Hadoop/current/share/hadoop/common/lib/commons-httpclient-3.1.jar]

You must adapt the path to the Hadoop lib according to your environment. Moreover, Hadoop must be installed on each datanode/workernode of your cluster otherwise it will not work.

Anyway, it’s strange to require this version of commons-httpclient instead of the Hive built-in version. Both have the class: [root@uabigspark01 ~]# strings -f /opt/application/Hadoop/current/share/hadoop/tools/lib/* |grep "org/apache/commons/httpclient/URIException" /opt/application/Hadoop/current/share/hadoop/tools/lib/commons-httpclient-3.1.jar: org/apache/commons/httpclient/URIException.class /opt/application/Hadoop/current/share/hadoop/tools/lib/commons-httpclient-3.1.jar: org/apache/commons/httpclient/URIException.classPK [root@uabigspark01 ~]# strings -f /opt/application/Hive/current/lib/* |grep "org/apache/commons/httpclient/URIException" /opt/application/Hive/current/lib/commons-httpclient-3.0.1.jar: org/apache/commons/httpclient/URIException.class /opt/application/Hive/current/lib/commons-httpclient-3.0.1.jar: org/apache/commons/httpclient/URIException.classPK

Maybe this can be modified in ESHadoop? Or maybe a few lines can be added in the documentation to help people with Hive on Tez?

[sasauz]

@costin thank you for your answer. I will try it next week and give you a feedback. I use Hortonworks (HDP 2.3 deployed with Ambari) and I run my queries with Cloudera Hue 3.9.