question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

upgrade to Hadoop 3.1.1 to be compatible with current HDP version

See original GitHub issue

Currently, the latest version of Hadoop supported seems to be 2.7.1, so Halyard 2.2 will not work with the latest version 3.1 of Hadoop which is included in the latest HDP 3.0.1 (released already last November). This makes initial setup, testing and adoption unnecessarily hard, as you have to dig out an old version from the archive. Newer HDP versions also include Docker images which would provide an additional benefit for prototyping and quick deployment. See: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.1/release-notes/content/comp_versions.html

It may be because Halyard 2.2 still uses deprecated methods (see #6), or because of the major version change, didn’t check closely.

So, creating the repository works, but opening fails:

> create repository
No template called repository found in /root/.RDF4J/console/templates
> create hbase
Please specify values for the following variables:
Repository ID: halyardproto
Repository title: Halyard Prototype
HBase Table Name: 
Create HBase Table if missing (true|false) [true]: 
HBase Table presplit bits [0]: 
Use Halyard Push Evaluation Strategy (true|false) [true]: 
Query Evaluation Timeout [180]: 
Optional ElasticSearch Index URL: 
Repository created
> open halyardproto
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Admin.createTable(Lorg/apache/hadoop/hbase/HTableDescriptor;[[B)V
	at com.msd.gin.halyard.common.HalyardTableUtils.getTable(HalyardTableUtils.java:166)
	at com.msd.gin.halyard.common.HalyardTableUtils.getTable(HalyardTableUtils.java:144)
	at com.msd.gin.halyard.sail.HBaseSail.initialize(HBaseSail.java:193)
	at org.eclipse.rdf4j.repository.sail.SailRepository.initializeInternal(SailRepository.java:151)
	at org.eclipse.rdf4j.repository.base.AbstractRepository.initialize(AbstractRepository.java:34)
	at org.eclipse.rdf4j.repository.manager.LocalRepositoryManager.createRepository(LocalRepositoryManager.java:273)
	at org.eclipse.rdf4j.repository.manager.RepositoryManager.getRepository(RepositoryManager.java:424)
	at org.eclipse.rdf4j.console.command.Open.openRepository(Open.java:80)
	at org.eclipse.rdf4j.console.command.Open.execute(Open.java:65)
	at org.eclipse.rdf4j.console.Console.executeCommand(Console.java:376)
	at org.eclipse.rdf4j.console.Console.start(Console.java:337)
	at org.eclipse.rdf4j.console.Console.main(Console.java:239)

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:6

github_iconTop GitHub Comments

1reaction
asotonacommented, Jun 2, 2019

Halyard 3.0 has been released with HDP 3.1 support.

0reactions
asotonacommented, Apr 18, 2019

Preliminary testing progress of the 3.0 nightly build on HDP 3.1 seems to be quite promising 😃 Please report any inconveniences observed.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Solved: Hadoop 3.1.1 fixes in HDP 3.0 - Cloudera Community
I would like to know if these fixes (3.1.1 release) are already included with HDP 3.0 when I install it via Ambari-2.7.0.0? If...
Read more >
Apache Hadoop 3.3.4 – HDFS Rolling Upgrade
Introduction. HDFS rolling upgrade allows upgrading individual HDFS daemons. For examples, the datanodes can be upgraded independent of the ...
Read more >
HDP 2.6.4 to HDP 3.1.0.0 - IBM
Supports upgrading only from HDP 2.6.4 to HDP 3.1.0.0 version and not to a ... versions are compatible with the HDP, Mpack and...
Read more >
Isilon: Upgrade Hortonworks HDP2.6.5 to HDP3.* on Dell ...
Prerequisites · Hadoop cluster running HDP 2.6.5 and Ambari Server 2.6.2.2. · DellEMC PowerScale OneFS updated to 8.1.2 and patch 240163 installed ...
Read more >
Apache Spark 2.3.1 compatibility with Hadoop 3.0 in HDP 3.0
I am plannig to upgrade from Hortonworks Data platform[HDP] (version 2.6.x) to HDP 3.0. But, there seems to be some major bugs in...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found