Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Bug while executing example "Analyzing the Ethereum Blockchain with Apache Flink"
See original GitHub issueHi everyone !
I followed the steps written in the wiki but I have an error when I execute the jar with Flink.
- I downloaded some Ethereum blockchain with geth and converted it to bin files.
- I put it on HDFS (Sandbox HDP 2.6)
- I built the example using the following command :
sbt +clean +assembly +it:test. I do get the jar file in target directory - I downloaded the flink tar.gz file here -> http://www.apache.org/dyn/closer.lua/flink/flink-1.5.0/flink-1.5.0-bin-scala_2.11.tgz (I choose the Without bundled Hadoop version)
- I run the jar with the following command :
bin/flink run example-hcl-flink-scala-ethereumblock.jar --input hdfs://localhost:8020/user/ethereum/input --output hdfs://localhost:8020/user/ethereum/output
And I get this output :
Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR was set. Starting execution of program
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error. at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:545) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:420) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:404) at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:781) at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:275) at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210) at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020) at org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096) at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096) Caused by: java.lang.RuntimeException: Cannot create Hadoop WritableTypeInfo. at org.apache.flink.api.java.typeutils.TypeExtractor.createHadoopWritableTypeInfo(TypeExtractor.java:2155) at org.zuinnote.flink.ethereum.example.FlinkScalaEthereumBlockCounter$.countTotalTransactions(FlinkScalaEthereumBlockCounter.scala:41) at org.zuinnote.flink.ethereum.example.FlinkScalaEthereumBlockCounter$.main(FlinkScalaEthereumBlockCounter.scala:34) at org.zuinnote.flink.ethereum.example.FlinkScalaEthereumBlockCounter.main(FlinkScalaEthereumBlockCounter.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528) … 9 more Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable at org.apache.flink.api.java.typeutils.WritableTypeInfo.<init>(WritableTypeInfo.java:54) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.flink.api.java.typeutils.TypeExtractor.createHadoopWritableTypeInfo(TypeExtractor.java:2148) … 17 more Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.Writable at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) … 23 more
Am I missing something ?
Issue Analytics
- State:
- Created 5 years ago
- Comments:16 (10 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
It should not be localhost if you work on a HDP vm. If you look in the log files you will see that it cannot resolve it.
You have to use instead of localhost the dns I Provided above. On older HDPs you had to replace localhost by sandbox.hortonworks.com
besides a cluster, i tried with the above build.sbt and the HDP 2.6.5 (newer then yours, but it does not matter). I downloaded the flink 1.5 distribution with hadoop 2.7. I added the hadoopcompatibility for 1.5 in the lib folder of the flink distribution. I started the local flink cluster using the script start-cluster.sh in the bin folder of the flink distribution. I ran the following command (you need to adapt path according to the folders that you have): ./flink-1.5.0/bin/flink run example-hcl-flink-scala-ethereumblock.jar --input hdfs://sandbox-hdp.hortonworks.com:8020/home/input/ethereum/ethgenesis --output hdfs://sandbox-hdp.horto nworks.com:8020/home/output/ethereum
I find the correct output in /home/output/ethereum