MapReduce On Alluxio-Http2Exception: First received frame was not SETTINGS

Alluxio Version: 2.6

Describe the bug

• Failed to run MapReduce App on alluxio

My environment is like this.

Node leaf runs hadoop：

$ jps6992 ResourceManager
24804 NameNode
7108 NodeManager
25124 SecondaryNameNode
21406 Jps
24942 DataNode

Node alluxio01 allxio02 alluxio03 runs alluxio:

alluxio01 $ jps
8417 AlluxioJobMaster
7890 AlluxioMaster
8985 AlluxioProxy
22831 Jps
alluxio02 $ jps
9954 Jps
29749 AlluxioWorker
30278 AlluxioProxy
29982 AlluxioJobWorker
alluxio03 $ jps
28916 AlluxioWorker
9111 Jps
29434 AlluxioProxy
29149 AlluxioJobWorker

I built a 3-node Aluxio cluster according to the official documentation.

The four nodes mentioned above are all accessible to each other, and I have turned off the firewall (for research on Aluxio).Then I submit an application using the following command according to the official documentation 《Runing Hadoop MapReduce on Alluxio》:

leaf $ yarn jar hadoop-mapreduce-examples-2.7.7.jar wordcount alluxio://alluxio01:19999/wordcount/input.txt alluxio://alluxio01:19999/wordcount/output

The following error occurred:

21/06/28 23:42:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/06/28 23:42:44 INFO hadoop.AbstractFileSystem: Creating Alluxio configuration from Hadoop configuration {}, uri configuration {alluxio.zookeeper.address=null, alluxio.zookeeper.enabled=false, alluxio.master.hostname=alluxio01, alluxio.master.rpc.addresses=null, alluxio.master.embedded.journal.addresses=null, alluxio.master.rpc.port=19999}
21/06/28 23:42:45 INFO hadoop.AbstractFileSystem: Initializing filesystem with connect details alluxio01:19999
21/06/28 23:42:45 INFO network.TieredIdentityFactory: Initialized tiered identity TieredIdentity(node=leaf, rack=null)
21/06/28 23:42:45 INFO client.RMProxy: Connecting to ResourceManager at /10.215.129.73:8032
21/06/28 23:42:46 INFO network.NettyUtils: EPOLL_MODE is available
alluxio.exception.status.UnavailableException: Failed to connect to master (alluxio01:19999) after 44 attempts.Please check if Alluxio master is currently running on "alluxio01:19999". Service="FileSystemMasterClient"
        at alluxio.AbstractClient.connect(AbstractClient.java:279)
        at alluxio.client.file.BaseFileSystem.rpc(BaseFileSystem.java:573)
        at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:275)
        at alluxio.client.file.FileSystem.getStatus(FileSystem.java:342)
        at alluxio.hadoop.AbstractFileSystem.getFileStatus(AbstractFileSystem.java:359)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1425)
        at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:145)
        at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:266)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
        at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
        at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
Caused by: alluxio.exception.status.UnavailableException: Failed to handshake with master alluxio01:19999 to load cluster default configuration values: INTERNAL: http2 exception
        at alluxio.util.ConfigurationUtils.loadConfiguration(ConfigurationUtils.java:504)
        at alluxio.ClientContext.loadConf(ClientContext.java:135)
        at alluxio.ClientContext.loadConfIfNotLoaded(ClientContext.java:159)
        at alluxio.AbstractClient.beforeConnect(AbstractClient.java:175)
        at alluxio.AbstractClient.connect(AbstractClient.java:224)
        ... 29 more
Caused by: alluxio.shaded.client.io.grpc.StatusRuntimeException: INTERNAL: http2 exception
        at alluxio.shaded.client.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)
        at alluxio.shaded.client.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)
        at alluxio.shaded.client.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)
        at alluxio.grpc.MetaMasterConfigurationServiceGrpc$MetaMasterConfigurationServiceBlockingStub.getConfiguration(MetaMasterConfigurationServiceGrpc.java:371)
        at alluxio.util.ConfigurationUtils.loadConfiguration(ConfigurationUtils.java:498)
        ... 33 more
Caused by: alluxio.shaded.client.io.netty.handler.codec.http2.Http2Exception: First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f
        at alluxio.shaded.client.io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:103)
        at alluxio.shaded.client.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.verifyFirstFrameIsSettings(Http2ConnectionHandler.java:338)
        at alluxio.shaded.client.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:239)
        at alluxio.shaded.client.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438)
        at alluxio.shaded.client.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
        at alluxio.shaded.client.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
        at alluxio.shaded.client.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at alluxio.shaded.client.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at alluxio.shaded.client.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at alluxio.shaded.client.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at alluxio.shaded.client.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at alluxio.shaded.client.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at alluxio.shaded.client.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at alluxio.shaded.client.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at alluxio.shaded.client.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
        at alluxio.shaded.client.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
        at alluxio.shaded.client.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        at alluxio.shaded.client.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at alluxio.shaded.client.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.lang.Thread.run(Thread.java:748)

Seems to be the GRPC problem? But then I really did not know which direction to search, I think this is a very simple example.Do I have some JAR conflicts? The version of Hadoop I am using is 2.7.7, and only one node is deployed for demonstration purposes.

I hope you can help to check this problem, thank you very much!

To Reproduce Refer to the above.

Expected behavior Refer to the above.

Urgency Refer to the above.

Additional context Refer to the above.

[xpleaf]

@HelloHorizon Hello, I finally solved the problem. First of all, I always had a misunderstanding about Aluxio. I always thought that the port number of 19998 was Worker (I was very surprised why I formed such an impression, and it was so deep that I didn’t even doubt that it was a wrong understanding), so my command to submit the task was as follows:

hadoop jar ~/app/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.7.jar  wordcount -libjars /home/hadoop/jars/alluxio/alluxio-2.6.0-client.jar alluxio://alluxio01:19998/wordcount/input.txt alluxio://alluxio01:19998/wordcount/output

But there are still errors, and I still can’t recognize the error of alluxio scheme, so I still add the following configuration to core-site.xml:

<property>
  <name>fs.alluxio.impl</name>
  <value>alluxio.hadoop.FileSystem</value>
  <description>The Alluxio FileSystem</description>
</property>

Obviously I’m using a 2.x version of Hadoop, and the official documentation says that you don’t need to add this configuration for the 2.x version, but in my case it does.

In the end, thanks a lot anyway!

[HelloHorizon]

@xpleaf Glad you solve the issue. Yeah, we use 19999 as web UI port and 19998 to server master RPCs.