ERROR AlluxioWorker - Fatal error: Failed to create worker process #14153

Alluxio Version: HDFS3.3.0 + Complied Alluxio

Describe the bug When I start the Alluxio cluster, all the worker nodes fail to start, showing a java initialization error, but I have configured the Java environment variables, and again specified JAVA_HOME and JAVA in alluxio-env.sh. Have you ever encountered such problems? Please help me.

To Reproduce

(base) [root@clu00 bin]# ./alluxio-start.sh all
Assuming NoMount by default.
Executing the following command on all master nodes and logging to /root/alluxio/logs/task.log: /root/alluxio/bin/alluxio-stop.sh proxy
Waiting for tasks to finish...
All tasks finished
Executing the following command on all worker nodes and logging to /root/alluxio/logs/task.log: /root/alluxio/bin/alluxio-stop.sh proxy
Waiting for tasks to finish...
All tasks finished
Executing the following command on all worker nodes and logging to /root/alluxio/logs/task.log: /root/alluxio/bin/alluxio-stop.sh job_worker
Waiting for tasks to finish...
All tasks finished
Executing the following command on all worker nodes and logging to /root/alluxio/logs/task.log: /root/alluxio/bin/alluxio-stop.sh worker
Waiting for tasks to finish...
All tasks finished
Executing the following command on all master nodes and logging to /root/alluxio/logs/task.log: /root/alluxio/bin/alluxio-stop.sh job_master
Waiting for tasks to finish...
All tasks finished
Executing the following command on all master nodes and logging to /root/alluxio/logs/task.log: /root/alluxio/bin/alluxio-stop.sh master
Waiting for tasks to finish...
All tasks finished
Executing the following command on all master nodes and logging to /root/alluxio/logs/task.log: /root/alluxio/bin/alluxio-start.sh -a master
Waiting for tasks to finish...
All tasks finished
Executing the following command on all master nodes and logging to /root/alluxio/logs/task.log: /root/alluxio/bin/alluxio-start.sh -a job_master
Waiting for tasks to finish...
All tasks finished
Executing the following command on all worker nodes and logging to /root/alluxio/logs/task.log: /root/alluxio/bin/alluxio-start.sh -a worker NoMount
Waiting for tasks to finish...
All tasks finished
Executing the following command on all worker nodes and logging to /root/alluxio/logs/task.log: /root/alluxio/bin/alluxio-start.sh -a job_worker
Waiting for tasks to finish...
All tasks finished
Executing the following command on all master nodes and logging to /root/alluxio/logs/task.log: /root/alluxio/bin/alluxio-start.sh -a proxy
Waiting for tasks to finish...
All tasks finished
Executing the following command on all worker nodes and logging to /root/alluxio/logs/task.log: /root/alluxio/bin/alluxio-start.sh -a proxy
Waiting for tasks to finish...
All tasks finished
-----------------------------------------
Starting to monitor all remote services.
-----------------------------------------
--- [ OK ] The master service @ clu00 is in a healthy state.
--- [ OK ] The job_master service @ clu00 is in a healthy state.
--- [ FAILED ] The worker @ clu02 is not serving requests.
--- Printing the log tail for /root/alluxio/logs/worker.log
>>> BEGIN
2021-09-26 22:36:54,162 ERROR AlluxioWorker - Fatal error: Failed to create worker process
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.ExceptionInInitializerError
	at alluxio.worker.AlluxioWorkerProcess.<init>(AlluxioWorkerProcess.java:162)
	at alluxio.worker.WorkerProcess$Factory.create(WorkerProcess.java:46)
	at alluxio.worker.WorkerProcess$Factory.create(WorkerProcess.java:38)
	at alluxio.worker.AlluxioWorker.main(AlluxioWorker.java:72)
Caused by: java.util.concurrent.ExecutionException: java.lang.ExceptionInInitializerError
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at alluxio.util.CommonUtils.invokeAll(CommonUtils.java:593)
	at alluxio.util.CommonUtils.invokeAll(CommonUtils.java:555)
	at alluxio.worker.AlluxioWorkerProcess.<init>(AlluxioWorkerProcess.java:113)
	... 3 more
Caused by: java.lang.ExceptionInInitializerError
	at alluxio.worker.block.DefaultBlockWorker.<init>(DefaultBlockWorker.java:185)
	at alluxio.worker.block.DefaultBlockWorker.<init>(DefaultBlockWorker.java:152)
	at alluxio.worker.block.BlockWorkerFactory.create(BlockWorkerFactory.java:43)
	at alluxio.worker.block.BlockWorkerFactory.create(BlockWorkerFactory.java:26)
	at alluxio.worker.AlluxioWorkerProcess.lambda$new$0(AlluxioWorkerProcess.java:105)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException
	at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1314)
	at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1237)
	at alluxio.worker.grpc.GrpcExecutors.<clinit>(GrpcExecutors.java:46)
	... 9 more

2021-09-26 22:37:41,748 INFO  NettyUtils - EPOLL_MODE is available
<<< EOF
--- Printing the log tail for /root/alluxio/logs/worker.out
    --- EMPTY ---
--- [ FAILED ] The worker @ clu01 is not serving requests.
--- Printing the log tail for /root/alluxio/logs/worker.log
>>> BEGIN
2021-09-26 22:36:53,971 ERROR AlluxioWorker - Fatal error: Failed to create worker process
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.ExceptionInInitializerError
	at alluxio.worker.AlluxioWorkerProcess.<init>(AlluxioWorkerProcess.java:162)
	at alluxio.worker.WorkerProcess$Factory.create(WorkerProcess.java:46)
	at alluxio.worker.WorkerProcess$Factory.create(WorkerProcess.java:38)
	at alluxio.worker.AlluxioWorker.main(AlluxioWorker.java:72)
Caused by: java.util.concurrent.ExecutionException: java.lang.ExceptionInInitializerError
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at alluxio.util.CommonUtils.invokeAll(CommonUtils.java:593)
	at alluxio.util.CommonUtils.invokeAll(CommonUtils.java:555)
	at alluxio.worker.AlluxioWorkerProcess.<init>(AlluxioWorkerProcess.java:113)
	... 3 more
Caused by: java.lang.ExceptionInInitializerError
	at alluxio.worker.block.DefaultBlockWorker.<init>(DefaultBlockWorker.java:185)
	at alluxio.worker.block.DefaultBlockWorker.<init>(DefaultBlockWorker.java:152)
	at alluxio.worker.block.BlockWorkerFactory.create(BlockWorkerFactory.java:43)
	at alluxio.worker.block.BlockWorkerFactory.create(BlockWorkerFactory.java:26)
	at alluxio.worker.AlluxioWorkerProcess.lambda$new$0(AlluxioWorkerProcess.java:105)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException
	at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1314)
	at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1237)
	at alluxio.worker.grpc.GrpcExecutors.<clinit>(GrpcExecutors.java:46)
	... 9 more

2021-09-26 22:37:41,578 INFO  NettyUtils - EPOLL_MODE is available
<<< EOF
--- Printing the log tail for /root/alluxio/logs/worker.out
    --- EMPTY ---
--- [ OK ] The job_worker service @ clu02 is in a healthy state.
--- [ OK ] The job_worker service @ clu01 is in a healthy state.
--- [ OK ] The proxy service @ clu02 is in a healthy state.
--- [ OK ] The proxy service @ clu01 is in a healthy state.
--- [ OK ] The proxy service @ clu00 is in a healthy state.

Additional context alluxio-env.sh

JAVA_HOME=/usr/java/jdk1.8.0_301
JAVA=/usr/java/jdk1.8.0_301/bin/java

alluxio-site.properties

alluxio.master.hostname=clu00
alluxio.master.mount.table.root.ufs=hdfs://clu00:9090/alluxio
alluxio.worker.ramdisk.size=1GB

/etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_301
export JRE_HOME=/usr/java/jdk1.8.0_301/jre
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export HADOOP_HOME=/home/hadoop/hadoop-3.3.0
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

export ALLUXIO_HOME=/root/alluxio
export MAVEN_HOME=/usr/maven/apache-maven-3.8.2

export PATH=.:${HADOOP_HOME}/bin:${JAVA_HOME}/bin:${JRE_HOME}/bin:${MAVEN_HOME}/bin:${ALLUXIO_HOME}/bin:$PATH

[j-buczak]

Thanks, I didn’t check this PR, but this: https://github.com/Alluxio/alluxio/issues/14243#issuecomment-941441855 worked (explicitly set resources limits and requests)

[yyongycy]

BTW, issue with kubernetes should be fixed by : https://github.com/Alluxio/alluxio/commit/fe0420c6c4bc65310637351aae2df3d01ca11ff2