[alluxio-2.0-rc2]workers failed to start if config with multiple value to dirs.path
See original GitHub issueAlluxio Version: alluxio-2.0-rc2
Describe the bug workers failed to start if config with multiple value to dirs.path
To Reproduce
config a worker with below configurations, then try to start worker with alluxio-start.sh script alluxio.worker.tieredstore.level1.dirs.path=/mnt/ssd1,/mnt/ssd2,/mnt/ssd3
Expected behavior
worker should started properly
Urgency high
Additional context
#alluxio-start.sh worker Mount
Killed 0 processes on xxx.xx.x
Ramdisk not detected. Mounting...
Formatting RamFS: /mnt/ramdisk (42949672960)
Starting worker @ xxx.xx.x. Logging to /var/log/alluxio/
--- [ FAILED ] The worker @ xxx.xx.x is not serving requests.
--- Printing the log tail for /var/log/alluxio//worker.log
>>> BEGIN
2019-06-13 11:28:48,014 INFO ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2019-06-13 11:28:48,014 INFO ZooKeeper - Client environment:host.name=xxx.xx.x
2019-06-13 11:28:48,014 INFO ZooKeeper - Client environment:java.version=1.8.0_212
2019-06-13 11:28:48,014 INFO ZooKeeper - Client environment:java.vendor=Oracle Corporation
2019-06-13 11:28:48,014 INFO ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/jre
2019-06-13 11:28:48,014 INFO ZooKeeper - Client environment:java.class.path=/usr/local/alluxio-2.0.0-RC2/conf/::/usr/local/alluxio-2.0.0-RC2/assembly/alluxio-server-2.0.0-RC2.jar
2019-06-13 11:28:48,014 INFO ZooKeeper - Client environment:java.library.path=:/usr/local/hadoop-2.7.2/lib/native:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/lib/amd64/server:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-06-13 11:28:48,014 INFO ZooKeeper - Client environment:java.io.tmpdir=/tmp
2019-06-13 11:28:48,014 INFO ZooKeeper - Client environment:java.compiler=<NA>
2019-06-13 11:28:48,014 INFO ZooKeeper - Client environment:os.name=Linux
2019-06-13 11:28:48,014 INFO ZooKeeper - Client environment:os.arch=amd64
2019-06-13 11:28:48,014 INFO ZooKeeper - Client environment:os.version=5.0.10-1.el7.elrepo.x86_64
2019-06-13 11:28:48,014 INFO ZooKeeper - Client environment:user.name=root
2019-06-13 11:28:48,015 INFO ZooKeeper - Client environment:user.home=/root
2019-06-13 11:28:48,015 INFO ZooKeeper - Client environment:user.dir=/root
2019-06-13 11:28:48,015 INFO ZooKeeper - Initiating client connection, connectString=AA:2181,BB:2181,CC:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@3c0f93f1
2019-06-13 11:28:48,032 INFO ClientCnxn - Opening socket connection to server xxx.xx.x/xxx.xx.x:2181. Will not attempt to authenticate using SASL (unknown error)
2019-06-13 11:28:48,033 INFO CuratorFrameworkImpl - Default schema
2019-06-13 11:28:48,038 INFO ClientCnxn - Socket connection established to xxx.xx.x/xxx.xx.x:2181, initiating session
2019-06-13 11:28:48,043 INFO ClientCnxn - Session establishment complete on server xxx.xx.x/xxx.xx.x:2181, sessionid = 0x1000000f43a0030, negotiated timeout = 40000
2019-06-13 11:28:48,050 INFO ConnectionStateManager - State change: CONNECTED
2019-06-13 11:28:48,505 INFO ConfigurationUtils - Alluxio client has loaded configuration from meta master xxx.xx.x/xxx.xx.x:19998
2019-06-13 11:28:48,510 INFO ConfigurationUtils - Alluxio client (version 2.0.0-RC2) is trying to load cluster level configurations
2019-06-13 11:28:48,528 INFO ConfigurationUtils - Alluxio client has loaded cluster level configurations
2019-06-13 11:28:48,547 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=xxx.xx.x, rack=null)
2019-06-13 11:28:48,748 INFO BlockWorkerFactory - Creating alluxio.worker.block.BlockWorker
2019-06-13 11:28:48,748 INFO FileSystemWorkerFactory - Creating alluxio.worker.file.FileSystemWorker
2019-06-13 11:28:48,776 INFO StorageDir - Folder /mnt/ramdisk/alluxioworker was created!
2019-06-13 11:28:48,791 WARN StorageTier - Failed to verify memory capacity
2019-06-13 11:28:48,793 INFO StorageDir - Folder /mnt/ssd1/alluxioworker was created!
<<< EOF
--- Printing the log tail for /var/log/alluxio//worker.out
>>> BEGIN
at alluxio.worker.WorkerProcess$Factory.create(WorkerProcess.java:38)
at alluxio.worker.AlluxioWorker.main(AlluxioWorker.java:68)
Caused by: java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException: 1
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at alluxio.util.CommonUtils.invokeAll(CommonUtils.java:493)
at alluxio.worker.AlluxioWorkerProcess.<init>(AlluxioWorkerProcess.java:118)
... 3 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
at alluxio.worker.block.meta.StorageTier.initStorageTier(StorageTier.java:95)
at alluxio.worker.block.meta.StorageTier.newStorageTier(StorageTier.java:186)
at alluxio.worker.block.BlockMetadataManager.<init>(BlockMetadataManager.java:68)
at alluxio.worker.block.BlockMetadataManager.createBlockMetadataManager(BlockMetadataManager.java:83)
at alluxio.worker.block.TieredBlockStore.<init>(TieredBlockStore.java:126)
at alluxio.worker.block.DefaultBlockWorker.<init>(DefaultBlockWorker.java:134)
at alluxio.worker.block.BlockWorkerFactory.create(BlockWorkerFactory.java:43)
at alluxio.worker.block.BlockWorkerFactory.create(BlockWorkerFactory.java:26)
at alluxio.worker.AlluxioWorkerProcess.lambda$new$0(AlluxioWorkerProcess.java:110)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Heap
par new generation total 613440K, used 229045K [0x00000005c0000000, 0x00000005e9990000, 0x00000005e9990000)
eden space 545344K, 42% used [0x00000005c0000000, 0x00000005cdfad660, 0x00000005e1490000)
from space 68096K, 0% used [0x00000005e1490000, 0x00000005e1490000, 0x00000005e5710000)
to space 68096K, 0% used [0x00000005e5710000, 0x00000005e5710000, 0x00000005e9990000)
concurrent mark-sweep generation total 3512768K, used 0K [0x00000005e9990000, 0x00000006c0000000, 0x00000007c0000000)
Metaspace used 21897K, capacity 22156K, committed 22292K, reserved 1069056K
class space used 2610K, capacity 2704K, committed 2760K, reserved 1048576K
<<< EOF
Issue Analytics
- State:
- Created 4 years ago
- Comments:8 (6 by maintainers)
Top Results From Across the Web
error: cannot overwrite multiple values with a single value
first steps are to make a backup of current settings. The simplest way is using these listings, then copy paste results. git config...
Read more >Multiple values in solrconfig.xml cause error with solr 4.5+
Starting with 4.5, configuration parsing will fail with an error in ... such as "solrconfig.xml contains more than one value for config path:...
Read more >unset-all name [value-pattern] git config [<file-option>]
When writing, the new value is written to the repository local configuration file by default, and options --system , --global , --worktree ,...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Hi Frank,
Do you have a tiered store configured for level0? Our indices start with 0 so changing the property to alluxio.worker.tieredstore.level0.dirs.path should fix it. To have two tiers, you need to set alluxio.worker.tieredstore.levels=2. Then you could set alluxio.worker.tieredstore.level1.dirs.path to configure your second tier.
This was an issue with worker registration. This was fixed by this PR: https://github.com/Alluxio/alluxio/pull/9517