question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[alluxio-2.0-rc2]workers failed to start if config with multiple value to dirs.path

See original GitHub issue

Alluxio Version: alluxio-2.0-rc2

Describe the bug workers failed to start if config with multiple value to dirs.path

To Reproduce

config a worker with below configurations, then try to start worker with alluxio-start.sh script alluxio.worker.tieredstore.level1.dirs.path=/mnt/ssd1,/mnt/ssd2,/mnt/ssd3

Expected behavior

worker should started properly

Urgency high

Additional context

#alluxio-start.sh worker Mount
Killed 0 processes on xxx.xx.x
Ramdisk not detected. Mounting...
Formatting RamFS: /mnt/ramdisk (42949672960)
Starting worker @ xxx.xx.x. Logging to /var/log/alluxio/
--- [ FAILED ] The worker @ xxx.xx.x is not serving requests.
--- Printing the log tail for /var/log/alluxio//worker.log
>>> BEGIN
2019-06-13 11:28:48,014 INFO  ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2019-06-13 11:28:48,014 INFO  ZooKeeper - Client environment:host.name=xxx.xx.x
2019-06-13 11:28:48,014 INFO  ZooKeeper - Client environment:java.version=1.8.0_212
2019-06-13 11:28:48,014 INFO  ZooKeeper - Client environment:java.vendor=Oracle Corporation
2019-06-13 11:28:48,014 INFO  ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/jre
2019-06-13 11:28:48,014 INFO  ZooKeeper - Client environment:java.class.path=/usr/local/alluxio-2.0.0-RC2/conf/::/usr/local/alluxio-2.0.0-RC2/assembly/alluxio-server-2.0.0-RC2.jar
2019-06-13 11:28:48,014 INFO  ZooKeeper - Client environment:java.library.path=:/usr/local/hadoop-2.7.2/lib/native:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/lib/amd64/server:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-06-13 11:28:48,014 INFO  ZooKeeper - Client environment:java.io.tmpdir=/tmp
2019-06-13 11:28:48,014 INFO  ZooKeeper - Client environment:java.compiler=<NA>
2019-06-13 11:28:48,014 INFO  ZooKeeper - Client environment:os.name=Linux
2019-06-13 11:28:48,014 INFO  ZooKeeper - Client environment:os.arch=amd64
2019-06-13 11:28:48,014 INFO  ZooKeeper - Client environment:os.version=5.0.10-1.el7.elrepo.x86_64
2019-06-13 11:28:48,014 INFO  ZooKeeper - Client environment:user.name=root
2019-06-13 11:28:48,015 INFO  ZooKeeper - Client environment:user.home=/root
2019-06-13 11:28:48,015 INFO  ZooKeeper - Client environment:user.dir=/root
2019-06-13 11:28:48,015 INFO  ZooKeeper - Initiating client connection, connectString=AA:2181,BB:2181,CC:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@3c0f93f1
2019-06-13 11:28:48,032 INFO  ClientCnxn - Opening socket connection to server xxx.xx.x/xxx.xx.x:2181. Will not attempt to authenticate using SASL (unknown error)
2019-06-13 11:28:48,033 INFO  CuratorFrameworkImpl - Default schema
2019-06-13 11:28:48,038 INFO  ClientCnxn - Socket connection established to xxx.xx.x/xxx.xx.x:2181, initiating session
2019-06-13 11:28:48,043 INFO  ClientCnxn - Session establishment complete on server xxx.xx.x/xxx.xx.x:2181, sessionid = 0x1000000f43a0030, negotiated timeout = 40000
2019-06-13 11:28:48,050 INFO  ConnectionStateManager - State change: CONNECTED
2019-06-13 11:28:48,505 INFO  ConfigurationUtils - Alluxio client has loaded configuration from meta master xxx.xx.x/xxx.xx.x:19998
2019-06-13 11:28:48,510 INFO  ConfigurationUtils - Alluxio client (version 2.0.0-RC2) is trying to load cluster level configurations
2019-06-13 11:28:48,528 INFO  ConfigurationUtils - Alluxio client has loaded cluster level configurations
2019-06-13 11:28:48,547 INFO  TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=xxx.xx.x, rack=null)
2019-06-13 11:28:48,748 INFO  BlockWorkerFactory - Creating alluxio.worker.block.BlockWorker
2019-06-13 11:28:48,748 INFO  FileSystemWorkerFactory - Creating alluxio.worker.file.FileSystemWorker
2019-06-13 11:28:48,776 INFO  StorageDir - Folder /mnt/ramdisk/alluxioworker was created!
2019-06-13 11:28:48,791 WARN  StorageTier - Failed to verify memory capacity
2019-06-13 11:28:48,793 INFO  StorageDir - Folder /mnt/ssd1/alluxioworker was created!
<<< EOF
--- Printing the log tail for /var/log/alluxio//worker.out
>>> BEGIN
	at alluxio.worker.WorkerProcess$Factory.create(WorkerProcess.java:38)
	at alluxio.worker.AlluxioWorker.main(AlluxioWorker.java:68)
Caused by: java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException: 1
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at alluxio.util.CommonUtils.invokeAll(CommonUtils.java:493)
	at alluxio.worker.AlluxioWorkerProcess.<init>(AlluxioWorkerProcess.java:118)
	... 3 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
	at alluxio.worker.block.meta.StorageTier.initStorageTier(StorageTier.java:95)
	at alluxio.worker.block.meta.StorageTier.newStorageTier(StorageTier.java:186)
	at alluxio.worker.block.BlockMetadataManager.<init>(BlockMetadataManager.java:68)
	at alluxio.worker.block.BlockMetadataManager.createBlockMetadataManager(BlockMetadataManager.java:83)
	at alluxio.worker.block.TieredBlockStore.<init>(TieredBlockStore.java:126)
	at alluxio.worker.block.DefaultBlockWorker.<init>(DefaultBlockWorker.java:134)
	at alluxio.worker.block.BlockWorkerFactory.create(BlockWorkerFactory.java:43)
	at alluxio.worker.block.BlockWorkerFactory.create(BlockWorkerFactory.java:26)
	at alluxio.worker.AlluxioWorkerProcess.lambda$new$0(AlluxioWorkerProcess.java:110)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Heap
 par new generation   total 613440K, used 229045K [0x00000005c0000000, 0x00000005e9990000, 0x00000005e9990000)
  eden space 545344K,  42% used [0x00000005c0000000, 0x00000005cdfad660, 0x00000005e1490000)
  from space 68096K,   0% used [0x00000005e1490000, 0x00000005e1490000, 0x00000005e5710000)
  to   space 68096K,   0% used [0x00000005e5710000, 0x00000005e5710000, 0x00000005e9990000)
 concurrent mark-sweep generation total 3512768K, used 0K [0x00000005e9990000, 0x00000006c0000000, 0x00000007c0000000)
 Metaspace       used 21897K, capacity 22156K, committed 22292K, reserved 1069056K
  class space    used 2610K, capacity 2704K, committed 2760K, reserved 1048576K
<<< EOF

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:8 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
ns1123commented, Jun 13, 2019

Hi Frank,

Do you have a tiered store configured for level0? Our indices start with 0 so changing the property to alluxio.worker.tieredstore.level0.dirs.path should fix it. To have two tiers, you need to set alluxio.worker.tieredstore.levels=2. Then you could set alluxio.worker.tieredstore.level1.dirs.path to configure your second tier.

0reactions
gpangcommented, Aug 14, 2019

This was an issue with worker registration. This was fixed by this PR: https://github.com/Alluxio/alluxio/pull/9517

Read more comments on GitHub >

github_iconTop Results From Across the Web

error: cannot overwrite multiple values with a single value
first steps are to make a backup of current settings. The simplest way is using these listings, then copy paste results. git config...
Read more >
Multiple values in solrconfig.xml cause error with solr 4.5+
Starting with 4.5, configuration parsing will fail with an error in ... such as "solrconfig.xml contains more than one value for config path:...
Read more >
unset-all name [value-pattern] git config [<file-option>]
When writing, the new value is written to the repository local configuration file by default, and options --system , --global , --worktree ,...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found