Load data failed due to java.net.UnknownHostException

Alluxio Version: 2.0.0

Describe the bug

Using /opt/alluxio/bin/alluxio fs -Dalluxio.user.ufs.block.read.location.policy=alluxio.client.block.policy.DeterministicHashPolicy load /training-data/images/ to load data from the under storage system to alluxio system, it’s failed and report

WARNING: [io.grpc.internal.ManagedChannelImpl-235] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host iZhp3bku0ru8vuxq08lorxZ, cause=java.lang.RuntimeException: java.net.UnknownHostException: iZhp3bku0ru8vuxq08lorxZ
	at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
	at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
	at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: iZhp3bku0ru8vuxq08lorxZ
	at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
	at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
	at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
	... 5 more

To Reproduce

1. Deploy the alluxio cluster successfully, and check the status. It looks good.

# /opt/alluxio/bin/alluxio fsadmin report
Alluxio cluster summary:
    Master Address: alluxio-master:19998
    Web Port: 19999
    Rpc Port: 19998
    Started: 06-30-2019 09:38:27:895
    Uptime: 2 day(s), 16 hour(s), 33 minute(s), and 45 second(s)
    Version: 2.0.0
    Safe Mode: false
    Zookeeper Enabled: false
    Live Workers: 4
    Lost Workers: 0
    Total Capacity: 2400.00GB
        Tier: MEM  Size: 400.00GB
        Tier: SSD  Size: 2000.00GB
    Used Capacity: 9.90GB
        Tier: MEM  Size: 9.90GB
        Tier: SSD  Size: 0B
    Free Capacity: 2390.10GB

1. And check the capacity, the worker is already registered.

/opt/alluxio/bin/alluxio fsadmin report capacity
Capacity information for all workers:
    Total Capacity: 2400.00GB
        Tier: MEM  Size: 400.00GB
        Tier: SSD  Size: 2000.00GB
    Used Capacity: 9.90GB
        Tier: MEM  Size: 9.90GB
        Tier: SSD  Size: 0B
    Used Percentage: 0%
    Free Percentage: 100%

Worker Name                  Last Heartbeat   Storage       Total            MEM           SSD
iZhp3bku0ru8vuxq08loruZ      0                capacity      600.00GB         100.00GB      500.00GB
                                              used          0B (0%)          0B            0B
iZhp3bku0ru8vuxq08lorvZ      0                capacity      600.00GB         100.00GB      500.00GB
                                              used          0B (0%)          0B            0B
iZhp3bku0ru8vuxq08lorxZ      0                capacity      600.00GB         100.00GB      500.00GB
                                              used          0B (0%)          0B            0B
iZhp3bku0ru8vuxq08lorwZ      0                capacity      600.00GB         100.00GB      500.00GB
                                              used          9.90GB (1%)      9.90GB        0B

1. And run load data

# /opt/alluxio/bin/alluxio fs -Dalluxio.user.ufs.block.read.location.policy=alluxio.client.block.policy.DeterministicHashPolicy  load /training-data/images/
Jul 03, 2019 12:14:37 AM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
WARNING: [io.grpc.internal.ManagedChannelImpl-137] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host iZhp3bku0ru8vuxq08loruZ, cause=java.lang.RuntimeException: java.net.UnknownHostException: iZhp3bku0ru8vuxq08loruZ
	at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
	at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
	at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: iZhp3bku0ru8vuxq08loruZ
	at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
	at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
	at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
	... 5 more
}

Expected behavior I think the worker is already registered into the alluxio master, it’s no need to use DNS to discover it. Can you enhance it? Thanks.

Urgency Describe the impact and urgency of the bug.

Additional context Add any other context about the problem here.

[madanadit]

@cheyang can you update the job worker JAVA_OPTS from what you had before to

- name: ALLUXIO_JOB_WORKER_JAVA_OPTS
   value: " -Dalluxio.worker.hostname=$(ALLUXIO_WORKER_HOSTNAME) "

seems like the job worker ignores alluxio.job.worker.hostname and uses alluxio.worker.hostname instead.

i’ll fix the k8s spec to use ALLUXIO_JOB_WORKER_JAVA_OPTS and the job worker code to respect alluxio.job.worker.hostname. thx again!

[apc999]

Great! Close this issue now