Unable to connect to skein driver

Hello,

as requested by @jcrist, this issue is on the problem I posted on stack overflow. Not sure if I should copy-paste the entire thing here.

However, the environment I am trying to get dask-yarn/skein working on is a 3 Node EMR v.5.21.0 cluster.

Kind regards Vadim

---

Edit: copied from stackoverflow below:

I am trying to get dask-yarn running. I figured out that, the skein module is used for submitting applications to the YARN cluster and it is failing to do so.

import skein
c = skein.Client()

This code fails with:

raise ConnectionError("Unable to connect to %s" % self._server_name)
skein.exceptions.ConnectionError: Unable to connect to driver

Starting the driver from CLI with skein driver start --log-level debug --log log.txt produces the following log while printing localhost:someport to the console:

19/04/18 10:34:04 DEBUG skein.Driver: Logging in using ticket cache
19/04/18 10:34:05 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
19/04/18 10:34:05 INFO client.RMProxy: Connecting to ResourceManager at ip/ip:8032
19/04/18 10:34:06 INFO skein.Driver: Driver started, listening on 36963
19/04/18 10:34:06 DEBUG skein.Driver: Reporting gRPC server port back to 
the launching process

Stopping the driver or trying to get an empty list of running applications writes Error: Unable to connect to driver into the shell.

Some time ago dask-yarn was based on knit, so I tried that one out for applications submission. In knit I successfully submitted some hello world app, but with some manual configuration of the name node/resource manager in the client. Autodetection would not work. Maybe it is a similar problem? The skein driver is written in Java and I can’t find a way to configure it the same way as knit.

Thanks

[ottomata]

AH! OK. The issue was on my end. I had an $https_proxy env var set, which was causing skein to not be able to reach the driver on localhost!

[ottomata]

@jcrist any thoughts? Thanks!