Full Support for Kerberos secured Hadoop Cluster

Hello everyone, Recently I tried to set up petastorm on my company’s hadoop cluster. However as the cluster uses Kerberos for authentication using petastorm failed. I figured out that petastorm relies on pyarrow which actually supports kerberos authentication.

I hacked “petastorm/petastorm/hdfs/namenode.py” line 250 and replaced it with

driver = 'libhdfs'
return pyarrow.hdfs.connect("cluserurl", clusterport, user=usernamer, kerb_ticket=ticket_cache_path, driver=driver)

This setup actually made it possible to work on a kerberos secured cluster. For future releases a better support in petastorm for kerberos would be great as the depending pyarrow perfectly works fine with kerberos.

I also used libhdfs instead of libhdfs3 because libhdfs3 doesn’t work for me.

Kind regards, Jonathan

[selitvin]

Jonathan,

Thanks for the suggestion. Let’s make get the kerberos authentication supported then.

[cupdike]

I was able to get past this error. Seems like it was actually caused by the file supplied to the kerbTicketCachePath not actually being at the specified location. I was using the --files option with spark-submit when it should have been --py-files.