Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Feature request: a public dataset

See original GitHub issue

Similar to df = dd.read_parquet('gs://dask-nyc-taxi/yellowtrip.parquet', storage_options={'token': 'anon'}).

Something like df = dd.read_parquet('az://yellowtrip.parquet', storage_options={'account_name': 'dask-nyc-taxi'})

cc. @martindurant @TomAugspurger

Issue Analytics

State:
Created 3 years ago
Comments:7 (5 by maintainers)

Top GitHub Comments

2reactions

lostmygithubaccountcommented, Sep 29, 2020

for Open Datasets it is @meyetman

you can access via adlfs with the anonymous access:

from adlfs import AzureBlobFileSystem

storage_options = {'account_name': 'azureopendatastorage'}

fs = AzureBlobFileSystem(**storage_options)
fs.ls('isdweatherdatacontainer')

1reaction

TomAugspurgercommented, Jan 26, 2022

@rabernat you provide the “storage account name” which is the https://<storage-account-name>.blob.windows.net part. The “storage container” is in the path. It’s a bit confusing at first:

In [1]: import adlfs

In [2]: fs = adlfs.AzureBlobFileSystem("ai4edataeuwest")

In [3]: fs.ls("/ecmwf")
Out[3]:
['ecmwf/20220121',
 'ecmwf/20220122',
 'ecmwf/20220123',
 'ecmwf/20220124',
 'ecmwf/20220125',
 'ecmwf/20220126']