question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Chunking works only with dask installed

See original GitHub issue

I am not sure if this is a bug and, if so, of what.

>>> path = "tests/data/S1B_IW_SLC__1SDV_20210401T052622_20210401T052650_026269_032297_EFA4.SAFE/manifest.safe"
>>> group = 'IW1/R168-N471-E0118'
>>> xs.sentinel1.open_dataset(path, group=group)
<xarray.Dataset>
Dimensions:           (azimuth_time: 1501, slant_range_time: 21632)
Coordinates:
    line              (azimuth_time) float64 0.5 1.5 2.5 ... 1.5e+03 1.5e+03
    pixel             (slant_range_time) float64 0.5 1.5 ... 2.163e+04 2.163e+04
  * azimuth_time      (azimuth_time) datetime64[ns] 2021-04-01T05:26:24.20999...
  * slant_range_time  (slant_range_time) float64 0.005343 0.005343 ... 0.005679
Data variables:
    VH                (azimuth_time, slant_range_time) complex128 ...
    VV                (azimuth_time, slant_range_time) complex128 ...
Attributes: (12/14)
    constellation:              sentinel-1
    platform:                   sentinel-1b
    instrument:                 ['c-sar']
    sat:orbit_state:            descending
    sat:absolute_orbit:         26269
    sat:relative_orbit:         168
    ...                         ...
    sar:instrument_mode:        IW
    sar:polarizations:          ['VV', 'VH']
    sar:product_type:           SLC
    xs:instrument_mode_swaths:  ['IW1', 'IW2', 'IW3']
    Conventions:                CF-1.7
    history:                    created by xarray_sentinel-999

>>> xs.sentinel1.open_dataset(path, group=group, chunks=1000)
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-5-79175c680e53> in <module>
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-7-90f5963f6ea0> in <module>
----> 1 xs.sentinel1.open_dataset(path, group=group, chunks=1000)

~\xarray-sentinel\xarray_sentinel\sentinel1.py in open_dataset(product_urlpath, drop_variables, group, chunks, fs)
    384         else:
    385             annotation_path = list(groups[group]["annotation_path"].values())[0]
--> 386             ds = open_burst_dataset(
    387                 manifest_path,
    388                 measurement_paths=groups[group]["measurement_path"],

~\xarray-sentinel\xarray_sentinel\sentinel1.py in open_burst_dataset(manifest_path, burst_position, measurement_paths, annotation_path, chunks)
    297     data_vars = {}
    298     for pol, data_path in measurement_paths.items():
--> 299         arr = rioxarray.open_rasterio(data_path, chunks=chunks)
    300
    301         arr = arr.squeeze("band").drop_vars(["band", "spatial_ref"])

~\miniconda3\envs\xr-sentinel\lib\site-packages\rioxarray\_io.py in open_rasterio(filename, parse_coordinates, chunks, cache, lock, masked, mask_and_scale, variable, group, default_name, **open_kwargs)
    863
    864     if chunks is not None:
--> 865         result = _prepare_dask(result, riods, filename, chunks)
    866
    867     # Make the file closeable

~\miniconda3\envs\xr-sentinel\lib\site-packages\rioxarray\_io.py in _prepare_dask(result, riods, filename, chunks)
    573     """
    574     # pylint: disable=import-outside-toplevel
--> 575     from dask.base import tokenize
    576
    577     # augment the token with the file modification time

ModuleNotFoundError: No module named 'dask'

Issue Analytics

  • State:open
  • Created 2 years ago
  • Comments:5 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
aurghscommented, Apr 28, 2021
  1. we could add an xfails on the tests with dask
  2. I agree
  3. I think @corrado9999 was preparing the PR.
0reactions
alexamicicommented, Apr 28, 2021
  1. we do install dask for testing already, no need to test without dask, we don’t have any dask related logic to test. (I propose no action)
  2. dask is not a needed dependency so it must not be listed in the install_requires (no action)
  3. we agree, @aurghs would you please take care of it?
Read more comments on GitHub >

github_iconTop Results From Across the Web

Chunks - Dask documentation
However, data stores often chunk more finely than is ideal for Dask array, so it is common to choose a chunking that is...
Read more >
Choosing good chunk sizes in Dask
First, start by choosing a chunk size similar to data you know can be processed entirely within memory (i.e. without Dask), using these...
Read more >
Best Practices - Dask documentation
A common performance problem among Dask Array users is that they have chosen a chunk size that is either too small (leading to...
Read more >
Slicing - Dask documentation
If we slice that with a sorted sequence of integers, Dask will return one chunk per input chunk (notice the output chunksize is...
Read more >
dask.array.rechunk - Dask documentation
dask.array.rechunk¶ ... Convert blocks in dask array x for new chunks. ... If True, try to make each chunk to be the same...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found