question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

hvplot is not using lazy loading for zarr-backed xarray dataset

See original GitHub issue

ALL software version info

xr.__version__
'0.21.1'
hvplot.__version__
'0.7.3'
holoviews.__version__
'1.14.7'

Description of expected behavior and the observed behavior

When I visualize zarr-backed xarray datasets with hvplot, I expect the loading to be lazy. I don’t want to load the entire array into memory. I expect this to happen even without dask being involved.

Instead, I am noticing that hvplot is loading data eagerly.

Complete, minimal, self-contained example code that reproduces the issue

import xarray as xr
import hvplot.xarray

url = "https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/swot_adac/FESOM/surf/fma.zarr"
ds = xr.open_dataset(url, engine='zarr') # note that ds is not chunked but still uses lazy loading

# this will trigger loading the whole array into memory
ds.sst.hvplot(x='lon', y='lat', dynamic=True, rasterize=True)

Note that lazyness can be achieved by chunking the xarray dataset. But I don’t want to have to do that. Dask introduces additional latency that I would like to avoid here.

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:19 (12 by maintainers)

github_iconTop GitHub Comments

1reaction
philippjfrcommented, Apr 19, 2022

I’m on xarray 2022.3.0 and this is sufficient to reproduce:

import xarray as xr

url = "https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/swot_adac/FESOM/surf/fma.zarr"
ds = xr.open_dataset(url, engine='zarr') # note that ds is not chunked but still uses lazy loading
ds.chunks
1reaction
rabernatcommented, Feb 18, 2022

There is a time dimension.

ds.isel(time=0)

will lazily grab a single time step, which is all that should be required for a single image plot.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Gridded Data — hvPlot 0.8.2 documentation
This user guide will cover how to leverage xarray and hvplot to visualize and explore data of different dimensionality ranging from simple 1D...
Read more >
Can I force xarray.open_dataset() to do a lazy load?
A colleague of mine gave me the answer: I needed to pass the option decode_times=False . With this option, xarray.open_dataset() does lazy ......
Read more >
hvplot - bytemeta
Odd behavior with invert_yaxis, invert_xaxis · hvplot is not using lazy loading for zarr-backed xarray dataset.
Read more >
Gprof2dot is not recognized as an internal or external ...
1 dev BSD-3-Clause X Machine learning with dirty categories. · 2 MIT X X X Python 2.7 backport of the "dis" module from...
Read more >
A high-level plotting API for pandas, dask, xarray, and ...
When I visualize zarr-backed xarray datasets with hvplot, I expect the loading to be lazy. I don't want to load the entire array...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found