Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

How to avoid the auto convert variable dtype from float32 to float64 when read netCDF file use open_dataset?

See original GitHub issue

when i read some netCDF4 file using xr.open_dataset, it seems this method would auto convert the variable type from float32 to float64, how to avoid it ?

Use xarry.open_dataset

import xarray as xr
xr.open_dataset('cat.20151003200633.nc')

will yield output as follow

<xarray.Dataset>
Dimensions:  (x: 461, y: 461, z: 9)
Coordinates:
  * x        (x) float32 -230.0 -229.0 -228.0 -227.0 -226.0 -225.0 -224.0 ...
  * y        (y) float32 -230.0 -229.0 -228.0 -227.0 -226.0 -225.0 -224.0 ...
  * z        (z) float32 0.4 1.4 2.3 3.2 4.3 5.8 9.7 14.5 19.3
Data variables:
    dbz      (z, y, x) float64 nan nan nan nan nan nan nan nan nan nan nan ...
    vr       (z, y, x) float64 nan nan nan nan nan nan nan nan nan nan nan ...
    sw       (z, y, x) float64 nan nan nan nan nan nan nan nan nan nan nan ...

Variables dtype of dbz, vr and sw in this file have been convert to float64, which actually is float32.

Use netCDF4.Dataset

import netCDF4 as ncf
ncf.Dataset('cat.20151003200633.nc')

will yield output as follow

   <type 'netCDF4._netCDF4.Dataset'>
    root group (NETCDF4 data model, file format HDF5): 
    ..........
    dimensions(sizes): x(461), y(461), z(9)
    variables(dimensions): float32 x(x), float32 y(y), float32 z(z), float32 dbz(z,y,x), float32 vr(z,y,x), float32 sw(z,y,x)
    groups:

The netCDF4.Dataset produce the right variable type, while the xarray.open_dataset not.

Issue Analytics

State:
Created 7 years ago
Comments:6 (3 by maintainers)

Top GitHub Comments

1reaction

shoyercommented, Mar 28, 2018

Yes, this was fixed by @Zac-HD in https://github.com/pydata/xarray/pull/1840

1reaction

shoyercommented, Sep 20, 2016

OK, that makes sense. I agree, we could keep such arrays as float32. We don’t need to guard against overflow when only decoding a _FillValue.

If you’re interested in taking a look at a fix, this is where the current logic is: https://github.com/pydata/xarray/blob/551a7bca7b42a6d6db976b01d2eee1131735785d/xarray/conventions.py#L795-L802

Top Results From Across the Web

Reading and writing files - Xarray

This is important when converting floating point with missing values to integers on disk, because NaN is not a valid value for integer...

netCDF4 API documentation

The variable primitive datatypes correspond to the dtype attribute of a numpy array. You can specify the datatype as a numpy dtype object,...

How to convert dtype of time axis from float32 to datetime64 ...

I have separate monthly netcdf files from 1940-2021 and I want to merge them in one single file. But the issue is that...

Serialization and IO — xray 0.5.0 documentation

You are not reading the most recent version of this documentation. v2022.12.0 is ... We can load netCDF files to create a new...

Variable Type changing when using Xarray - Data - Pangeo

In this specific example, the data values change from float32 to float64, ... import xarray as xr data = xr.open_dataset('test_slice.nc4') print(data) ...