to_netcdf() fails to append to an existing file

The following code used to work well in v0.8.2:

import os
import xarray as xr

path = 'test.nc'
if os.path.exists(path):
    os.remove(path)
    
ds = xr.Dataset()
ds['dim'] = ('dim', [0, 1, 2])
ds['var1'] = ('dim', [10, 11, 12])
ds.to_netcdf(path)

ds = xr.Dataset()
ds['dim'] = ('dim', [0, 1, 2])
ds['var2'] = ('dim', [10, 11, 12])
ds.to_netcdf(path, 'a')

On master, it fails with:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-fce5f5e876aa> in <module>()
     14 ds['dim'] = ('dim', [0, 1, 2])
     15 ds['var2'] = ('dim', [10, 11, 12])
---> 16 ds.to_netcdf(path, 'a')

/home/mowglie/Documents/git/xarray/xarray/core/dataset.py in to_netcdf(self, path, mode, format, group, engine, encoding)
    927         from ..backends.api import to_netcdf
    928         return to_netcdf(self, path, mode, format=format, group=group,
--> 929                          engine=engine, encoding=encoding)
    930 
    931     def __unicode__(self):

/home/mowglie/Documents/git/xarray/xarray/backends/api.py in to_netcdf(dataset, path, mode, format, group, engine, writer, encoding)
    563     store = store_cls(path, mode, format, group, writer)
    564     try:
--> 565         dataset.dump_to_store(store, sync=sync, encoding=encoding)
    566         if isinstance(path, BytesIO):
    567             return path.getvalue()

/home/mowglie/Documents/git/xarray/xarray/core/dataset.py in dump_to_store(self, store, encoder, sync, encoding)
    873             variables, attrs = encoder(variables, attrs)
    874 
--> 875         store.store(variables, attrs, check_encoding)
    876         if sync:
    877             store.sync()

/home/mowglie/Documents/git/xarray/xarray/backends/common.py in store(self, variables, attributes, check_encoding_set)
    219         cf_variables, cf_attrs = cf_encoder(variables, attributes)
    220         AbstractWritableDataStore.store(self, cf_variables, cf_attrs,
--> 221                                         check_encoding_set)
    222 
    223 

/home/mowglie/Documents/git/xarray/xarray/backends/common.py in store(self, variables, attributes, check_encoding_set)
    194     def store(self, variables, attributes, check_encoding_set=frozenset()):
    195         self.set_attributes(attributes)
--> 196         self.set_variables(variables, check_encoding_set)
    197 
    198     def set_attributes(self, attributes):

/home/mowglie/Documents/git/xarray/xarray/backends/common.py in set_variables(self, variables, check_encoding_set)
    204             name = _encode_variable_name(vn)
    205             check = vn in check_encoding_set
--> 206             target, source = self.prepare_variable(name, v, check)
    207             self.writer.add(source, target)
    208 

/home/mowglie/Documents/git/xarray/xarray/backends/netCDF4_.py in prepare_variable(self, name, variable, check_encoding)
    293             endian='native',
    294             least_significant_digit=encoding.get('least_significant_digit'),
--> 295             fill_value=fill_value)
    296         nc4_var.set_auto_maskandscale(False)
    297 

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Dataset.createVariable (netCDF4/_netCDF4.c:18740)()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.__init__ (netCDF4/_netCDF4.c:30713)()

RuntimeError: NetCDF: String match to name in use

[shoyer]

Is it now possible to append to a netCDF file using xarray?

No, it is not. This issue is about appending new variables to an existing netCDF file.

I think what you are looking for is to append along existing dimensions to a netCDF file. This is possible in the netCDF data model, but not yet supported by xarray. See https://github.com/pydata/xarray/issues/1398 for some discussion.

For these types of use cases, I would generally recommend writing a new netCDF file, and then loading everything afterwards using xarray.open_mfdataset.

[shoyer]

I will give xarray.open_mfdataset a shot. Just one question - is this approach memory conservative? My reasoning for chunking in the first place is large file size.

Yes, open_mfdataset uses dask, which allows for streaming computation.