Raise nice error when attempting to concatenate CFTimeIndex & DatetimeIndex

MCVE Code Sample

import subprocess
import sys
import wget
import glob

def install(package):
    subprocess.check_call([sys.executable, "-m", "pip", "install", package])
try:
    from xclim import ensembles
except:
    install('xclim')
    from xclim import ensembles

outdir = 'tmp'
url = []
url.append('https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/EnsembleStats/BCCAQv2+ANUSPLIN300_ACCESS1-0_historical+rcp45_r1i1p1_1950-2100_tg_mean_YS.nc')
url.append('https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/EnsembleStats/BCCAQv2+ANUSPLIN300_BNU-ESM_historical+rcp45_r1i1p1_1950-2100_tg_mean_YS.nc')
url.append('https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/EnsembleStats/BCCAQv2+ANUSPLIN300_CCSM4_historical+rcp45_r1i1p1_1950-2100_tg_mean_YS.nc')
url.append('https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/EnsembleStats/BCCAQv2+ANUSPLIN300_CCSM4_historical+rcp45_r2i1p1_1950-2100_tg_mean_YS.nc')
for u in url:
    wget.download(u,out=outdir)
datasets = glob.glob(f'{outdir}/*1950*.nc')
ens1 = ensembles.create_ensemble(datasets)
print(ens1)

Expected Output

Following advice of @dcherian (https://github.com/Ouranosinc/xclim/issues/281#issue-508073942) we have started testing builds of xclim against the master branch as well as the current release:

Using xarray 0.14.1 via pip the above code generates a concatenated dataset with new added dimension ‘realization’

Problem Description

using xarray@master the xclim.ensembles.create_ensemble call gives the following error:

Traceback (most recent call last):
  File "/home/travis/.PyCharmCE2019.3/config/scratches/scratch_26.py", line 23, in <module>
    ens1 = ensembles.create_ensemble(datasets)
  File "/home/travis/github_xclim/xclim/xclim/ensembles.py", line 83, in create_ensemble
    data = xr.concat(list1, dim=dim)
  File "/home/travis/.conda/envs/xclim_dev/lib/python3.7/site-packages/xarray/core/concat.py", line 135, in concat
    return f(objs, dim, data_vars, coords, compat, positions, fill_value, join)
  File "/home/travis/.conda/envs/xclim_dev/lib/python3.7/site-packages/xarray/core/concat.py", line 439, in _dataarray_concat
    join=join,
  File "/home/travis/.conda/envs/xclim_dev/lib/python3.7/site-packages/xarray/core/concat.py", line 303, in _dataset_concat
    *datasets, join=join, copy=False, exclude=[dim], fill_value=fill_value
  File "/home/travis/.conda/envs/xclim_dev/lib/python3.7/site-packages/xarray/core/alignment.py", line 298, in align
    index = joiner(matching_indexes)
  File "/home/travis/.conda/envs/xclim_dev/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 2385, in __or__
    return self.union(other)
  File "/home/travis/.conda/envs/xclim_dev/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 2517, in union
    return self._union_incompatible_dtypes(other, sort=sort)
  File "/home/travis/.conda/envs/xclim_dev/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 2436, in _union_incompatible_dtypes
    return Index.union(this, other, sort=sort).astype(object, copy=False)
  File "/home/travis/.conda/envs/xclim_dev/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 2517, in union
    return self._union_incompatible_dtypes(other, sort=sort)
....
  File "/home/travis/.conda/envs/xclim_dev/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 498, in __new__
    return DatetimeIndex(subarr, copy=copy, name=name, **kwargs)
  File "/home/travis/.conda/envs/xclim_dev/lib/python3.7/site-packages/pandas/core/indexes/datetimes.py", line 334, in __new__
    int_as_wall_time=True,
  File "/home/travis/.conda/envs/xclim_dev/lib/python3.7/site-packages/pandas/core/arrays/datetimes.py", line 446, in _from_sequence
    int_as_wall_time=int_as_wall_time,
  File "/home/travis/.conda/envs/xclim_dev/lib/python3.7/site-packages/pandas/core/arrays/datetimes.py", line 1854, in sequence_to_dt64ns
    data, copy = maybe_convert_dtype(data, copy)
  File "/home/travis/.conda/envs/xclim_dev/lib/python3.7/site-packages/pandas/core/arrays/datetimes.py", line 2060, in maybe_convert_dtype
    elif is_extension_type(data) and not is_datetime64tz_dtype(data):
  File "/home/travis/.conda/envs/xclim_dev/lib/python3.7/site-packages/pandas/core/dtypes/common.py", line 1734, in is_extension_type
    if is_categorical(arr):
  File "/home/travis/.conda/envs/xclim_dev/lib/python3.7/site-packages/pandas/core/dtypes/common.py", line 387, in is_categorical
    return isinstance(arr, ABCCategorical) or is_categorical_dtype(arr)
  File "/home/travis/.conda/envs/xclim_dev/lib/python3.7/site-packages/pandas/core/dtypes/common.py", line 708, in is_categorical_dtype
    return CategoricalDtype.is_dtype(arr_or_dtype)
  File "/home/travis/.conda/envs/xclim_dev/lib/python3.7/site-packages/pandas/core/dtypes/base.py", line 256, in is_dtype
    if isinstance(dtype, (ABCSeries, ABCIndexClass, ABCDataFrame, np.dtype)):
  File "/home/travis/.conda/envs/xclim_dev/lib/python3.7/site-packages/pandas/core/dtypes/generic.py", line 9, in _check
    return getattr(inst, attr, "_typ") in comp
RecursionError: maximum recursion depth exceeded while calling a Python object

Output of xr.show_versions()

INSTALLED VERSIONS

commit: None python: 3.7.5 (default, Oct 25 2019, 15:51:11) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 4.15.0-74-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_CA.UTF-8 LOCALE: en_CA.UTF-8 libhdf5: 1.10.4 libnetcdf: 4.6.3 xarray: 0.14.1+37.gdb36c5c0 pandas: 0.25.3 numpy: 1.17.4 scipy: 1.3.1 netCDF4: 1.5.3 pydap: None h5netcdf: 0.7.4 h5py: 2.9.0 Nio: None zarr: None cftime: 1.0.4.2 nc_time_axis: None PseudoNetCDF: None rasterio: 1.1.1 cfgrib: None iris: None bottleneck: 1.3.1 dask: 2.6.0 distributed: 2.6.0 matplotlib: 3.1.1 cartopy: None seaborn: None numbagg: None setuptools: 41.6.0.post20191030 pip: 19.3.1 conda: None pytest: 5.2.2 IPython: None sphinx: None

[dcherian]

@tlogan specifying use_cftime as True or False in the open_dataset call might be useful.

This suggests that the RecursionError in @keewis example is a result of trying to concatenate datasets with mixed DatetimeIndex & CFTimeIndex indexes.

In [18]: files = glob.glob("*.nc") 
    ...: datasets = [xr.open_dataset(file) for file in files] 
    ...: xr.concat(datasets, dim="new_dim")
RecursionError

[tlogan2000]

FYI - @dcherian @keewis … thanks for the suggestions and help This final small change (dropping time values before replacing with the common calendar) has gotten all cases in the xclim code test suite running against the xarray@master… Feel free to close

for d in datasets:
    ds = xr.open_dataset(d, chunks=dict(time=10), decode_times=False)
    cal1 = xr.decode_cf(ds).time
    ds = ds.drop_vars('time')
    ds["time"] = pd.to_datetime(
        {
            "year": cal1.time.dt.year,
            "month": cal1.time.dt.month,
            "day": cal1.time.dt.day,
        }
    ).values
    dslist.append(ds)

ens1 = xr.concat(dslist,dim='realization')

#ens1 = ensembles.create_ensemble(datasets)
print(ens1)