str object has no attribute __array_priority__

Unfortunately I don’t have a simple way to reproduce this currently, but it does seem to be something I am running into repeatedly. Traceback and environment details below. Am using the distributed backend with the latest version of dask (0.14.3).

It seems have some issue when I tell it to compute. In particular, it looks like there is some issue internally converting some array representation. Though this is pretty deep in the traceback, so it could be a red herring. Had used a development version of dask (past 0.14.1) with multidimensional FFT support previously, which did not encounter this issue.

Traceback:

distributed.utils - ERROR - 'str' object has no attribute '__array_priority__'
Traceback (most recent call last):
  File "/opt/conda/lib/python2.7/site-packages/distributed/utils.py", line 212, in f
    result[0] = yield make_coro()
  File "/opt/conda/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/opt/conda/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/opt/conda/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/opt/conda/lib/python2.7/site-packages/distributed/client.py", line 1551, in _get
    result = yield self._gather(packed)
  File "/opt/conda/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/opt/conda/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/opt/conda/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/opt/conda/lib/python2.7/site-packages/distributed/client.py", line 996, in _gather
    traceback)
  File "/opt/conda/lib/python2.7/site-packages/toolz/functoolz.py", line 466, in __call__
    ret = self.first(*args, **kwargs)
  File "/opt/conda/lib/python2.7/site-packages/dask/array/core.py", line 468, in _concatenate2
    module = package_of(type(max(arrays, key=lambda x: x.__array_priority__))) or np
  File "/opt/conda/lib/python2.7/site-packages/dask/array/core.py", line 468, in <lambda>
    module = package_of(type(max(arrays, key=lambda x: x.__array_priority__))) or np
AttributeError: 'str' object has no attribute '__array_priority__'

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-10-9b9e82bc1130> in <module>()
    116             # Compute change
    117             shifts = shifts.persist()
--> 118             avg_rel_dist = avg_rel_dist.compute()
    119             i += 1
    120 

/opt/conda/lib/python2.7/site-packages/dask/base.pyc in compute(self, **kwargs)
     94             Extra keywords to forward to the scheduler ``get`` function.
     95         """
---> 96         (result,) = compute(self, traverse=False, **kwargs)
     97         return result
     98 

/opt/conda/lib/python2.7/site-packages/dask/base.pyc in compute(*args, **kwargs)
    201     dsk = collections_to_dsk(variables, optimize_graph, **kwargs)
    202     keys = [var._keys() for var in variables]
--> 203     results = get(dsk, keys, **kwargs)
    204 
    205     results_iter = iter(results)

/opt/conda/lib/python2.7/site-packages/distributed/client.pyc in get(self, dsk, keys, restrictions, loose_restrictions, resources, **kwargs)
   1588         return sync(self.loop, self._get, dsk, keys, restrictions=restrictions,
   1589                     loose_restrictions=loose_restrictions,
-> 1590                     resources=resources)
   1591 
   1592     def _optimize_insert_futures(self, dsk, keys):

/opt/conda/lib/python2.7/site-packages/distributed/utils.pyc in sync(loop, func, *args, **kwargs)
    221         e.wait(1000000)
    222     if error[0]:
--> 223         six.reraise(*error[0])
    224     else:
    225         return result[0]

/opt/conda/lib/python2.7/site-packages/distributed/utils.pyc in f()
    210                 raise RuntimeError("sync() called from thread of running loop")
    211             yield gen.moment
--> 212             result[0] = yield make_coro()
    213         except Exception as exc:
    214             logger.exception(exc)

/opt/conda/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053 
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/opt/conda/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/opt/conda/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1061                     if exc_info is not None:
   1062                         try:
-> 1063                             yielded = self.gen.throw(*exc_info)
   1064                         finally:
   1065                             # Break up a reference to itself

/opt/conda/lib/python2.7/site-packages/distributed/client.pyc in _get(self, dsk, keys, restrictions, loose_restrictions, resources, raise_on_error)
   1549         packed = pack_data(keys, futures)
   1550         try:
-> 1551             result = yield self._gather(packed)
   1552         except Exception as e:
   1553             if raise_on_error:

/opt/conda/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053 
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/opt/conda/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/opt/conda/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1061                     if exc_info is not None:
   1062                         try:
-> 1063                             yielded = self.gen.throw(*exc_info)
   1064                         finally:
   1065                             # Break up a reference to itself

/opt/conda/lib/python2.7/site-packages/distributed/client.pyc in _gather(self, futures, errors)
    994                             six.reraise(type(exception),
    995                                         exception,
--> 996                                         traceback)
    997                     if errors == 'skip':
    998                         bad_keys.add(key)

/opt/conda/lib/python2.7/site-packages/toolz/functoolz.pyc in __call__()
    464 
    465     def __call__(self, *args, **kwargs):
--> 466         ret = self.first(*args, **kwargs)
    467         for f in self.funcs:
    468             ret = f(ret)

/opt/conda/lib/python2.7/site-packages/dask/array/core.pyc in _concatenate2()
    466     if len(axes) > 1:
    467         arrays = [_concatenate2(a, axes=axes[1:]) for a in arrays]
--> 468     module = package_of(type(max(arrays, key=lambda x: x.__array_priority__))) or np
    469     return module.concatenate(arrays, axis=axes[0])
    470 

/opt/conda/lib/python2.7/site-packages/dask/array/core.pyc in <lambda>()
    466     if len(axes) > 1:
    467         arrays = [_concatenate2(a, axes=axes[1:]) for a in arrays]
--> 468     module = package_of(type(max(arrays, key=lambda x: x.__array_priority__))) or np
    469     return module.concatenate(arrays, axis=axes[0])
    470 

AttributeError: 'str' object has no attribute '__array_priority__'

Environment:

name: root
channels: !!python/tuple
- !!python/unicode
  'nanshe'
- !!python/unicode
  'conda-forge'
- !!python/unicode
  'defaults'
dependencies:
- apr=1.5.2=1
- conda-forge::anaconda-client=1.6.3=py27_0
- conda-forge::backports.shutil_get_terminal_size=1.0.0=py27_1
- conda-forge::backports_abc=0.5=py27_0
- conda-forge::beautifulsoup4=4.5.3=py27_0
- conda-forge::blas=1.1=openblas
- conda-forge::bleach=1.5.0=py27_0
- conda-forge::bokeh=0.12.5=py27_2
- conda-forge::boost=1.63.0=np112py27_6
- conda-forge::boost-cpp=1.63.0=2
- conda-forge::boto3=1.4.4=py27_0
- conda-forge::botocore=1.5.39=py27_0
- conda-forge::bottleneck=1.2.0=np112py27_0
- conda-forge::bzip2=1.0.6=1
- conda-forge::ca-certificates=2017.1.23=1
- conda-forge::certifi=2017.4.17=py27_0
- conda-forge::chardet=3.0.2=py27_1
- conda-forge::click=6.7=py27_0
- conda-forge::cloudpickle=0.2.2=py27_2
- conda-forge::clyent=1.2.1=py27_0
- conda-forge::conda=4.2.13=py27_0
- conda-forge::conda-build=2.1.10=py27_0
- conda-forge::conda-env=2.6.0=0
- conda-forge::conda-verify=2.0.0=py27_0
- conda-forge::configparser=3.5.0=py27_0
- conda-forge::contextlib2=0.5.5=py27_0
- conda-forge::curl=7.52.1=0
- conda-forge::cycler=0.10.0=py27_0
- conda-forge::dask=0.14.3=py27_0
- conda-forge::dask-ndfilters=0.1.0=py27_0
- conda-forge::dask-ndfourier=0.1.1=py27_0
- conda-forge::dbus=1.10.10=3
- conda-forge::decorator=4.0.11=py27_0
- conda-forge::distributed=1.16.3=py27_0
- conda-forge::docutils=0.13.1=py27_0
- conda-forge::drmaa=0.7.7=py27_0
- conda-forge::entrypoints=0.2.2=py27_1
- conda-forge::enum34=1.1.6=py27_1
- conda-forge::expat=2.1.0=2
- conda-forge::fasteners=0.14.1=py27_2
- conda-forge::fftw=3.3.6=0
- conda-forge::filelock=2.0.6=py27_0
- conda-forge::fontconfig=2.12.1=4
- conda-forge::freetype=2.7=1
- conda-forge::funcsigs=1.0.2=py27_0
- conda-forge::functools32=3.2.3.2=py27_1
- conda-forge::future=0.16.0=py27_0
- conda-forge::futures=3.0.5=py27_0
- conda-forge::gettext=0.19.7=1
- conda-forge::git=2.12.2=4
- conda-forge::glances=2.9.1=py27_0
- conda-forge::glib=2.51.4=0
- conda-forge::gmp=6.1.2=0
- conda-forge::gst-plugins-base=1.8.0=0
- conda-forge::gstreamer=1.8.0=1
- conda-forge::h5py=2.7.0=np112py27_0
- conda-forge::hdf5=1.8.17=10
- conda-forge::heapdict=1.0.0=py27_0
- conda-forge::html5lib=0.999=py27_0
- conda-forge::icu=58.1=1
- conda-forge::imageio=2.1.2=py27_0
- conda-forge::imgroi=0.0.2=py27_0
- conda-forge::ipykernel=4.6.1=py27_0
- conda-forge::ipyparallel=6.0.2=py27_0
- conda-forge::ipython=5.3.0=py27_0
- conda-forge::ipython_genutils=0.2.0=py27_0
- conda-forge::ipywidgets=6.0.0=py27_0
- conda-forge::jinja2=2.9.5=py27_0
- conda-forge::jmespath=0.9.2=py27_0
- conda-forge::jpeg=9b=0
- conda-forge::jsonschema=2.5.1=py27_0
- conda-forge::jupyter_client=5.0.1=py27_0
- conda-forge::jupyter_core=4.3.0=py27_0
- conda-forge::kenjutsu=0.5.1=py27_0
- conda-forge::libffi=3.2.1=3
- conda-forge::libiconv=1.14=4
- conda-forge::libpng=1.6.28=0
- conda-forge::libsodium=1.0.10=0
- conda-forge::libtiff=4.0.6=7
- conda-forge::libxcb=1.12=1
- conda-forge::libxml2=2.9.4=4
- conda-forge::locket=0.2.0=py27_1
- conda-forge::mahotas=1.4.3=np112py27_1
- conda-forge::markupsafe=0.23=py27_1
- conda-forge::matplotlib=2.0.2=np112py27_0
- conda-forge::mercurial=4.2=py27_1
- conda-forge::metawrap=0.0.2=py27_0
- conda-forge::mistune=0.7.4=py27_0
- conda-forge::mock=2.0.0=py27_0
- conda-forge::monotonic=1.3=py27_0
- conda-forge::mplview=0.0.2=py27_0
- conda-forge::msgpack-python=0.4.8=py27_0
- conda-forge::nbconvert=5.1.1=py27_1
- conda-forge::nbformat=4.3.0=py27_0
- conda-forge::ncurses=5.9=10
- conda-forge::networkx=1.11=py27_0
- conda-forge::notebook=5.0.0=py27_0
- conda-forge::npctypes=0.0.4=py27_0
- conda-forge::numpy=1.12.1=py27_blas_openblas_200
- conda-forge::olefile=0.44=py27_0
- conda-forge::openblas=0.2.19=2
- conda-forge::openssl=1.0.2k=0
- conda-forge::pandas=0.20.1=np112py27_0
- conda-forge::pandoc=1.19.2=0
- conda-forge::pandocfilters=1.4.1=py27_0
- conda-forge::partd=0.3.8=py27_0
- conda-forge::patchelf=0.9=1
- conda-forge::pathlib2=2.2.1=py27_0
- conda-forge::pbr=3.0.0=py27_0
- conda-forge::pcre=8.39=0
- conda-forge::pexpect=4.2.1=py27_0
- conda-forge::pickleshare=0.7.3=py27_0
- conda-forge::pillow=4.1.1=py27_0
- conda-forge::pims=0.4=py27_0
- conda-forge::pip=9.0.1=py27_0
- conda-forge::pkginfo=1.2.1=py27_0
- conda-forge::prompt_toolkit=1.0.14=py27_0
- conda-forge::psutil=5.2.1=py27_0
- conda-forge::ptyprocess=0.5.1=py27_0
- conda-forge::pycosat=0.6.1=py27_0
- conda-forge::pycrypto=2.6.1=py27_0
- conda-forge::pyfftw=0.10.4=np112py27_1
- conda-forge::pygments=2.2.0=py27_0
- conda-forge::pyparsing=2.2.0=py27_0
- conda-forge::pyqt=5.6.0=py27_2
- conda-forge::python=2.7.13=1
- conda-forge::python-dateutil=2.6.0=py27_0
- conda-forge::pytz=2017.2=py27_0
- conda-forge::pywavelets=0.5.2=np112py27_0
- conda-forge::pyyaml=3.12=py27_1
- conda-forge::pyzmq=16.0.2=py27_2
- conda-forge::qt=5.6.2=2
- conda-forge::rank_filter=0.4.13=np112py27_0
- conda-forge::readline=6.2=0
- conda-forge::requests=2.13.0=py27_0
- conda-forge::ruamel_yaml=0.11.14=py27_0
- conda-forge::runipy=0.1.5=py27_1
- conda-forge::s3fs=0.1.0=py27_0
- conda-forge::s3transfer=0.1.10=py27_1
- conda-forge::scandir=1.5=py27_1
- conda-forge::scikit-image=0.13.0=np112py27_0
- conda-forge::scikit-learn=0.18.1=np112py27_blas_openblas_200
- conda-forge::scipy=0.19.0=np112py27_blas_openblas_200
- conda-forge::setuptools=33.1.1=py27_0
- conda-forge::simplegeneric=0.8.1=py27_0
- conda-forge::singledispatch=3.4.0.3=py27_0
- conda-forge::sip=4.18=py27_1
- conda-forge::six=1.10.0=py27_1
- conda-forge::slicerator=0.9.8=py27_0
- conda-forge::sortedcontainers=1.5.3=py27_0
- conda-forge::sqlite=3.13.0=1
- conda-forge::ssl_match_hostname=3.5.0.1=py27_1
- conda-forge::subprocess32=3.2.7=py27_0
- conda-forge::tblib=1.3.2=py27_0
- conda-forge::terminado=0.6=py27_0
- conda-forge::testpath=0.3=py27_0
- conda-forge::tifffile=0.12.0=np112py27_0
- conda-forge::tk=8.5.19=1
- conda-forge::toolz=0.8.2=py27_0
- conda-forge::tornado=4.5.1=py27_0
- conda-forge::traitlets=4.3.2=py27_0
- conda-forge::vigra=1.11.0=np112py27_11
- conda-forge::wcwidth=0.1.7=py27_0
- conda-forge::webcolors=1.7=py27_0
- conda-forge::webencodings=0.5=py27_0
- conda-forge::wheel=0.29.0=py27_0
- conda-forge::widgetsnbextension=2.0.0=py27_0
- conda-forge::xnumpy=0.1.2=py27_0
- conda-forge::xorg-libxau=1.0.8=3
- conda-forge::xorg-libxdmcp=1.1.2=3
- conda-forge::xz=5.2.2=0
- conda-forge::yail=0.0.2=py27_0
- conda-forge::yaml=0.1.6=0
- conda-forge::zarr=2.1.4=py27_0
- conda-forge::zeromq=4.2.1=1
- conda-forge::zict=0.1.2=py27_0
- conda-forge::zlib=1.2.11=0
- libgfortran=3.0.0=1
- nanshe::nanshe=0.1.0a58=py27_0
- serf=1.3.8=1
- svn=1.9.4=0
- pip:
  - backports-abc==0.5
  - backports.shutil-get-terminal-size==1.0.0
  - backports.ssl-match-hostname==3.5.0.1
  - ipython-genutils==0.2.0
  - jupyter-client==5.0.1
  - jupyter-core==4.3.0
  - prompt-toolkit==1.0.14
  - rank-filter==0.4.13
  - ruamel-yaml==0.11.14

[jcrist]

Ok, I’ve heard a report from someone else using dask.distributed that errors during serialization to disk (using the spill to disk functionality) can cause a key to be dropped from Worker.data, resulting in the above behavior. Their issue was fixed in https://github.com/dask/distributed/pull/946, but there may be more issues especially around memory errors.

I’ve opened an issue in the distributed repo: https://github.com/dask/distributed/issues/1088. If you could comment there regarding your specific situation, and if possible provide logs from the offending workers that would be helpful. Also note if you’re using the spill to disk functionality.

I’m closing this issue, as it’s related to the distributed scheduler instead.

[jakirkham]

SGTM. Thanks @jcrist. Will follow up with some logs.