question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

xr.concat loses coordinate dtype information with recarrays in 0.9

See original GitHub issue

Simple example script:

import numpy as np
import xarray as xr

p1 = np.array([('A', 180), ('B', 150), ('C', 200)], dtype=[('name', '|S256'), ('height', int)])
p2 = np.array([('D', 170), ('E', 250), ('F', 150)], dtype=[('name', '|S256'), ('height', int)])

data1 = np.arange(50, 80, 1, dtype=np.float)
data2 = data1 * 2

dims = ['measurement', 'participant']

da1 = xr.DataArray(
    data1.reshape(10, 3),
    coords={
        'measurement': np.arange(10),
        'participant': p1,
        'samplerate': 1
    },
    dims=dims
)

da2 = xr.DataArray(
    data2.reshape(10, 3),
    coords={
        'measurement': np.arange(10),
        'participant': p2,
        'samplerate': 1
    },
    dims=dims
)

combined = xr.concat([da1, da2], dim='participant')
print(da1.participant.dtype)
print(da2.participant.dtype)
print(combined.participant.dtype)

With xarray version 0.9.5 (and 0.8.2), this results in:

[('name', 'S256'), ('height', '<i8')]
[('name', 'S256'), ('height', '<i8')]
object

We have to go all the way back to xarray version 0.7 to get the expected behavior:

[('name', 'S256'), ('height', '<i8')]
[('name', 'S256'), ('height', '<i8')]
[('name', 'S256'), ('height', '<i8')]

Issue Analytics

  • State:open
  • Created 6 years ago
  • Comments:7 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
shoyercommented, May 30, 2017

Is there an advantage to using pandas for concatenation over numpy.concatenate?

Yes, otherwise we loose pandas (d)type information. For example, a MultiIndex gets turned into a dtype=object array of tuples.

0reactions
aldanorcommented, Mar 20, 2019

Looks like this is still a problem, just tested on 0.11.3 and it still results in object

Read more comments on GitHub >

github_iconTop Results From Across the Web

Combining data — xarray 0.9.6 documentation
Concatenate ¶. To combine arrays along existing or new dimension into a larger array, you can use concat() . concat takes an iterable...
Read more >
What's New — xarray 0.9.6+dev240.g5a28b89 documentation
These methods return a new Dataset (or DataArray) with updated data or coordinate variables. sel() now supports the method parameter, which works like...
Read more >
xarray.concat
Concatenate xarray objects along a new or existing dimension. ... Each object is expected to consist of variables and coordinates with matching shapes ......
Read more >
What's New - 《Xarray v0.12.3 Document》 - 书栈网 · BookStack
See Coarsen large arraysand Computation using Coordinates for details. ... Fixed dtype promotion rules in where() and concat() tomatch ...
Read more >
Structured arrays — NumPy v1.24 Manual
repack_fields converts an aligned dtype or array to a packed one and vice versa. It takes either a dtype or structured ndarray as...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found