question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

concat automagically outer-joins coordinates

See original GitHub issue

I would like to concatenate two netCDF files that have float64 coordinate variables. I thought the coordinate values were the same, but in fact they differ by something in the order of 1e-14.

Using open_mfdataset or concat to merge the datasets, I get a completely different output shape than the two input files have.

This is because concat is somewhere performing an outer join on the coordinates. Now I am wondering where else in my workflows this might happen without my notice…

It would be awesome if there was an option to change this behaviour on concat, open_mfdataset, and auto_combine. I would actually rather make these functions fail if any dimension other than the concatenation dimension differs.

Note: This could also be a special case, because

>>> (ds1.lon == ds2.lon).all()
<xarray.DataArray 'lon' ()>
array(True, dtype=bool)

while

>>> (ds1.lon.values == ds2.lon.values).all()
False

Maybe an interface change could be considered together with that discussed in #1340?

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Comments:8 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
shoyercommented, Jul 5, 2017

I think the right solution here is to relabel the coordinates on each of the datasets that you are concatenating. The alternative behavior here is xarray raising an error in the case of conflicting coordinate labels.

1reaction
darothencommented, Jul 5, 2017

@wqshen, a workaround until a more complete modification to align is available would be to explicitly copy/set the coordinate values on your arrays before using xr.concat(). Alternatively, if it’s as simple as stacking along a new tailing axis, you could stack via dask/numpy and then construct a new DataArray passing the coordinates explicitly.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Combining data - Xarray
To combine concat . concat takes an iterable of DataArray or Dataset objects, ... Vacant cells as a result of the outer-join are...
Read more >
Oracle SQL - Outer join based on concat - Stack Overflow
I want to outer join the column To_Number(Concat(Location, Channel_Id)) with ItemLoc.Loc where to put the (+) sign.
Read more >
xarray.combine_by_coords — xarray 0.13.0 documentation
Attempt to auto-magically combine the given datasets into one by using dimension coordinates. This method attempts to combine a group of datasets along...
Read more >
Concatenation (Combining Data Tables) in Python and Pandas
result = pd.concat([list of DataFrames], axis=0, join='outer', ... as the americas DataFrame (pandas automatically alphabetized them).
Read more >
Join Your Data - Tableau Help
When you use a full outer join to combine tables, the result is a table that ... For example, the join clause could...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found