Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Support for datetime[*] numpy dtype

See original GitHub issue

Currently, a pandas DataFrame with a column of eg. datetime[ns] dtype is resulting in an obscure ValueError: ValueError: cannot include dtype 'M' in a buffer

values = {'time':  ['20190902093000','20190913093000','20190921200000']}
df = pd.DataFrame(values, columns = ['time'])
df['time'] = pd.to_datetime(df['time'], format='%Y%m%d%H%M%S')
df.dtypes

time       datetime64[ns]
dtype: object

# current workaround
df['time'] = df.time.values.astype(int)
akarr = ak.Record({c: df[c].values for c in df.columns})

Issue Analytics

State:
Created 3 years ago
Reactions:4
Comments:18 (12 by maintainers)

Top GitHub Comments

1reaction

jpivarskicommented, Aug 11, 2020

I’d like to thank you, @YannickJadoul, for your help in the above. Feel free to “unwatch” this thread, since giving you credit also pulled you into the GitHub issue as a side-effect.

1reaction

jpivarskicommented, Aug 4, 2020

This is actually waiting on a pybind11 feature: it currently can’t ingest datetime64 ('M') or timedelta64 ('m') arrays as py::buffer_info. I think that’s being handled in pybind/pybind11#2078 or pybind/pybind11#2085 or both, I’m not sure. (Issue pybind/pybind11#1970 might also be related.)

Either way, we have two dtype enums ready and waiting:

https://github.com/scikit-hep/awkward-1.0/blob/46767d776dcdc825621f8151dbd4289225b09659/include/awkward/util.h#L26-L47

The bigger question, though, is what reducers would mean for these types. ak.sum could make sense for timedelta64, but not datetime64, ak.prod wouldn’t make sense for either, though ak.min and ak.max could be extended for either. (Maybe the datetime64 could reuse the uint64 min/max and the timedelta64 could reuse the int64 min/max…)

All the type coersions when concatenating non-temporal and temporal numbers would also have to be added, which is a lot of boilerplate, but straightforward.

Top Results From Across the Web

Datetimes and Timedeltas — NumPy v1.24 Manual

Starting in NumPy 1.7, there are core array data types which natively support datetime functionality. The data type is called datetime64 , so...

Datetimes and Timedeltas — NumPy v1.8 Manual

Starting in NumPy 1.7, there are core array data types which natively support datetime functionality. The data type is called “datetime64”, so named...

How can I make a python numpy arange of datetime

The key point here is to use astype(datetime) , otherwise the result will be datetime64 . Share.

Support for datetime[*] numpy dtype - scikit-hep/awkward

Currently, a pandas DataFrame with a column of eg. datetime[ns] dtype is resulting in an obscure ValueError: ValueError: cannot include dtype 'M' in...

Python Datetime Vs. NumPy Datetime: 8 Differences You ...

The range of native Python datetime has been set, hardcoded, between 1 to 9999 Years. It is done so because the Python integer...