question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

to_sql method turns datetime64 index to time zone aware in postgres

See original GitHub issue

I have a pandas Dataframe with an index which is of type datetime64[ns]. when I use the to_sql method i expect the index to be created as a timestamp postgres column, however it is creates a timestamptz column.

Note that when the datetime64[ns] column is not an index but rather a normal column, this doesn’t happen.

pandas 0.23.4, psycopg2 2.7.4, sqlalchemy 1.2.7, PostgreSQL 9.6.6


example

dates = pd.date_range('2018-01-01', periods=5, freq='6h')
df_test = pd.DataFrame({'nums': range(5)}, index=dates)
nums
2018-01-01 00:00:00 0
2018-01-01 06:00:00 1
2018-01-01 12:00:00 2

inserting to postgres

df_test.to_sql('foo_table',postgres_uri,schema='data_test',index_label='info_date')

when reading from the database I get the index with different type

df_db = pd.read_sql_table('foo_table',postgres_uri,schema='data_test',index_col='info_date')
nums
info_date
2018-01-01 00:00:00+00:00 0
2018-01-01 06:00:00+00:00 1
2018-01-01 12:00:00+00:00 2
As you can see `info_date` is of type `datetime64[ns, UTC]`

not sure if it’s a sqlalchemy or pandas question. reported because of my question on SO

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:5 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
mroeschkecommented, Nov 7, 2018

I was able to confirm this behavior on master. I’ve included a fix in #22654 since it was a simple fix to handle.

0reactions
finetecommented, Nov 6, 2018

@mroeschke it’s a normal schema, I just didn’t want to create a table in my public schema.

Read more comments on GitHub >

github_iconTop Results From Across the Web

pandas to sql turns datetime64 index to time zone aware ...
[Solved]-pandas to sql turns datetime64 index to time zone aware-postgresql. Search. score:0. I use mysql , but maybe if you set index=True?
Read more >
pandas to sql turns datetime64 index to time zone aware
when I use the to_sql method i expect the index to be created as a timestamp postgres column, however it is creates a...
Read more >
Documentation: 15: 8.5. Date/Time Types - PostgreSQL
PostgreSQL supports the full set of SQL date and time types, shown in Table ... In most cases, a combination of date ,...
Read more >
14 Time Zone Handling — Pandas Doc - GitHub Pages
To remove timezone from tz-aware DatetimeIndex , use tz_localize(None) or tz_convert(None) . tz_localize(None) will remove timezone holding local time ...
Read more >
Exploring Postgres date formats and their different functions
Date format: This is where you specify the new date format. The following query converts existing date values stored in the [SalesOrders] table ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found