TST: dtype for BigQuery TIMESTAMP unexpectedly using datetime64[ns, UTC] dtype
See original GitHub issue$ pytest 'tests/system/test_gbq.py::TestReadGBQIntegration::test_return_correct_types[env-current_timestamp()-datetime64[ns]]'
=============================== test session starts ===============================
platform darwin -- Python 3.6.4, pytest-4.2.0, py-1.8.0, pluggy-0.8.1
rootdir: /Users/swast/src/pandas/pandas-gbq, inifile:
collected 1 item
tests/system/test_gbq.py F [100%]
==================================== FAILURES =====================================
_ TestReadGBQIntegration.test_return_correct_types[env-current_timestamp()-datetime64[ns]] _
self = <tests.system.test_gbq.TestReadGBQIntegration object at 0x10c3277b8>
project_id = 'swast-scratch', expression = 'current_timestamp()'
type_ = 'datetime64[ns]'
@pytest.mark.parametrize(
"expression, type_",
[
("current_date()", "<M8[ns]"),
("current_timestamp()", "datetime64[ns]"),
("current_datetime()", "<M8[ns]"),
("TRUE", bool),
("FALSE", bool),
],
)
def test_return_correct_types(self, project_id, expression, type_):
"""
All type checks can be added to this function using additional
parameters, rather than creating additional functions.
We can consolidate the existing functions here in time
TODO: time doesn't currently parse
("time(12,30,00)", "<M8[ns]"),
"""
query = "SELECT {} AS _".format(expression)
df = gbq.read_gbq(
query,
project_id=project_id,
credentials=self.credentials,
dialect="standard",
)
> assert df["_"].dtype == type_
E AssertionError: assert datetime64[ns, UTC] == 'datetime64[ns]'
E + where datetime64[ns, UTC] = 0 2019-03-22 22:35:32.398261+00:00\nName: _, dtype: datetime64[ns, UTC].dtype
tests/system/test_gbq.py:392: AssertionError
============================ 1 failed in 2.68 seconds =============================
It’s odd that we explicitly specify the datetime64[ns] dtype, but it comes back as datetime64[ns, UTC] on the latest pandas version. I know to_dataframe from google-cloud-bigquery returns datetime objects with the UTC timezone, but I’d expect an explicit dtype of datetime64[ns] to take precedence.
Issue Analytics
- State:
- Created 4 years ago
- Comments:6 (1 by maintainers)
Top Results From Across the Web
dtype for BigQuery TIMESTAMP unexpectedly using ... - GitHub
I know to_dataframe from google-cloud-bigquery returns datetime objects with the UTC timezone, but I'd expect an explicit dtype of datetime64[ns] ...
Read more >python - Importing dataframes with dtypes datetime64[ns] and ...
I'd like to update a bigquery table from a dataframe with the following datatypes: datetime64[ns] timedelta64[ns].
Read more >Query About Datetime64[Ns] And Timestamp - ADocLib
The CData Python Connector for Snowflake enables you use pandas and other ... TST: dtype for BigQuery TIMESTAMP unexpectedly using datetime64[ns UTC] dtype...
Read more >What's New — pandas 0.19.2 documentation
Bug in pd.read_csv() in which the dtype parameter was not being respected for ... in Index creation from Timestamp with mixed tz coerces...
Read more >Comparison with pandas-gbq | BigQuery - Google Cloud
pandas.Timestamp("2015-10-02T16:00:00") ], } ) client = bigquery.Client() table_id = 'my_dataset.new_table' # Since string columns use the "object" dtype, ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found

Filed upstream at https://github.com/pandas-dev/pandas/issues/25843
Should we put this upstream? That second example is a good one…