question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

TST: dtype for BigQuery TIMESTAMP unexpectedly using datetime64[ns, UTC] dtype

See original GitHub issue
$ pytest 'tests/system/test_gbq.py::TestReadGBQIntegration::test_return_correct_types[env-current_timestamp()-datetime64[ns]]'
=============================== test session starts ===============================
platform darwin -- Python 3.6.4, pytest-4.2.0, py-1.8.0, pluggy-0.8.1
rootdir: /Users/swast/src/pandas/pandas-gbq, inifile:
collected 1 item                                                                  

tests/system/test_gbq.py F                                                  [100%]

==================================== FAILURES =====================================
_ TestReadGBQIntegration.test_return_correct_types[env-current_timestamp()-datetime64[ns]] _

self = <tests.system.test_gbq.TestReadGBQIntegration object at 0x10c3277b8>
project_id = 'swast-scratch', expression = 'current_timestamp()'
type_ = 'datetime64[ns]'

    @pytest.mark.parametrize(
        "expression, type_",
        [
            ("current_date()", "<M8[ns]"),
            ("current_timestamp()", "datetime64[ns]"),
            ("current_datetime()", "<M8[ns]"),
            ("TRUE", bool),
            ("FALSE", bool),
        ],
    )
    def test_return_correct_types(self, project_id, expression, type_):
        """
        All type checks can be added to this function using additional
        parameters, rather than creating additional functions.
        We can consolidate the existing functions here in time
    
        TODO: time doesn't currently parse
        ("time(12,30,00)", "<M8[ns]"),
        """
        query = "SELECT {} AS _".format(expression)
        df = gbq.read_gbq(
            query,
            project_id=project_id,
            credentials=self.credentials,
            dialect="standard",
        )
>       assert df["_"].dtype == type_
E       AssertionError: assert datetime64[ns, UTC] == 'datetime64[ns]'
E        +  where datetime64[ns, UTC] = 0   2019-03-22 22:35:32.398261+00:00\nName: _, dtype: datetime64[ns, UTC].dtype

tests/system/test_gbq.py:392: AssertionError
============================ 1 failed in 2.68 seconds =============================

It’s odd that we explicitly specify the datetime64[ns] dtype, but it comes back as datetime64[ns, UTC] on the latest pandas version. I know to_dataframe from google-cloud-bigquery returns datetime objects with the UTC timezone, but I’d expect an explicit dtype of datetime64[ns] to take precedence.

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:6 (1 by maintainers)

github_iconTop GitHub Comments

1reaction
tswastcommented, Mar 22, 2019
1reaction
max-sixtycommented, Mar 22, 2019

Should we put this upstream? That second example is a good one…

Read more comments on GitHub >

github_iconTop Results From Across the Web

dtype for BigQuery TIMESTAMP unexpectedly using ... - GitHub
I know to_dataframe from google-cloud-bigquery returns datetime objects with the UTC timezone, but I'd expect an explicit dtype of datetime64[ns] ...
Read more >
python - Importing dataframes with dtypes datetime64[ns] and ...
I'd like to update a bigquery table from a dataframe with the following datatypes: datetime64[ns] timedelta64[ns].
Read more >
Query About Datetime64[Ns] And Timestamp - ADocLib
The CData Python Connector for Snowflake enables you use pandas and other ... TST: dtype for BigQuery TIMESTAMP unexpectedly using datetime64[ns UTC] dtype...
Read more >
What's New — pandas 0.19.2 documentation
Bug in pd.read_csv() in which the dtype parameter was not being respected for ... in Index creation from Timestamp with mixed tz coerces...
Read more >
Comparison with pandas-gbq | BigQuery - Google Cloud
pandas.Timestamp("2015-10-02T16:00:00") ], } ) client = bigquery.Client() table_id = 'my_dataset.new_table' # Since string columns use the "object" dtype, ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found