Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

PostgresToGCSOperator doesn't convert DATE correctly

See original GitHub issue

What happened:

Time datatypes are treated as datetime in the operator this create problems when dumping to csv (so can be loaded to BigQuery) because the data will be date type yet bigquery will expect datetime format.

How to reproduce it: dump DATE column to csv using the operator and create a table from it in bigquery. The column will not be of DATE format.

Anything else we need to know:

probably need fix at https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/postgres_to_gcs.py#L46 changing it from TIMESTAMP to DATE as:

  Date                  -> 1082
  Time                  -> 1083
  Timestamp             -> 1114
  TimestampWithTimeZone -> 1184

https://github.com/psycopg/psycopg2/blob/master/psycopg/pgtypes.h#L41

and make some adjustments in convert_type function.

Issue Analytics

State:
Created 3 years ago
Comments:8 (5 by maintainers)

Top GitHub Comments

1reaction

eladkalcommented, Feb 8, 2022

Recently https://github.com/apache/airflow/pull/20807 was merged. It added a generic SqlToS3Operator and deprecated MySqlToS3Operator.

I would say that probably the best thing to do is to implement similar generic operator for Gcs cc @mariotaddeucci

0reactions

pierrejeambruncommented, Mar 27, 2022

Normally it does indeed solve this issue. In my tests I could export and then load into bigquery each of these different field types.

Top Results From Across the Web

Convert date to specific format using Postgresql - Stack Overflow

Your approach doesn't work because the value is apparently already stored as a string, so converting it back to a string with a...

Cloud Composer release notes | Google Cloud

Migrate from Mainframe ... Service to convert live video and package for streaming. ... Convert video files and package them for optimized delivery....

8.1: Data Type Formatting Functions - PostgreSQL

The PostgreSQL formatting functions provide a powerful set of tools for converting various data types (date/time, integer, floating point, ...

airflow.providers.google.cloud.transfers.postgres_to_gcs

[docs]class PostgresToGCSOperator(BaseSQLToGCSOperator): """ Copy data from Postgres to Google Cloud Storage in JSON or CSV format.

Exploring Postgres date formats and their different functions

The format or data type of the date column should always match the user's input. Additionally, you should convert the date format display...