Big Number with Trendline coudn't work with PostgreSQL's date field

Big Number with Trendline coudn’t work with PostgreSQL’s date field.

How to reproduce the bug

1. Create a table with a column which type is “date” in postgresql
2. Add that database and table as dataset to Superset
3. Try to create a Big Number with Trendline chart with that dataset
4. It shows

Unexpected error
Error: '+08'

which +08 is the timezone about your pgsql settings

Expected results

The chart should be draw successfully

Actual results

Unexpected error
Error: '+08'

Screenshots

Additional Error Log

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/security/decorators.py", line 190, in wraps
    return f(self, *args, **kwargs)
  File "/app/superset/utils/log.py", line 245, in wrapper
    value = f(*args, **kwargs)
  File "/app/superset/views/core.py", line 2351, in results
    return self.results_exec(key)
  File "/app/superset/views/core.py", line 2419, in results_exec
    obj = _deserialize_results_payload(
  File "/app/superset/views/utils.py", line 625, in _deserialize_results_payload
    df = result_set.SupersetResultSet.convert_table_to_df(pa_table)
  File "/app/superset/result_set.py", line 177, in convert_table_to_df
    return table.to_pandas(integer_object_nulls=True)
  File "pyarrow/array.pxi", line 757, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow/table.pxi", line 1748, in pyarrow.lib.Table._to_pandas
  File "/usr/local/lib/python3.8/site-packages/pyarrow/pandas_compat.py", line 789, in table_to_blockmanager
    blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
  File "/usr/local/lib/python3.8/site-packages/pyarrow/pandas_compat.py", line 1130, in _table_to_blocks
    return [_reconstruct_block(item, columns, extension_columns)
  File "/usr/local/lib/python3.8/site-packages/pyarrow/pandas_compat.py", line 1130, in <listcomp>
    return [_reconstruct_block(item, columns, extension_columns)
  File "/usr/local/lib/python3.8/site-packages/pyarrow/pandas_compat.py", line 733, in _reconstruct_block
    dtype = make_datetimetz(item['timezone'])
  File "/usr/local/lib/python3.8/site-packages/pyarrow/pandas_compat.py", line 758, in make_datetimetz
    tz = pa.lib.string_to_tzinfo(tz)
  File "pyarrow/types.pxi", line 1927, in pyarrow.lib.string_to_tzinfo
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "/usr/local/lib/python3.8/site-packages/pytz/__init__.py", line 188, in timezone
    raise UnknownTimeZoneError(zone)
pytz.exceptions.UnknownTimeZoneError: '+08'
2022-05-23 03:27:32,051:ERROR:superset.views.base:'+08'

Environment

• browser type and version: any browser
• superset version: 1.5.0
• python version: 3.8.12
• any feature flags active:
  • VERSIONED_EXPORT
  • ENABLE_TEMPLATE_PROCESSING

Checklist

• I have checked the superset logs for python stacktraces and included it here as text if there are any.
• I have reproduced the issue with at least the latest released version of superset.
• I have checked the issue tracker for the same issue and I haven’t found one similar.

Additional context

• This issue doesn’t exists in Superset-v1.4.0

The SQL in the chart is

SELECT DATE_TRUNC('day', publish_date) AS __timestamp,
       count(*) AS count
FROM
  (SELECT publish_date
   FROM software.package_file) AS virtual_table
GROUP BY DATE_TRUNC('day', publish_date)
LIMIT 50000;

The error part seems to be the DATE_TRUNC function, it returns something like “2022-05-23 12:00:00+08”. Which format seems to be errorly processed by pyarrow project.

Accourding to pyarrow project, the timezone format seems to be like +08:00 format, maybe it’s the root cuase? https://github.com/apache/arrow/blob/apache-arrow-5.0.0/python/pyarrow/types.pxi#L1915

[uwine-fr]

It looks like an issue that popped somewhere else, and that was fixed by upgrading psycopg2 to 2.9.1 in requirements.txt

[michael-s-molina]

So it seems to be the problem about k8s setup example located in https://superset.apache.org/docs/installation/running-on-kubernetes , and which instructed to pip install psycopg2==2.8.5 in dependency section.

Thanks for reporting this @ensky and also for submitting a PR to update the guide 🙂