question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

error with pd.read_sql and sqlalchemy connection

See original GitHub issue

I’m setting up an sqlalchemy engine using this connection string

conn_str = awsathena+rest://:@athena.{region_name}.amazonaws.com:443/{schema_name}?s3_staging_dir={s3_staging_dir}

which creates an sqlalchemy engine using pyathena for the dialect thing. I just switched to sqlalchemy as pandas from 1.4.0 on in read_sql requires an sqlalchemy connectionable, before it accepted a pyathena connection as well. Now, when running pd.read_sql(query, my_new_sqlalchemy_connection(conn_str)) I get somewhere from within pyathena:

Traceback (most recent call last):
  File "/home/henning/my-project/.venv/lib/python3.8/site-packages/pyathena/common.py", line 413, in _execute
    query_id = retry_api_call(
  File "/home/henning/my-project/.venv/lib/python3.8/site-packages/pyathena/util.py", line 84, in retry_api_call
    return retry(func, *args, **kwargs)
  File "/home/henning/my-project/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 423, in __call__
    do = self.iter(retry_state=retry_state)
  File "/home/henning/my-project/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 360, in iter
    return fut.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/home/henning/my-project/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 426, in __call__
    result = fn(*args, **kwargs)
  File "/home/henning/my-project/.venv/lib/python3.8/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/henning/my-project/.venv/lib/python3.8/site-packages/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidRequestException: An error occurred (InvalidRequestException) when calling the StartQueryExecution operation: line 23:44: mismatched input '2019'. Expecting: '%', '*', '+', '-', '.', '/', 'AND', 'AT', 'EXCEPT', 'GROUP', 'HAVING', 'INTERSECT', 'LIMIT', 'OFFSET', 'OR', 'ORDER', 'UNION', '[', '||', <EOF>

I debugged through pyathena and found when changing

query = table_name in pyathena.sqlalchemy_athena.AthenaDialect.get_columns it works as expected. It this a bug or am I setting up something wrongly?

edit:

name         : pyathena
version      : 2.4.1

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:6 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
laughingman7743commented, May 19, 2022

Using the read_sql method directly is the best choice in terms of performance. If for some reason it is difficult to use it, you can modify the log output of the pyathena package as follows.

import logging
from logging import CRITICAL
logging.getLogger("pyathena").setLevel(CRITICAL)
0reactions
plazmakekscommented, May 19, 2022

Thanks @laughingman7743. It actually works. However this error message is indeed appearing. Is there any way to mute it?

Read more comments on GitHub >

github_iconTop Results From Across the Web

How to create sql alchemy connection for pandas read_sql ...
I am using SQL server with SQLAlchemy import pyodbc import pandas as pd cnxn = pyodbc.connect('DRIVER=ODBC Driver 17 for SQL Server ...
Read more >
Connecting Pandas to a Database ... - Hackers and Slackers
We need to install a database connector as our third and final library, but the library you need depends on the type of...
Read more >
pandas.read_sql — pandas 1.5.2 documentation
Read SQL query or database table into a DataFrame. ... conSQLAlchemy connectable, str, or sqlite3 connection. Using SQLAlchemy makes it possible to use...
Read more >
Usage error in pandas.read_sql with sqlalchemy-Pandas,Python
Error when trying to create new database table in SQL Server 2016 from csv file while using python 3.5 with pandas and sqlalchemy...
Read more >
How to Connect to SQL Databases from Python Using ...
How to Connect to SQL Databases from Python Using SQLAlchemy and Pandas · Install Libraries · Import Libraries · Create Connection to the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found