error with pd.read_sql and sqlalchemy connection
See original GitHub issueI’m setting up an sqlalchemy engine using this connection string
conn_str = awsathena+rest://:@athena.{region_name}.amazonaws.com:443/{schema_name}?s3_staging_dir={s3_staging_dir}
which creates an sqlalchemy engine using pyathena for the dialect thing. I just switched to sqlalchemy as pandas
from 1.4.0 on in read_sql
requires an sqlalchemy connectionable, before it accepted a pyathena connection as well. Now, when running pd.read_sql(query, my_new_sqlalchemy_connection(conn_str))
I get somewhere from within pyathena:
Traceback (most recent call last):
File "/home/henning/my-project/.venv/lib/python3.8/site-packages/pyathena/common.py", line 413, in _execute
query_id = retry_api_call(
File "/home/henning/my-project/.venv/lib/python3.8/site-packages/pyathena/util.py", line 84, in retry_api_call
return retry(func, *args, **kwargs)
File "/home/henning/my-project/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 423, in __call__
do = self.iter(retry_state=retry_state)
File "/home/henning/my-project/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 360, in iter
return fut.result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/home/henning/my-project/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 426, in __call__
result = fn(*args, **kwargs)
File "/home/henning/my-project/.venv/lib/python3.8/site-packages/botocore/client.py", line 391, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/henning/my-project/.venv/lib/python3.8/site-packages/botocore/client.py", line 719, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidRequestException: An error occurred (InvalidRequestException) when calling the StartQueryExecution operation: line 23:44: mismatched input '2019'. Expecting: '%', '*', '+', '-', '.', '/', 'AND', 'AT', 'EXCEPT', 'GROUP', 'HAVING', 'INTERSECT', 'LIMIT', 'OFFSET', 'OR', 'ORDER', 'UNION', '[', '||', <EOF>
I debugged through pyathena and found when changing
query = table_name
in pyathena.sqlalchemy_athena.AthenaDialect.get_columns
it works as expected. It this a bug or am I setting up something wrongly?
edit:
name : pyathena
version : 2.4.1
Issue Analytics
- State:
- Created 2 years ago
- Comments:6 (4 by maintainers)
Top Results From Across the Web
How to create sql alchemy connection for pandas read_sql ...
I am using SQL server with SQLAlchemy import pyodbc import pandas as pd cnxn = pyodbc.connect('DRIVER=ODBC Driver 17 for SQL Server ...
Read more >Connecting Pandas to a Database ... - Hackers and Slackers
We need to install a database connector as our third and final library, but the library you need depends on the type of...
Read more >pandas.read_sql — pandas 1.5.2 documentation
Read SQL query or database table into a DataFrame. ... conSQLAlchemy connectable, str, or sqlite3 connection. Using SQLAlchemy makes it possible to use...
Read more >Usage error in pandas.read_sql with sqlalchemy-Pandas,Python
Error when trying to create new database table in SQL Server 2016 from csv file while using python 3.5 with pandas and sqlalchemy...
Read more >How to Connect to SQL Databases from Python Using ...
How to Connect to SQL Databases from Python Using SQLAlchemy and Pandas · Install Libraries · Import Libraries · Create Connection to the...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Using the read_sql method directly is the best choice in terms of performance. If for some reason it is difficult to use it, you can modify the log output of the pyathena package as follows.
Thanks @laughingman7743. It actually works. However this error message is indeed appearing. Is there any way to mute it?