BUG: using dtype=str in pd.read_sql_query casts nans to strings instead of nan
See original GitHub issuePandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
I have confirmed this bug exists on the master branch of pandas.
Reproducible Example
# Table contains columns with nulls
import sqlite3
import pandas as pd
con = sqlite3.connect('example.db')
cur = con.cursor()
cur.execute('''CREATE TABLE IF NOT EXISTS Sample
(col1,col2)''')
cur.execute("INSERT INTO Sample VALUES ('val1',NULL)")
con.commit()
df = pd.read_sql_query("SELECT * from Sample", con, dtype=str)
con.close()
print(df.info())
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2 entries, 0 to 1
Data columns (total 2 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 col1 2 non-null object
1 col2 2 non-null object
dtypes: object(2)
memory usage: 160.0+ bytes
print(df.to_markdown())
| | col1 | col2 |
|---:|:-------|:-------|
| 0 | val1 | None |
| 1 | val1 | None |
Issue Description
Parsing dtypes to str on all columns results on ignoring nans and parsing nulls to strings instead of keeping them as nans.
Expected Behavior
Similar to the pd.read_csv(…,dtype=str) , where nans are taken into account.
Installed Versions
pandas=1.3.5
Python=3.8
pyodbc=4.0.3
Issue Analytics
- State:
- Created 2 years ago
- Comments:8 (5 by maintainers)
Top Results From Across the Web
How to turn unwanted string values into NaNs in pandas
Per the Documentation it will replace any value which cannot be converted with NaN import pandas as pd df = pd.
Read more >Pandas Read SQL Query or Table with Examples
pandas read_sql() function is used to read SQL query or database table into DataFrame. This is a wrapper on read_sql_query() and read_sql_table()
Read more >pandas.read_sql_query — pandas 1.5.2 documentation
Read SQL query into a DataFrame. Returns a DataFrame corresponding to the result set of the query string. Optionally provide an index_col parameter...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Hi,
Could you provide something reproducible/describe the steps necessary to reproduce?
Using object solved the issue thanks !