to_sql 2x slower in pandas 0.20.3 then 0.19.2 (reproducable)
See original GitHub issueCode Sample, a copy-pastable example if possible
import numpy as np
import pandas as pd
index = pd.date_range('1970-01-01',periods=100000, freq='H')
values = np.random.randn(100000,10)
df = pd.DataFrame(values, index = index)
df.to_sql('table_name','connection_string', if_exists='append')
Problem description
Since upgrading to 0.20+ I am finding that to_sql
takes twice as long.
I used two environments to test the code on the same machine, below are the results:
pandas 0.20.3
CPU times: user 15.9 s, sys: 3.67 s, total: 19.6 s
Wall time: 1min 13s
pandas 0.19.2
CPU times: user 5.2 s, sys: 728 ms, total: 5.93 s
Wall time: 36.8 s
Please note that each test was performed on an empty table.
Output of pd.show_versions()
INSTALLED VERSIONS
commit: None python: 3.5.2.final.0 python-bits: 64 OS: Linux OS-release: 4.4.0-87-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8
pandas: 0.19.2 nose: None pip: 9.0.1 setuptools: 27.2.0 Cython: 0.25.2 numpy: 1.11.2 scipy: 0.18.1 statsmodels: 0.6.1 xarray: None IPython: 5.1.0 sphinx: 1.5.1 patsy: 0.4.1 dateutil: 2.6.0 pytz: 2016.10 blosc: None bottleneck: 1.2.0 tables: None numexpr: None matplotlib: 1.5.3 openpyxl: None xlrd: None xlwt: None xlsxwriter: None lxml: None bs4: None html5lib: 0.9999999 httplib2: None apiclient: None sqlalchemy: 1.1.6 pymysql: None psycopg2: 2.6.2 (dt dec pq3 ext lo64) jinja2: 2.9.4 boto: None pandas_datareader: 0.2.1
INSTALLED VERSIONS
commit: None python: 3.6.1.final.0 python-bits: 64 OS: Linux OS-release: 4.4.0-87-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8
pandas: 0.20.3 pytest: 3.0.7 pip: 9.0.1 setuptools: 27.2.0 Cython: 0.25.2 numpy: 1.13.1 scipy: 0.19.0 xarray: None IPython: 5.3.0 sphinx: 1.5.6 patsy: 0.4.1 dateutil: 2.6.1 pytz: 2017.2 blosc: None bottleneck: 1.2.1 tables: 3.3.0 numexpr: 2.6.2 feather: None matplotlib: 2.0.2 openpyxl: 2.4.7 xlrd: 1.0.0 xlwt: 1.2.0 xlsxwriter: 0.9.6 lxml: 3.7.3 bs4: 4.6.0 html5lib: 0.999 sqlalchemy: 1.1.9 pymysql: None psycopg2: 2.7.3 (dt dec pq3 ext lo64) jinja2: 2.9.6 s3fs: None pandas_gbq: None pandas_datareader: None
Issue Analytics
- State:
- Created 6 years ago
- Comments:7 (4 by maintainers)
Top GitHub Comments
Judging from this conversation, it appears that we’re in the clear for this problem. Closing for the time being, but can reopen if it turns out that
pandas
is to blame.Thank you for doing that. I will raise a ticket in the
psycopg2
library for further investigation.