to_sql 2x slower in pandas 0.20.3 then 0.19.2 (reproducable)

Code Sample, a copy-pastable example if possible

import numpy as np
import pandas as pd

index = pd.date_range('1970-01-01',periods=100000, freq='H')
values = np.random.randn(100000,10)
df = pd.DataFrame(values, index = index)
df.to_sql('table_name','connection_string', if_exists='append')

Problem description

Since upgrading to 0.20+ I am finding that to_sql takes twice as long. I used two environments to test the code on the same machine, below are the results:

pandas 0.20.3

CPU times: user 15.9 s, sys: 3.67 s, total: 19.6 s
Wall time: 1min 13s

pandas 0.19.2

CPU times: user 5.2 s, sys: 728 ms, total: 5.93 s
Wall time: 36.8 s

Please note that each test was performed on an empty table.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None python: 3.5.2.final.0 python-bits: 64 OS: Linux OS-release: 4.4.0-87-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.19.2 nose: None pip: 9.0.1 setuptools: 27.2.0 Cython: 0.25.2 numpy: 1.11.2 scipy: 0.18.1 statsmodels: 0.6.1 xarray: None IPython: 5.1.0 sphinx: 1.5.1 patsy: 0.4.1 dateutil: 2.6.0 pytz: 2016.10 blosc: None bottleneck: 1.2.0 tables: None numexpr: None matplotlib: 1.5.3 openpyxl: None xlrd: None xlwt: None xlsxwriter: None lxml: None bs4: None html5lib: 0.9999999 httplib2: None apiclient: None sqlalchemy: 1.1.6 pymysql: None psycopg2: 2.6.2 (dt dec pq3 ext lo64) jinja2: 2.9.4 boto: None pandas_datareader: 0.2.1

INSTALLED VERSIONS

commit: None python: 3.6.1.final.0 python-bits: 64 OS: Linux OS-release: 4.4.0-87-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.20.3 pytest: 3.0.7 pip: 9.0.1 setuptools: 27.2.0 Cython: 0.25.2 numpy: 1.13.1 scipy: 0.19.0 xarray: None IPython: 5.3.0 sphinx: 1.5.6 patsy: 0.4.1 dateutil: 2.6.1 pytz: 2017.2 blosc: None bottleneck: 1.2.1 tables: 3.3.0 numexpr: 2.6.2 feather: None matplotlib: 2.0.2 openpyxl: 2.4.7 xlrd: 1.0.0 xlwt: 1.2.0 xlsxwriter: 0.9.6 lxml: 3.7.3 bs4: 4.6.0 html5lib: 0.999 sqlalchemy: 1.1.9 pymysql: None psycopg2: 2.7.3 (dt dec pq3 ext lo64) jinja2: 2.9.6 s3fs: None pandas_gbq: None pandas_datareader: None

[gfyoung]

Judging from this conversation, it appears that we’re in the clear for this problem. Closing for the time being, but can reopen if it turns out that pandas is to blame.

[gregsifr]

Thank you for doing that. I will raise a ticket in the psycopg2 library for further investigation.