question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

to_sql 2x slower in pandas 0.20.3 then 0.19.2 (reproducable)

See original GitHub issue

Code Sample, a copy-pastable example if possible

import numpy as np
import pandas as pd

index = pd.date_range('1970-01-01',periods=100000, freq='H')
values = np.random.randn(100000,10)
df = pd.DataFrame(values, index = index)
df.to_sql('table_name','connection_string', if_exists='append')

Problem description

Since upgrading to 0.20+ I am finding that to_sql takes twice as long. I used two environments to test the code on the same machine, below are the results:

pandas 0.20.3

CPU times: user 15.9 s, sys: 3.67 s, total: 19.6 s
Wall time: 1min 13s

pandas 0.19.2

CPU times: user 5.2 s, sys: 728 ms, total: 5.93 s
Wall time: 36.8 s

Please note that each test was performed on an empty table.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None python: 3.5.2.final.0 python-bits: 64 OS: Linux OS-release: 4.4.0-87-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.19.2 nose: None pip: 9.0.1 setuptools: 27.2.0 Cython: 0.25.2 numpy: 1.11.2 scipy: 0.18.1 statsmodels: 0.6.1 xarray: None IPython: 5.1.0 sphinx: 1.5.1 patsy: 0.4.1 dateutil: 2.6.0 pytz: 2016.10 blosc: None bottleneck: 1.2.0 tables: None numexpr: None matplotlib: 1.5.3 openpyxl: None xlrd: None xlwt: None xlsxwriter: None lxml: None bs4: None html5lib: 0.9999999 httplib2: None apiclient: None sqlalchemy: 1.1.6 pymysql: None psycopg2: 2.6.2 (dt dec pq3 ext lo64) jinja2: 2.9.4 boto: None pandas_datareader: 0.2.1

INSTALLED VERSIONS

commit: None python: 3.6.1.final.0 python-bits: 64 OS: Linux OS-release: 4.4.0-87-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.20.3 pytest: 3.0.7 pip: 9.0.1 setuptools: 27.2.0 Cython: 0.25.2 numpy: 1.13.1 scipy: 0.19.0 xarray: None IPython: 5.3.0 sphinx: 1.5.6 patsy: 0.4.1 dateutil: 2.6.1 pytz: 2017.2 blosc: None bottleneck: 1.2.1 tables: 3.3.0 numexpr: 2.6.2 feather: None matplotlib: 2.0.2 openpyxl: 2.4.7 xlrd: 1.0.0 xlwt: 1.2.0 xlsxwriter: 0.9.6 lxml: 3.7.3 bs4: 4.6.0 html5lib: 0.999 sqlalchemy: 1.1.9 pymysql: None psycopg2: 2.7.3 (dt dec pq3 ext lo64) jinja2: 2.9.6 s3fs: None pandas_gbq: None pandas_datareader: None

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Comments:7 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
gfyoungcommented, Nov 18, 2017

Judging from this conversation, it appears that we’re in the clear for this problem. Closing for the time being, but can reopen if it turns out that pandas is to blame.

0reactions
gregsifrcommented, Aug 7, 2017

Thank you for doing that. I will raise a ticket in the psycopg2 library for further investigation.

Read more comments on GitHub >

github_iconTop Results From Across the Web

What's New — pandas 0.20.3 documentation
This is a major release from 0.19.2 and includes a number of API changes, deprecations, new features, enhancements, and performance improvements along with ......
Read more >
Speeding up pandas.DataFrame.to_sql with ... - Stack Overflow
Now the code is more readable, but the upload is at least 150 times slower... Is there a way to flip the fast_executemany...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found