question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

df.to_stata should automatically write in format 117 with wide strings

See original GitHub issue

Code Sample, a copy-pastable example if possible

import pandas as pd
df = pd.DataFrame({'a': ['x' * 250]})
df.to_stata('test')

Problem description

The last line above raises an exception:

ValueError:
Fixed width strings in Stata .dta files are limited to 244 (or fewer)
characters.  Column 'a' does not satisfy this restriction.

but is solved with:

df.to_stata('test', version=117)

This functionality (writing in dta format 117) was added in version 0.23. In my opinion, the Stata writer should automatically switch to version 117 if one of the columns is wider than 244 characters. At the least, the error message should be changed to note that as of version 0.23, it’s possible to write long strings to Stata files by adding version=117.

I’d be happy to submit a PR if this functionality is desired.

Expected Output

Stata file written to disk.

Output of pd.show_versions()

pd.show_versions()
No module named 'dask'

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.7.final.0
python-bits: 64
OS: Darwin
OS-release: 18.2.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.0.dev0+948.g82120016e
pytest: 3.10.0
pip: 18.1
setuptools: 40.5.0
Cython: 0.29
numpy: 1.15.4
scipy: 1.1.0
pyarrow: 0.11.1
xarray: 0.10.9
IPython: 7.1.1
sphinx: 1.8.1
patsy: 0.5.1
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: 1.2.1
tables: 3.4.4
numexpr: 2.6.8
feather: None
matplotlib: 3.0.1
openpyxl: 2.5.9
xlrd: 1.1.0
xlwt: 1.2.0
xlsxwriter: 1.1.2
lxml: 4.2.5
bs4: 4.6.3
html5lib: 1.0.1
sqlalchemy: 1.2.13
pymysql: 0.9.2
psycopg2: None
jinja2: 2.10
s3fs: 0.1.6
fastparquet: 0.1.6
pandas_gbq: None
pandas_datareader: None
gcsfs: 0.1.2

cc: @bashtage

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:16 (15 by maintainers)

github_iconTop GitHub Comments

1reaction
gfyoungcommented, Nov 8, 2018

@bashtage : Deprecation cycle also works. However, keep in mind that the proposal of auto-changing to 117 would have been accompanied by a warning message if we had to auto-convert, for transparency reasons (anti-magic) as you have eluded.

0reactions
kylebarroncommented, Nov 20, 2018

You probably need to update pandas

Read more comments on GitHub >

github_iconTop Results From Across the Web

Pandas DataFrame: to_stata() function - w3resource
Writes the DataFrame to a Stata dataset file. ... Storing strings in the StrL format can produce smaller dta files if strings have...
Read more >
pandas.DataFrame.to_stata — pandas 0.23.1 documentation
List of column names to convert to string columns to Stata StrL format. Only available if version is 117. Storing strings in the...
Read more >
reshape — Convert data from wide to long form and vice versa
reshape will determine them automatically from the data. string specifies that j() may contain string values. atwl(chars), available only with the advanced ...
Read more >
10.12 Stata Format — Pandas Doc
10.12.1 Writing to Stata format. The method to_stata() will write a DataFrame into a .dta file. The format version of this file is...
Read more >
readstata13: Import 'Stata' Data Files
Description Function to read and write the 'Stata' file format. ... Strings can be converted from Windows-1252 or UTF-8 to system encoding.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found