question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

encoding error reading back dataframe i just saved

See original GitHub issue

When I try to read back a file saved using geopandas I get an encoding error. It works if I specify encoding=utf8 on save. However it would make more sense to fail on save. Otherwise you save some data and lose access because you cannot read it back.

gdf.to_file("popshapes.geojson", driver="GeoJSON")
gpd.read_file("popshapes.geojson")

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:9 (6 by maintainers)

github_iconTop GitHub Comments

2reactions
sgilliescommented, Jan 6, 2020

@simonm3 @martinfleis I’ve found the problem and am looking for it to be solved in GDAL/OGR: https://github.com/OSGeo/gdal/issues/2151. The way this affects fiona and GeoPandas is that this line https://github.com/Toblerity/Fiona/blob/maint-1.8/fiona/ogrext.pyx#L566 evaluates to False and so we fall back to the default encoding on your computer, which is presumably latin-1 or windows-1252.

It might be determined that fiona is misusing the OGR API, in which case we’ll need to make a special case for GeoJSON in that function.

Meanwhile, the workaround is to make sure that your environment’s default encoding is UTF-8 when you run your program.

0reactions
martinfleiscommented, Feb 19, 2022

This was most likely fixed in https://github.com/OSGeo/gdal/pull/2152. We can reopen if not.

Read more comments on GitHub >

github_iconTop Results From Across the Web

UnicodeDecodeError when reading CSV file in Pandas with ...
You know the encoding, and there is no encoding error in the file. Great: you have just to specify the encoding: file_encoding =...
Read more >
pandas.DataFrame.to_csv — pandas 1.5.2 documentation
Specifies how encoding and decoding errors are to be handled. See the errors argument for open() for a full list of options. New...
Read more >
Error - unable to read the csv file in pandas
i am unable to read the csv file. import pandas as pd df=pd.read_csv(“data.csv”). the error i am getting is :- ...
Read more >
Pandas read_csv() Tutorial: Importing Data - DataCamp
Importing Data is the first important step in any data science project. Learn how pandas' read_csv() function is perfect for this.
Read more >
11 Data import - R for Data Science - Hadley Wickham
Both functions increase the chances of the output file being read back in correctly by: Always encoding strings in UTF-8. Saving dates and...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found