question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

BUG: Exporting a GeoDataframe with column 'fid' to GeoPackage will delete that column

See original GitHub issue

Steps:

  1. Create a GeoDataframe with a column 'fid'

  2. Export as a GeoPackage using to_file()

  3. Read the exported GeoPackage

Bug

The 'fid' column should exist, but is missing.

Hunches

A scan of both the GeoPandas and the Fiona source code reveal no reason this bug should occur, so I suspect that it’s actually an issue with GDAL.

Issue Analytics

  • State:open
  • Created 4 years ago
  • Comments:10 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
theroggycommented, Mar 23, 2022

I forgot about this back then… but a new case where a solution for this would have made life a lot easier arrived…

So, in response to @jorisvandenbossche: I think your second option sounds great. It is safe backwards-compatibility wise, and as it is an index, it sounds reasonable that it would end up as the index of the resulting GeoDataFrame as well.

I also noticed pyogrio.read_dataframe supports it already (on the master branch, not yet in the released latest version), via a parameter called “fid_as_index=False”: https://pyogrio.readthedocs.io/en/latest/api.html#geopandas-integration

1reaction
jorisvandenbosschecommented, Jun 12, 2020

I am still a bit hesitant to always convert the id to an "fid" column, when reading from a GeoPackage file: 1) it introduces GPKG specific code in geopandas and 2) it adds an additional column for all users, also when you actually don’t use/need it.

But I certainly understand that, if you want access to the FID of the GeoPackage file, the current situation is quite annoying…

So some option I can think of:

  • Simply always include a “fid” column on GPKG file reading (but only for GPKG, I think?)
  • Have an option to indicate to preserve the id when reading (then this could be done for all formats, I think, and we don’t necessarily need to make it specific to GPK?)
  • Warn users when they write a GeoDataFrame with an “fid” column to GPKG that their column will be lost when reading back (this would be an option if we decide to not change reading, to avoid the roundtrip surprise, but of course in itself doesn’t solve the use case of accessing GPKG’s FID)

Thoughts? Somebody interested in doing a PR for one of those?

Read more comments on GitHub >

github_iconTop Results From Across the Web

How to make necessary changes to Geodataframe to export ...
it is when I am trying to convert a geodataframe which has a string/object field with lists in it to shapefile. Do you...
Read more >
Reading and Writing Files - GeoPandas
GeoDataFrames can be exported to many different standard formats using the geopandas.GeoDataFrame.to_file() method. For a full list of supported formats, type ...
Read more >
Debugging GeoPandas GeoPackage IO - kuan butts
When you write to a GPKG file and then re-read the output file back in as a GeoDataFrame, a column named fid will...
Read more >
OSMNX graph_from_gdfs KeyError: 'x' when converting a ...
Your problem appears to be that the u , v , and key columns in your edges GeoDataFrame contain null values, presumably from...
Read more >
Writing to GeoPackage fails when two attributes have the ...
Here is the error message I get with geopandas/fiona (the ... so that the resulting GeoDataFrame contained two columns with the same name ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found