big integer i/o handling

If I create a GeoDataFrame like this:

import geopandas as gpd
import pandas as pd
from shapely.geometry import Point
d = {'id' : [3694063472, 3694063473],
     'geometry' : [Point((-122.278015,37.868995)),
                   Point((-122.278876,37.868895))]}
gdf = gpd.GeoDataFrame(d)
gdf

Returns:

			     geometry	        id
0	POINT (-122.278015 37.868995)	3694063472
1	POINT (-122.278876 37.868895)	3694063473

If I save to csv as a pandas Dataframe and read it back in:

pd.DataFrame(gdf).to_csv('test.csv', index=False)
pd.read_csv('test.csv')

Returns:

			     geometry	        id
0	POINT (-122.278015 37.868995)	3694063472
1	POINT (-122.278876 37.868895)	3694063473

Everything’s the same. But if I save to disk as a shapefile:

gdf.to_file('test_shp')
gpd.read_file('test_shp')

Returns:

	        id			     geometry
0	2147483647	POINT (-122.278015 37.868995)
1	2147483647	POINT (-122.278876 37.868895)

Writing my GeoDataFrame to disk and reading it back in results in my id field’s values silently being changed to 2147483647, the max value for a 32-bit signed integer. This is in geopandas 0.3.0 with fiona 1.7.11.

[jdmcbr]

This is fixed on fiona master now.

In [1]: gdf = gpd.read_file(gpd.datasets.get_path("nybb"))

In [2]: gdf["big_int"] = arange(5) + 12345678910

In [3]: gdf.to_file("test.shp")

In [4]: gdf_back = gpd.read_file("test.shp")

In [5]: gdf_back["big_int"]
Out[5]: 
0    12345678910
1    12345678911
2    12345678912
3    12345678913
4    12345678914
Name: big_int, dtype: int64

[sgillies]

@jorisvandenbossche I’ve got a solution (for Fiona 1.8) coming up. Meanwhile, I suggest this mapping of Python/Numpy types to Fiona/GDAL types:

Python/Numpy : Fiona/GDAL
-------------------------
int, int64, long : 'int:18'
int32 : 'int:9'
bool : 'bool'