GeoDataFrame.to_file() raises ValueError: Invalid field type <class 'numpy.int64'>

Hi,

this is a problem I mentioned on the gis ML without answer so far: http://lists.gispython.org/pipermail/community/2015-April/003382.html

Here again the mwe. The shapefile contains only one entity. If someone ever wants to try it out, the shapefile can be downloaded here:

import geopandas as gpd

ifile = 'Hintereisferner.shp'
gdf = gpd.GeoDataFrame.from_file(ifile)

for index, entity in gdf.iterrows():
     # no problem here
     gpd.GeoDataFrame(entity).T.to_file('test1.shp')

# This raises an error
entity = gdf.iloc[0]
gpd.GeoDataFrame(entity).T.to_file('test2.shp')

The error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-1d31c9cdd3e2> in <module>()
       7
       8 entity = gdf.iloc[0]
----> 9 gpd.GeoDataFrame(entity).T.to_file('test2.shp')

/home/mowglie/.pyvirtualenvs/py3.3/lib/python3.3/site-packages/geopandas/geodataframe.py 
in to_file(self, filename, driver, **kwargs)
     303                         schema=schema, **kwargs) as c:
     304             for i, row in self.iterrows():
--> 305                 c.write(feature(i, row))
     306
     307     def to_crs(self, crs=None, epsg=None, inplace=False):

/home/mowglie/.pyvirtualenvs/py3.3/lib/python3.3/site-packages/fiona/collection.py 
in write(self, record)
     323     def write(self, record):
     324         """Stages a record for writing to disk."""
--> 325         self.writerecords([record])
     326
     327     def validate_record(self, record):

/home/mowglie/.pyvirtualenvs/py3.3/lib/python3.3/site-packages/fiona/collection.py 
in writerecords(self, records)
     317         if self.mode not in ('a', 'w'):
     318             raise IOError("collection not open for writing")
--> 319         self.session.writerecs(records, self)
     320         self._len = self.session.get_length()
     321         self._bounds = self.session.get_extent()

/home/mowglie/.pyvirtualenvs/py3.3/lib/python3.3/site-packages/fiona/ogrext.cpython-33m.so 
in fiona.ogrext.WritingSession.writerecs (fiona/ogrext.c:15333)()

/home/mowglie/.pyvirtualenvs/py3.3/lib/python3.3/site-packages/fiona/ogrext.cpython-33m.so 
in fiona.ogrext.OGRFeatureBuilder.build (fiona/ogrext.c:5751)()

ValueError: Invalid field type <class 'numpy.int64'>

What’s striking me is that I don’t see any difference between the entity object and the object returned by iloc…

[jorisvandenbossche]

What is actually passed to fiona is the output of GeoDataFrame.iterfeatures (https://github.com/geopandas/geopandas/blob/master/geopandas/io/file.py#L62).

Using your example:

In [45]: index, entity = next( gdf.iterrows())

In [46]: new1 = gpd.GeoDataFrame(entity).T

In [47]: new2 = gpd.GeoDataFrame(gdf.iloc[0]).T

In [48]: feat1 = next(new1.iterfeatures())

In [49]: feat2 = next(new2.iterfeatures())

In [50]: feat1['properties']
Out[50]:
{u'AREA': 8.036,
 u'ASPECT': 90L,
 u'BGNDATE': u'20030799',
 u'CENLAT': 46.8003,
 u'CENLON': 10.7584,
 u'ENDDATE': u'20030999',
 u'GLACTYPE': u'9099',
 u'GLIMSID': u'G010758E46800N',
 u'LMAX': 7178L,
 u'NAME': None,
 u'O1REGION': 11L,
 u'O2REGION': 1L,
 u'RGIFLAG': u'000',
 u'RGIID': u'RGI40-11.00897',
 u'SLOPE': 16.2,
 u'ZMAX': 3673L,
 u'ZMED': 3050L,
 u'ZMIN': 2435L}

In [51]: for prop in feat1['properties']:
    ...:     print(type(feat1['properties'][prop]))
    ...:
<type 'long'>
<type 'float'>
<type 'float'>
<type 'unicode'>
<type 'NoneType'>
<type 'unicode'>
<type 'float'>
<type 'long'>
<type 'unicode'>
<type 'unicode'>
<type 'long'>
<type 'long'>
<type 'unicode'>
<type 'long'>
<type 'float'>
<type 'long'>
<type 'unicode'>
<type 'long'>

In [52]: for prop in feat2['properties']:
    ...:     print(type(feat2['properties'][prop]))
    ...:
<type 'numpy.int64'>
<type 'numpy.float64'>
<type 'numpy.float64'>
<type 'unicode'>
<type 'NoneType'>
<type 'unicode'>
<type 'numpy.float64'>
<type 'numpy.int64'>
<type 'unicode'>
<type 'unicode'>
<type 'numpy.int64'>
<type 'numpy.int64'>
<type 'unicode'>
<type 'numpy.int64'>
<type 'numpy.float64'>
<type 'numpy.int64'>
<type 'unicode'>
<type 'numpy.int64'>

So the problem for fiona is that in the properties numpy scalars are passed instead of native python types.

[fmaussion]

Here is the fiona issue: https://github.com/Toblerity/Fiona/issues/365

Closing this now, and thanks again for the help @jorisvandenbossche