Spatially Enabled Dataframe - edit_features null integers uploading as 0s

Describe the bug pd.NA or np.nan values upload as 0s against integer fields (hosted tables and hosted feature layers - with null enabled).

To Reproduce Steps to reproduce the behavior:

from arcgis.gis import GIS
gis = GIS("home")

import pandas as pd

nullDF = pd.DataFrame({
  "nullString": [pd.NA],
  "nullInteger": [pd.NA],
  "nullDouble": [pd.NA]  
})

# Add 'nulDF' to AGOL table
nullFeatureSet = nullDF.spatial.to_featureset()
nullTableToEdit = gis.content.get("8966b5e40197499fa296a02950d11b6c").tables[0]
nullTableToEdit.edit_features(adds = nullFeatureSet)

error:

Link to the resulting hosted table (publicly shared with editing enabled for test purposes) - https://actgov.maps.arcgis.com/home/item.html?id=8966b5e40197499fa296a02950d11b6c#data

Expected behavior The integer field should be “null” or “” similar to the string or double fields.

Platform

• AGOL Notebok Standard and Advanced Runtimes
• MacOS ArcGIS Python api 2.0.1

[hildermesmedeiros]

You should Always handle int with extra care, first numpy does not suport casting nan to int. I mean 2010 they started arguing about int supporting nan, and here we are, long history short…you can’t do this conversion yet…

It should fail , your column would be object type if int and nan are in there.

df.colum.astype(‘int32’) #this is numpy

Howerver we can do it in Pandas.

df.column.astype(‘Int32’) #Pandas type

If you don’t care about performace you could use pandas int type, since they suport nan and the column would be:

https://pandas.pydata.org/pandas-docs/stable/user_guide/gotchas.html Int8Dtype Int16Dtype Int32Dtype Int64Dtype https://github.com/pandas-dev/pandas/blob/v1.5.1/pandas/core/arrays/integer.py#L174-L177

A field length propertie and type propertie would help you with pandas to arcgis type maping. I could have a object pandas column but in gdb, table i want it to be int…so your developers would go extra crazy trying to fix without a clue.

[nanaeaubry]

@GeeFernando We are looking into the issue on ArcPy side. Unfortunately we dug into the shapely side and we cannot fix it because it is in the shapely library’s logic to write a None int to 0. When they take the values and write them to the shapefile, it gets seen as a ‘missing’ value and is changed to 0.

I know in a previous issue you had mentioned working with shapely so I wanted to mention this. The best workaround at this moment is to cast the None values to random numbers like -1 or 9999 in order to see them as invalid.

As for the ArcPy side, we are working on a solution but since many parts of the API use this logic (like the learn module), we need to be careful of breaking changes. I will post here once we have integrated a fix for it.

One last thing, we added logic for the to_featurelayer to also publish tables now at the next version. This might help avoid using append_features and edit_features however it will create a new table each time. Just wanted to let you know.