Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Spatially Enabled Dataframe - edit_features null integers uploading as 0s

See original GitHub issue

Describe the bug pd.NA or np.nan values upload as 0s against integer fields (hosted tables and hosted feature layers - with null enabled).

To Reproduce Steps to reproduce the behavior:

from arcgis.gis import GIS
gis = GIS("home")

import pandas as pd

nullDF = pd.DataFrame({
  "nullString": [pd.NA],
  "nullInteger": [pd.NA],
  "nullDouble": [pd.NA]  

# Add 'nulDF' to AGOL table
nullFeatureSet = nullDF.spatial.to_featureset()
nullTableToEdit = gis.content.get("8966b5e40197499fa296a02950d11b6c").tables[0]
nullTableToEdit.edit_features(adds = nullFeatureSet)


Link to the resulting hosted table (publicly shared with editing enabled for test purposes) -

Expected behavior The integer field should be “null” or “” similar to the string or double fields.


  • AGOL Notebok Standard and Advanced Runtimes
  • MacOS ArcGIS Python api 2.0.1

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:8 (4 by maintainers)

github_iconTop GitHub Comments

hildermesmedeiroscommented, Oct 21, 2022

You should Always handle int with extra care, first numpy does not suport casting nan to int. I mean 2010 they started arguing about int supporting nan, and here we are, long history short…you can’t do this conversion yet…

It should fail , your column would be object type if int and nan are in there.

df.colum.astype(‘int32’) #this is numpy

Howerver we can do it in Pandas.

df.column.astype(‘Int32’) #Pandas type

If you don’t care about performace you could use pandas int type, since they suport nan and the column would be: Int8Dtype Int16Dtype Int32Dtype Int64Dtype

A field length propertie and type propertie would help you with pandas to arcgis type maping. I could have a object pandas column but in gdb, table i want it to be int…so your developers would go extra crazy trying to fix without a clue.

nanaeaubrycommented, Oct 21, 2022

@GeeFernando We are looking into the issue on ArcPy side. Unfortunately we dug into the shapely side and we cannot fix it because it is in the shapely library’s logic to write a None int to 0. When they take the values and write them to the shapefile, it gets seen as a ‘missing’ value and is changed to 0.

I know in a previous issue you had mentioned working with shapely so I wanted to mention this. The best workaround at this moment is to cast the None values to random numbers like -1 or 9999 in order to see them as invalid.

As for the ArcPy side, we are working on a solution but since many parts of the API use this logic (like the learn module), we need to be careful of breaking changes. I will post here once we have integrated a fix for it.

One last thing, we added logic for the to_featurelayer to also publish tables now at the next version. This might help avoid using append_features and edit_features however it will create a new table each time. Just wanted to let you know.

Read more comments on GitHub >

github_iconTop Results From Across the Web

edit_features null values becoming 0s - Esri Community
I am trying to use edit_features to update a hosted table with data from a pandas dataframe. The process runs without error, but...
Read more >
Spatially Enabled DataFrames - Advanced Topics
This method prints information about a DataFrame including the index dtype and column dtypes, non-null values and memory usage. Example: Displaying info on ......
Read more >
Pandas spatially enabled dataframe error. Could not load ...
Its strange to me, because it seems like the dataframe is trying to convert my floats to integers, even though I'm not doing...
Read more >
Data Engineering with Spatially Enabled DataFrames and ...
Pandas is a data analysis and data engineering library that is very popular in the scientific Python ecosystem. The ArcGIS API for Python ......
Read more >
Software Release Notes - Geocortex Viewer for HTML5 2.11.0
o Must match the spatial reference of the base map of the application ... GVH-12020 Heat map enabled on uploaded data features becomes ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found