Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Spatially Enabled Data Frame not working with pandas datetime

See original GitHub issue

Describe the bug Attempting sdf = pd.DataFrame.spatial.from_layer() returns a warning “FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime instead.” and gives me “Exception: Could not load the dataset: dtype ‘<class ‘pandas.__DatetimeSub’>’ not understood”

Are there any workarounds to downloading AGOL data with Python API? sdf = SpatialDataFrame.from_layer() is the deprecated version which also returns an error.

I’ve been running into an issue much like what’s happened here (code is essentially the same):

To Reproduce Steps to reproduce the behavior:

from arcgis.gis import *
from arcgis.features import GeoAccessor, GeoSeriesAccessor

gis = GIS("", ***, ***)
item = gis.content.get('The item id')
QB = item.layers[0]

# create a Spatially Enabled DataFrame object
sdf = pd.DataFrame.spatial.from_layer(QB)


TypeError                                 Traceback (most recent call last)
~\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone1\lib\site-packages\arcgis\features\geo\ in from_layer(layer)
   1966             from arcgis.features.geo._io.serviceops import from_layer
-> 1967             return from_layer(layer=layer)
   1968         except ImportError:

~\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone1\lib\site-packages\arcgis\features\geo\_io\ in from_layer(layer, query)
    122     else:
--> 123         return layer.query(where=query, as_df=True)#.sdf
    124     return res

~\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone1\lib\site-packages\arcgis\features\ in query(self, where, out_fields, time_filter, geometry_filter, return_geometry, return_count_only, return_ids_only, return_distinct_values, return_extent_only, group_by_fields_for_statistics, statistic_filter, result_offset, result_record_count, object_ids, distance, units, max_allowable_offset, out_sr, geometry_precision, gdb_version, order_by_fields, out_statistics, return_z, return_m, multipatch_option, quantization_parameters, return_centroid, return_all_records, result_type, historic_moment, sql_format, return_true_curves, return_exceeded_limit_features, as_df, **kwargs)
    735                 import pandas as pd
--> 736                 df = self._query_df(url, params)
    737                 dt_fields = [fld['name'] for fld in \

~\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone1\lib\site-packages\arcgis\features\ in _query_df(self, url, params)
   1478                     df.loc[q, fld['name']] = 0
-> 1479         df = df.astype(dtypes, False)
   1480         if 'geometryType' in featureset_dict:

~\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone1\lib\site-packages\pandas\core\ in astype(self, dtype, copy, errors)
   5530                     results.append(
-> 5531                         col.astype(dtype=dtype[col_name], copy=copy, errors=errors)
   5532                     )

~\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone1\lib\site-packages\pandas\core\ in astype(self, dtype, copy, errors)
   5545             # else, only a single dtype is given
-> 5546             new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors,)
   5547             return self._constructor(new_data).__finalize__(self, method="astype")

~\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone1\lib\site-packages\pandas\core\internals\ in astype(self, dtype, copy, errors)
    594     ) -> "BlockManager":
--> 595         return self.apply("astype", dtype=dtype, copy=copy, errors=errors)

~\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone1\lib\site-packages\pandas\core\internals\ in apply(self, f, align_keys, **kwargs)
    405             else:
--> 406                 applied = getattr(b, f)(**kwargs)
    407             result_blocks = _extend_blocks(applied, result_blocks)

~\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone1\lib\site-packages\pandas\core\internals\ in astype(self, dtype, copy, errors)
    547         if dtype is not None:
--> 548             dtype = pandas_dtype(dtype)

~\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone1\lib\site-packages\pandas\core\dtypes\ in pandas_dtype(dtype)
   1777     elif npdtype.kind == "O":
-> 1778         raise TypeError(f"dtype '{dtype}' not understood")

TypeError: dtype '<class 'pandas.__DatetimeSub'>' not understood

During handling of the above exception, another exception occurred:

Exception                                 Traceback (most recent call last)
<ipython-input-48-b148856d5a67> in <module>
      1 # Convert to Spatial Data Frame to make a local data source for processing
----> 2 sdf = pd.DataFrame.spatial.from_layer(QB)
      3 #sdf.to_featureclass("QB_Output")
      5"C:\Data\GIS_Projects_Local\ACHRIS\AGOL_to_QB") #downloads as 'file', not readable

~\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone1\lib\site-packages\arcgis\features\geo\ in from_layer(layer)
   1971             raise Exception("Malformed response from server, could not load the dataset: %s" % str(je))
   1972         except Exception as e:
-> 1973             raise Exception("Could not load the dataset: %s" % str(e))
   1974     #----------------------------------------------------------------------
   1975     @staticmethod

Exception: Could not load the dataset: dtype '<class 'pandas.__DatetimeSub'>' not understood

Platform (please complete the following information):

  • Browser Chrome
  • Python API Version 1.5.3

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:5 (2 by maintainers)

github_iconTop GitHub Comments

AtmaManicommented, Dec 15, 2020

If you are using Pro, you may have to clone your environment and then update the API. Start the ‘Pro Python command prompt’ with elevated privileges (as administrator) and run

conda create --name myclone --clone arcgispro-py3

then activate it

conda activate myclone

then install Python API. You may have to unpin as shown below:

conda upgrade --no-pin arcgis
mw1mcommented, Jan 17, 2021

Thanks for your help @priyankatuteja and @AtmaMani Updating using the conda upgrade --no-pin arcgis ended up doing the trick!

Read more comments on GitHub >

github_iconTop Results From Across the Web

ArcGIS API for Python Spatially Enabled Data Frame...
I am trying to create a Spatially Enabled Data Frame object from a FeatureLayer object using the ArcGIS API for python version 1.5.3...
Read more >
Pandas spatially enabled dataframe error. Could not load ...
I did find a work-around. The layer has a method for sdf of which I wasn't aware. Instead of: agol_df = pd.DataFrame.spatial.from_layer(fLayer).
Read more >
Introduction to the Spatially Enabled DataFrame
This document outlines some fundamentals of using the Spatially Enabled DataFrame object for working with GIS data. import pandas as pd from arcgis.features ......
Read more >
Getting to Know Pandas and the Spatial Enabled DataFrame
This presentation will cover modern Pandas best practices by using examples as well as how to use the new Spatially Enabled DataFrame ......
Read more >
Updating Data in an Enterprise GIS with Python and Pandas
I read that into a pandas dataframe, then used the GeoAccessor to create a ... and enable decision-makers to look at their issues...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found