Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

`to_featureset` does not preserve datetime

See original GitHub issue

Describe the bug When attempting to append data by converting features in a spatially enabled dataframe to a featureset (using to_featureset()) and then edit_features to update an AGOL layer, datetime attrributes are converted to unix timestamps. If an AGOL layer contains date columns, then that data is not appended to the hosted feature service.

The code below is connected to another github issue.

To Reproduce Steps to reproduce the behavior:

## pull layer to update
item_id = ## item_id
to_update = gis.content.get(itemid=item_id) 
to_update_layer = to_update.layers[0]

## add new data
format = "%Y-%m-%d %H:%M:%S"

df = pd.DataFrame([
    {'test' : 'update4',
     'longitude' : -117.15246191281217,
     'latitude' : 32.70969794547845,
     'date' : today, #str(
     {'test' : 'update5',
     'longitude' : -115.15246191281217,
     'latitude' : 32.70969794547845,
     'date' : today, #str(
     {'test' : 'update6',
     'longitude' : -117.15246191281217,
     'latitude' : 34.70969794547845,
     'date' : today, #str(

## create timezone cols
df['date_datetime'] = pd.to_datetime(df['date'])
df['date_localized_utc'] = df['date_datetime'].dt.tz_localize('UTC')
df['date_localized_est'] = df['date_localized_utc'].dt.tz_convert('US/Eastern')
df['date_localized_pst'] = df['date_localized_utc'].dt.tz_convert('US/Pacific')
df['date_datetime_epoch'] = df['date_datetime'].map(pd.Timestamp.timestamp)

update_with = pd.DataFrame.spatial.from_xy(df=df, x_column='longitude', y_column='latitude', sr=4326)
print(list(update_with.dropna(how='all', axis=1).spatial.to_featureset()))

## update data
results = []
for i, row in update_with.iterrows():
        featureset = update_with.iloc[i:i+1].dropna(how='all', axis=1).spatial.to_featureset()
        result = to_update.layers[0].edit_features(adds=featureset)
        print(f'Feature {i} not added')

Screenshots image

Expected behavior Datetime columns are maintained as dates and appended to AGOL successfully (this just may not be possible via to_featureset / edit_features; if so, please let me know!😄).

Platform (please complete the following information):

  • OS: Windows
  • Browser Chrome/Firefox
  • Python API Version 2.0.0

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:9 (4 by maintainers)

github_iconTop GitHub Comments

Ray-Mangancommented, Nov 15, 2022

For the column truncation I am not sure why Shapefile does that. If I find anything out about it or a way around I will let you know.

@nanaeaubry - I know this issue is closed but the column truncation issue is caused by the shapefile format. The format uses .dbf as the datatable storage format and has a 10 character limit. See the limitations section part way down the page here:

nanaeaubrycommented, Aug 1, 2022

Great to hear that it works! Creating a shapefile occurs when arcpy is not in your environment. I assumed it was which is why I spoke about fgdb but if it is not then we use the Shapely library and create a shapefile instead so there is no workflow difference just the result is a Feature Layer and a Shapefile instead.

For the column truncation I am not sure why Shapefile does that. If I find anything out about it or a way around I will let you know.

I will go ahead and close this issue since we were able to resolve one problem and the other is linked to the other issue you have open. You can still post here if needed but for timezone updates they will be on your other issue 😃 Thanks!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Changes to time zone info? (Appending to feature s...
My workflow has been to add new records to a local geodatabase (because that preserves the subtype-based default values), and then append them ......
Read more >
When writing FME DateTime to ESRI FGDB How can I retain ...
I have a workbench which builds FME date time attributes and writes them to ESRI FGDB Classes. So, for 20191009115225 I get 09/10/2019 ......
Read more >
ArcGIS: Working with Date-Time Data - YouTube
This video demonstrates how data containing dates and times are handled and represented in ArcGIS. You will learn best practices for sharing ...
Read more >
Incident Response - Hybrid Analysis
This report is generated from a file or URL submitted to this webservice on March 5th 2020 ... Not all malicious and suspicious...
Read more >
FeatureSet, DotSpatial.Data C# (CSharp) Code Examples
These are the top rated real world C# (CSharp) examples of DotSpatial.Data. ... //The current version does not preserve any attribute info.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found