KeyError: 'type' when publishing CSV to Enterprise Portal

I have a CSV file that I cannot publish to ArcGIS Enterprise Portal from a ArcGIS Pro Notebook. I have a stand-alone script that requests JSON from a URL and writes it to a CSV. The content has no special characters that I’m aware of. When opening the CSV from my stand-alone script, I encoded it to utf-8. But, when subsequently attempting to publish to the Portal I get the KeyError. I’m suspecting it’s a character encoding issue in the CSV, but I’m in no way certain.

To Reproduce I’m not sure if this is necessary but here is my script that writes to the CSV. It’s completely reproducible assuming you have the relevant modules.

import json,requests
import os,sys
import csv

#change file path to new directory
path_to_file = os.path.join(r'C:\Users\jpilbeam\Downloads', 'c19_Vaccine_Current.csv')
#idph url goes here
idph_data = 'https://idph.illinois.gov/DPHPublicInformation/api/covidVaccine/getVaccineAdministrationCurrent'

##get json data from idph
response = requests.get(idph_data,verify=True)
#read the json response and keep the VaccineAdministration part
data = response.json()['VaccineAdministration']

#write to file
with open(path_to_file, 'w', newline='', encoding='UTF-8') as csvfile:
    f = csv.writer(csvfile) 
    #write the headers of the csv file
    f.writerow(['County','AdminCount','AdminCountChange', 'RollAvg', 'AllocDoses', 'FullyVaccinated', 
                'FullyVaccinatedChange', 'ReportDate', 'Pop', 'PctVaccinated', 'LHDInventory', 'CommInventory',
                'TotalInventory', 'InventoryDate'])
    for elem in data:
        #get the values for all the keys (i.e. CountyName, AdministeredCount, etc...)
        f.writerow([elem['CountyName'], elem['AdministeredCount'], elem['AdministeredCountChange'], 
                    elem['AdministeredCountRollAvg'], elem['AllocatedDoses'], 
                    elem['PersonsFullyVaccinated'], elem['PersonsFullyVaccinatedChange'], 
                    elem['Report_Date'], elem['Population'], elem['PctVaccinatedPopulation'], 
                    elem['LHDReportedInventory'], elem['CommunityReportedInventory'], 
                    elem['TotalReportedInventory'], elem['InventoryReportDate']])>

In a Pro Notebook I then run the following code on the resulting CSV. It’s when I try publishing that I get the KeyError.

from IPython.display import display
from arcgis.gis import GIS
import os
gis = GIS('Home') #uses current Notebook as workspace

csv_file = r'C:\Users\jpilbeam\Downloads\c19_Vaccine_Current.csv' #path to CSV
csv_item = gis.content.add({}, csv_file) #add CSV to Enterprise Portal
display(csv_item) #display it here 

#location parameter needs to be set to "none" when publishing to Enterprise Portal
params={"type":"csv","locationType":"none"} 
csv_item.publish(publish_parameters=params) #publish to Enterprise Portal

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
In  [7]:
Line 3:     csv_item.publish(publish_parameters=params) #publish to Enterprise Portal

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py, in publish:
Line 10353: elif not buildInitialCache and ret[0]['type'].lower() == 'image service':

KeyError: 'type'
--------------------------------------------------------------------------->

If it makes any difference I was able to publish only by opening the CSV in Notepad --> setting Encoding to utf-8 --> giving the filename a *.csv extension --> then saving. Needless to say, I don’t intend to go this route.

Expected behavior I expected the CSV to publish once I set the encoding to utf-8 in the with open() as csvfile part. I’ve attached my posts from GeoNet and StackExchange relating to this issue. It shows how I’ve come to this point.

Platform

• OS: Windows 10
• Browser FireFox
• Python API Version 1.8.4

Additional context My related posts on the issue: https://gis.stackexchange.com/questions/390282/encode-csv-to-utf-8-before-publishing-to-enterprise-portal https://community.esri.com/t5/arcgis-api-for-python-questions/publish-csv-to-enterprise-portal-errors/td-p/1035018 https://community.esri.com/t5/arcgis-api-for-python-questions/exception-job-failed-when-publishing-csv-item/m-p/1036154

[UnderJollyRoger]

@achapkowski @Detteor After some additional help from an ESRI analyst I have a more or less stable script to publish a CSV to Portal as a feature service. And because it’s a reoccurring process there’s an additional script that overwrites the feature service. See this post.

[Detteor]

@UnderJollyRoger

@Detteor By the way, this is from an ESRI blog: it is not recommended that you have both Anaconda and ArcGIS Pro installed on a single machine

This Definitely might have been the case when using ArcGIS Pro 2.2, but now with the metapackages and being able to install arcpy in an environment it doesn’t seem as impactful anymore. ENV and PATH variables are always a pain in windows, so I don’t even let conda set its own variables. Also, they don’t want you to mess with the base install since that can cause a lot of issues, this is one of the reasons that they don’t let you change the base install for ArcGIS Pro.

@Detteor Good to know. I’m already using a conda environment in Pro 2.7.2 as fas as I can tell. I didn’t bother going through the Anaconda site, as I installed the API with Pro. I’m connected to the same environment in my IDE. This is from Visual Studio 2019.

When I start to work on new projects, I always use a new environment that has the latest package versions. I actually never touch the conda environment that comes with ArcGIS Pro since I have found that it causes a lot of problems. I do now that the version of ArcGIS API that comes with ArcGIS Pro is locked at 1.8.3 due to compatibility issues. You might want to create a whole new environment that isn’t going to be used in ArcGIS Pro and that will only be used in your IDE.

You can create an env.yml file that you can use to install the packages that you want in your new environment. You will need to activate your environment and type conda env export > env.yml, this will save a yml file in your current working directory of the packages currently installed. You can also use conda env export --from-history > env.yml, this will export only the packages that you have installed.Then you can use conda create -n myenv python=3.7 use 3.7 since that is what arcpy is running in ArcGIS Pro.