OutputFileDatasetConfig results in error 'Exception: No temp file found.'

• Package Name: azureml.data.output_dataset_config.OutputFileDatasetConfig
• Package Version: azure-batch 9.0.0 azure-cli 2.16.0 azure-cli-core 2.16.0 azure-cli-telemetry 1.0.6 azure-common 1.1.26 azure-core 1.9.0 azure-cosmos 3.2.0 azure-datalake-store 0.0.51 azure-functions-devops-build 0.0.22 azure-graphrbac 0.61.1 azure-identity 1.4.1 azure-keyvault 1.1.0 azure-keyvault-administration 4.0.0b2 azure-loganalytics 0.1.0 azure-mgmt-advisor 2.0.1 azure-mgmt-apimanagement 0.2.0 azure-mgmt-appconfiguration 0.6.0 azure-mgmt-applicationinsights 0.1.1 azure-mgmt-authorization 0.61.0 azure-mgmt-batch 9.0.0 azure-mgmt-batchai 2.0.0 azure-mgmt-billing 1.0.0 azure-mgmt-botservice 0.2.0 azure-mgmt-cdn 5.2.0 azure-mgmt-cognitiveservices 6.3.0 azure-mgmt-compute 14.0.0 azure-mgmt-consumption 2.0.0 azure-mgmt-containerinstance 1.5.0 azure-mgmt-containerregistry 2.8.0 azure-mgmt-containerservice 9.4.0 azure-mgmt-core 1.2.1 azure-mgmt-cosmosdb 1.0.0 azure-mgmt-datalake-analytics 0.2.1 azure-mgmt-datalake-nspkg 3.0.1 azure-mgmt-datalake-store 0.5.0 azure-mgmt-datamigration 0.1.0 azure-mgmt-deploymentmanager 0.2.0 azure-mgmt-devtestlabs 4.0.0 azure-mgmt-dns 2.1.0 azure-mgmt-eventgrid 3.0.0rc7 azure-mgmt-eventhub 4.1.0 azure-mgmt-hdinsight 2.0.0 azure-mgmt-imagebuilder 0.4.0 azure-mgmt-iotcentral 3.0.0 azure-mgmt-iothub 0.12.0 azure-mgmt-iothubprovisioningservices 0.2.0 azure-mgmt-keyvault 2.2.0 azure-mgmt-kusto 0.3.0 azure-mgmt-loganalytics 0.7.0 azure-mgmt-managedservices 1.0.0 azure-mgmt-managementgroups 0.2.0 azure-mgmt-maps 0.1.0 azure-mgmt-marketplaceordering 0.2.1 azure-mgmt-media 2.2.0 azure-mgmt-monitor 0.11.0 azure-mgmt-msi 0.2.0 azure-mgmt-netapp 0.14.0 azure-mgmt-network 13.0.0 azure-mgmt-nspkg 3.0.2 azure-mgmt-policyinsights 0.5.0 azure-mgmt-privatedns 0.1.0 azure-mgmt-rdbms 3.1.0rc1 azure-mgmt-recoveryservices 0.4.0 azure-mgmt-recoveryservicesbackup 0.6.0 azure-mgmt-redhatopenshift 0.1.0 azure-mgmt-redis 7.0.0rc2 azure-mgmt-relay 0.1.0 azure-mgmt-reservations 0.6.0 azure-mgmt-resource 10.3.0 azure-mgmt-search 2.1.0 azure-mgmt-security 0.4.1 azure-mgmt-servicebus 0.6.0 azure-mgmt-servicefabric 0.5.0 azure-mgmt-signalr 0.4.0 azure-mgmt-sql 0.21.0 azure-mgmt-sqlvirtualmachine 0.5.0 azure-mgmt-storage 11.2.0 azure-mgmt-synapse 0.5.0 azure-mgmt-trafficmanager 0.51.0 azure-mgmt-web 0.48.0 azure-multiapi-storage 0.5.2 azure-nspkg 3.0.2 azure-storage-blob 12.6.0 azure-storage-common 1.4.2 azure-storage-queue 12.1.4 azure-synapse-accesscontrol 0.2.0 azure-synapse-artifacts 0.3.0 azure-synapse-spark 0.2.0 azureml-accel-models 1.19.0 azureml-automl-core 1.19.0 azureml-automl-runtime 1.19.0 azureml-cli-common 1.19.0 azureml-contrib-dataset 1.19.0 azureml-contrib-fairness 1.19.0 azureml-contrib-gbdt 1.19.0 azureml-contrib-interpret 1.19.0 azureml-contrib-notebook 1.19.0 azureml-contrib-pipeline-steps 1.19.0 azureml-contrib-reinforcementlearning 1.19.0 azureml-contrib-server 1.19.0 azureml-contrib-services 1.19.0 azureml-core 1.19.0 azureml-datadrift 1.19.0 azureml-dataprep 2.6.1 azureml-dataprep-native 26.0.0 azureml-dataprep-rslex 1.4.0 azureml-dataset-runtime 1.19.0 azureml-defaults 1.19.0 azureml-explain-model 1.19.0 azureml-interpret 1.19.0 azureml-mlflow 1.19.0 azureml-model-management-sdk 1.0.1b6.post1 azureml-opendatasets 1.19.0 azureml-pipeline 1.19.0 azureml-pipeline-core 1.19.0 azureml-pipeline-steps 1.19.0 azureml-samples 0+unknown azureml-sdk 1.19.0 azureml-telemetry 1.19.0 azureml-tensorboard 1.19.0 azureml-train 1.19.0 azureml-train-automl 1.19.0 azureml-train-automl-client 1.19.0 azureml-train-automl-runtime 1.19.0 azureml-train-core 1.19.0 azureml-train-restclients-hyperdrive 1.19.0 azureml-widgets 1.19.0
• Operating System: Ubuntu (Default Azure ML Studio Env)
• Python Version: 3.6.9

Describe the bug When using OutputFileDatasetConfig, sometimes the job fails with error

Exception: No temp file found.

To Reproduce Steps to reproduce the behavior:

1. Create a pipeline using the Python SDK and ParallelRunStep
2. Use OutputFileDatasetConfig for output
3. Run Pipeline

Expected behavior For the pipeline to run without error.

Screenshots If applicable, add screenshots to help explain your problem. N/A.

Additional context Detailed StackOverflow question I posted when I was having issues with OutputFileDatasetConfig: https://stackoverflow.com/questions/65240603/azure-ml-studio-ml-pipeline-exception-no-temp-file-found

[GuXiaoMing]

Hi @yeamusic21, that’s a known issue already fixed in all regions since 1/17/2021. And the original context in stackoverflow shows that all cases occurred in 2020. I’ll close the issue since it’s fixed and no more occurrence now. Thanks for raising the problem!

[ChunyuMSFT]

Hi @yeamusic21 , thank you for your feedback, in current version, OutputDatasetConfig can’t work with ParallelRunStep, we are working on fixing it.