question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Definitions created using 'az synapse spark-job-definition create' do not show in the Workspace UI with Git Integration enabled

See original GitHub issue

Related command az synapse spark-job-definition create --workspace-name <workspace> --name <job-name> --file @"folder\sparkjob.json"

Describe the bug When using ‘az synapse spark-job-definition create’ command I am able to create a sparkjob definition, however that definition is not then visible in the Synapse Workspace UI. I am however able to see it when calling ‘az synapse spark-job-definition list’

I have Git integration enabled on that workspace, checked all the branches and can’t find the job definition I’ve created in the UI.

To Reproduce

  1. Export existing job definition from Synapse UI
  2. Change jobProperties.name to be unique.
  3. Run `az synapse spark-job-definition create --workspace-name <workspace> --name <job-name> --file @“folder\sparkjob.json” using the previously exported file.
  4. Run az synapse spark-job-definition list --workspace-name <workspace> and verify newly create job definition is visible.
  5. Check UI.

Expected behavior The created spark job definition is also visible in Synapse Workspace UI

Environment summary

Windows-10-10.0.19044-SP0
Python 3.10.5
Installer: MSI

azure-cli 2.38.0

Extensions:
azure-cli-ml 1.41.0

Dependencies:
msal 1.18.0b1
azure-mgmt-resource 21.1.0b1

Additional context

Issue Analytics

  • State:open
  • Created a year ago
  • Comments:16 (8 by maintainers)

github_iconTop GitHub Comments

1reaction
kevinzz6commented, Aug 17, 2022

Hi @alexszym, I tried to use cli to create other artifacts like SQL script with git enable and there is the same issue as you mentioned above. But something like spark pools that are managed by “azure-mgmt-synapse” module are good. So, I think this issue is related to git CI/CD background.

1reaction
RakeshMohanMSFTcommented, Jul 29, 2022

@alexszym Let me engage the synapse engineering team (CLI), before we resort to support.

Read more comments on GitHub >

github_iconTop Results From Across the Web

az synapse spark-job-definition - Microsoft Learn
Manage spark job definitions in a synapse workspace. In this article. Commands; az synapse spark-job-definition create; az synapse spark-job-definition delete ...
Read more >
Azure Synapse Analytics - Git Integration and CI/CD
Azure Synapse Analytics is a data services tool that provides data integration, warehousing, and analytical functionality.
Read more >
Synapse – Choosing Between Spark Notebook vs Spark Job ...
Introduction Apache Spark applications are used by businesses to perform ... You can integrate spark job definition in the Synapse pipeline.
Read more >
azurerm_synapse_workspace | Resources | hashicorp/azurerm
Example Usage - creating a workspace with Customer Managed Key and Azure AD ... the name of the Resource Group where the synapse...
Read more >
Access issue with Azure Synapse studio - Stack Overflow
This could be an intermittent issue while opening synapse workspace. Could you please confirm the permission on the Synapse workspace which ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found