eco-system terraform fails with 'Invalid provider argument'

I’m trying to setup Dependabot to monitor a Terraform-project that is hosted in Azure DevOps. The container fails with the message " Error: Invalid provider argument" which is output of the command “terraform providers lock” that is executed by Dependabot-Core.

Running Dependabot in GitHub using the same repo, works as expected.

I have created a test setup do reproduce the error: Test-repository in Azure DevOps: https://dev.azure.com/humanprinter/AzureDependabot/_git/terraformtest Test-repository in GitHub: https://github.com/HumanPrinter/terraformtest

Running ‘dependabot-azure-devops’

Command

$ docker run --rm -i \
	-e DEPENDABOT_PACKAGE_MANAGER=terraform \
	-e DEPENDABOT_OPEN_PULL_REQUESTS_LIMIT=5 \
	-e DEPENDABOT_DIRECTORY=terraform/container_registry \
	-e DEPENDABOT_CLOSE_PULL_REQUESTS=false \
	-e AZURE_ORGANIZATION=humanprinter \
	-e AZURE_PROJECT=AzureDependabot \
	-e AZURE_REPOSITORY=terraformtest \
	-e AZURE_ACCESS_TOKEN=xxxxxx \
	-e AZURE_MERGE_STRATEGY=squash \
	tingle/dependabot-azure-devops:latest

Output

warning: parser/current is loading parser/ruby31, which recognizes3.1.3-compliant syntax, but you are running 3.1.2.
Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.
Using 'https://dev.azure.com:443/' as API endpoint
Working in humanprinter/AzureDependabot/_git/terraformtest, 'default' branch under 'terraform/container_registry' directory
Looking for configuration file in the repository ...
GET https://dev.azure.com/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest
200  ://dev.azure.com:443/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest
GET https://dev.azure.com/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/stats/branches?name=master
200  ://dev.azure.com:443/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/stats/branches
GET https://dev.azure.com/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items?path=.github/dependabot.yml&versionDescriptor.versionType=commit&versionDescriptor.version=7a53efb1751859edafca08686df1ff1d45f00d31
404  ://dev.azure.com:443/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items
GET https://dev.azure.com/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items?path=.github/dependabot.yaml&versionDescriptor.versionType=commit&versionDescriptor.version=7a53efb1751859edafca08686df1ff1d45f00d31
404  ://dev.azure.com:443/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items
Configuration file was not found, a default config will be used. 😔
Fetching terraform dependency files ...
GET https://dev.azure.com/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest
200  ://dev.azure.com:443/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest
GET https://dev.azure.com/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/stats/branches?name=master
200  ://dev.azure.com:443/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/stats/branches
GET https://dev.azure.com/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items?path=terraform/container_registry&versionDescriptor.versionType=commit&versionDescriptor.version=7a53efb1751859edafca08686df1ff1d45f00d31
200  ://dev.azure.com:443/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items
GET https://dev.azure.com/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/trees/c635ae408ae30b017ad01cce1e01f5854ebd1671?recursive=false
200  ://dev.azure.com:443/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/trees/c635ae408ae30b017ad01cce1e01f5854ebd1671
GET https://dev.azure.com/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items?path=terraform/container_registry/azuredevops_serviceendpoint.tf&versionDescriptor.versionType=commit&versionDescriptor.version=7a53efb1751859edafca08686df1ff1d45f00d31
200  ://dev.azure.com:443/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items
GET https://dev.azure.com/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items?path=terraform/container_registry/configuration.tf&versionDescriptor.versionType=commit&versionDescriptor.version=7a53efb1751859edafca08686df1ff1d45f00d31
200  ://dev.azure.com:443/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items
GET https://dev.azure.com/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items?path=terraform/container_registry/container_registry.tf&versionDescriptor.versionType=commit&versionDescriptor.version=7a53efb1751859edafca08686df1ff1d45f00d31
200  ://dev.azure.com:443/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items
GET https://dev.azure.com/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items?path=terraform/container_registry/external_resources.tf&versionDescriptor.versionType=commit&versionDescriptor.version=7a53efb1751859edafca08686df1ff1d45f00d31
200  ://dev.azure.com:443/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items
GET https://dev.azure.com/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items?path=terraform/container_registry/locals.tf&versionDescriptor.versionType=commit&versionDescriptor.version=7a53efb1751859edafca08686df1ff1d45f00d31
200  ://dev.azure.com:443/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items
GET https://dev.azure.com/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items?path=terraform/container_registry/main.tf&versionDescriptor.versionType=commit&versionDescriptor.version=7a53efb1751859edafca08686df1ff1d45f00d31
200  ://dev.azure.com:443/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items
GET https://dev.azure.com/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items?path=terraform/container_registry/outputs.tf&versionDescriptor.versionType=commit&versionDescriptor.version=7a53efb1751859edafca08686df1ff1d45f00d31
200  ://dev.azure.com:443/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items
GET https://dev.azure.com/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items?path=terraform/container_registry/resourcegroup.tf&versionDescriptor.versionType=commit&versionDescriptor.version=7a53efb1751859edafca08686df1ff1d45f00d31
200  ://dev.azure.com:443/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items
GET https://dev.azure.com/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items?path=terraform/container_registry/variables.tf&versionDescriptor.versionType=commit&versionDescriptor.version=7a53efb1751859edafca08686df1ff1d45f00d31
200  ://dev.azure.com:443/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items
GET https://dev.azure.com/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items?path=terraform/container_registry/.terraform.lock.hcl&versionDescriptor.versionType=commit&versionDescriptor.version=7a53efb1751859edafca08686df1ff1d45f00d31
200  ://dev.azure.com:443/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/items
Found 10 dependency file(s) at commit 7a53efb1751859edafca08686df1ff1d45f00d31
 - /terraform/container_registry/azuredevops_serviceendpoint.tf
 - /terraform/container_registry/configuration.tf
 - /terraform/container_registry/container_registry.tf
 - /terraform/container_registry/external_resources.tf
 - /terraform/container_registry/locals.tf
 - /terraform/container_registry/main.tf
 - /terraform/container_registry/outputs.tf
 - /terraform/container_registry/resourcegroup.tf
 - /terraform/container_registry/variables.tf
 - /terraform/container_registry/.terraform.lock.hcl
Parsing dependencies information
Found 3 dependencies
 - hashicorp/azuread (2.31.0)
 - hashicorp/azurerm (3.35.0)
 - microsoft/azuredevops (0.3.0)
GET https://dev.azure.com/humanprinter/_apis/connectionData
200  ://dev.azure.com:443/humanprinter/_apis/connectionData
GET https://dev.azure.com/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest
200  ://dev.azure.com:443/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest
GET https://dev.azure.com/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/pullrequests?api-version=6.0&searchCriteria.status=active&searchCriteria.creatorId=7fca59c0-b12c-4fcf-a6a5-8321cd76e161&searchCriteria.targetRefName=refs/heads/master
200  ://dev.azure.com:443/humanprinter/AzureDependabot/_apis/git/repositories/terraformtest/pullrequests
Checking if hashicorp/azuread 2.31.0 needs updating
GET https://registry.terraform.io/.well-known/terraform.json
200  ://registry.terraform.io:443/.well-known/terraform.json
GET https://registry.terraform.io/v1/providers/hashicorp/azuread/versions
200  ://registry.terraform.io:443/v1/providers/hashicorp/azuread/versions
No update needed for hashicorp/azuread 2.31.0
Checking if hashicorp/azurerm 3.35.0 needs updating
GET https://registry.terraform.io/.well-known/terraform.json
200  ://registry.terraform.io:443/.well-known/terraform.json
GET https://registry.terraform.io/v1/providers/hashicorp/azurerm/versions
200  ://registry.terraform.io:443/v1/providers/hashicorp/azurerm/versions
Requirements to unlock own
Requirements update strategy
Updating hashicorp/azurerm from 3.35.0 to 3.37.0
Error working on updates for hashicorp/azurerm 3.35.0 (continuing)
/home/dependabot/dependabot-script/vendor/ruby/3.1.0/gems/dependabot-common-0.215.0/lib/dependabot/shared_helpers.rb:316:in `run_shell_command': Error: Invalid provider argument (Dependabot::SharedHelpers::HelperSubprocessFailed)

The provider registry.terraform.io/hashicorp/azurerm is not required by the
current configuration.
        from /home/dependabot/dependabot-script/vendor/ruby/3.1.0/gems/dependabot-terraform-0.215.0/lib/dependabot/terraform/file_updater.rb:176:in `block (2 levels) in lookup_hash_architecture'
        from /home/dependabot/dependabot-script/vendor/ruby/3.1.0/gems/dependabot-terraform-0.215.0/lib/dependabot/terraform/file_updater.rb:169:in `each'
        from /home/dependabot/dependabot-script/vendor/ruby/3.1.0/gems/dependabot-terraform-0.215.0/lib/dependabot/terraform/file_updater.rb:169:in `block in lookup_hash_architecture'
        from /home/dependabot/dependabot-script/vendor/ruby/3.1.0/gems/dependabot-common-0.215.0/lib/dependabot/shared_helpers.rb:49:in `block in in_a_temporary_directory'
        from /home/dependabot/dependabot-script/vendor/ruby/3.1.0/gems/dependabot-common-0.215.0/lib/dependabot/shared_helpers.rb:49:in `chdir'
        from /home/dependabot/dependabot-script/vendor/ruby/3.1.0/gems/dependabot-common-0.215.0/lib/dependabot/shared_helpers.rb:49:in `in_a_temporary_directory'
        from /home/dependabot/dependabot-script/vendor/ruby/3.1.0/gems/dependabot-common-0.215.0/lib/dependabot/shared_helpers.rb:38:in `in_a_temporary_repo_directory'
        from /home/dependabot/dependabot-script/vendor/ruby/3.1.0/gems/dependabot-terraform-0.215.0/lib/dependabot/terraform/file_updater.rb:168:in `lookup_hash_architecture'
        from /home/dependabot/dependabot-script/vendor/ruby/3.1.0/gems/dependabot-terraform-0.215.0/lib/dependabot/terraform/file_updater.rb:211:in `architecture_type'
        from /home/dependabot/dependabot-script/vendor/ruby/3.1.0/gems/dependabot-terraform-0.215.0/lib/dependabot/terraform/file_updater.rb:227:in `block in update_lockfile_declaration'
        from /home/dependabot/dependabot-script/vendor/ruby/3.1.0/gems/dependabot-common-0.215.0/lib/dependabot/shared_helpers.rb:49:in `block in in_a_temporary_directory'
        from /home/dependabot/dependabot-script/vendor/ruby/3.1.0/gems/dependabot-common-0.215.0/lib/dependabot/shared_helpers.rb:49:in `chdir'
        from /home/dependabot/dependabot-script/vendor/ruby/3.1.0/gems/dependabot-common-0.215.0/lib/dependabot/shared_helpers.rb:49:in `in_a_temporary_directory'
        from /home/dependabot/dependabot-script/vendor/ruby/3.1.0/gems/dependabot-common-0.215.0/lib/dependabot/shared_helpers.rb:38:in `in_a_temporary_repo_directory'
        from /home/dependabot/dependabot-script/vendor/ruby/3.1.0/gems/dependabot-terraform-0.215.0/lib/dependabot/terraform/file_updater.rb:225:in `update_lockfile_declaration'
        from /home/dependabot/dependabot-script/vendor/ruby/3.1.0/gems/dependabot-terraform-0.215.0/lib/dependabot/terraform/file_updater.rb:36:in `updated_dependency_files'
        from ./update-script.rb:578:in `block in <main>'
        from ./update-script.rb:476:in `each'
        from ./update-script.rb:476:in `<main>'
Checking if microsoft/azuredevops 0.3.0 needs updating
GET https://registry.terraform.io/.well-known/terraform.json
200  ://registry.terraform.io:443/.well-known/terraform.json
GET https://registry.terraform.io/v1/providers/microsoft/azuredevops/versions
200  ://registry.terraform.io:443/v1/providers/microsoft/azuredevops/versions
No update needed for microsoft/azuredevops 0.3.0
Done

Running ‘dependabot-core’ for GitHub

I’ve tested with a local copy of the Dependabot repository as documented by dependabot

Commands

$ git clone https://github.com/dependabot/dependabot-core.git $ docker pull ghcr.io/dependabot/dependabot-core-development:latest $ docker tag ghcr.io/dependabot/dependabot-core-development dependabot/dependabot-core-development $ bin/docker-dev-shell => running docker development shell [dependabot-core-dev] ~/dependabot-core $ bin/dry-run.rb terraform HumanPrinter/terraformtest --dir /terraform/container_registry

Output

warning: parser/current is loading parser/ruby31, which recognizes3.1.3-compliant syntax, but you are running 3.1.2.
Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.
=> cloning into /home/dependabot/dependabot-core/tmp/HumanPrinter/terraformtest
=> parsing dependency files
=> updating 3 dependencies: hashicorp/azuread, hashicorp/azurerm, microsoft/azuredevops

=== hashicorp/azuread (2.31.0)
 => checking for updates 1/3
🌍 https://registry.terraform.io/.well-known/terraform.json
🌍 https://registry.terraform.io/v1/providers/hashicorp/azuread/versions
 => latest available version is 2.31.0
    (no update needed as it's already up-to-date)

=== hashicorp/azurerm (3.35.0)
 => checking for updates 2/3
🌍 https://registry.terraform.io/.well-known/terraform.json
🌍 https://registry.terraform.io/v1/providers/hashicorp/azurerm/versions
 => latest available version is 3.37.0
 => latest allowed version is 3.37.0
 => requirements to unlock: own
 => requirements update strategy:
 => updating hashicorp/azurerm from 3.35.0 to 3.37.0

    ± .terraform.lock.hcl
    ~~~
    25c25
    <   version     = "3.35.0"
    ---
    >   version     = "3.37.0"
    28,38c28,39
    <     "h1:UgjTFMinFpuXO3brreeB8B0ZucSaPyeHdPAgUqHTANs=",
    <     "zh:17a0322fc329b3d7cedf76e3cbaea3ffcd11fe7f806608cece646844c9645be1",
    <     "zh:501d3b0d29803099c68c6e17d462e93c4362c3b0ab5024924becc275b595bd86",
    <     "zh:64b69bdbfef1e326677ca238f8090d9b04e457840c4388c84b51fce96799c13c",
    <     "zh:842bf9f3e7eb69ea4aac783d94bee2a153b43e49e26290f933df3a39f69038b8",
    <     "zh:8cf78f65af5335aa44b1dd27ab47dfe9c01f040ab82b209f54fea6b46e93f6f4",
    <     "zh:959277ddff0620e5ea0441ed275da5b4e959ec63040d53988560c2fbbc58f713",
    <     "zh:badb39ef631ffc5d337a98e168a67224e4bbd75b96d034d6f15764c4d14b76f4",
    <     "zh:d100ca6bd7d851d0e5e3289f05a3a7ef049f1af8ede5b64fcda6497e3e4a13ef",
    <     "zh:dd490138be6166c8128f5be1345ee62e4fa508568659c74d22a2a3bbc4826d61",
    <     "zh:e2e7af79f9adde9b7882ad2780936b7ff599abd2dd154e246fb419f9ed7cae64",
    ---
    >     "h1:83XTgyPKUKt706IjTLHo9HL0KN5m+DwmSKuVQv6dNb4=",
    >     "zh:2a7bda0b7679d1c791c762103a22f333b544b6e6776c4177f33bafc9cc28c919",
    >     "zh:49ff49670c349f918017315838a43ece09bf6f1bf7721b992f1cadbceb273c62",
    >     "zh:55c9346d03380585e17616b79c4233b726d6fb9efa1921848834fc881e5d7d54",
    >     "zh:5ab117b56a4236ea29926e9d95c27d7bf8ae6706d0fffb76c0b1bfe67bf3a78e",
    >     "zh:5cfc086d5d56308edb3e68aac5f8a448ddc6e56541be7b152ae886399e9b2c69",
    >     "zh:7a8929ed38152aac6652711f32193c8582bc996f8fa73879a3ac7a9bf88d2460",
    >     "zh:895294e90a37f719975fcd2269b95e973147e48ec0ebb9c2fe472bc93531b49c",
    >     "zh:8baa5e2b6e5b02df5b45d253a3aea93f22619920cf9577290d682b59a6d5664b",
    >     "zh:b146a732c7909238c10d216b92a35092be4f72a0509a4c6742cc3245bf3b3bf3",
    >     "zh:cedef898ccd512a6519eae3dff7eb0d581d2c3dad8e0001992da16ad1d7fded8",
    >     "zh:f016d9ba94ea88476883b4d63cff88a0225974e0a8b8c3e8555f73c5de6f7119",
    40d40
    <     "zh:f797b58f9a136c5754eab0d6c05e7171f3cf0b8c117b4bd26caa68926b9e1eeb",
    ~~~

=== microsoft/azuredevops (0.3.0)
 => checking for updates 3/3
🌍 https://registry.terraform.io/.well-known/terraform.json
🌍 https://registry.terraform.io/v1/providers/microsoft/azuredevops/versions
 => latest available version is 0.3.0
    (no update needed as it's already up-to-date)
🌍 Total requests made: '6'

Investigated soo far

From what I can tell (I have no experience with Ruby-development what so ever), is that the container executes the update-script.rb up to line 572 where it initializes the FileUpdater for terraform. From what I can tell (using some debug printlines), it initializes the FileUpdater with the correct data. However, when line 579 (updated_files = updater.updated_dependency_files) is executed, it fails with the error as described above. Looking at the code for the Terraform handler (from the dependabot-core code base), it will update the .terraform.lock.hcl file with the new Terraform-provider version and then execute the command terraform providers lock -platform=linux_amd64 registry.terraform.io/hashicorp/azurerm -no-color. This command fails with the message Error: Invalid provider argument The provider registry.terraform.io/hashicorp/azurerm is not required by the current configuration.

However, the same code runs fine when using the default Dependabot-code against my GitHub-code.

Any help is greatly appreciated.

[HumanPrinter]

I ran a couple of tests (both manual runs as runs through a pipeline) and I can confirm that is now works as expected. Good work and many thanks

[mburumaxwell]

This now works, see https://dev.azure.com/tingle/dependabot/_build/results?buildId=54323

@HumanPrinter Could you test using dockerImageTag: '0.13.3-ci.12' in the inputs?

Example:

steps:
- task: dependabot@1
  inputs:
    dockerImageTag: '0.13.3-ci.12'