x-ms-identifiers not allowed when generating ARM template schemas

Before filling a bug

• have you checked the faq for known issues.
• have you checked existing issues

Describe the bug

Please provide as much information as you can. This would be:

• OpenAPI files having the issues: https://github.com/leni-msft/azure-rest-api-specs/blob/d76689fc4d/specification/oep/resource-manager/readme.md
• Autorest command used:

➜  azure-rest-api-specs git:(pr-17105) autorest --version=3.0.6375 --azureresourceschema --multiapi ~/workspace/azure-rest-api-specs/specification/oep/resource-manager/readme.md --debug
SourceMapConsumer.initialize is a no-op when running in node.js
AutoRest code generation utility [cli version: 3.5.1; node: v14.15.5]
(C) 2018 Microsoft Corporation.
https://aka.ms/autorest
info    | AutoRest core version selected from configuration: 3.0.6375.
debug   | [0.67 s] Network Enabled: true
debug   | [0.74 s] The most recent installed version is 3.7.3
debug   | [0.74 s] Starting @autorest/core from /home/leni/.autorest/@autorestcore@3.0.6375/nodemodules/@autorest/core
   Loading AutoRest core      '/home/leni/.autorest/@autorestcore@3.0.6375/nodemodules/@autorest/core/dist' (3.0.6375)
   Loading AutoRest extension '@autorest/azureresourceschema' (~3.0.45->3.0.103)
Processing batch task - {"tag":"schema-2021-06-01-preview"} .
DEBUG [1.51 s]: swagger-document/loader-swagger - START inputs = 0
DEBUG [1.51 s]: openapi-document/loader-openapi - START inputs = 0
DEBUG [1.51 s]: configuration-emitter - START inputs = 0
DEBUG [1.51 s]: pipeline-emitter - START inputs = 0
DEBUG [1.55 s]: configuration-emitter - END [0.03 s]
DEBUG [1.55 s]: pipeline-emitter - END [0.03 s]
DEBUG [1.59 s]: openapi-document/loader-openapi - END [0.07 s]
DEBUG [1.59 s]: openapi-document/individual/transform - SKIPPING
DEBUG [1.59 s]: openapi-document/individual/schema-validator-openapi - SKIPPING
DEBUG [1.59 s]: openapi-document/individual/identity - SKIPPING
DEBUG [1.68 s]: swagger-document/loader-swagger - END [0.16 s]
DEBUG [1.68 s]: swagger-document/individual/transform - START inputs = 3
DEBUG [1.77 s]: swagger-document/individual/transform - END [0.09 s]
DEBUG [1.77 s]: swagger-document/individual/schema-validator-swagger - START inputs = 3
ERROR: Schema violation: Additional properties not allowed: x-ms-identifiers (definitions > OepProperties > properties > dataPartitionNames)
    - file:///home/leni/workspace/azure-rest-api-specs/specification/oep/resource-manager/Microsoft.OpenEnergyPlatform/preview/2021-06-01-preview/oep.json:668:4
/home/leni/.autorest/@autorest_core@3.0.6375/node_modules/@autorest/core/dist/src_lib_autorest-core_ts.js - FAILURE {"exitCode":1}
ERROR: Schema violation: Additional properties not allowed: x-ms-identifiers (definitions > OepProperties > properties > dataPartitionNames)
    - file:///home/leni/workspace/azure-rest-api-specs/specification/oep/resource-manager/Microsoft.OpenEnergyPlatform/preview/2021-06-01-preview/oep.json:668:4
Process() cancelled due to exception : [OperationAbortedException] Error occurred. Exiting. / Error: [OperationAbortedException] Error occurred. Exiting.
    at /home/leni/.autorest/@autorestcore@3.0.6375/node_modules/@autorest/core/dist/src_lib_autorest-corets.js:7062:23
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async /home/leni/.autorest/@autorestcore@3.0.6375/node_modules/@autorest/core/dist/src_lib_autorest-corets.js:1855:29
    at async ScheduleNode (/home/leni/.autorest/@autorestcore@3.0.6375/node_modules/@autorest/core/dist/src_lib_autorest-corets.js:2543:33)
Failure during batch task - {"tag":"schema-2021-06-01-preview"} -- false.
[1.82 s] Shutting Down.
[1.82 s] Exiting.

Expected behavior ARM template schemas should be successfully generated without errors.

Additional context This x-ms-identifiers is a new swagger extension, and it seems the swagger-extension.json doesn’t define it yet. CI pipeline output can be found here.

[anthony-c-martin]

The commonmark extension seems to “sanitize” markdown to plain text, according to 5decd2c , and only takes affect on DocumentationField. @anthony-c-martin Do you know if the extension is needed in generating schemas?

I think we could probably get away without sanitizing markdown - it’ll probably just make the odd description a bit less readable.

Also @timotheeguerin since there’re quite several issues with the autorest csharp code, do you think it’s okay to hotfix the swagger schema with #4391 ? and (if necessary? @anthony-c-martin) resolve the issues in the future by upgrading the ARM schemas extension to modelerfour?

FYI https://github.com/Azure/bicep-types-az/tree/main/src/autorest.bicep is a good starting point for migrating to modelerfour, as it was originally forked from the autorest.azureresourceschema (C#) extension, and then rewritten for modelerfour.

[timotheeguerin]

that PR makes sense to be added(for validation purposes) but it won’t fix anything. It update the latest schema which shouldn’t complain about this already. Can you check if the resource schema works with autorest core --v2?

There is 3 issues here:

• Version you are using is not supporting unknown x-ms extension: this should be fixed in the latest
• Using the latest is missing the commonmark plugin(We can discuss what we do here)
• THere is issue with the old CSHarp code. I think this is due to some unsupported thing in the spec.