Smoke Testing fails for KV in Azure US Gov Cloud

Seeing failure relating to the API version available: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=488075&view=logs&j=da44f339-192b-547d-b580-2f53ccd7d29e&t=d042d7e7-4b06-5096-46cd-6ebdaa3f6dba

Proposal: Smoke Tests use samples so it might be possible to provide the API version to the sample (and if no API version is provided use a “safe default” that works for the Azure Public Cloud) with an environment variable. Several things to think about:

• Minimizing sample complexity
• Changes to the “safe default” version over time in the SDK will need to be synchronized the sample code
• We will probably run into the same problem when onboarding the SDK for live testing in sovereign clouds so live tests should be able to make use of this configuration

##[error]Test Failed - Package: keyvault-certificates - Sample File:helloWorld.js
{
  sample: {
    entrypoint: [AsyncFunction: main],
    name: 'keyvault-certificates',
    sampleFile: 'helloWorld.js',
    directory: '/home/vsts/work/1/s/sdk/keyvault/keyvault-certificates/dist-samples/javascript'
  },
  result: {
    success: false,
    exception: RestError: The specified version (7.1) is not recognized. Consider using the latest supported version (2016-10-01). 
     {
      "name": "RestError",
      "code": "BadParameter",
      "statusCode": 400,
      "request": {
        "streamResponseBody": false,
        "url": "***certificates/cert1597077148327/create?api-version=7.1",
        "method": "POST",
        "headers": {
          "_headersMap": {
            "content-type": "application/json; charset=utf-8",
            "user-agent": "azsdk-js-keyvault-certificates/4.1.0-preview.2 core-http/1.1.6 Node/v12.18.3 OS/(x64-Linux-5.3.0-1034-azure)",
            "x-ms-client-request-id": "1bda04c6-1cbe-408a-bef1-19c4e0c0d0f1",
            "cookie": "REDACTED",
            "authorization": "REDACTED"
          }
        },
        "withCredentials": false,
        "timeout": 0,
        "keepAlive": true,
        "requestId": "1bda04c6-1cbe-408a-bef1-19c4e0c0d0f1"
      },
      "details": {
        "error": {
          "code": "BadParameter",
          "message": "The specified version (7.1) is not recognized. Consider using the latest supported version (2016-10-01)."
        }
      },
      "message": "The specified version (7.1) is not recognized. Consider using the latest supported version (2016-10-01)."
    }
  }
}

[witemple-msft]

Just want to revisit this since @sadasant asked for my input.

I can’t remember exactly what we discussed or where we landed on this, but knowing what I know now and given the developments since (Samples V2, which we didn’t introduce until several months after this issue), I wouldn’t advise the proposed solution anymore.

At the time, we had a different samples process. Currently, prep is only used by smoke tests. At this point, with Samples V2, I would just suggest using an environment variable if customizing the API version in the samples is very important.

Keep in mind that samples serve as example programs for customers first and as supplementary test surface second, so ask first if the change you want to make to the sample is good for customers and then if it’s needed for testing. If it’s needed for testing, but bad for customers, then we should revisit our testing strategy.

[danieljurek]

From our discussion:

• Update samples to include something like:

// Additional client constructor options
const clientConstructorOptions = {}; 
...
const client = new CertificateClient(url, credential, clientConstructorOptions);

• Update https://github.com/Azure/azure-sdk-for-js/blob/master/common/tools/dev-tool/src/commands/samples/prep.ts to include the ability to put some collection of options, probably a string, into the above code. Possibly something like:

dev-tool samples prep --directory $PackageDirectory --use-packages --constructor-options '{ serviceVersion: "1.2.3"}' 

cc: @willmtemple