ERROR: 'charmap' codec can't encode characters in position 4790-4801: character maps to <undefined>

az feedback auto-generates most of the information requested below, as of CLI version 2.0.62

Describe the bug

Recently, my script run into this problem. My az script is run on Azure Pipelines. Here is my PowerShell script on Windows Server 2019 agent:

az deployment sub what-if `
    --name "main$(Get-Date -Format "yyyyMMddHHmmss")" `
    --location 'japaneast' `
    --template-file 'main.bicep' `
    --parameters  `
        "env=$(env.name)" `
        "projectName=$(projectName)" `
        "dbAdminAccount=$(sql.serveradmin.name)" `
        "dbAdminPwd=$(sql.serveradmin.password)"

Here is the error output:

ERROR: 'charmap' codec can't encode characters in position 4790-4801: character maps to <undefined>

For the following command, it works.

az deployment sub validate `
    --name "main$(Get-Date -Format "yyyyMMddHHmmss")" `
    --location 'japaneast' `
    --template-file 'main.bicep' `
    --parameters  `
        "env=$(env.name)" `
        "projectName=$(projectName)" `
        "dbAdminAccount=$(sql.serveradmin.name)" `
        "dbAdminPwd=$(sql.serveradmin.password)"

So the error only shows when running az deployment sub what-if command.

To Reproduce

Expected behavior

My script is running for more than 1 month. It works before. I found it’s broken today.

Environment summary

C:\Windows\system32\cmd.exe /D /S /C ""C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin\az.cmd" --version"
azure-cli                         2.32.0 *
WARNING: You have 2 updates available. Consider updating your CLI installation with 'az upgrade'


core                              2.32.0 *
Please let us know how we are doing: https://aka.ms/azureclihats
telemetry                          1.0.6
and let us know if you're interested in trying out our newest features: https://aka.ms/CLIUXstudy

Extensions:
azure-devops                      0.22.0

Dependencies:
msal                              1.16.0
azure-mgmt-resource               20.0.0

Python location 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Program Files\Common Files\AzureCliExtensionDirectory'

Python (Windows) 3.8.9 (tags/v3.8.9:a743f81, Apr  6 2021, 13:22:56) [MSC v.1928 32 bit (Intel)]

Legal docs and information: aka.ms/AzureCliLegal

Additional context

[doggy8088]

@jiasli I confirmed the ubuntu-latest agent works. Thanks.

[jiasli]

Thanks @ian90911 for the debug log which is extremely helpful.

Judging by the log:

DEBUG: cli.azure.cli.core.util: Traceback (most recent call last):
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 692, in _run_job
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 328, in __call__
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/resource/custom.py", line 815, in what_if_deploy_arm_template_at_subscription_scope
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/resource/custom.py", line 835, in _what_if_deploy_arm_template_at_subscription_scope_core
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/resource/custom.py", line 930, in _what_if_deploy_arm_template_core
  File "encodings\cp1252.py", line 19, in encode
UnicodeEncodeError: 'charmap' codec can't encode characters in position 4665-4676: character maps to <undefined>

The failed code is at

https://github.com/Azure/azure-cli/blob/67568014e3df1f183b92ee1ad7699b3b86f35dc0/src/azure-cli/azure/cli/command_modules/resource/custom.py#L930

Apparently print throws this error.

Also from the log, I noticed you have a resource name with Unicode characters:

"resourceId":"/subscriptions/***/resourceGroups/em-s/providers/microsoft.insights/activityLogAlerts/\u65e5\u672c\u6771\u5340\u57df\u7684\u670d\u52d9\u5065\u5eb7\u8b66\u793a"

I used Python to decode it to

python -c "print('\u65e5\u672c\u6771\u5340\u57df\u7684\u670d\u52d9\u5065\u5eb7\u8b66\u793a')"
日本東區域的服務健康警示

I tested without Azure CLI but only with Python on Windows agent and reproduced this issue.

jobs:
- job: TestAzureCLITaskUbuntu
  pool:
    vmImage: 'ubuntu-latest'
  steps:
  - task: AzureCLI@2
    displayName: Azure CLI
    inputs:
      azureSubscription: ...
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        python3 -c "import sys; print(sys.stdout.encoding); print('汉字')"
- job: TestAzureCLITaskWindows
  pool:
    vmImage: 'windows-latest'
  steps:
  - task: AzureCLI@2
    displayName: Azure CLI
    inputs:
      azureSubscription: ...
      scriptType: batch
      scriptLocation: inlineScript
      inlineScript: |
        "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe" -c "import sys; print(sys.stdout.encoding); print('汉字')"

The Ubuntu agent gives output

utf-8
汉字

The Windows agent gives output

cp1252
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "encodings\cp1252.py", line 19, in encode
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-1: character maps to <undefined>

Solution

For end user, the are 2 possible solutions:

1. Use a Ubuntu agent on which stdout is UTF-8.
2. Remove Unicode characters in the resource name since Azure DevOps’s stdout’s encoding is cp1252 which is not able to handle Unicode. For example, you may rename it to japan-north-service-health-alert.

For DevOps, Windows agent should use UTF-8 as stdout’s encoding. We will internally work with Azure DevOps team on this topic.