Start-Transcript uses lossy ASCII character encoding instead of BOM-less UTF-8 if the target file happens to exist

Note:

• Windows PowerShell is not affected - UTF-8 with BOM is used consistently there.
• A fix for the closely related #13678 could probably encompass a fix for this issue as well.

When the target file happens to exist and doesn’t have a BOM - even without -Append - Start-Transcript mistakenly uses ASCII encoding to write the file.

Without -Append, Start-Transcript shouldn’t even look at the existing file - it should simply replace it, and use the default encoding (BOM-less UTF-8). See #13678 for a closely related bug with -Append.

Instead, Start-Transcript apparently looks for a BOM at the start of an existing file and uses ASCII encoding if it doesn’t find one.

Steps to reproduce

# To surface the bug, make sure that the target file exists and doesn't have a BOM.
$null > temp:/$PID.txt

$null = Start-Transcript temp:/$PID.txt
'ü' # output a string with a non-ASCII-range character
$null = Stop-Transcript

Select-String -Quiet 'ü' temp:/$PID.txt | Should -BeTrue

Remove-Item temp:/$PID.txt

Expected behavior

The test should succeed.

Actual behavior

The test fails, because ü was transliterated to verbatim ?, suggesting that ASCII encoding was used to write the file.

Environment data

PowerShell Core 7.1.0-preview.7

[Gimly]

@mklement0 I’ve opened a PR that, I think, fixes this issue. It’s quite simple as I think the only part that was not working as you described was that the “clear file” that was done when the -Append flag isn’t set wasn’t fixing a specific encoding, but simply using the one that existed.

Now I’ve fixed the encoding (using the static readonly instance present in Utils) to UTF8 No Bom, as we had done in the PR #13732.

[mklement0]

Thanks for tackling this, @Gimly - I haven’t even looked at the original code yet, but @iSazonov has, so perhaps he has thoughts.