Roslyn should provide better error message than "error:03000098:digital envelope routines::invalid digest" on RSA+SHA1 unavailability

Version Used:

Originally observed with .NET 6 (see https://github.com/dotnet/runtime/issues/65874). That should correspond, roughly, to something along commit 487283bcd8d66693091f2800dcf1c8ae37cccdee.

Steps to Reproduce:

Attempt to use full signing on CentOS Stream 9

1. docker run -it quay.io/centos/centos:stream9-development /bin/bash
2. curl -sSLO https://dot.net/v1/dotnet-install.sh
3. bash ./dotnet-install.sh --channel LTS
4. export DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1
5. cd ~
6. ~/.dotnet/dotnet new console -o test
7. cd test
8. echo '[assembly:System.Reflection.AssemblyKeyFile("testkey.snk")]' > Program.cs
9. echo 'Console.WriteLine("Hello!");' >> Program.cs
10. curl -sSLO https://github.com/dotnet/runtime/raw/release/6.0/src/mono/mono/tests/testing_gac/testkey.snk
11. ~/.dotnet/dotnet publish

Expected Behavior:

Either signing works, or I get an actionable message that tells me what I should do instead.

Actual Behavior:

Microsoft (R) Build Engine version 17.2.0+41abc5629 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

Determining projects to restore...                                                                                                  
  All projects are up-to-date for restore.

/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error : Unhandled exception. Interop+Crypto+OpenSslCryptographicException: error:03000098:digital envelope routines::invalid digest [/root/test/test.csproj]                                          
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at Interop.Crypto.RsaSignHash(SafeEvpPKeyHandle pkey, RSASignaturePaddingMode paddingMode, IntPtr digestAlgorithm, ReadOnlySpan`1 hash, Span`1 destination) [/root/test/test.csproj]       
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.TrySignHash(ReadOnlySpan`1 hash, Span`1 destination, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding, Boolean allocateSignature, Int32& bytesWritten, Byte[]& signature) [/root/test/test.csproj]                                                      
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.SignHash(Byte[] hash, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding) [/root/test/test.csproj]
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at Microsoft.CodeAnalysis.SigningUtilities.CalculateRsaSignature(IEnumerable`1 content, RSAParameters privateKey) [/root/test/test.csproj]                                                 
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at Microsoft.CodeAnalysis.DesktopStrongNameProvider.<>c__DisplayClass12_0.<SignBuilder>b__0(IEnumerable`1 content) [/root/test/test.csproj]                                                
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at System.Reflection.PortableExecutable.PEBuilder.Sign(BlobBuilder peImage, Blob strongNameSignatureFixup, Func`2 signatureProvider) [/root/test/test.csproj]                              
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at System.Reflection.PortableExecutable.ManagedPEBuilder.Sign(BlobBuilder peImage, Func`2 signatureProvider) [/root/test/test.csproj]                                                      
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at Microsoft.CodeAnalysis.DesktopStrongNameProvider.SignBuilder(ExtendedPEBuilder peBuilder, BlobBuilder peBlob, RSAParameters privateKey) [/root/test/test.csproj]                        
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at Microsoft.Cci.PeWriter.WritePeToStream(EmitContext context, CommonMessageProvider messageProvider, Func`1 getPeStream, Func`1 getPortablePdbStreamOpt, PdbWriter nativePdbWriterOpt, String pdbPathOpt, Boolean metadataOnly, Boolean isDeterministic, Boolean emitTestCoverageData, Nullable`1 privateKeyOpt, CancellationToken cancellationToken) [/root/test/test.csproj]                                                                                        
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at Microsoft.CodeAnalysis.Compilation.SerializePeToStream(CommonPEModuleBuilder moduleBeingBuilt, DiagnosticBag metadataDiagnostics, CommonMessageProvider messageProvider, Func`1 getPeStream, Func`1 getMetadataPeStreamOpt, Func`1 getPortablePdbStreamOpt, PdbWriter nativePdbWriterOpt, String pdbPathOpt, RebuildData rebuildData, Boolean metadataOnly, Boolean includePrivateMembers, Boolean isDeterministic, Boolean emitTestCoverageData, Nullable`1 privateKeyOpt, CancellationToken cancellationToken) [/root/test/test.csproj]
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at Microsoft.CodeAnalysis.Compilation.SerializeToPeStream(CommonPEModuleBuilder moduleBeingBuilt, EmitStreamProvider peStreamProvider, EmitStreamProvider metadataPEStreamProvider, EmitStreamProvider pdbStreamProvider, RebuildData rebuildData, Func`2 testSymWriterFactory, DiagnosticBag diagnostics, EmitOptions emitOptions, Nullable`1 privateKeyOpt, CancellationToken cancellationToken) [/root/test/test.csproj]                                           
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at Microsoft.CodeAnalysis.CommonCompiler.CompileAndEmit(TouchedFileLogger touchedFilesLogger, Compilation& compilation, ImmutableArray`1 analyzers, ImmutableArray`1 generators, ImmutableArray`1 additionalTextFiles, AnalyzerConfigSet analyzerConfigSet, ImmutableArray`1 sourceFileAnalyzerConfigOptions, ImmutableArray`1 embeddedTexts, DiagnosticBag diagnostics, CancellationToken cancellationToken, CancellationTokenSource& analyzerCts, Boolean& reportAnalyzer, AnalyzerDriver& analyzerDriver) [/root/test/test.csproj]                                                                        
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at Microsoft.CodeAnalysis.CommonCompiler.RunCore(TextWriter consoleOutput, ErrorLogger errorLogger, CancellationToken cancellationToken) [/root/test/test.csproj]        
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at Microsoft.CodeAnalysis.CommonCompiler.Run(TextWriter consoleOutput, CancellationToken cancellationToken) [/root/test/test.csproj]                                                       
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc.<>c__DisplayClass1_0.<Run>b__0(TextWriter tw) [/root/test/test.csproj]
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at Microsoft.CodeAnalysis.CommandLine.ConsoleUtil.RunWithUtf8Output[T](Boolean utf8Output, TextWriter textWriter, Func`2 func) [/root/test/test.csproj]
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc.Run(String[] args, BuildPaths buildPaths, TextWriter textWriter, IAnalyzerAssemblyLoader analyzerLoader) [/root/test/test.csproj]
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at Microsoft.CodeAnalysis.CommandLine.BuildClient.RunCompilation(IEnumerable`1 originalArguments, BuildPaths buildPaths, TextWriter textWriter, String pipeName) [/root/test/test.csproj]
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at Microsoft.CodeAnalysis.CommandLine.BuildClient.Run(IEnumerable`1 arguments, RequestLanguage language, CompileFunc compileFunc, CompileOnServerFunc compileOnServerFunc) [/root/test/test.csproj]
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at Microsoft.CodeAnalysis.CSharp.CommandLine.Program.MainCore(String[] args) [/root/test/test.csproj]
/root/.dotnet/sdk/6.0.301/Roslyn/Microsoft.CSharp.Core.targets(75,5): error :    at Microsoft.CodeAnalysis.CSharp.CommandLine.Program.Main(String[] args) [/root/test/test.csproj]

[jaredpar]

Anyone have any other ideas?

On Windows at least if the signing service is unavailable, if it’s stopped for example, this operation will throw. We’ve seen this occur several times in production. That is a case where I wouldn’t want to recommend /publicSign. It’s not that the machine proactively disabled it, instead the machine is in an error state.

Hoping @bartonjs @GrabYourPitchforks can chime in here with ways to make this specific case detectable.

[jaredpar]

the best I can think of for detection would be that after a failure you try signing nothing with a new ephemeral key. If that fails, it seems more likely that it’s the algorithm.

That’s an interesting thought. We could look at doing that.