With many large files: stops progressing, OOMs or InvalidOperation in Roslyn code

Steps to reproduce

1. Convert a proprietary codebase (which I can’t share, my apologies).
2. Wait a couple of hours
3. In phase 2, simplification, it crashes.

Error message shown

The crash is

https://github.com/dotnet/roslyn threw an exception: System.InvalidOperationException: Exception of type 'System.InvalidOperationException' was thrown.
   at System.Runtime.CompilerServices.ConditionalWeakTable`2.VerifyIntegrity()
   at System.Runtime.CompilerServices.ConditionalWeakTable`2.Add(TKey key, TValue value)
   at Microsoft.CodeAnalysis.GreenNode..ctor(ObjectReader reader)

Full log at https://gist.github.com/jnnnnn/2b7a739170467e125d0a05b0a263774a (with filenames hidden, sorry)

Details

VS extension version 8.5.0.0 (installed 2022-04-14)

• Did you see it working in a previous version, which? no
• I can follow reasonable requests to debug, but can’t share the code it is crashing on

[GrahamTheCoder]

Thanks @EoflaOE this definitely seems to repro the issue even in total isolation. I’ve created a draft PR with an initial workaround which skips files that take longer than the configurable timeout. Hopefully I’ll find a way to improve the performance by calling roslyn in a different way (or possibly making a fix to roslyn)

Here’s a snippet of C# which takes more than a minute to simplify (though admittedly contains about 10K characters in a single expression.

using System;using System.Collections;using System.Collections.Generic;using System.Data;using System.Diagnostics;

// Kernel Simulator  Copyright (C) 2018-2022  Aptivi
// 
// This file is part of Kernel Simulator
// 
// Kernel Simulator is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// 
// Kernel Simulator is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
// GNU General Public License for more details.
// 
// You should have received a copy of the GNU General Public License
// along with this program.  If not, see <https://www.gnu.org/licenses/>.
using global::System.IO;using System.Linq;using System.Net;using System.Xml.Linq;using ColorSeq;using Extensification.DictionaryExts;using Extensification.StringExts;using FluentFTP;using FluentFTP.Helpers;using KS.ConsoleBase;using KS.ConsoleBase.Colors;using KS.ConsoleBase.Inputs;using KS.ConsoleBase.Themes;using KS.ConsoleBase.Themes.Studio;using KS.Files;using KS.Kernel;using KS.Languages;using KS.Login;using KS.Misc.Platform;using KS.Misc.Probers;using KS.Misc.Threading;using KS.Misc.Writers.ConsoleWriters;using KS.Misc.Writers.DebugWriters;using KS.Misc.Writers.FancyWriters;using KS.Shell;using KS.Shell.ShellBase;using KS.Shell.ShellBase.Commands;using KS.Shell.ShellBase.Shells;using global::KS.Shell.Shells.UESH.Commands;using Microsoft.VisualBasic.Constants;using Newtonsoft.Json;using Renci.SshNet;

namespace KS.Shell.Shells.UESH
{
    public static class UESHShellCommon
    {

        internal readonly static global::System.Collections.Generic.Dictionary<global::System.String, global::KS.Shell.ShellBase.Commands.CommandInfo> ModCommands = new global::System.Collections.Generic.Dictionary<global::System.String, global::KS.Shell.ShellBase.Commands.CommandInfo>();

        /// <summary>
        /// List of commands
        /// </summary>
                                                                                                                                                                                                                                                                public readonly static global::System.Collections.Generic.Dictionary<global::System.String, global::KS.Shell.ShellBase.Commands.CommandInfo> Commands = new global::System.Collections.Generic.Dictionary<global::System.String, global::KS.Shell.ShellBase.Commands.CommandInfo>() { { "adduser", new global::KS.Shell.ShellBase.Commands.CommandInfo("adduser", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Adds users", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<userName> [password] [confirm]" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.AddUserCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "alias", new global::KS.Shell.ShellBase.Commands.CommandInfo("alias", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Adds aliases to commands", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { $"<rem/add> <{global::System.String.Join("/", global::System.Enum.GetNames(typeof(global::KS.Shell.ShellBase.Shells.ShellType)))}> <alias> <cmd>" }, true, 3), new global::KS.Shell.Shells.UESH.Commands.AliasCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "arginj", new global::KS.Shell.ShellBase.Commands.CommandInfo("arginj", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Injects arguments to the kernel (reboot required)", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "[Arguments separated by spaces]" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.ArgInjCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "beep", new global::KS.Shell.ShellBase.Commands.CommandInfo("beep", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Beep in 'n' Hz and time in 'n' milliseconds", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<37-32767 Hz> <milliseconds>" }, true, 2), new global::KS.Shell.Shells.UESH.Commands.BeepCommand()) }, { "blockdbgdev", new global::KS.Shell.ShellBase.Commands.CommandInfo("blockdbgdev", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Block a debug device by IP address", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<ipaddress>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.BlockDbgDevCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "calc", new global::KS.Shell.ShellBase.Commands.CommandInfo("calc", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Calculator to calculate expressions.", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<expression>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.CalcCommand()) }, { "calendar", new global::KS.Shell.ShellBase.Commands.CommandInfo("calendar", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Calendar, event, and reminder manager", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<show> [year] [month]", "<event> <add> <date> <title>", "<event> <remove> <eventid>", "<event> <list>", "<event> <saveall>", "<reminder> <add> <dateandtime> <title>", "<reminder> <remove> <reminderid>", "<reminder> <list>", "<reminder> <saveall>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.CalendarCommand()) }, { "cat", new global::KS.Shell.ShellBase.Commands.CommandInfo("cat", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Prints content of file to console", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "[-lines|-nolines] <file>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.CatCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Wrappable) }, { "cdbglog", new global::KS.Shell.ShellBase.Commands.CommandInfo("cdbglog", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Deletes everything in debug log", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(), new global::KS.Shell.Shells.UESH.Commands.CdbgLogCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "chattr", new global::KS.Shell.ShellBase.Commands.CommandInfo("chattr", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes attribute of a file", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<file> +/-<attributes>" }, true, 2), new global::KS.Shell.Shells.UESH.Commands.ChAttrCommand()) }, { "chdir", new global::KS.Shell.ShellBase.Commands.CommandInfo("chdir", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes directory", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<directory/..>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.ChDirCommand()) }, { "chhostname", new global::KS.Shell.ShellBase.Commands.CommandInfo("chhostname", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes host name", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<HostName>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.ChHostNameCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "chlang", new global::KS.Shell.ShellBase.Commands.CommandInfo("chlang", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes language", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "[-alwaystransliterated|-alwaystranslated|-force] <language>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.ChLangCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "chmal", new global::KS.Shell.ShellBase.Commands.CommandInfo("chmal", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes MAL, the MOTD After Login", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "[Message]" }, false, 0), new global::KS.Shell.Shells.UESH.Commands.ChMalCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "chmotd", new global::KS.Shell.ShellBase.Commands.CommandInfo("chmotd", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes MOTD, the Message Of The Day", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "[Message]" }, false, 0), new global::KS.Shell.Shells.UESH.Commands.ChMotdCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "choice", new global::KS.Shell.ShellBase.Commands.CommandInfo("choice", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Makes user choices", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "[-o|-t|-m|-a] [-multiple|-single] <$variable> <answers> <input> [answertitle1] [answertitle2] ..." }, true, 3), new global::KS.Shell.Shells.UESH.Commands.ChoiceCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.SettingVariable) }, { "chpwd", new global::KS.Shell.ShellBase.Commands.CommandInfo("chpwd", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes password for current user", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<Username> <UserPass> <newPass> <confirm>" }, true, 4), new global::KS.Shell.Shells.UESH.Commands.ChPwdCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "chusrname", new global::KS.Shell.ShellBase.Commands.CommandInfo("chusrname", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes user name", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<oldUserName> <newUserName>" }, true, 2), new global::KS.Shell.Shells.UESH.Commands.ChUsrNameCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "clearfiredevents", new global::KS.Shell.ShellBase.Commands.CommandInfo("clearfiredevents", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Clears all fired events", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(), new global::KS.Shell.Shells.UESH.Commands.ClearFiredEventsCommand()) }, { "cls", new global::KS.Shell.ShellBase.Commands.CommandInfo("cls", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Clears the screen", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(), new global::KS.Shell.Shells.UESH.Commands.ClsCommand()) } };

    }
}

[GrahamTheCoder]

@EoflaOE thanks, that example is likely going to be a massive help! I’ll try to repro with it as soon as I can find time and hopefully will finally be able to make progress on this issue 😊