Out of memory crash in 0.0.6

This issue is a follow-up to https://github.com/igorshubovych/markdownlint-cli/issues/108

Trying markdownlint-cli2 "**/*" in a project that has a large folder in .markdownlintignore results this output in about 5 mins:

<--- Last few GCs --->

[57132:0x102d52000]   340958 ms: Mark-sweep 2346.6 (2408.0) -> 2348.5 (2399.5) MB, 1070.8 / 0.0 ms  (average mu = 0.113, current mu = 0.028) allocation failure scavenge might not succeed
[57132:0x102d52000]   341633 ms: Mark-sweep 2348.5 (2399.5) -> 2346.6 (2391.3) MB, 675.3 / 0.0 ms  (average mu = 0.070, current mu = 0.000) allocation failure scavenge might not succeed


<--- JS stacktrace --->
Cannot get stack trace in GC.
FATAL ERROR: MarkCompactCollector: semi-space copy, fallback in old gen Allocation failed - JavaScript heap out of memory
 1: 0x1011c2ff5 node::Abort() (.cold.1) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
 2: 0x10009fbc9 node::Abort() [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
 3: 0x10009fd2f node::OnFatalError(char const*, char const*) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
 4: 0x1001e3907 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
 5: 0x1001e38a7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
 6: 0x1003695e5 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
 7: 0x1003bd837 v8::internal::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject, int) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
 8: 0x10039b58b void v8::internal::LiveObjectVisitor::VisitBlackObjectsNoFail<v8::internal::EvacuateNewSpaceVisitor, v8::internal::MajorNonAtomicMarkingState>(v8::internal::MemoryChunk*, v8::internal::MajorNonAtomicMarkingState*, v8::internal::EvacuateNewSpaceVisitor*, v8::internal::LiveObjectVisitor::IterationMode) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
 9: 0x10039acc8 v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
10: 0x10039aa06 v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
11: 0x1003c24ce v8::internal::PageEvacuationTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
12: 0x10038b7f2 v8::internal::ItemParallelJob::Task::RunInternal() [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
13: 0x10038bc78 v8::internal::ItemParallelJob::Run() [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
14: 0x10039dfb5 void v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator, v8::internal::MarkCompactCollector>(v8::internal::MarkCompactCollector*, v8::internal::ItemParallelJob*, v8::internal::MigrationObserver*, long) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
15: 0x10039daa5 v8::internal::MarkCompactCollector::EvacuatePagesInParallel() [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
16: 0x10038ff17 v8::internal::MarkCompactCollector::Evacuate() [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
17: 0x10038d92e v8::internal::MarkCompactCollector::CollectGarbage() [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
18: 0x10036992f v8::internal::Heap::MarkCompact() [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
19: 0x1003671b5 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
20: 0x100365670 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
21: 0x10037149a v8::internal::Heap::AllocateRawWithLightRetry(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
22: 0x100371521 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
23: 0x10033f73a v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
24: 0x10068e808 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
25: 0x1009d39b9 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
26: 0x1009d488d Builtins_StringAdd_CheckNone [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
27: 0x100a1060f Builtins_RegExpReplace [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
error Command failed with signal "SIGABRT".

Repro steps are the same as in markdownlint-cli2 "**/*" except that the ignored folder is even bigger (354K files distributed across about 6K subfolders)

[DavidAnson]

Great, thank you, I will close this! By the way, I made a couple of other refactorings intended to avoid unnecessary memory pressure. They will be in the next release.

Regarding your point about progress, note that the kind of progress used here is not the fancy kind that updates the same line in the console or anything like that. It just outputs three lines of plain text at the relevant parts of execution. And they can be turned off entirely if wanted.

Regarding double negation in the ignores property, I’m afraid that might get confusing. Remember that ignores are applied after the original globs from the command line, so they can not add back in files that were excluded there. They could add back in files that were excluded by the ignore pattern itself, but I’m afraid trying to explain and understand that might be difficult. I’m going to wait and see if this becomes an issue for users.

[DavidAnson]

I’ve added progress to stdout here: https://github.com/DavidAnson/markdownlint-cli2/commit/bb7c4ddf563c549deead9e056b3c2697470eef00

It can be turned off via the noProgress option.

Example output:

Finding: **/*.md
Linting: 3 file(s)
Summary: 9 error(s)