question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Memory leak when foreach-object parallel

See original GitHub issue

Steps to reproduce

$scriptThrottleLimit = 4
$scriptTimeoutLimit = 0
$ErrorActionPreference = Continue
$test = Get-Aduser -Filter *
$test.count
14483

 $test.samaccountname | Foreach-Object -Parallel {
     Write-Output $_
 } -ThrottleLimit $scriptThrottleLimit -TimeoutSeconds $scriptTimeoutLimit

Expected behavior

works normally, process all object

Actual behavior

After processing some pack of objects, server goes to out of memory, then pwsh stop using cpu (0-1%) and stop processing pipe. Only stop process works. I tried different numbers in throttlelimit, no change, it just slow memory leak.
https://prnt.sc/ueti4c
http://prntscr.com/uetj45
i think it not clean after each of parallel instance

Environment data


Name                           Value
----                           -----
PSVersion                      7.0.3
PSEdition                      Core
GitCommitId                    7.0.3
OS                             Microsoft Windows 10.0.14393
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Issue Analytics

  • State:open
  • Created 3 years ago
  • Comments:9 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
PaulHigincommented, Sep 10, 2020

This version of PowerShell does not have the perf improvement for ForEach-Object -Parallel, and creates and destroys a runspace for each loop iteration (perf improvement in 7.1 preview reuses runspaces by default).

But I have tested both versions for leaks, and haven’t found any.

However, running scripts in parallel does use a lot of resources. Try adding [System.GC]::Collect() to your loop to help ensure CLR recovers resources.

I notice two things in your test repro above:

  1. You store output from Get-ADUser, which may use a lot of resources depending on how large the returned objects are. Instead you could just stream directly to For-EachObject -Parallel.

  2. Your loop script writes to output/console, which is a serial operation. So your repro expends a tremendous amount of resources to parallelize something that is immediately re-serialized, and essentially provides no benefit.

1reaction
iSazonovcommented, Sep 10, 2020

/cc @PaulHigin for information.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Memory usage powershell 7.03 Foreach-object Parallel
My solution was to move all of the code I had inside the forEach parallel into a function outside and above the forEach...
Read more >
Huge memory consumption using ForEach-Object -Parallel
I've a function that updates the stats in databases using ForEach-Object -Parallel. When I run it, to loop through the stats in a...
Read more >
PowerShell 7 ForEach-Object -Parallel Memory Consumption
I'm undertaking a project where one of of the tasks involves processing some 350000 XML files. If each file takes one second to...
Read more >
Powershell 7's parallel ForEach-Object is mind blowing.
I just installed v7 yesterday and have been putting it through the paces to see what I can use it for by overhauling...
Read more >
Curious Case of Parallel.ForEach with BlockingCollection
ForEach together with a BlockingCollection. The application was continuously eating up the memory until no more incoming data could be processed ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found