ForEach-Object -parallel missing elements during durn

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest released version
• Search the existing issues.
• Refer to the FAQ.
• Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

I am missing elements during a ForEach-Object parallel run. I expect to see all runs succeed, but I am missing random items.

$result = @{"start"=0; "end"=100000; "count"= 0; "concurrent" = 100}; ($result.start)..($result.end) | ForEach-Object -Parallel { ($using:result).count += 1; } -ThrottleLimit $result.concurrent Start-Sleep -Seconds 2 Write-Host "Total # of elements: $($result.count); Expected # of elements: $($result.end - $result.start + 1)" -ForegroundColor Blue if($result.count -ne ($result.end - $result.start + 1)) { Write-Host "Missing Element Detected" -ForegroundColor Red }

Expected behavior

Total # of elements: 100001; Expected # of elements: 100001

Actual behavior

Total # of elements: 99995; Expected # of elements: 100001
Missing Element Detected

Error details

No response

Environment data

Name                           Value
----                           -----
PSVersion                      7.2.5
PSEdition                      Core
GitCommitId                    7.2.5
OS                             Microsoft Windows 10.0.19044
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Visuals

No response

[jborean93]

At least here I don’t think so, += isn’t an atomic operation, the code still needs to:

• Get the original value from the dict
• Increment the value
• Set the new value to the dict

The problem is the getting and setting are 2 distinct operations. If those types had something like ($using:result).TryUpdate('count', {$_ + 1}) where the TryUpdate is doing the locking internally and calling the delegate to update the value but AFAIK this isn’t possible through any builtin type.

What you could do is use something like a ConcurrentBag and just add a random value and get the final count at the end

$result = @{
    start = 0
    end = 100000
    count = [System.Collections.Concurrent.ConcurrentBag[Object]]::new()
    concurrent = 100
}

($result.start)..($result.end) | ForEach-Object -Parallel {
    ($using:result).count.Add($null)
} -ThrottleLimit $result.concurrent

$result.count.Count

The downside is you now have a collection storing useless values and taking up memory for no reason. It still does the locking for you internally so technically simpler code.

Depending on OPs use case it may be possible, it sounds like this is just an example to reproduce their problem and they have an actual scenario that may find some use using some of the thread safe collections. But without knowing what the actual scenario is it’s hard to recommend something.

[pirion]

Depending on OPs use case it may be possible, it sounds like this is just an example to reproduce their problem and they have an actual scenario that may find some use using some of the thread safe collections. But without knowing what the actual scenario is it’s hard to recommend something.

Exactly as you said, use case of what we’re doing here calling a child script with various parameters. Co-worker running this was using latest and was finding that various instances were missing, so this was an attempt to put together a test case that would show the issue without all the overhead. I’ve made note of the thread safety issue and will reattempt with that in place to ensure it is still occurring in our environment.