Feature Request: ForEach-Object BatchSize Parameter

Summary of the new feature/enhancement

As a user I want my ForEach-Object -Parallel jobs to be evenly distributed so that they finish at about the same time (and not all except one and that one continues running for an excessive extra amount of time).

Example use case

1. This one works, but creates a thread for each and every host and because of the “initialization part” (loading the module and connecting to the server) it becomes quite bottlenecked.

$ansibleInventory["all"]["children"].GetEnumerator() ` # List of hosts grouped by tenant
| Where-Object {$null -ne $_.Value["hosts"]} ` # Filter out all tenants without hosts
| ForEach-Object {$_.Value["hosts"].GetEnumerator()} ` # Get a list of all host objects.
| ForEach-Object -Throttlelimit 20 -UseNewRunspace -AsJob -Parallel {
  Import-Module "VMware.PowerCLI"
  Connect-ViServer ...
  Do-Stuff
}

1. Iterating over each tenant in parallel and within that loop iterate over each customer vm. This however results in an uneven distribution. For example customer 0-9 have 100 vms each, but customer 10 has 1000 vms. We’re now basically back at single threaded performance.

$ansibleInventory["all"]["children"].GetEnumerator() ` # List of hosts grouped by tenant
| Where-Object {$null -ne $_.Value["hosts"]} ` # Filter out all tenants without hosts
| ForEach-Object -Throttlelimit 20 -UseNewRunspace -AsJob -Parallel {
  $_.Value["hosts"].GetEnumerator() `
  | ForEach-Object {
    Import-Module "VMware.PowerCLI"
    Connect-ViServer ...
    Do-Stuff
  }
} `

Proposed technical implementation details (optional)

Adding either a Batch-Object -Size command, or add -BatchSize as an additional parameter to ForEach-Object

$ansibleInventory["all"]["children"].GetEnumerator() `
| Where-Object {$null -ne $_.Value["hosts"]} `
| ForEach-Object {$_.Value["hosts"].GetEnumerator()} `
| ForEach-Object -Throttlelimit 20 -BatchSize 20 -UseNewRunspace -AsJob -Parallel {
  Import-Module "VMware.PowerCLI"
  Connect-ViServer ...
  Do-Stuff
}

or

$ansibleInventory["all"]["children"].GetEnumerator() `
| Where-Object {$null -ne $_.Value["hosts"]} `
| ForEach-Object {$_.Value["hosts"].GetEnumerator()} `
| Batch-Object -Size 20`
| ForEach-Object -Throttlelimit 20 -UseNewRunspace -AsJob -Parallel {
  $_ | ForEach-Object {
    Import-Module "VMware.PowerCLI"
    Connect-ViServer ...
    Do-Stuff
  }
}

[MartinGC94]

Not to shill myself too much but I recently published my SplitCollection module to the PS gallery which splits arrays into smaller arrays like this:

PS C:\> $DemoArray=1..20
PS C:\> $SplitArray=Split-Collection -InputObject $DemoArray -ChunkSize 2
PS C:\> $SplitArray.Count
10
PS C:\> $SplitArray[0].Count
2
PS C:\> $SplitArray=Split-Collection -InputObject $DemoArray -AmountOfParts 2
PS C:\> $SplitArray.Count
2
PS C:\> $SplitArray[0].Count
10

I think this should work in your scenario OP. For anyone curious the source code is available here: https://github.com/MartinGC94/SplitCollection/blob/master/SplitCollection/SplitCollectionCommand.cs

<div> GitHub</div><div>MartinGC94/SplitCollection</div><div>Contribute to MartinGC94/SplitCollection development by creating an account on GitHub.</div>

[mklement0]

If I understand your intent correctly, then a general-purpose batching mechanism may solve your problem. Such a mechanism is proposed in #8270.

Also, can you please update the OP to provide a syntax-highlighting hint for the code blocks? Currently, the code is hard to read; use ```powershell