Многопоточный foreach в powershell core

One of the advanced features that can significantly enhance the efficiency of your scripts is multi-threading. In this blog post, we’ll explore how to use multi-threaded ForEach loops in PowerShell Core, with practical examples to illustrate the concepts. We’ll also use the Measure-Command cmdlet to compare the execution time of single-threaded and multi-threaded operations.

Edited with last version code:

This script runs just fine when steps are done in serial setting. Original script with while loop taking input items one be one, connect to API, get answer, save JSON answer to a file.

The problem: However it is enormously slow this way. Each cycle take around 1 minute and i have around 11,000 entries to process. It would take almost 7 days to finish.

The solution: based on this article: text. I have decide to use multi-thread job solution, where I can run multiple jobs in parallel.

The Issue: In this multi-tread solution, it seems, like each standalone JOB is finished prematurely. In few milliseconds whole batch of 11K files are saved, with proper notion of each variable. But all files are empty.
I would still expect (even for parallel run), that each JOB last 1 min in average. I am convinced, that script simply does not wait till it gets answer from API and immediately jump to next step save an empty file.

Here is the PowerShell script:

Attempt to solve the issue: I have tried to use various wait statements, to give each JOB to get answer from API, but nothing seems to help. Still even that would not be desired solution. I would like enforce script to wait till API part is finished. I am clueless. I will emphasize that script it self is running just fine, if it is serialized.

This is one that I should have learnt way earlier than I actually did if i’m being completely honest. One of my colleagues wrote a cool function for multithreading, and although this it, it did made me look into how it all worked.

Some of the techniques can be quite complicated, especially for a DBA who’s not as experienced with Powershell yet (and a ForEach loop just works right?) so I thought i’d post about the simplier methods that can be employed, and tweaks to get of the benefits, without the complexity.

:/> Windows 10 флешка вне сети

So, lets jump straight into it.

All good and well, unless you have a huge estate of servers to loop through, and/or a limited time to run the collection. For example if you wanted to collect metrics or perform a heartbeat check every 1 minute. A loop against a large set would take a while, we hate (c/o Jeff Moden ) in SQL, so why do it in Powershell?

Thankfully, Powershell has been making it easier for us for years, and continues to do so through through new functionality.

I’m going to focus on and an enhancement to them that does away with most of their limitations,

What is a Powershell Job?

A job is basically an asynchronous session running commands in the background. The key word here is asynchronous, so the job statement returns immediately, while the heavy lifting of the job is being done in the background. This allows for you to call multiple jobs, and have them all running in parallel. Lovely.

Well, potentially lovely, as these are all powershell.exe processes running, each with their memory requirements. Try and run too many of these at once, and your host OS may grind to a halt through resource starvation. Tough love.

Lets start with defining some basic info, and our target list.

So here, i’ve just set $throttle to the number of cores we have, and created an array ($Serverlist) with our list of servers. This could easily be your Invoke-sqlcmd command to your inventory tables, or a call to your CMS using Get-DbaRegServer from dbatools.

This is just

Output will look something like this.

:/> Как удалить все данные пользователя в Windows 7 отключение учетной записи? Как создать и удалять администратора на Win 10

Now we want to pull our results from those jobs. To do that, you use:

. Our results.

Returnedserver : SERVER1
Returnedtime : 27/01/2024 17:19:08

Returnedserver : SERVER2
Returnedtime : 27/01/2024 17:19:08

Returnedserver : SERVER3
Returnedtime : 27/01/2024 17:19:09

Thats it, your first parallel query.

But you told us to be careful of Jobs!

I did, as if you try to create too many at once, they can steal all your resource. So you may have to be careful and use a throttling method (using a $throttle variable and checking against current running jobs) as above. But this can slow things down lots, and seriously hurt the reasoning for running in parallel in the first place.

Where Angels Fear to Thread – ThreadJob

I like this as there is virtually no change to the code that you use for ‘normal’ Powershell Jobs.

Essentially what this does is massively lower resource requirements, as each job now runs as a separate , rather than a separate . Great, we can now eat more, faster. Yum.

As with any module – Install and Import.

Install-Module -Name ThreadJob
Import-module ThreadJob

Oh, and add a -ThrottleLimit $throttle parameter to throttle it fully at that level too.

Yup, that’s it. No other changes. Done.

Next up is a post around using the above to do a connectivity check of many SQL servers at the same time, and making the output a little more friendly.

Thanks for reading as ever.

(I’m not posh, seriously)

Using ForEach-Object -Parallel

To leverage parallel processing, you’ll use the ForEach-Object cmdlet with the -Parallel parameter. Here’s how you can rewrite the previous example to use multi-threading:

:/> Как удалить всё с компьютера Windows кроме Windows

In this example:

The Problem with Standard ForEach

Consider a scenario where you need to retrieve the content of multiple web pages. Using a standard ForEach loop, the script processes each URL sequentially, which can be time-consuming if you have a long list of URLs:

Measuring Execution Time

To understand the performance benefits of multi-threading, we can use the Measure-Command cmdlet to measure the time taken to execute both single-threaded and multi-threaded operations.

Single-threaded execution time: 5.02 seconds

Multi-threaded execution time: 1.68 seconds

As shown in the screenshots, the single-threaded execution took 5.02 seconds, whereas the multi-threaded execution took only 1.68 seconds. This demonstrates a significant improvement in execution time when using multi-threading.

This functionality can improve the performance of your scripts significantly. You might be wondering if there is a way to control the number of threads when running this? The answer is Yes!, you can use ThrottleLimit parameter specifies the maximum number of parallel threads that can run at the same time.

I hope this was useful and informative for you !!

Why Multi-Threading?

Multi-threading allows you to run multiple operations simultaneously, making your scripts run faster, especially when dealing with tasks that can be performed in parallel. This is particularly useful for tasks like checking the status of multiple web pages, file processing, and other repetitive tasks that don’t depend on each other.