Parallel Processing in PowerShell

by Apr 11, 2014

If a script needs some speed-up, you might find background jobs helpful. They can be used if a script does a number of things that also could run concurrently.

PowerShell is single-threaded and can only do one thing at a time. With background jobs, additional PowerShell processes run in the background and can share the workload. This works well only if the jobs you need to do are completely independent from each other, and if your background job does not need to produce a lot of data. Sending back data from a background job is an expensive procedure that can easily eat up all the saved time, resulting in an even slower script.

Here are three tasks that all can run concurrently:

$start = Get-Date

# get all hotfixes
$task1 = { Get-Hotfix }

# get all scripts in your profile
$task2 = { Get-Service | Where-Object Status -eq Running }

# parse log file
$task3 = { Get-Content -Path $env:windir\windowsupdate.log | Where-Object { $_ -like '*successfully installed*' } }

# run 2 tasks in the background, and 1 in the foreground task
$job1 =  Start-Job -ScriptBlock $task1 
$job2 =  Start-Job -ScriptBlock $task2 
$result3 = Invoke-Command -ScriptBlock $task3

# wait for the remaining tasks to complete (if not done yet)
$null = Wait-Job -Job $job1, $job2

# now they are done, get the results
$result1 = Receive-Job -Job $job1
$result2 = Receive-Job -Job $job2

# discard the jobs
Remove-Job -Job $job1, $job2

$end = Get-Date
Write-Host -ForegroundColor Red ($end - $start).TotalSeconds

On a sample system, executing all three tasks took 5.9 seconds. The results for all three tasks are available in $result1, $result2, and $result3.

Let's check how long it takes for all three tasks to run consecutively in the foreground.

$start = Get-Date

# get all hotfixes
$task1 = { Get-Hotfix }

# get all scripts in your profile
$task2 = { Get-Service | Where-Object Status -eq Running }

# parse log file
$task3 = { Get-Content -Path $env:windir\windowsupdate.log | Where-Object { $_ -like '*successfully installed*' } }

# run them all in the foreground:
$result1 = Invoke-Command -ScriptBlock $task1 
$result2 = Invoke-Command -ScriptBlock $task2 
$result3 = Invoke-Command -ScriptBlock $task3

$end = Get-Date
Write-Host -ForegroundColor Red ($end - $start).TotalSeconds

As it turns out, this time it only took 5.05 seconds. So background jobs really pay off for long running tasks that all take almost the same time. Since the three sample tasks returned a lot of data, the benefit of executing them concurrently was eliminated by the overhead that it took to serialize the return data and transport it back to the foreground process.

Twitter This Tip! ReTweet this Tip!