Written by Luke

Concatenate PowerShell arrays on the fly

The context

Sometimes you need to process data coming from various parts of a tree.
For example:

  • Different folders of one or several volumes 
    C:\Program Files\App1\Logs
D:\App2\Logs
E:\Logs
  • Different OUs of one or several Active Directories 
    OU=Management,DC=MyDomain,DC=Com
OU=InternalUsers,DC=MyDomain,DC=Com
OU=ExternalUsers,DC=MyDomain,DC=Com

Once the data is collected, you have two options for the processing part, each one having its pros and cons.

  • Repeat the processing for each source of collected data
    • The code executes faster because the data collection is only processed once
    • You have a redundancy of code because you repeat the code for each collection of data
  • Concatenate all data into one array and process the whole in one bunch
    • The code execution is slower because you preprocess the data to create a single collection
    • The code is optimized, cleaner and easier to maintain

Concatenation methods

There are many ways to concatenate arrays.
Here are some of the most popular and well-known methods.

Note: I will not comment on all these methods except the last one (Enumeration on the fly), because you can find many articles explaining the pros and cons for each one of them.

The destroying method

The array $ConcatenatedArrays is each time destroyed and replaced by the concatenation of both itself and the other array.

$ConcatenatedArrays += $Array1
$ConcatenatedArrays += $Array2
$ConcatenatedArrays += $Array3

The addition method

$ConcatenatedArrays = $Array1 + $Array2 + $Array3

Note:
Depending on the objects you handle, this method may give unexpected results, like getting an array of arrays instead of an array of all elements as single elements.

Enumeration

foreach($item in $Array1){$ConcatenatedArrays = $ConcatenatedArrays + $item}
foreach($item in $Array2){$ConcatenatedArrays = $ConcatenatedArrays + $item}
foreach($item in $Array3){$ConcatenatedArrays = $ConcatenatedArrays + $item}

Enumeration on the fly

Among all those know methods I recently proposed this one as answer on StackOverFlow.

$ConcatenatedArrays = $($Array1;$Array2;$Array3)

You need:

  • Parentheses preceded by a dollar sign
  • Elements inside the parentheses separated by semicolons (and NOT commas)

Technically, and without going too much in depth, this method executes the arrays inside the parentheses as code, by enumerating the values.
And, the $ sign before the parentheses indicates to PowerShell that the whole thing between the parentheses is to be considered as a value.
We can even list all arrays inside the parentheses without semicolons and use new lines instead.

$ConcatenatedArrays = $(
  $Array1
  $Array2
  $Array3
)

Why “on the fly”

This method can be used to build the global array at the same time you collect the data.
No need to do an extra operation to concatenate your data!
Let’s use the examples from the beginning of this article.
Example 1:

$AllFiles = $(
  Get-ChildItem -Path 'C:\Program Files\App1\Logs'
  Get-ChildItem -Path 'D:\App2\Logs'
  Get-ChildItem -Path 'E:\Logs'
)

Example 2:

$AllUsers = $(
  Get-AdUser -Filter * -SearchBase 'OU=Management,DC=MyDomain,DC=Com'
  Get-AdUser -Filter * -SearchBase 'OU=InternalUsers,DC=MyDomain,DC=Com'
  Get-AdUser -Filter * -SearchBase 'OU=ExternalUsers,DC=MyDomain,DC=Com'
)

Wait there is more!

This method is especially useful for one-liners.
Example 1:

$(
  Get-ChildItem -Path 'C:\Program Files\App1\Logs'
  Get-ChildItem -Path 'D:\App2\Logs'
  Get-ChildItem -Path 'E:\Logs'
) | ForEach-Object -Process {
    #Doing my stuff here on all collected files
}

Example 2:

$(
  Get-AdUser -Filter * -SearchBase 'OU=Management,DC=MyDomain,DC=Com'
  Get-AdUser -Filter * -SearchBase 'OU=InternalUsers,DC=MyDomain,DC=Com'
  Get-AdUser -Filter * -SearchBase 'OU=ExternalUsers,DC=MyDomain,DC=Com'
) | ForEach-Object -Process {
    #Doing my stuff here on all collected user accounts
}

And what about the speed?

Another important topic is to know if the concatenation on the fly is faster or slower than the other methods.
For this purpose, I will collect four lists of files and concatenate them into a single list.
You can unfold the line below to see the whole code:

$MaxIteration = 100

#region Destroying method
'Destroying method'

Measure-Command -Expression {
  1..$MaxIteration|%{

    [array]$a = Get-ChildItem -path C:\ProgramData -File
    [array]$b = Get-ChildItem -path C:\Windows -File
    [array]$c = Get-ChildItem -path C:\Windows\System32 -File

    $ConcatenatedArray = $a
    $ConcatenatedArray += $b
    $ConcatenatedArray += $c

    $ConcatenatedArray |
    ForEach-Object -Process {
      $PSItem | Out-Null
    }
  }
}
#endregion Destroying method

#region Addition method
'Addition method'

Measure-Command -Expression {
  1..$MaxIteration|%{

    [array]$a = Get-ChildItem -path C:\ProgramData -File
    [array]$b = Get-ChildItem -path C:\Windows -File
    [array]$c = Get-ChildItem -path C:\Windows\System32 -File

    $ConcatenatedArray = $a + $b + $c

    $ConcatenatedArray |
    ForEach-Object -Process {
      $PSItem | Out-Null
    }
  }
}
#endregion Addition method

#region Enumeration method
'Enumeration method'

Measure-Command -Expression {
  1..$MaxIteration|%{

    [array]$a = Get-ChildItem -path C:\ProgramData -File
    [array]$b = Get-ChildItem -path C:\Windows -File
    [array]$c = Get-ChildItem -path C:\Windows\System32 -File

    [array]$ConcatenatedArray = $null

    foreach($i in $a){
      $ConcatenatedArray = $ConcatenatedArray + $i
    }

    foreach($i in $b){
      $ConcatenatedArray = $ConcatenatedArray + $i
    }

    foreach($i in $c){
      $ConcatenatedArray = $ConcatenatedArray + $i
    }

    $ConcatenatedArray |
    ForEach-Object -Process {
      $PSItem | Out-Null
    }
  }
}
#endregion Enumeration method

#region Concatenation on the fly
'Concatenation on the fly'

Measure-Command -Expression {
  1..$MaxIteration|%{

    $(
      Get-ChildItem -path C:\ProgramData -File
      Get-ChildItem -path C:\Windows -File
      Get-ChildItem -path C:\Windows\System32 -File
    )|
    ForEach-Object -Process {
      $PSItem | Out-Null
    }
  }
}
#endregion Concatenation on the fly

And you can unfold the line below to see the result:

Destroying method

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 33
Milliseconds      : 333
Ticks             : 333337604
TotalDays         : 0.000385807412037037
TotalHours        : 0.00925937788888889
TotalMinutes      : 0.555562673333333
TotalSeconds      : 33.3337604
TotalMilliseconds : 33333.7604

Addition method

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 32
Milliseconds      : 899
Ticks             : 328990928
TotalDays         : 0.000380776537037037
TotalHours        : 0.00913863688888889
TotalMinutes      : 0.548318213333333
TotalSeconds      : 32.8990928
TotalMilliseconds : 32899.0928

Enumeration method

Days              : 0
Hours             : 0
Minutes           : 1
Seconds           : 57
Milliseconds      : 36
Ticks             : 1170368437
TotalDays         : 0.00135459309837963
TotalHours        : 0.0325102343611111
TotalMinutes      : 1.95061406166667
TotalSeconds      : 117.0368437
TotalMilliseconds : 117036.8437

Concatenation on the fly

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 33
Milliseconds      : 418
Ticks             : 334184004
TotalDays         : 0.000386787041666667
TotalHours        : 0.009282889
TotalMinutes      : 0.55697334
TotalSeconds      : 33.4184004
TotalMilliseconds : 33418.4004

As you can notice, except the enumeration method, which is very slower, all other methods are equivalent in speed.
So it’s up to you to choose the method you feel the most comfortable with… :)

3 thoughts on “Concatenate PowerShell arrays on the fly”

  1. I would also want to express the idea of using a different type of list instead of a fixed array using an array list.

    Baseline: using the ‘Destroying method’ (27.423157 seconds)

    The command structure would be one of 2 methods:

    List of Arrays ( This is technically 3 objects in that list but was highest on cost savings for larger amounts of items individually)

    Measure-Command -Expression {
    1..$maxiteration | % {
    $ConcatenatedArray = New-Object System.Collections.ArrayList

          [Array]$a = Get-ChildItem -Path C:\ProgramData -File
      [Array]$b = Get-ChildItem -path C:\Windows -file
      [Array]$c = Get-ChildItem -path C:\Windows\System32 -File

      $ConcatenatedArray.Add($a) | out-null
      $ConcatenatedArray.Add($b) | out-null
      $ConcatenatedArray.Add($c) | out-null

      $ConcatenatedArray | Foreach-Object -Process {
                $psitem | out-null
      }
 }

    }

    Result: 14.9025176 seconds

    Enumerated Array List

    Measure-Command -Expression {
    1..$maxiteration | % {
    $ConcatenatedArray = New-Object System.Collections.ArrayList

          [Array]$a = Get-ChildItem -Path C:\ProgramData -File
      [Array]$b = Get-ChildItem -path C:\Windows -file
      [Array]$c = Get-ChildItem -path C:\Windows\System32 -File

      $a, $b, $c | foreach-Object -Process { $ConcatenatedArray.Add($psitem) }

      $ConcatenatedArray | Foreach-Object -Process {
                $psitem | out-null
      }
 }

    }

    Result: 16.7182658 Seconds

    Might give that one a try :). Love the blog. Keep it up

    Like

    Reply

  2. Apologies, a minor issue came up when I was testing larger arrays. (Basically creating the array outside of the loop and adding it to it over 100 times).

      $a, $b, $c | foreach-Object -Process { $ConcatenatedArray.Add($psitem) }

    Should be —
    $a | foreach-Object -Process { $ConcatenatedArray.Add($psitem) }
    $b | foreach-Object -Process { $ConcatenatedArray.Add($psitem) }
    $c | foreach-Object -Process { $ConcatenatedArray.Add($psitem) }

    Which using this method was ~485,700 items. Using this above method took ~40.8 seconds to create the list. Oh could also use the object type System.Collections.Generic.list[object] (Which got it down to ~31 seconds.)

    ~Enjoy~

    Like

    Reply

Leave a comment