Путь к целостности и эффективности данных

By the end, you’ll be familiar with several .NET classes and understand how to discover and use others that might better meet your specific needs.

Listing Available Methods and Properties of .NET Classes

If you’re wondering what other static methods LINQ has, you may have tried to pipe the class to Get-Member. You may have also tried to do the same for a List<T> object and found unexpected results.

What you’ll find with collections like List<T> is Get-Member returns data of the type of the first item inside the collection.

 C:\>  = ::new

 C:\>   
: You must specify an object  the  cmdlet

 C:\> Add1

 C:\>   

   TypeName: SystemInt32 

Name                 MemberType Definition
                  
CompareTo            Method     int CompareToSystemObject value int CompareToint value int IComparableCompareToSystemObject obj int IComparableCompareToint other
Equals               Method     bool EqualsSystemObject obj bool Equalsint obj bool IEquatableEqualsint other
GetByteCount         Method     int IBinaryIntegerGetByteCount
GetHashCode          Method     int GetHashCode
GetShortestBitLength Method     int IBinaryIntegerGetShortestBitLength

We can use GetMembers() to get the list of methods and properties. When calling it on an already instantiated object we first need to bubble up the class using GetType(), and then call GetMembers().

 C:\>  C:\> GetTypeGetMembers  ? MemberType     ExpandProperty Name Unique
get_Capacity
set_Capacity
get_Count
get_Item
set_Item
Add
AddRange
AsReadOnly
BinarySearch
Clear
Contains
ConvertAll
CopyTo
EnsureCapacity
Exists
Find

If calling it on a type, we can omit the GetType() call. Here are some of the LINQ methods available:

 C:\> GetMembers  ? MemberType    ? IsStatic   ExpandProperty Name Unique
Empty
Aggregate
Any
All
Append
Prepend
Average
OfType
Cast
Chunk
Concat
Contains
Count
TryGetNonEnumeratedCount
LongCount
DefaultIfEmpty
Distinct
DistinctBy
ElementAt
ElementAtOrDefault
AsEnumerable
Except
ExceptBy
First
$cyberarkAccounts = Get-ADUser -identity users01 -Properties samaccountname,enabled,memberof

$cyberarkAccountTable = @{}

foreach ($account in $cyberarkAccounts) {

$cyberarkAccountTable[$account.samaccountname] = $account.MemberOf  | %{(Get-ADGroup $_).sAMAccountName} |  Where-Object { $_ -like 'apps Users*' -or $_ -like 'apps test Users*'  }

}
Name       Value
---        ----
Users01
Users02    apps users
Users03    apps users,apps test Users

My desired output:

Name       Value
---        ----
Users01     NULL
Users02    apps users
Users03    apps users,apps test Users

Santiago Squarzon's user avatar

Arbelac's user avatar

$cyberarkAccounts = Get-ADUser -Identity users01 -Properties samaccountname, enabled, memberOf
$cyberarkAccountTable = @{}
foreach ($account in $cyberarkAccounts) {
    $result = $account.MemberOf |
        ForEach-Object { (Get-ADGroup $_).sAMAccountName } |
        Where-Object { $_ -like 'apps Users*' -or $_ -like 'apps test Users*' }
    $cyberarkAccountTable[$account.samaccountname] = if ($result) { $result -join ',' } else { 'NULL' }
}
$targetGroups = 'apps Users', 'apps test Users'
$groupMap = @{} # maps key: each item in `$targetGroups` value: hashset with each group member samAccountName
$targetGroups | ForEach-Object {
    $groupDn = (Get-ADGroup $_).DistinguishedName
    $members = (Get-ADUser -LDAPFilter "(memberOf=$groupDn)").SamAccountName
    $groupMap[$_] = [System.Collections.Generic.HashSet[string]]::new(
        [string[]] $members)
}

$cyberarkAccounts = 'user1', 'user2', 'useretc' # these should be samAccountNames!
foreach ($account in $cyberarkAccounts) {
    $membership = foreach ($group in $targetGroups) {
        if ($groupMap[$group].Contains($account)) {
            $group
        }
    }

    [pscustomobject]@{
        User       = $account
        Membership = if ($membership) { $membership -join ',' } else { 'NULL' }
    }
}

Santiago Squarzon's user avatar


PowerShell is the best option for this kind of file operation. Let’s check each method with a complete example.

Method 1: Using Get-Unique Cmdlet

The PowerShell Get-Unique cmdlet is designed to compare each item in a sorted list to the next item, eliminate duplicates, and return only one instance of each item. This method requires the input to be sorted first.

Here is how it will work:

  1. Read the file and sort the content: $lines = Get-Content “C:\MyFolder\MyExample.txt” | Sort-Object
  2. Get unique lines: $uniqueLines = $lines | Get-Unique
  3. Output the unique lines to a new file: $uniqueLines | Set-Content “C:\MyFolder\MyExample_unique.txt”

This method ensures you get unique lines, below is the complete script.

$lines = Get-Content "C:\MyFolder\MyExample.txt" | Sort-Object

$uniqueLines = $lines | Get-Unique

$uniqueLines | Set-Content "C:\MyFolder\MyExample_unique.txt"

$uniqueLines

I executed the above script, and it showed me the output in the console and written in the text file. Check out the screenshot below.

Get Unique Lines from a File Using PowerShell

Method 2: Using Select-Object with -Unique Parameter

Let me show you another relatively simple method. In PowerShell, you can use the Select-Object with the -Unique parameter. When we use the -Unique parameter, we do not need to sort the input.

This is how it will work:

  1. Read the file and get unique lines: $uniqueLines = Get-Content “C:\MyFolder\MyExample.txt” | Select-Object -Unique
  2. Output the unique lines to a new file: $uniqueLines | Set-Content “C:\MyFolder\MyExample_unique.txt”

Here is the complete PowerShell script:

$uniqueLines = Get-Content "C:\MyFolder\MyExample.txt" | Select-Object -Unique
$uniqueLines | Set-Content "C:\MyFolder\MyExample_unique.txt"
$uniqueLines

Check out the screenshot below, it is giving the output like the above after I executed the PowerShell script using VS code.

PowerShell Get Unique Lines from a File

Method 3: Using HashSet

If you are working with a large text file, you can use the HashSet to get unique lines in PowerShell.

Here is how it works.

  1. Initialize the HashSet and read the file: $hashSet = [System.Collections.Generic.HashSet[string]]::new() $lines = Get-Content "C:\MyFolder\MyExample.txt"
  2. Add each line to the HashSet: foreach ($line in $lines) { $hashSet.Add($line) | Out-Null }
  3. Output the unique lines to a new file: $hashSet | Set-Content "C:\MyFolder\MyExample_unique.txt"

Here is the complete script.

# Step 1: Initialize the HashSet and read the file
$hashSet = [System.Collections.Generic.HashSet[string]]::new()
$lines = Get-Content "C:\MyFolder\MyExample.txt"

# Step 2: Add each line to the HashSet
foreach ($line in $lines) {
    $hashSet.Add($line) | Out-Null
}

# Step 3: Output the unique lines to a new file
$hashSet | Set-Content "C:\MyFolder\MyExample_unique.txt"

Method 4: Using Group-Object Cmdlet

Let me show you the final method.

You can also use the Group-Object cmdlet in PowerShell to group and then select unique lines.

Here is how it works:

  1. Read the file and group by line content: $groupedLines = Get-Content "C:\MyFolder\MyExample.txt" | Group-Object
  2. Select unique lines: $uniqueLines = $groupedLines | ForEach-Object { $_.Name }
  3. Output the unique lines to a new file: $uniqueLines | Set-Content "C:\MyFolder\MyExample_unique.txt"

Below is the complete script.

# Step 1: Read the file and group by line content
$groupedLines = Get-Content "C:\MyFolder\MyExample.txt" | Group-Object

# Step 2: Select unique lines
$uniqueLines = $groupedLines | ForEach-Object { $_.Name }

# Step 3: Output the unique lines to a new file
$uniqueLines | Set-Content "C:\MyFolder\MyExample_unique.txt"

$uniqueLines

After I executed the above script, you can see the output in the screenshot below.

How to Get Unique Lines from a File Using PowerShell

Conclusion

I hope now you have a complete idea of how to get unique lines from a file in PowerShell using various methods, such as the Get-Unique Cmdlet, the Select-Object with -Unique Parameter, and the Group-Object Cmdlet.

PowerShell is an interactive shell and scripting language from Microsoft. It’s object-oriented — and that’s not just a buzzword, that’s a big difference to how the standard Unix shells work. And it is actually usable as an interactive shell.

Getting Started

PowerShell is so nice, Microsoft made it twice.

Specifically, there concurrently exist two products named PowerShell:

  • Windows PowerShell (5.1) is a built-in component of Windows. It is proprietary, Windows-only, and is based on the equally proprietary and equally Windows-only .NET Framework 4.x. It has a blue icon.
  • PowerShell (7.x), formerly known as PowerShell Core, is a stand-alone application. It is MIT-licensed (developed on GitHub), available for Windows, Linux, and macOS, and is based on the equally MIT-licensed and equally multi-platform .NET (formerly .NET Core). It has a black icon.

Windows PowerShell development stopped when PowerShell (Core) came out. There are some niceties and commands missing in it, but it is still a fine option for trying it out or for when one can’t install PowerShell on a Windows system but need to solve something with code.

All examples in this post should work in either version of PowerShell on any OS (unless explicitly noted otherwise).

Install the modern PowerShell: Windows, Linux, macOS.

Objects? In my shell?

Let’s try getting a directory listing. This is Microsoft land, so let’s try the DOS command for a directory listing — that would be dir:

 

     

                           
                           
                              
                            
                             
                            
                             
                             

This looks like a typical (if slightly verbose) file listing.

Now, let’s try to do something useful with this. Let’s get the total size of all .txt files.

In a Unix shell, one option is du -bc *.txt. The arguments: -b (--bytes) gives the real byte size, and -c (--summarize) produces a total. The result is this:




Let’s try something in PowerShell. If we do $x = dir, we’ll have the output of the dir command in $x. Let’s try to analyse it further, is the first character a newline?

   
 

     

                           
                           
                              
 

     

                           
                           
                            

What if we try getting the Length property out of that?

 

It turns out that dir returns an array of objects, and PowerShell knows how to format this array (and a single item from the array) into a nice table. What can we do with it? This:

    
      
   

              
           
                
           
           
 
          

    
      
   

    
     

     
     

       
     
   

We can iterate over all file objects, get their length (using ForEach-Object and a lambda), and then use Measure-Object to compute the sum (Measure-Object returns an object, we need to get its Sum property). We can replace the ForEach-Object call with the -Property argument in Measure-Object. And if we want to look into subdirectories, we can easily add -Recurse to Get-ChildItem. We get actual integers we can do math on.

:/>  Включение телнет в windows 7

You might have noticed I used Get-ChildItem instead of dir in the previous example. Get-ChildItem is the full name of the command (cmdlet). dir is one of its aliases, alongside gci and ls (Windows-only to avoid shadowing /bin/ls). Many common commands have aliases defined for easier typing and ease of use — Copy-Item can be written as cp (for compatibility with Unix), copy (for compatibility with MS-DOS), and ci. In our examples, we could also use measure for Measure-Object and foreach or % for ForEach-Object. Those aliases are a nice thing to have for interactive use, but for scripts, it’s best to use the full names for readability, and to avoid depending on the environment for those aliases.

More filesystem operations

Files per folder

There’s a photo collection in a Photos folder, grouped into folders. The objective is to see how many .jpg files are in each folder. Here’s the PowerShell solution:

   
      
     
                       
                       
                             
                         
  • Group by $_.Directory.Name (same as before) to get two
  • Group by Split-Path -Parent ([System.IO.Path]::GetRelativePath("$PWD/Photos", $_.FullName)) to get one/two
  • Group by ([System.IO.Path]::GetRelativePath("$PWD/Photos", $_.FullName)).Split([System.IO.Path]::DirectorySeparatorChar)[0] to get one

(All of the above examples work for a single folder as well. The latter two examples don’t work on Windows PowerShell.)

Duplicate finder

Let’s build a simple tool to detect byte-for-byte duplicated files. Get-FileHash is a shell built-in. We can use Group-Object again, and Where-Object to filter only matching objects. Computing the hash of every file is quite inefficient, so we’ll group by the file length first, and then ensure the hashes match. This gives us a nice pipeline of 6 commands:

# Fully spelled out
   
      
        
     
         
       

# Using aliases
  
     
       
     
        
      

# Using less readable aliases
  
     
        
     
        
       

Serious Scripting: Software Bill of Materials

Software Bills of Materials (SBOMs) and supply chain security are all the rage these days. The boss wants to have something like that, i.e. a CSV file with a list of packages and versions, and only the direct production dependencies. Sure, there exist standards like SPDX, but the boss does not like those pesky “standards”. The backend is written in C#, and the frontend is written in Node.js. Since we care only about the production dependencies, we can look at the .csproj and package.json files. For Node packages, we’ll also try to fetch the license name from the npm API (the API is a bit more complicated for NuGet, so we’ll keep it as a TODO in this example).

   # stop execution on any error
  

  
        
     
      
          
          
          
          
      
    


  
        
      
    
        
        
        
          
    
  


  
   
       
      
    
     
  


      
      

      
        
      

    
      

Just like every well-written shell script starts with set -euo pipefail, every PowerShell script should start with $ErrorActionPreference = "Stop" so that execution is stopped as soon as something goes wrong. Note that this does not affect native commands, you still need to check $LASTEXITCODE. Another useful early command is Set-StrictMode -Version 3.0 to catch undefined variables.

For .csproj files, which are XML, we look for PackageReference elements using XPath, and then build a PSCustomObject out of a hashmap — extracting the appropriate attributes from the PackageReference nodes.

For package.json, we read the file, parse the JSON, and extract the properties of the dependencies object (it’s a map of package names to versions). To get the license, we use Invoke-RestMethod, which takes care of parsing JSON for us.

In the main body of the script, we look for the appropriate files (skipping things under node_modules) and call our parser functions. After retrieving all data, we concatenate the two arrays, convert to CSV, and use Tee-Object to output to a file and to standard output. We get this:














Could it be done in a different language? Certainly, but PowerShell is really easy to integrate with CI, e.g. GitHub Actions or Azure Pipelines. On Linux, you might be tempted to use Python — and you could get something done equally simply, as long as you don’t mind using the ugly urllib.request library, or alternatively ensuring requests is installed (and then you get into the hell that is Python package management).

Using .NET classes

PowerShell is built on top of .NET. This isn’t just the implementation technology — PowerShell gives access to everything the .NET standard library offers. For example, the alternate ways to group photos in multiple subdirectories we’ve explored above involve a call to a static method of the .NET System.IO.Path class.

Other .NET types are also available. Need a HashSet? Here goes:

    
 

 

   
 

   

   

It is also possible to load any .NET DLL into PowerShell (as long as it’s compatible with the .NET version PowerShell is built against) and use it as usual from C# (although possibly with slightly ugly syntax).

Sick Windows Tricks

Microsoft supposedly killed off Internet Explorer last year. Attempting to launch iexplore.exe will bring up Microsoft Edge. But you see, Internet Explorer is a crucial part of Windows, and has been so for over two decades. Software vendors have built software that depends on IE being there and being able to show web content. Some of them are using web views, but some of them prefer something else: COM.

COM, or Component Object Model, is Microsoft’s thing for interoperability between different applications and/or components. COM is basically a way for classes offered by different vendors and potentially written in different languages to talk to one another. Under the hood, COM is C++ vtables plus standard reference counting and class loading/discovery mechanisms. The .NET Framework, and its successor .NET, have always included COM interoperability. The modern WinRT platform is COM on steroids.

Coming back to Internet Explorer, it exposes some COM classes. They were not removed with iexplore.exe. This means you can bring up a regular Internet Explorer window in just two lines of PowerShell:

    
  

We have already explored the possibility of using classes from .NET. .NET comes with a GUI framework named Windows Forms, which can be loaded from PowerShell and used to build a GUI. There is no form designer, so it requires manually defining and positioning controls, but it actually works.

   
    

Getting out of PowerShell land

As a shell, PowerShell can obviously launch subprocesses. Unlike something like Python, running a subprocess is as simple as running anything else. If you need to git pull, you just type that. Or you can make PowerShell interact with non-PowerShell commands, reading output and passing arguments:

     
    
   "Not a git repository"
     
   "Getting changes from git failed"


    
   "No changes found"
  
    
     
        
        
  

  # Alternate spelling for regex fans:
    
     
          
         
  

    
      untracked files in VS Code"
     
    
     "No untracked files"
  

I chose to compute untracked files with the help of standard .NET string manipulation methods, but there’s also a regex option. On a related note, there are three content check operators: -match uses regex, -like uses wildcards, and -contains checks collection membership.

Profile script

I use a fairly small profile script that adds some behaviours I’m used to from Unix, and to make Tab completion show a menu. Here are the most basic bits:

 
    
    
    
    
  
  
  # Commands starting with space are not remembered.
      

Apart from that, I use a few aliases and a pretty prompt with the help of oh-my-posh.

The unusual and sometimes confusing parts

PowerShell can be verbose. Some of its syntax is a little quirky, compared to other languages, e.g. the equality and logic operators (for example, -eq, -le, -and). The aliases usually help with remembering commands, but they can’t always be depended on — ls is defined as an alias only on Windows, and Windows PowerShell aliases wget and curl to Invoke-WebRequest, even though all three have completely different command line arguments and outputs (this was removed in PowerShell).

Moreover, the Unix/DOS aliases do not change the argument handling. rm -rf foo is invalid. rm -r foo is, since argument names can be abbreviated as long as the abbreviation is unambiguous. rm -r -f foo is not valid, because -f can be an abbreviation of -Filter or -Force (so rm -r -fo foo) will do. rm foo bar does not work, an array is needed: rm foo,bar.

C:\Windows\regedit.exe launches the Registry editor. "C:\Program Files\Mozilla Firefox\firefox.exe" is a string. Launching something with spaces in its name requires the call operator: & "C:\Program Files\Mozilla Firefox\firefox.exe". PowerShell’s tab completion will add the & if necessary.

:/>  Максимум ядер процессора и памяти в конфигурации windows почему не нужно настраивать

There are two function call syntaxes. Calling a function/cmdlet uses the shell-style syntax with argument names: Some-Function -Arg1 value1 -Arg2 value2, and argument names can be abbreviated, and can sometimes be omitted. Calling a method requires a more traditional syntax: $obj.SomeMethod(value1, value2). Names are case-insensitive in either case.

The escape character is the backtick. The backslash is the path separator in Windows, so making it an escape character would make everything painful on Windows. At least it makes it easy to write regex.

The ugliest part

Conclusion

PowerShell is a fine interactive shell and scripting language. While it does have some warts, it is more powerful than your usual Unix shell, and its strongly-typed, object-oriented code beats stringly-typed sh spaghetti any day.

.NET HashSet in PowerShell

A HashSet<T> is another generic .NET collection class similar to a List<T> with a few notable differences.

  1. It is unordered (though there is also a SortedSet<T>)
  2. All items in the HashSet must be unique (no duplicates)
  3. It cannot be indexed into using $HashSet[<int>]

As we’re now familiar with most of the syntax, let’s jump straight into the examples:

Creating the HashSet<T> object:

 namespace SystemCollectionsGeneric

 = ::New

Adding, searching for, and removing items:

# Adding an item
 C:\> Add1
True

# Checking if the set contains an item
 C:\> Contains1
True

# Duplicates are not permitted
 C:\> Add1
False

# Removing an item
 C:\> Remove1
True

As HashSet<T> implements IEnumerable<T> we can use LINQ without any casting. Let’s look at slightly more complex example than before. We’re going to create two HashSets, the first will be a set of IP addresses we “trust”. The second will be a list of IP addresses perhaps pulled from some access logs. What we want to do is return a collection of unique IPs in the second set, but exclude the “trusted” IPs from the first set.

HashSet Venn diagram showing overlap between trusted IP set and Access Log IP set

 namespace SystemCollectionsGeneric

 = ::New  \trustedtxt
 = ::New  \accesslogtxt

Here are the contents of those two files:


102501100
102501101


192168150
102501120
102501100
215714595
102501100
102501100
102501100
125286035
125286035

We can see our accesslog file has several duplicates, but once we add it to our set the duplicate values are no longer present.

 C:\>    ExpandProperty IPAddressToString

192168150
102501120
102501100
215714595
125286035

Now let’s use LINQ to extract the logged IPs but exclude our trusted addresses.

 C:\>  = ::Except 
 C:\>    Expand IPAddressToString

192168150
102501120
215714595
125286035

That wraps it up. Hopefully you’ve not only discovered some useful .NET classes but also have a bit of insight into how to find and use other types that haven’t been covered here.

Using LINQ with other .NET collection types

LINQ works with any collection class that implements IEnumerable<T>, not just lists. So what is IEnumerable<T> and how do we find out what “implements” it?

.NET Interfaces

Interfaces are a contractual blueprint for classes that define what methods and properties implementing classes must provide. The classes that implement that interface then define how those members are implemented.

For example, we could have an IAnimal interface (the convention for interface naming is to begin with a capital I) which states that any classes implementing it must have a MakeNoise method that returns a string (e.g., “Woof”).

When we then create our various animal classes we make them implement the IAnimal interface, and in each animal class we will need to write the MakeNoise method (or we will get IDE / compilation errors) and have it return the noise of that specific animal.

Using interfaces means we can have confidence that regardless of the animal, there will be a MakeNoise method that returns a string. There are other advantages to interfaces such as decoupling which helps facilitate changes but none of that is important for our needs here.

Back to LINQ – it works with classes that implement the IEnumerable<T> interface. How do we find what those are?

There are two ways in PowerShell.

GetInterfaces

We can use the GetInterfaces method:

 C:\> GetInterfaces

IsPublic IsSerial Name                                     BaseType
                                       
True     False    IList`1
True     False    ICollection`1
True     False    IEnumerable`1True     False    IEnumerable
True     False    IList
True     False    ICollection
True     False    IReadOnlyList`1
True     False    IReadOnlyCollection`1

We can see List<T> list implements both IEnumerable and IEnumerable'1, so what’s the difference? The the backtick and digit indicates this is a generic type (List<T> instead of just List) and refers to the number of type parameters it accepts.

For example, List<T> only accepts one, but a Dictionary<T key,T value> accepts two (one for the key and one for the value), as seen below:

 C:\> GetInterfaces

IsPublic IsSerial Name                                     BaseType
                                       
True     False    IDictionary`2True     False    ICollection`1
True     False    IEnumerable`1
True     False    IEnumerable
True     False    IDictionary
True     False    ICollection
True     False    IReadOnlyDictionary`2
True     False    IReadOnlyCollection`1
True     False    ISerializable
True     False    IDeserializationCallback

PowerShell native arrays only implement IEnumerable, not IEnumerable<T>, so they don’t work with LINQ without casting (more on this below).

 C:\> GetInterfaces

IsPublic IsSerial Name                                     BaseType
                                       
True     False    ICloneable
True     False    IList
True     False    ICollection
True     False    IEnumerableTrue     False    IStructuralComparable
True     False    IStructuralEquatable

Let’s try it out:

 C:\>  = @123456789
 C:\> ::Sum

MethodException: Cannot find an overload   and the argument count: 

The second method to determine whether something implements an interface is to use IsAssignableFrom.

IsAssignableFrom

First we need to define our type, then the interface, and then call the IsAssignableFrom method.

 C:\>  = 
 C:\>  = 
 C:\> IsAssignableFrom

True

Let’s check PowerShell arrays.

 C:\>  = 
 C:\>  = 
 C:\> IsAssignableFrom

False

We can also check the class documentation.

Documentation

Here is a screenshot showing what interfaces List<T> implements.

List<T> interfaces” title=”” src=”https://xkln.net/static/e70d03a18219d8c611456685b5daaa2c/8c557/list-interfaces.png” srcset=”https://xkln.net/static/e70d03a18219d8c611456685b5daaa2c/4edbd/list-interfaces.png 175w,<br /> https://xkln.net/static/e70d03a18219d8c611456685b5daaa2c/13ae7/list-interfaces.png 350w,<br /> https://xkln.net/static/e70d03a18219d8c611456685b5daaa2c/8c557/list-interfaces.png 700w,<br /> https://xkln.net/static/e70d03a18219d8c611456685b5daaa2c/69476/list-interfaces.png 926w” sizes=”(max-width: 700px) 100vw, 700px” loading=”lazy” decoding=”async”></p><p>Let’s cover casting PowerShell arrays to an object that enables us to use LINQ.</p><h2>Using List<T> instead of PowerShell Arrays</h2><p>Before we go further let’s quickly cover <code>using</code> statements.</p><h3 id=‘Using’ statements

‘Using’ statements allow us to use types without having to reference the entire namespace each time we want to interact with the class. For example, if we wanted to create a List<T> (a .NET class we will cover below) we can do so in two ways.

The first is to specify the full namespace path:

 = ::New

The second is to place a using statement at the top of the script, and then reference the type only by its name.

 namespace SystemCollectionsGeneric

 = ::New

We already have a similar concept to this in PowerShell. We can run Get-Process by itself, or we can reference the module-qualified cmdlet with Microsoft.PowerShell.Management\Get-Process. If we had two commands that mapped to Get-Process we could remove ambiguity by writing out the full path.

Back to using List<T>.

List<T> in PowerShell

However, when we attempt to perform some task against each item in the collection (as is common) we’re likely to encounter errors.

 C:\> @1020     2
5
10
InvalidArgument: Cannot convert value  to   Error: "The input string 'thirty' was not in a correct format."

With generics we specify what type will be contained inside the collection. Let’s begin with creating a List<T> to hold integers.

 namespace SystemCollectionsGeneric

 = ::New
Add10
Add20

Now when we add “thirty” we will receive an error.

 C:\> Add

MethodException: Cannot convert argument  with value:    to  : "Cannot convert value "thirty" to type "SystemInt32". Error: "The input string  was not in a correct format

List Caveats

 = ::New
Add
Add
Add30

Here the integer 30 was automatically cast to the string "30".

 C:\> 
ten
twenty
30

 C:\> 2GetType

IsPublic IsSerial Name                                     BaseType
                                       
True     True     String                                   SystemObject

This appears to be a PowerShell specific behaviour as the equivalent C# code will throw an exception.

 

 Numbers  
Numbers
Numbers
Numbers
Error   CS1503 Argument  cannot convert from '' to ''

Another caveat is that List<T> accepts null values regardless of the type specified.

 C:\>  = ::New
 C:\> Add
 C:\> 
0

 C:\>  = ::New
 C:\> Add
 C:\> Count
1

Setting aside tangents and caveats, List<T> vastly outperforms PowerShell arrays, offering significant performance improvements.

Using LINQ in PowerShell

LINQ (Language Integrated Query) is a .NET component that allows for data querying and filtering. Think of a combination PowerShell’s Where, Measure-Object, and Select cmdlets, but usually a lot faster than native PowerShell cmdlets, especially when dealing with large collections.

Let’s look at a couple of simple examples.

List comparison with LINQ in PowerShell

We’re going to create a few lists and compare their contents with LINQ, including the order of the items in the lists.

 =   
 =   
 =    # Same items, different order

::SequenceEqual 


::SequenceEqual 
# False due to ordering being different

There is a Sort method we can use if we have lists where the items may have different ordering:





::SequenceEqual 


::SequenceEqual 

Getting minimum, maximum, average values from a collection

 = ::new
1100   Add Minimum 1 Maximum 1000

 C:\> ::Min
32

 C:\> ::Max
983

 C:\> ::Average
51395

 C:\> ::Sum
51395

As a very quick performance comparison let’s see how LINQ compares with Measure-Object for calculating the sum of one million numbers.

 = ::new
11000000   Add Minimum 1 Maximum 100


 C:\> 110       Sum    ExpandProperty TotalMilliseconds   Average

Average           : 85756657


 C:\> 110    ::Sum    ExpandProperty TotalMilliseconds   Average

Average           : 045215

In this example LINQ is ~1897 times faster than piping to Measure-Object

There are too many LINQ methods to cover here, and I’ve only scratched the surface to show the most basic syntax. Michael Sorens has written a fantastic article on using LINQ with PowerShell titled High Performance PowerShell with LINQ that I recommend you check out.

Determining what version of .NET CLR PowerShell is using

Different .NET/Core versions support different classes and behaviors. Knowing the .NET CLR (Common Language Runtime) version PowerShell is using helps you refer to the correct documentation and troubleshoot unexpected behaviors.

PowerShell 5.1 uses .NET Framework 4.0.30319.42000.

 C:\> PSVersionToString
51190413996

 C:\> ::VersionToString
403031942000

PowerShell 7.4.1 uses .NET Core 8.0.1

 C:\> PSVersionToString
741

 C:\> ::VersionToString
801

We can already see a big difference here, v5.1 uses the older .NET Framework while 7.x uses the more modern .NET Core. This is why PowerShell 7.x can be cross platform, and why it can’t ship with Windows as Microsoft does not want to bundle the .NET Core runtime due to support lifecycles of the products being different (at least that is my understanding).

There will be an example below where the .NET CLR version is important.

Let’s begin with something simple, validating whether a string is null, empty, or whitespace.

IP Address validation using [IPAddress]

 C:\> 

Address            : 16843018
AddressFamily      : InterNetwork
ScopeId            :
IsIPv6Multicast    : False
IsIPv6LinkLocal    : False
IsIPv6SiteLocal    : False
IsIPv6Teredo       : False
IsIPv4MappedToIPv6 : False
IPAddressToString  : 10111

However, it has limitations; for example, 10.1 is considered a valid value:

 C:\> 

Address            : 16777226
AddressFamily      : InterNetwork
ScopeId            :
IsIPv6Multicast    : False
IsIPv6LinkLocal    : False
IsIPv6SiteLocal    : False
IsIPv6Teredo       : False
IsIPv4MappedToIPv6 : False
IPAddressToString  : 10001

However, we can get around this by comparing the input against the IPAddressToString output:

  IPAddressToString

Here are some examples:

 C:\> @           IPAddressToString

True
False
False
False
InvalidArgument: Cannot convert value  to   Error: "An invalid IP address was specified."
InvalidArgument: Cannot convert value  to   Error: "An invalid IP address was specified."
InvalidArgument: Cannot convert value  to   Error: "An invalid IP address was specified."

The second value of 10.1 while accepted as valid input for the class failed the equality check, and the same goes for the third hex value and the fourth decimal value. Both are ‘valid’ input for the class as they’re just different ways a valid IP can be represented, but if we’re expecting the usual dotted-decimal IP representation in our input they fail the string equality check.

The last three are invalid IP addresses and return errors.

While we’re on the topic of networking, let’s look at MAC Addresses.

Why use .NET classes in PowerShell?

There are two main reasons:.

  1. Extended Functionality: Access features in .NET not available natively in PowerShell.
  2. Enhanced Performance: Many .NET classes perform faster than their PowerShell equivalents.

Exploring .NET also helps deepen your understanding of the foundation upon which PowerShell is built.

Casting PowerShell arrays to an IEnumerable object

We may already have an existing array that we would like to use with LINQ, and in these cases LINQ provides a casting method that returns a usable objects which implements IEnumerable<T>.

 = @1 2 3 4 5
 = ::Cast

# We can now use LINQ
 C:\> ::Average
3

 C:\> GetType

IsPublic IsSerial Name                                     BaseType
                                       
False    False    <CastIterator>d__68`1                    SystemObject

Let’s move onto the final .NET class for this post: HashSets

.NET Syntax in PowerShell

We have explored several .NET classes, which might raise questions about their syntax in PowerShell.

Understanding [ClassName]::MethodName() Syntax

Here it is shown in the Microsoft documentation for the PhysicalAddress class.

Microsoft documentation showing Parse as a static method

When we use the usual . notation to invoke a method (as we did with $List.Add()) this the same as invoking a method on a PowerShell object, for example $SomeString.ToUpper().

Default imports

 C:\> ::CurrentDomainGetAssemblies  ? Location  

GAC    Version        Location
            
False  v4030319     C:\Program Files\PowerShell\7\SystemCollectionsdll
False  v4030319     C:\Program Files\PowerShell\7\SystemCollectionsConcurrentdll
False  v4030319     C:\Program Files\PowerShell\7\SystemCollectionsSpecializeddll
False  v4030319     C:\Program Files\PowerShell\7\SystemCollectionsNonGenericdll
False  v4030319     C:\Program Files\PowerShell\7\SystemCollectionsImmutabledll

 C:\> ::New
InvalidOperation: Unable to find  

In short, some namespaces will be imported by default, others you’ll need to specify even if the assembly is loaded.

Microsoft’s documentation is an excellent resource for mapping classes to namespaces and assemblies.

List class documentation

And here is a list of all classes in the System.Collections.Generic namespace. I will briefly cover HashSets, but I’d recommend exploring others as a learning exercise.

Loading custom or third party assemblies

Sometimes the classes we want to work with are in third party assemblies. We can load these using Add-Type. As this is slightly out of scope for this post and I’ve shown examples of it before, you can check it out here.

Fast TCP port testing with [System.Net.Sockets.TcpClient]

This is one of my favourites simply because it’s so much faster than Test-NetConnection thanks to the configurable timeout.

Here is an example:

 C:\>  = 
 C:\>  = 443
 C:\>  = 100
 C:\>  = ::New
 C:\> ConnectAsyncWait

True

 C:\> Dispose

It returns $true if the connection was successfully established, and $false if not.

String validation

.NET has very two very convenient string validation methods built into the String class, IsNullOrEmpty, and IsNullOrWhiteSpace. While not difficult, performing this kind of validation in PowerShell 5.1 requires a bit more work than just calling a method.

Here are a few examples:

 = 
 = 
 = 
 = 

 
::IsNullOrEmpty
::IsNullOrEmpty
::IsNullOrEmpty
::IsNullOrEmpty

 
::IsNullOrWhiteSpace
::IsNullOrWhiteSpace
::IsNullOrWhiteSpace
::IsNullOrWhiteSpace
IsNullOrEmpty
True
True
False
False

IsNullOrWhiteSpace
True
True
True
False

String validation natively in PowerShell

PowerShell 5.1 has ValidateNotNullOrEmpty built in, while PowerShell 7 contains both validators in the form of attributes. Here is an example:

 =        # Works in PowerShell 5.1+
 =   # Works in PowerShell 7+

Using the .NET method however works across all versions.

Moving on, let’s tackle IP address validation

Generating passwords with [System.Web.Security]

I’m sure at some stage we’ve all needed to programmatically generate passwords. The System.Web.Security namespace has a nice method for doing this. The downside is that it did not make it across to .NET Core, so it only works on PowerShell 5.1.

The syntax is GeneratePassword (int length, int minimumNumberOfNonAlphanumericCharacters)

Here are a couple of examples:

 C:\> ::GeneratePassword201
7!=vvIOh45ib3p+dU&AGSvHy42+

 C:\> ::GeneratePassword2010
g2Ht@AM-pv&iz+dNU

Let’s move onto some .NET classes that have a wider use cases.

MAC Address Validation and Normalisation using [PhysicalAddress]

As with IP addresses we can use regex and some string manipulation here, but .NET again has a convenient method to make our code a little cleaner and easier to read.

Valid MAC address example:

 C:\> ::Parse

34ED1BAABBCC

Invalid MAC address example:

 C:\> ::Parse

Exception calling  with  arguments: "An invalid physical address was specified."
At line:1 char:1
 ::Parse
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     CategoryInfo          : NotSpecified: :  MethodInvocationException
     FullyQualifiedErrorId : FormatException

Compatibility Across .NET Versions

If you recall earlier the underlying version of .NET CLR can make a difference, this the is one class where Microsoft has made improvements between .NET Framework (used by PowerShell 5.1) and .NET Core.

Unfortunately the above format (all uppercase, - delimited) is the only format valid under .NET Framework. All of these valid and commonly formatted MAC address values fail:

 C:\> ::Parse
Exception calling  with  arguments: "An invalid physical address was specified."

 C:\> ::Parse
Exception calling  with  arguments: "An invalid physical address was specified."

 C:\> ::Parse
Exception calling  with  arguments: "An invalid physical address was specified."

However, they all work under PowerShell 7.x which is built on top of .NET Core.

 C:\> ::Parse
34ED1BAABBCC

 C:\> ::Parse
34ED1BAABBCC

 C:\> ::Parse
34ED1BAABBCC

These changes are documented in the Microsoft documentation.

parse remarks

Sticking with the networking theme, let’s test TCP ports

Оставьте комментарий