Windows OS Hub / PowerShell / Invoke-WebRequest: Perform HTTP Requests, Download Files, Parse Web with PowerShell
The Invoke-WebRequest cmdlet can be used to request HTTP/HTTPS/FTP resources directly from the PowerShell console. You can use this command to send HTTP requests (GET and POST), download files from a website, parse HTML web pages, perform authentication, fill out and submit web forms, etc. In this article, we’ll cover basic examples of using the Invoke-WebRequest cmdlet in PowerShell to interact with web services.
Imagine, you have to verify the response of an API in an integration test. The response of the API is master data in JSON format, so it is static, but it is quite huge and the only “source of truth” you can verify the data against is a table on a website of your favorite wiki software (e.g. wikipedia.com or a Confluence page). The idea is, to parse the table with Powershell and compare it to the JSON response of the API call.
Long story short, the API is described as a table on a Confluence page and your task is, to “understand” the meaning of the table’s cells and parse it, so Powershell can handle this huge amount of data.
Sounds exciting, doesn’t it?
I’m using this script below to extract the message of the body from an Event Log.
$eventId = 3000
$logName = 'Microsoft-Windows-SMBServer/Audit'
$logs = Get-WinEvent -FilterXPath "*[System/EventID=$eventId]" -LogName $logName -ErrorAction SilentlyContinue | Select-Object TimeCreated, Id, Message
TimeCreated Id Message
----------- --- --------
12/31/2023 3:41:09 AM 3000 SMB1 access...
I would like to only extract certain part of the message such as Client Address: 10.10.7.7.
SMB1 access
Client Address: 10.10.7.7
Guidance:
This event indicates that a client attempted to access the server using SMB1. To stop auditing SMB1 access, use the Windows PowerShell cmdlet Set-SmbServerConfiguration.
My desired output:
TimeCreated, Id, Message
12/31/2023 3:41:09 AM ,3000,10.10.7.7
12/31/2023 3:41:09 AM ,3000,10.10.7.15
You can use a regex -replace to get only the IP address from the message like:
$eventId = 3000
$logName = 'Microsoft-Windows-SMBServer/Audit'
$logs = Get-WinEvent -FilterXPath "*[System/EventID=$eventId]" -LogName $logName -ErrorAction SilentlyContinue |
Select-Object TimeCreated, Id,
@{Name = 'IPAddress'; Expression = {$_.Message -replace '(?sm)^.*Client Address:\s*(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}).*', '$1'}}
8 gold badges26 silver badges43 bronze badges
With the rise of configuration files in various applications, understanding the ins and outs of INI files can elevate your productivity as a developer or software enthusiast. Despite their roots in the 16-bit Microsoft Windows platforms, INI files remain a crucial part of application configurations.
But, when it comes to handling INI files with PowerShell things can be a bit tricky. PowerShell, even though it’s a great tool, doesn’t have ready-made commands to work with these files. But don’t worry, we’re here to help!
In this article, we’ll show you how you can use PowerShell to read INI files and turn their content into a format that’s easy to use.
How to read INI files with PowerShell
You can read INI files with PowerShell using the Get-Content cmdlet:
However, there is a difference between reading the file content and transforming the file content into a usable format under a hash table. Let’s go into that.
Converting INI files into PowerShell hash tables
Microsoft offers a PowerShell cmdlet, ConvertFrom-StringData, which converts a string with key-value pairs into a hash table.
The sections return an error!
The cmdlet doesn’t capture the INI file structure accurately, as it expects a name=value format.
It ignores any characters where the # character is the first non-whitespace character, making it unsuitable for files with sections and comments or when there are duplicate element names in different sections.
For simpler INI files with no sections or comments, this will work perfectly:
But if your INI file is more complex, we’ve got you covered.
Inspired by a script developed by Doctor Scripto in the Microsoft devblogs, we present an improved method for reading all types of INI files. We leverage regular expressions (regex) to determine whether a line is a comment, section, or key.
Here’s the magical function:
function Get-IniContent ($filePath)
{
$ini = @{}
switch -regex -file $FilePath
{
“^\[(.+)\]” # Section
{
$section = $matches[1]
$ini[$section] = @{}
$CommentCount = 0
}
“^(;.*)$” # Comment
{
$value = $matches[1]
$CommentCount = $CommentCount + 1
$name = “Comment” + $CommentCount
$ini[$section][$name] = $value
}
“(.+?)\s*=(.*)” # Key
{
$name,$value = $matches[1..2]
$ini[$section][$name] = $value
}
}
return $ini
}To call this function, store the output in a variable and then call the elements in the hash table as shown:
function Get-IniContent ($filePath)
{
$ini = @{}
switch -regex -file $FilePath
{
“^\[(.+)\]” # Section
{
$section = $matches[1]
$ini[$section] = @{}
$CommentCount = 0
}
“^(;.*)$” # Comment
{
$value = $matches[1]
$CommentCount = $CommentCount + 1
$name = “Comment” + $CommentCount
$ini[$section][$name] = $value
}
“(.+?)\s*=(.*)” # Key
{
$name,$value = $matches[1..2]
$ini[$section][$name] = $value
}
}
return $ini
}
$test = Get-IniContent "C:\Users\User\Desktop\test.ini"
write-host $test.owner.name
write-host $test.database.port
How to Read INI files with MSI
Microsoft’s built-in INI table for the MSI databases provides an alternative way to manage INI files.
And, with tools like Advanced Installer, reading and modifying INI files becomes even easier.
Here’s how you can use Advanced Installer to import the test INI file that we used to try our PowerShell scripts:
- Open Advanced Installer
- Create a project
- Navigate to the Files and Folders page.
- From the Add section in the top menu, select Import Files > Import INI (Advanced) and select the INI file.
As you can see, Advanced Installer reads the INI file and adds it to the INIFile table. From here, you can easily manipulate the INIFile table by using:
- dehardcoding,
- Properties,
- Custom Actions.
Excited about your new INI file skills? Try Advanced Installer’s 30-day free trial now and put them into play!
Conclusion
Even though INI files have been with us for a long time, they present certain hurdles when it comes to reading and modifying them with PowerShell. But as we’ve demonstrated, there are handy workarounds that make the process less stressful and more efficient. We hope this guide offers a valuable resource in your journey of mastering INI files with PowerShell.
Ensure to bookmark this page for future reference, and do not hesitate to leave a comment if you have any queries or suggestions.
Alex Marin
Application Packaging and SCCM Deployments specialist, solutions finder, Technical Writer at Advanced Installer.
Popular Articles
Setup
For the sake of demonstrating, how to parse an HTML table with Powershell, I created a GitHub repo: ms_powershellTableCoverter.
To have a working example, you’ll find a Dockerfile in the docker directory, which starts up a nginx server with a static index.html file, containing the table containing the fake specification for the API. Here are the commands to boot up the web server:
> cd docker
> docker build -t nginxwebsrvstatictable:0.0.1 .
> docker run -p 8080:80 nginxwebsrvstatictable:0.0.1If those two requests are successful, we can go on parsing the table with Powershell.
Get Web Page Content with Invoke-WebRequest Cmdlet
The Invoke-WebRequest PowerShell cmdlet allows you to send the HTTP, HTTPS, or FTP request with the GET method to the specified web page and receive a response from the server.
There are two aliases for the Invoke-WebRequest command in Windows: iwk and wget.
Invoke-WebRequest -Uri "https://woshub.com"
Tip. If you are connected to the Internet via a proxy server, you must properly configure PowerShell to access the Web through the proxy server. If you do not set proxy parameters, you will receive an error when you run the IWK command:
Invoke-WebRequest: Unable to connect to the remote server CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
The command loaded the page and displayed its contents in the PowerShell console. The returned response is not just the HTML code of the page. The Invoke-WebRequest cmdlet returns an object of type HtmlWebResponseObject. Such an object is a collection of forms, links, images, and other important elements of an HTML document. Let’s look at all the properties of this object:
To get the raw HTML code of the web page that is contained in the HtmlWebResponseObject object, run:
You can list the HTML code along with the HTTP headers returned by the web server:
You can check only the web server HTTP status code and the HTTP headers of the HTML page:
As you can see, the server has returned a response 200. This means that the request has been successful, and the web server is available and works correctly.
Key Value --- ----- Transfer-Encoding chunked Connection keep-alive Vary Accept-Encoding,Cookie Cache-Control max-age=3, must-revalidate Content-Type text/html; charset=UTF-8 Date Wed, 13 Jul 2022 02:28:32 GMT Server nginx/1.20.2 X-Powered-By PHP/5.6.40
To get the last modification time of a web page:
The list of available agents in PowerShell can be displayed like this:
Ignore SSL/TLS Certificate Checks
Another issue is that the Invoke-WebRequest cmdlet is closely related to Internet Explorer. For example, in Windows Server Core editions in which IE is not installed (or removed), the Invoke-WebRequest cmdlet cannot be used.
Invoke-WebRequest: The response content cannot be parsed because the Internet Explorer engine is not available, or Internet Explorer’s first-launch configuration is not complete. Specify the UseBasicParsing parameter and try again.
In this case, the WebClient class can be used instead of Invoke-WebRequest. For example, to download a file from a specified URL, use the command.
(New-Object -TypeName 'System.Net.WebClient').DownloadFile($Url, $FileName)
If an invalid SSL certificate is used on an HTTPS site, or PowerShell doesn’t support this type of SSL/TLS protocol, then the Invoke-WebRequest cmdlet drops the connection.
Invoke-WebRequest : The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
Invoke-WebRequest : The request was aborted: Could not create SSL/TLS secure channel.
By default, Windows PowerShell (in early builds of Windows 10, Windows Server 2016, and older versions of Windows) uses the legacy and insecure TLS 1.0 protocol for connections (check the blog post describing the PowerShell module installation error: Install-Module: Unable to download from URI).
In PowerShell Core, the Invoke-WebRequest cmdlet has an additional parameter –SkipCertificateCheck that allows you to ignore invalid SSL/TLS certificates.
Another significant drawback of the Invoke-WebRequest cmdlet is its rather low performance. When downloading a file, the HTTP stream is entirely buffered into memory, and only after the full download is completed is saved to disk. Thus, when downloading large files using Invoke-WebReques, you may run out of RAM.
How to Fill and Submit HTML Form with PowerShell?
Using the next command, display the list of the fields to fill in the login HTML form (login_form):
Assign the desired values to all fields:
To submit (sent) the filled form to the server, call the action attribute of the HTML form:
You can also use the JSON format to send data to a web page with the POST method:
Prerequisites
> Install-Module -Name PowerHTMLAfter a few moments, the package should be available on your system. You can test it if you type “ConvertFrom-HTML” and hit enter. If no errors are displayed, the installation procedure was successful.
How to Download File over HTTP/FTP with PowerShell Wget (Invoke-WebRequest)?
wget "https://download-installer.cdn.mozilla.net/pub/firefox/releases/102.0.1/win64/en-US/Firefox%20Setup%20102.0.1.exe" -outfile “c:\tools\firefox_setup.exe”
This command will download the file from the HTTP site and save it to the specified directory.
You can get the size of a file in MB before downloading it with wget:
$url = "https://download-installer.cdn.mozilla.net/pub/firefox/releases/102.0.1/win64/en-US/Firefox%20Setup%20102.0.1.exe"
(Invoke-WebRequest $url -Method Head).Headers.'Content-Length'/1Mb
Below is an example of a PowerShell script that will find all links to *.pdf files on a target web page and bulk download all found files from the website to your computer (each pdf file is saved under a random name):
As a result of the script in the target directory, all PDF files from the page will be downloaded. Each file is saved under a random name.
In modern PowerShell Core (6.1 and newer), the Invoke-WebRequest cmdlet supports resume mode. Update your version of PowerShell Core and you can use the -Resume option on the Invoke-WebRequest command to automatically resume downloading a file in case a communication channel or server is unavailable:
Invoke-WebRequest -Uri $Uri -OutFile $OutFile –Resume
Using Invoke-WebRequest with Authentication
Some web resources require authentication to access. You can use various types of authentication with the Invoke-WebRequest cmdlet (Basic, NTLM, Kerberos, OAuth, or certificate authentication).
$cred = Get-Credential
wget -Uri 'https://somesite.com' -Credential $cred
Invoke-WebRequest 'https://somesite.com' -UseDefaultCredentials
DefaultCredentials not working with Basic authentication.
To authenticate with a certificate, you need to specify its thumbprint:
Invoke-WebRequest 'https://somesite.com' -CertificateThumbprint xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
You can use modern Bearer/OAuth token authentication in your PowerShell scripts.
- First, you need to get an OAuth token from your REST API provider (out of the scope of this post);
- Convert token with ConvertTo-SecureString cmdlet:
$Token = "12345678912345678901234567890" | ConvertTo-SecureString -AsPlainText –Force - Now you can perform OAuth authentication:
$Params = @{
Uri = "https://somesite.com"
Authentication = "Bearer"
Token = $Token }
Invoke-RestMethod @Params
Parse the HTML document
First things first, we need to parse the entire HTML web page with this command:
We assume, that there is only one table on the page respectively the data is stored in the first table of the page. Hence, we can get the table headers with this command
Next, we initialize the array, where the table’s data will be stored.
Afterward, when everything is set and initialized correctly, we can start grabbing the data from the table:
The main part of the script counts the table’s rows, excluding the header row. Then it starts a loop, beginning at row two, right after the table’s header. The script fetches the cells of a particular row into an array and creates a new empty hash for it. Now with the cells in place, it loops over the table’s header and sets a key for every header column with the associated value of the table row. Last, it “returns” the hash back from the loop to the pwshTable variable.
Now, when calling pwshTable on the command prompt, you may see something like this:
Quite handy and efficient!
Hence, the call to Invoke-RestMethod returns an object, which contains an array “catbreeds”, which contains information about cat breeds. The rest of the comparison is quite easy:
For each entry in the array pwshTable, get the object, where the breed matches the property “Breed”. Write to the console, if the breed was found or not.
Thats it! Now, with the output of the console, you can easily observe, if the static response of the API and the content of the table match!
Please comment on your thoughts on the article. I’m curious what you guys think about it and if it could be useful.
Parse and Scrape HTML a Web Page with PowerShell
The Invoke-WebRequest cmdlet allows you to quickly and conveniently parse the content of any web page. When processing an HTML page, collections of links, web forms, images, scripts, etc., are created.
Let’s look at how to access specific objects on a web page. For example, I want to get a list of all outgoing links (A HREF objects) on the target HTML web page:
You can only select links with a specific CSS class:
Or specific text in the URL address:
Then display a list of all images on this page:
Create a collection of full URL paths to these images:
Initialize a new instance of WebClient class:
$wc = New-Object System.Net.WebClient
Download all the image files from the page (with their original filenames) to the c:\too1s\ folder:
