Проверка доступности uri адреса средствами powershell на подобие curl

 Windows OS Hub / PowerShell / Invoke-WebRequest: Perform HTTP Requests, Download Files, Parse Web with PowerShell

The Invoke-WebRequest cmdlet can be used to request HTTP/HTTPS/FTP resources directly from the PowerShell console. You can use this command to send HTTP requests (GET and POST), download files from a website, parse HTML web pages, perform authentication, fill out and submit web forms, etc. In this article, we’ll cover basic examples of using the Invoke-WebRequest cmdlet in PowerShell to interact with web services.

А решение, как оказалось, достаточно простое:

powershell -nologo -noprofile "%{[Net.ServicePointManager]::SecurityProtocol = 'Tls12, Tls11, Tls, Ssl3'} ;(Invoke-WebRequest -UseBasicParsing -Uri <ULR>).StatusCode;exit [int]$Error[0].Exception.Status"

Результат считывается из значение свойства “Status” члена “Exception” в переменной $Error. Это помогает обработать ситуации типа такой

Invoke-WebRequest : Невозможно разрешить удаленное имя: ‘cle.linux.org.tw’
строка:1 знак:77
+ %{[Net.ServicePointManager]::SecurityProtocol = ‘Tls12, Tls11, Tls, Ssl3’} ;Invo …
+ ~
~~~
+ CategoryInfo: InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

Status Property System.Net.WebExceptionStatus Status {get;}

, которое и даёт ответ, было соединение или нет.

Ну, вот, как‐то так. Почти полный аналог “curl”.

Introduced in 2022.2

Copy the script and replace the placeholders with your actual API URL and the credentials of an technician account.

To get the proper output, make sure that the ticket exists in your database and the file is available; otherwise, replace the sample values with your actual IDs.

$ApiUrl = 'https://API_URL'	# API URL without a trailing slash, i.e. https://example.com/api
$username = 'your_username'	# technician username
$password = 'your_password'	# technician password
$postParams = @{ 'grant_type' = 'password'; 'username' = $username; 'password' = $password; }
$htmlResponse = Invoke-WebRequest -UseBasicParsing -Uri "$ApiUrl/token" -Method POST -Body $postParams
$token = (ConvertFrom-Json $htmlResponse.Content).access_token
$htmlResponse = Invoke-WebRequest -Uri "$ApiUrl/Object/T000030/Attachments/7B5B7027-E73C-4003-8AD6-4A0478D3AAED/download" `	-Method GET `	-Headers @{ 'authorization' = "bearer $token"; 'content-type' = 'application/json'; }
$outFle = New-Item -Name $htmlResponse.Headers['Attachment-File-Name'] -ItemType File -Force
$fileStream = $outFle.OpenWrite();
$htmlResponse.RawContentStream.WriteTo($fileStream);
$fileStream.Dispose();

• To support extracting content that gets loaded dynamically, via JavaScript, you need a full web browser that you can control programmatically.

• As you’ve discovered yourself, Chromium-based browsers do offer a CLI method of outputting the dynamically generated / augmented HTML, as it would render in interactively in a browser, using the --headless and --dump-dom options.

• You can capture this HTML in a variable and then process it via an HTML parser such as provided by the AngleSharp .NET library, as offered via the PSParseHTML module, which also wraps the HTML Agility Pack (which is used by default).

• It assumes that you’re running Windows 11 with the modern, Chromium-based version of Microsoft Edge, located at:

  "C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe"

  • Alternatively, you can download a different Chromium-based browser, such as Brave, or Google Chrome, whose executable you can then find at:

    "C:\Program Files (x86)\Google\Chrome\Application\chrome.exe"

• It downloads HTML from sample URL http://www.nptcstudents.co.uk/andrewg/jsweb/dynamicpages.html, which dynamically fills in various elements using client-side JavaScript, including one with the current timestamp.

• It then ensures that the PSParseHTML module is installed and uses it to parse the rendered HTML, and extracts the element that was dynamically populated with the current timestamp to verify that client-side rendering was indeed performed.

# Create an 'msedge' alias for Microsoft Edge.
Set-Alias msedge 'C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe'
# Sample URL that includes dynamic content.
$url = 'http://www.nptcstudents.co.uk/andrewg/jsweb/dynamicpages.html'
# Use Microsoft Edge in headless mode to download from the URL
# and run its client-side scripts.
# Note:
# * --disable-gpu prevents any GPU-related errors from appearing in the output.
# * ... | Out-String captures all output as a *single, multiline string*
# and additionally ensures *synchronous* execution on Windows,
# which in turn enables capturing the output.
# * Since a full web browser must be launched, as a child process,
# followed by downloading and rendering a web page, this takes
# a while, especially if the browser isn't already running.
Write-Verbose -Verbose "Downloading and rendering $url..."
$dynamicHtml = msedge --headless --dump-dom --disable-gpu $url | Out-String
# Now you can use the PSParseHTML module to parse the captured HTML.
# Install the module on demand.
if (-not (Get-Module -ErrorAction Ignore -ListAvailable PSParseHTML)) { Write-Verbose "Installing PSParseHTML module for the current user..." Install-Module -ErrorAction Stop -Scope CurrentUser PSParseHTML
}
# Parse the HTML.
Write-Verbose -Verbose "Parsing the rendered HTML..."
$parsedHtml = ConvertFrom-Html -Engine AngleSharp -Content $dynamicHtml
# Now extract the dynamically populated element to verify that it contains the current timestamp.
Write-Verbose -Verbose "Extracting a dynamically populated element..."
$parsedHtml.QuerySelectorAll('div.exampleblock')[1].InnerHtml

The above should print something like (note the timestamp):

<script type="text/javascript"> document.write("The date is " + Date());
</script>The date is Tue Jan 16 2024 23:48:55 GMT-0500 (Eastern Standard Time) 

In this quick guide, we’ll walk through the utilities necessary to make an HTTP request to Twilio’s API, which is secured with HTTP basic authentication. We go over Invoke-WebRequest and finish by sending an outgoing SMS message.

PowerShell allows developers to write command line scripts using the full power of the .NET framework. However, some tasks you may be used to performing in *nix environments, like making an authenticated HTTP request with curl, are not as straightforward.

# Pull in Twilio account info, previously set as environment variables

$sid = $env:TWILIO_ACCOUNT_SID

$token = $env:TWILIO_AUTH_TOKEN

$number = $env:TWILIO_NUMBER

# Twilio API endpoint and POST params

$url = "https://api.twilio.com/2010-04-01/Accounts/$sid/Messages.json"

# Create a credential object for HTTP basic auth

$credential = New-Object System.Management.Automation.PSCredential($sid, $p)

# Make API request, selecting JSON properties from the response

Let’s break down the moving parts of this script.

Здравствуйте.
Я пользуюсь сервисом для сокращения ссылок exe.io.
В личном кабинете где находятся все ссылки есть возможность с помощью кнопки Hide скрыть те ссылки, которые не нужны. Пытаюсь с помощью команды PowerShell автоматизировать эту задачу, чтобы не приходилось вручную нажимать кнопку Hide.
Пользуюсь вот этим кодом (я получил его с помощью инструмента Монитор Сети Ctrl+Shift+E)

$session = New-Object Microsoft.PowerShell.Commands.WebRequestSession
$session.Cookies.Add((New-Object System.Net.Cookie("AppSession", "7d17c52a80d9c25d288ab1585", "/", "exe.io")))
$session.Cookies.Add((New-Object System.Net.Cookie("csrfToken", "c7d46cce802f956bde8762f34b65daa4fbcf6d6abdfe555b880fb10f9c114c8e4c23c2bd6934cb3", "/", "exe.io")))
Invoke-WebRequest -UseBasicParsing -Uri "https://exe.io/member/links/hide/CdrhjyEjVRs" `
-Method POST `
-WebSession $session `
-Headers @{ "Accept-Encoding" = "gzip, deflate, br" "Upgrade-Insecure-Requests" = "1" "Sec-Fetch-Dest" = "document" "Sec-Fetch-Mode" = "navigate" "Sec-Fetch-Site" = "same-origin" "Sec-Fetch-User" = "?1" "TE" = "trailers"
} `
-Body "_method=POST&_csrfToken=c7d46cce802f956bde8762f34b65daa4fbcf6d6abdfe555b880fb10f9c114c8e4c23c2bd6934cb3&_Token%5Bfields%5D=28181cf9aceff27%3A&_Token%5Bunlocked%5D=adcopy_challenge%257Cadcopy_response%257Ccaptcha_code%257Ccaptcha_namespace%257Cg-recaptcha-response"

Этот код работает.
Чтобы скрыть тысячи ссылок я копирую этот код и вставляю его ниже уже для другой ссылки. Получается примерно так (здесь пример для двух ссылок):

$session = New-Object Microsoft.PowerShell.Commands.WebRequestSession
$session.Cookies.Add((New-Object System.Net.Cookie("AppSession", "7d17c52a80d9c25d288ab1585", "/", "exe.io")))
$session.Cookies.Add((New-Object System.Net.Cookie("csrfToken", "c7d46cce802f956bde8762f34b65daa4fbcf6d6abdfe555b880fb10f9c114c8e4c23c2bd6934cb3", "/", "exe.io")))
Invoke-WebRequest -UseBasicParsing -Uri "https://exe.io/member/links/hide/CdrhjyEjVRs" `
-Method POST `
-WebSession $session `
-Headers @{ "Accept-Encoding" = "gzip, deflate, br" "Upgrade-Insecure-Requests" = "1" "Sec-Fetch-Dest" = "document" "Sec-Fetch-Mode" = "navigate" "Sec-Fetch-Site" = "same-origin" "Sec-Fetch-User" = "?1" "TE" = "trailers"
} `
-Body "_method=POST&_csrfToken=c7d46cce802f956bde8762f34b65daa4fbcf6d6abdfe555b880fb10f9c114c8e4c23c2bd6934cb3&_Token%5Bfields%5D=28181cf9aceff27%3A&_Token%5Bunlocked%5D=adcopy_challenge%257Cadcopy_response%257Ccaptcha_code%257Ccaptcha_namespace%257Cg-recaptcha-response"
$session = New-Object Microsoft.PowerShell.Commands.WebRequestSession
$session.Cookies.Add((New-Object System.Net.Cookie("AppSession", "7d17c52a80d9c25d288ab1585", "/", "exe.io")))
$session.Cookies.Add((New-Object System.Net.Cookie("csrfToken", "c7d46cce802f956bde8762f34b65daa4fbcf6d6abdfe555b880fb10f9c114c8e4c23c2bd6934cb3", "/", "exe.io")))
Invoke-WebRequest -UseBasicParsing -Uri "https://exe.io/member/links/hide/nckeDrhwF" `
-Method POST `
-WebSession $session `
-Headers @{ "Accept-Encoding" = "gzip, deflate, br" "Upgrade-Insecure-Requests" = "1" "Sec-Fetch-Dest" = "document" "Sec-Fetch-Mode" = "navigate" "Sec-Fetch-Site" = "same-origin" "Sec-Fetch-User" = "?1" "TE" = "trailers"
} `
-Body "_method=POST&_csrfToken=c7d46cce802f956bde8762f34b65daa4fbcf6d6abdfe555b880fb10f9c114c8e4c23c2bd6934cb3&_Token%5Bfields%5D=bd7fafd79bf9531%3A&_Token%5Bunlocked%5D=adcopy_challenge%257Cadcopy_response%257Ccaptcha_code%257Ccaptcha_namespace%257Cg-recaptcha-response"

Этот способ работает.
Я меняю только саму ссылку в строке Invoke-WebRequest и Token 28181cf9aceff27%3A в строке -Body (эти данные относятся только к определенной ссылке, а остальная часть кода универсальна.)

Но как сделать, чтобы универсальная часть кода не повторялась тысячи раз?

P.S. Код PowerShell я размещаю в файле myscript.ps1 и запускаю этот файл с помощью команды в *.bat файле:

powershell -executionpolicy RemoteSigned -file myscript.ps1

• 
  Вопрос задан
  
  28 окт. 2023
  

Вопрос решен!
Спасибо за помощь!

$appsession = Get-Content -Path .\appsession.txt
1..(Get-Content -Path .\url.txt | measure).Count | ForEach-Object {
$url = (Get-Content -Path .\url.txt)[$_-1];
$token = (Get-Content -Path .\token.txt)[$_-1]
$session = New-Object Microsoft.PowerShell.Commands.WebRequestSession
$session.Cookies.Add((New-Object System.Net.Cookie("AppSession", "$appsession", "/", "exe.io")))
$session.Cookies.Add((New-Object System.Net.Cookie("csrfToken", "c7d46cce802f956bde8762f34b65daa4fbcf6d6abdfe555b880fb10f9c114c8e4c23c2bd6934cb3", "/", "exe.io")))
Invoke-WebRequest -UseBasicParsing -Uri "$url" `
-Method POST `
-WebSession $session `
-Headers @{ "Accept-Encoding" = "gzip, deflate, br" "Upgrade-Insecure-Requests" = "1" "Sec-Fetch-Dest" = "document" "Sec-Fetch-Mode" = "navigate" "Sec-Fetch-Site" = "same-origin" "Sec-Fetch-User" = "?1" "TE" = "trailers"
} `
-Body "_method=POST&_csrfToken=c7d46cce802f956bde8762f34b65daa4fbcf6d6abdfe555b880fb10f9c114c8e4c23c2bd6934cb3&_Token%5Bfields%5D=$token&_Token%5Bunlocked%5D=adcopy_challenge%257Cadcopy_response%257Ccaptcha_code%257Ccaptcha_namespace%257Cg-recaptcha-response"
}

DBI

от 40 000 ₽

16 июл. 2024, в 23:57

20000 руб./за проект

16 июл. 2024, в 22:38

2000 руб./за проект

16 июл. 2024, в 22:20

100000 руб./за проект

Ignore SSL/TLS Certificate Checks

Another issue is that the Invoke-WebRequest cmdlet is closely related to Internet Explorer. For example, in Windows Server Core editions in which IE is not installed (or removed), the Invoke-WebRequest cmdlet cannot be used.

Invoke-WebRequest: The response content cannot be parsed because the Internet Explorer engine is not available, or Internet Explorer’s first-launch configuration is not complete. Specify the UseBasicParsing parameter and try again.

In this case, the WebClient class can be used instead of Invoke-WebRequest. For example, to download a file from a specified URL, use the command.

(New-Object -TypeName 'System.Net.WebClient').DownloadFile($Url, $FileName)

If an invalid SSL certificate is used on an HTTPS site, or PowerShell doesn’t support this type of SSL/TLS protocol, then the Invoke-WebRequest cmdlet drops the connection.

Invoke-WebRequest : The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

Invoke-WebRequest : The request was aborted: Could not create SSL/TLS secure channel.

By default, Windows PowerShell (in early builds of Windows 10, Windows Server 2016, and older versions of Windows) uses the legacy and insecure TLS 1.0 protocol for connections (check the blog post describing the PowerShell module installation error: Install-Module: Unable to download from URI).

In PowerShell Core, the Invoke-WebRequest cmdlet has an additional parameter –SkipCertificateCheck that allows you to ignore invalid SSL/TLS certificates.

Another significant drawback of the Invoke-WebRequest cmdlet is its rather low performance. When downloading a file, the HTTP stream is entirely buffered into memory, and only after the full download is completed is saved to disk. Thus, when downloading large files using Invoke-WebReques, you may run out of RAM.

Using Invoke-WebRequest with Authentication

Some web resources require authentication to access. You can use various types of authentication with the Invoke-WebRequest cmdlet (Basic, NTLM, Kerberos, OAuth, or certificate authentication).

$cred = Get-Credential
wget -Uri 'https://somesite.com' -Credential $cred

Invoke-WebRequest 'https://somesite.com' -UseDefaultCredentials

DefaultCredentials not working with Basic authentication.

To authenticate with a certificate, you need to specify its thumbprint:

Invoke-WebRequest 'https://somesite.com' -CertificateThumbprint xxxxxxxxxxxxxxxxxxxxxxxxxxxxx

You can use modern Bearer/OAuth token authentication in your PowerShell scripts.

1. First, you need to get an OAuth token from your REST API provider (out of the scope of this post);
2. Convert token with ConvertTo-SecureString cmdlet: $Token = "12345678912345678901234567890" | ConvertTo-SecureString -AsPlainText –Force
3. Now you can perform OAuth authentication:
   $Params = @{
Uri = "https://somesite.com"
Authentication = "Bearer"
Token = $Token }
Invoke-RestMethod @Params

How to Fill and Submit HTML Form with PowerShell?

Using the next command, display the list of the fields to fill in the login HTML form (login_form):

Assign the desired values to all fields:

To submit (sent) the filled form to the server, call the action attribute of the HTML form:

You can also use the JSON format to send data to a web page with the POST method:

Parse and Scrape HTML a Web Page with PowerShell

The Invoke-WebRequest cmdlet allows you to quickly and conveniently parse the content of any web page. When processing an HTML page, collections of links, web forms, images, scripts, etc., are created.

Let’s look at how to access specific objects on a web page. For example, I want to get a list of all outgoing links (A HREF objects) on the target HTML web page:

You can only select links with a specific CSS class:

Or specific text in the URL address:

Then display a list of all images on this page:

Create a collection of full URL paths to these images:

Initialize a new instance of WebClient class:

$wc = New-Object System.Net.WebClient

Download all the image files from the page (with their original filenames) to the c:\too1s\ folder:

How to Download File over HTTP/FTP with PowerShell Wget (Invoke-WebRequest)?

wget "https://download-installer.cdn.mozilla.net/pub/firefox/releases/102.0.1/win64/en-US/Firefox%20Setup%20102.0.1.exe" -outfile “c:\tools\firefox_setup.exe”

This command will download the file from the HTTP site and save it to the specified directory.

You can get the size of a file in MB before downloading it with wget:

$url = "https://download-installer.cdn.mozilla.net/pub/firefox/releases/102.0.1/win64/en-US/Firefox%20Setup%20102.0.1.exe"
(Invoke-WebRequest $url -Method Head).Headers.'Content-Length'/1Mb

Below is an example of a PowerShell script that will find all links to *.pdf files on a target web page and bulk download all found files from the website to your computer (each pdf file is saved under a random name):

As a result of the script in the target directory, all PDF files from the page will be downloaded. Each file is saved under a random name.

In modern PowerShell Core (6.1 and newer), the Invoke-WebRequest cmdlet supports resume mode. Update your version of PowerShell Core and you can use the -Resume option on the Invoke-WebRequest command to automatically resume downloading a file in case a communication channel or server is unavailable:

Invoke-WebRequest -Uri $Uri -OutFile $OutFile –Resume

Get Web Page Content with Invoke-WebRequest Cmdlet

The Invoke-WebRequest PowerShell cmdlet allows you to send the HTTP, HTTPS, or FTP request with the GET method to the specified web page and receive a response from the server.

There are two aliases for the Invoke-WebRequest command in Windows: iwk and wget.

Invoke-WebRequest -Uri "https://woshub.com"

Tip. If you are connected to the Internet via a proxy server, you must properly configure PowerShell to access the Web through the proxy server. If you do not set proxy parameters, you will receive an error when you run the IWK command:

Invoke-WebRequest:  Unable to connect to the remote server
CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

The command loaded the page and displayed its contents in the PowerShell console. The returned response is not just the HTML code of the page. The Invoke-WebRequest cmdlet returns an object of type HtmlWebResponseObject. Such an object is a collection of forms, links, images, and other important elements of an HTML document. Let’s look at all the properties of this object:

To get the raw HTML code of the web page that is contained in the HtmlWebResponseObject object, run:

You can list the HTML code along with the HTTP headers returned by the web server:

You can check only the web server HTTP status code and the HTTP headers of the HTML page:

As you can see, the server has returned a response 200. This means that the request has been successful, and the web server is available and works correctly.

Key               Value
---               -----
Transfer-Encoding chunked
Connection        keep-alive
Vary              Accept-Encoding,Cookie
Cache-Control     max-age=3, must-revalidate
Content-Type      text/html; charset=UTF-8
Date              Wed, 13 Jul 2022 02:28:32 GMT
Server            nginx/1.20.2
X-Powered-By      PHP/5.6.40

To get the last modification time of a web page:

The list of available agents in PowerShell can be displayed like this:

Похожее:

Веб-запрос http внезапно не ответит Windows 10 — Wikipedia Republished // WIKI 2 <strong>как мне выбрать элементы, которые приходят из запроса на веб-сайт?</strong> Выполнять http-запросы загрузить файлы анализ веб-сайта с помощью power shell