Веб-запрос http внезапно не ответит

I have been running a HttpWebRequest on a public website to return JSon for at least a couple of years. Last week it suddenly stopped working and would not give a response via this code (it is called Old because I am using a workaround with a powershell script, but it is a clumsy solution). It eventually times out after the default 100 seconds but a longer timeout would not fix it.

    Function CreateRequestOld(ByVal sMESUser As String, ByVal sMeeting As String, ByVal sRaceDate As String, ByVal iRaceNumber As Integer, ByVal iTrade As Integer, Optional ByVal sJuristriction As String = "VIC") As String
        Dim Url As String
        Dim request As HttpWebRequest = Nothing
        Dim dataStream As Stream = Nothing
        Dim response As HttpWebResponse = Nothing
        Dim strResponseStatus As String = ""
        Dim reader As StreamReader = Nothing
        Dim responseFromServer As String = ""
        Dim sFile As String = "C:\logs\MESRaceLog_" & sMESUser & ".txt"
        Dim Util As New Utilities
        Dim sMeetingCode As String
        Dim RI As New RaceInfo
        Dim dte As New DateClass
        Dim dtRaceDate As Date

        If (iTrade = 1) Then
            sFile = "C:\logs\MESRaceTradeLog_" & sMESUser & ".txt"
        End If
        If (iTrade = 2) Then
            sFile = "C:\logs\MESRaceOddsLog_" & sMESUser & ".txt"
        End If

        Try
            sMeetingCode = RI.GetTABCouseCode(sMeeting)
            If (sMeetingCode = "") Then
                Util.LogMsg(sFile, "TAB CreateRequest Error no meeting code for Meeting name " & sMeeting)
                Return ""
            End If

            dtRaceDate = dte.ConvertStringToDate(sRaceDate, "dd/mm/yyyy")
            sRaceDate = dte.Get_Date_String(dtRaceDate, "yyyy-mm-dd")

            Url = "https://api.beta.tab.com.au/v1/tab-info-service/racing/dates/" & sRaceDate & "/meetings/R/" & sMeetingCode & "/races/" & iRaceNumber & "?jurisdiction=" & sJuristriction
            Util.LogMsg(sFile, "TAB CreateRequest URL " & Url)

            request = HttpWebRequest.Create(New Uri(Url))
            request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
            request.Method = "GET"
            request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
            request.AllowAutoRedirect = True
            response = request.GetResponse()

            strResponseStatus = CType(response, HttpWebResponse).StatusDescription
            dataStream = response.GetResponseStream()
            reader = New StreamReader(dataStream)
            responseFromServer = reader.ReadToEnd()
            reader.Close()
            dataStream.Close()
            response.Close()

        Catch ex As Exception
            Util.LogMsg(sFile, "CreateRequest Error" & vbCrLf & ex.Message)
        End Try
        Return responseFromServer
    End Function

However, if I paste the URL into a browser (Chrome in my case) it responds with Json as expected.

So I tried Postman and it also does not respond if it is just the plain URL. If however, I copy the curl script and use Postman it does respond correctly. I notice with the curl script it has 23 request header elements, whereas with the plain URL (the one that fails) there are only 5 header elements.

I suspect that the website has changed to HTTP/2 and therefore requires all these extra header elements. So I copied the URL as Windows Powershell from Chrome (under developer tools, right click, copy as Windows Powershee) and created a script which is below and this runs well and grabs the response.

$session = New-Object Microsoft.PowerShell.Commands.WebRequestSession
$session.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
$session.Cookies.Add((New-Object System.Net.Cookie("_cs_c", "0", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_tgpc", "d077be37-9bfe-58ce-bb34-2219fe18534c", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("s_ecid", "MCMID%7C42816005962630671342879997247751714175", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_gcl_au", "1.1.1461016396.1701314243", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("__qca", "P0-115178265-1701314243095", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_fbp", "fb.2.1701314243684.710804252", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_scid", "173c7c5f-d2aa-4407-9b6d-5eb48d7d5326", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_scid_r", "173c7c5f-d2aa-4407-9b6d-5eb48d7d5326", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_sctr", "1%7C1706792400000", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("at_check", "true", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("AMCVS_5C750C3A53DB4D070A490D4D%40AdobeOrg", "1", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_abck", "E98C9DC841935F14412438E52AEE528E~0~YAAQBcfOF59EHWuNAQAAsVHf0gvjwuRSy0ElwTKYqvAoJLQPTUBCXrzWbnUSm592eN9RKleGSHb6N1G4sOpkqms4YYyspcf+3j26XCyZ5cUO/PtysyIwLocjO/mDFZ0aQTYZIcMXSuuNycWI0uarkBykoIE5TBfeo6kPD1jD7LecoSHOPVRT/z/FC/0XUS8QcgfqHESSh7Gk6D8Dh3OBQ67qpu6GXq2enMxvn/8v0vn9u0DfH739OwrQyIfO82L11CRPswZtzXj0Wk5Qx5z9ObdAZ+eM9Edz+ts4SjQVXm585PwP9aam/xPL6Lqhov+8+htIFbaKT9O9WoemaBfthkvFk90cFfZ4tv57iPDPwT7+dIDGIcwph6wK5gsd9DS4kCpPXiOIgDcJIV4r6/tL2h8xgm+Ui+Sx~-1~-1~-1", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("bm_sz", "55EEBE064EF9FD3569F82C5AE9083857~YAAQBcfOF6FEHWuNAQAAsVHf0had2HtUmx7KB2ZnSoNh2rj59HNbyA1TAzJCjLmz8aXSPpV1Q699Pfux/9L01gXAmWItcxkttXyzNTYJBwN5/0YXC7Z5KMVxPA4YnDI+DPEUDgbo9pls/PxE86j6qFbQsHPk5mjH5trCuBl4I6lLa4we6cHSZZ2D8Rb8hNSdi3hfkgNtXDISvW35NyOo3EMgBYkb3SZL5BgjWsI6D/Bw9rpj80MRJs/NPZPHOaCvj76NQEHEr2FNclSS1j5sKa3dCeLQC3CfFtw705FlSHX+fxLCaJr0k+V3guclD9JQxD64qzawCSqOur3H8sGNdd6eRHPULHeijC+naKplTbnYgTtwUX2KdjE=~3621429~4469301", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("AKA_A2", "A", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("ak_bmsc", "F23866DFAFE8CC0C588C07C641D011C3~000000000000000000000000000000~YAAQOU5haKnFDcqNAQAAbNln0xZzvuAFiiDMoeSqPqgw/Qyd1vMSeWW1tBvtNwt1/1xz7GqrMaHlb6o/7sgtgmAm+DDYW43r4aLL0IpxZ2znjHkqMe9WUQGFW1z66zd5FWnbVJph+HPgdO38rC0bnaxlGp4EjsgLMVSp3JoveyRNFernNgur79/hEqApGaNw0wIdq+0uT1GIN7MzFitWaLYEXdbwgWmGYroXCzL4f2Jcxuz4jlsrvo+BhP36wRHKjmflq36j+D9bcg1E6QLT94auind7POTHnvh/uERvVKxGJdW6b4DpuHYiWo8s0EwK2PE2d3+7nXCH1iTxV7+2sDPmcMmqrPbGcrIH0UXR5bb7d2P64v4CgiynMUNx3AlK9JJBSm1czBQFKlMQ", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("www.tab.com.au", "179643557%7CMCIDTS%7C19777%7CMCMID%7C42816005962630671342879997247751714175%7CMCAAMLH-1709253617%7C8%7CMCAAMB-1709253617%7CRKhpRz8krg2tLO6pguXWp5olkAcUniQYPHaMWWgdJ3xzPWQmdj0y%7CMCOPTOUT-1708656017s%7CNONE%7CMCAID%7CNONE%7CMCCIDH%7C-394740951%7CvVersion%7C5.5.0", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("bm_mi", "7E78507F65544A6BEB911B6594252269~YAAQOU5haLLFDcqNAQAA1N1n0xaR8VACVmX0VDfQyabr26S83xcjsuZDySCB0Ba3KF8V7EBTrDO2I0q0LqCGVCXBIEpm4kqRUrsWAnZ2ku6pa3LWedJ/oPEPdqXXQz4MRQoqBv3xk/iKpQsEscblrzmKbEh1S+MoRjUsDv05k7V8/YypSTJlyfrhrA4XFBRJcyKThMUSgVmKmftm6lFi2Iv5X32BWk9tdClyDo1WcoR1gf3benLG+0dVsRJlJi3GhxxJh6NzUnbl5iuFh+p7VQtjevNJYtVX66OX773tSVIteUnl/rYsf5u/6dPm1F8FllaIkx0BtA==~1", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_cs_mk_aa", "0.1392149211434872_1708648818354", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("adcloud", "{%22_les_v%22:%22y%2Ctab.com.au%2C1708650618%22}", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_tguatd", "eyJzYyI6IihkaXJlY3QpIn0=", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_tgidts", "eyJzaCI6ImQ0MWQ4Y2Q5OGYwMGIyMDRlOTgwMDk5OGVjZjg0MjdlIiwiY2kiOiJkODhmOTQ2ZS0xNGY2LTU2MDUtODJlOS02N2FiZjA0NDQ0MzIiLCJzaSI6IjBhYzA0OGMzLTliZjgtNTBkZC04ZmZhLTQzY2U3N2ZkNGQ5MCJ9", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_tglksd", "eyJzIjoiMGFjMDQ4YzMtOWJmOC01MGRkLThmZmEtNDNjZTc3ZmQ0ZDkwIiwic3QiOjE3MDg2NDg4MTg1MTUsInNvZCI6IihkaXJlY3QpIiwic29kdCI6MTcwMTMxNDI0MjY2NSwic29kcyI6Im8iLCJzb2RzdCI6MTcwMTMxNDI0MjY2NX0=", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_gid", "GA1.3.286042675.1708648819", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_ga", "GA1.3.1084240241.1701314243", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_tgsid", "eyJscGQiOiJ7XCJscHVcIjpcImh0dHBzOi8vd3d3LnRhYi5jb20uYXUlMkZcIixcImxwdFwiOlwiT25saW5lJTIwU3BvcnRzJTIwJTI2JTIwSG9yc2UlMjBSYWNpbmclMjBCZXR0aW5nJTIwQXVzdHJhbGlhJTIwJTdDJTIwVEFCXCIsXCJscHJcIjpcIlwifSIsInBzIjoiZjk5OWNmMDktNDlkOC00OTM3LTgyMmQtNzVkYWJhMzlhYjM1IiwicHZjIjoiMSIsInNjIjoiMGFjMDQ4YzMtOWJmOC01MGRkLThmZmEtNDNjZTc3ZmQ0ZDkwOi0xIiwiZWMiOiI0IiwicHYiOiIxIiwidGltIjoiMGFjMDQ4YzMtOWJmOC01MGRkLThmZmEtNDNjZTc3ZmQ0ZDkwOjE3MDg2NDg4MjE4MDU6LTEifQ==", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_cs_id", "8c885f2f-4ec7-a81f-a51d-0c7fbc4e5253.1701314242.13.1708649137.1708648818.1.1735478242634.1", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_cs_s", "4.0.0.1708650937050", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("mbox", "PC#4d529f94d644416b83c7f3f4420ea6c1.36_0#1771893939|session#e85cb33e18f1458fae30265336746b93#1708650999", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_uetsid", "1e537e60d1e411eeb9d3b9dc7b7b2ef3", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_uetvid", "fa7b46408f2e11ee94a259059b2087fb", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("bm_sv", "FA3401F77595CB3BFA48E9B3EB68D923~YAAQB1rWF90I/s+NAQAAvYR40xY/4Sg4442TcILWMORP16M+CUypxE/1Q8GoIPyAaR3Dw6QFQYepBzb6OzgAuHqjwVwgKffGPGlNSIakay0I67nwTvg22/n4RLbWrXHiKR1WUnyy0xS1ldQdlsQLEU9TbDrhJbIKxcuaoIvFSlWb13RSwFgigd6I77vhvfog4dYam2Prx93Z9S5AsP/Yt8mjF+UnQjukoloUsVSrB/tmd1rbngxGNFwKDA08xL7MVjU=~1", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_ga_ZMHBYSQ22N", "GS1.1.1708648818.12.1.1708649915.60.0.0", "/", ".tab.com.au")))
Invoke-WebRequest -UseBasicParsing -Uri "https://api.beta.tab.com.au/v1/tab-info-service/racing/dates/ThisRaceDate/meetings/R/ThisRaceCode/races/999?jurisdiction=VIC" `
-WebSession $session `
-Headers @{
"authority"="api.beta.tab.com.au"
  "method"="GET"
  "path"="/v1/tab-info-service/racing/dates/today/meetings?jurisdiction=VIC"
  "scheme"="https"
  "accept"="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
  "accept-encoding"="gzip, deflate, br"
  "accept-language"="en-US,en;q=0.9"
  "cache-control"="max-age=0"
  "dnt"="1"
  "if-none-match"="W/`"850d3337bda07909d530fde6a483487e`""
  "sec-ch-ua"="`"Not A(Brand`";v=`"99`", `"Google Chrome`";v=`"121`", `"Chromium`";v=`"121`""
  "sec-ch-ua-mobile"="?0"
  "sec-ch-ua-platform"="`"Windows`""
  "sec-fetch-dest"="document"
  "sec-fetch-mode"="navigate"
  "sec-fetch-site"="none"
  "sec-fetch-user"="?1"
  "upgrade-insecure-requests"="1"
} -OutFile Race.json

My question is how can I decorate the code HTTPWebRequest with all these 30 odd elements which are set with Powershell using the session.Cookies.Add command within code, and appear as items in the request header when run from a browser (or as a curl script in Postman).

I have tried adding a CookieContainer in my code but to no avail but maybe I didn’t implement it properly. I never needed to set cookies in the past so my knowledge of how to implement them is limited.

I don’t need a solution necessarily in VB.Net. I can change to C#. I only have VB.Net because of some legacy code and have not spent the time changing it all to C#

Update 1
I can now get this code to get a rsponse by adding a bunch of headers, the same as I find in the Powershell command.

This is now my code

'Using request As HttpWebRequest
        request = HttpWebRequest.Create(New Uri(Url))
        request.Pipelined = True
        request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
        request.Method = "GET"
        request.Timeout = 5000
        request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
        request.Headers.Add("authority", "api.beta.tab.com.au")
        request.Headers.Add("method", "GET")
        request.Headers.Add("scheme", "https")
        request.Headers.Add("accept-encoding", "gzip, deflate, br, zstd")
        request.Headers.Add("accept-language", "en-US,en;q=0.9")
        request.Headers.Add("cache-control", "max-age=0")
        request.Headers.Add("dnt", "1")

        request.AutomaticDecompression = DecompressionMethods.GZip
        request.CookieContainer = cookies
        response = request.GetResponse()

        strResponseStatus = CType(response, HttpWebResponse).StatusDescription
        dataStream = response.GetResponseStream()
        reader = New StreamReader(dataStream)
        responseFromServer = reader.ReadToEnd()

        reader.Close()
        dataStream.Close()
        response.Close()

This will run successfully and return a response the first time, but I have this in a loop every 15 seconds. Sometimes the next loop will also work, but sometimes it won’t. Sometimes 3 loops in a row work and then it fails and the next loop works. There is no consistency.

What am I doing wrong? I close the read stream so I don’t think it is that. I am using just a single instance of the HTTPWebRequest and pass the object to this code.

I agree it is not HTTP/2 like I first suspected. It isn;t the timeout set to 5 seconds becuase even at 120 seconds it times out. When it works it takes milliseconds.

When it fails I’ve noticed the Pipelined is set to false, hence why I set it to true, but that seems to make no difference.

А решение, как оказалось, достаточно простое:

powershell -nologo -noprofile "%{[Net.ServicePointManager]::SecurityProtocol = 'Tls12, Tls11, Tls, Ssl3'} ;(Invoke-WebRequest -UseBasicParsing -Uri <ULR>).StatusCode;exit [int]$Error[0].Exception.Status"

Результат считывается из значение свойства “Status” члена “Exception” в переменной $Error. Это помогает обработать ситуации типа такой

Invoke-WebRequest : Невозможно разрешить удаленное имя: ‘cle.linux.org.tw’
строка:1 знак:77
+ %{[Net.ServicePointManager]::SecurityProtocol = ‘Tls12, Tls11, Tls, Ssl3’} ;Invo …
+ ~
~~~
+ CategoryInfo: InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

Status           Property   System.Net.WebExceptionStatus Status {get;}

, которое и даёт ответ, было соединение или нет.

Ну, вот, как‐то так. Почти полный аналог “curl”.

11 min read

At the end of this Invoke-WebRequest PowerShell proxy guide, you will know:

Let’s dive in! 

What Is PowerShell Invoke-WebRequest?

Invoke-WebRequest is a PowerShell cmdlet for sending HTTP, HTTPS, and FTP requests to web servers and web services. By default, it automatically parses the response produced by the server and returns collections of forms, links, images, or other significant HTML elements. 

Usually, it is used for accessing REST APIs, downloading files from the web, or interacting with web services. This is the basic syntax of an Invoke-WebRequest request:

Invoke-WebRequest [-Uri] <Uri> [-Method <WebRequestMethod>] [-Headers <IDictionary>] [-Body <Object>]

The key parameters to remember are:

• -Uri: The URI of the web resource to which the request is sent.
• -Method: The HTTP method to use for the request (e.g., GET, POST, PUT, DELETE). Invoke-WebRequest sends GET requests by default.
• -Headers: The additional HTTP headers to include in the request.
• -Body: The body of the request to send to the server.

As you can see, the only required argument is <Uri>. Thus, in short, the simplest syntax to perform a GET request to a given URI is:

Invoke-WebRequest <Uri>

This cmdlet was introduced in PowerShell 3.0, in 2012. 

Installing Invoke-WebRequest

To use Invoke-WebRequest, you need PowerShell. So, let’s learn how to install PowerShell and get access to the Invoke-WebRequest cmlet!

Windows

You can verify the current version of PowerShell installed on your Windows machine with this command:

$PSVersionTable

On PowerShell 7.x, that should print something similar to:

PSVersion                      7.4.1

PSEdition                      Core

GitCommitId                    7.4.1

OS                             Microsoft Windows 10.0.22631

Platform                       Win32NT

PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}

PSRemotingProtocolVersion      2.3

SerializationVersion           1.1.0.1

WSManStackVersion              3.0

macOS and Linux

Prerequisites to Get Started with a Proxy in PowerShell

A proxy acts as an intermediary between a client and the destination server. It intercepts your requests, forwards them to the server, receives the responses from the server, and sends them back to you. This way, the destination server will see the requests as coming from the IP and location of the chosen proxy server and not from you.

To get started using a PowerShell proxy with Invoke-WebRequest, you need to understand what a proxy server URL looks like. 

This is the URL of a PowerShell Invoke-WebRequest proxy:

<PROTOCOL>://[<USERNAME>:<PASSWORD>]@<HOST>[:<PORT>]

That consists of:

• <PROTOCOL>: The protocol to use to connect to the proxy server.
• <HOST>: The IP address or URL of the proxy server’s hostname.
• <PORT>: The port number the proxy server listens to.
• <USERNAME>: The optional username to specify when proxy authentication is required.
• <PASSWORD>: The optional password to specify when proxy authentication is required.

Invoke-WebRequest : This operation is not supported for a relative URI.

As for proxy protocols, the most popular ones are HTTP, HTTPS, and SOCKS. Invoke-WebRequest in PowerShell 5.1 only supports HTTP while in PowerShell 7.x it also supports HTTPS and SOCKS.

Time to retrieve a valid HTTP proxy!

You can find one for free online, as below:

Protocol: HTTP; IP Address: 190.6.23.219; Port: 999

http://190.6.23.219:999

Warning

Opting for free proxies is okay for learning purposes, but you cannot rely on them in real-world scenarios. Free proxies are unreliable, error-prone, slow, data-greedy, and short-lived. Do not use them!

The solution? Premium proxies from Bright Data, the best provider in the market. Subscribe and try our reliable proxies for free.

http://admin-4521:@rUuH3tJqf45.103.203.109:9571

How to Specify an HTTP Proxy in Invoke-WebRequest

Before getting started, launch the command below in PowerShell:

Invoke-WebRequest "https://httpbin.org/ip"

That should print something like:

StatusCode        : 200

StatusDescription : OK

Content           : {

                      "origin": "194.34.233.12"

                    }

RawContent        : HTTP/1.1 200 OK

                    Connection: keep-alive

                    Access-Control-Allow-Origin: *

                    Access-Control-Allow-Credentials: true

                    Content-Length: 32

                    Content-Type: application/json

                    Date: Thu, 01 Feb 2024 10:46:14 GMT...

Forms             : {}

Headers           : {[Connection, keep-alive], [Access-Control-Allow-Origin, *], [Access-Control-Allow-Credentials,

                    true], [Content-Length, 32]...}

Images            : {}

InputFields       : {}

Links             : {}

ParsedHtml        : mshtml.HTMLDocumentClass

RawContentLength  : 32

Focus on the Content field. That will contain your IP. 

Why? Because the /ip endpoint from the HTTPBin project returns the origin IP of the request. In other words, it returns the IP address of the machine that performed the request. In this case, that is your machine’s IP.

If you want to access only the Content field, you can do it with:

$response = Invoke-WebRequest "https://httpbin.org/ip"

$response.Content

This would print:

{

  "origin": "194.34.233.12"

}

If you route that request through a proxy, you should see the IP address of the proxy server and not yours. Calling that endpoint is therefore a good test to verify that the specified PowerShell Invoke-WebRequest proxy is working as expected. 

There are a couple of ways to set a PowerShell proxy in Invoke-WebRequest. Find out more in the step-by-step guided sections below!

Using a Command Line Option

Invoke-WebRequest offers the -Proxy flag to specify a proxy URL for your request.

So, the syntax to use Invoke-WebRequest with a proxy server becomes:

Invoke-WebRequest -Proxy "<PROTOCOL>://[<USERNAME>:<PASSWORD>]@<HOST>[:<PORT>]" <Uri>

If you now execute this PowerShell command:

Invoke-WebRequest -Proxy "http://190.6.23.219:999" "https://httpbin.org/ip"
Invoke-WebRequest -Uri "http://httpbin.org/ip" -Proxy "http://brd.superproxy.io:22225" -ProxyCredential (New-Object System.Management.Automation.PSCredential("brd-customer-CUSTOMER_ID-zone-ZONE’S_NAME", ("ZONE’S_PASSWORD" | ConvertTo-SecureString -AsPlainText -Force)))

The result should be:

StatusCode        : 200

StatusDescription : OK

Content           : {

                      "origin": "190.6.23.219"

                    }

RawContent        : HTTP/1.1 200 OK

                    Connection: keep-alive

                    Access-Control-Allow-Origin: *

                    Access-Control-Allow-Credentials: true

                    Content-Length: 31

                    Content-Type: application/json

                    Date: Thu, 01 Feb 2024 12:36:56 GMT...

Forms             : {}

Headers           : {[Connection, keep-alive], [Access-Control-Allow-Origin, *], [Access-Control-Allow-Credentials,

                    true], [Content-Length, 31]...}

Images            : {}

InputFields       : {}

Links             : {}

ParsedHtml        : mshtml.HTMLDocumentClass

RawContentLength  : 31

Notice that the origin in Content matches the proxy server IP. This proves that the target server sees the request as coming from the proxy, as expected. Awesome!

Note: Do not forget that free proxies are short-lived! By the time you read this guide, it is unlikely that the above server is still alive. In case of an error, replace it with a fresh proxy.

Using Environment Variables

Since PowerShell 7.0, Invoke-WebRequest supports proxy configuration via environment variables. 

• HTTP_PROXY:The URL of the proxy server to use in case of HTTP requests.
• HTTPS_PROXY: The URL of the proxy server to use in case of HTTPS requests.

On Windows, you can set the two environment variables with this PowerShell syntax:

$env:HTTP_PROXY = "<PROTOCOL>://[<USERNAME>:<PASSWORD>]@<HOST>[:<PORT>]"

$env:HTTPS_PROXY = "<PROTOCOL>://[<USERNAME>:<PASSWORD>]@<HOST>[:<PORT>]"

In our example, the commands will become:

$env:HTTP_PROXY = "http://190.6.23.219:999"

$env:HTTPS_PROXY = "http://190.6.23.219:999"

On macOS and Linux, you need to use the syntax below:

export HTTP_PROXY="<PROTOCOL>://[<USERNAME>:<PASSWORD>]@<HOST>[:<PORT>]"

export HTTPS_PROXY="<PROTOCOL>://[<USERNAME>:<PASSWORD>]@<HOST>[:<PORT>]"

So, the two commands will be:

export http_proxy="http://190.6.23.219:999"

export https_proxy="http://190.6.23.219:999"

From now on, every Invoke-WebRequest request will go through the specified proxies without having to add the -Proxy option. After setting the envs, launch the command below:

Invoke-WebRequest "https://httpbin.org/ip"

You will get the same result as before:

StatusCode        : 200

StatusDescription : OK

Content           : {

                      "origin": "190.6.23.219"

                    }

RawContent        : HTTP/1.1 200 OK

                    Connection: keep-alive

                    Access-Control-Allow-Origin: *

                    Access-Control-Allow-Credentials: true

                    Content-Length: 31

                    Content-Type: application/json

                    Date: Thu, 01 Feb 2024 12:36:56 GMT...

Forms             : {}

Headers           : {[Connection, keep-alive], [Access-Control-Allow-Origin, *], [Access-Control-Allow-Credentials,

                    true], [Content-Length, 31]...}

Images            : {}

InputFields       : {}

Links             : {}

ParsedHtml        : mshtml.HTMLDocumentClass

RawContentLength  : 31

To turn off the Invoke-WebRequest proxies, unset the environment variables with:

$env:HTTP_PROXY = ""

$env:HTTPS_PROXY = ""

Or on macOS and Linux:

unset HTTP_PROXY

unset HTTPS_PROXY

Invoke-WebRequest will go back to its standard behavior, and https://httpbin.org/ip will now expose your IP.

How To Use HTTPS and SOCKS Proxies in PowerShell

If you need to use an HTTPS or SOCKS proxy, you must upgrade to version 7.x+ of PowerShell. Otherwise, Invoke-WebRequest will fail with:

Invoke-WebRequest : The ServicePointManager does not support proxies with the https scheme.

Or in the case of a SOCKS proxy:

Invoke-WebRequest : The ServicePointManager does not support proxies with the socks scheme.

When dealing with HTTPS or SOCKS proxies in PowerShell 7.x, the Invoke-WebRequest command structure remains the same:

Invoke-WebRequest -Proxy "<PROTOCOL>://[<USERNAME>:<PASSWORD>]@<HOST>[:<PORT>]" <Uri>

What changes is that <PROTOCOL> will be https, socks4, socks4a, socks5, or socks5a, instead of http.

If you try to invoke a request with a proxy involving a protocol other than those mentioned above, you will get this error:

Invoke-WebRequest: Only the 'http', 'https', 'socks4', 'socks4a' and 'socks5' schemes are allowed for proxies.

Thus, a complete example of an Invoke-WebRequest SOCKS proxy request is:

Invoke-WebRequest -Proxy "socks5://94.14.109.54:3567" "http://httpbin.org/ip"

As you can expect, the result will be:

StatusCode        : 200

StatusDescription : OK

Content           : {

                      "origin": "94.14.109.54"

                    }

RawContent        : HTTP/1.1 200 OK

                    Connection: keep-alive

                    Access-Control-Allow-Origin: *

                    Access-Control-Allow-Credentials: true

                    Content-Length: 31

                    Content-Type: application/json

                    Date: Thu, 01 Feb 2024 12:47:56 GMT...

Forms             : {}

Headers           : {[Connection, keep-alive], [Access-Control-Allow-Origin, *], [Access-Control-Allow-Credentials,

                    true], [Content-Length, 31]...}

Images            : {}

InputFields       : {}

Links             : {}

ParsedHtml        : mshtml.HTMLDocumentClass

RawContentLength  : 31

Tips and Tricks You Need to Know

See some useful tricks and valuable tips for dealing with a PowerShell Invoke-WebRequest proxy like a pro.

Ignore the PowerShell Proxy Configuration

Invoke-WebRequest -NoProxy <Uri>

This instructs Invoke-WebRequest to contact <Uri> without using a proxy.

To verify that this approach works, set up a proxy in the envs and run:

Invoke-WebRequest -NoProxy "https://httpbin.org/ip"

The resulting origin will contain your IP, not that of the proxy server.

Avoid SSL Certificate Errors

When using HTTP proxies, your requests may fail because of SSL certificate errors. To avoid that, specify the -SkipCertificateCheck option:

Invoke-WebRequest -SkipCertificateCheck -Proxy "<PROTOCOL>://[<USERNAME>:<PASSWORD>]@<HOST>[:<PORT>]" <Uri>

-SkipCertificateCheck helps you avoid certificate errors by allowing insecure server connections. Keep in mind that using this parameter is not secure. Set it only when dealing with known hosts.

For example, you can contact HTTPBin through a proxy while bypassing SSL issues with:

Invoke-WebRequest -SkipCertificateCheck -Proxy "http://190.6.23.219:999" "https://httpbin.org/ip"

Which PowerShell Proxy Should You Use?

The answer to this question changes based on what you want to achieve with your Invoke-WebRequest request. To find the right PowerShell proxy for your needs, take a look at the different types of proxies available:

• Datacenter proxies: They are fast and cheap, but can be easily detected and blocked by sites because of their identifiable IP ranges.
• Residential proxies: They offer rotating genuine IP addresses from real devices in specific locations. This means they can guarantee a high level of anonymity. Residential proxies are perfect for accessing sites that rely on geo-restriction blocks or to avoid anti-bot measures.
• ISP proxies: They are secure, fast, and highly reliable as they provide static IPs from devices registered with ISPs. ISP proxies are also called residential static proxies and are a perfect solution for SEO monitoring and market research.
• Mobile proxies: They provide IPs from real mobile devices for a high level of anonymity. They are useful for accessing applications, sites, or content specifically designed for mobile devices.

Conclusion

In this PowerShell proxy guide, you learned what Invoke-WebRequest is, how it works, and how to use it with an HTTP/HTTPS/SOCKS proxy. As it turned out, you cannot rely on proxies from free providers. Therefore, the only decision to be made is which proxy provider you should adopt. Save time and energy and go directly for the best on the market, Bright Data.

Bright Data controls the best proxy servers in the world, serving Fortune 500 companies and over 20,000 customers. Its worldwide proxy network involves:

Overall, that is one of the largest and most reliable scraping-oriented proxy networks on the market. 

No credit card required

 Windows OS Hub / PowerShell / Invoke-WebRequest: Perform HTTP Requests, Download Files, Parse Web with PowerShell

The Invoke-WebRequest cmdlet can be used to request HTTP/HTTPS/FTP resources directly from the PowerShell console. You can use this command to send HTTP requests (GET and POST), download files from a website, parse HTML web pages, perform authentication, fill out and submit web forms, etc. In this article, we’ll cover basic examples of using the Invoke-WebRequest cmdlet in PowerShell to interact with web services.

Get Web Page Content with Invoke-WebRequest Cmdlet

The Invoke-WebRequest PowerShell cmdlet allows you to send the HTTP, HTTPS, or FTP request with the GET method to the specified web page and receive a response from the server.

There are two aliases for the Invoke-WebRequest command in Windows: iwk and wget.

Invoke-WebRequest -Uri "https://woshub.com"

Tip. If you are connected to the Internet via a proxy server, you must properly configure PowerShell to access the Web through the proxy server. If you do not set proxy parameters, you will receive an error when you run the IWK command:

Invoke-WebRequest:  Unable to connect to the remote server
CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

The command loaded the page and displayed its contents in the PowerShell console. The returned response is not just the HTML code of the page. The Invoke-WebRequest cmdlet returns an object of type HtmlWebResponseObject. Such an object is a collection of forms, links, images, and other important elements of an HTML document. Let’s look at all the properties of this object:

To get the raw HTML code of the web page that is contained in the HtmlWebResponseObject object, run:

You can list the HTML code along with the HTTP headers returned by the web server:

You can check only the web server HTTP status code and the HTTP headers of the HTML page:

As you can see, the server has returned a response 200. This means that the request has been successful, and the web server is available and works correctly.

Key               Value
---               -----
Transfer-Encoding chunked
Connection        keep-alive
Vary              Accept-Encoding,Cookie
Cache-Control     max-age=3, must-revalidate
Content-Type      text/html; charset=UTF-8
Date              Wed, 13 Jul 2022 02:28:32 GMT
Server            nginx/1.20.2
X-Powered-By      PHP/5.6.40

To get the last modification time of a web page:

The list of available agents in PowerShell can be displayed like this:

Using Invoke-WebRequest with Authentication

Some web resources require authentication to access. You can use various types of authentication with the Invoke-WebRequest cmdlet (Basic, NTLM, Kerberos, OAuth, or certificate authentication).

$cred = Get-Credential
wget -Uri 'https://somesite.com' -Credential $cred

Invoke-WebRequest 'https://somesite.com' -UseDefaultCredentials

DefaultCredentials not working with Basic authentication.

To authenticate with a certificate, you need to specify its thumbprint:

Invoke-WebRequest 'https://somesite.com' -CertificateThumbprint xxxxxxxxxxxxxxxxxxxxxxxxxxxxx

You can use modern Bearer/OAuth token authentication in your PowerShell scripts.

1. First, you need to get an OAuth token from your REST API provider (out of the scope of this post);
2. Convert token with ConvertTo-SecureString cmdlet: $Token = "12345678912345678901234567890" | ConvertTo-SecureString -AsPlainText –Force
3. Now you can perform OAuth authentication:
   $Params = @{
Uri = "https://somesite.com"
Authentication = "Bearer"
Token = $Token }
Invoke-RestMethod @Params

Parse and Scrape HTML a Web Page with PowerShell

The Invoke-WebRequest cmdlet allows you to quickly and conveniently parse the content of any web page. When processing an HTML page, collections of links, web forms, images, scripts, etc., are created.

Let’s look at how to access specific objects on a web page. For example, I want to get a list of all outgoing links (A HREF objects) on the target HTML web page:

You can only select links with a specific CSS class:

Or specific text in the URL address:

Then display a list of all images on this page:

Create a collection of full URL paths to these images:

Initialize a new instance of WebClient class:

$wc = New-Object System.Net.WebClient

Download all the image files from the page (with their original filenames) to the c:\too1s\ folder:

How to Download File over HTTP/FTP with PowerShell Wget (Invoke-WebRequest)?

wget "https://download-installer.cdn.mozilla.net/pub/firefox/releases/102.0.1/win64/en-US/Firefox%20Setup%20102.0.1.exe" -outfile “c:\tools\firefox_setup.exe”

This command will download the file from the HTTP site and save it to the specified directory.

You can get the size of a file in MB before downloading it with wget:

$url = "https://download-installer.cdn.mozilla.net/pub/firefox/releases/102.0.1/win64/en-US/Firefox%20Setup%20102.0.1.exe"
(Invoke-WebRequest $url -Method Head).Headers.'Content-Length'/1Mb

Below is an example of a PowerShell script that will find all links to *.pdf files on a target web page and bulk download all found files from the website to your computer (each pdf file is saved under a random name):

As a result of the script in the target directory, all PDF files from the page will be downloaded. Each file is saved under a random name.

In modern PowerShell Core (6.1 and newer), the Invoke-WebRequest cmdlet supports resume mode. Update your version of PowerShell Core and you can use the -Resume option on the Invoke-WebRequest command to automatically resume downloading a file in case a communication channel or server is unavailable:

Invoke-WebRequest -Uri $Uri -OutFile $OutFile –Resume

How to Fill and Submit HTML Form with PowerShell?

Using the next command, display the list of the fields to fill in the login HTML form (login_form):

Assign the desired values to all fields:

To submit (sent) the filled form to the server, call the action attribute of the HTML form:

You can also use the JSON format to send data to a web page with the POST method:

Ignore SSL/TLS Certificate Checks

Another issue is that the Invoke-WebRequest cmdlet is closely related to Internet Explorer. For example, in Windows Server Core editions in which IE is not installed (or removed), the Invoke-WebRequest cmdlet cannot be used.

Invoke-WebRequest: The response content cannot be parsed because the Internet Explorer engine is not available, or Internet Explorer’s first-launch configuration is not complete. Specify the UseBasicParsing parameter and try again.

In this case, the WebClient class can be used instead of Invoke-WebRequest. For example, to download a file from a specified URL, use the command.

(New-Object -TypeName 'System.Net.WebClient').DownloadFile($Url, $FileName)

If an invalid SSL certificate is used on an HTTPS site, or PowerShell doesn’t support this type of SSL/TLS protocol, then the Invoke-WebRequest cmdlet drops the connection.

Invoke-WebRequest : The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

Invoke-WebRequest : The request was aborted: Could not create SSL/TLS secure channel.

By default, Windows PowerShell (in early builds of Windows 10, Windows Server 2016, and older versions of Windows) uses the legacy and insecure TLS 1.0 protocol for connections (check the blog post describing the PowerShell module installation error: Install-Module: Unable to download from URI).

In PowerShell Core, the Invoke-WebRequest cmdlet has an additional parameter –SkipCertificateCheck that allows you to ignore invalid SSL/TLS certificates.

Another significant drawback of the Invoke-WebRequest cmdlet is its rather low performance. When downloading a file, the HTTP stream is entirely buffered into memory, and only after the full download is completed is saved to disk. Thus, when downloading large files using Invoke-WebReques, you may run out of RAM.

New to powerhsell.
Have this json file: Cannot change json as the api is implemented on this.

Code: Select all

[
  {
    "Id": "yourfirstid",
    "Name": "Alex"
  },

  {
    "Id": "yoursecondid",
    "Name": "Peter"
  }
]

Created hashtable using powerhsell as:

Code: Select all

$Body = {@{Id=yourfirstid; Name=Alex}, @{Id=yoursecondid; Name=Peter}}

Powershell Code to post data:

Code: Select all

$token = "somevalue"
$url= "http://somepath"
$headers = @{Authorization = "Bearer $token"} 
$Body = {@{Id=yourfirstid; Name=Alex}, @{Id=yoursecondid; Name=Peter}}
$JsonBody = $Body | ConvertTo-Json
$response = Invoke-RestMethod -ContentType "application/json" -Uri $url -Method Post -Headers $headers -Body $JsonBody
Write-Output $response

Now when I run this script, it gives the below error. However, the $response shows both ids and Name fine. The data didn’t post to the url though.

Code: Select all

Invoke-RestMethod : The remote server returned an error: (409) Conflict.
At C:\Users\test\Documents\PostData.ps1:9 char:13
+ $response = Invoke-RestMethod -ContentType "application/json" -Uri $u ...
+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Last edited by Simon Sheppard on 2023-Oct-17, 11:24 am, edited 1 time in total.

Reason: typo

Похожее:

В powershell 5 1 как переходить предупреждения из команды удаленного вызова <strong>как мне выбрать элементы, которые приходят из запроса на веб-сайт?</strong> Где хранятся файлы cookie в Mozilla, что такое файлы cookie и как они облегчают нашу жизнь в Интернете Проверка доступности uri адреса средствами powershell на подобие curl