I'm having trouble getting a WebRequest to pass authentication when trying to download a file from a URL. I have verified that, hitting the url directly and being prompted with a username / password form, my credentials go through just fine and the file is downloaded to my computer.
However, when using a WebRequest (with a proxy) i'm getting a (401) Authentication failed.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Credentials = new NetworkCredential(username, password);
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
WebProxy proxy = new WebProxy("myProxyAddress:myProxyPort");
proxy.UseDefaultCredentials = false;
proxy.Credentials = System.Net.CredentialCache.DefaultNetworkCredentials;
request.Proxy = proxy;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
It's definitely passing proxy auth as i was a getting a 407 before. I've tried a myriad of settings such as setting PreAuthenticate to true, UseDefaultCredentials to false etc but to no avail.
Is there something i'm missing here? Could it be down to the type of auth the 3rd party url is requiring?
Related
I am posting the following JSON to my Flask server:
{"comment": "astute observation", "ua": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36", "cookie": "ua=f8256d37159e3faf28ae61a6406601c3; platform=pc; bs=bxa7ziiq07dufk31prvoo0mbfm44sayh; ss=994139249854417186; il=v1qyca8PU7X6jSwgiqceXsySwWB60HnCjSJ1HjNmsSxRUxNjQzNjA3OTEycUVTTFoxWUpSZDFTZ3kwclIzenBHa19FbnlmMTlIN0hZeHNDQ1FOLQ..; expiredEnterModalShown=1", "parent": "860245961"}
When sent from Postman, the request works flawlessly. However, when sent from my C# app, like so, the server returns a 400 Bad Request error code.
var data = $"{{\"comment\": \"{text}\", \"ua\": \"{userAgent}\", \"cookie\": \"{cookie}\", \"parent\": \"{parent}\"}}";
var url = "http://127.0.0.1:5000/";
var request = HttpWebRequest.CreateHttp(url);
request.Method = "POST";
request.ContentType = "application/json";
await using var sw = new StreamWriter(request.GetRequestStream());
await sw.WriteAsync(data);
var response = await request.GetResponseAsync();
To make sure that the JSON is properly formatted, I set a breakpoint and inspected the "data" variable. I copied/pasted that value into Postman, set the Content-Type to application/json, and the request succeeds from there, but fails from my C# application.
Server code:
from flask import Flask, request
app = Flask(__name__)
#app.route('/', methods=['GET', 'POST'])
def process():
print(request.json)
return 'Hello'
app.run(debug=True)
Edit: After posting this, I realized I wasn't sending a user agent from my C# app. I tried adding one and the server still responds with a 400 Bad Request error.
Edit 2: I tried simplifying the data variable for testing purposes. var data = "{\"msg\": \"david\"}"; and setting request.ContentLength = data.Length, still having the same problem. Really strange.
Very odd, but this solved the problem. Using the StreamWriter class was causing the issue. Doing this instead results in a 200 OK response finally.
using var stream = request.GetRequestStream();
await stream.WriteAsync(Encoding.UTF8.GetBytes(data));
Consider the following URL: "http://www.bestbuy.com". This resource is quickly and correctly loaded in all browsers and from all locations.
However, basic C# code currently stucks (ends by timeout for any timeout) for this URL:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
var req = WebRequest.CreateHttp("http://www.bestbuy.com");
req.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
req.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36";
req.Timeout = 30000; // you can set any timeout
using (var resp = (HttpWebResponse)req.GetResponse()) // stucks and finally ends by timeout here
{
Console.WriteLine(resp.StatusCode);
Console.WriteLine(resp.ResponseUri.AbsoluteUri);
}
Fiddle: https://dotnetfiddle.net/M7NZgG
The same code works fine for most other URLs.
Tried different things, but all of them did not help:
direct loading of HTTPS version ("https://www.bestbuy.com")
remove calls for UserAgent, AutomaticDecompression and SecurityProtocol setters
HttpClient class also stucks and ends by timeout for that resource.
In Fiddler the response is quickly returned, but it looks strange - it is completely empty:
I have a website on a local network that I am trying to write a little client for.
I am trying to use WebClient for this purpose, however, it seems that the website somehow detects it and does not allow to continue cutting the connection, which results in WebException.
To counter this, I have tried adding headers like:
WebClient wc = new WebClient();
wc.Headers["User-Agent"] = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
wc.Headers.Add("Accept", "text/html,application/xhtml+xml,application/xml");
wc.Headers.Add("Accept-Encoding", "deflate, sdch, br");
wc.Headers.Add("Accept-Charset", "ISO-8859-1");
wc.Headers.Add("Accept-Language", "en-us;q=0.7,en;q=0.3");
However, the website still cut of the connection and I managed to notice that not all headers were sent, then, I have tried to override WebRequest:
public class MyWebClient : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
var castRequest = request as HttpWebRequest;
if (castRequest != null)
{
castRequest.KeepAlive = true;
castRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
castRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
castRequest.Headers.Add("Accept-Encoding", "deflate, sdch, br");
castRequest.Headers.Add("Accept-Charset", "ISO-8859-1");
castRequest.Headers.Add("Accept-Language", "en-US,en;q=0.8");
}
return request;
}
}
This managed to send all the headers, however, I still could not access the website.
I can access the website just fine using any browser like Firefox or Chrome from which I copied the headers or even WebBrowser control, I can also access other websites using WebClient without any issue.
Is there anything specific why I cannot access such website using WebClient?
Is there anything else to make WebClient request look more like browser for a website?
I have figured out that I was looking in the wrong direction.
It seems the website in question does not support default SecurityProtocol, therefore I had to enable TLS12:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
For some reason it has caused the same issue with another local website I was parsing, which I have solved by enabling all TLS versions:
ServicePointManager.SecurityProtocol =
SecurityProtocolType.Tls
| SecurityProtocolType.Tls11
| SecurityProtocolType.Tls12;
I'm trying to get ASP.NET_SessionId from a HttpWebResponse but it seems that no such data comes on the response.
Basically I'm trying to simulate some steps over some pages, where authentication is required. The problem is not in the authentication, my problem is to get ASP.NET_SessionId after the authentication so I can use it in my future requests/steps.
From Chrome on developer tools > network, I can see the ASP.NET_SessionId on the headers, but it does not come in my HttpWebResponse. Any ideia why this is happening?
There is my code:
var httpWebRequest = (HttpWebRequest) WebRequest.Create(url);
httpWebRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36"; httpWebRequest.Method = "POST";
httpWebRequest.ContentType = "application/x-www-form-urlencoded";
httpWebRequest.ContentLength = 0;
var httpWebResponse = (HttpWebResponse) httpWebRequest.GetResponse();
After my request I should see a ASP.NET_SessionId header Set-Cookie, but no luck. Any ideia?
I've seen some people say that
httpWebResponse.Headers["ASP.NET_SessionId"]
or
httpWebResponse.Headers["SESSION_ID"]
should work but no, no session id header is set nor any Cookie.
After many research, the answer was here
Basically we have to keep the same CookieContainer object reference across all requests. I was extracting some Set-Cookie from the responses and adding them into my requests, but now I don't need to do anything, CookieContainer manages all of it transparently.
Set-Cookie from responses are set on CookieContainer of your request. It's the way they found to resolve possible security issues, so don't lose more time and just keep a reference to your CookieContainer because you will not be able to access session id (and you don't need it).
There's the example of my code now.
var cookieContainer = new CookieContainer();
var httpWebRequest1 = (HttpWebRequest) WebRequest.Create(url);
httpWebRequest1.CookieContainer = cookieContainer;
// do the request and some logic
var httpWebRequest2 = (HttpWebRequest) WebRequest.Create(anotherUrl);
httpWebRequest2.CookieContainer = cookieContainer; // same cookieContainer reference
Everything is working great now, hope it helps someone.
Trying To Loading Html Content Of "http://links.casemakerlegal.com/states/CA/books/Case_Law/results?search[Cite]=214 Cal.App.3d 533" but HttpWebResponse object Giving This Error "(500) Internal Server Error"
And Code Is------
request = WebRequest.Create(urlCheck); request.Timeout = 100000; response = request.GetResponse(); strmRead = new StreamReader(response.GetResponseStream(),System.Text.Encoding.UTF8); result = strmRead.ReadToEnd();
You need to use a tool like Wireshark or Ethereal, or the developer tools in your browser to investigate this further. It is likely the browser is sending some values in the HTTP Header that your code is not, and the server is returning a 500 due to these missing values. Try replicating all of the headers that the browser is using in your code to see if this resolves the problem.
It is usually browser agent. try adding a valid browser agent to your request headers along with Accept and Accept-Encoding headers,
*Edit: For example:
request.UserAgent = "Mozilla/5.0 (Windows NT 5.2; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1";
request.Headers.Add("Accept-Encoding: gzip, deflate");
request.Headers.Add("Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
request.Headers.Add("Accept-Language: en;q=0.8");