Trying To Loading Html Content Of "http://links.casemakerlegal.com/states/CA/books/Case_Law/results?search[Cite]=214 Cal.App.3d 533" but HttpWebResponse object Giving This Error "(500) Internal Server Error"
And Code Is------
request = WebRequest.Create(urlCheck); request.Timeout = 100000; response = request.GetResponse(); strmRead = new StreamReader(response.GetResponseStream(),System.Text.Encoding.UTF8); result = strmRead.ReadToEnd();
You need to use a tool like Wireshark or Ethereal, or the developer tools in your browser to investigate this further. It is likely the browser is sending some values in the HTTP Header that your code is not, and the server is returning a 500 due to these missing values. Try replicating all of the headers that the browser is using in your code to see if this resolves the problem.
It is usually browser agent. try adding a valid browser agent to your request headers along with Accept and Accept-Encoding headers,
*Edit: For example:
request.UserAgent = "Mozilla/5.0 (Windows NT 5.2; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1";
request.Headers.Add("Accept-Encoding: gzip, deflate");
request.Headers.Add("Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
request.Headers.Add("Accept-Language: en;q=0.8");
Related
I am posting the following JSON to my Flask server:
{"comment": "astute observation", "ua": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36", "cookie": "ua=f8256d37159e3faf28ae61a6406601c3; platform=pc; bs=bxa7ziiq07dufk31prvoo0mbfm44sayh; ss=994139249854417186; il=v1qyca8PU7X6jSwgiqceXsySwWB60HnCjSJ1HjNmsSxRUxNjQzNjA3OTEycUVTTFoxWUpSZDFTZ3kwclIzenBHa19FbnlmMTlIN0hZeHNDQ1FOLQ..; expiredEnterModalShown=1", "parent": "860245961"}
When sent from Postman, the request works flawlessly. However, when sent from my C# app, like so, the server returns a 400 Bad Request error code.
var data = $"{{\"comment\": \"{text}\", \"ua\": \"{userAgent}\", \"cookie\": \"{cookie}\", \"parent\": \"{parent}\"}}";
var url = "http://127.0.0.1:5000/";
var request = HttpWebRequest.CreateHttp(url);
request.Method = "POST";
request.ContentType = "application/json";
await using var sw = new StreamWriter(request.GetRequestStream());
await sw.WriteAsync(data);
var response = await request.GetResponseAsync();
To make sure that the JSON is properly formatted, I set a breakpoint and inspected the "data" variable. I copied/pasted that value into Postman, set the Content-Type to application/json, and the request succeeds from there, but fails from my C# application.
Server code:
from flask import Flask, request
app = Flask(__name__)
#app.route('/', methods=['GET', 'POST'])
def process():
print(request.json)
return 'Hello'
app.run(debug=True)
Edit: After posting this, I realized I wasn't sending a user agent from my C# app. I tried adding one and the server still responds with a 400 Bad Request error.
Edit 2: I tried simplifying the data variable for testing purposes. var data = "{\"msg\": \"david\"}"; and setting request.ContentLength = data.Length, still having the same problem. Really strange.
Very odd, but this solved the problem. Using the StreamWriter class was causing the issue. Doing this instead results in a 200 OK response finally.
using var stream = request.GetRequestStream();
await stream.WriteAsync(Encoding.UTF8.GetBytes(data));
All I'm trying to do is create a program that gets a web response from Nike's upcoming shoe's page, however I keep running into an error saying this is forbidden. No other threads on this topic have been of use to me, is there anything I can do for this or am I just screwed? This is the code:
WebRequest request = WebRequest.Create("https://www.nike.com/launch/?s=upcoming");
WebResponse response = request.GetResponse();
and this is the error:
System.Net.WebException: 'The remote server returned an error: (403) Forbidden.'
Seems like a header issue, try this:
WebClient client = new WebClient();
client.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
client.Headers.Add("Content-Type", "application / zip, application / octet - stream");
client.Headers.Add("Referer", "http://whatevs");
client.Headers.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
String someStuff = client.DownloadString("https://www.hassanhabib.com");
Console.WriteLine(someStuff);
Console.Read();
Removed the Accept-Encoding line, should be fine now.
this is quiet different than the answers given on the other questions on the stack.
i m confused why i'm getting the exception "Too many requests" when sending a GET request to the server .
the request works fine on BURP-SUIT/PostMan even without headers . and i tried to setup 10 continuous requests on Postman and i received OK status code on all of them with interval of 700 ms . even tho in c# Code i'm still getting this exception .
any help is really appreciated .
EDIT :
var req = WebRequest.Create("example.com") as HttpWebRequest;
req.Method = "GET";
req.UserAgent = "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0";
req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
req.GetResponse();
Add this to your request header:
Retry-After: 120
Like this on code :
request.Headers.Add("Retry-After", "120");
My code is :
string result = new WebClient().DownloadString("https://www.facebook.com/profile.php?id=123456789");
the result giving me supported browser.
body class=\"unsupportedBrowser
What i intend to do:
Download a source code from facebook for the particular page.
Problem encounter :
I get the stream from facebook, facebook block me since i access from the apps due to this is not a valid browser.
Expectation : How i can submit the browser type like chrome to cheat the facebook as this is a valid browser.
You can add headers to the webclient object. However, I prefer to go the HttpWebRequest/HttpWebResponse method of scraping since I believe it gives more options.
Headers.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
Headers.Add("Accept-Encoding", "gzip, deflate");
Headers.Add("Accept-Language", "en-US,en;q=0.5");
Headers.Add("Cookie", "has_js=1");
Headers.Add("DNT", "1");
Headers.Add("Host", host);
Headers.Add("Referer", url);
Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0");
I'd like my application to query a csv file from a secure website. I have no experience with web programming so I'd appreciate detailed instructions. Currently I have the user login to the site, manually query the csv, and have my application load the file locally. I'd like to automate this by having the user enter his login information, authenticating him on the website, and querying the data. The application is written in C# .NET.
I've tested the following code already and am able to access the file once the user has already authenticated himself and created a manual query.
System.Net.WebClient Client = new WebClient();
Stream strm = Client.OpenRead("https://<URL>/file.csv");
Here is the request sent to the site for authentication. I've angle bracketed the real userid and password.
POST /pwdVal.asp HTTP/1.1
Accept: image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/x-shockwave-flash, */*
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; Tablet PC 2.0; OfficeLiveConnector.1.4; OfficeLivePatch.1.3; .NET4.0C; .NET4.0E)
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate
Cookie: ASPSESSIONID<unsure if this data contained password info so removed>; ClientId=<username>
Host: www3.emidas.com
Content-Length: 36
Connection: Keep-Alive
Cache-Control: no-cache
Accept-Language: en-US
client_id=<username>&password=<password>
Most likely the server sends a cookie once login is performed. You need to submit the same values as the login form. (this can be done using UploadValues()) However, you need to save the resulting cookies in a CookieContainer.
When I did this, I did it using HttpWebRequest, however per http://couldbedone.blogspot.com/2007/08/webclient-handling-cookies.html you can subclass WebClient and override the GetWebRequest() method to make it support cookies.
Oh, also, I found it useful to use Fiddler while manually accessing the web site to see what actually gets sent back and forth to the web site, so I knew what I was trying to reproduce.
edit, elaboration requested: I can only elaborate how to do it using HttpWebRequest, I have not done it using WebClient. Below is the code snippet I used for login.
private CookieContainer _jar = new CookieContainer();
private string _password;
private string _userid;
private string _url;
private string _userAgent;
...
string responseData;
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(_url);
webRequest.CookieContainer = _jar;
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.UserAgent = _userAgent;
string requestBody = String.Format(
"client_id={0}&password={1}", _userid, _password);
try
{
using (StreamWriter requestWriter = new StreamWriter(webRequest.GetRequestStream()))
{
requestWriter.Write(requestBody);
requestWriter.Close();
using (HttpWebResponse res = (HttpWebResponse)webRequest.GetResponse())
{
using (StreamReader responseReader = new StreamReader(res.GetResponseStream()))
{
responseData = responseReader.ReadToEnd();
responseReader.Close();
if (res.StatusCode != HttpStatusCode.OK)
throw new WebException("Logon failed", null, WebExceptionStatus.Success, res);
}
}
}
Before you go down this rabbit hole, contact the web site and ask them if they provide a web service to query user account info from. The simulated login method you are proposing should be a last resort only.
Another way you can do it is to automate IE, e.g. use a WebBrowser control. That will more accurately simulate all the clever stuff that IE does like running Javascript, which might be necessary. Although if Javascript or other clever stuff isn't necessary then using IE is a little heavy-handed and possibly prone to other problems.