Here i do a Post request and i know the address (i am not the owner) and it is not malicious, I just want to Post the request and get the desired response.
Web request code:
HttpWebRequest oHTTP = (HttpWebRequest)WebRequest.Create("https://some-random-website.com/");
string data = Uri.EscapeDataString(parameters);
oHTTP.Method = "POST";
oHTTP.ContentType = "application/x-www-form-urlencoded";
oHTTP.UserAgent = "Mozilla/5.0 (Windows NT 9; WOW64; rv:38.0) Firefox:40.1";
oHTTP.ContentLength = parameters.Length;
using (Stream stream = oHTTP.GetRequestStream())
stream.Write(Encoding.ASCII.GetBytes(parameters), 0, parameters.Length);
HttpWebResponse response = (HttpWebResponse)oHTTP.GetResponse();
string oReceived = new StreamReader(response.GetResponseStream() ?? throw new InvalidOperationException()).ReadToEnd();
Response title:
Warning: Suspected Phishing Site Ahead!
Then there is a button that says:
Dismiss this warning and enter site
So my question is how can i ignore this warnings and post my request successfully? Should i change my UserAgent?
Note1: I use Fiddler to inspect both request and response header and content.
Note2: I have done the same thing in AutoIt but it uses WinHttp and there is no issue on this website.
Related
This is my C# code:
var url = "http://10.2.0.2/api";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Referer = url;
request.Host = "wwww.abc.com";
request.Accept = "*/*";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
while I run the code, and open fiddler to capture request,
I found fiddler discard header: Host,
so my IIS returned an error!
How did I avoid it?
Why fiddler discards Host but keep other headers?
Question: another
was header be changed, my question was header be discard.
I found my solution, open FiddlerScript, and add these script:
static function OnBeforeRequest(oSession: Session) {
var sOverride = oSession["X-Original-Host"];
if (!String.IsNullOrEmpty(sOverride))
{
oSession.oRequest.headers["Host"] = sOverride;
}
So I am currently trying to log into my account on a website using WebRequest.
I have been reading about it to the point where I feel like I wanted to use an example to learn by trial and error.
This is the example I am using
Login to website, via C#
So when I try to execute my code it returns an unhandled exception and its this one
System.Net.WebException: 'The remote server returned an error: (404)
Not Found.'
I tried stepping through the code and I THINK it might be that it's trying to POST somewhere where it can't.
I wanted to fix this before moving onto getting a confirmation that it successfully logged in.
I changed the username and password to dummy text for the sake of this question.
What did I do wrong here and whats the most logical way of fixing this issue?
Thanks in advance.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
string formUrl = "https://secure.runescape.com/m=weblogin/login.ws"; // NOTE: This is the URL the form POSTs to, not the URL of the form (you can find this in the "action" attribute of the HTML's form tag
string formParams = string.Format("login-username={0}&login-password={1}", "myUsername", "password");
string cookieHeader;
WebRequest req = WebRequest.Create(formUrl);
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
When you scrape a website, you have to make sure you mimic everything that happens. That includes any client-side state (Cookies) that is sent earlier before a form is POST-ed. As most sites don't like to be scraped or steered by bots they are often rather picky about what is the payload. Same is true for the site you're trying to control.
Three important things you have missed:
You didn't start with an initial GET so you have the required cookies in a CookieContainer.
on the post you missed an header (Referrer) and three hidden fields in the form.
The form fields are named username and password (as can be seen in the name attribute of the input tags). You have used the id's.
Fixing those omissions will result in the following code:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
string useragent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36";
// capture cookies, this is important!
var cookies = new CookieContainer();
// do a GET first, so you have the initial cookies neeeded
string loginUrl = "https://secure.runescape.com/m=weblogin/loginform.ws?mod=www&ssl=0&dest=community";
// HttpWebRequest
var reqLogin = (HttpWebRequest) WebRequest.Create(loginUrl);
// minimal needed settings
reqLogin.UserAgent = useragent;
reqLogin.CookieContainer = cookies;
reqLogin.Method = "GET";
var loginResp = reqLogin.GetResponse();
//loginResp.Dump(); // LinqPad testing
string formUrl = "https://secure.runescape.com/m=weblogin/login.ws"; // NOTE: This is the URL the form POSTs to, not the URL of the form (you can find this in the "action" attribute of the HTML's form tag
// in ther html the form has 3 more hidden fields, those are needed as well
string formParams = string.Format("username={0}&password={1}&mod=www&ssl=0&dest=community", "myUsername", "password");
string cookieHeader;
// notice the cast to HttpWebRequest
var req = (HttpWebRequest) WebRequest.Create(formUrl);
// put the earlier cookies back on the request
req.CookieContainer = cookies;
// the Referrer is mandatory, without it a timeout is raised
req.Headers["Referrer"] = "https://secure.runescape.com/m=weblogin/loginform.ws?mod=www&ssl=0&dest=community";
req.UserAgent = useragent;
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
This returns for me success. It is up to you parse the resulting HTML to plan your next steps.
I have tried many ways to login to an https website programmatically, but I am having issues. Every time I get an error stating that my login and password are incorrect. I am sure they are correct because I can login to the site via the browser using the same credentials.
Failing Code
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("https://www.majesticseo.com/account/login?EmailAddress=myemail&Password=mypass&RememberMe=1");
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,**;q=0.8";
request.UnsafeAuthenticatedConnectionSharing = true;
request.Method = "POST";
request.KeepAlive = true;
request.ContentType = "application/x-www-form-urlencoded";
request.AllowAutoRedirect = true;
request.CookieContainer = container;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
//String tmp;
foreach(Cookie cookie1 in response.Cookies)
{
container.Add(cookie1);
}
Stream stream = response.GetResponseStream();
string html = new StreamReader(stream).ReadToEnd();
Console.WriteLine("" + html);
That site uses HTTP POST for login, and does not send the username and password in the URL.
The correct login URL is https://www.majesticseo.com/account/login
You need to create a string of data to post, convert it to a byte array, set the content length and then do your request. It is very important that the content-length is sent. Without it the post will not work.
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("https://www.majesticseo.com/account/login?EmailAddress=myemail&Password=mypass&RememberMe=1");
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0";
request.Referer = "https://www.majesticseo.com/account/login";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,**;q=0.8";
request.UnsafeAuthenticatedConnectionSharing = true;
request.Method = "POST";
request.KeepAlive = true;
request.ContentType = "application/x-www-form-urlencoded";
request.AllowAutoRedirect = true;
// the post string for login form
string postData = "redirect=&EmailAddress=EMAIL&Password=PASS";
byte[] postBytes = System.Text.Encoding.ASCII.GetBytes(postData);
request.ContentLength = postBytes.Length;
System.IO.Stream str = request.GetRequestStream();
str.Write(postBytes, 0, postBytes.Length);
str.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
System.IO.Stream stream = response.GetResponseStream();
string html = new System.IO.StreamReader(stream).ReadToEnd();
Console.WriteLine("" + html);
You are trying to post something (I don't see, what, from your code) but not credentials. I guess that your web page shows you a web form where you enter username (email address?) and password. Then the browsers posts this form. Consequently you need to replicate browser behavior - encode form contents and send them in your post request. Use some webmaster developer tools for popular browsers to see what exactly the client browser sends to the server and how it encodes form data. Next, it's very likely that your request requires special cookies which you can collect by visiting another page (eg. login page). Sending preset cookies (like you do in commented code) won't work for most sites.
In other words, proper mechanism is:
GET the login web page
collect cookies
POST form data and pass collected cookies in the request.
collect other cookies, which could have been sent after login.
I am able to read the url and entire page but not able to read the HTTP POST Request Message Parameters in c#.
In my situation i am posting a post url to a site after they verify they send me a HTTP Post message with parameters like id.
here is my code in c#
HttpWebRequest request1 = (HttpWebRequest)WebRequest.Create(uri);
postsourcedata = "processing=true&Sal=5000000";
request1.Method = "POST";
request1.ContentType = "application/x-www-form-urlencoded";
request1.ContentLength = postsourcedata.Length;
request1.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)";
Stream writeStream1 = request1.GetRequestStream();
UTF8Encoding encoding1 = new UTF8Encoding();
byte[] bytes1 = encoding1.GetBytes(postsourcedata);
writeStream1.Write(bytes1, 0, bytes1.Length);
writeStream1.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader readStream = new StreamReader(responseStream, Encoding.UTF8);
string page = readStream.ReadToEnd();
//page.Close();
return page.ToString();
They are sending me request parameters like id and text , how to read these parameters on my side.I am posting to the website through a web service.
Can anyone help me with this?
If they are sending you an HTTP Post message that means that you either need to have a web server or something that understands HTTP protocol to handle the requests, correct?
What I mean is that by your description, it looks like they are sending you an HTTP Request to port 80 or port 443 (https) and you should have asp.net page to handle the request. Once they hit that page, you can simply do:
Request.Parameters("Id")
Request.Parameters("Text")
And so on.
For several days I've tried to write a program that remote upload image to an image host (imgur.com). I used Wireshark to sniff http requests sent by browser, then create HttpWebRequest with similar headers and parameters. But the server always send back to me something weird. Please look at the code (this code is simplified):
static void Main(string[] args)
{
ServicePointManager.Expect100Continue = false;
CookieContainer cc = new CookieContainer();
List<string> formData = new List<string>();
//The first request - login
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://imgur.com/signin");
configRequest(request, cc);
//add POST params
add(formData, "username", "abcdefgh"); //this is a working account,
add(formData, "password", "abcdefgh"); //feel free to use it if you
add(formData, "remember", "remember"); //want to test
add(formData, "submit", "");
writeToRequestStream(request, formData);
//send request
request.GetResponse();
//The second request - remote upload image
request = (HttpWebRequest)WebRequest.Create("http://imgur.com/upload?sid_hash=9efff36179fef47dc5e078a4575fd96a");
configRequest(request, cc);
//add POST params
formData = new List<string>();
add(formData, "url", "http://img34.imageshack.us/img34/8425/89948070152259768406.jpg");
add(formData, "create_album", "0");
add(formData, "album_title", "Optional Album Title");
add(formData, "album_layout", "b");
add(formData, "edit_url", "0");
writeToRequestStream(request, formData);
//send request
Stream s = request.GetResponse().GetResponseStream();
StreamReader sr = new StreamReader(s);
string html = sr.ReadToEnd();
sr.Close();s.Close();
Console.WriteLine(html + "\n\n");
}
static void add(List<string> formData, string key, string value)
{
formData.Add(HttpUtility.UrlEncode(key) + "=" + HttpUtility.UrlEncode(value));
}
static void configRequest(HttpWebRequest request, CookieContainer cc)
{
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
request.CookieContainer = cc;
request.Credentials = CredentialCache.DefaultCredentials;
request.Accept = "*/*";
request.KeepAlive = true;
request.Referer = "http://imgur.com/";
request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.15) Gecko/20110303 Firefox/3.6.15";
request.Headers.Add("Accept-Language", "en-us,en;q=0.5");
request.Headers.Add("Accept-Encoding", "gzip,deflate");
request.Headers.Add("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.7");
request.Headers.Add("Keep-Alive", "115");
request.Headers.Add("X-Requested-With", "XMLHttpRequest");
request.Headers.Add("Pragma", "no-cache");
request.Headers.Add("Cache-Control", "no-cache");
}
static void writeToRequestStream(HttpWebRequest request, List<string> formData)
{
//build request stream
string queryString = String.Join("&", formData.ToArray());
byte[] byteArray = Encoding.UTF8.GetBytes(queryString);
//write to stream
request.ContentLength = byteArray.Length;
Stream rs = request.GetRequestStream();
rs.Write(byteArray, 0, byteArray.Length);
rs.Close();
}
Now I sniff my uploading request (2nd request) and compare it to the browser's request, there're only 2 differences:
Browser's 'Connection' header ='keep-alive' but mine doesn't exist (I don' know why although request.Keep-alive is set to 'true')
Some browser's cookies doesn't appear in mine.
The response should be a JSON, something like this:
{"hashes":"[\"QcvII\"]","hash":"QcvII","album":false,"edit":false}
But the server responses to my request by a pile of special characters... I can't find out which in above 2 differences makes my code doesn't work. I will extremely appreciate if you can help me making this code work. I'm a newbie so please don't blame me if my code or my expression's silly.
Can anybody help to make this code work?
P/S: i'm using .net framework 4
My guess is that the sid_hash url parameter in your attempt to upload the image is a session id that needs to change when you log in.
OK, now I've found out the solution, fortunately. Forget all things in my function configRequest() (except 3 first lines), they just make things go wrong. The solution is, after sending the login request, send another request to the homepage (no parameter needed, but remember to include the cookies received from the 1st request). The sid_hash can be found in the returned HTML. Use that sid_hash to make the remote uploading request.
Thank you all, guys.
Not sure about your code, but ClipUpload is an open source project that seems to already do about what you want:
Quickly upload anything thats on your clipboard to the internet. It supports FTP, Imgur.com, Pastebin.com and SendSpace.com. Usage? Step 1: Copy. Step 2: Click system tray icon. Step 3: Paste public link. The easiest way to share your clipboard!
Most likely, the second request contains the session ID cookies. Without those cookies, server will not be able to recognise you hence upload will not work.
You can set the keep-alive yourself but my suggestion is to post snippet of the response headers to the first request so we could help.
UPDATE
According to your updates, you need to include this cookie:
IMGURSESSION=iliutpm33rhl2rugn5vcr8jq60
Obviously the value will change with each logging.