Develop Reference

C# How to keep connection alive while parsing html

I'm trying to create a little script, that lets you login automatically to a php website.
When I looked at the POST script that is being sent through https, I realized that there is a hidden inputform with a hash string (in the login form hidden with css visibility) which is being send in the data form with the email + password
Example:
login=USERNAME&passwort=PASSWORD&loginhash=1b9f29d68152619b469c3744de3a1f54
So actually I would need to do a simple GET action before the post to parse the string out and then do the POST with the loginhash but how do I keep that connection alive without getting a new hash that is not matching anymore?
//GET
WebRequest req = WebRequest.Create(url);
WebResponse resp = req.GetResponse();
using (StreamReader sr = new StreamReader(resp.GetResponseStream()))
{
reqHtml = sr.ReadToEnd();
}
cookieHeader = resp.Headers["Set-cookie"];
//Parsing html and storing to var loginhash
//POST
string formData = string.Format("login={0}&loginhash={1}&passwort={2}", "USERNAME", loginhash, "PASSWORD");
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
req.Headers.Add("Cookie", cookieHeader);
byte[] bytes = Encoding.ASCII.GetBytes(formData);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
using (StreamReader sr = new StreamReader(resp.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
Update:
Realized, that my theory is actually wrong as I sent two GET's with curl so that curl would pipeline/reuse the connection before. At the end there were still two different hashes in the responses. Maybe the Hash is just a distraction ?

Trying to upload with a PUT request

I am trying to change the price of an article using the websites API
the documentation for it is at https://www.mkmapi.eu/ws/documentation/API_1.1:Stock
When run the class I get an error 417 Expectation Failed, which is described from the documentation as:
Typically you get a 417 Expectation Failed HTTP status code, when your request has an XML body without the corresponding header and/or the body not sent as text, but its byte representation. Another possible reason for a 417 is, when you send body data with more than 1.024 bytes without adding the header Expect: to your request.
Any help would be appreciated. I should also say that the authentication is not the problem I can download my article prices.
public void UpdateMarketPrice(string MarketID, string NewPrice)
{
// https://www.mkmapi.eu/ws/documentation/API_1.1:Stock
String finalResult;
String method = "PUT";
String url = "https://www.mkmapi.eu/ws/v1.1/stock";
HttpWebRequest request = WebRequest.CreateHttp(url) as HttpWebRequest;
OAuthHeader header = new OAuthHeader();
request.Headers.Add(HttpRequestHeader.Authorization, header.getAuthorizationHeader(method, url));
request.Method = method;
request.ContentType = "text/xml; encoding='utf-8'";
XElement xmlDoc =
new XElement("request",
new XElement("article",
new XElement("idArticle", MarketID),
new XElement("idLanguage", 1),
new XElement("comments", "Edited through the API"),
new XElement("count", 7),
new XElement("price", 11),
new XElement("condition", "NM"),
new XElement("isFoil", false),
new XElement("isSigned", false),
new XElement("isPlayset", false)
)
);
String finalXML = "<?xml version=\"1.0\" encoding=\"UTF-8\" ?>\n" + xmlDoc.ToString();
MessageBox.Show(finalXML);
byte[] bytes = Encoding.ASCII.GetBytes(finalXML);
request.ContentLength = bytes.Length;
using (Stream putStream = request.GetRequestStream())
{
putStream.Write(bytes, 0, bytes.Length);
}
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
finalResult = reader.ReadToEnd();
}
MessageBox.Show(finalResult);
}

I have read that HttpWebRequest adds an "expect 100 continue" header to requests unless you turn it off. There are servers that possibly don't support this header. And will produce this 417 Expectation Failed message.
You could try setting it to false:
System.Net.ServicePointManager.Expect100Continue = false;
So the header isn't sent.
I've seen this suggested sollution to other similar questions also.

Maybe use StreamWriter ?
using (Stream putStream = request.GetRequestStream())
{
using (var writeStream = new StreamWriter(putStream, Encoding.ASCII))
{
writeStream.Write(finalXML);
}
request.ContentLength = putStream.Length; // I am not sure about that
}

WP8 - Login to Website and parse HTML from answer

I want to login to this site: http://subcard.subway.co.uk/de_cardholder/JSP/login_reg.jsp
And i found out that I have to send a POST request to the server and I also know that I have to work with this POST request:
POST /de_cardholder/servlet/SPLoginServlet HTTP/1.1
Host: subcard.subway.co.uk
language=de&userID=ID&password=PASSWORD&transIdentType=1&programID=6
And after the login I want to parse the HTML data. But how can I implement the POST request on WP in C# and is it as easy as I think?

Try this,
Uri RequestUri = new Uri("subcard.subway.co.uk/de_cardholder/servlet/SPLoginServlet HTTP/1.1?language=de&userID=ID&password=PASSWORD&transIdentType=1&programID=6", UriKind.Absolute);
string PostData = "";
WebRequest webRequest;
webRequest = WebRequest.Create(RequestUri);
webRequest.Method = "POST";
webRequest.ContentType = "text/xml";
HttpWebResponse response;
string Response;
using (response = (HttpWebResponse)await webRequest.GetResponseAsync()) ;
using (Stream streamResponse = response.GetResponseStream())
using (StreamReader streamReader = new StreamReader(streamResponse))
{
Response = await streamReader.ReadToEndAsync();
}
if(Response != null)
{
//data should be available in Response
}

var postRequest = (HttpWebRequest)WebRequest.Create("your Url here");
postRequest.ContentType = "application/x-www-form-urlencoded";// you can give the type of request content here
postRequest.Method = "POST";
if you have any data to be posted along with the request as part of content and not as part of URL itself you can add this. example:- ( language=de&userID=ID&password=PASSWORD&transIdentType=1&programID=6)
using (var requestStream = await postRequest.GetRequestStreamAsync())
{
byte[] postDataArray = Encoding.UTF8.GetBytes("your request data");
await requestStream.WriteAsync(postDataArray, 0, postDataArray.Length);
}
if you do not have any data to be a send as content ignore the above code
WebResponse postResponse = await postRequest.GetResponseAsync();
if (postResponse != null)
{
var postResponseStream = postResponse.GetResponseStream();
var postStreamReader = new StreamReader(postResponseStream);
string response = await postStreamReader.ReadToEndAsync();// Result comes here
}

scrape site after login

I'm trying to scrape a website that requires a login. Getting an error that I haven't received before, copied the code from another forum successfully in the past:
Exception Details: System.Net.ProtocolViolationException: Cannot send a content-body with this verb-type.
with the code:
Stream newStream = http.GetRequestStream(); //open connection
Here's the entire code:
#{
var strUserId = "userName";
var strPassword = "password";
var url = "formSubmitLandingSite";
var url2 = "pageToScrape";
HttpWebRequest http = WebRequest.Create(url) as HttpWebRequest;
http.KeepAlive = true;
http.Method = "POST";
http.ContentType = "application/x-www-form-urlencoded";
string postData = "email=" + strUserId + "&password=" + strPassword;
byte[] dataBytes = UTF8Encoding.UTF8.GetBytes(postData);
http.ContentLength = dataBytes.Length;
using (Stream postStream = http.GetRequestStream())
{
postStream.Write(dataBytes, 0, dataBytes.Length);
}
HttpWebResponse httpResponse = http.GetResponse() as HttpWebResponse;
// Probably want to inspect the http.Headers here first
http = WebRequest.Create(url2) as HttpWebRequest;
http.CookieContainer = new CookieContainer();
http.CookieContainer.Add(httpResponse.Cookies);
HttpWebResponse httpResponse2 = http.GetResponse() as HttpWebResponse;
Stream newStream = http.GetRequestStream(); //open connection
newStream.Write(dataBytes, 0, dataBytes.Length); // Send the data.
newStream.Close();
string sourceCode;
HttpWebResponse getResponse = (HttpWebResponse)http.GetResponse();
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
sourceCode = sr.ReadToEnd();
}
Response.Write(sourceCode);
}

You're creating a new request object here:
http = WebRequest.Create(url2) as HttpWebRequest;
Keep in mind that the default HTTP verb used is GET. Then you try to open the request stream here:
Stream newStream = http.GetRequestStream();
This method is used to enable writing data to the request's content. However, GET requests don't have content. As you do in the code above the error, you'll need to use a different HTTP verb. POST is most common for this, and is what you're using above:
http.Method = "POST";
So just use a POST request again. (Assuming, of course, that's what the server is expecting. In any event, if the server is expecting content then it's definitely not expecting a GET request.)

HtmlAgilityPack Post Login

I'm trying to login to a site using HtmlAgilityPack (site:http://html-agility-pack.net).
Now, I can't exactly figure out how to go about this.
I've tried setting the Html form values via
m_HtmlDoc.DocumentNode.SelectSingleNode("//input[#name='EMAIL']").SetAttributeValue("value", "myemail.com");
I then submit the form with
m_HtmlWeb.Load("http://example.com/", "POST");
This isn't working though. It's not logging in or anything. Does anyone else have any other insight?
Thank you

The HTML Agility Pack is used to parse HTML - you cannot use it to submit forms. Your first line of code changes the parsed nodes in memory. The second line does not post the page to the server - it loads the DOM again, but using the POST method instead of the default GET.
It doesn't look like you need to parse the page at all at this point, since you already know the name of the control. Use the HttpWebRequest class to send a post request to the server, with the string email=acb#example.com in the request.
Here's a sample I wrote when I needed something similar:
/// <summary>
/// Append a url parameter to a string builder, url-encodes the value
/// </summary>
/// <param name="sb"></param>
/// <param name="name"></param>
/// <param name="value"></param>
protected void AppendParameter(StringBuilder sb, string name, string value)
{
string encodedValue = HttpUtility.UrlEncode(value);
sb.AppendFormat("{0}={1}&", name, encodedValue);
}
private void SendDataToService()
{
StringBuilder sb = new StringBuilder();
AppendParameter(sb, "email", "hello#example.com");
byte[] byteArray = Encoding.UTF8.GetBytes(sb.ToString());
string url = "http://example.com/"; //or: check where the form goes
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
//request.Credentials = CredentialCache.DefaultNetworkCredentials; // ??
using (Stream requestStream = request.GetRequestStream())
{
requestStream.Write(byteArray, 0, byteArray.Length);
}
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
// do something with response
}

If you want to do this with Html Agility Pack. Here's the code.
CookieCollection Cookies = new CookieCollection();
var web = new HtmlWeb();
web.OverrideEncoding = Encoding.Default;
web.UseCookies = true;
web.PreRequest += (request) =>
{
if (request.Method == "POST")
{
string payload = request.Address.Query;
byte[] buff = Encoding.UTF8.GetBytes(payload.ToCharArray());
request.ContentLength = buff.Length;
request.ContentType = "application/x-www-form-urlencoded";
System.IO.Stream reqStream = request.GetRequestStream();
reqStream.Write(buff, 0, buff.Length);
}
request.CookieContainer.Add(Cookies);
return true;
};
web.PostResponse += (request, response) =>
{
if (request.CookieContainer.Count > 0 || response.Cookies.Count > 0)
{
Cookies.Add(response.Cookies);
}
};
string baseUrl = "Your Website URL";
string urlToHit = baseUrl + "?QueryString with Login Credentials";
HtmlDocument doc = web.Load(urlToHit, "POST");

I spent some hours about this topic and found a very simple solution actually.
I have :
.net core 1.1.2
HttmlAgilityPack 1.4.9.5
login urlLogin: "www.url.com/login".
url for urlData: "www.url.com/data/3" => to get this you should be connected.
Here what I did and it just did work:
HttpClient hc = new HttpClient();
HttpResponseMessage resultLogin = await hc.PostAsync(urlLogin, new StringContent("login=myUserName&password=myPaswordValue", Encoding.UTF8, "application/x-www-form-urlencoded"));
HttpResponseMessage resultPlaylist = await hc.GetAsync(urlData);
Stream stream = await resultPlaylist.Content.ReadAsStreamAsync();
HtmlDocument doc = new HtmlDocument();
doc.Load(stream);
string webContent = doc.DocumentNode.InnerHtml; => it works
I think It needs to login first your HttpClient then you can send the request you want.
Enjoy