Develop Reference

C# reading data off html

I've been trying to see if I could get timetable data of a school website, and make a little application of it. At the moment this is what I have :
string userInput = "/*My username will be here*/";
string passInput = "/*My password will be here */";
string formUrl = "https://portal.gc.ac.nz/student/index.php/process-login";
string formParams = string.Format("username={0}&password={1}", userInput, passInput);
string cookieHeader;
WebRequest req = WebRequest.Create(formUrl);
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
string pageSource;
string getUrl = "https://portal.gc.ac.nz/student/index.php/timetable";
WebRequest getRequest = WebRequest.Create(getUrl);
getRequest.Headers.Add("Cookie", cookieHeader);
WebResponse getResponse = getRequest.GetResponse();
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
I couldn't find a way to check if above code works, however my question is:
How can you access the data(texts) you want from the page? I want to get the subject names. Part of the html looks like this :

There are a few ways to do this: one would be regexp matching and taking the contents of the tags and another would be to just use HtmlAgilityPack library.
If you don't need to do it in C# I would strongly recommend a different language like Python or Perl. It seems to me that you are trying to scrape the data and in this case I strongly recommend to use the Scrapy framework from Python if possible. It's the best tool I encountered for scraping and you can use XPath to get your data easily. Here is the link to Scrapy's website.

Problems encoding string data that contains percent sign

I'm working on a project where I have to send product information via HTTP POST in XML string to a web server. It turns out that certain product names may have a % sign in their name, such as ".05% Topical Cream". Whenever I attempt to send XML data that contained a % sign in the product name, I get an error apparently because when encoding the XML string data the percent sign caused the data to become malformed.
How can I encode and send the XML string data with % sign in product name safely?
XML Data:
<node>
<product>
<BrandName>amlodipine besylate (bulk) 100 % Powder</BrandName>
</product>
</node>
Web request code:
public string MakeWebServerRequest(string url, string data)
{
var parms = System.Web.HttpUtility.UrlEncode(data);
byte[] bytes = Encoding.UTF8.GetBytes("xml=" + parms);
string webResponse = String.Empty;
try
{
System.Web.HttpUtility.UrlEncode(data);
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
req.ContentLength = bytes.Length;
using (Stream reqStream = req.GetRequestStream())
{
reqStream.WriteTimeout = 3000;
reqStream.Write(bytes, 0, bytes.Length);
reqStream.Close();
}
using (HttpWebResponse response = (HttpWebResponse)req.GetResponse())
{
using (StreamReader rdr = new StreamReader(response.GetResponseStream()))
{
webResponse = rdr.ReadToEnd();
rdr.Close();
}
response.Close();
}
}
Should I be creating the web request differently? What can I do to resolve, while maintaining the product name?
Corrected - and working now. Thanks
Thanks

You need to construct request propely. application/x-www-form-urlencoded means that each parameter is Url-encoded. In your case xml parameter must have value correctly encoded, not just blindly concatenated. Below is sample that should give you stared... hopefully you'll be able to avoid string concateneation to construct XML (and insane way of constructing string constant with queotes you have in original code):
var parameterValue = System.Web.HttpUtility.UrlEncode("<xml>" + data);
byte[] bytes = Encoding.UTF8.GetBytes("xml=" + parameterValue);
There are also plenty of samples how to correctly construct requests of this kind. I.e. C# web request with POST encoding question

How to escape URL-encoded data in POST with HttpWebRequest

I am trying to send an URL-encoded post to a REST API implemented in PHP. The POST data contains two user-provided strings:
WebRequest request = HttpWebRequest.Create(new Uri(serverUri, "rest"));
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
request.Headers.Add("Content-Transfer-Encoding", "binary");
// Form the url-encoded credentials we'll use to log in
StringBuilder builder = new StringBuilder();
builder.Append("user=");
builder.Append(user);
builder.Append("&password=");
builder.Append(password);
byte[] credentials = Encoding.UTF8.GetBytes(builder.ToString());
// Write the url-encoded post data into the request stream.
request.ContentLength = credentials.Length;
using (Stream requestStream = request.GetRequestStream()) {
requestStream.Write(credentials, 0, credentials.Length);
}
This sends a HTTP request to the server containing user=myusername&password=mypassword in UTF-8 as its POST data.
How can I escape the user-provided strings?
For example, if I had a user named big&mean, how should the ampersand be escaped so that it does not mess up the request line?

You can use the static HttpUtility class in System.Web for encoding and decoding HTML and Url related values.
Try HttpUtility.UrlEncode().

It would seem that System.Web is obsolete - the newer way to access it is System.Net.WebUtility.HtmlEncode

Problems consuming WebService in .Net (ReCaptcha)

I am having difficulty in consuming the reCaptcha Web Service using C#/.Net 3.5. Although I think the problem is with consuming web services in general.
String validate = String.Format("http://api-verify.recaptcha.net/verify?privatekey={0}&remoteip={1}&challenge={2}&response={3}", PrivateKey, UserIP, Challenge, Response);
WebClient serviceRequest = new WebClient();
serviceRequest.Headers.Add("ContentType","application/x-www-form-urlencoded")
String response = serviceRequest.DownloadString(new Uri(validate ));
It keeps telling me that the error is: nverify-params-incorrect. Which means:
The parameters to /verify were incorrect, make sure you are passing all the required parameters.
But it's correct. I am using the private key, the IP address (locally) is 127.0.0.1, and the challenge and response seem fine. However the error keeps occurring.
I am pretty sure this is a issue with how I am requesting the service as this is the first time I have actually used webservices and .Net.
I also tried this as it ensures the data is posted:
String queryString = String.Format("privatekey={0}&remoteip={1}&challenge={2}&response={3}",PrivateKey, UserIP, Challenge, Response);
String Validate = "http://api-verify.recaptcha.net/verify" + queryString;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(new Uri(Validate));
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = Validate.Length;
**HttpWebResponse captchaResponse = (HttpWebResponse)request.GetResponse();**
String response;
using (StreamReader reader = new StreamReader(captchaResponse.GetResponseStream()))
response = reader.ReadToEnd();
Seems to stall at the point where I get response.
Any advice?
Thanks in advance

Haven't worked with the recaptcha service previously, but I have two troubleshooting recommendations:
Use Fiddler or Firebug and watch what you're sending outbound. Verifying your parameters would help you with basic troubleshooting, i.e. invalid characters, etc.
The Recaptcha Wiki has an entry about dealing with development on Vista. It doesn't have to be limited to Vista, though; if you're system can handle IPv6, then your browser could be communicating in that format as a default. It appears as if Recaptcha deals with IPv4. Having Fiddler/Firebug working would tell you about those other parameters that could be causing you grief.
This may not help solve your problem but it might provide you with better troubleshooting info.

So got this working, for some reason I needed to write the request to a stream like so:
//Write data to request stream
using (Stream requestSteam = request.GetRequestStream())
requestSteam.Write(byteData, 0, byteData.Length);
Could anyone explain why this works. I didn't think I would need to do this, don't completely understand what's happening behind the scenes..

Damien's answer is correct of course, but just to be clear about the order of things (I was a little confused) and to have a complete code sample...
var uri = new Uri("http://api-verify.recaptcha.net/verify");
var queryString = string.Format(
"privatekey={0}&remoteip={1}&challenge={2}&response={3}",
privateKey,
userIP,
challenge,
response);
var request = (HttpWebRequest)HttpWebRequest.Create(uri);
request.Method = WebRequestMethods.Http.Post;
request.ContentLength = queryString.Length;
request.ContentType = "application/x-www-form-urlencoded";
using (var writer = new StreamWriter(request.GetRequestStream()))
{
writer.Write(queryString);
}
string result;
using (var webResponse = (HttpWebResponse)request.GetResponse())
{
var reader = new StreamReader(webResponse.GetResponseStream());
result = reader.ReadToEnd();
}
There's a slight difference in that I'm writing the post variables to the request, but the core of it is the same.

C# to PHP base64 encode/decode

So I have this c# application that needs to ping my web server thats running linux/php stack.
I am having problems with the c# way of base 64 encoding bytes.
my c# code is like:
byte[] encbuff = System.Text.Encoding.UTF8.GetBytes("the string");
String enc = Convert.ToBase64String(encbuff);
and php side:
$data = $_REQUEST['in'];
$raw = base64_decode($data);
with larger strings 100+ chars it fails.
I think this is due to c# adding '+'s in the encoding but not sure.
any clues

You should probably URL Encode your Base64 string on the C# side before you send it.
And URL Decode it on the php side prior to base64 decoding it.
C# side
byte[] encbuff = System.Text.Encoding.UTF8.GetBytes("the string");
string enc = Convert.ToBase64String(encbuff);
string urlenc = Server.UrlEncode(enc);
and php side:
$data = $_REQUEST['in'];
$decdata = urldecode($data);
$raw = base64_decode($decdata);

Note that + is a valid character in base64 encoding, but when used in URLs it is often translated back to a space. This space may be confusing your PHP base64_decode function.
You have two approaches to solving this problem:
Use %-encoding to encode the + character before it leaves your C# application.
In your PHP application, translate space characters back to + before passing to base64_decode.
The first option is probably your better choice.

This seems to work , replacing + with %2B...
private string HTTPPost(string URL, Dictionary<string, string> FormData)
{
UTF8Encoding UTF8encoding = new UTF8Encoding();
string postData = "";
foreach (KeyValuePair<String, String> entry in FormData)
{
postData += entry.Key + "=" + entry.Value + "&";
}
postData = postData.Remove(postData.Length - 1);
//urlencode replace (+) with (%2B) so it will not be changed to space ( )
postData = postData.Replace("+", "%2B");
byte[] data = UTF8encoding.GetBytes(postData);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = data.Length;
Stream strm = request.GetRequestStream();
// Send the data.
strm.Write(data, 0, data.Length);
strm.Close();
WebResponse rsp = null;
// Send the data to the webserver
rsp = request.GetResponse();
StreamReader rspStream = new StreamReader(rsp.GetResponseStream());
string response = rspStream.ReadToEnd();
return response;
}

Convert.ToBase64String doesn't seem to add anything extra as far as I can see. For instance:
byte[] bytes = new byte[1000];
Console.WriteLine(Convert.ToBase64String(bytes));
The above code prints out a load of AAAAs with == at the end, which is correct.
My guess is that $data on the PHP side doesn't contain what enc did on the C# side - check them against each other.

in c#
this is a <B>long</b>string. and lets make this a3214 ad0-3214 0czcx 909340 zxci 0324#$##$%%13244513123
turns into
dGhpcyBpcyBhIDxCPmxvbmc8L2I+c3RyaW5nLiBhbmQgbGV0cyBtYWtlIHRoaXMgYTMyMTQgYWQwLTMyMTQgMGN6Y3ggOTA5MzQwIHp4Y2kgMDMyNCMkQCMkJSUxMzI0NDUxMzEyMw==
for me. and i think that + is breaking it all.

The PHP side should be:
$data = $_REQUEST['in'];
// $decdata = urldecode($data);
$raw = base64_decode($decdata);
The $_REQUEST should already be URLdecoded.