Develop Reference

C# reading data off html

I've been trying to see if I could get timetable data of a school website, and make a little application of it. At the moment this is what I have :
string userInput = "/*My username will be here*/";
string passInput = "/*My password will be here */";
string formUrl = "https://portal.gc.ac.nz/student/index.php/process-login";
string formParams = string.Format("username={0}&password={1}", userInput, passInput);
string cookieHeader;
WebRequest req = WebRequest.Create(formUrl);
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
string pageSource;
string getUrl = "https://portal.gc.ac.nz/student/index.php/timetable";
WebRequest getRequest = WebRequest.Create(getUrl);
getRequest.Headers.Add("Cookie", cookieHeader);
WebResponse getResponse = getRequest.GetResponse();
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
I couldn't find a way to check if above code works, however my question is:
How can you access the data(texts) you want from the page? I want to get the subject names. Part of the html looks like this :

There are a few ways to do this: one would be regexp matching and taking the contents of the tags and another would be to just use HtmlAgilityPack library.
If you don't need to do it in C# I would strongly recommend a different language like Python or Perl. It seems to me that you are trying to scrape the data and in this case I strongly recommend to use the Scrapy framework from Python if possible. It's the best tool I encountered for scraping and you can use XPath to get your data easily. Here is the link to Scrapy's website.

Non english characters in Kodi API

I am using the Kodi API, to control my htpc via asp.net.
Especialy the functio named "Playlist.Add".
The Json I send is like this:
{"jsonrpc":"2.0","method":"Playlist.Insert","params":{"playlistid":0,"position":0,"item":{"file":"smb://server/Ferry Corsten/Beautiful/Ferry Corsten - Beautiful (Extended).mp3"}},"id":1}
This is working fine. But when there are some none english characters in the string like this:
{"jsonrpc":"2.0","method":"Playlist.Insert","params":{"playlistid":0,"position":0,"item":{"file":"smb://server/01-Zum Geburtstag viel Glück.mp3"}},"id":1}
It is just throwing a "RequestCanceled" Exception.
My c# source is like this:
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(_url);
string authInfo = Convert.ToBase64String(Encoding.Default.GetBytes(_username + ":" + _password));
webRequest.Headers["Authorization"] = "Basic " + authInfo;
webRequest.Method = "POST";
webRequest.UserAgent = "KodiControl";
webRequest.ContentType = "application/json";
webRequest.ContentLength = json.Length;
using (var streamWriter = new StreamWriter(webRequest.GetRequestStream()))
{
streamWriter.Write(json);
streamWriter.Flush();
streamWriter.Close();
}
The Exception is thrown at streamWriter.Flush().
So what do I have to do to send this request?``

I suggest you look into Kodi addon unicode paths
Following that guide will help you prevent common problems with non-latin characters in Kodi.
Python only works with unicode strings internally, converting to a particular encoding on output. (or input)". To make string literals unicode by default, add
from __future__ import unicode_literals
Addon path
path = addon.getAddonInfo('path').decode('utf-8')
.decode('utf-8') tells kodi to decode the given function using utf-8.
Kodi's getAddonInfo returns an UTF-8 encoded string and we decode it an unicode.
Browse dialog
dialog = xbmcgui.Dialog()
directory = dialog.browse(0, 'Title' , 'pictures').decode('utf-8')
dialog.browse() returns an UTF-8 encoded string which perhaps contains some non latin characters. Therefore decode it to unicode!

Get Romaji from google translation website

i am trying to use the translation code bellow to get the romaji words for a specific set of japanese characters, but i can't get the romaji character to even show up the url i download, it's not even in the Google Translate page source code, this is my code:
string languagePair = "jp|en";
string url = String.Format("http://www.google.com/translate_t?hl=en&ie=UTF8&text={0}&langpair={1}", "本", languagePair);
WebClient webClient = new WebClient();
webClient.Encoding = Encoding.UTF8;
string result = webClient.DownloadString(url);
Clipboard.SetText(result);
the character in my code is just an example, it's supposed to say Hon.

For japanese language you must use ja ISO 639-1 Code as described here:
Notes:
1. the language pairs are listed in this FAQ, while the language codes are included in this long list.
So, you must change your code to this:
string languagePair = "ja|en";
string url = String.Format("http://www.google.com/translate_t?hl=en&ie=UTF8&text={0}&langpair={1}", "本", languagePair);
WebClient webClient = new WebClient();
webClient.Encoding = Encoding.UTF8;
string result = webClient.DownloadString(url);
Clipboard.SetText(result);
Result page:

C# Escape Plus Sign (+) in POST using HttpWebRequest

I'm having issues to send POST data that includes characteres like "+" in the password field,
string postData = String.Format("username={0}&password={1}", "anyname", "+13Gt2");
I'm using HttpWebRequest and a webbrowser to see the results, and when I try to log in from my C# WinForms using HttpWebRequest to POST data to the website, it tells me that password is incorrect. (in the source code[richTexbox1] and the webBrowser1). Trying it with another account of mine, that does not contain '+' character, it lets me log in correctly (using array of bytes and writing it to the stream)
byte[] byteArray = Encoding.ASCII.GetBytes(postData); //get the data
request.Method = "POST";
request.Accept = "text/html";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = byteArray.Length;
Stream newStream = request.GetRequestStream(); //open connection
newStream.Write(byteArray, 0, byteArray.Length); // Send the data.
newStream.Close(); //this works well if user does not includes symbols
From this Question I found that HttpUtility.UrlEncode() is the solution to escape illegal characters, but I can't find out how to use it correctly, my question is, after url-encoding my POST data with urlEncode() how do I send the data to my request correctly?
This is how I've been trying for HOURS to make it work, but no luck,
First method
string urlEncoded = HttpUtility.UrlEncode(postData, ASCIIEncoding.ASCII);
//request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = urlEncoded.Length;
StreamWriter wr = new StreamWriter(request.GetRequestStream(),ASCIIEncoding.ASCII);
wr.Write(urlEncoded); //server returns for wrong password.
wr.Close();
Second method
byte[] urlEncodedArray = HttpUtility.UrlEncodeToBytes(postData,ASCIIEncoding.ASCII);
Stream newStream = request.GetRequestStream(); //open connection
newStream.Write(urlEncodedArray, 0, urlEncodedArray.Length); // Send the data.
newStream.Close(); //The server tells me the same thing..
I think I'm doing wrong on how the url-encoded must be sent to the request, I really ask for some help please, I searched through google and couldn't find more info about how to send encoded url to an HttpWebRequest.. I appreciate your time and attention, hope you can help me. Thank you.

I found my answer using Uri.EscapeDataString it exactly solved my problem, but somehow I couldn't do it with HttpUtility.UrlEncode. Stackoverflowing around, I found this question that is about urlEncode method, in msdn it documentation tells that:
Encodes a URL string.
But I know now that is wrong if used to encode POST data (#Marvin, #Polity, thanks for the correction). After discarding it, I tried the following:
string postData = String.Format("username={0}&password={1}", "anyname", Uri.EscapeDataString("+13Gt2"));
The POST data is converted into:
// **Output
string username = anyname;
string password = %2B13Gt2;
Uri.EscapeDataString in msdn says the following:
Converts a string to its escaped representation.
I think this what I was looking for, when I tried the above, I could POST correctly whenever there's data including the '+' characters in the formdata, but somehow there's much to learn about them.
This link is really helpful.
http://blogs.msdn.com/b/yangxind/archive/2006/11/09/don-t-use-net-system-uri-unescapedatastring-in-url-decoding.aspx
Thanks a lot for the answers and your time, I appreciate it very much. Regards mates.

I'd recommend you a WebClient. Will shorten your code and take care of encoding and stuff:
using (var client = new WebClient())
{
var values = new NameValueCollection
{
{ "username", "anyname" },
{ "password", "+13Gt2" },
};
var url = "http://foo.com";
var result = client.UploadValues(url, values);
}

the postdata you are sending should NOT be URL encoded! it's formdata, not the URL
string url = #"http://localhost/WebApp/AddService.asmx/Add";
string postData = "x=6&y=8";
WebRequest req = WebRequest.Create(url);
HttpWebRequest httpReq = (HttpWebRequest)req;
httpReq.Method = WebRequestMethods.Http.Post;
httpReq.ContentType = "application/x-www-form-urlencoded";
Stream s = httpReq.GetRequestStream();
StreamWriter sw = new StreamWriter(s,Encoding.ASCII);
sw.Write(postData);
sw.Close();
HttpWebResponse httpResp =
(HttpWebResponse)httpReq.GetResponse();
s = httpResp.GetResponseStream();
StreamReader sr = new StreamReader(s, Encoding.ASCII);
Console.WriteLine(sr.ReadToEnd());
This uses the System.Text.Encoding.ASCII to encode the postdata.
Hope this helps,

"The format of the URI could not be determined" with WebRequest

I'm trying to perform a POST to a site using a WebRequest in C#. The site I'm posting to is an SMS site, and the messagetext is part of the URL. To avoid spaces in the URL I'm calling HttpUtility.Encode() to URL encode it.
But I keep getting an URIFormatException - "Invalid URI: The format of the URI could not be determined" - when I use code similar to this:
string url = "http://www.stackoverflow.com?question=a sentence with spaces";
string encoded = HttpUtility.UrlEncode(url);
WebRequest r = WebRequest.Create(encoded);
r.Method = "POST";
r.ContentLength = encoded.Length;
WebResponse response = r.GetResponse();
The exception occurs when I call WebRequest.Create().
What am I doing wrong?

You should only encode the argument, not the entire url, so try:
string url = "http://www.stackoverflow.com?question=" + HttpUtility.UrlEncode("a sentence with spaces");
WebRequest r = WebRequest.Create(url);
r.Method = "POST";
r.ContentLength = encoded.Length;
WebResponse response = r.GetResponse();
Encoding the entire url would mean the :// and the ? get encoded too. The encoded string is then no longer a valid url.

UrlEncode should only be used on the query string. Try this:
string query = "a sentence with spaces";
string encoded = "http://www.stackoverflow.com/?question=" + HttpUtility.UrlEncode(query);
The current version of your code is urlencoding the slashes and colon in the URL, which is confusing webrequest.