Develop Reference

C# reading data off html

I've been trying to see if I could get timetable data of a school website, and make a little application of it. At the moment this is what I have :
string userInput = "/*My username will be here*/";
string passInput = "/*My password will be here */";
string formUrl = "https://portal.gc.ac.nz/student/index.php/process-login";
string formParams = string.Format("username={0}&password={1}", userInput, passInput);
string cookieHeader;
WebRequest req = WebRequest.Create(formUrl);
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
string pageSource;
string getUrl = "https://portal.gc.ac.nz/student/index.php/timetable";
WebRequest getRequest = WebRequest.Create(getUrl);
getRequest.Headers.Add("Cookie", cookieHeader);
WebResponse getResponse = getRequest.GetResponse();
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
I couldn't find a way to check if above code works, however my question is:
How can you access the data(texts) you want from the page? I want to get the subject names. Part of the html looks like this :

There are a few ways to do this: one would be regexp matching and taking the contents of the tags and another would be to just use HtmlAgilityPack library.
If you don't need to do it in C# I would strongly recommend a different language like Python or Perl. It seems to me that you are trying to scrape the data and in this case I strongly recommend to use the Scrapy framework from Python if possible. It's the best tool I encountered for scraping and you can use XPath to get your data easily. Here is the link to Scrapy's website.

C# Web API REST Service POST

I originally asked a question regarding a WCF web service that I was trying to write and then found that the ASP.net web API was more appropriate to my needs, due to some feedback on here.
I've now found a good tutorial that tells me how to create a simple REST service using Web API which works well pretty much out of the box.
My question
I have a POST method in my REST service server:
// POST api/values/5
public string Post([FromBody]string value)
{
return "Putting value: " + value;
}
I can POST to this using POSTER and also my C# client code.
However the bit I don't understand is why I have to prepend an '=' sign to the POST data so that it reads: "=Here is my data which is actually a JSON string"; rather than just sending: "Here is my data which is actually a JSON string";
My C# Client that talks to the REST service is written as follows:
public string SendPOSTRequest(string sFunction, string sData)
{
string sResponse = string.Empty;
// Create the request string using the data provided
Uri uriRequest = GetFormRequest(m_sWebServiceURL, sFunction, string.Empty);
// Data to post
string sPostData = "=" + sData;
// The Http Request obj
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uriRequest);
request.Method = m_VERB_POST;
System.Text.UTF8Encoding encoding = new System.Text.UTF8Encoding();
Byte[] byteArray = encoding.GetBytes(sPostData);
request.ContentLength = byteArray.Length;
request.ContentType = m_APPLICATION_FORM_URLENCODED;
try
{
using (Stream dataStream = request.GetRequestStream())
{
dataStream.Write(byteArray, 0, byteArray.Length);
}
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
StreamReader reader = new StreamReader(stream, Encoding.UTF8);
sResponse = reader.ReadToEnd();
}
}
}
catch (WebException ex)
{
//Log exception
}
return sResponse;
}
private static Uri GetFormRequest(string sURL, string sFunction, string sParam)
{
StringBuilder sbRequest = new StringBuilder();
sbRequest.Append(sURL);
if ((!sURL.EndsWith("/") &&
(!string.IsNullOrEmpty(sFunction))))
{
sbRequest.Append("/");
}
sbRequest.Append(sFunction);
if ((!sFunction.EndsWith("/") &&
(!string.IsNullOrEmpty(sParam))))
{
sbRequest.Append("/");
}
sbRequest.Append(sParam);
return new Uri(sbRequest.ToString());
}
Is anybody able to explain why I have to prepend the '=' sign as in the above code (string sPostData = "=" + sData;)?
Many thanks in advance!

The content type x-www-form-urlencoded is a key-value format. With form bodies you are only able to read a single simple type from a request body. As a name is expected, but in this case not allowed, you have to prefix the equal sign to indicate that there is no name with the followed value.
However, you should lean away from accepting simple types within the body of your web-api controller actions.
You are limited to only a single simple type if you attempt to pass data in the body of an HttpPost/HttpPut without directly implementing your own MediaTypeFormatter, which is unlikely to be reliable. Creating a light-weight complex type is generally much more preferable, and will make interoperating with other content-types, like application/json, much easier.

RESTful WCF receiving Base64 message. Need to convert message back to XML

I am having some trouble trying to write code within a RESTful WCF service. I have made a method available to a calling client application and I am receiving a message that is of the format Ax27834...... which is a Base64 Binary message. The issue is that following receiving this I need to be able to convert it back to the original xml version of that message that was sent from the client. How can I achieve this in the code snippet below. On line 6 below you will see where the code needs to go. I have searched for a solution but not managed to find anything suitable. I have to receive a message rather than a stream.
I should highlight that the service works fine in the respect of receiving the request. I am just struggling to get the message into a form that I can use.
The receiving code
public Message StoreMessage(Message request)
{
//Store the message
try
{
string message = [NEED SOLUTION HERE]
myClass.StoreNoticeInSchema(message, DateTime.Now);
}
catch (Exception e)
{
log4net.Config.XmlConfigurator.Configure();
ILog log = LogManager.GetLogger(typeof(Service1));
if (log.IsErrorEnabled)
{
log.Error(String.Format("{0}: Notice was not stored. {1} reported an exception. {2}", DateTime.Now, System.Reflection.MethodBase.GetCurrentMethod().DeclaringType, e.Message));
}
}
XElement responseElement = new XElement(XName.Get("elementName", "url"));
XDocument resultDocument = new XDocument(responseElement);
return Message.CreateMessage(OperationContext.Current.IncomingMessageVersion, "elementName", resultDocument.CreateReader());
}
The client code
public string CallPostMethod()
{
const string action = "StoreNotice/New";
TestNotice testNotice = new TestNotice();
const string url = "http://myaddress:myport/myService.svc/StoreNotice/New";
string contentType = String.Format("application/soap+xml; charset=utf-8; action=\"{0}\"", action);
string xmlString = CreateSoapMessage(url, action, testNotice.NoticeText);
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
ASCIIEncoding encoding = new ASCIIEncoding();
byte[] bytesToSend = encoding.GetBytes(xmlString);
request.Method = "POST";
request.ContentLength = bytesToSend.Length;
request.ContentType = contentType;
using (Stream requestStream = request.GetRequestStream())
{
requestStream.Write(bytesToSend, 0, bytesToSend.Length);
requestStream.Close();
}
string responseFromServer;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
using (Stream dataStream = response.GetResponseStream())
{
using (StreamReader reader = new StreamReader(dataStream))
responseFromServer = reader.ReadToEnd();
dataStream.Close();
}
XDocument document = XDocument.Parse(responseFromServer);
string nameSpace = "http://www.w3.org/2003/05/soap-envelope";
XElement responseElement = document.Root.Element(XName.Get("Body", nameSpace))
.Element(XName.Get(#action + "Response", "http://www.wrcplc.co.uk/Schemas/ETON"));
return responseElement.ToString();
}
Code to create SOAP message
protected string CreateSoapMessage(string url, string action, string messageContent)
{
return String.Format(
#"<?xml version=""1.0"" encoding=""utf-8""?>
<soap12:Envelope xmlns:xsi=""http://www.w3.org/2001/XMLSchema-instance""
xmlns:xsd=""http://www.w3.org/2001/XMLSchema"" xmlns:soap12=""http://www.w3.org/2003/05/soap-envelope""><soap12:Body>{0}</soap12:Body>
</soap12:Envelope>
", messageContent, action, url);
}
NOTE: The TestNotice() object contains a large xml string which is the body of the message.

With a Message object you usually use GetReaderAtBodyContents() to get an XML representation of the body content, unless you know what type the body has then you can use GetBody<>. Try using those to get the string, and then decode it if you still need to. Which you can do as follows:
byte[] encodedMessageAsBytes = System.Convert.FromBase64String(requestString);
string message = System.Text.Encoding.Unicode.GetString(encodedMessageAsBytes);
From there you can reconstruct the xml from the string
Edit: to answer the last part from the comment, the content type should be: text/xml

C# Escape Plus Sign (+) in POST using HttpWebRequest

I'm having issues to send POST data that includes characteres like "+" in the password field,
string postData = String.Format("username={0}&password={1}", "anyname", "+13Gt2");
I'm using HttpWebRequest and a webbrowser to see the results, and when I try to log in from my C# WinForms using HttpWebRequest to POST data to the website, it tells me that password is incorrect. (in the source code[richTexbox1] and the webBrowser1). Trying it with another account of mine, that does not contain '+' character, it lets me log in correctly (using array of bytes and writing it to the stream)
byte[] byteArray = Encoding.ASCII.GetBytes(postData); //get the data
request.Method = "POST";
request.Accept = "text/html";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = byteArray.Length;
Stream newStream = request.GetRequestStream(); //open connection
newStream.Write(byteArray, 0, byteArray.Length); // Send the data.
newStream.Close(); //this works well if user does not includes symbols
From this Question I found that HttpUtility.UrlEncode() is the solution to escape illegal characters, but I can't find out how to use it correctly, my question is, after url-encoding my POST data with urlEncode() how do I send the data to my request correctly?
This is how I've been trying for HOURS to make it work, but no luck,
First method
string urlEncoded = HttpUtility.UrlEncode(postData, ASCIIEncoding.ASCII);
//request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = urlEncoded.Length;
StreamWriter wr = new StreamWriter(request.GetRequestStream(),ASCIIEncoding.ASCII);
wr.Write(urlEncoded); //server returns for wrong password.
wr.Close();
Second method
byte[] urlEncodedArray = HttpUtility.UrlEncodeToBytes(postData,ASCIIEncoding.ASCII);
Stream newStream = request.GetRequestStream(); //open connection
newStream.Write(urlEncodedArray, 0, urlEncodedArray.Length); // Send the data.
newStream.Close(); //The server tells me the same thing..
I think I'm doing wrong on how the url-encoded must be sent to the request, I really ask for some help please, I searched through google and couldn't find more info about how to send encoded url to an HttpWebRequest.. I appreciate your time and attention, hope you can help me. Thank you.

I found my answer using Uri.EscapeDataString it exactly solved my problem, but somehow I couldn't do it with HttpUtility.UrlEncode. Stackoverflowing around, I found this question that is about urlEncode method, in msdn it documentation tells that:
Encodes a URL string.
But I know now that is wrong if used to encode POST data (#Marvin, #Polity, thanks for the correction). After discarding it, I tried the following:
string postData = String.Format("username={0}&password={1}", "anyname", Uri.EscapeDataString("+13Gt2"));
The POST data is converted into:
// **Output
string username = anyname;
string password = %2B13Gt2;
Uri.EscapeDataString in msdn says the following:
Converts a string to its escaped representation.
I think this what I was looking for, when I tried the above, I could POST correctly whenever there's data including the '+' characters in the formdata, but somehow there's much to learn about them.
This link is really helpful.
http://blogs.msdn.com/b/yangxind/archive/2006/11/09/don-t-use-net-system-uri-unescapedatastring-in-url-decoding.aspx
Thanks a lot for the answers and your time, I appreciate it very much. Regards mates.

I'd recommend you a WebClient. Will shorten your code and take care of encoding and stuff:
using (var client = new WebClient())
{
var values = new NameValueCollection
{
{ "username", "anyname" },
{ "password", "+13Gt2" },
};
var url = "http://foo.com";
var result = client.UploadValues(url, values);
}

the postdata you are sending should NOT be URL encoded! it's formdata, not the URL
string url = #"http://localhost/WebApp/AddService.asmx/Add";
string postData = "x=6&y=8";
WebRequest req = WebRequest.Create(url);
HttpWebRequest httpReq = (HttpWebRequest)req;
httpReq.Method = WebRequestMethods.Http.Post;
httpReq.ContentType = "application/x-www-form-urlencoded";
Stream s = httpReq.GetRequestStream();
StreamWriter sw = new StreamWriter(s,Encoding.ASCII);
sw.Write(postData);
sw.Close();
HttpWebResponse httpResp =
(HttpWebResponse)httpReq.GetResponse();
s = httpResp.GetResponseStream();
StreamReader sr = new StreamReader(s, Encoding.ASCII);
Console.WriteLine(sr.ReadToEnd());
This uses the System.Text.Encoding.ASCII to encode the postdata.
Hope this helps,

What is the correct way in .NET to encode an XML string for Posting to a URL?

For the first time, I am having to use a vendor's API by posting an XML dataset to a URL. If I encode the dataset using http://meyerweb.com/eric/tools/dencoder/, and manually paste the URL & dataset into a browser, I get a successful response.
However, when trying to do the same thing with .NET, I get a response from the vendor API indicating that my dataset is invalid.
Here's my code (which I copied from elsewhere on the good ol' interweb):
public void PostXML(string value1, string value2, string value3, out string responseStatus, out string responseBody)
{
// Create a request using a URL that can receive a post.
WebRequest request = WebRequest.Create("https://www.vendorURL.com/API/page.html?input=");
// Set the Method property of the request to POST.
request.Method = "POST";
// Create POST data and convert it to a byte array.
string postData = "<DATASET>";
postData += "<FIELD1>" + value1+ "</FIELD1>";
postData += "<FIELD2>" + value2 + "</FIELD2>";
postData += "<FIELD3>" + value3 + "</FIELD3>";
postData += "</DATASET>";
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
// Set the ContentType property of the WebRequest.
request.ContentType = "application/x-www-form-urlencoded";
// Set the ContentLength property of the WebRequest.
request.ContentLength = byteArray.Length;
// Get the request stream.
Stream dataStream = request.GetRequestStream();
// Write the data to the request stream.
dataStream.Write(byteArray, 0, byteArray.Length);
// Close the Stream object.
dataStream.Close();
// Get the response.
WebResponse response = request.GetResponse();
// Assign the status.
responseStatus = ((HttpWebResponse)response).StatusDescription;
// Get the stream containing content returned by the server.
dataStream = response.GetResponseStream();
// Open the stream using a StreamReader for easy access.
StreamReader reader = new StreamReader(dataStream);
// Read the content.
responseBody = reader.ReadToEnd();
// Clean up the streams.
reader.Close();
dataStream.Close();
response.Close();
}
Keep in mind, I am getting a response from the vendor API. It's just telling me that my dataset is invalid. I have already contacted the vendor's customer support. That have confirmed that my dataset itself is correct, but somewhere along the way I'm encoding it incorrectly for it to be recognizable by the vendor API.

Several things are missing.
You should probably start with an xml declaration
<?xml version="1.0"?>
Also are you sure the dataset tags are upper case? Case is significant in Xml.
<?xml version="1.0"?>
<dataset><field>value</field></dataset>
Additionally you might try XmlWriter:
using(var writer = XmlWriter.Create(stream))
{
writer.WriteStartDocument();
writer.WriteStartElement("dataset");
writer.WriteElementString("field1", "value");
writer.WriteElementString("field2", "value");
writer.WriteElementString("field3", "value");
writer.WriteEndElement();
writer.WriteEndDocument();
}
Finally contact the vendor to send you sample code/API...

Try using System.Web.HttpUtility.UrlEncode on your post data before invoking "Encoding.UTF8.GetBytes(postData);".