Develop Reference

C# HttpWebRequest removing any encoding on an URL

I'm trying to send percent-encoded ASCII characters using HttpWebRequest, for example https://example.com/%74%65%73%74
Using POSTMAN I'm able to send a request and see in charles that it's still encoded, however, regardless of what I do, any C# library will simplify this to https://example.com/test.
Is there any workaround or async compatible library to work around this?
Thanks in advance
Example Code
string newURL = "";
string path = "test";
foreach (char character in path)
{
string charac = "";
string finished = "";
if (character != '-' || character != '/')
{
if (random.Next(1, 10) >= 5)
{
charac = "%" + Convert.ToInt32(character).ToString("X");
}
else
{
charac = character.ToString();
}
}
newURL = newURL + charac;
}
string main = "http://www.example.com";
HttpWebRequest request = HttpWebRequest.CreateHttp(main + "/en/" + newURL);

code working well when i put a breakpoint on getresponsedata, but without a breakpoint it does not work

//webservice connection
CRB_WEBSERVICE.CBS_SERVICE.BatchPublicServiceClient proxy = new CRB_WEBSERVICE.CBS_SERVICE.BatchPublicServiceClient("BasicHttpBinding_IBatchPublicService");
proxy.ClientCredentials.UserName.UserName = "";
proxy.ClientCredentials.UserName.Password = "";
string currdte = DateTime.Now.ToString("yyyyMMdd");
//Open Connection for webservice
proxy.Open();
//batchid is a string requested in format the client wants
string batchid = "" + currdte + "_" + "D" + "_" + "**" + "_" + num + "_" + "XML" + "_" + "T";
//generatexml is generating an XML file which is then zipped and sent via //webservice
GenerateXML(batchid);
try
{
//process to send zipped file via webservice
byte[] data = File.ReadAllBytes(batchid + ".zip");
proxy.Begin(batchid);
proxy.PutData(batchid, 1, data);
proxy.Finish(batchid, 1);
var b = proxy.GetBatchInfo(batchid);
// thread so it waits for the response
Thread.Sleep(11000);
// this is where my issue is.if i put a breakpoint i get a response of a //zipped file. if i dont nothing happens
**var result = proxy.GetResponseData(batchid, 1);**
//Thread.Sleep(100);
System.IO.File.WriteAllBytes(#"C:\CBS_WEBSERVICE\Response.zip", result);
}
Please note this is not the whole code. I need assistance as my code is running perfectly when I put a breakpoint on the GetResponseData but if I do not it does not bring a response. I tried adding thread.sleep so it waits for a response from the webservice but still no luck. I have added comments.

I was supposed to check the first parameter: BatchStatus before calling the method GetDataResponse. I read the documentation of the API again. Without the batchstatus GetDataResponse will be null hence code was hanging in release mode and only working in debug mode.
var b = proxy.GetBatchInfo(batchid);
while (b.BatchStatus != CRB_WEBSERVICE.CBS_SERVICE.BatchStatuses.Finished)
{
b = proxy.GetBatchInfo(batchid);
}
if (b.BatchStatus == CRB_WEBSERVICE.CBS_SERVICE.BatchStatuses.Finished)
{
if (b.BatchResult == CRB_WEBSERVICE.CBS_SERVICE.BatchResults.Success)
{
var result = proxy.GetResponseData(batchid, 1);
}
}

Unity3D XML(-RPC) and C#

I'm actually answering my own question here.
I must be the only person in the world who tried to do this but given that it has taken me about a week to work this out - I figured that if there is ever another person who wants to use XML(-RPC) in Unity - I'll save them a weeks hassle.
What I wanted to do is talk to one of our Game servers for things like leaderboards. This server "talks" XML-RPC and I soon figured out that that's not easy in Unity.

Build XML to send to our servers
I couldn't find a standard function in Unity to do this without adding very large amounts of overhead. So I build the following procedure instead.
public string buildXMLRPCRequest(Hashtable FieldArray,string MethodName)
{
string ReturnString = "";
ReturnString += "<?xml version=\"1.0\" encoding=\"iso-8859-1\"?>" +
"\n" + "<simpleRPC version=\"0.9\">" +
"\n" + "<methodCall>" +
"\n" + "<methodName>" + MethodName + "</methodName>" +
"\n" + "<vector type=\"struct\">";
ReturnString += buildNode(FieldArray);
ReturnString += "\n</vector>" +
"\n</methodCall>" +
"\n</simpleRPC>";
return ReturnString;
}
public string buildNode(Hashtable FieldArray)
{
string ReturnList = "";
foreach (DictionaryEntry Item in FieldArray) {
string TypeName = "int";
string NodeType = "scalar";
Type myType = Item.Value.GetType();
string fieldValue = "";
if (myType == typeof(string) ) {
TypeName = "string";
fieldValue = Item.Value.ToString();
}
if (myType == typeof(Hashtable) ) {
fieldValue = buildNode(Item.Value as Hashtable);
NodeType = "vector";
TypeName = "struct";
}
if (myType == typeof(int) ) {
fieldValue = Item.Value.ToString();
TypeName = "int";
}
var ThisNode = "\n<" + NodeType + " type=\"" + TypeName + "\" id=\"" + Item.Key + "\">" + fieldValue + "</" + NodeType + ">";
ReturnList += ThisNode;
}
return ReturnList;
}
The buildXMLRPCRequest is used to build XML. You hand it a HashTable with fields you want to encode which may include objects of the types: int, string or Hashtable. It will return a beautifully formated (Simple) XML-RPC string which is ready to go to our server.
Send
To send XML to our servers, you need to issue a POST request with the mime type set to text/xml. None of the standard C# methods can be used in Unity but using this with the output of the buildXMLRPCRequest logic works perfectly. What it does:
Sending in Unity
I used this code:
private void UnityPostXML( int Staging,
string WebServer,
string MethodName,
Hashtable FieldArray)
{
string WebServiceURL = "http://LIVESERVER/";
if (Staging == 1) {
WebServiceURL = "http://TESTSERVER";
}
// Encode the text to a UTF8 byte arrray
string XMLRequest = buildXMLRPCRequest(FieldArray,MethodName);
System.Text.Encoding enc = System.Text.Encoding.UTF8;
byte[] myByteArray = enc.GetBytes(XMLRequest);
// Get the Unity WWWForm object (a post version)
var form = new WWWForm();
var url = WebServiceURL;
// Add a custom header to the request.
// Change the content type to xml and set the character set
var headers = form.headers;
headers["Content-Type"]="text/xml;charset=UTF-8";
// Post a request to an URL with our rawXMLData and custom headers
var www = new WWW(WebServiceURL, myByteArray, headers);
// Start a co-routine which will wait until our servers comes back
StartCoroutine(WaitForRequest(www));
}
IEnumerator WaitForRequest(WWW www)
{
yield return www;
// check for errors
if (www.error == null)
{
Debug.Log("WWW Ok!: " + www.text);
} else {
Debug.Log("WWW Error: "+ www.error);
}
}
encode the XML to a ByteArray using UTF8
Create a new Unity WWWForm
Create a HashTable, store the current http headers (if any) and overwrite the content type to text/xml
Send that lot to the server
Set up a Coroutine which waits for the reply
Sending without Unity
I found that developing a library in C# (I use the standards version of MonoDevelop) is much simpler then using Unity for everything so the equivelant send logic in C# is below if wnat to do the same.
private string NormalXMLCall(int Staging,
string WebServer,
string MethodName,
Hashtable Fields)
{
// Figure out who to call
string WebServiceURL = "http://LIVSERVER";
if (Staging == 1) {
WebServiceURL = "http://TESTSERVER";
}
WebServiceURL += WebServer;
// Build the request
XmlRpcParser parser = new XmlRpcParser();
string XMLRequest = parser.buildXMLRPCRequest(Fields,MethodName);
// Fire it off
HttpWebRequest httpRequest =(HttpWebRequest)WebRequest.Create(WebServiceURL);
httpRequest.Method = "POST";
//Defining the type of the posted data as XML
httpRequest.ContentType = "text/xml";
// string data = xmlDoc.InnerXml;
byte[] bytedata = Encoding.UTF8.GetBytes(XMLRequest);
// Get the request stream.
Stream requestStream = httpRequest.GetRequestStream();
// Write the data to the request stream.
requestStream.Write(bytedata, 0, bytedata.Length);
requestStream.Close();
//Get Response
HttpWebResponse httpResponse = (HttpWebResponse)httpRequest.GetResponse();
// Get the stream associated with the response.
Stream receiveStream = httpResponse.GetResponseStream ();
// Pipes the stream to a higher level stream reader with the required encoding format.
StreamReader readStream = new StreamReader (receiveStream, Encoding.UTF8);
string ReceivedData = readStream.ReadToEnd ();
httpResponse.Close ();
readStream.Close ();
return ReceivedData;
}
}
Extract data from XML
I wrote a simple parser. The constructor for the below findNode function should be given the raw XML data and the child node object you want to find. It will return the value of that node (as a string) if that node can be found on the highest level of the XML string or null if it can't find it. This parser is specific to "Simple XML-RPC" and needs a bit of work to decode encoded characters but that should be simple to add.
public string findNode(string Xml,string SearchForTag) {
int NestCounter = 0;
bool FoundTag = false;
int FoundTagLevel = 0;
string ReturnValue = null;
// Break it down by "<"
string [] TagArray = Xml.Split('<');
for (int i=0;i<TagArray.Length;i++) {
if (i>175 && i<180) {
int Hello=1;
}
string ThisLine = "<" + TagArray[i];
if (ThisLine.Length <= 1) continue;
if ((ThisLine.Length >= 2) && (ThisLine.Substring(0,2) == "<?")) continue;
if ((ThisLine.Length >= 3) && (ThisLine.Substring(0,3) == "<--")) continue;
// It can be a vector or a scalar - vectors are full of scalars so we'll
ThisLine = ThisLine.Replace(" "," ");
ThisLine = ThisLine.Replace("</","</");
string [] FieldArray = ThisLine.Split(' ');
bool AddLineToResult = FoundTag;
// Nest counter is the level we are operating on. We only check the first
// Level. When a vector is found we increase the NestCount and we won't
// search for the ID
if (NestCounter <= 1) { // Initial array we are looking on level 1
for (int a=0;a<FieldArray.Length;a++) {
string ThisTag = FieldArray[a];
string [] TagValue = ThisTag.Split("=\"".ToCharArray(),5);
// Every TagValue is xx=yy pair... we want "ID=\"xxx\"
if (TagValue.Length >= 3) {
string TagName = TagValue[2];
if (TagName == SearchForTag) {
FoundTag = true;
FoundTagLevel = NestCounter;
// This could be a vector or Scalar so find the ">" in this string
// and start adding from there
int TerminatePos = ThisLine.IndexOf(">");
if ((TerminatePos >= 0) && (TerminatePos < ThisLine.Length)) {
ReturnValue = ThisLine.Substring(TerminatePos+1);
}
break;
}
}
}
}
if (FieldArray.Length > 0) {
string ThisField = FieldArray[0].ToLower();
/*
* If we are in the loop where we have found the tag,
* we haven't changed level and this is the end of a scalar it must
* mean that the tag was a scalar so we can safely leave now.
*/
if ((FoundTag) && (FoundTagLevel == NestCounter) && (ThisField == "</scalar>")) {
break;
// return ReturnValue;
}
// If we end or leave a vector we change the NestCounter
if (ThisField.IndexOf("<vector") >= 0) {
NestCounter++;
}
else if (ThisField.IndexOf("</vector>") >= 0) {
NestCounter--;
}
}
// If we have found our tag and the nest counte goes below the level
// we where looking at - it's time to leave
if (FoundTag) {
if (NestCounter <= FoundTagLevel) {
break;
//return ReturnValue;
}
}
if (AddLineToResult) {
ReturnValue += ThisLine;
}
}
// You may wanna do some url decoding here....
return ReturnValue;
}

401 Error when trying to get Request Token

resp = (HttpWebResponse)request.GetResponse();Right I know this is probably some stupid mistake or just me not being well versed in Oauth stuff but I have come to a grinding halt and have no idea where to go from here, after many searches and attempts I humbly ask for some help.
I am attempting to get a request token from Twitter to try and get a users twitter feeds, I cannot use a library for other business reasons...
Here is the code so far:
Dictionary<string, string> parameters = new Dictionary<string, string>();
parameters.Add("oauth_callback", "www.url.com/redirect.aspx");
parameters.Add("oauth_consumer_key", <Consumer_KEY>);
parameters.Add("oauth_nonce", generateNonce());
parameters.Add("oauth_signature_method", "HMAC-SHA1");
parameters.Add("oauth_timestamp", CurrentUNIXTimestamp.Get());
parameters.Add("oauth_version", "1.0");
parameters = parameters.OrderBy(x => x.Key).ToDictionary(v => v.Key, v => v.Value);
string concat = "";
string OAuthHeader = "OAuth ";
foreach (string k in parameters.Keys)
{
if (k == "oauth_callback")
{
concat += k + "%3D" + EncodeToUpper(parameters[k]) + "%26";
OAuthHeader += k + "=" + "\"" + EncodeToUpper(parameters[k]) + "\", ";
}
else
{
concat += k + "%3D" + parameters[k] + "%26";
OAuthHeader += k + "=" + "\"" + parameters[k] + "\", ";
}
}
concat = concat.Remove(concat.Length - 3, 3);
concat = "POST&" + EncodeToUpper("https://api.twitter.com/oauth/request_token" ) + "&" + concat;
//byte[] content = Encoding.UTF8.GetBytes(concat);
HMACSHA1 hmacsha1 = new HMACSHA1();
hmacsha1.Key = Encoding.ASCII.GetBytes(string.Format("{0}&{1}", EncodeToUpper(<CONSUMER SECRET>, ""));
byte[] dataBuffer = System.Text.Encoding.ASCII.GetBytes(concat);
byte[] hashBytes = hmacsha1.ComputeHash(dataBuffer);
string hash = Convert.ToBase64String(hashBytes);
OAuthHeader += "oauth_signature=\"" + EncodeToUpper(hash) + "\"";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://api.twitter.com/oauth/request_token");
request.Method = "POST";
request.Headers["Authorization"] = OAuthHeader;
StringBuilder responding = new StringBuilder();
HttpWebResponse resp = null;
try
{
resp = (HttpWebResponse)request.GetResponse();
}
catch (WebException exc)
{
lblError.Text = "Error Connecting to Social Network " + exc.Message;
}
if (resp != null)
{
using (StreamReader reader = new StreamReader(resp.GetResponseStream()))
{
responding.Append(reader.ReadToEnd());
}
}
An Example of the Nonce is "ne8ehvVr0pW2EUxNHdxdyqbi8Fwphatt3SW1yerTyXH" and the CurrentUNIXTimestamp is generated by
public static class CurrentUNIXTimestamp
{
public static string Get()
{
return Convert.ToString((int)(DateTime.UtcNow - new DateTime(1970, 1, 1, 0, 0, 0)).TotalSeconds);
}
}
I have tried as many things as I can think of, there is no longer a client/browser issue (which was many other answers), the server time is correct to British Summer Time (I don't know whether that would be an issue but I tried adding an hour to the unix time stamp still a 401.), and i have defined the callback url on the twitter app page
I have got the same app working with facebook (I know its different oauth but may help)
The actual error comes at resp = (HttpWebResponse)request.GetResponse();, which comes up with a 401 error. I couldn't get any further details from the exc.response object, could anyone say how to get something useful out of the error in VS2008?
Thanks for any answers

The hardest problem I had implementing OAuth was dealing with character encoding. It's very particular.
Here is the code I ended up writing for it.
private static string UrlEncode(IEnumerable<KeyValuePair<string, object>> parameters)
{
StringBuilder parameterString = new StringBuilder();
var paramsSorted = from p in parameters
orderby p.Key, p.Value
select p;
foreach (var item in paramsSorted)
{
if (parameterString.Length > 0)
{
parameterString.Append("&");
}
if(item.Value.GetType() == typeof(string) )
parameterString.Append(
string.Format(
CultureInfo.InvariantCulture,
"{0}={1}",
UrlEncode(item.Key),
UrlEncode(item.Value as string)));
}
return UrlEncode(parameterString.ToString());
}
public static string UrlEncode(string value)
{
if (string.IsNullOrEmpty(value))
{
return string.Empty;
}
value = Uri.EscapeDataString(value);
// UrlEncode escapes with lowercase characters (e.g. %2f) but oAuth needs %2F
value = Regex.Replace(value, "(%[0-9a-f][0-9a-f])", c => c.Value.ToUpper());
// these characters are not escaped by UrlEncode() but needed to be escaped
value = value
.Replace("(", "%28")
.Replace(")", "%29")
.Replace("$", "%24")
.Replace("!", "%21")
.Replace("*", "%2A")
.Replace("'", "%27");
// these characters are escaped by UrlEncode() but will fail if unescaped!
value = value.Replace("%7E", "~");
return value;
}
If you really get fed up, you can use the WebRequestBuilder class from my library to do all the OAuth stuff for you: http://www.twitterizer.net/

Payment succeeds, and our data matches querystring data, but falls thru to INVALID anyway!

Why does my ipn script I wrote always fail? It always goes to INVALID even though it matches everything in the query string that paypal sends to me?
notification.cshtml?tx=b78v54b5b55rby92S&st=Completed&amt=3.04&cc=USD&cm=&item_number=&merchant_return_link=Return+to+web+site+name&form_charset=UTF-8
And the part that checks it is:
string LiveURL = "https://www.paypal.com/cgi-bin/webscr";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(LiveURL);
// Set request back values.
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
byte[] parameters = Request.BinaryRead(HttpContext.Current.Request.ContentLength);
string RequestString = System.Text.Encoding.ASCII.GetString(parameters);
RequestString += "&cmd=_notify-validate";
request.ContentLength = RequestString.Length;
// Send request to PP and get response.
StreamWriter Sout = new StreamWriter(request.GetRequestStream(), System.Text.Encoding.ASCII);
Sout.Write(RequestString);
Sout.Close();
StreamReader Sin = new StreamReader(request.GetResponse().GetResponseStream());
string response = Sin.ReadToEnd();
Sin.Close();
if(result != null && result.OrderStatus == "Confirmed")
{
switch(response)
{
case "VERIFIED":
if(Request["st"] == "Completed")
{
var PPQuery = "SELECT TransactionId, OrderTotal FROM Orders WHERE OrderId = '" + Session["OSFOID"] + "' AND UserId = '" + WebSecurity.CurrentUserId + "'";
var ppQueryResult = database.Query(PPQuery);
foreach(var item in ppQueryResult)
{
decimal fff = 3.04M;
if(item["TransactionId"] != Request["tx"])
{
if(item["OrderTotal"] == TotalPrice)
{
// Payment was a success. Convey that to the user.
output = "Thanks. Order complete.";
}
else
{
// Possible fraud. Log it.
}
}
else
{
// This is a duplicate transaction. Log it and Redirect to homepage.
}
}
}
break;
case "INVALID":
output = "Invalid was returned. Investigate further.";
break;
default:
output = "Other exception has occured. Investigate further and log.";
break;
}
}

The code looks fine. The problem must be with response not matching "VERIFIED".
You're not in Turkey by chance, and changing response to uppercase prior to the comparison? *
*) If the locale is Turkey, uppercasing a string turns i into İ, not I (just one of the many traps with string manipulation)

Within the "VERIFIED" block, check:
if (Request.Params["payment_status"] == "Completed")
{
...
}
Request["st"] is incorrect.
Be sure to set IPN URL in one place in PayPal admin and do not use the other form of return URL checking (can't remember the name of it offhand) and IPN at the same time.
There is no "merchant_return_link" parameter; I think it should be "notify_url"... the URL string and the list of params doesn't look right to me; for example: &cm=&item_number
I know your list of params will be unique for your situation, but here's some sample code where I construct the URL to be passed to PayPal:
protected string GetPayPalURL(string SERVER_URL, string business, string[] itemNames,
int[] quantities, decimal[] amounts, double[] weight, string invoiceID, string transID, string NOTIFY_URL)
{
// Customer will be required to specify delivery address to PayPal - VERY IMPORTANT
const string NO_SHIPPING = "2";
StringBuilder url = new StringBuilder();
url.Append(SERVER_URL + "?cmd=_cart&upload=1");
url.Append("&business=" + HttpUtility.UrlEncode(business));
for (int i = 0; i < itemNames.Length; i++)
{
url.Append("&item_name" + "_" + (i + 1).ToString() + "=" + HttpUtility.UrlEncode(itemNames[i]));
url.Append("&quantity" + "_" + (i + 1).ToString() + "=" + quantities[i].ToString().Replace(",", "."));
url.Append("&amount" + "_" + (i + 1).ToString() + "=" + amounts[i].ToString().Replace(",", "."));
url.Append("&weight" + "_" + (i + 1).ToString() + "=" + weight[i].ToString().Replace(",", "."));
}
url.Append("&no_shipping=" + HttpUtility.UrlEncode(NO_SHIPPING));
url.Append("&custom=" + HttpUtility.UrlEncode(invoiceID));
url.Append("&txn_id=" + HttpUtility.UrlEncode(transID));
url.Append("&notify_url=" + HttpUtility.UrlEncode(NOTIFY_URL));
return url.ToString();
}

I think the Paypal method you are trying to do is as follows on code project
and if you get payment_status = INVALID, then check the reason in payment_reason

i dont see in the code where you are defining result which is checked in the if, also in the switch you are checking against request, surely this should be against response?