Develop Reference

UriBuilder().Query will wrongly encode non-ASCII characters

I am working on an asp.net mvc 4 web application. and i am using .net 4.5. now i have the following WebClient() class:
using (var client = new WebClient())
{
var query = HttpUtility.ParseQueryString(string.Empty);
query["model"] = Model;
//code goes here for other parameters....
string apiurl = System.Web.Configuration.WebConfigurationManager.AppSettings["ApiURL"];
var url = new UriBuilder(apiurl);
url.Query = query.ToString();
string xml = client.DownloadString(url.ToString());
XmlDocument doc = new XmlDocument();
//code goes here ....
}
now i have noted a problem when one of the parameters contain non-ASCII charterers such as £, ¬, etc....
now the final query will have any non-ASCII characters (such as £) encoded wrongly (as %u00a3). i read about this problem and seems i can replace :-
url.Query = query.ToString();
with
url.Query = ri.EscapeUriString(HttpUtility.UrlDecode(query.ToString()));
now using the later approach will encode £ as %C2%A3 which is the correct encoded value.
but the problem i am facing with url.Query = Uri.EscapeUriString(HttpUtility.UrlDecode(query.ToString())); in that case one of the parameters contains & then the url will have the following format &operation=AddAsset&assetName=&.... so it will assume that I am passing empty assetName parameter not value =&??
EDIT
Let me summarize my problem again. I want to be able to pass the following 3 things inside my URL to a third part API :
Standard characters such as A,B ,a ,b ,1, 2, 3 ...
Non-ASCII characters such as £,¬ .
and also special characters that are used in url encoding such as & , + .
now i tried the following 2 approaches :
Approach A:
using (var client = new WebClient())
{
var query = HttpUtility.ParseQueryString(string.Empty);
query["model"] = Model;
//code goes here for other parameters....
string apiurl = System.Web.Configuration.WebConfigurationManager.AppSettings["ApiURL"];
var url = new UriBuilder(apiurl);
url.Query = query.ToString();
string xml = client.DownloadString(url.ToString());
XmlDocument doc = new XmlDocument();
//code goes here ....
}
In this approach i can pass values such as & ,+ since they are going to be url encoded ,,but if i want to pass non-ASCII characters they will be encoded using ISO-8859-1 ... so if i have £ value , my above code will encoded as %u00a3 and it will be saved inside the 3rd party API as %u00a3 instead of £.
Approach B :
I use :
url.Query = Uri.EscapeUriString(HttpUtility.UrlDecode(query.ToString()));
instead of
url.Query = query.ToString();
now I can pass non-ASCII characters such as £ since they will be encoded correctly using UTF8 instead of ISO-8859-1. but i can not pass values such as & because my url will be read wrongly by the 3rd party API.. for example if I want to pass assetName=& my url will look as follow:
&operation=Add&assetName=&
so the third part API will assume I am passing empty assetName, while I am trying to pass its value as &...
so not sure how I can pass both non-ASCII characters + characters such as &, + ????

You could use System.Net.Http.FormUrlEncodedContent instead.
This works with a Dictionary for the Name/Value pairing and the Dictionary, unlike the NameValueCollection, does not "incorrectly" map characters such as £ to an unhelpful escaping (%u00a3, in your case).
Instead, the FormUrlEncodedContent can take a dictionary in its constructor. When you read the string out of it, it will have properly urlencoded the dictionary values.
It will correctly and uniformly handle both of the cases you were having trouble with:
£ (which exceeds the character value range of urlencoding and needs to be encoded into a hexadecimal value in order to transport)
& (which, as you say, has meaning in the url as a parameter separator, so that values cannot contain it--so that it has to be encoded as well).
Here's a code example, that shows that the various kinds of example items you mentioned (represented by item1, item2 and item3) now end up correctly urlencoded:
String item1 = "£";
String item2 = "&";
String item3 = "xyz";
Dictionary<string,string> queryDictionary = new Dictionary<string, string>()
{
{"item1", item1},
{"item2", item2},
{"item3", item3}
};
var queryString = new System.Net.Http.FormUrlEncodedContent(queryDictionary)
.ReadAsStringAsync().Result;
queryString will contain item1=%C2%A3&item2=%26&item3=xyz.

Maybe you could try to use an Extension method on the NameValueCollection class. Something like this:
using System.Collections.Specialized;
using System.Text;
using System.Web;
namespace Testing
{
public static class NameValueCollectionExtension
{
public static string ToUtf8UrlEncodedQuery(this NameValueCollection nv)
{
StringBuilder sb = new StringBuilder();
bool firstIteration = true;
foreach (var key in nv.AllKeys)
{
if (!firstIteration)
sb.Append("&");
sb.Append(HttpUtility.UrlEncode(key, Encoding.UTF8))
.Append("=")
.Append(HttpUtility.UrlEncode(nv[key], Encoding.UTF8));
firstIteration = false;
}
return sb.ToString();
}
}
}
Then, in your code you can do this:
url.Query = query.ToUtf8UrlEncodedQuery();
Remember to add a using directive for the namespace where you put the NameValueCollectionExtension class.

The problem here isn't UriBuilder.Query, it's UriBuilder.ToString(). Read the documentation here: https://msdn.microsoft.com/en-us/library/system.uribuilder.tostring(v=vs.110).aspx. The property is defined as returning the "display string" of the builder, not a validly encoded string. Uri.ToString() has a similar problem, in that it doesn't perform proper encoding.
Use the following instead: url.Uri.AbsoluteUri, that will always be a properly encoded string. You shouldn't have to do any encoding on the way into the builder (that's part of it's purpose, after all, to properly encode things).

You need to use:
System.Web.HttpUtility.UrlEncode(key)
Change your code to this:
using (var client = new WebClient())
{
var query = HttpUtility.ParseQueryString(string.Empty);
query["model"] = Model;
//code goes here for other parameters....
string apiurl = System.Web.Configuration.WebConfigurationManager.AppSettings["ApiURL"];
var url = new UriBuilder(apiurl);
url.Query = HttpUtility.UrlEncode(query.ToString());
string xml = client.DownloadString(url.ToString());
XmlDocument doc = new XmlDocument();
//code goes here ....
}

If nothing helps, then just manually convert those problematic chars inside values of parameters
& to %26
+ to %2B
? to %3F

Is there any way to check if oauth token is expired or not?

I am accessing web api using oauth token.
Token expires after 1 hour. But I want to add functionality to generate new token when it expires.
I found that in case of expired token it sends StatusCode as unauthorized.
Please let me know if its the only statuscode which tells about expiration of token.

Resurrecting this post once again and taking #Lavandysh's solution and pulling in the System.IdentityModel.Tokens.Jwt class; call this method to have an idea if the token is even valid anymore before the request:
public bool _isEmptyOrInvalid (string token)
{
if (string.IsNullOrEmpty(token))
{
return true;
}
var jwtToken = new JwtSecurityToken(token);
return (jwtToken == null) || (jwtToken.ValidFrom > DateTime.UtcNow) || (jwtToken.ValidTo < DateTime.UtcNow);
}

I know this is an old post, but there is a more direct way. If you decode the token from Base64 format you can see what the expiration date is, then you can compare it with the date right now.
If you want to check if this method works you can use this website to decode the token.
Now, in C# you run against some difficulties since the token contains '.' that cannot be decoded and if you only take out the middle part between the two points the length becomes invalid (it should always be a multitude of 4). To make the middle part have a correct length you can add '=' signs until it is a multitude of 4.
In the end I compare the date with the current dat plus a minute. Because I don't want a token that will expire in a second to be considered valid. You might want to adjust that time to suit your purposes.
So here's my code:
/***
* Check if the string token is expired by decoding it from the Base64 string
* Some adjustements to the string are necessairy since C# can only decode certain strings.
* It cannot handle '.' and requires a string has a length that is a multitude of 4.
* The information we are interrested in is found between the dots.
*
* The token that is given as a parameter should have approximately the following structure:
* ddddddddddddddddddddd.ddddddddddddddddddddddddddddddddddddd.dddddddddddddddddddddddddddd
* And should be a valid Oauth token that may or may not be expired
*/
public static bool isExpired(String token)
{
if (token == null || ("").Equals(token))
{
return true;
}
/***
* Make string valid for FromBase64String
* FromBase64String cannot accept '.' characters and only accepts stringth whose length is a multitude of 4
* If the string doesn't have the correct length trailing padding '=' characters should be added.
*/
int indexOfFirstPoint = token.IndexOf('.') + 1;
String toDecode = token.Substring(indexOfFirstPoint, token.LastIndexOf('.') - indexOfFirstPoint);
while (toDecode.Length % 4 != 0)
{
toDecode += '=';
}
//Decode the string
string decodedString = Encoding.ASCII.GetString(Convert.FromBase64String(toDecode));
//Get the "exp" part of the string
String beginning = "\"exp\":\"";
int startPosition = decodedString.LastIndexOf(beginning) + beginning.Length;
decodedString = decodedString.Substring(startPosition);
int endPosition = decodedString.IndexOf("\"");
decodedString = decodedString.Substring(0, endPosition);
long timestamp = Convert.ToInt64(decodedString);
DateTime date = new DateTime(1970, 1, 1).AddMilliseconds(timestamp);
DateTime compareTo = DateTime.Now.AddMinutes(1);
int result = DateTime.Compare(date, compareTo);
return result < 0;
}

The easiest way is to just try to call the service with it. It will reject it if it is expired and then you can request a new one.
You can also keep the time you received the token and use the expires_in to calculate when it will approximately expire. Then you request a new token before making a new request after the expiration date.

I got that problem when I was developing a Xamarin Forms application. The main problem here is that I couldn't install any Nuget in the project. So I decide to improve #Lavandysg answer, since It isn't extracting the expiration timestamp correctly and It isn't calculating the expiration time correctly. Here is my final code using regex to extract expiration time from token:
public bool IsTokenExpired(string token)
{
if (token == null || ("").Equals(token))
{
return true;
}
/***
* Make string valid for FromBase64String
* FromBase64String cannot accept '.' characters and only accepts stringth whose length is a multitude of 4
* If the string doesn't have the correct length trailing padding '=' characters should be added.
*/
int indexOfFirstPoint = token.IndexOf('.') + 1;
String toDecode = token.Substring(indexOfFirstPoint, token.LastIndexOf('.') - indexOfFirstPoint);
while (toDecode.Length % 4 != 0)
{
toDecode += '=';
}
//Decode the string
string decodedString = Encoding.ASCII.GetString(Convert.FromBase64String(toDecode));
//Get the "exp" part of the string
Regex regex = new Regex("(\"exp\":)([0-9]{1,})");
Match match = regex.Match(decodedString);
long timestamp = Convert.ToInt64(match.Groups[2].Value);
DateTime date = new DateTime(1970, 1, 1).AddSeconds(timestamp);
DateTime compareTo = DateTime.UtcNow;
int result = DateTime.Compare(date, compareTo);
return result < 0;
}

To check if the token is expired I made an Http request to check if the returned value is null. If it is null, it means that the token is expired but if the returned value is not null then the token is still available.
List<Values> values = get.GetAllValues(); if(values is null) { return RedirectToPage("Logout"); }
I hope this helps :D

Is there a simpler way to calculate this MD5 hash?

I'm working on implementing a hosted checkout, and the hosted checkout is supposed to redirect the user back to my website so that I can show a custom receipt page.
This is a sample querystring that I'd get back:
trnApproved=0&trnId=10000000&messageId=71&messageText=Declined&authCode=000000&responseType=T&trnAmount=20.00&trnDate=9%2f23%2f2011+9%3a30%3a56+AM&trnOrderNumber=1000000&trnLanguage=eng&trnCustomerName=FirstName+LastName&trnEmailAddress=something_something%40gmail.com&trnPhoneNumber=1235550123&avsProcessed=0&avsId=0&avsResult=0&avsAddrMatch=0&avsPostalMatch=0&avsMessage=Address+Verification+not+performed+for+this+transaction.&cvdId=3&cardType=VI&trnType=P&paymentMethod=CC&ref1=9dae6af7-7c22-4697-b23a-413d8a129a75&ref2=&ref3=&ref4=&ref5=&hashValue=33dacf84682470f267b2cc6d528b1594
To validate the request, I'm supposed to remove &hashValue=f3cf58ef0fd363e0c2241938b04f1068 from the end of the querystring, and then append a key. I then perform an MD5 hash of the entire string, and the result should be 33dacf84682470f267b2cc6d528b1594, same as the original.
This is easy, except that a few of the fields are causing a problem for me. This is the code I use (taken from a dummy application, so you can ignore some of the bad coding):
// Split up the query string parameters
string[] parameters = GetQueryString().Split(new[] { "&" }, StringSplitOptions.None);
var querySet = new List<string>();
// Rebuild the query string, encoding the values.
foreach (string s in parameters)
{
// Every field that contains a "." will need to be encoded except for trnAmount
querySet.Add(param.Contains("trnAmount") ? param : UrlEncodeToUpper(param));
}
// Create the querystring without the hashValue, we need to calculate our hash without it.
string qs = string.Join("&", querySet.ToArray());
qs = qs.Substring(0, qs.IndexOf("&hashValue"));
qs = qs + "fb76124fea73488fa11995dfa4cbe89b";
var encoding = new UTF8Encoding();
var md5 = new MD5CryptoServiceProvider();
var hash = md5.ComputeHash(encoding.GetBytes(qs));
var calculatedHash = BitConverter.ToString(hash).Replace("-", String.Empty).ToLower();
This is the UrlEncode method I use.
private static string UrlEncodeToUpper(string value)
{
// Convert their encoding into uppercase so we can do our hash
value = Regex.Replace(value, "(%[0-9af][0-9a-f])", c => c.Value.ToUpper());
// Encode the characters that they missed
value = value.Replace("-", "%2D").Replace(".", "%2E").Replace("_", "%5F");
return value;
}
This all works (until someone enters a character I haven't accounted for), except this seems more complicated than it should be. I know I'm not the only one who has to implement this HCO into an ASP.NET application, so I don't think the simple validation should be so complicated.
Am I missing an easier way to do this? Having to loop through the fields, encoding some of them while skipping others, converting their encoding to uppercase and then selectively replacing characters seems a little... odd.

Here's a better way to work with query strings:
var queryString = "trnApproved=0&trnId=10000000&messageId=71&messageText=Declined&authCode=000000&responseType=T&trnAmount=20.00&trnDate=9%2f23%2f2011+9%3a30%3a56+AM&trnOrderNumber=1000000&trnLanguage=eng&trnCustomerName=FirstName+LastName&trnEmailAddress=something_something%40gmail.com&trnPhoneNumber=1235550123&avsProcessed=0&avsId=0&avsResult=0&avsAddrMatch=0&avsPostalMatch=0&avsMessage=Address+Verification+not+performed+for+this+transaction.&cvdId=3&cardType=VI&trnType=P&paymentMethod=CC&ref1=9dae6af7-7c22-4697-b23a-413d8a129a75&ref2=&ref3=&ref4=&ref5=&hashValue=33dacf84682470f267b2cc6d528b1594";
var values = HttpUtility.ParseQueryString(queryString);
// remove the hashValue parameter
values.Remove("hashValue");
var result = values.ToString();
// At this stage result = trnApproved=0&trnId=10000000&messageId=71&messageText=Declined&authCode=000000&responseType=T&trnAmount=20.00&trnDate=9%2f23%2f2011+9%3a30%3a56+AM&trnOrderNumber=1000000&trnLanguage=eng&trnCustomerName=FirstName+LastName&trnEmailAddress=something_something%40gmail.com&trnPhoneNumber=1235550123&avsProcessed=0&avsId=0&avsResult=0&avsAddrMatch=0&avsPostalMatch=0&avsMessage=Address+Verification+not+performed+for+this+transaction.&cvdId=3&cardType=VI&trnType=P&paymentMethod=CC&ref1=9dae6af7-7c22-4697-b23a-413d8a129a75&ref2=&ref3=&ref4=&ref5=
// now add some other query string value
values["foo"] = "bar"; // you can stuff whatever you want it will be properly url encoded
Then I didn't quite understand what you wanted to do. You want to calculate an MD5 on the result? You could do that and then append to the query string.

Encoding and decoding a string that may have slashes in it

I have strings like this:
RowKey = "Local (Automatic/Manual) Tests",
When I try to store in Windows Azure then this fails as I assume it does not accept the "/" as part of the row key.
Is there a simple way that I can encode the value before putting into RowKey?
Also once the data is in the table I get it out with the following:
var Stores = storeTable.GetAll(u => u.PartitionKey == "ABC");
Is there a simple way that I can get out the value of RowKey and decode it?

One possible way for handling is this by converting the PartitionKey and RowKey values in Base64 encoded string and save it. Later when you retrieve the values, you just decode it. In fact I have had this issue some days back in our tool and Base64 encoding was suggested to me on MSDN forums: http://social.msdn.microsoft.com/Forums/en-US/windowsazuredata/thread/a20cd3ce-20cb-4273-a1f2-b92a354bd868. But again it is not fool proof.

I'm not familiar with Azure, so I don't know if there is an existing API for that. But it's not hard to code:
encode:
const string escapeChar='|';
RowKey.Replace(escapeChar,escapeChar+escapeChar).Replace("/",escapeChar+"S");
decode:
StringBuilder sb=new StringBuilder(s.Length);
bool escape=false;
foreach(char c in s)
{
if(escape)
{
if(c=='S')
sb.Append('/');
else if(c==escapeChar)
sb.Append(escapeChar);
else
throw new ArgumentException("Invalid escape sequence "+escapeChar+c);
}
else if(c!=escapeChar)
{
sb.Append(c);
escape=false;
}
else
escape=true;
return sb.ToString();

When a string is Base64 encoded, the only character that is invalid in an Azure Table Storage key column is the forward slash ('/'). To address this, simply replace the forward slash character with another character that is both (1) valid in an Azure Table Storage key column and (2) not a Base64 character. The most common example I have found (which is cited in other answers) is to replace the forward slash ('/') with the underscore ('_').
private static String EncodeToKey(String originalKey)
{
var keyBytes = System.Text.Encoding.UTF8.GetBytes(originalKey);
var base64 = System.Convert.ToBase64String(keyBytes);
return base64.Replace('/','_');
}
When decoding, simply undo the replaced character (first!) and then Base64 decode the resulting string. That's all there is to it.
private static String DecodeFromKey(String encodedKey)
{
var base64 = encodedKey.Replace('_', '/');
byte[] bytes = System.Convert.FromBase64String(base64);
return System.Text.Encoding.UTF8.GetString(bytes);
}
Some people have suggested that other Base64 characters also need encoding. According to the Azure Table Storage docs this is not the case.

Determine if a string contains a base64 string inside of it

I'm trying to figure out a way to parse out a base64 string from with a larger string.
I have the string "Hello <base64 content> World" and I want to be able to parse out the base64 content and convert it back to a string. "Hello Awesome World"
Answers in C# preferred.
Edit: Updated with a more real example.
--abcdef
\n
Content-Type: Text/Plain;
Content-Transfer-Encoding: base64
\n
<base64 content>
\n
--abcdef--
This is taken from 1 sample. The problem is that the Content.... vary quite a bit from one record to the next.

There is no reliable way to do it. How would you know that, for instance, "Hello" is not a base64 string ? OK, it's a bad example because base64 is supposed to be padded so that the length is a multiple of 4, but what about "overflow" ? It's 8-character long, it is a valid base64 string (it would decode to "¢÷«~Z0"), even though it's obviously a normal word to a human reader. There's just no way you can tell for sure whether a word is a normal word or base64 encoded text.
The fact that you have base64 encoded text embedded in normal text is clearly a design mistake, I suggest you do something about it rather that trying to do something impossible...

In short form you could:
split the string on any chars that are not valid base64 data or padding
try to convert each token
if the conversion succeeds, call replace on the original string to switch the token with the converted value
In code:
var delimiters = new char[] { /* non-base64 ASCII chars */ };
var possibles = value.Split(delimiters, StringSplitOptions.RemoveEmptyEntries);
//need to tweak to include padding chars in matches, but still split on padding?
//maybe better off creating a regex to match base64 + padding
//and using Regex.Split?
foreach(var match in possibles)
{
try
{
var converted = Convert.FromBase64String(match);
var text = System.Text.Encoding.UTF8.GetString(converted);
if(!string.IsNullOrEmpty(text))
{
value = value.Replace(match, text);
}
}
catch (System.ArgumentNullException)
{
//handle it
}
catch (System.FormatException)
{
//handle it
}
}
Without a delimiter though, you can end up converting non-base64 text that happens to be also be valid as base64 encoded text.
Looking at your example of trying to convert "Hello QXdlc29tZQ== World" to "Hello Awesome World" the above algorithm could easily generate something like "ée¡Ý•Í½µ”¢¹]" by trying to convert the whole string from base64 since there is no delimiter between plain and encoded text.
Update (based on comments):
If there are no '\n's in the base64 content and it is always preceded by "Content-Transfer-Encoding: base64\n", then there is a way:
split the string on '\n'
iterate over all the tokens until a token ends in "Content-Transfer-Encoding: base64"
the next token (if there are any) should be decoded (if possible) and then the replacement should be made in the original string
return to iterating until out of tokens
In code:
private string ConvertMixedUpTextAndBase64(string value)
{
var delimiters = new char[] { '\n' };
var possibles = value.Split(delimiters,
StringSplitOptions.RemoveEmptyEntries);
for (int i = 0; i < possibles.Length - 1; i++)
{
if (possibles[i].EndsWith("Content-Transfer-Encoding: base64"))
{
var nextTokenPlain = DecodeBase64(possibles[i + 1]);
if (!string.IsNullOrEmpty(nextTokenPlain))
{
value = value.Replace(possibles[i + 1], nextTokenPlain);
i++;
}
}
}
return value;
}
private string DecodeBase64(string text)
{
string result = null;
try
{
var converted = Convert.FromBase64String(text);
result = System.Text.Encoding.UTF8.GetString(converted);
}
catch (System.ArgumentNullException)
{
//handle it
}
catch (System.FormatException)
{
//handle it
}
return result;
}