Develop Reference

OAuth Invalid Bas64

I am trying to implement an OAuth flow and in my Services for my client I have the following lines of code:
services.AddAuthentication(config => {
// We check the cookie to confirm that we are authenticated
config.DefaultAuthenticateScheme = "ClientCookie";
// When we sign in we will deal out a cookie
config.DefaultSignInScheme = "ClientCookie";
// use this to check if we are allowed to do something.
config.DefaultChallengeScheme = "OurServer";
})
.AddCookie("ClientCookie")
.AddOAuth("OurServer", config => {
config.ClientId = "client_id";
config.ClientSecret = "client_secret";
config.CallbackPath = "/oauth/callback";
config.AuthorizationEndpoint = "https://localhost:44360/oauth/authorize";
config.TokenEndpoint = "https://localhost:44360/oauth/token";
config.SaveTokens = true;
config.Events = new OAuthEvents()
{
OnCreatingTicket = context =>
{
var accessToken = context.AccessToken;
var base64payload = accessToken.Split('.')[1];
var modified = base64payload.Replace(" ", "");
var bytes = Convert.FromBase64String(base64payload);
var jsonPayload = Encoding.UTF8.GetString(bytes);
var claims = JsonConvert.DeserializeObject<Dictionary<string, string>>(jsonPayload);
foreach (var claim in claims)
{
context.Identity.AddClaim(new Claim(claim.Key, claim.Value));
}
return Task.CompletedTask;
}
};
});
When it goes to try and convert the string from its base64 representation it throws a: System.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.
This confuses me as I am following a tutorial on YouTube that, as far as I can tell from following it and looking at the github code is IDENTICAL. The supposedly invalid Base64 string is:
eyJzdWIiOiJzb21lX2lkIiwiR3Jhbm55IjoiQ29va2llIiwibmJmIjoxNjEwMTYwMjE5LCJleHAiOjE2MTAyNDY2MTksImlzcyI6Imh0dHA6Ly9sb2NhbGhvc3Q6NDQzNjAvIiwiYXVkIjoiaHR0cDovL2xvY2FsaG9zdDo0NDM2MC8ifQ
This is odd as checking it on https://www.base64decode.org/ properly returns the following content:
{"sub":"some_id","Granny":"Cookie","nbf":1610160219,"exp":1610246619,"iss":"http://localhost:44360/","aud":"http://localhost:44360/"}
Which appears to be valid as it matches the claims and content of the JWT token i've set in my controller on the Server side. What exactly am I doing wrong? I've checked other places that deal with issues of tokens, but I don't see any invalid characters there such as -, +, etc.
EDIT:
The fix for it was to do the following, thanks to LinkedListT for mentioning that it isn't a multiple a 4.
private string AddPaddingToBase64(string base64)
{
if(base64.Length % 4 == 0)
return base64;
else
{
var builder = new StringBuilder(base64);
builder.Append("=");
return AddPaddingToBase64(builder.ToString());
}
}
Then where we parse the string first have it go through that method and then pass the modified base64 to the converter. This is also the first time in 3+ years of professional development where recursion is actually simpler and more straight forward than iterative.

The problem is your base64 string isn't valid.
The length of a base64 encoded string is always a multiple of 4. If it is not a multiple of 4, then = characters are appended until it is. A query string of the form ?name=value has problems when the value contains = characters (some of them will be dropped, I don't recall the exact behavior). You may be able to get away with appending the right number of = characters before doing the base64 decode.
https://stackoverflow.com/a/2925959/9936356
So you need to append '==' to the base64 string in question. I.e.
Convert.FromBase64String("eyJzdWIiOiJzb21lX2lkIiwiR3Jhbm55IjoiQ29va2llIiwibmJmIjoxNjEwMTYwMjE5LCJleHAiOjE2MTAyNDY2MTksImlzcyI6Imh0dHA6Ly9sb2NhbGhvc3Q6NDQzNjAvIiwiYXVkIjoiaHR0cDovL2xvY2FsaG9zdDo0NDM2MC8ifQ==");

How to parse query string of a Uri by keeping the encoding?

I'm tryig to parse a query string's parameters into a NameValueCollection, using HttpUtility.ParseQueryString method:
var queryParams = HttpUtility.ParseQueryString(queryString);
The problem is, when I have an encoded parameter value, such as "%3Cscript%3Ealert(%27abc%27)%3C%2Fscript%3E" and I try to get it via a key, I get the decoded version.
var queryString = "?name=%3Cscript%3Ealert(%27abc%27)%3C%2Fscript%3E";
var queryParams = HttpUtility.ParseQueryString(queryString);
var nameValue = queryParams["name"] //<script>alert('abc')</script>
When I look at the reference source I can see that this is because of a boolean parameter urlencoded which I have no way to change. So, I'm guessing this is the expected behaviour, although I'm not sure why there is no option to change this value.
Is there any other method that will allow me to parse a query string without messing with the encoding, or am I missing something here?

UriBuilder().Query will wrongly encode non-ASCII characters

I am working on an asp.net mvc 4 web application. and i am using .net 4.5. now i have the following WebClient() class:
using (var client = new WebClient())
{
var query = HttpUtility.ParseQueryString(string.Empty);
query["model"] = Model;
//code goes here for other parameters....
string apiurl = System.Web.Configuration.WebConfigurationManager.AppSettings["ApiURL"];
var url = new UriBuilder(apiurl);
url.Query = query.ToString();
string xml = client.DownloadString(url.ToString());
XmlDocument doc = new XmlDocument();
//code goes here ....
}
now i have noted a problem when one of the parameters contain non-ASCII charterers such as £, ¬, etc....
now the final query will have any non-ASCII characters (such as £) encoded wrongly (as %u00a3). i read about this problem and seems i can replace :-
url.Query = query.ToString();
with
url.Query = ri.EscapeUriString(HttpUtility.UrlDecode(query.ToString()));
now using the later approach will encode £ as %C2%A3 which is the correct encoded value.
but the problem i am facing with url.Query = Uri.EscapeUriString(HttpUtility.UrlDecode(query.ToString())); in that case one of the parameters contains & then the url will have the following format &operation=AddAsset&assetName=&.... so it will assume that I am passing empty assetName parameter not value =&??
EDIT
Let me summarize my problem again. I want to be able to pass the following 3 things inside my URL to a third part API :
Standard characters such as A,B ,a ,b ,1, 2, 3 ...
Non-ASCII characters such as £,¬ .
and also special characters that are used in url encoding such as & , + .
now i tried the following 2 approaches :
Approach A:
using (var client = new WebClient())
{
var query = HttpUtility.ParseQueryString(string.Empty);
query["model"] = Model;
//code goes here for other parameters....
string apiurl = System.Web.Configuration.WebConfigurationManager.AppSettings["ApiURL"];
var url = new UriBuilder(apiurl);
url.Query = query.ToString();
string xml = client.DownloadString(url.ToString());
XmlDocument doc = new XmlDocument();
//code goes here ....
}
In this approach i can pass values such as & ,+ since they are going to be url encoded ,,but if i want to pass non-ASCII characters they will be encoded using ISO-8859-1 ... so if i have £ value , my above code will encoded as %u00a3 and it will be saved inside the 3rd party API as %u00a3 instead of £.
Approach B :
I use :
url.Query = Uri.EscapeUriString(HttpUtility.UrlDecode(query.ToString()));
instead of
url.Query = query.ToString();
now I can pass non-ASCII characters such as £ since they will be encoded correctly using UTF8 instead of ISO-8859-1. but i can not pass values such as & because my url will be read wrongly by the 3rd party API.. for example if I want to pass assetName=& my url will look as follow:
&operation=Add&assetName=&
so the third part API will assume I am passing empty assetName, while I am trying to pass its value as &...
so not sure how I can pass both non-ASCII characters + characters such as &, + ????

You could use System.Net.Http.FormUrlEncodedContent instead.
This works with a Dictionary for the Name/Value pairing and the Dictionary, unlike the NameValueCollection, does not "incorrectly" map characters such as £ to an unhelpful escaping (%u00a3, in your case).
Instead, the FormUrlEncodedContent can take a dictionary in its constructor. When you read the string out of it, it will have properly urlencoded the dictionary values.
It will correctly and uniformly handle both of the cases you were having trouble with:
£ (which exceeds the character value range of urlencoding and needs to be encoded into a hexadecimal value in order to transport)
& (which, as you say, has meaning in the url as a parameter separator, so that values cannot contain it--so that it has to be encoded as well).
Here's a code example, that shows that the various kinds of example items you mentioned (represented by item1, item2 and item3) now end up correctly urlencoded:
String item1 = "£";
String item2 = "&";
String item3 = "xyz";
Dictionary<string,string> queryDictionary = new Dictionary<string, string>()
{
{"item1", item1},
{"item2", item2},
{"item3", item3}
};
var queryString = new System.Net.Http.FormUrlEncodedContent(queryDictionary)
.ReadAsStringAsync().Result;
queryString will contain item1=%C2%A3&item2=%26&item3=xyz.

Maybe you could try to use an Extension method on the NameValueCollection class. Something like this:
using System.Collections.Specialized;
using System.Text;
using System.Web;
namespace Testing
{
public static class NameValueCollectionExtension
{
public static string ToUtf8UrlEncodedQuery(this NameValueCollection nv)
{
StringBuilder sb = new StringBuilder();
bool firstIteration = true;
foreach (var key in nv.AllKeys)
{
if (!firstIteration)
sb.Append("&");
sb.Append(HttpUtility.UrlEncode(key, Encoding.UTF8))
.Append("=")
.Append(HttpUtility.UrlEncode(nv[key], Encoding.UTF8));
firstIteration = false;
}
return sb.ToString();
}
}
}
Then, in your code you can do this:
url.Query = query.ToUtf8UrlEncodedQuery();
Remember to add a using directive for the namespace where you put the NameValueCollectionExtension class.

The problem here isn't UriBuilder.Query, it's UriBuilder.ToString(). Read the documentation here: https://msdn.microsoft.com/en-us/library/system.uribuilder.tostring(v=vs.110).aspx. The property is defined as returning the "display string" of the builder, not a validly encoded string. Uri.ToString() has a similar problem, in that it doesn't perform proper encoding.
Use the following instead: url.Uri.AbsoluteUri, that will always be a properly encoded string. You shouldn't have to do any encoding on the way into the builder (that's part of it's purpose, after all, to properly encode things).

You need to use:
System.Web.HttpUtility.UrlEncode(key)
Change your code to this:
using (var client = new WebClient())
{
var query = HttpUtility.ParseQueryString(string.Empty);
query["model"] = Model;
//code goes here for other parameters....
string apiurl = System.Web.Configuration.WebConfigurationManager.AppSettings["ApiURL"];
var url = new UriBuilder(apiurl);
url.Query = HttpUtility.UrlEncode(query.ToString());
string xml = client.DownloadString(url.ToString());
XmlDocument doc = new XmlDocument();
//code goes here ....
}

If nothing helps, then just manually convert those problematic chars inside values of parameters
& to %26
+ to %2B
? to %3F

How to retrieve the locale(country) code from URL?

I have a URL, which is like http://example.com/UK/Deal.aspx?id=322
My target is to remove the locale(country) part, to make it like http://example.com/Deal.aspx?id=322
Since the URL may have other similar formats like: https://ssl.example.com/JP/Deal.aspx?id=735, using "substring" function is not a good idea.
What I can think about is to use the following method for separating them, and map them back later.
HttpContext.Current.Request.Url.Scheme
HttpContext.Current.Request.Url.Host
HttpContext.Current.Request.Url.AbsolutePath
HttpContext.Current.Request.Url.Query
And, suppose HttpContext.Current.Request.Url.AbsolutePath will be:
/UK/Deal.aspx?id=322
I am not sure how to deal with this since my boss asked me not to use "regular expression"(he thinks it will impact performance...)
Except "Regular Expression", is there any other way to remove UK from it?
p.s.: the UK part may be JP, DE, or other country code.
By the way, for USA, there is no country code, and the url will be http://example.com/Deal.aspx?id=322
Please also take this situation into consideration.
Thank you.

Assuming that you'll have TwoLetterCountryISOName in the Url. yYou can use UriBuilder class to remove the path from Uri without using the Regex.
E.g.
var originalUri = new Uri("http://example.com/UK/Deal.aspx?id=322");
if (IsLocaleEnabled(sourceUri))
{
var builder = new UriBuilder(sourceUri);
builder.Path
= builder.Path.Replace(sourceUri.Segments[1] /* remove UK/ */, string.Empty);
// Construct the Uri with new path
Uri newUri = builder.Uri;;
}
Update:
// Cache the instance for performance benefits.
static readonly Regex regex = new Regex(#"^[aA-zZ]{2}\/$", RegexOptions.Compiled);
/// <summary>
/// Regex to check if Url segments have the 2 letter
/// ISO code as first ocurrance after root
/// </summary>
private bool IsLocaleEnabled(Uri sourceUri)
{
// Update: Compiled regex are way much faster than using non-compiled regex.
return regex.IsMatch(sourceUri.Segments[1]);
}
For performance benefits you must cache it (means keep it in static readonly field). There's no need to parse a pre-defined regex on every request. This way you'll get all the performance benefits you can get.
Result - http://example.com/Deal.aspx?id=322

It all depends on whether the country code always has the same position. If it's not, then some more details on the possible formats are required.. Maybe you could check, if the first segment has two chars or something, to be sure it really is a country code (not sure if this is reliable though). Or you start with the filename, if it's always in the format /[optionalCountryCode]/deal.aspx?...
How about these two approaches (on string level):
public string RemoveCountryCode()
{
Uri originalUri = new Uri("http://example.com/UK/Deal.aspx?id=322");
string hostAndPort = originalUri.GetLeftPart(UriPartial.Authority);
// v1: if country code is always there, always has same position and always
// has format 'XX' this is definitely the easiest and fastest
string trimmedPathAndQuery = originalUri.PathAndQuery.Substring("/XX/".Length);
// v2: if country code is always there, always has same position but might
// not have a fixed format (e.g. XXX)
trimmedPathAndQuery = string.Join("/", originalUri.PathAndQuery.Split('/').Skip(2));
// in both cases you need to join it with the authority again
return string.Format("{0}/{1}", hostAndPort, trimmedPathAndQuery);
}

If the AbsolutePath will always have the format /XX/...pagename.aspx?id=### where XX is the two letter country code, then you can just strip off the first 3 characters.
Example that removes the first 3 characters:
var targetURL = HttpContext.Current.Request.Url.AbsolutePath.Substring(3);
If the country code could be different lengths, then you could find the index of the second / character and start the substring from there.
var sourceURL = HttpContext.Current.Request.Url.AbsolutePath;
var firstOccurance = sourceURL.IndexOf('/')
var secondOccurance = sourceURL.IndexOf('/', firstOccurance);
var targetURL = sourceURL.Substring(secondOccurance);

The easy way would be to treat as string, split it by the "/" separator, remove the fourth element, and then join them back with the "/" separator again:
string myURL = "https://ssl.example.com/JP/Deal.aspx?id=735";
List<string> myURLsplit = myURL.Split('/').ToList().RemoveAt(3);
myURL = string.Join("/", myURLsplit);
RESULT: https://ssl.example.com/Deal.aspx?id=735

Encoding and decoding a string that may have slashes in it

I have strings like this:
RowKey = "Local (Automatic/Manual) Tests",
When I try to store in Windows Azure then this fails as I assume it does not accept the "/" as part of the row key.
Is there a simple way that I can encode the value before putting into RowKey?
Also once the data is in the table I get it out with the following:
var Stores = storeTable.GetAll(u => u.PartitionKey == "ABC");
Is there a simple way that I can get out the value of RowKey and decode it?

One possible way for handling is this by converting the PartitionKey and RowKey values in Base64 encoded string and save it. Later when you retrieve the values, you just decode it. In fact I have had this issue some days back in our tool and Base64 encoding was suggested to me on MSDN forums: http://social.msdn.microsoft.com/Forums/en-US/windowsazuredata/thread/a20cd3ce-20cb-4273-a1f2-b92a354bd868. But again it is not fool proof.

I'm not familiar with Azure, so I don't know if there is an existing API for that. But it's not hard to code:
encode:
const string escapeChar='|';
RowKey.Replace(escapeChar,escapeChar+escapeChar).Replace("/",escapeChar+"S");
decode:
StringBuilder sb=new StringBuilder(s.Length);
bool escape=false;
foreach(char c in s)
{
if(escape)
{
if(c=='S')
sb.Append('/');
else if(c==escapeChar)
sb.Append(escapeChar);
else
throw new ArgumentException("Invalid escape sequence "+escapeChar+c);
}
else if(c!=escapeChar)
{
sb.Append(c);
escape=false;
}
else
escape=true;
return sb.ToString();

When a string is Base64 encoded, the only character that is invalid in an Azure Table Storage key column is the forward slash ('/'). To address this, simply replace the forward slash character with another character that is both (1) valid in an Azure Table Storage key column and (2) not a Base64 character. The most common example I have found (which is cited in other answers) is to replace the forward slash ('/') with the underscore ('_').
private static String EncodeToKey(String originalKey)
{
var keyBytes = System.Text.Encoding.UTF8.GetBytes(originalKey);
var base64 = System.Convert.ToBase64String(keyBytes);
return base64.Replace('/','_');
}
When decoding, simply undo the replaced character (first!) and then Base64 decode the resulting string. That's all there is to it.
private static String DecodeFromKey(String encodedKey)
{
var base64 = encodedKey.Replace('_', '/');
byte[] bytes = System.Convert.FromBase64String(base64);
return System.Text.Encoding.UTF8.GetString(bytes);
}
Some people have suggested that other Base64 characters also need encoding. According to the Azure Table Storage docs this is not the case.