Develop Reference

Prevent unescaping url in outbound request

If I do this in .Net Core 3.1:
await new HttpClient().GetAsync("http://test.com/page?parameter=%2D%2E%5F%2E%2D");
then this happens:
GET http://test.com/page?parameter=-._.- HTTP/1.1
but this is what I want:
GET http://test.com/page?parameter=%2D%2E%5F%2E%2D HTTP/1.1
The background is that I get a signed Url from a third party and I need to use the url as it is, non-unescaped. I manage to find the resource with the unescaped url, but the signature check fails on the other end because the url they see in the request is not the url that was signed.
I can paste the url into any browser and get the resource, but the signature check fails when I do it programatically in .Net Core 3.1.
The unescaping is supposed to happen according to documentation on the Uri Class:
Escaped characters (also known as percent-encoded octets) that don't
have a reserved purpose are decoded (also known as being unescaped).
These unreserved characters include uppercase and lowercase letters
(%41-%5A and %61-%7A), decimal digits (%30-%39), hyphen (%2D), period
(%2E), underscore (%5F), and tilde (%7E).
I have tried solutions listed in these questions:
GETting a URL with an url-encoded slash. But the schemeSetting seems not to work for .Net Core 3.1 and and neither does the workaround ForceCanonicalPathAndQuery.
How to make System.Uri not to unescape %2f (slash) in path?. Again schemeSetting seems not to work for .Net Core 3.1, and neither does the workaround LeaveDotsAndSlashesEscaped.
So, does anyone know how I can use the signed url as is, non-unescaped, on .Net Core 3.1?

So after fiddling around a bit I came up with this:
private static Uri CreateNonUnescapedUri(string url)
{
// Initiate Uri as e.g "http://test.com" so internal flags will indicate that the url does not include characters that needs unescaping
int offset = url.IndexOf("://");
offset = url.IndexOf('/', offset + 4);
var uri = new Uri(url.Substring(0, offset));
// Then replace internal field with complete url that would otherwise be unescaped
typeof(Uri).GetField("_string", BindingFlags.Instance | BindingFlags.NonPublic).SetValue(uri, url);
return uri;
}
I tested it on 300 signed url's.
Offcourse changing the internal state of the Uri which is 5600 lines of pure madness is bound to fail in the future, but I need this working by monday and this is what I've got. Let me know if anyone has a real solution.
Edit April 2022:
In .Net 6 there is a new constructor that will keep the original url as is, using UriCreationOptions:
var uri = new Uri("http://test.com/page?parameter=%2D%2E%5F%2E%2D",
new UriCreationOptions { DangerousDisablePathAndQueryCanonicalization = true });
I have no idea whats supposedly dangerous about it though.
For .Net Core 3.1 I'm still using the hack above, I never did find a better solution for it.

server.htmlencode of asp.net equivalent in asp.net mvc

I have a ASP.NET MVC controller which is making call to another service using HttpClient class.
var url = "some url";
var client = new HttpClient();
var result= client.GetAsync(url);
The URL I am sending contains some special characters. How can encode special characters in ASP.NET MVC controller?
Thanks!!1

Try this:
url = HttpUtility.UrlEncode(url);

As you are considering a URL that will be used as such for a request (with HttpClient.getAsync) -- not as an argument within another URL -- you should use Uri.EscapeUriString.
Here is a comparison of three methods for the following URL:
var url = "http://some url?data=x y+z&user=1#ok";
HttpUtility.UrlEncode
Console.WriteLine(HttpUtility.UrlEncode(url));
http%3a%2f%2fsome+url%3fdata%3dx+y%2bz%26user%3d1%23ok
Obviously, this is not desired: the URL got damaged with / escaped, a + entered in the path, ...etc. The method seems useful for the query part of the URL, but not for the whole lot.
HttpUtility.UrlPathEncode
Console.WriteLine(HttpUtility.UrlPathEncode(url));
http://some%20url?data=x y+z&user=1#ok
This looks useful, although the space is a bit of a problem in the query part (notice the broken hyperlinking here, although browser can deal with it). But more importantly, the method is being deprecated:
Do not use; intended only for browser compatibility. Use UrlEncode.
Uri.EscapeUriString
Console.WriteLine(Uri.EscapeUriString(url));
http://some%20url?data=x%20y+z&user=1#ok
This seems to do the job well: %20 is an escape sequence that all modern browsers should support, also when occurring in the query part of the URL.

There is no need it encoding in Razor View Engine starting from 3rd version, and it's very convenient. Instead if you want to use tags you should use:
#Html.Raw(myString)
So basically just using Razor comes with encoding by default.

You should use HttpUtility.UrlPathEncode
When you use url = HttpUtility.UrlEncode(url) it doesn't work fine with spaces.

System.Net.WebRequest.Create and URL encoding

I have to make a call to an API and the URL hasd this format: http://apiurl.com/method/parameter1/parameter2
Now, parameter2 may contain characters that could mess with the URL such as forward slashes, so I URL encode parameter2:
string parameter2 = HttpUtility.UrlEncode("string with a / in it");
System.Net.WebRequest request = System.Net.WebRequest.Create(String.Format("http://apiurl.com/method/{0}/{1}", parameter1, parameter2));
The problem is that if I inspect the URL of the request, parameter2 is NOT URL encoded and the call fails (unsurprisingly). However, when I inspect the value of parameter2 it is correctly URL encoded with the forward slash replaced with %2f
I am using .NET 3.5 so many of the usual workarounds don't apply (genericUriParserOptions for example is .NET 4 and above only)
Thanks

Encode header value using WebClient.Headers.Add()

I am using HTTP headers to send a string which contains Unicode characters (such as ñ) to a custom http server.
When I add the string as a header:
webClient.Headers.Add("Custom-Data", "señor");
It is interpreted by the server as:
se�or
Obviously I need to encode the value differently, but I am unsure what encoding to use.
How should I encode this HTTP header to preserve extended/special characters?
porneL's answer to a related question is confusing.
Unanswered, related: C# WebClient non-english request header value encoding

As #Jordan suggested, representing the string as base64 (with UTF8 encoding) worked well:
On the client side:
webClient.Headers.Add("Custom-Data",
Convert.ToBase64String(Encoding.UTF8.GetBytes("señor")));
And on the server:
string customData = Encoding.UTF8.GetString(Convert.FromBase64String(customHeader.Value));

Creating an Uri in .NET automatically urldecodes all parameters from passed string

Suppose I want to create an Uri object from the following string:
string url = #"http://someserver.com?param1=1&url=http%3a%2f%2fwww.otherserver.com";
Uri uri = new Uri(url, UriKind.Absolute);
Expected result would be:
http://someserver.com?param1=1&url=http%3a%2f%2fwww.otherserver.com
Obtained:
http://someserver.com/?param1=1&url=http://www.otherserver.com
The same behavior is noticed in many related methods that allow Uri creation: Uri.TryCreate, UriBuilder.Uri, etc.
How would I get an Uri that preserve initial encoded parameter?

In .NET4 you can disable Uri compaction for certain scheme via a configuration:
<configuration>
<uri>
<schemeSettings>
<add name="http" genericUriParserOptions="DontUnescapePathDotsAndSlashes"/>
</schemeSettings>
</uri>
</configuration>
Note that there are security implications related to disabling of the default behaviour.

How did you "obtain" the URL? If I hover my mouse over it in Visual Studio, it indeed shows the decoded URL.
But whenever I access it through the AbsoluteUri property, it shows the encoded URL.

This behavior is documented:
As part of canonicalization in the constructor for some schemes,
escaped representations are compacted. The schemes for which URI will
compact escaped sequences include the following: file, http, https,
net.pipe, and net.tcp. For all other schemes, escaped sequences are
not compacted. For example: if you percent encode the two dots ".." as
"%2E%2E" then the URI constructor will compact this sequence for some
schemes. For example, the following code sample shows a URI
constructor for the http scheme.
So one workaround might be temporarily using a custom scheme (e.g. leavemealone://) to construct the URL objects (possibly through UriBuilder?).

In my case I solved it by returning ToString() method of UriBuilder class instead of using Uri property of the same class.

I recommend using the library Flurl. It allows a fluent and correct build and analysis of URIs. See their examples. https://flurl.dev/docs/fluent-url/
It is available as NuGet package. You don't need the HTTP-Stuff
using Flurl;
var url = "http://www.some-api.com"
.AppendPathSegment("endpoint")
.SetQueryParams(new {
api_key = ConfigurationManager.AppSettings["SomeApiKey"],
max_results = 20,
q = "Don't worry, I'll get encoded!"
})
.SetFragment("after-hash");