I have a ASP.NET MVC controller which is making call to another service using HttpClient class.
var url = "some url";
var client = new HttpClient();
var result= client.GetAsync(url);
The URL I am sending contains some special characters. How can encode special characters in ASP.NET MVC controller?
Thanks!!1
Try this:
url = HttpUtility.UrlEncode(url);
As you are considering a URL that will be used as such for a request (with HttpClient.getAsync) -- not as an argument within another URL -- you should use Uri.EscapeUriString.
Here is a comparison of three methods for the following URL:
var url = "http://some url?data=x y+z&user=1#ok";
HttpUtility.UrlEncode
Console.WriteLine(HttpUtility.UrlEncode(url));
http%3a%2f%2fsome+url%3fdata%3dx+y%2bz%26user%3d1%23ok
Obviously, this is not desired: the URL got damaged with / escaped, a + entered in the path, ...etc. The method seems useful for the query part of the URL, but not for the whole lot.
HttpUtility.UrlPathEncode
Console.WriteLine(HttpUtility.UrlPathEncode(url));
http://some%20url?data=x y+z&user=1#ok
This looks useful, although the space is a bit of a problem in the query part (notice the broken hyperlinking here, although browser can deal with it). But more importantly, the method is being deprecated:
Do not use; intended only for browser compatibility. Use UrlEncode.
Uri.EscapeUriString
Console.WriteLine(Uri.EscapeUriString(url));
http://some%20url?data=x%20y+z&user=1#ok
This seems to do the job well: %20 is an escape sequence that all modern browsers should support, also when occurring in the query part of the URL.
There is no need it encoding in Razor View Engine starting from 3rd version, and it's very convenient. Instead if you want to use tags you should use:
#Html.Raw(myString)
So basically just using Razor comes with encoding by default.
You should use HttpUtility.UrlPathEncode
When you use url = HttpUtility.UrlEncode(url) it doesn't work fine with spaces.
Related
If I do this in .Net Core 3.1:
await new HttpClient().GetAsync("http://test.com/page?parameter=%2D%2E%5F%2E%2D");
then this happens:
GET http://test.com/page?parameter=-._.- HTTP/1.1
but this is what I want:
GET http://test.com/page?parameter=%2D%2E%5F%2E%2D HTTP/1.1
The background is that I get a signed Url from a third party and I need to use the url as it is, non-unescaped. I manage to find the resource with the unescaped url, but the signature check fails on the other end because the url they see in the request is not the url that was signed.
I can paste the url into any browser and get the resource, but the signature check fails when I do it programatically in .Net Core 3.1.
The unescaping is supposed to happen according to documentation on the Uri Class:
Escaped characters (also known as percent-encoded octets) that don't
have a reserved purpose are decoded (also known as being unescaped).
These unreserved characters include uppercase and lowercase letters
(%41-%5A and %61-%7A), decimal digits (%30-%39), hyphen (%2D), period
(%2E), underscore (%5F), and tilde (%7E).
I have tried solutions listed in these questions:
GETting a URL with an url-encoded slash. But the schemeSetting seems not to work for .Net Core 3.1 and and neither does the workaround ForceCanonicalPathAndQuery.
How to make System.Uri not to unescape %2f (slash) in path?. Again schemeSetting seems not to work for .Net Core 3.1, and neither does the workaround LeaveDotsAndSlashesEscaped.
So, does anyone know how I can use the signed url as is, non-unescaped, on .Net Core 3.1?
So after fiddling around a bit I came up with this:
private static Uri CreateNonUnescapedUri(string url)
{
// Initiate Uri as e.g "http://test.com" so internal flags will indicate that the url does not include characters that needs unescaping
int offset = url.IndexOf("://");
offset = url.IndexOf('/', offset + 4);
var uri = new Uri(url.Substring(0, offset));
// Then replace internal field with complete url that would otherwise be unescaped
typeof(Uri).GetField("_string", BindingFlags.Instance | BindingFlags.NonPublic).SetValue(uri, url);
return uri;
}
I tested it on 300 signed url's.
Offcourse changing the internal state of the Uri which is 5600 lines of pure madness is bound to fail in the future, but I need this working by monday and this is what I've got. Let me know if anyone has a real solution.
Edit April 2022:
In .Net 6 there is a new constructor that will keep the original url as is, using UriCreationOptions:
var uri = new Uri("http://test.com/page?parameter=%2D%2E%5F%2E%2D",
new UriCreationOptions { DangerousDisablePathAndQueryCanonicalization = true });
I have no idea whats supposedly dangerous about it though.
For .Net Core 3.1 I'm still using the hack above, I never did find a better solution for it.
I add route to project.All work correct, but if i send normal url as first param its not correctly work.
Get["/{url}/{digit}"
If i send this params to server-all work correctly.
localhost:8888/google.com/2
But if i send param with http://www its not work.
localhost:8888/https://www.google.com/2
How correct pass url param to route? I think it because Nancy think that i send 3 input param.
If you really need to use GET instead of POST try HttpUtility.UrlEncode("https://google.com/2") to urlencode your url.
You have to encode your url which is send as a paramater:
Use:
var encodedString = Uri.EscapeDataString("https://www.google.com/2");
then your url will look like this and it shouldn't get any errors:
https%3A%2F%2Fwww.google.com%2F2
Sending the request:
localhost:8888/https%3A%2F%2Fwww.google.com%2F2
Or ou can use the
HttpUtility.UrlEncode();
method. For further information have a look at this.
Since you insist on changing the backend only, you could try using a regex to capture your route
Get["^(?<url>.*<digit>[0-9]+)$"]
This should match any url ending with atleast one number, and put everything before it in url like so:
Get["^(?<url>.*<digit>[0-9]+)$"] = parameters =>
{
var url = parameters.url;
var digit = parameters.digit;
};
I am currently unable to verify if this works as you want it to though, and to make sure you can adjust this yourself make sure to look into how to write regex
I'm using the Amazon .NET SDK to generate a pre-signed URL like this:
public System.Web.Mvc.ActionResult AsActionResult(string contentType, string contentDisposition)
{
ResponseHeaderOverrides headerOverrides = new ResponseHeaderOverrides();
headerOverrides.ContentType = contentType;
if (!string.IsNullOrWhiteSpace(contentDisposition))
{
headerOverrides.ContentDisposition = contentDisposition;
}
GetPreSignedUrlRequest request = new GetPreSignedUrlRequest()
.WithBucketName(bucketName)
.WithKey(objectKey)
.WithProtocol(Protocol.HTTPS)
.WithExpires(DateTime.Now.AddMinutes(6))
.WithResponseHeaderOverrides(headerOverrides);
string url = S3Client.GetPreSignedURL(request);
return new RedirectResult(url, permanent: false);
}
This works perfectly, except if my contentType contains a + in it. This happens when I try to get an SVG file, for example, which gets a content type of image/svg+xml. In this case, S3 throws a SignatureDoesNotMatch error.
The error message shows the StringToSign like this:
GET 1234567890 /blah/blabh/blah.svg?response-content-disposition=filename="blah.svg"&response-content-type=image/svg xml
Notice there's a space in the response-content-type, where it now says image/svg xml instead of image/svg+xml. It seems to me like that's what is causing the problem, but what's the right way to fix it?
Should I be encoding my content type? Enclose it within quotes or something? The documentation doesn't say anything about this.
Update
This bug has been fixed as of Version 1.4.1.0 of the SDK.
Workaround
This is a confirmed bug in the AWS SDK, so until they issue a fix I'm going with this hack to make things work:
Specify the content type exactly how you want it to look like in the response header. So, if you want S3 to return a content type of image/svg+xml, set it exactly like this:
ResponseHeaderOverrides headerOverrides = new ResponseHeaderOverrides();
headerOverrides.ContentType = "image/svg+xml";
Now, go ahead and generate the pre signed request as usual:
GetPreSignedUrlRequest request = new GetPreSignedUrlRequest()
.WithBucketName(bucketName)
.WithKey(objectKey)
.WithProtocol(Protocol.HTTPS)
.WithExpires(DateTime.Now.AddMinutes(6))
.WithResponseHeaderOverrides(headerOverrides);
string url = S3Client.GetPreSignedURL(request);
Finally, "fix" the resulting URL with the properly URL encoded value for your content type:
url = url.Replace(contentType, HttpUtility.UrlEncode(contentType));
Yes, it's a dirty workaround but, hey, it works for me! :)
Strange indeed - I've been able reproduce this easily, with the following observed behavior:
replacing + in the the URL generated by GetPreSignedURL() with its encoded form %2B yields a working URL/signature
this holds true, no matter whether / is replaced with its encoded form %2F or not
encoding the contentType upfront before calling GetPreSignedURL(), e.g. via the HttpUtility.UrlEncode Method, yields invalid signatures regardless of any variation of the generated URL
Given how long this functionality is available already, this is somewhat surprising, but I'd still consider it to be a bug - accordingly it might be best to inquiry about this in the Amazon Simple Storage Service forum.
Update: I just realized you asked the same question there already and the bug got confirmed indeed, so the correct answer can be figured out over time by monitoring the AWS team response ;)
Update: This bug has been fixed as of Version 1.4.1.0 of the SDK.
I want to crawl/scrape a webpage which has a form
to be precise following is the URL
http://lafayetteassessor.com/propertysearch.cfm
The problem is, i want to make a search and save the result in a webpage.
my search string will always give a unique page, so result count won't be a problem.
the search over there doesn't search on URL (e.g. google searching url contains parameters to search). How can i search from starting page (as above) and get the result page ?
please give me some idea.
I am using C#/.NET.
If you look at the forms on that page, you will notice that they use the POST method, rather than the GET method. As I'm sure you know, GET forms pass their parameters as part of the URL, eg mypage?arg1=value&arg2=value
However, for POST requests, you need to pass the parameters as the request body. It takes the same format, it's just passed in differently. To do this, use code similar to this:
HttpRequest myRequest = (HttpRequest)WebRequest.Create(theURL);
myRequest.Method = "post";
using(TextWriter body = new StreamWriter(myRequest.GetRequestStream())) {
body.Write("arg1=value1&arg2=value2");
}
WebResponse theResponse = myRequest.GetResponse();
//do stuff with the response
Don't forget that you still need to escape the arguments, etc.
I'm using HttpListener to provide a web server to an application written in another technology on localhost. The application is using a simple form submission (application/x-www-form-urlencoded) to make its requests to my software. I want to know if there is already a parser written to convert the body of the html request document into a hash table or equivalent.
I find it hard to believe I need to write this myself, given how much .NET already seems to provide.
Thanks in advance,
You mean something like HttpUtility.ParseQueryString that gives you a NameValueCollection? Here's some sample code. You need more error checking and maybe use the request content type to figure out the encoding:
string input = null;
using (StreamReader reader = new StreamReader (listenerRequest.InputStream)) {
input = reader.ReadToEnd ();
}
NameValueCollection coll = HttpUtility.ParseQueryString (input);
If you're using HTTP GET instead of POST:
string input = listenerRequest.Url.QueryString;
NameValueCollection coll = HttpUtility.ParseQueryString (input);
The magic bits that fill out HttpRequest.Form are in System.Web.HttpRequest, but they're not public (Reflector the method "FillInFormCollection" on that class to see). You have to integrate your pipeline with HttpRuntime (basically write a simple ASP.NET host) to take full advantage.
If you want to avoid the dependency on System.Web that is required to use HttpUtility.ParseQueryString, you could use the Uri extension method ParseQueryString found in System.Net.Http.
Make sure to add a reference (if you haven't already) to System.Net.Http in your project.
Note that you have to convert the response body to a valid Uri so that ParseQueryString (in System.Net.Http)works.
string body = "value1=randomvalue1&value2=randomValue2";
// "http://localhost/query?" is added to the string "body" in order to create a valid Uri.
string urlBody = "http://localhost/query?" + body;
NameValueCollection coll = new Uri(urlBody).ParseQueryString();