Develop Reference

414. The request URL is too long. asp.net

I'm getting the error "HTTP Error 414. The request URL is too long." From the following article, I understand that this is due to a very long query string:
http://www.mytecbits.com/microsoft/iis/query-string-too-long
In web.config, I have maxQueryStringLength="2097151". Is this the maximum value?
In order to solve this problem, should I set maxUrl in web.config? If so, what's the maximum value supported?
What should I do to fix this error?

The GET request should never be this long. You need to change it to POST method instead since it was designed to transmit block of data such as forms.
An excerpt from the RFC 2616: Hypertext Transfer Protocol -- HTTP/1.1:
The POST method is used to request that the origin server accept the
entity enclosed in the request as a new subordinate of the resource
identified by the Request-URI in the Request-Line. POST is designed to
allow a uniform method to cover the following functions:
Annotation of existing resources;
Posting a message to a bulletinboard, newsgroup, mailing list, or
similar group of articles;
Providing a block of data, such as the result of submitting a
form, to a data-handling process;
Extending a database through an appendoperation.

How to properly encode Web API parameter in URL path?

I have endpoint used to find book by book reference.
The book reference is string that can contain white-space and any kind of special characters, e.g. mybook, my book, my-book, my/book, book++
// GET api/books/reference/{reference}
[HttpGet("reference/{reference}")]
public ActionResult<BookItem> FindByReference(string reference)
This is what I get when testing:
GET api/books/reference/mybook
OK
GET api/books/reference/my book
OK
GET api/books/reference/my-book
OK
GET api/books/reference/my+book
404 Not found
GET api/books/reference/my/book
404 Not found
GET api/books/reference/book++
404 Not found
What is a proper way to encode this reference parameter IN THE URL PATH so that it gets properly resolved by routing? Is that even possible?

The encoding of the URLs is the responsibility of the client calling the API, a client must supply a valid URL if it wants a proper response. If you encode your examples you will get:
GET api/books/reference/mybook
GET api/books/reference/my%20book
GET api/books/reference/my-book
GET api/books/reference/my%2Bbook
GET api/books/reference/my%2Fbook
GET api/books/reference/book%2B%2B
These should now work.
If you want help with the actual encoding you will have to edit your question with the client source code.

HTTP 404 Error after encoding Url

When I use HttpUtility.UrlEncode to encode a Url I end up getting a server error.
ASP.Net code:
NavigateUrl=<%# HttpUtility.UrlEncode(string.Concat("UpdateMember.aspx","?groupId=", DataBinder.Eval(Container.DataItem, "GroupID").ToString())) %>
Url:
http://localhost/UITest/MM/UpdateMember.aspx%3fgroupId%3d0032409901
which results in "HTTP Error 404.0 - Not Found
The resource you are looking for has been removed, had its name changed, or is temporarily unavailable."
However using:
NavigateUrl=<%# string.Concat("UpdateMember.aspx","?groupId=", DataBinder.Eval(Container.DataItem, "GroupID").ToString()) %>
results in the Url:
http://localhost/UITest/MM/UpdateMember.aspx?groupId=0032409901
which works out fine. Am I doing something incorrectly?

You shouldn't encode the entire URL, atleast not the 1st "?" symbol. If you encode the ? too then your application looks for a file with the name & extension "UpdateMember.aspx%3fgroupId%3d0032409901" which doesn't exist.
Probably, this is what you should do.
http://localhost/UITest/MM/UpdateMember.aspx?groupId%3d0032409901

HttpUtility.UrlEncode() URL-encodes a string
That means that it escapes all special characters from the string so that you can insert it as part of a URL without any characters being parsed as URL modifiers.
You use this kind of escape function when inserting arbitary text as part of a URL.

Why is my S3 pre-signed request invalid when I set a response header override that contains a "+"?

I'm using the Amazon .NET SDK to generate a pre-signed URL like this:
public System.Web.Mvc.ActionResult AsActionResult(string contentType, string contentDisposition)
{
ResponseHeaderOverrides headerOverrides = new ResponseHeaderOverrides();
headerOverrides.ContentType = contentType;
if (!string.IsNullOrWhiteSpace(contentDisposition))
{
headerOverrides.ContentDisposition = contentDisposition;
}
GetPreSignedUrlRequest request = new GetPreSignedUrlRequest()
.WithBucketName(bucketName)
.WithKey(objectKey)
.WithProtocol(Protocol.HTTPS)
.WithExpires(DateTime.Now.AddMinutes(6))
.WithResponseHeaderOverrides(headerOverrides);
string url = S3Client.GetPreSignedURL(request);
return new RedirectResult(url, permanent: false);
}
This works perfectly, except if my contentType contains a + in it. This happens when I try to get an SVG file, for example, which gets a content type of image/svg+xml. In this case, S3 throws a SignatureDoesNotMatch error.
The error message shows the StringToSign like this:
GET 1234567890 /blah/blabh/blah.svg?response-content-disposition=filename="blah.svg"&response-content-type=image/svg xml
Notice there's a space in the response-content-type, where it now says image/svg xml instead of image/svg+xml. It seems to me like that's what is causing the problem, but what's the right way to fix it?
Should I be encoding my content type? Enclose it within quotes or something? The documentation doesn't say anything about this.

Update
This bug has been fixed as of Version 1.4.1.0 of the SDK.
Workaround
This is a confirmed bug in the AWS SDK, so until they issue a fix I'm going with this hack to make things work:
Specify the content type exactly how you want it to look like in the response header. So, if you want S3 to return a content type of image/svg+xml, set it exactly like this:
ResponseHeaderOverrides headerOverrides = new ResponseHeaderOverrides();
headerOverrides.ContentType = "image/svg+xml";
Now, go ahead and generate the pre signed request as usual:
GetPreSignedUrlRequest request = new GetPreSignedUrlRequest()
.WithBucketName(bucketName)
.WithKey(objectKey)
.WithProtocol(Protocol.HTTPS)
.WithExpires(DateTime.Now.AddMinutes(6))
.WithResponseHeaderOverrides(headerOverrides);
string url = S3Client.GetPreSignedURL(request);
Finally, "fix" the resulting URL with the properly URL encoded value for your content type:
url = url.Replace(contentType, HttpUtility.UrlEncode(contentType));
Yes, it's a dirty workaround but, hey, it works for me! :)

Strange indeed - I've been able reproduce this easily, with the following observed behavior:
replacing + in the the URL generated by GetPreSignedURL() with its encoded form %2B yields a working URL/signature
this holds true, no matter whether / is replaced with its encoded form %2F or not
encoding the contentType upfront before calling GetPreSignedURL(), e.g. via the HttpUtility.UrlEncode Method, yields invalid signatures regardless of any variation of the generated URL
Given how long this functionality is available already, this is somewhat surprising, but I'd still consider it to be a bug - accordingly it might be best to inquiry about this in the Amazon Simple Storage Service forum.
Update: I just realized you asked the same question there already and the bug got confirmed indeed, so the correct answer can be figured out over time by monitoring the AWS team response ;)
Update: This bug has been fixed as of Version 1.4.1.0 of the SDK.

HttpWebRequest long URI workaround?

I've encountered an issue with HttpWebRequest that if the URI is over 2048 characters long the request fails and returns a 404 error even though the server is perfectly capable of servicing a request with a URI that long. I know this since the same URI that causes an error if submitted via HttpWebRequest works fine when pasted directly into a browser address bar.
My current workaround is to allow users to set a compatability flag to say that it's safe to send the parameters as a POST request instead in the case where the URI would be too long but this is not ideal since the protocol I'm using is RESTful and GET should be used for queries. Plus there is no guarentee that other implementors of the protocol will accept POSTed queries
Is there another class in .Net that has equivalent functionality to HttpWebRequest that doesn't suffer from the URI length limit that I could use?
I'm aware of WebClient but I don't really want to use that as I need to be able to fully control the HTTP Headers which WebClient restricts the ability to do.
Edit
Because Shoban asked for it:
http://localhost/BBCDemo/sparql/?query=PREFIX+rdf%3A+%3Chttp%3A%2F%2Fwww.w3.org%2F1999%2F02%2F22-rdf-syntax-ns%23%3E%0D%0APREFIX+rdfs%3A+%3Chttp%3A%2F%2Fwww.w3.org%2F2000%2F01%2Frdf-schema%23%3E%0D%0APREFIX+xsd%3A+%3Chttp%3A%2F%2Fwww.w3.org%2F2001%2FXMLSchema%23%3E%0D%0APREFIX+skos%3A+%3Chttp%3A%2F%2Fwww.w3.org%2F2004%2F02%2Fskos%2Fcore%23%3E%0D%0APREFIX+dc%3A+%3Chttp%3A%2F%2Fpurl.org%2Fdc%2Felements%2F1.1%2F%3E%0D%0APREFIX+po%3A+%3Chttp%3A%2F%2Fpurl.org%2Fontology%2Fpo%2F%3E%0D%0APREFIX+timeline%3A+%3Chttp%3A%2F%2Fpurl.org%2FNET%2Fc4dm%2Ftimeline.owl%23%3E%0D%0ASELECT+*+WHERE+{%0D%0A++++%3Chttp%3A%2F%2Fwww.bbc.co.uk%2Fprogrammes%2Fb00n4d6y%23programme%3E+dc%3Atitle+%3Ftitle+.%0D%0A++++%3Chttp%3A%2F%2Fwww.bbc.co.uk%2Fprogrammes%2Fb00n4d6y%23programme%3E+po%3Ashort_synopsis+%3Fsynopsis-short+.%0D%0A++++%3Chttp%3A%2F%2Fwww.bbc.co.uk%2Fprogrammes%2Fb00n4d6y%23programme%3E+po%3Amedium_synopsis+%3Fsynopsis-med+.%0D%0A++++%3Chttp%3A%2F%2Fwww.bbc.co.uk%2Fprogrammes%2Fb00n4d6y%23programme%3E+po%3Along_synopsis+%3Fsynopsis-long+.%0D%0A++++%3Chttp%3A%2F%2Fwww.bbc.co.uk%2Fprogrammes%2Fb00n4d6y%23programme%3E+po%3Amasterbrand+%3Fchannel+.%0D%0A++++%3Chttp%3A%2F%2Fwww.bbc.co.uk%2Fprogrammes%2Fb00n4d6y%23programme%3E+po%3Agenre+%3Fgenre+.%0D%0A++++%3Fchannel+dc%3Atitle+%3Fchanneltitle+.%0D%0A++++OPTIONAL+{%0D%0A++++++++%3Chttp%3A%2F%2Fwww.bbc.co.uk%2Fprogrammes%2Fb00n4d6y%23programme%3E+po%3Abrand+%3Fbrand+.%0D%0A++++++++%3Fbrand+dc%3Atitle+%3Fbrandtitle+.%0D%0A++++}%0D%0A++++OPTIONAL+{%0D%0A++++++++%3Chttp%3A%2F%2Fwww.bbc.co.uk%2Fprogrammes%2Fb00n4d6y%23programme%3E+po%3Aversion+%3Fver+.%0D%0A++++++++%3Fver+po%3Atime+%3Finterval+.%0D%0A++++++++%3Finterval+timeline%3Astart+%3Fstart+.%0D%0A++++++++%3Finterval+timeline%3Aend+%3Fend+.%0D%0A++++}%0D%0A}&default-graph-uri=&timeout=30000
Which is the following encoded onto the querystring:
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
PREFIX dc: <http://purl.org/dc/elements/1.1/>
PREFIX po: <http://purl.org/ontology/po/>
PREFIX timeline: <http://purl.org/NET/c4dm/timeline.owl#>
SELECT * WHERE {
<http://www.bbc.co.uk/programmes/b00n4d6y#programme> dc:title ?title .
<http://www.bbc.co.uk/programmes/b00n4d6y#programme> po:short_synopsis ?synopsis-short .
<http://www.bbc.co.uk/programmes/b00n4d6y#programme> po:medium_synopsis ?synopsis-med .
<http://www.bbc.co.uk/programmes/b00n4d6y#programme> po:long_synopsis ?synopsis-long .
<http://www.bbc.co.uk/programmes/b00n4d6y#programme> po:masterbrand ?channel .
<http://www.bbc.co.uk/programmes/b00n4d6y#programme> po:genre ?genre .
?channel dc:title ?channeltitle .
OPTIONAL {
<http://www.bbc.co.uk/programmes/b00n4d6y#programme> po:brand ?brand .
?brand dc:title ?brandtitle .
}
OPTIONAL {
<http://www.bbc.co.uk/programmes/b00n4d6y#programme> po:version ?ver .
?ver po:time ?interval .
?interval timeline:start ?start .
?interval timeline:end ?end .
}
}

the protocol I'm using is RESTful and GET should be used for queries.
There's no reason POST can't also be used for queries; for really long request data you have to, as very-long-URIs aren't globally supported, and have never been. This is one area where HTTP does not live up to the REST ideal.
The reason POST generally isn't used on a plain-HTML level is to stop the browser prompting for reloads, and promote eg. bookmarking. But for HttpWebRequest you don't have either of those concerns, so go ahead and POST it. Web applications should use a parameter or a URI path part to distinguish write requests from queries, not merely the request method. (Of course a write request from a GET method should still be denied.)

I don't think HttpWebRequest is actually incompatible with GET URLs of the size you are talking about. I say this based on two things:
In my own work I use HttpWebRequest to send HTTP GET requests longer than 2048 characters without trouble. I'm not sure what my longest ones are, but we're talking 10,000+ characters. (This is primarily between a web application and an instance of Solr running under Tomcat.)
.NET does have some limits on GET URL lengths, but the ones I'm aware of are much higher than 2048 characters. For example, I learned today from my profiler that WebRequest.Create(string url) calls the Uri class constructor, and that is documented to throw a UriFormatException if "the length of uriString exceeds 65534 characters."
I'm not sure where your problem might be, if it's not HttpWebRequest itself. Do you know under what conditions your web service will return HTTP 404 (i.e. "not found")? (I assume the 404 is coming from your web service, rather than being faked inside the depths of .NET.) I'd also want to double-check that the address you're pasting into the browser is actually the same one that's being sent by .NET; as feroze suggested, you should use a network sniffing tool for this. If the two addresses are the same, then maybe next compare how the HTTP headers vary between the .NET case and the browser case. (Incidentally, I personally find Fiddler a bit handier than wireshark for HTTP debugging tasks along these lines.)
See also this somewhat related question: How does HttpWebRequest differ (functional) from pasteing a URL into an address bar?

Here's a snippet which constructs HttpWebRequest instances with bigger and bigger url values until an exception gets thrown:
using System.Net;
...
StringBuilder url = new StringBuilder("http://example.com?p=");
try
{
for (int i = 1; i < Int32.MaxValue; i++)
{
url.Append("0");
HttpWebRequest request = HttpWebRequest.CreateHttp(url.ToString());
}
}
catch (Exception ex)
{
Console.Out.WriteLine("Error occurred at url length: " + url.Length);
Console.Out.WriteLine(ex.GetType().ToString() + ": " + ex.Message);
return;
}
Console.Out.WriteLine("Completed without error!");
On my machine (in LINQPad running .Net 4.5), this snippet outputs:
Error occurred at url length: 65520
System.UriFormatException: Invalid URI: The Uri string is too long.

Your query string is wrong according to RFC3986. '{' and '}' characters are not allowed in a URI.