Develop Reference

.NET 5 - Correctly handling expected 404 or 400 errors in a web API

I'm looking for a correct way to handle the expected errors in my web API.
For example, i have a endpoint with 2 possible responses.
200 OK, with a result object
404 Not Found, with a ApiError object, containing a small (localized) message and title.
Currently i'm throwing an custom exception at the place in my code where i know if the resource is a Ok or Not found. I got a custom middleware in place to catch these exceptions and create a nice formatted ApiError.
This works perfect, except all these (known) messages are also logged inside my log files/Application Insights. It feels weird to add a boolean to this exception where i can specify if the exception should be logged. But currently this feels like the only possible way.
Is there any advice that could make this implementation beter or maybe i should change the implementation?

Well, the issue is not .NET 5 - it is application insight.
Your middleware basically should tell AI that those specific errors ARE NOT ERRORS.
https://peter.intheazuresky.com/2020/08/21/let-application-insights-focus-on-real-problems-and-not-missing-data-404-in-your-apis/
Basically, get the ITelemetryInterface from ApplicationInsight and tell of that despite the error code, the request was a success.
Voila, problem solved. AI can not magically know that specific 404 errors are "expected" as normal operations - you will have to help it out.
It makes sense to track them on a website / web application level. It does not make sense to treat them as errors on the backend level. The response (not found) is perfectly valid, from the point of an API. Just configure AI accordingly.

The requested url was rejected

I am working on a simulation site using selenium .
I launched on a server to start testing it,
after a will the targeted url is giving this error message while it is navigated to by chrome
The requested URL was rejected. Please consult with your administrator.
Your support ID is: 8410459054496748680
I tried to navigate to this site as a user and cleaned my browsing history and cache
and it keeps giving me this error

You are block by a firewall ASM.
The data you are trying to pass are detected as threats and your request is denied. You need to resolve this with you Network engineer by providing him the ID.
NOTE: Cookies cleanup might also work sometimes!

Soap Error: "Server was unable to process request" "Object reference not set to an instance of an object"

When I send a SOAP request to my service in the IIS locally, everything works fine.
When I send a SOAP request to the same service that running on IIS on another host, everything works fine.
But when another programmer sends a SOAP request to my service, he generally gets the right response except one method in the service that returns:
<soap:Body>
<soap:Fault>
<faultcode>soap:Server</faultcode>
<faultstring>Server was unable to process request. ---> Object reference not set to an instance of an object.</faultstring>
<detail />
</soap:Fault>
I need to understand why he is receiving this error.
His SOAP request is exactly the same as SOAP Request yet mine works and his does not.

The "Object reference" error is the hardest to troubleshoot if you don't know exactly where the error is occurring. Be sure to test for null values, as the most obvious cause for this error is an object not being initialized or a variable having a null value when it's expected to contain an object or a value.
Also, be sure that you're following best practices for error handling. Try to catch the errors at the earliest possible level unless you intentionally intend to let a higher-level error handler catch the error. The fact that you're getting this error means that the error handling is not sufficient.
Finally, I'd strongly recommend including some sort of logging mechanism for your error handlers. Save the errors to the Event Log, or a database, or even email them - anything, so that you can tell exactly where the error is. Do NOT display the error back to the user.
There are several options for this. Asp.Net Health Monitoring works fine in web services as well as standard asp.net sites. There's also ELMAH, or you can write your own code to do this.
If you follow these guidelines, you won't be in a situation where you don't understand how the error is happening. You'll have the information you need to troubleshoot properly, and your apps will be more solid.

Thanks guys for all your help. I solved the problem.
The 'SoapAction' field in the soap request had the wrong URI
I recommend using a HTTP sniffer. The http sniffer will show the SOAP properties.
The "SOAPAction" property should have something like : http://tempuri.org/YourServiceMethodName

Erratic Invalid Viewstate issue in a .NET application

I seem to be getting a "invalid viewstate" every now and then in the event viewer for my ASP.NET application.
Most of them (95%) seem to be referencing ScriptResource.axd (the application
uses the ASP.NET AJAX library). There is no way I can remove the Ajax library either as Ajax is used everywhere..
How can I reduce these errors? I'm getting ~ 100-200 errors a day and I have no idea how to fix them! They come from different browsers, different IPs and geographical locations.
It's difficult for me to reproduce the problem because it barely even happened to me, it has only happened to me 3-4 times out of the blue.
Error:
Process information:
Process ID: 4004
Process name: w3wp.exe
Account name: NT AUTHORITY\NETWORK SERVICE
Exception information:
Exception type: HttpException
Exception message: Invalid viewstate.
Request information:
Request URL: http://domainnamehere/ScriptResource.axd?d=W1R6x9VzZ2C9SKnIkOmX9VRLhSjJ3nOF1GSQvPwKS3html
Request path: /ScriptResource.axd
User host address: 124.177.170.75
User:
Is authenticated: False
Authentication Type:
Thread account name: NT AUTHORITY\NETWORK SERVICE
Thread information:
Thread ID: 1
Thread account name: NT AUTHORITY\NETWORK SERVICE
Is impersonating: False
Stack trace: at System.Web.UI.Page.DecryptStringWithIV(String s, IVType ivType)
at System.Web.UI.Page.DecryptString(String s)
at System.Web.Handlers.ScriptResourceHandler.DecryptParameter(NameValueCollection queryString)
at System.Web.Handlers.ScriptResourceHandler.ProcessRequestInternal(HttpResponse response, NameValueCollection queryString, VirtualFileReader fileReader)
at System.Web.Handlers.ScriptResourceHandler.ProcessRequest(HttpContext context)
at System.Web.Handlers.ScriptResourceHandler.System.Web.IHttpHandler.ProcessRequest(HttpContext context)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
Custom event details:
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
I also get this error every now and then in my .NET code which happens at the same time which might be related:
Exception raised in GLOBAL.ASAX.Application_Error(): 'Padding is invalid and cannot be removed.' at System.Security.Cryptography.RijndaelManagedTransform.DecryptData(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount, Byte[]& outputBuffer, Int32 outputOffset, PaddingMode paddingMode, Boolean fLast)
at System.Security.Cryptography.RijndaelManagedTransform.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount)
at System.Security.Cryptography.CryptoStream.FlushFinalBlock()
at System.Web.Configuration.MachineKeySection.EncryptOrDecryptData(Boolean fEncrypt, Byte[] buf, Byte[] modifier, Int32 start, Int32 length, IVType ivType, Boolean useValidationSymAlgo)
at System.Web.UI.ObjectStateFormatter.Deserialize(String inputString)

This appears to be the same IE8 issue that many people have been experiencing. What appears to happen is that somehow IE8 (in both IE8 rendering mode and IE7 compatibility mode) will lose 4096 bytes out of the middle of the HTML document and this missing data causes this exception (you usually see this in a ScriptResource or WebResource call).
Here is a Microsoft bug report on the issue:
https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=434997
Also there are plenty of forum, blog etc posts on this issue:
Invalid Webresource.axd parameters being generated
IE 8 dropping memory pages?
http://forums.asp.net/t/1373410.aspx?PageIndex=1
http://forums.asp.net/p/1409964/3085329.aspx
Microsoft has responded to this issue:
Note is a bug in Internet Explorer 8. The Internet Explorer team has been investigating this issue.
Impact: Thus far, we believe the problem has no impact on the end-user's experience with the web application; the only negative effect is the spurious/malformed requests sent by the JavaScript speculative-download engine. When the script is actually needed by the parser, it will properly be downloaded and used at that time.
Circumstances: The spurious-request appears to occur only in certain timing situations, only when a META HTTP-EQUIV tag containing a Content-Type with a CHARSET directive appears in the document, and only when a JavaScript SRC URL spans the 4096th byte of the HTTP response body.
Workaround: Hence, we currently believe this issue can be mitigated by declaring the CHARSET of the page using the HTTP Content-Type header rather than specifying it within the page.
So, rather than putting
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=utf-8">
In your head tag, instead, send the following HTTP response header:
Content-Type: text/html; charset=utf-8
Note that specification of the charset in the HTTP header results in improved performance in all browsers, because the browser's parsers need not restart parsing from the beginning upon encountering the character set declaration. Furthermore, using the HTTP header helps mitigate certain XSS attack vectors.
NOTE: There have been reports that this problem still happens when the META HTTP-EQUIV is not on the page. We will update this comment when we have more investigation.
Posted by Microsoft on 6/30/2009 at 12:25 PM.
Edit:
I still see this exception occasionally, but this bug is reported as being fixed:
Link

I guess you are using ASP.NET AJAX. I am having the same problem. Sporadically I would find this exception in my Event Log, and the requested path is ALWAYS ScriptResource.axd.
Using fixed validationKey and decryptionKey in machineKey did not fix the problem for me.
Based on what I was able to gather, I tend to believe that this error has nothing to do with the ViewState whatsoever; my theory is that for some reason, certain UAs somehow mess up the "d" parameter of the ScriptResource.axd. The problem is easily replicable by requesting the offending path manually. This gives an "Invalid ViewState" exception, even though ViewState doesn't even apply here.
Digging through my logs, I found for example:
This request is served OK (200):
/ScriptResource.axd?d=oFCAB7_vUyp7Hhe9lxZBz37lpoAxhfbWwwdfFy3Zd3z41W_33Y_9Dq6i10g9Q1NRCY1n0_DNg1nE6-DDbsD6r4EiuwoeDzp9mjDDfBNLb1k1&t=41df03cc
This slightly different request is also served OK (200):
/ScriptResource.axd?d=oFCAB7_vUyp7Hhe9lxZBz37lpoAxhfbWwwdfFy3Zd3z41W_33Y_9Dq6i10g9Q1NR5ijsxQts4AfbJdACRwmQ8sHt6UAzui3spEnooPneTz01&t=41df03cc
This request fails with a 500 response and the Invalid ViewState exception:
/ScriptResource.axd?d=oFCAB7_vUyp7Hhe9lxZBz37lpoAxhfbWwwdfFy3Zd3z41W_3products$ctl00$AddToCart1$id
If you look closely, the first few characters on all three request are the same, but the last few characters of the last request (in bold) clearly is Control ID "products$ctl00$AddToCart1$id" (I have a controls named products and AddToCart). I don't know how this ID got there, but in my case this is what is causing all these Invalid ViewState exceptions.
I'm not sure whether this is the same case as the OP or not, but I notice Martin's Request URL ends in "html", which is a bit of a coincidence for a parameter that is supposed to be a key...
I already have a headache thanks to this problem. And so far, the most insightful post I came across is this one http://bytes.com/topic/asp-net/answers/861764-invalid-viewstate-system-string-decryptstringwithiv
Any insights?

Viewstate issues are annoying and frustrating - I've noticed a few people have talked about having Viewstate issues in this thread. So, here are some suggestions you can look at in order.
I'd echo what Freddy Rios has said
in the thread already. Make sure
that you've hardcoded the machine
key. This will solve the vast
majority of these issues. The
important thing about the
ScriptResource link is that it
should have a d parameter and a t
parameter in the querystring. If it
doesn't something else is wrong!
Don't let the user postback until
your done. You could probably do
this with javascript and a bit of
css. From memory, I think there is a
way to do this with a meta tag but
it might be IE only.
I would look at is flushing the
response early. I would think after
the script manager would be best.
But you might need to experiment a
bit.
If your viewstate looks bloated,
turn on GZip compression on in IIS.
If your viewstate has became really
bloated and you can't get GZip
compression turned on/or it has an
undesired side affect. Then you can
compress and uncompress the
viewstate.
http://www.codeproject.com/KB/viewstate/ViewStateCompression.aspx
If that still leaves you with a
bloated viewstate, you could look at
storing the viewstate locally.
http://blog.arctus.co.uk/articles/2007/04/23/advanced-asp-net-storing-viewstate-in-a-database/
is a good starting point.

Use a fixed machine key (even when doing single server).
The issue occurs when using the auto configuration for the machine key, you get a new one each time the app domain is recycled. This affects viewstate validation, dynamic resources query string decryption and authentication tickets [insert other uses of the machine key].

I've seen problems like this when the Viewstate is too large. I've seen it happen becaue of the problem Freddy describes.
I generally dislike the idea of using Viewstate. Can you turn Viewstate off altogether?

I am also having this issue, and I've tried everything mentioned in all the blogs I've found (fixed machine key, viewstate size, etc). 99% of the time the error is logged on requests to ScriptResource.axd. I am using .net 3.5 SP1, on Win 2003 server. The app is hosted on two parallel identical servers, balanced 50/50. Each server has the same machine key.
Normally this error does not concern me much, however, over a 3 month period, the trend on occurance has been going way up.
Does anyone think this error is related to the Viewstate not being UrlEncoded/HtmlEncoded or UrlDecoded correctly. Perhaps there is character subset within the viewstate that some browsers are replacing with some encoded value. I'm not sure if that even makes sense..

I think, you must use in aspx page:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
This solve my problem

Are you running a non-english Operation System?
For some reasons, the account name of "NT Authority\Network Service" has been localized in other languages.
Sadly, a lot of programs have the account name hard coded to the english name, and won't find the Network Service when running on foreign versions of Windows, leading to all kind of funky errors in the event log.

I have just narrowed this issue down for me to a user with Trend Micro antivirus and the errors just started to occur after he updated his trend micro software on 5/21/2009. No errors before this date.

The problem seems to be the lookahead downloader in IE8.
It seems to only affect IE8 in a fairly obscure set of circumstances. One of the reasons it can go unnoticed is that a 4k chunk of data appended to a URL is often discarded by the server. One of the things that seems to make it more likely is a slow network connection. Someone in one of the below resources noted that he only had the issue with his clients in New Zealand.
Lots of people explaining two separate problems, one of which is described in the question above (malformed URLs sent to server):
http://connect.microsoft.com/VisualStudio/feedback/details/434997/invalid-webresource-axd-parameters-being-generated
Article explaining that the lookahead downloader is fixed:
http://blogs.msdn.com/b/ieinternals/archive/2010/04/01/ie8-lookahead-downloader-fixed.aspx
KB980182 – Cumulative update in which issue is fixed:
http://support.microsoft.com/kb/980182
NOTE: Because the script is automatically re-downloaded if it couldn’t be retrieved by the lookahead downloader, it should be possible to change back to the old validation mode in which .axd files were not checked for validity and remove the symptoms of the issue:
<httpRuntime requestValidationMode="2.0" />

Posting forms to a 404 + HttpHandler in IIS7: why has all POST data gone missing?

OK, this might sound a bit confusing and complicated, so bear with me.
We've written a framework that allows us to define friendly URLs. If you surf to any arbitrary URL, IIS tries to display a 404 error (or, in some cases, 403;14 or 405). However, IIS is set up so that anything directed to those specific errors is sent to an .aspx file. This allows us to implement an HttpHandler to handle the request and do stuff, which involves finding the an associated template and then executing whatever's associated with it.
Now, this all works in IIS 5 and 6 and, to an extent, on IIS7 - but for one catch, which happens when you post a form.
See, when you post a form to a non-existent URL, IIS says "ah, but that url doesn't exist" and throws a 405 "method not allowed" error. Since we're telling IIS to redirect those errors to our .aspx page and therefore handling it with our HttpHandler, this normally isn't a problem. But as of IIS7, all POST information has gone missing after being redirected to the 405. And so you can no longer do the most trivial of things involving forms.
To solve this we've tried using a HttpModule, which preserves POST data but appears to not have an initialized Session at the right time (when it's needed). We also tried using a HttpModule for all requests, not just the missing requests that hit 404/403;14/405, but that means stuff like images, css, js etc are being handled by .NET code, which is terribly inefficient.
Which brings me to the actual question: has anyone ever encountered this, and does anyone have any advice or know what to do to get things working again? So far someone has suggested using Microsoft's own URL Rewriting module. Would this help solve our problem?
Thanks.

Microsoft released a hotfix for this :
http://support.microsoft.com/default.aspx/kb/956578

Since IIS7 uses .net from the top down there would not be any performance overhead of using an HttpModule, In fact there are several Managed HttpModules that are always used on every request. When the BeginRequest event is fired, the SessionStateModule may not have been added to the Modules collection, so if you try to handle the request during this event no session state info will be available. Setting the HttpContext.Handler property will initialize the session state if the requested handler needs it, so you can just set the handler to your fancy 404 page that implements IRequiresSessionState. The code below should do the trick, though you may need to write a different implementation for the IsMissing() method:
using System.Web;
using System.Web.UI;
class Smart404Module : IHttpModule
{
public void Dispose() {}
public void Init(HttpApplication context)
{
context.BeginRequest += new System.EventHandler(DoMapping);
}
void DoMapping(object sender, System.EventArgs e)
{
HttpApplication app = (HttpApplication)sender;
if (IsMissing(app.Context))
app.Context.Handler = PageParser.GetCompiledPageInstance(
"~/404.aspx", app.Request.MapPath("~/404.aspx"), app.Context);
}
bool IsMissing(HttpContext context)
{
string path = context.Request.MapPath(context.Request.Url.AbsolutePath);
if (System.IO.File.Exists(path) || (System.IO.Directory.Exists(path)
&& System.IO.File.Exists(System.IO.Path.Combine(path, "default.aspx"))))
return true;
return false;
}
}
Edit: I added an implementation of IsMissing()
Note: On IIS7, The session state module does not run globally by default. There are two options: Enable the session state module for all requests (see my comment above regarding running managed modules for all request types), or you could use reflection to access internal members inside System.Web.dll.

The problem in IIS 7 of post variables not being passed through to custom error handlers is fixed in service pack 2 for Vista. Haven't tried it on Windows Server but I'm sure it will be fixed there too.

Just a guess: the handler specified in IIS7's %windir%\system32\inetsrv\config\applicationhost.config which is handling your request is not allowing the POST verb to get through at all, and it is evaluating that rule before determining whether the URL doesn't exist.

Yes, I would definitely recommend URL rewriting (using Microsoft's IIS7 one or one of the many alternatives). This is specifically designed for providing friendly URLs, whereas error documents are a last-ditch backstop for failures, which tends to munge the incoming data so it may not be what you expect.