I have built ASP .NET Web API with SignalR to handle chat feature. The problem is, when client start SignalR connection, the application start some compilation (csc.exe or VBCSCompiler.exe), take 100% CPU usage and block other requests. What is more, the problem exist on all environments except my own and does not happen on all page refreshes - sometimes it connects without compile.
So far I have tried:
Remove my code which handle connections (OnConnected, OnDisconnected, OnReconnected)
Check differences in Web.config files between environments
Block csc.exe or VBCSCompiler.exe with permissions.
Try different SignalR transport method. LongPolling works the same way
Check SignalR connection log on client and server side (here you can see how long it takes to negotiate - 17s)
Check what process take so much time with a profiler. Here you can see StartApplication method which run some compile methods.
Do you have any idea what more should I check or do with that case?
I have found a solution.
The problem was that logs was saving in bin folder. Because of that, every entry in log file caused app pool recycle. You cannot keep your logs in bin folder!
In my case, there was some info entry during connection, and because of that my application was compiling something on almost every request.
Related
I have a WCF service application hosted in IIS. On startup, it reads the SQL connection information from the web.config and connects to the SQL Server to get the necessary information to bootstrap the application. If the SQL Server is unavailable for some reason, we want the service to shutdown - which I currently accomplish by throwing an exception.
The problem we have with this is that, if a service fails to start more than N times, IIS helpfully turns off the application pool - which means that, after the SQL Server connection issue is corrected (possibly entirely external to the IIS server), a sysadmin has to think to go an turn the app pool back on.
What I'd like to do is have the service gracefully exit. I know that there are ways to do this with ASP.Net Core (i.e., ways to hook into the lifecycle management of the process and tell it to exit), but have been unable to find anything like that for .Net Framework/ASP.Net.
Is there such an API that I can use?
Turns out that this method is available and works.
https://learn.microsoft.com/en-us/dotnet/api/system.web.hosting.hostingenvironment.initiateshutdown?view=netframework-4.8
Our Web Application uses an .net-core web api running on a loab balancer and an angular client. We access the DB using EF core.
We have a long running background-task that does a great amount of calculation and takes about 2-3 hours to do so, but will only be initiated by administrators of the application 3-4 times a year.
While the job is running we want to prevent users from adding/editing/deleting data and our client told us its even fine if the application is not avaliable for the duration as they will mostly do it overnight.
The easiest way to do this is to redirect users to an informationpage while the job is running but I have found no way of actually getting to the information if the task is running or not.
I could set a flag whether the job is running or not and just check that flag at every request but I found no way to access an applicationwide state.
I cannot save a flag to the DB because while the transaction is commiting at the end of the job (~1 hour) we cannot read from the DB
What baffles me most is that I have not found a single article or question about a problem like that which doesn't seem to be too outlandish to me, so I guess I'm missing something very obvious.
The simplest way is to store the value for your "Maintenance Mode" in a Singleton class on the server. (No database call needed). The value will remain there for as long as the server is actively running.
If distributed cache (as already mentioned) is not an option, you can run long running task in (uniquely) named transaction and then the check list of active transactions to determine if task is still running.
This is completely dependent on your setup but a simple way to approach this problem might be to make it the long-running job's responsibility to divert traffic from your site while it is running, and then undo that once it is finished.
As an example, if you were running this with an old-school .NET site in IIS the job could drop an app_offline.htm file into the site folder, run, then delete it again. Your setup is different, but if you could do something similar with your load-balancer (configure it to serve some static file instead of routing the requests to your servers) then it could work for you.
I have a project where users upload multiple 4-6MB AccessDb files to migrate into a AWS hosted SQL Server database. I think what is happening is that IIS maybe shutting down the process after some amount of time.
Its basically a file upload and then a c# static class that reads the mdb file and uses EF .core to copy into SQL Server. Using Visual Studio, i don't see any shutdowns and all test files migrate successfully.
Website built using .netcore 2.1 / c#
I have Web.config timeout set to requestTimeout="00:59:00" so that basically stops some 503 errors.
What I'm seeing is that if all browsers are closed(or maybe not) and maybe technically no connections to IIS, that after a few minutes, the migration stops. I'm not seeing new rows added to the database.
I want my c# .netcore processes to continue migrating whatever databases the user uploaded without IIS or whatever other process killing the connection.
Looking for tips I can update. I'm assuming there's other Web.config settings I can update?
It sounds like your code that processes the files is a background job in your application, so I think I know what's going on.
By default, IIS app pools are configured to shut down after 20 minutes of inactivity (no incoming requests). You need to disable that.
The option is called "Idle Time-out (minutes)" in the Advanced Settings of the application pool. Set it to 0 to disable it.
There are also options in there to automatically refresh the app pool, so review those and make sure they are acceptable to you (I forget what the defaults are).
It's not a good design decision to keep an http connection open for that long. First of all asp.net core will drop the thread after a while, and second the user could drop the connection by mistake. I would recommend doing this async to the user:
Receive the file, validate it.
Schedule a background job using either Hangfire or the builtin
IHostedService interface.
When the migration is complete email or notify the user some other
way.
I have a problem I'm fighting for a week now. I have a WCF service running in IIS 8.5 on Windows Server 2012 R2 and a windows service client who is making one or two requests at each 30 seconds. At some point (usually withing two hours of the service running) one of the requests is causing the service app pool (separated from other app pools) process to gain CPU usage. In IIS worker process section can be seen that this request never ends and is hanging in ServiceModel-4 module in AuthenticateRequest state (i.e most likely it is in infinite loop somewhere). At some point another such request is added to the first one, until they become four, staying forever and causing 100 % CPU usage (there are 4 logical processors on the machine). What I did to investigate , fix this problem:
used wcf tracing and custom logging to determine where the problem is. Wcf tracing actually shows all the requests made to the server passed succesfully in milliseconds (!) (at the same time wcf tracing on the client side shows of course time out on the same requests). Custom logging also is showing that the service code is calling returtn of the requested operation. The result of the method are two simple dto objects, so no possible serialization issue and also there are no enpoint behaviors or wathever custom code which is execting before sending reply from the service (except the method code, which, as I mentioned returns successfully).
used iis failed request tracing which shows the request reaching the ServiceModel-4 without continuing with the following information:
ModuleName : ServiceModel-4.0
Notification: AUTHENTICATE_REQUEST
HttpStatus: 500
HttpReason: Internal Server Error
HttpSubStatus: 0
ErrorCode: The operation completed succesfully (0x0)
used Debug Diag for tracing requests continuing more than 10 minutes and saw the threads which are running long time. The stack trace is as follows:
or as follows:
I've seen these are called from iis process. Since thiese are .Net function I suspected first corrupted .Net installation, moreover there were both .Net4.5 and .Net4 installed on the server (which I don't know how exactly could happen). So:
I deinstalled .Net4 and From windows features on/off i turned off .Net4.5 features, restarted and after that i turned them on, restarted, without success
after that I by same way reinstalled the IIS (from Windows features). Again no success.
Does not have any more ideas.
it seems I have found the answer (but havent used Dot Trace or other tools). There was an access to a Generic Dictionary from multiple threads. This seems to be a known problem:
https://blogs.msdn.microsoft.com/tess/2009/12/21/high-cpu-in-net-app-using-a-static-generic-dictionary/
https://blogs.msdn.microsoft.com/asiatech/2009/05/11/asp-net-application-100-cpu-caused-by-system-collections-generic-dictionary/
Actually I noticed this problem in the beggining of the research but ruled it out, because i couldn't reproduce it (probably because I havent't testing the dictionary in iis app, of course I received various exceptions, but not a 100 % Cpu) and mainly because all logs showed that the code, accessing the dictionary has passed, also the stack trace above has nothing to do with the dictionary.
However I think that the problem happened during the serialization of this dictionary (which is data contract) which explains the logged information.
Still cannot explain how this exactly is happening. If anyone can explain it I think it will be a good knowledge for everyone.
Our team has an application in Android, with a .NET c# backend, hosted in IIS.
Recently, we have observed sudden and unexplainable latencies in our customers with the following scenario:
Without any warning, users are enable to change the channel (Zapping) , since the product has to do with Live Media Streaming, and they can not even log out of the application
The mobile application connected to another backend (still a c# backend) , is working properly, without any problem
After some time (which varies from 6 hours of the first incident, to 5 minutes of the last one), it all turns back to normal.
I have enabled Failed Request Tracing logs, to see if I can get anything from there, and I have results as follows:
<failedRequest url="https://ourDNS.com:443/servertime.aspx"
siteId="1"
appPoolId="DefaultAppPool"
processId="22232"
verb="POST"
remoteUserName=""
userName=""
tokenUserName="NT AUTHORITY\IUSR"
authenticationType="anonymous"
activityId="{80013C53-0802-B500-B63F-84710C7967BB}"
failureReason="TIME_TAKEN"
statusCode="200"
triggerStatusCode="0"
timeTaken="45141"
xmlns:freb="http://schemas.microsoft.com/win/2006/06/iis/freb"
>
The page described above is a simple page, that first gets the server's timezone, and then after getting the customer's timezone (that can be set manually from the client), returns the exact date and time of the device where the application is hosted, for further calculations of stream program, what is playing now etc. However, for this page, that returns a simple JSON with a string in it, it requires some times more than 45 seconds (to me this is insane).
Another log from Client side in the moment is one Exception as above:
java.net.SocketTimeoutException
at java.net.PlainSocketImpl.read(PlainSocketImpl.java:491)
at java.net.PlainSocketImpl.access$000(PlainSocketImpl.java:46)
at java.net.PlainSocketImpl$PlainSocketInputStream.read(PlainSocketImpl.java:240)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:103)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:191)
at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:82)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:174)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:180)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:235)
at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:259)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:279)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:121)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:428)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:555)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:487)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:465)
at com.framework.utilityframe.webhelper.HttpRequest.getHttpResponse(HttpRequest.java:316)
at com.framework.utilityframe.webhelper.HttpRequest.httpRequest(HttpRequest.java:393)
at com.tibo.webtv.web.TiboLog.logBufferingError(TiboLog.java:319)
at com.tibo.webtv.CustomVideoView$Buffering_Problem.doInBackground(CustomVideoView.java:324)
at com.tibo.webtv.CustomVideoView$Buffering_Problem.doInBackground(CustomVideoView.java:307)
at android.os.AsyncTask$2.call(AsyncTask.java:287)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:305)
at java.util.concurrent.FutureTask.run(FutureTask.java:137)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1076)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:569)
at java.lang.Thread.run(Thread.java:856)
Reading through different forums, I have seen different causes of performance leaks, starting from database to IIS and even a misconfiguration of the application. I have discarded database as a cause because:
At the moment of the problem, database parameters were absolutely fine, no changes in queries time execution, no waiting tasks, no locking
Secondly, the mobile and Decoder application connect to the same database, and the mobile application is running just fine with the same queries
Now, if I think of IIS, every Application hosted at that AppPool, was running fine and without delays, but still there may be something I am missing over there
And at least, something that makes me suspicious is the fact that the mobile application differs in two ways with the Decoder application:
First, the mobile application takes the responses from the Backend in XML format, the Decoder uses JSON.
Second,the mobile application uses http requests, and the Decoder uses https (SSL)
If anyone has experienced similar issues, their help would be greatly appreciated. And for any other detail you need, just ask and I will provide.
So,
Today, our team made another test, which included :
Application hosted in one server and database in another
Application and database hosted in a completely different server (Azure environment)
In both cases, the result was the same: Latencies and problem at the service.
The problem was neither at the backend nor the server. First, the Java application by mistake executed Sync Tasks when saving the logs to another server(dedicated, with full potential to keep as much data as you can give). Second, the log server had a full HDD, with more than 1 TB of only DB Logs, so when the application executed those Sync Tasks (which came as the first call, before any interaction with the channels), they received the Socket exceptions. So, maybe for someone else who may see this post: PLEASE,ALWAYS CHECK YOUR TASKS IN YOUR APPLICATION,AND ALWAYS CHECK ANY SERVER RELATED TO YOUR APPLICATION!!! Thank you very much :D