Active connections number reached maximum Mongo Atlas - c#

I have an .NetCore application. My DB is Mongo and hosted on Mongo Atlas. I reach the number of maximum connections very easy. I have a lot of request made to the db but my connection with the DB is Singletone.
I connect to Mongo through Mongo Client
MongoClient client = new MongoClient(config.GetConnectionString("Dbconnectionstring"));
IMongoDatabase database = client.GetDatabase("database");
All this in a class that is registered as a singletone in Startup.cs
services.AddSingleton<MongoUnitOfWork>();
I did not have this problem before. On this current application I have to make a lot of requests to the db but if I registered the connection as I singleton I expected to use the same connection and not open a new one for every request. At least this is what I think it happens.
UPDATE
I used a recurrent service named HangFire (intended to use for recurring jobs) that needed a connection string to the DB. Even though I didn't put any function to be automatically called every 1 minute or something similar, I think this was the problem. For now I commented every thing about this service and everything went back to normal.
I will get back as soon as I'm 100% sure that this was the problem.
UPDATE 2
Today I had the problem again so I figured that the Recurrent Service wasn't the problem.
UPDATE 3
Last time when I got rid of that service I also deleted all the connections. Until now I didn't have another problem and when I checked them in the day that I deleted the service all seemed fine. Today I checked again the connections and I had around 70. Apparently the connections don't close.
Also, yes, I'm sure I'm using a singleton to instantiate the MongoClient. I also put breakpoints to see if the they are hit on request. Not the requests are the problem.

Related

Apparent delay in Azure KeyVault access

We have an Azure-based ASP.NET Web Service that accesses an Azure KeyVault. We are seeing two instances in which a method "hangs" on a first try, and then works a minute or so later.
In both instances, a KeyVault access occurs. In both instances the problem started when we started using the KeyVault in these methods.
We have done very careful logging in the first instance, and cannot see anything else in our code that could cause the hang. The KeyVault access is the primary suspect.
In addition, if we run the app from our local servers (from Visual Studio), the KeyVault access works fine on the "first try". It only produces the "hang" error when it runs in production on Azure, and only on that "first try".
By "hang" I mean that in one instance, which is triggered by an external API, it takes at least 60 seconds (we can tell that because the external API times out.) In the other instance, which is triggered by a page request, several minutes can pass and the page just spins, at which point we assume the DB request or something else has timed out.
When I say "a minute or so later", that's as fast as we have timed the retry.
Is there some kind of issue or function where the KeyVault needs to be "warmed up" before it works on the first try?
Update: I'm looking at the code more carefully, and I see at least a couple of places where we can insert still more logging to get a more exact picture of where the failure occurs. I'm going to do that, and then I'll report back here.
Update: See answer below - major newbie error, has been corrected.
Found the problem, and the solution.
Key Vault access needs to be called from an async task, because there is a multi-second delay.
private async Task<string> GetKeyVaultSecretValue(varSecretParms) {
I don't understand the underlying technology, however, apparently, if the call is from within a standard code sequence, the server doesn't like to wait, and so the thread is abandoned/halts.
According to your description, it seems that it dues to WebApp that does not enable Always on .
By default, web apps are unloaded if they are idle for some period of time. This lets the system conserve resources. In Basic or Standard mode, you can enable Always On to keep the app loaded all the time
If possible, please have a try to enable Always on and try it again.

ODP.Net OracleConnection.Open cold start very slow

I have a WebAPI service which connects to an Oracle database using Oracle.ManagedDataAccess.dll. Each time after a reset of the application pool (or a deployment) there is a long delay on the first OracleConnection.Open() statement. It's typically around 8 seconds. Subsequent calls are around ~0.5 seconds each.
After reading lots of suggestions regarding server OS and networking issues, i have narrowed it down to the oracle client itself. If I remote debug my code, set a breakpoint on the open statement, and then run Sysinternals Process Monitor i can confirm that the first open statement produces 544 entries, second and subsequent tests produce 2 entries.
The entries are quite random, but mostly relate to Cryptography. A quick overview of the logs:
RegOpenKey, HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider Types\Type 001
RegOpenKey, HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider\Microsoft Strong Cryptographic Provider
RegSetInfoKey, HKLM\SOFTWARE\Microsoft\Bryptography\MachineGuid
These are repeated several times, then there are sections like below:
RegQueryValue, HKLM\System\CurrentControlSet\WinSock2\Parameters\Protocol_Catalog9
RegCreateKey, HKLM\System\CurrentControlSet\Services\Tcpip\Parameters
RegCreateKey, HKLM\System\CurrentControlSet\Services\DnsCache\Parameters
RegOpenKey, HKLM\SOFTWARE\Policies\Microsoft\Windows NT\DNSClient
Then there are several reads of the machine.config file followed by multiple TCP connect and receive to the oracle port 1521. Following this is a section reading the time zone from the registry.
My question is, why is the oracle client doing all of this at first open? Is there any way i can predetermine the answer to some of these questions? (like configure the time zone so it doesn't have to 'ask' Oracle for it)?
Only time I have seen something like this was when the address in the tns connect descriptor was not fully qualified, ie host=computername, instead of host=computername.domain.com.
Issue is likely dns resolution as it goes thru suffixes. I imagine you could put in an ip and eliminate dns altogether as a test. Consider posting your tns entry and connection string as well.
FYI, a lot of things are happening when that first connection is created, ie the pool is established and connections are actually opened vs just fetched from the pool, initial parameters for self-tuning are initialized, etc, so i think number of reg reads is probably a red hearing.

Properly shutting down MongoDB database connection from C# 2.1 driver?

I am just getting started with integrating MongoDB into my application and I have ran into a few questions. In my application I am using the newest 2.1 version of the MongoDB C# driver and only using MongoDB for application logging.
Currently before showing my main application Form I first check to see if mongod.exe is running and if not I start it. Then when my main Form is shown it opens a connection to the database for use seen below.
public void Open()
{
Client = new MongoClient("mongodb://localhost:27017");
Database = Client.GetDatabase(DBName);
Collection = Database.GetCollection<BsonDocument>(ColName);
}
My question is how I should properly shutdown this connection when my application is closing?
Also are there in considerations I should take into account in leaving mongod.exe running versus exiting it each time the application closes?
I have searched a few times trying to figure out if there is a proper way to shutdown the connection but have found nothing very specific. There is an old SO post (that I can't seem to find now) mentioning a .Dispose method, though I cannot seem to find it in the newest driver nor from my IDE's auto complete.
As of today's version of MongoDB (v2.0.1.27 for MongoDB.Driver), there's no need to close or dispose of connections. The client handles it automatically.
From the docs:
A MongoClient object will be the root object. It is thread-safe and is all that is needed to handle connecting to servers, monitoring servers, and performing operations against those servers.
[...]
It is recommended to store a MongoClient instance in a global place, either as a static variable or in an IoC container with a singleton lifetime. However, multiple MongoClient instances created with the same settings will utilize the same connection pools underneath.
There's a partial/old list of thread-safe MongoDB classes in this SO answer.
The question seems to have been already kinda asked here at When should i be opening and closing MongoDB connections?
If it's accepted answer,
I would leave the connection open as re-creating the connection is
costly. Mongo is fine with lots of connections, open for a long time.
What you ideally should do is to share the connection with all parts
of your application as a persistent connection. The C# driver should
be clever enough to do this itself, so that it does not create too
many connections, as internally it uses "connection pooling" that
makes it even re-use connections. The docs say: "The connections to
the server are handled automatically behind the scenes (a connection
pool is used to increase efficiency)."
works fine for you then all well and good. Even the MongoDB C# driver's quick tour page lends the same advice -
Typically you only create one MongoClient instance for a given cluster
and use it across your application. Creating multiple MongoClients
will, however, still share the same pool of connections if and only if
the connection strings are identical.
Otherwise, I think you can simply put your call to create the connection in a using(){} code block. It automatically calls the dispose method for you (as it implements the IDisposable pattern). You should use this block for any resource you want disposed.
From my experience, the correct way is as answered, but even following these recommendations, I still was having random EndOfStreamException. It seems that some problems are caused by the internet provider closing the connection after some time.
I Solved it by adding:
MongoClientSettings settings = MongoClientSettings.FromUrl(new MongoUrl(connectionString));
settings.SslSettings = new SslSettings() { EnabledSslProtocols = SslProtocols.Tls12 };
settings.MaxConnectionIdleTime = TimeSpan.FromSeconds(30);

Selfhosted Asp.Net WebApi stops receiving reuqests

I have a REST service in a self hosted ASP.Net WebApi application (Console).
Some clients poll the server in specific intervals to fetch new data. In general all is working fine.
The problem is, that the server stops responding to requests after some random duration (~30mins - 2.5 hours). All client requests start to time out.
The weird thing is, the server doesn't seem to receive the requests anymore as no controller method is invoked anymore). Server didn't throw any exceptions and the console app is still responsive. So I can only suppose there is a problem, before the request reaches the API controller.
In the debugger everything seems fine.
How can I diagnose such an issue?
What else can I try to fix the described behavior?
Notes:
Tested on multiple systems
.Net 4.5.1
Asp.Net WebApi 5.1.2
I have found the issue, the reason this is happening is because of connection leaks. If you are sending requests and aren't closing them correctly, either after the request is finished, or within an exception, the amount of open connections will eventuelly reach it's max value. Either you change the max amount of open connections in the connectionstring or(the prefered way) make sure your code is handling the closing part:
SqlConnection myConnection = new SqlConnection(ConnectionString);
try
{
conn.Open();
someCall (myConnection);
}
finally
{
myConnection.Close();
}
Credit goes to How can I solve a connection pool problem between ASP.NET and SQL Server? Where you can read more about this.
In my case, the issue was caused by never ending tasks. Due a misusage of the ReactiveExtensions Api, I randomly created never ending tasks. It seems, at some point the task scheduler simply couldn't handle them anymore, although I'm not completely sure about that.
Thing learned: It seems, by doing bad things in your app code (too many tasks, SQL connections ...) you can kill the WebApi infrastructure, so that it doesn't handle requests - at any level - anymore.

ADO.NET How to keep my connection Active

I have created a windows service using c#.NET, The service will updated oracle tables whenever it receives new files. I have kept timer control and the time limit as 30 seconds. I am using ODP.NET as data access layer.
The very first time I will get error, but subsequently the service will work fine. If service is Idle for a long time if it receives a file, I will get "connection lost error", but after if we receives file it will loaded successfully.
Kindly suggest me do I need add any properties in connection string to fix this error?
Hello Karthik Two issues here it seems.
You are best to open and close a new connection each time your service is called.
Windows services quickly go to a latent state if not called and they will respond slower on the next call. If the caller does not have a sufficient timeout value to accomodate this lag then it will return a time out error. If you address these two points you should be fine.

Categories

Resources