I am experiencing a problem where by when connecting to an Oracle11g database using NHibernate, old connections in the pool are not being closed.
I am fairly sure that all the NHibernate sessions are disposed properly, however the connections still remain in an INACTIVE status. I know this will be because of connection pooling, however surley they should be removed after a certain amount of time? If not how can I configure this to happen.
I have tried adding the following settings into my connection string:
Max Pool Size=10;
Connection Lifetime=120;
Incr Pool Size=1;
Decr Pool Size=10;`
This seems to stop as many connections being created, I guess because this increase size is an increment of 1, however once the connections have been put back into the pool they are never closed.
I have looked at the v$session table and some of the LASST_CALL_ET values were as much as 786465s or 9 days!!
I am fairly sure all the sessions are being disposed, here is an example of the code:
public class DoSomethingToDb(ISessionFactory sessionFactory)
{
using (ISession session = sessionFactory.OpenSession())
{
session.Transaction.Begin();
//Do Stuff
session.Transaction.Commit();
}
}
How can I setup my program/NHibernate/Ado.Net/Oracle to close connections that are no longer in use.
The server we were testing on crashed yesterday as there were over 800 INACTVIE connections and no more could be issued.
The reason you are having problems is because your Decr Pool Size value is too large. It will not be able to close any connections unless all of them are available to close since your Decr Pool Size is the same as your Max Pool Size.
When i have set this value to 1, it takes for ever to release unused connections. I am currently setting mine to 5 and it still takes just as long between each decrements, but it will release more at once.
Pooling=true;
Min Pool Size=0;
Max Pool Size=10;
Incr Pool Size=1;
Decr Pool Size=3;
Also, with Connection Lifetime being set to 120, it will not keep any sessions open for more than 2 minutes.
It would surprise me if you could do this in hibernate since I think the connections are leaked connections. For some reason they got out of control and won't ever be reused.
What you can do is configure an session idle timeout in resource manager in the Oracle database. See Managing Resources with Oracle Database Resource Manager
Make sure that a resource consumer group is defined for your pooled sessions and that the idle timeout is big enough to not unexpectedly interrupt a working healthy session.
Oracle Database Resource Manager is a very flexible and powerful tool that can help your system in many ways.
It seems the problem was being caused by the use of transactions. Removing the transactions from the above code produced the following:
public class DoSomethingToDb(ISessionFactory sessionFactory)
{
using (ISession session = sessionFactory.OpenSession())
{
//Do Stuff
session.Flush();
}
}
Which seems to cause no issues.
Related
I am getting this exception -Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use.
I have set ServiceStack.Redis.PoolSizeMultiplier to 100 and ServiceStack.Redis.PoolTimeOutSeconds to 50.
But i am still getting errors. What do you suggest. To increase or decrease this timeout value? I need some explanation about this Redis.TimeOutValue. What are advantages of increasing these values?
This error message indicates that all Redis connections are currently in use and that the client timed out waiting for a free connection from the pool to be available.
This can also happen if your redis client connections are not properly disposed of after usage. Make sure all your Redis Client instances are disposed after usage, e.g. by using a using {} statement:
using (var redis = redisManager.GetClient())
{
//...
}
Another solution to avoid this error is to switch to using the RedisManagerPool which will create a new connection outside of the pool after the pool size has been exceeded, however this could be masquerading an issue that your Redis clients are not being properly disposed.
I've written a service that occasionally has to poll a database very often. Usually I'd create a new SqlConnection with the SqlDataAdapter and fire away like this:
var table = new DataTable();
using(var connection = new SqlConnection(connectionString))
{
connection.Open();
using(var adapter = new SqlDataAdapter(selectStatement, connection))
{
adapter.Fill(table);
}
}
However in a heavy load situation (which occurs maybe once a week), the service might actually use up the entire connection pool and the service records the following exception.
System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
Multiple threads in the service have to access the SQL server for various queries and I'd like as much of them to run in parallel as possible (and that obviously works too well sometimes).
I thought about several possible solutions:
I thought about increasing the connection pool size, however that might just delay the problem.
Then I thought about using a single connection for the service and keep that open for the remainder of the service running, which might a simple option, however it will keep the connection open even if there is no workload to be done and would have to handle connection resets by the server etc. which I do not know the effect of.
Lastly I thought about implementing my own kind of pool that manages the number of concurrent connections and keeps the threads on hold until there is a free slot.
What would be the recommended procedure or is there a best practice way of handling this?
Well the solution in the end was not exactly ideal (fixing the issue on the SQL Server side) so I ended up checking the number of concurrent connections in the job queuing system.
The service will now not create another thread for document generation unless it can guarantee that the connection pool is actually available. The bad bottleneck on the SQL server is still in place, however the service now no longer generates exceptions.
The downside of course is, that the queue gets longer while there is some blocking query executing on the SQL Server, which might delay document generation for a minute or two. So it isn't an ideal solution but a workable one, since the delay isn't critical as the documents aren't needed directly but stored for archival purpose.
The better solution would have been to fix it SQL Server side.
When using ServiceStack.Redis, TimeoutException is thrown since all the pools are being used. How to resolve the timeout exception.
Possible connection leak
The TimeoutException only happens when the pools are full, i.e. when there are no connections available which is usually an indication that you have a connection leak in which case make sure that all redis clients resolved from the pool are released, which you can ensure by wrapping each usage in a using statement, e.g:
using (var redis = redisManager.GetClient())
{
//...
}
RedisManagerPool creates new connections instead when full
Another option to is to use the RedisManagerPool which instead of throwing a TimeoutException will create a new connection outside the pool, however this will usually just mask the problem of a connection leak and it will result in a high open/closed connection count.
Increase Pool Resource Settings
If you believe the TimeoutException is due to heavy load you can increase the pool size and timeout seconds, e.g:
var redisManager = new PooledRedisClientManager(
redisReadWriteHosts, redisReadOnlyHosts,
poolSizeMultiplier: 40, //connection pool size * no of hosts
poolTimeOutSeconds: 5); //how long to block for available connection
I'm trying to understand how MS Enterprise Library's data access block manages its connections to SQL. The issue I have is that under a steady load (from a load test), at 10 minute intervals the number of connections to SQL increases quickly - which causes noticeable jump in page response times from the website.
This is the scenario I'm running:
Visual Studio load test tools, running against 3 web servers behind a load balancer
The tools give full visibility over the performance counters to all web + DB boxes
The tests take ~10 seconds each, and perform 4 inserts (form data), and some trivial selects
There are 60 tests running concurrently. There is no increase or decrease in load during the entire test.
The test is run for between 20 minutes and 3 hours, with consistent results.
And this is the issue we see:
Exactly every 10 minutes, the performance counter from SQL for SQL General: User Connections increases - by ~20 connections total
The pages performing the HTTP post / DB insert are the ones most significantly affected. The other pages show moderate, but noticeable rises.
The CPU/memory load on the web servers is unaffected
This increase corresponds with a notable bump in page response times - E.g. from .3 seconds average to up to 5 seconds
After ~5 minutes it releases many of the connections, with no affect on web performance
The following 5 minutes of testing gives the same (normal) web performance
Ultimately, the graph looks like a square wave
Happens all over again, 10 minutes after the first rise
What I've looked at:
Database calls:
All calls in the database start with:
SqlDatabase database = new SqlDatabase([...]);
And execute either proc with no required output:
return database.ExecuteScalar([...], [...]);
Or read wrapped in a using statement:
using (SqlDataReader reader = (SqlDataReader)database.ExecuteReader([...], [...]))
{
[...]
}
There are no direct uses of SqlConnection, no .Open() or .Close() methods, and no exceptions being thrown
Database verification:
We've run SQL profiler over the login / logout events, and taken snapshots with the sp_who2 command, showing who owns the connections. The latter shows that indeed the web site (seen by machine + credential) are holding the connections.
There are no scheduled jobs (DB or web server), and the user connection load is stable when there is no load from the web servers.
Connection pool config
I know the min & max size of the connection pool can be altered with the connection string.
E.g.:
"Data Source=[server];Initial Catalog=[x];Integrated Security=SSPI;Max
Pool Size=75;Min Pool Size=5;"
A fall back measure may be to set the minimum size to ~10
I understand the default max is 100, and the default min is 0 (from here)
I'm a little bit lithe to think of connection pooling (specific to this setting) and the User Connections performance counter from SQL. This article introduces these connection pools as being used to manage connection string, which seems different to what I assume it does (hold a pool of connections generally available, to avoid the cost of re-opening them on SQL)
I still haven't seen any configuration parameters that are handily defaulting to 5 or 10 minutes, to zero in on...
So, any help is appreciated.
I know that 10 minute spikes sounds like a change in load, or new activity is happening - but we've worked quite hard to isolate those & any other factors - and for this question, I am hoping to understand EL scaling its connections up & down.
Thanks.
So, it turns out that SQL user connections are created & added to the pool whenever all other connections are busy. So when long-running queries occur, or the DB is otherwise unresponsive, it will choose to expand to manage the load.
The cause of this in our case happened to be a SQL replication job (unfortunate, but found...) - And the changes in the # of User Connections was just a symptom, not a possible cause.
Although the cause turned out to be elsewhere, I now feel I understand the connection pool management, from this (and assumably other) SQL libraries.
Both the name of the connection string parameter and this blog post - http://fxjr.blogspot.co.uk/2010/04/npgsql-connection-pool-explained.html - lead me to believe that Npgsql wont exceed the MaxPoolSize value set in the connection string. However the docs (http://npgsql.projects.postgresql.org/docs/manual/UserManual.html) say "Max size of connection pool. Pooled connections will be disposed of when returned to the pool if the pool contains more than this number of connections. Default: 20"
This suggests that the pool can actually grow larger than MaxPoolSize and it is in fact just a level at which Npgsql starts to aggressively remove connections from the pool as soon as they are returned.
I've been searching to try and find an answer but I can find out exactly what happens when you reach MaxPoolSize. Anyone else know?
edit: I should add we are using Npgsql 2.0.6.0 due to another dependency being supported only up to that version.
I think this may be a copy paste issue regarding the minpoolsize. Npgsql doesn't create more than maxpoolsize connections. When this value is reached, new connection requests are queued until there is a free one.
Which issue are you depending on which only works on 2.0.6?