When using ServiceStack.Redis, TimeoutException is thrown since all the pools are being used. How to resolve the timeout exception.
Possible connection leak
The TimeoutException only happens when the pools are full, i.e. when there are no connections available which is usually an indication that you have a connection leak in which case make sure that all redis clients resolved from the pool are released, which you can ensure by wrapping each usage in a using statement, e.g:
using (var redis = redisManager.GetClient())
{
//...
}
RedisManagerPool creates new connections instead when full
Another option to is to use the RedisManagerPool which instead of throwing a TimeoutException will create a new connection outside the pool, however this will usually just mask the problem of a connection leak and it will result in a high open/closed connection count.
Increase Pool Resource Settings
If you believe the TimeoutException is due to heavy load you can increase the pool size and timeout seconds, e.g:
var redisManager = new PooledRedisClientManager(
redisReadWriteHosts, redisReadOnlyHosts,
poolSizeMultiplier: 40, //connection pool size * no of hosts
poolTimeOutSeconds: 5); //how long to block for available connection
Related
We use Service stack as our RedisClient.
Our application handles above 50 requests per second and the current architecture is that 12 load balanced application instances all connect to a single Redis instance with get/set operations queued using Hangfire.
We use PooledRedisClientManager with the configuration below:
services.AddSingleton<IRedisClientsManager>(p =>
new PooledRedisClientManager(Configuration.GetValue<long>("Redis:DatabaseId"), Configuration.GetValue<string>("Redis:master"))
{
ConnectTimeout = Configuration.GetValue<int>(5000),
IdleTimeOutSecs = Configuration.GetValue<int>(30),
PoolTimeout = Configuration.GetValue<int>(3)
});
Our Applications have been throwing this exception:
Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use.
at ServiceStack.Redis.PooledRedisClientManager.GetClient(Boolean forAsync) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\PooledRedisClientManager.cs:line 264
I'm aware the maximum number of connected clients to Redis is 10000 and we only had 594 clients connected to the Redis Instance.
Kindly assist
You've reached the Pool Size of the PooledRedisClientManager which when it reaches its limit blocks until a new client is released back into the pool or times out if the Timeout is exceeded.
You can configure the pool size with:
RedisConfig.DefaultMaxPoolSize = ...
Alternatively you may want to switch to the RedisManagerPool which when it exceeds the pool size will create/release clients outside of the pool, instead of blocking.
I am getting this exception -Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use.
I have set ServiceStack.Redis.PoolSizeMultiplier to 100 and ServiceStack.Redis.PoolTimeOutSeconds to 50.
But i am still getting errors. What do you suggest. To increase or decrease this timeout value? I need some explanation about this Redis.TimeOutValue. What are advantages of increasing these values?
This error message indicates that all Redis connections are currently in use and that the client timed out waiting for a free connection from the pool to be available.
This can also happen if your redis client connections are not properly disposed of after usage. Make sure all your Redis Client instances are disposed after usage, e.g. by using a using {} statement:
using (var redis = redisManager.GetClient())
{
//...
}
Another solution to avoid this error is to switch to using the RedisManagerPool which will create a new connection outside of the pool after the pool size has been exceeded, however this could be masquerading an issue that your Redis clients are not being properly disposed.
I've written a service that occasionally has to poll a database very often. Usually I'd create a new SqlConnection with the SqlDataAdapter and fire away like this:
var table = new DataTable();
using(var connection = new SqlConnection(connectionString))
{
connection.Open();
using(var adapter = new SqlDataAdapter(selectStatement, connection))
{
adapter.Fill(table);
}
}
However in a heavy load situation (which occurs maybe once a week), the service might actually use up the entire connection pool and the service records the following exception.
System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
Multiple threads in the service have to access the SQL server for various queries and I'd like as much of them to run in parallel as possible (and that obviously works too well sometimes).
I thought about several possible solutions:
I thought about increasing the connection pool size, however that might just delay the problem.
Then I thought about using a single connection for the service and keep that open for the remainder of the service running, which might a simple option, however it will keep the connection open even if there is no workload to be done and would have to handle connection resets by the server etc. which I do not know the effect of.
Lastly I thought about implementing my own kind of pool that manages the number of concurrent connections and keeps the threads on hold until there is a free slot.
What would be the recommended procedure or is there a best practice way of handling this?
Well the solution in the end was not exactly ideal (fixing the issue on the SQL Server side) so I ended up checking the number of concurrent connections in the job queuing system.
The service will now not create another thread for document generation unless it can guarantee that the connection pool is actually available. The bad bottleneck on the SQL server is still in place, however the service now no longer generates exceptions.
The downside of course is, that the queue gets longer while there is some blocking query executing on the SQL Server, which might delay document generation for a minute or two. So it isn't an ideal solution but a workable one, since the delay isn't critical as the documents aren't needed directly but stored for archival purpose.
The better solution would have been to fix it SQL Server side.
I am experiencing a problem where by when connecting to an Oracle11g database using NHibernate, old connections in the pool are not being closed.
I am fairly sure that all the NHibernate sessions are disposed properly, however the connections still remain in an INACTIVE status. I know this will be because of connection pooling, however surley they should be removed after a certain amount of time? If not how can I configure this to happen.
I have tried adding the following settings into my connection string:
Max Pool Size=10;
Connection Lifetime=120;
Incr Pool Size=1;
Decr Pool Size=10;`
This seems to stop as many connections being created, I guess because this increase size is an increment of 1, however once the connections have been put back into the pool they are never closed.
I have looked at the v$session table and some of the LASST_CALL_ET values were as much as 786465s or 9 days!!
I am fairly sure all the sessions are being disposed, here is an example of the code:
public class DoSomethingToDb(ISessionFactory sessionFactory)
{
using (ISession session = sessionFactory.OpenSession())
{
session.Transaction.Begin();
//Do Stuff
session.Transaction.Commit();
}
}
How can I setup my program/NHibernate/Ado.Net/Oracle to close connections that are no longer in use.
The server we were testing on crashed yesterday as there were over 800 INACTVIE connections and no more could be issued.
The reason you are having problems is because your Decr Pool Size value is too large. It will not be able to close any connections unless all of them are available to close since your Decr Pool Size is the same as your Max Pool Size.
When i have set this value to 1, it takes for ever to release unused connections. I am currently setting mine to 5 and it still takes just as long between each decrements, but it will release more at once.
Pooling=true;
Min Pool Size=0;
Max Pool Size=10;
Incr Pool Size=1;
Decr Pool Size=3;
Also, with Connection Lifetime being set to 120, it will not keep any sessions open for more than 2 minutes.
It would surprise me if you could do this in hibernate since I think the connections are leaked connections. For some reason they got out of control and won't ever be reused.
What you can do is configure an session idle timeout in resource manager in the Oracle database. See Managing Resources with Oracle Database Resource Manager
Make sure that a resource consumer group is defined for your pooled sessions and that the idle timeout is big enough to not unexpectedly interrupt a working healthy session.
Oracle Database Resource Manager is a very flexible and powerful tool that can help your system in many ways.
It seems the problem was being caused by the use of transactions. Removing the transactions from the above code produced the following:
public class DoSomethingToDb(ISessionFactory sessionFactory)
{
using (ISession session = sessionFactory.OpenSession())
{
//Do Stuff
session.Flush();
}
}
Which seems to cause no issues.
I have a web app deployed under IIS. It's been getting timeout errors, and I believe I've tracked the problem down to code that doesn't properly dispose of SQL connections.
This will require quite a lot of surgery to fix, so at least until I can finish implementing and testing the new code, using using blocks correctly, I decided to bump up the connection pool size to something astronomical. But that doesn't seem to have helped. My connection string is:
Data Source=MyServer;Initial Catalog=MyDb;User ID=MyUser;Password=MyPwd;Max Pool Size=10000
And my log file shows:
System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
I have been keeping Performance Monitor open over this period, and the number of user connections maxed out at around 150. So I don't think it can be that we're running out of connections.
What else could be causing this error?
Unfortunately SqlCommand does not obey ConnectionString or SqlConnection's time-out. And it has to be set manually. It's because the time-out mentioned in connection string is used for connecting, not executing. So that please consider setting up it's time-out by code.
var conn = new SqlConnection(cs);
var cmd = conn.CreateCommand();
cmd.CommandTimeout = conn.ConnectionTimeout;
...