Develop Reference

Redis Timeout optimization. Need suggestions

I am getting this exception -Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use.
I have set ServiceStack.Redis.PoolSizeMultiplier to 100 and ServiceStack.Redis.PoolTimeOutSeconds to 50.
But i am still getting errors. What do you suggest. To increase or decrease this timeout value? I need some explanation about this Redis.TimeOutValue. What are advantages of increasing these values?

This error message indicates that all Redis connections are currently in use and that the client timed out waiting for a free connection from the pool to be available.
This can also happen if your redis client connections are not properly disposed of after usage. Make sure all your Redis Client instances are disposed after usage, e.g. by using a using {} statement:
using (var redis = redisManager.GetClient())
{
//...
}
Another solution to avoid this error is to switch to using the RedisManagerPool which will create a new connection outside of the pool after the pool size has been exceeded, however this could be masquerading an issue that your Redis clients are not being properly disposed.

Redis ServiceStack TimeoutException C# Asp.Net

When using ServiceStack.Redis, TimeoutException is thrown since all the pools are being used. How to resolve the timeout exception.

Possible connection leak
The TimeoutException only happens when the pools are full, i.e. when there are no connections available which is usually an indication that you have a connection leak in which case make sure that all redis clients resolved from the pool are released, which you can ensure by wrapping each usage in a using statement, e.g:
using (var redis = redisManager.GetClient())
{
//...
}
RedisManagerPool creates new connections instead when full
Another option to is to use the RedisManagerPool which instead of throwing a TimeoutException will create a new connection outside the pool, however this will usually just mask the problem of a connection leak and it will result in a high open/closed connection count.
Increase Pool Resource Settings
If you believe the TimeoutException is due to heavy load you can increase the pool size and timeout seconds, e.g:
var redisManager = new PooledRedisClientManager(
redisReadWriteHosts, redisReadOnlyHosts,
poolSizeMultiplier: 40, //connection pool size * no of hosts
poolTimeOutSeconds: 5); //how long to block for available connection

MongoDB connection problems on Azure

We have an ASP.NET MVC application deployed to an Azure Website that connects to MongoDB and does both read and write operations. The application does this iteratively. A few thousand times per minute.
We initialize the C# driver using Autofac and we set the MaxConnectionIdleTime to 45 seconds as suggested in https://groups.google.com/forum/#!topic/mongodb-user/_Z8YepNHnbI and a few other places.
We are still getting a large number of the below error:
Unable to read data from the transport connection: A connection
attempt failed because the connected party did not properly respond
after a period of time, or established connection failed because
connected host has failed to respond. Method
Message:":{"ClassName":"System.IO.IOException","Message":"Unable to
read data from the transport connection: A connection attempt failed
because the connected party did not properly respond after a period of
time, or established connection failed because connected host has
failed to respond.
We get this error while connecting to both a MongoDB instance deployed on a VM in the same datacenter/region on Azure and also while connecting to an external PaaS MongoDB provider.
I run the same code in my local computer and connect to the same DB and I don't receive these errors. It's only when I deploy the code to an Azure Website.
Any suggestions?

A few thousand requests per minute is a big load, and the only way to do it right, is by controlling and limiting the maximum number of threads which could be running at any one time.
As there's not much information posted as to how you've implemented this. I'm going to cover a few possible circumstances.
Time to experiment...
The constants:
Items to process:
50 per second, or in other words...
3,000 per minute, and one more way to look at it...
180,000 per hour
The variables:
Data transfer rates:
How much data you can transfer per second is going to play a role no matter what we do, and this will vary through out the day depending on the time of day.
The only thing we can do is fire off more requests from different cpu's to distribute the weight of traffic we're sending back n forth.
Processing power:
I'm assuming you have this in a WebJob as opposed to having this coded inside the MVC site it's self. It's highly inefficient and not fit for the purpose that you're trying to achieve. By using a WebJob we can queue work items to be processed by other WebJobs. The queue in question is the Azure Queue Storage.
Azure Queue storage is a service for storing large numbers of messages
that can be accessed from anywhere in the world via authenticated
calls using HTTP or HTTPS. A single queue message can be up to 64 KB
in size, and a queue can contain millions of messages, up to the total
capacity limit of a storage account. A storage account can contain up
to 200 TB of blob, queue, and table data. See Azure Storage
Scalability and Performance Targets for details about storage account
capacity.
Common uses of Queue storage include:
Creating a backlog of work to process asynchronously
Passing messages from an Azure Web role to an Azure Worker role
The issues:
We're attempting to complete 50 transactions per second, so each transaction should be done in under 1 second if we were utilising 50 threads. Our 45 second time out serves no purpose at this point.
We're expecting 50 threads to run concurrently, and all complete in under a second, every second, on a single cpu. (I'm exaggerating a point here, just to make a point... but imagine downloading 50 text files every single second. Processing it, then trying to shoot it back over to a colleague in the hopes they'll even be ready to catch it)
We need to have a retry logic in place, if after 3 attempts the item isn't processed, they need to be placed back in to the queue. Ideally we should be providing more time to the server to respond than just one second with each failure, lets say that we gave it a 2 second break on first failure, then 4 seconds, then 10, this will greatly increase the odds of us persisting / retrieving the data that we needed.
We're assuming that our MongoDb can handle this number of requests per second. If you haven't already, start looking at ways to scale it out, the issue isn't in the fact that it's a MongoDb, the data layer could have been anything, it's the fact that we're making this number of requests from a single source that is going to be the most likely cause of your issues.
The solution:
Set up a WebJob and name it EnqueueJob. This WebJob will have one sole purpose, to queue items of work to be process in the Queue Storage.
Create a Queue Storage Container named WorkItemQueue, this queue will act as a trigger to the next step and kick off our scaling out operations.
Create another WebJob named DequeueJob. This WebJob will also have one sole purpose, to dequeue the work items from the WorkItemQueue and fire out the requests to your data store.
Configure the DequeueJob to spin up once an item has been placed inside the WorkItemQueue, start 5 separate threads on each and while the queue is not empty, dequeue work items for each thread and attempt to execute the dequeued job.
Attempt 1, if fail, wait & retry.
Attempt 2, if fail, wait & retry.
Attempt 3, if fail, enqueue item back to WorkItemQueue
Configure your website to autoscale out to x amount of cpu's (note that your website and web jobs share the same resources)
Here's a short 10 minute video that gives an overview on how to utilise queue storages and web jobs.
Edit:
Another reason you may be getting those errors could be because of two other factors as well, again caused by it being in an MVC app...
If you're compiling the application with the DEBUG attribute applied but pushing the RELEASE version instead, you could be running into issues due to the settings in your web.config, without the DEBUG attribute, an ASP.NET web application will run a request for a maximum of 90 seconds, if the request takes longer than this, it will dispose of the request.
To increase the timeout to longer than 90 seconds you will need to change the [httpRuntime][3] property in your web.config...
<!-- Increase timeout to five minutes -->
<httpRuntime executionTimeout="300" />
The other thing that you need to be aware of is the request timeout settings of your browser > web app, I'd say that if you insist on keeping the code in MVC as opposed to extracting it and putting it into a WebJob, then you can use the following code to fire a request off to your web app and offset the timeout of the request.
string html = string.Empty;
string uri = "http://google.com";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Timeout = TimeSpan.FromMinutes(5);
using (HttpWebResponse response = (HttpWebResonse)request.GetResponse())
using (Stream stream = response.GetResponseStream())
using (StreamReader reader = new StreamReader(stream))
{
html = reader.ReadToEnd();
}

Are you using mongoDB in a VM? It seems to be a network problem. This kind of transient faults should occur, so the best you can do is implement a retry pattern or use a lib such as Polly to do that:
Policy
.Handle<IOException>()
.Retry(3, (exception, retryCount) =>
{
// do something
});
https://github.com/michael-wolfenden/Polly

Database connections not being closed on oracle server

I am experiencing a problem where by when connecting to an Oracle11g database using NHibernate, old connections in the pool are not being closed.
I am fairly sure that all the NHibernate sessions are disposed properly, however the connections still remain in an INACTIVE status. I know this will be because of connection pooling, however surley they should be removed after a certain amount of time? If not how can I configure this to happen.
I have tried adding the following settings into my connection string:
Max Pool Size=10;
Connection Lifetime=120;
Incr Pool Size=1;
Decr Pool Size=10;`
This seems to stop as many connections being created, I guess because this increase size is an increment of 1, however once the connections have been put back into the pool they are never closed.
I have looked at the v$session table and some of the LASST_CALL_ET values were as much as 786465s or 9 days!!
I am fairly sure all the sessions are being disposed, here is an example of the code:
public class DoSomethingToDb(ISessionFactory sessionFactory)
{
using (ISession session = sessionFactory.OpenSession())
{
session.Transaction.Begin();
//Do Stuff
session.Transaction.Commit();
}
}
How can I setup my program/NHibernate/Ado.Net/Oracle to close connections that are no longer in use.
The server we were testing on crashed yesterday as there were over 800 INACTVIE connections and no more could be issued.

The reason you are having problems is because your Decr Pool Size value is too large. It will not be able to close any connections unless all of them are available to close since your Decr Pool Size is the same as your Max Pool Size.
When i have set this value to 1, it takes for ever to release unused connections. I am currently setting mine to 5 and it still takes just as long between each decrements, but it will release more at once.
Pooling=true;
Min Pool Size=0;
Max Pool Size=10;
Incr Pool Size=1;
Decr Pool Size=3;
Also, with Connection Lifetime being set to 120, it will not keep any sessions open for more than 2 minutes.

It would surprise me if you could do this in hibernate since I think the connections are leaked connections. For some reason they got out of control and won't ever be reused.
What you can do is configure an session idle timeout in resource manager in the Oracle database. See Managing Resources with Oracle Database Resource Manager
Make sure that a resource consumer group is defined for your pooled sessions and that the idle timeout is big enough to not unexpectedly interrupt a working healthy session.
Oracle Database Resource Manager is a very flexible and powerful tool that can help your system in many ways.

It seems the problem was being caused by the use of transactions. Removing the transactions from the above code produced the following:
public class DoSomethingToDb(ISessionFactory sessionFactory)
{
using (ISession session = sessionFactory.OpenSession())
{
//Do Stuff
session.Flush();
}
}
Which seems to cause no issues.

Nhibernate Connection Pool Problems

we are having some connection pool issues with Nhibernate on an MVC3 web application which is running with SQL Express and dealing with multiple concurrent AJAX based requests.
Every so often (hours in between) we see errors starting which show:
NHibernate.Util.ADOExceptionReporter
Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
then a load of
While preparing select TOP (#p0)
....
an error occurred
We have to recycle the IIS app pool to stop 500 errors from being thrown after that.
Looking at the SQL Server we see:
select * from sys.dm_exec_sessions
... gives about 30 sessions with IDs above 51 (i.e. user sessions)
select * from sys.dm_exec_connections
... gives around the same amount
BUT
select ##connections
... gives results with 79022
Is this indicating that the connections are never released?
The Nhibernate sessions are for the lifetime of the request.
Does anyone have any experience of anything like this or can point us in the right direction?
Many thanks
Richard

You can't have more then 32767 connection to SQL Server.
##CONNECTIONS also gives (my bold)
Returns the number of attempted connections, either successful or unsuccessful since SQL Server was last started.
Not current connection
I suspect that your pool is not set up correctly so it's exhausted too quickly.
Or you are not releasing connections correctly and you're checking SQL Server after you recycle IIS.