I have a code like below :
public void Do
{
using (var myConnection = new SqlConnection(connectionString))
{
for (int i = 0; i < 4; i++)
{
MyProcess.Execute(myConnection);
}
myConnection.Close();
myConnection.dispose();
}
}
public class MyProcess
{
public void Execute(SqlConnection myConnection)
{
if (myConnection.State == ConnectionState.Closed)
myConnection.Open();
//long running code
}
}
Execute methods sometimes take 5-10 minutes,sometimes run in 1-2 minutes for each iteration.
Now here i am confused that whether i shall open and close connection for each iteration and this will be efficient or whether i open and close connection 1 time only.
But with opening and closing connection once, this will hold resource for each of the iteration and will consume resource.
So i am not getting what should be the proper way to handle this
Can anybody please give me some advice on this?
ADO.NET uses Connection Pooling under the hood. That is why opening a new connection each time should not be an issue. Your call to myConnection.Open() will not actually result in opening of physical connection to the Database each time.
Check also this question. It's author made a measure test for connection opening. I have shown him that time of subsequent calls to DbConnection.Open() approaches to 0.
The most preferred way of using DbConnection is to open it for the minimal time period. If your MyProcess.Execute() has a lot of other non-DB related logic that takes considerable time, you should not keep an opened connection during this execution.
Check also Best Practices for Using ADO.NET article. It has following explicit statement:
High performance applications keep connections to the data source in
use for a minimal amount of time, as well as take advantage of
performance enhancing technology such as connection pooling.
So when does Ado.net removes the connection from connection pool?
This article provides some internal details on connection pooling:
The connection pooler removes a connection from the pool after it has
been idle for approximately 4-8 minutes, or if the pooler detects that
the connection with the server has been severed. Note that a severed
connection can be detected only after attempting to communicate with
the server. If a connection is found that is no longer connected to
the server, it is marked as invalid. Invalid connections are removed
from the connection pool only when they are closed or reclaimed.
This article also provides some details on pooling tuning if it's required. However you should consider such manual configuration only if you experience some performance issues caused by the pooling.
Related
I've written a service that occasionally has to poll a database very often. Usually I'd create a new SqlConnection with the SqlDataAdapter and fire away like this:
var table = new DataTable();
using(var connection = new SqlConnection(connectionString))
{
connection.Open();
using(var adapter = new SqlDataAdapter(selectStatement, connection))
{
adapter.Fill(table);
}
}
However in a heavy load situation (which occurs maybe once a week), the service might actually use up the entire connection pool and the service records the following exception.
System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
Multiple threads in the service have to access the SQL server for various queries and I'd like as much of them to run in parallel as possible (and that obviously works too well sometimes).
I thought about several possible solutions:
I thought about increasing the connection pool size, however that might just delay the problem.
Then I thought about using a single connection for the service and keep that open for the remainder of the service running, which might a simple option, however it will keep the connection open even if there is no workload to be done and would have to handle connection resets by the server etc. which I do not know the effect of.
Lastly I thought about implementing my own kind of pool that manages the number of concurrent connections and keeps the threads on hold until there is a free slot.
What would be the recommended procedure or is there a best practice way of handling this?
Well the solution in the end was not exactly ideal (fixing the issue on the SQL Server side) so I ended up checking the number of concurrent connections in the job queuing system.
The service will now not create another thread for document generation unless it can guarantee that the connection pool is actually available. The bad bottleneck on the SQL server is still in place, however the service now no longer generates exceptions.
The downside of course is, that the queue gets longer while there is some blocking query executing on the SQL Server, which might delay document generation for a minute or two. So it isn't an ideal solution but a workable one, since the delay isn't critical as the documents aren't needed directly but stored for archival purpose.
The better solution would have been to fix it SQL Server side.
I have a long-running .NET process (a Windows service), which talks to a SQL Server.
Originally, I opened a connection when the service started up, and just kept it open. However, occasionally a network hiccup or reboot of the SQL Server would break that connection, and my process wouldn't know this. It would attempt to keep throwing SQL against a closed connection.
Trying to maintain an open connection like that was clearly me trying to be too clever. So, do I:
Create (and open) a new SqlConnection object every time
Create the connection once on service start-up, and just attempt to re-open the connection every time
For the latter (#2), I have this code:
if(connection.State != ConnectionState.Open)
{
connection.Open();
}
I only have to do this because the connection already exists. If I created the connection fresh each time (#1), clearly it would be closed and I would need to open it.
What I'm hoping to do is take advantage of SQL connection pooling to actually not open a new connection every time, but just let the connection pool manage that -- give me an open connection when it has one, or open a new one when it doesn't.
To achieve this, does it matter if I create the connection fresh each time (#1), or if I just-reuse a connection and attempt to re-open it each time (#2)?
Connection pooling means that even if you do (1), under the hood, the framework will do (2) for you for improved performance. So you can feel free to create a new connection each time.
It's worth pointing out that, for the most part, the pooling only applies to identical connection strings. So if you change certain options, those connections may not be pooled.
I'm not using LINQ-to-SQL or Entity Framework bits in a web app, and have currently been using something like this (this is for a class project):
using System.Data;
using System.Data.SqlClient;
namespace StackOverflowClone.Models
{
public class Database
{
public static SqlConnection ActiveConnection { get; private set; }
static Database()
{
ActiveConnection = new SqlConnection(
"Data Source=********.database.windows.net;" +
"Initial Catalog=EECS341;Uid=*****;Pwd=*******;" +
"MultipleActiveResultSets=True;");
ActiveConnection.Open();
}
}
}
However this seems to cause threading issues because the static initializer runs once per server process, rather than once per request.
Does the framework provide a built in method of handling this or should I just have a function that coughs up database connections new'd up each time?
or should I just have a function that coughs up database connections new'd up each time?
Yes, do this. Let ADO.NET connection pooling handle the details for you. Your goal should be to keep the connection open for as short a period of time as possible.
Connection pooling reduces the number of times that new connections
must be opened. The pooler maintains ownership of the physical
connection. It manages connections by keeping alive a set of active
connections for each given connection configuration. Whenever a user
calls Open on a connection, the pooler looks for an available
connection in the pool. If a pooled connection is available, it
returns it to the caller instead of opening a new connection. When the
application calls Close on the connection, the pooler returns it to
the pooled set of active connections instead of closing it. Once the
connection is returned to the pool, it is ready to be reused on the
next Open call.
So, create a static GetConnection() method that returns a new open connection. Use this within a using statement so it can be closed and returned to the connection pool as soon as possible.
using(var cn = Database.GetConnection())
{
//query your data here, Dapper example below
cn.Execute("update MyTable set MyField = #newValue", new {newValue});
}
Always create new connections and destroy them with using. They are not really created from scratch, they are fetched from a connection pool. There is no performance penalty. Actually that's the best and correct way to go.
See my answer about using: https://stackoverflow.com/a/9811911/290343
Does the framework provide a built in method of handling this or
should I just have a function that coughs up database connections
new'd up each time?
Both, actually.
The web server is multi threaded, so each thread needs its own database connection. Just create one when needed.
The actual connections to the database are pooled. When you dispose a connection object, the actual connection isn't closed, but returned to the pool. If you create a new connection object with the same connection string, it will just reuse a connection from the pool.
When opening a connection to SQL Server 2005 from our web app, we occasionally see this error:
"Impersonate Session Security Context" cannot be called in this batch because a simultaneous batch has called it.
We use MARS and connection pooling.
The exception originates from the following piece of code:
protected SqlConnection Open()
{
SqlConnection connection = new SqlConnection();
connection.ConnectionString = m_ConnectionString;
if (connection != null)
{
try
{
connection.Open();
if (m_ExecuteAsUserName != null)
{
string sql = Format("EXECUTE AS LOGIN = {0};", m_ExecuteAsUserName);
ExecuteCommand(connection, sql);
}
}
catch (Exception exception)
{
connection.Close();
connection = null;
}
}
return connection;
}
I found an MS Connect article which suggests that the error is caused when a previous command has not yet terminated before the EXECUTE AS LOGIN command is sent. Yet how can this be if the connection has only just been opened?
Could this be something to do with connection pooling interacting strangely with MARS?
UPDATE: For the short-term we have implemented a workaround by clearing out the connection pool whenever this happens, to get rid of the bad connection, as it otherwise keeps getting handed back to various users. (This now happens a 5-10 times a day with only a small number of simultaneous users, so it is fairly annoying.) But if anyone has any further ideas, we are still looking out for a real solution...
I would say it's MARS rather then pooling
From "Using Multiple Active Result Sets (MARS)"
Applications can have multiple default
result sets open and can interleave
reading from them.
Applications can
execute other statements (for example,
INSERT, UPDATE, DELETE, and stored
procedure calls) while default result
sets are open.
Connection pooling in it's basic form means the connection open/close overhead is minimised, but any connection (until MARS) has one thing going on at any one time. Pooling has been around for some time and just works out of the box.
MARS (I've not used it BTW) introduces overlapping "stuff" going on for any single connection. So it's probably MARS rather than connection pooling is the bigger culprit of the 2.
From "Extending Database Impersonation by Using EXECUTE AS"
When impersonating a principal by
using the EXECUTE AS LOGIN statement,
or within a server-scoped module by
using the EXECUTE AS clause, the scope
of the impersonation is server-wide.
This may explain why MARS is causing it: the same principal in 2 session both running EXECUTE AS.
There may be something in that article of use, or try this:
IF ORIGINAL_LOGIN() = SUSER_SNAME() EXECUTE AS LOGIN = {0};
On reflection and after reading for this answer, I've not convinced that trying to change execution context for each session (MARS) in one connections is a good idea...
Don't blame connection pooling - MARS is quite notorious for wreaking a havoc. It's not entirely it's blame but it's kind of half and half. The key thing to remember is that MARS is designed, and only works with "normal" DB use (meaning, regular CRUD stuff no admin batches). Any commands that have a wide effect on DB engine can trip MARS even if it's just one connection and single threaded (like running a setup batch to create tables or a nested transaction).
Having said that, one can easily just blame MARS, but it works perfecly fine for normal CRUD scenarios which are like 99% (and things with low efficiencey like ORM-s and LINQ depend on it for life). Meaning that it's important for people to learn that if they want to hack SQL through a connection they can't use MARS. For example I had a setup code that was creating whole DB from scratch, beceuse it's very convenient for deployment, but it was sharing connection sting with web service it was deploying - oops :-) Took me a few days of digging to learn my lesson. So I just maintain the separation of concerns (which is always good) and problems went away.
Have you tried to use a revert at the end of your sql statement?
http://msdn.microsoft.com/en-us/library/ms178632.aspx
I always do this to just make sure the current context is back to normal.
We have some client code which is using the SqlConnection class in .NET to talk to a SQLServer database. It is intermittently failing with this error:
"ExecuteReader requires an open and available Connection. The connection's current state is Closed"
The "temporary" solution is to reboot the process, after which everything works - however, that's obviously unsatisfactory.
The code is keeping a cache of SqlConnection instances, one for each database.
We'd like to re-write the code, but before I do, I need to know a few things:
My first question is: Is it inefficient to repeatedly connect and disconnect SqlConnection objects, or does the underlying library perform connection pooling on our behalf?
// Is this bad/inefficient?
for(many-times)
{
using(SQLConnection conn = new SQLConnection(connectionString))
{
// do stuff with conn
}
}
Because our code does not do the above, what seems the likely cause of the problem is that something happens to the underlying SQLServer database during the "lifetime" of the connection that causes the connection to be closed...
If it turns out that it is worthwile to "cache" SqlConnection objects, what is the recommended way to handle all errors that could be resolved simply by "reconnecting" to the database. I'm talking about scenarios such as:
The database is taken offline and brought back online, but the client process had no open transactions while this was happening
The database was "disconnected", then "reconnected"
I notice that there is a "State" property on SqlConnection... is there an appropriate way to query that?
Finally, I have a test SQLServer instance set up with full access rights: how can I go about reproducing the exact error "ExecuteReader requires an open and available Connection. The connection's current state is Closed"
No, it's not inefficient to create lots of SqlConnection objects and close each of them when you're done. That's exactly the right thing to do. Let the .NET framework connection pooling do its job - don't try to do it yourself. You don't need to do anything specific to enable connection pooling (although you can disable it by setting Pooling=false in your connection string).
There are many things that could go wrong if you try to cache the connection yourself. Just say no :)
You should enable connection pooling on your connection string. In that case the runtime will add back your connections to the 'pool' when you close them, instead of really disconencting. When a 'new' connection is taken out of the pool it will be reset (ie. sp_reset_connection is called ) then presented to your application as a brand new, fresh connection. The pool is handling transparently such cases as if the connection is closed while idling in the pool.
The cost of creating a new connection 'from scratch' is significant because the authentication requires several roundtrips between client and server (depending on the authentication method and on SSL settings it can be 1 roundtrip in best case vs. about 10 in worse).
And to answer your question, connection raise the OnStateChange event when their state changes, but you shouldn't care about this if you use the pooling.
In my recent experience if you use this code:
using(SQLConnection conn = new SQLConnection(connectionString))
{
// do stuff with conn
}
have an error, and do not explicitly close the connection, it will not be closed or checked back into the pool. So use a catch or finally block to close the connection