I have a simple while loop that checks the database for inserts. If its in the loop too long my try catch gets "max pool size was reached" error. So at the bottom of loop I have a connectionclearallpools(); But that still doesn't solve it.
while (!quit)
{
connection to database strings timeout=400
read from database
connection.clearallpools();
}
Probably you are not closing your connections...you may want to use
while(!quit){
//do something here
using(var connection = GetMyConnection()){
//do your db reads here
}
//do validations and something more here
}
This would ensure your connections are disposed / closed properly.
Also, you do not need to clear your pools.
SQLConnection / DBConnection objects implements IDisposable. You may want to go thru these
Digging into IDisposable
C# Using Statement with SQL Connection
You most probably keep opening new connections in the loop.
Above the loop open the connection is a using statement and then use it in the loop. Also note the removal of the clearallpools:
using(create new connection)
{
while (!quit)
{
connection to database strings timeout=400
read from database
// connection.clearallpools(); REMOVE THIS!!!!
}
}
You're exhausting the connection pool. Following each read you should close the connection which will release it back to the pool.
Related
I'm trying to figure out the best way to batch insert about 37k rows into my Sql Server using DAPPER.
My problem is that when I use Parallel.ForEach - the number of connections to the database increases over a short period of time - finally hitting nearly or about 100 ... which gives connection pool errors. If I force the max degree of parall then it's hit that max number and stays there.
Setting the maxdegree feels wrong.
It currently is doing about 10-20 inserts a second. This is also in a simple Console App - so there's no other database activity besides what's happening in my Parallel.ForEach loop.
Is using Parallel.ForEach the incorrect thing in this case because this is not-CPU bound?
Should I be using async/await ? If so, what stopping this from doing hundreds of db calls in one go?
Sample code which is basically what I'm doing.
var items = GetItemsFromSomewhere(); // Returns 37K items.
Parallel.ForEach(items => item)
{
using (var sqlConnection = new SqlConnection(_connectionString))
{
var result = sqlConnection.Execute(myQuery, new { ... } );
}
}
My (incorrect) understanding of this was that there should on be about 8 or so connections at any time to the db. The Connection Pool will release the connection (which remains instantiated in the Connection Pool, waiting to be used). And if the Execute takes .. i donno .. lets say even a 1 second (the longest running time for an insert was about 500ms .. and that's 1 in every 100 or so) ... that's ok .. that thread is blocked and chills until the Execute completes. Then the scope completes (and Dispose is auto called) and the connection closed. With the connection closed, the Parallel.ForEach then grabs the next item in the collection, goes to the connection pool and then grabs a spare connection (remember - we just closed one, a split second ago) ... rinse.repeat.
Is this wrong?
Notes:
.NET 4.5
Sql 2012
Console app.
Using Dapper.NET for sql code.
First of all: If it is about performance, use SqlBulkCopy. This works with SQL-Server. If you are using other database servers, they might have their own SqlBulkCopy-solution (Oracle has one).
SqlBulkCopy works like a bulk-select: One state opens one connection and streams all the data from the server to the client. With an insert, it works the other way arround: It streams all the new records from the client to the server.
See: https://msdn.microsoft.com/en-us/library/ex21zs8x(v=vs.110).aspx
If you insist of using parallellism, you might want to consider the follow code:
void BulkInsert<T>(object p)
{
IEnumerator<T> e = (IEnumerator<T>)p;
using (var sqlConnection = new SqlConnection(_connectionString))
{
while(true)
{
T item;
lock(e)
{
if (!e.MoveNext())
return;
item = e.Current;
}
var result = sqlConnection.Execute(myQuery, new { ... } );
}
}
}
Now create your own threads and invoke this method on these threads with one and the same parameter: The iterator which runs through your collection. Each threat opens its own connection once, starts inserting, and after all items are inserted, the connection is closed. This solutions uses as many connections as your created threads.
PS: Multiple variants of above code are possible . You could call it from background threads, from Tasks, etc. I hope you get the point.
You should use SqlBulkCopy instead of inserting one by one. Faster and more efficient.
https://msdn.microsoft.com/en-us/library/ex21zs8x(v=vs.110).aspx
credits to the answer owner
Sql Bulk Copy/Insert in C#
Some first things that people learned in their early use of MySQL that closing connection right after its usage is important, but why is this so important? Well, if we do it on a website it can save some server resource (as described here) But why we should do that on a .NET desktop application? Does it share the same issues with web application? Or are there others?
If you use connection pooling you won't close the physical connection by calling con.Close, you just tell the pool that this connection can be used. If you call database stuff in a loop you'll get exceptions like "too many open connections" quickly if you don't close them.
Check this:
for (int i = 0; i < 1000; i++)
{
var con = new SqlConnection(Properties.Settings.Default.ConnectionString);
con.Open();
var cmd = new SqlCommand("Select 1", con);
var rd = cmd.ExecuteReader();
while (rd.Read())
Console.WriteLine("{0}) {1}", i, rd.GetInt32(0));
}
One of the possible exceptions:
Timeout expired. The timeout period elapsed prior to obtaining a
connection from the pool. This may have occurred because all pooled
connections were in use and max pool size was reached.
By the way, the same is true for a MySqlConnection.
This is the correct way, use the using statement on all types implementing IDsiposable:
using (var con = new SqlConnection(Properties.Settings.Default.ConnectionString))
{
con.Open();
for (int i = 0; i < 1000; i++)
{
using(var cmd = new SqlCommand("Select 1", con))
using (var rd = cmd.ExecuteReader())
while (rd.Read())
Console.WriteLine("{0}) {1}", i, rd.GetInt32(0));
}
}// no need to close it with the using statement, will be done in connection.Dispose
Yes I think it is important to close out your connection rather than leaving it open or allowing the garbage collector to eventually handle it. There are a couple of reason why you should do this and below that I'll describe the best method for how
WHY:
So you've opened a connection to the database and sent some data back and forth along this pipeline and now have the results you were looking for. Ideally at this point you do something else with the data and the end results of your application is achieved.
Once you have the data from the database you don't need it anymore, its part in this is done so leaving the connection open does nothing but hold up memory and increase the number of connections the database and your application has to keep track of and possibly pushing you closer to your maximum number of connections limit.
"But wait! I have to make a lot of database calls in rapid
succession!"
Okay no problem, open the connection run your calls and then close it out again. Opening a connection to a database in a "modern" application isn't going to cost you a significant amount of computing power/time, while explicitly closing out a connection does nothing but help (frees up memory, lowers your number of current connections).
So that is the why, here is the how
HOW:
So depending on how you are connecting to your MySQL database you a probably using an IDisposible object to help manage the connection. Here is what MSDN has to say on using an IDisposable:
As a rule, when you use an IDisposable object, you should declare and
instantiate it in a using statement. The using statement calls the
Dispose method on the object in the correct way, and (when you use it
as shown earlier) it also causes the object itself to go out of scope
as soon as Dispose is called. Within the using block, the object is
read-only and cannot be modified or reassigned.
Here is my personal take on the subject:
Using a using block helps to keep your code cleaner (readability)
Using a usingblock helps to keep your code clear (memory wise), it will "automagically" clean up unused items
With a usingblock it helps to prevent using a previous connection from being used accidentally as it will automatically close out the connection when you are done with it.
In short, I think it is important to close connections properly, preferably with a con.close() type statement method in combination with a using block
As pointed out in the comments this is also a very good question/answer similar to yours: Why always close Database connection?
We have a web server which connects to a database using a single connection string, which makes it a strong candidate for being able to use a Connection Pool.
Do we need one SqlConnection object, or many?
i.e. Should we set up one connection in shared memory, and use that each time, or should we create a new one each time we want to use any connection object?
Is it the call to .Open() which assigns it from a pool, or the creating of a new object with the same connection string?
Also, do we need to call .Close() on the connection before it's released back into the pool, or is the variable going out of scope enough?
I read somewhere (I forget where exactly - sorry) that you shouldn't call close on connections in a pool, because it removes them from the pool somehow.
It’s worth bearing in mind that an identical connection string will re-use a connection which has been returned the the pool, but changing it in any way will cause a new connection to be made to the server.
i.e.
Assuming that SQLBox has IP 10.0.0.1
using (var conn = new SqlConnection(#"Server=10.0.0.1;...") {
conn.Open();
.. Do work ..
conn.Close();
}
using (var conn = new SqlConnection(#"Server=SQLBox;…") {
conn.Open(); // This will *NOT* reuse the connection from the pool.
.. Do work ..
conn.Close();
}
using (var conn = new SqlConnection(#"Server=10.0.0.1;...") {
conn.Open(); // This *WILL* reuse the connection from the pool.
.. Do work ..
conn.Close();
}
You should open a separate connection every time you need to access a database and close it once you're done with all the access. Do not keep your connections open for too long. It is not necessary and modern databases are definitely very good at handling multiple connections.
You can leave the management of the connection pool to SQL Server - it will do a good job as long as you'll not try to prevent it.
It's best to use local connection object and not share it across multiple parts of the code. You can use using pattern to make sure your connection will be closed:
using (SqlConnection conn = new SqlConnection("connectionstring"))
{
conn.Open();
// use your connection here
}
lock (_connectionLock) {
if (conn == null) {
conn = GetOpenConnection(connectionString);
}
try {
PerformDbAction(conn);
} finally {
conn.Dispose();
}
}
I have run into a problem where multithreading can cause issues with null connections as they can be opened and closed by several threads running at once. I tried to solve the issue by locking the process (above, code simplified for clarity) but have found that this seriously slows down the performance.
I tried to get around this problem by using two separate locks for the creation/disposal of database connections and to perform database action outside of locking:
lock (_connectionLock) {
if (conn == null) {
conn = GetOpenConnection(connectionString);
}
}
try {
PerformDbAction(conn);
} finally {
lock(_connectionLock)
conn.Dispose();
}
}
Only I realized that the above doesn't work as another thread may try to perform a database action with a connection that has already been disposed by another thread.
Could anyone suggest an alternative or solution where I can safely lock access to the database connection strings without slowing everything down so much?
EDIT: Sorry for not including this previously, but the reason I am not just creating new connections and disposing of them immediately is that I am trying to avoid unwanted MSDTC escalation. In using GetOpenConnection I am reusing an existing connection as this is one of the things that triggers MSDTC escalation.
I have managed to avoid the escalation with the top code example, but it performs way too slow.
Simply don't have one shared connection variable. Instead, each time you need to do something, open a connection, use it, and close it as soon as you can. You don't need to use any locks in your code, and the connection pool will manage the real network connections to the database.
At the moment, you've essentially built a primitive connection pool containing exactly one connection, which means you've got no concurrency at all in the database (well, not per process).
At the moment our code looks like this
public void Close(bool saveChange)
{
if ((_Connection != null) && (_Connection.State == System.Data.ConnectionState.Open))
{
_Connection.Close();
_Connection.Dispose();
_Connection = null;
GC.Collect();
}
}
Where GC.Collect() is needed to close file properly and remove all .ldb files.
Is it possible to close db file and immideately remove all temp .ldb files without calling GC ?
As every class implementing IDisposable, you should use the using-statement to dispose objects. On this way connections are also getting closed.
For example:
using(var conn = new OleDbConnection("Provider=Microsoft.Jet.OLEDB.4.0; Data Source=" + Server.MapPath("Pets/Pets.mdb")))
{
conn.open();
// do something with it
}
You should also read this because the way you're using connections can be improved.
As a rule of thumb: always create,open,use and close connections at the same place and let the ADO.NET connection pool manage the underlying connections.
If you have a Class that is using the Data in several entry-points.
and you HAVE to keep the connection open.
(hence can not use using(Connection x=...){} statement)
.
your class should also inherit IDisposable and implement
void Dispose()
{
this.dbConnection.Dispose();
}
the dbConnection.Dispose() will take care of closing the connection gracefully unless you need some custom actions performed (e.g. Logout, Save Cached Info, Etc...)
Anyway, i am in favor of #Tim Schmelter idea:
Close the connection after any transaction.
The Connections will be cached in the Connection Pool => insignificant performance impact.
Calling GC.Collect() may not be the best solution, as this method collects all the "garbage". You may find it very helpful to make use of the using keyword. e.g.
using (OleDbConnection connection = new OleDbConnection(connectionString))
{
connection.Open();
// Do work here.
}
Just right after your using block, the connection object will be disposed!
using connections, msdn
using keyword, msdn
connection.Close();
OleDbConnection.ReleaseObjectPool();
This solved it for me eventually