I've got a sql server 2k8 database that's being populated with several hundred thousand records per day.
I'm currently writing some code that's going to make a call to the db, retrieve n records, process them, and write some data back to the db.
There are two ways I see to do this (psuedo code):
function xyz() {
conn = conn creation code
conn.open();
while(not last record) {
select next 1000 records
process each record
last record = true
}
conn.close();
xyz();
}
Essentially creating one connection per batch. The second method:
function xyz() {
conn = connection creation code
while(conn.open();) {
select next 1000 records
process each record
last record = true
}
conn.close();
xyz();
}
I'm curious what's better practice. I feel like it's the latter, but it's also going to have a more-or-less permanent/persistent connection to my db. I'm worried about possible memory overruns or some such.
Thoughts?
Scott
ADO.Net Sqlclient provider (which is I assume you will use, since you say is C#) automatically does connection pooling, see SQL Server Connection Pooling (ADO.NET). Pooled connection are not truly closed when you call Close, they are simply returned to the pool. 'Opening' and 'closing' pooled connection is very fast.
unrelated note: you should embed your connection in a using block:
using (SqlConnection conn = new SqlConnection(...))
{
conn.Open ();
...
}
This way you avoid leaking connecitons in exception conditions.
I would handle the connection outside of the function and pass it in. The function is designed to process entries, not to connect to the database and process entries. I would separate those two jobs.
As far as connection opening/closing goes: yes, you should avoid it. It's not too terribly slow on it's own, but if you're just doing processing, there's no reasons to open/close the connection over and over again. If it's due to memory usage (I'm not very familiar with C# by the way, so this part might just be wrong), you should be freeing the result sets. Closing the connection will free the memory associated with it, however, you should be able to get the same effect without having to reconnect.
It depends on what "process each record" means. If that takes a notable amount of time and does not need to maintain the connection, you might want to reconsider keeping the connection open and let the pool handle it.
But if it is constantly reading from/to the connection, you should not artificially disconnect/reconnect.
There is no reason to keep closing and re-opening the connection. It imposes needless load on the database and the network.
Related
I'm writing a program that generates Sitemaps. To avoid duplicates I'm using MSSQL Server to store links that are found. during this process It may read and write millions of links and the process of reading and writing may have very little gaps between them (I mean the time between each access to database is very tiny).
I want to know If I can open the connection when the program starts and close it at the end. please consider that creating some sitemaps may take days to finish. Or is it wise to open and close connection each time I want to access the db?
By the way, I'm using sql server locally.
The answer to your question is the Connection Pooling feature.
It is always preferable to open the connection just when needed, access your data, close and dispose the connection immediately
Indeed a specific work pattern exists
using(SqlConnection con = new SqlConnection(.....))
{
... do your work with the database here ....
}
The using statement ensures that the object created at the opening is closed and disposed at the closing braces. Even if an Exception occurs in database code.
This doesn't mean that you cannot try to optimize your code. For example you could gather a few of sitemaps together and write them in just one shot, but usually it is best to follow a simple work pattern, finish the work as soon as possible and worry for optimization later.
You can work with the following code :
SqlConnection conn;
try
{
conn = new SqlConnection(_dbconnstr));
}
catch
{
//exceptions are bubbled
throw;
}
finally
{
//Dispose is always called
conn.Dispose();
}
hope this helps!
I usually open and close connection each time I use it.
Releasing resources must be done when there is many connections around.
If you are the only one, you might keep your resources.
Be aware of setting timeout connection.
Good luck guy!
It is better to open/close the connection. There is no benefits in kipping connection open. ADO.NET Connection pooling (which is on by default) will take care of which connection to close and when.
From MSDN
CAUTION It is recommended that you always close the Connection when
you are finished using it in order for the connection to be returned
to the pool. This can be done using either the Close or Dispose
methods of the Connection object. Connections that are not explicitly
closed might not be added or returned to the pool. For example, a
connection that has gone out of scope but that has not been explicitly
closed will only be returned to the connection pool if the maximum
pool size has been reached and the connection is still valid.
If you know it's going to take a while, what about writing to a local data table and then when you reach a threshold, write all the records to the database.
Or even write the info to a local file (format of your choice) and then write that file to the database in one shot (or multiple shots if you want to do it every x records or y minutes.)
I work with Windows-Mobile and Windows-CE using SqlCE and I dont know what better to do.
To open connection when the program open, run any query's... update...delete database and close the connection after the program close?
Or open connection run any query's..update...delete database and close the connection immediately?
Nice. The answers are all over the place. Here's what I know from experience and interacting with the SQL Compact team:
Closing the connection flushes the changes you've made, otherwise the engine waits for the flush period before doing it. It's a good idea to close the connection when you're done using it to ensure that your changes actually go to the store. A power loss after a write and before a flush will lose data.
There is no official connection pool, but opening the first connection is expensive (i.e. slow), all others are quick. The recommendation I got from the team is to actually create a connection when the app starts up and just leave it open. You don't actually need to use it, but keeping it open keeps a lot of connection info cached so that subsequent connections to the same store are quick.
So the answer, actually, is both.
Edit
For those interested, a good example of how this works can be seen in the OpenNETCF ORM library. The library, by default, creates a "maintenance" connection that remains open and is used for doing things like schema queries. All other data operations use their own connection. You also have to option to configure the library to reuse a single connection for the life of the Store, or to use a new connection every time it touches the store. Perfomance and behavior has always been best in all of my projects using the default (which is why I made it the default).
Always keep a connection open for the lifetime of your Windows Mobile app. Opening a SQL Server Compact database is a costly operation.
You should close your connection each time you have completed an sql transaction to free connection ports. Always a good practice to avoid security breach.
Connection establishment is a slow operation, so, creating and closing it can slow down the application. On the opposite hand, if you have a lot of clients, the connection pool will be filled very quickly and other clients won't be able to connect.
There are already some conflicting answers here.
To be honest, I'm not enirely sure how WinCE deals with connections. I don't think there is a ConnectionPool.
But the general pattern in .NET is to keep connections open as short as possible. This improves reliability and prevents resource leaks. Make sure you know about the using (var conn = ...) { ... } pattern.
So I would say: go with your second option, and only keep connections longer if you really experience a performance problem, and if opening the connection is the cause. I don't think it will be with SqlCE
On a single-user platform such as wince, there's no harm in keeping the connection open, and you may get better performance.
If worry about data lost because you are not calling Close() frequently, you can execute your code within a transaction that commits changes to disk immediately:
using (SqlCeTransaction transaction = this.connection.BeginTransaction())
{
using (SqlCeCommand command = new SqlCeCommand(query, connection))
{
command.Transaction = transaction;
command.ExecuteNonQuery();
}
transaction.Commit(CommitMode.Immediate);
}
Of course, there is still some performance lost when using CommitMode.Immediate too frequently.
If connecting to a database consumes a lot of resources, why should a database connection always be closed in your application if you have to open it again? Can I just make this connection available globally throughout my application so that other classes and methods can reuse it?
For example (in pseudo code):
public class PopulateGridViews()
{
public SqlConnection conn = new SqlConnection(#"Database:DATABASE");
conn.Open();
void PopulateGrid1()
{
SqlCommand cmd = new SqlCommand("SELECT * FROM TABLE1");
cmd.Connection = conn;
cmd.ExecuteNonQuery();
cmd.Dispose();
// Populate Grid1
}
void PopulateGrid2()
{
SqlCommand cmd = new SqlCommand("SELECT * FROM TABLE2");
cmd.Connection = conn;
cmd.ExecuteNonQuery();
cmd.Dispose();
// Populate Grid2
}
}
You should not leave connections open.
You should:
Open connections as late as possible
Close connections as soon as possible
The connection itself is returned to the connection pool. Connections are a limited and relatively expensive resource. Any new connection you establish that has exactly the same connection string will be able to reuse the connection from the pool.
We strongly recommend that you always
close the connection when you are
finished using it so that the
connection will be returned to the
pool. You can do this using either the
Close or Dispose methods of the
Connection object, or by opening all
connections inside a using statement
in C#, or a Using statement in Visual
Basic. Connections that are not
explicitly closed might not be added
or returned to the pool. For more
information, see using Statement (C#
Reference) or How to: Dispose of a
System Resource for Visual Basic. Ref.
You should appropriately wrap anything that implements IDisposable in a using statement block:
using (SqlConnection connection = new SqlConnection(connectionString))
{
connection.Open();
...
command.ExecuteNonQuery();
}
Because (some) databases also hold open a connection until told by the calling app to close it. If you get hundreds of calls to a database then it's sitting there with 100 open connections tying up resources. It's not uncommon to have thousands or hundreds of thousands of calls to a database in a busy app, and sooner or later the DB performance will kill the app's performance.
It's really just common sense. If you have a valid reason to keep it open, then do so. If not, close it as soon as you're done with it. But it's better to be in the good habit of closing the connections so that you don't just leave them open when you don't intend to. It's a good habit like wearingyour seat belt or closing the refrigerator door when you're not getting food out.
This article states it well (even if it is a bit outdated):
http://www.bewebmaster.com/84.php
A common problem among hosting
companies is that ASP websites do not
close the database connections after
they are opened. This is a basic step
that you should consider to be part of
mandatory code. If you do not close
your database connections, many
problems can occur like web pages
hanging, slow page loads, and more.
Think of it as going through a door to
your house. Maybe the door will shut
by itself, but maybe it won't. If it
doesn't shut, who knows what will
happen. If you live in the country, a
bear could walk in. If you live in the
city, a mugger could walk in. Alright,
well maybe leaving a database
connection open won't lead to anything
that bad, but it will lead to a lot of
unnecessary headaches for both you and
your hosting company.
It is clear if you donot close the connection than it would consume your resource continuously and which would have overall impact on your apllication and it may also not be added or returned to the pool.
I understand that if I instantiate a SqlConnection object, I am really grabbing a connection from a connection pool. When I call Open(), it will open the connection. If I call the Close() or Dispose() method on that SqlConnection object, it is returned to the connection pool.
However, that doesn't really tell me if it's really closed, or if I still have an active connection to the database.
How can I force a SqlConnection to close at the network level, or at least tell when it closes?
Example:
using(SqlConnection conn = new SqlConnection(DBConnString)) {
conn.Open();
SqlCommand cmd = conn.CreateCommand();
...
cmd.ExecuteReader(CommandBehavior.CloseConnection);
...
}
First run: 300 ms
Second run: 100 ms
Third run: 100 ms
After waiting a long time (30 minutes): 300 ms
If the connection was TRULY closing, the second and third runs should also be 300 ms. But I know that the connection is not truly closed for those runs (I checked the SQL Server's activity monitor). It doesn't take the extra 200ms to perform authentication/etc.
How do I force the connection to truly close?
Ideas
Does CommandBehavior.CloseConnection work? (apparently not?)
Does setting "Max Pool Size = 0" in the connection string work? (this would be a pyrrhic solution)
Does Dispose() work?
References
Article on Connection Pooling
Here's another one that tells us that Close() doesn't really close the connection.
An article on pros and cons connection pooling
Maybe SqlConnection.ClearPool ?
Moe Sisko's answer (Call SqlConnection.ClearPool) is correct.
Sometimes you need a connection to really close rather than return to the pool. As an example, I have a unit test that creates a scratch database, builds the schema, tests some stuff, then drops the scratch database if the tests all pass.
When connection pooling is active, the drop database command fails because there are still active connections. From the point of view of programmer all SQLConnections are closed, but as the pool still holds one open, SQL Server won't allow the drop.
The best documentation for how connection pooling is handled is this page on SQL Server Connection Pooling on MSDN. One doesn't want to turn connection pooling off entirely because it improves performance with repeated opens and closes, but sometimes you need to call a "force close" on an SQLConnection so that it will let go of the database.
This is done with ClearPool. If you call SqlConnection.ClearPool(connection) before closing/disposing, when you do close/dispose it will really go away.
If you don't want to use the connection pool you have to specify it in your SqlConnection.ConnectionString property. For example
"Data Source=MSSQL1;Database=AdventureWorks;Integrated Security=true;Pooling=false;"
Disposing or closing the SqlConnection object is just going to close the connection and return it to the connection pool.
Generally, you want the connection pool to do its job - you don't want the connection to truly close.
Why specifically do you want the connection not to return to the pool?
I see that you are using .net but as this showed up in a google query allow me to give a java response...
Use a DataSource that implements Closeable() and call close on the DataSource. Hikari supports Closeable.
CommandBehavior.CloseConnection is usually discouraged because of this very fact - You can't be sure that the Connection will be closed. (I'll try to find some concrete evidence of this, I'm saying this from faint recall).
Dispose() is the surest way because it implicitly calls Close().
The using construct demonstrated by #Alex is just another (programmer friendly) way of writing the try-finally construct with the added implicit Disposal of objects.
Edit: (after edit to question)
Your concern over the connections actually closing seems unwarranted to me. The connection would simply return to the pool so that it can be reused easily without having to go through all the initialization. This does not mean that the Connection is still actively connected to the DB.
Robert's answer of SqlConnection.ClearPool(TheSqlConn) did exactly what I wanted. It's nice to know the pool CAN be interacted with when necessary.
My use case was: We have ruined a connection and let it go back into the pool, how to we detect that it's ruined and refresh it, so the next user won't have problems.
The solution was: Detect that we have just ruined the connection, and clear it from the pool entirely, letting the pool fill back up with fresh connections.
A decade of writing SqlClient.SqlConnection and I never even thought of interacting with the pool till today.
If you have a C# function with Sqlaccess, is it mandatory to close all objects/handles, or is everything cleaned up automatically once you exit the function
For example:
void DoSqlStuff()
{
SqlConnection sqlConn = new SqlConnection(...);
SqlCommand cmd = new SqlCommand(...);
SqlDataReader sqlData= null;
sqlConn,Open();
sqlData = cmd.ExecutReader();
while(sqlData.Read())
{
...
}
}
Is it optional, recommended or mandatory to close SqlConn and SqlData?
Thanks.
You should close the SqlConnection object as soon as you're done with it. If you don't then the connection will remain open, and will not be available to handle other requests.
The using statement is useful for this. It will call Dispose() on the object for you:
using (SqlConnection cn = new SqlConnection(connectionString))
{
SqlCommand cm = new SqlCommand(commandString, cn)
cn.Open();
cm.ExecuteNonQuery();
}
You don't need to have a separate using statement for the SqlDataReader (as well as one using statement for the connection) unless you plan do perform other operations with the connection after the SqlDataReader has fully read through the row set.
If you are just opening a connection, reading some data using the reader, and then closing the connection, then one using statement for the entire block of code (surrounding the connection) will suffice as the garbage collector will clean up all resources tied to the connection that is disposed by the first using statement.
Anyway, here's a good article that describes it all...
You should close everything before returning from the function. Open datareaders mean open cursors on the database, resulting in increased memory usage. Same goes for database connections.
Unused objects are not immediately freed in C#, but only when garbage collection is performed, which is not deterministic.
All three classes have a Dispose() method. Mandatory is too strong, but definitely highly recommended you use the using keyword so Dispose() is automatically called. Failing to do so makes your program run "heavy", using more system resources than necessary. And outright failure when you don't use the "new" keyword enough to trigger the garbage collector.
Calling Close on the SQL connection won't actually close it, but will return it to a connection pool to be reused, improving performance.
Additionally it is generally poor practice to not explicitly dispose of unmanaged resources when you are finished with them (asap).
Be careful with absolutes here. A lot depends on what you are doing & where the inefficiencies may lie.
In a Web Page where each user has a separate security context you may have no choice but to establish a new SQL connection with new security credentials with each page hit. Clearly nicer if you can use a pool of SQL connections with a shared security context & let the Web page filter the results but perhaps you can't.
In early versions of SQL Server ie (v6.5 or less) the Login Authentication was done by SQL Server. Also SQL was severely constrained by connection memory & the number of active connections it could handle. So it was a great idea to drop your connection when not in use.
Post v6.5, most people use Windows Authentication to login to SQL. This causes a lot of network calls between servers & some latency. Kerberos Security is even more chatty, Thus establishing a SQL connection is expensive. For that reason you need to find a balance between Holding a connection open for the life of your WinForms application vs Opening & closing it within each method call.
As a rough guide, if you think your app is going to want to talk to SQL in the next, say 30 secs. Keep the established connection open. If they've minimised your app, not touched it within a timeout period, or you've got all the data in RAM & they are unlikely to need anything more from the SQL system. Close the connection.
Consider creating a Class with a System Timer to hold the connection. Your class will always provide a valid connection, but perhaps the class will choose to drop it & free the connection load on SQL when appropriate.
Unless you are also writing Server based code, a small amount of memory inefficiency might not even be noticed. But 2-10,000 clients all poorly using your Security & Data Servers is likely to bring your Data Centre to its knees.
Explicit disposing in the finally statement is another approach, although the using statement is a much better solution. It produces a bit more code, but demonstrates the goal...
SqlConnection conn = null;
try
{
//create connection
SqlCommand cmd = null;
try
{
//create command
SqlDataReader reader = null;
try
{
//create reader
}
finally
{
reader.Dispose();
}
}
finally
{
cmd.Dispose();
}
}
finally
{
conn.Dispose();
}
Any class handling SQL stuff like Connections should implement the IDisposable interface as stated by Microsoft .NET coding guidelines.
Thus, you should probably close and dispose your connection in your Dispose method.