I am trying to write a background process that checks to see if a database is broken and I am a little bit confused on what exactly would constitute as "broken".
Looking at the official documentation found here on the Microsoft Developers Network for ConnectionState there is a member entitled "broken". At what point would this member result to true, or how exactly would it be used?
This is currently how I am checking if the DB is broken:
public bool DatabaseConnection()
{
bool statusUp = true;
using (var databaseConnection = new SqlConnection(ConfigData.ConnectionStrings.DatabaseConnectionString))
{
try
{
databaseConnection.Open()
}
catch (SqlException ex)
{
const string message = "Could not establish a connection with Database.";
Log.DatabaseStatusDown(message, ex);
statusUp = false;
}
finally { databaseConnection.Close(); }
}
return statusUp
}
I know Using statement leverages the IDisposable class and the connection will be disposed of but I am extra paranoid. Is this efficient? If not, would a more efficient way to determine if my connection is broken would be to do something like this?
public bool DatabaseConnection()
{
using (var databaseConnection = new SqlConnection(ConfigData.ConnectionStrings.DatabaseConnectionString))
{
return databaseConnection.State == ConnectionState.Broken;
}
}
I will be running this process every two minutes, and something tells me the first method I outlined will not be efficient. Would the second method work to determine if my DB is broken? What exactly does Microsoft define as broken for this particular enum?
I wouldn't use ConnectionState.Broken. It is reserved for future use.
The first technique is actually pretty lightweight. All it does is get a connection from the connection pool, which is held locally. Disposing the connection will return the connection to the pool for use by other processes.
I would perhaps consider actually sending a command to the SQL Server, e.g. "SELECT 'ping'" or something lightweight. If you don't get a resultset back it indicates that your SQL Server couldn't service the request for whatever reason.
Not an expert, but I believe broken would indicate that an already connected database connection had an unrecoverable connection issue (Server closed it, etc). It wouldn't be very reliable, and then possibly only detectable after a failed attempt to do something.
It doesn't make sense to check for Broken on a connection you just made. It's only useful for a long-living connection - and it basically tells you to reopen the connection.
It doesn't tell you anything about the state of the database, or the database server. The only thing it tells you is whether the concrete connection is working or not.
If you're always creating new connections, the only thing you care about is whether connection.Open throws an exception or not. And of course, the ExecuteXXX methods etc. - the connection can drop at any point.
I'm working on an ASP.NET application where, as part of some logic, I want to lock some tables and do work on them. The method runs in a separate thread running as a kind of background task, spawned via a Task. The problem comes in with the error handling...
The code looks more or less like this:
MySqlConnection connection = new MySqlConnection(ConfigurationManager.AppSettings["prDatabase"]);
try
{
connection.Open();
MySqlCommand lock_tables = new MySqlCommand(Queries.lockTables(), connection);
lock_tables.ExecuteNonQuery();
// do a bunch of work here
MySqlCommand unlock_tables = new MySqlCommand(Queries.unlockTables(), connection);
unlock_tables.ExecuteNonQuery();
}
catch (MySqlException mex)
{
// Mostly error logging here
}
finally
{
connection.Close();
}
Pretty simple stuff. Everything works fine and dandy assuming nothing goes wrong. That's a terrible assumption to make, though, so I deliberately set up a situation where things would foul up in the middle and move to the finally block.
The result was that my table locks remained until I closed the app, which I learned by trying to access the tables with a different client once the method completed. Needless to say this isn't my intention, especially since there's another app that's supposed to access those tables once I'm done with them.
I could quickly fix the problem by explicitly releasing the locks before closing the connection, but I'm still left curious about some things. Everything I've read before has sworn that closing a connection should implicitly release the table locks. Obviously in this case it isn't. Why is that? Does connection.Close() not actually completely close the connection? Is there a better way I should be closing my connections?
Try wrapping your Connection and MySqlCommand instance in a using statement. That will release the objects as soon as it leaves the brackets.
using(MySqlConnection conn = new MySqlConnection(connStr))
{
conn.Open();
using(MySqlCommand command = new MySqlCommand("command to execute",conn))
{
//Code here..
}
}
At the moment our code looks like this
public void Close(bool saveChange)
{
if ((_Connection != null) && (_Connection.State == System.Data.ConnectionState.Open))
{
_Connection.Close();
_Connection.Dispose();
_Connection = null;
GC.Collect();
}
}
Where GC.Collect() is needed to close file properly and remove all .ldb files.
Is it possible to close db file and immideately remove all temp .ldb files without calling GC ?
As every class implementing IDisposable, you should use the using-statement to dispose objects. On this way connections are also getting closed.
For example:
using(var conn = new OleDbConnection("Provider=Microsoft.Jet.OLEDB.4.0; Data Source=" + Server.MapPath("Pets/Pets.mdb")))
{
conn.open();
// do something with it
}
You should also read this because the way you're using connections can be improved.
As a rule of thumb: always create,open,use and close connections at the same place and let the ADO.NET connection pool manage the underlying connections.
If you have a Class that is using the Data in several entry-points.
and you HAVE to keep the connection open.
(hence can not use using(Connection x=...){} statement)
.
your class should also inherit IDisposable and implement
void Dispose()
{
this.dbConnection.Dispose();
}
the dbConnection.Dispose() will take care of closing the connection gracefully unless you need some custom actions performed (e.g. Logout, Save Cached Info, Etc...)
Anyway, i am in favor of #Tim Schmelter idea:
Close the connection after any transaction.
The Connections will be cached in the Connection Pool => insignificant performance impact.
Calling GC.Collect() may not be the best solution, as this method collects all the "garbage". You may find it very helpful to make use of the using keyword. e.g.
using (OleDbConnection connection = new OleDbConnection(connectionString))
{
connection.Open();
// Do work here.
}
Just right after your using block, the connection object will be disposed!
using connections, msdn
using keyword, msdn
connection.Close();
OleDbConnection.ReleaseObjectPool();
This solved it for me eventually
I have a simple while loop that checks the database for inserts. If its in the loop too long my try catch gets "max pool size was reached" error. So at the bottom of loop I have a connectionclearallpools(); But that still doesn't solve it.
while (!quit)
{
connection to database strings timeout=400
read from database
connection.clearallpools();
}
Probably you are not closing your connections...you may want to use
while(!quit){
//do something here
using(var connection = GetMyConnection()){
//do your db reads here
}
//do validations and something more here
}
This would ensure your connections are disposed / closed properly.
Also, you do not need to clear your pools.
SQLConnection / DBConnection objects implements IDisposable. You may want to go thru these
Digging into IDisposable
C# Using Statement with SQL Connection
You most probably keep opening new connections in the loop.
Above the loop open the connection is a using statement and then use it in the loop. Also note the removal of the clearallpools:
using(create new connection)
{
while (!quit)
{
connection to database strings timeout=400
read from database
// connection.clearallpools(); REMOVE THIS!!!!
}
}
You're exhausting the connection pool. Following each read you should close the connection which will release it back to the pool.
I'm performing a large number of INSERTS to a SQLite database. I'm using just one thread. I batch the writes to improve performance and have a bit of security in case of a crash. Basically I cache up a bunch of data in memory and then when I deem appropriate, I loop over all of that data and perform the INSERTS. The code for this is shown below:
public void Commit()
{
using (SQLiteConnection conn = new SQLiteConnection(this.connString))
{
conn.Open();
using (SQLiteTransaction trans = conn.BeginTransaction())
{
using (SQLiteCommand command = conn.CreateCommand())
{
command.CommandText = "INSERT OR IGNORE INTO [MY_TABLE] (col1, col2) VALUES (?,?)";
command.Parameters.Add(this.col1Param);
command.Parameters.Add(this.col2Param);
foreach (Data o in this.dataTemp)
{
this.col1Param.Value = o.Col1Prop;
this. col2Param.Value = o.Col2Prop;
command.ExecuteNonQuery();
}
}
this.TryHandleCommit(trans);
}
conn.Close();
}
}
I now employ the following gimmick to get the thing to eventually work:
private void TryHandleCommit(SQLiteTransaction trans)
{
try
{
trans.Commit();
}
catch (Exception e)
{
Console.WriteLine("Trying again...");
this.TryHandleCommit(trans);
}
}
I create my DB like so:
public DataBase(String path)
{
//build connection string
SQLiteConnectionStringBuilder connString = new SQLiteConnectionStringBuilder();
connString.DataSource = path;
connString.Version = 3;
connString.DefaultTimeout = 5;
connString.JournalMode = SQLiteJournalModeEnum.Persist;
connString.UseUTF16Encoding = true;
using (connection = new SQLiteConnection(connString.ToString()))
{
//check for existence of db
FileInfo f = new FileInfo(path);
if (!f.Exists) //build new blank db
{
SQLiteConnection.CreateFile(path);
connection.Open();
using (SQLiteTransaction trans = connection.BeginTransaction())
{
using (SQLiteCommand command = connection.CreateCommand())
{
command.CommandText = DataBase.CREATE_MATCHES;
command.ExecuteNonQuery();
command.CommandText = DataBase.CREATE_STRING_DATA;
command.ExecuteNonQuery();
//TODO add logging
}
trans.Commit();
}
connection.Close();
}
}
}
I then export the connection string and use it to obtain new connections in different parts of the program.
At seemingly random intervals, though at far too great a rate to ignore or otherwise workaround this problem, I get unhandled SQLiteException: Database file is locked. This occurs when I attempt to commit the transaction. No errors seem to occur prior to then. This does not always happen. Sometimes the whole thing runs without a hitch.
No reads are being performed on these files before the commits finish.
I have the very latest SQLite binary.
I'm compiling for .NET 2.0.
I'm using VS 2008.
The db is a local file.
All of this activity is encapsulated within one thread / process.
Virus protection is off (though I think that was only relevant if you were connecting over a network?).
As per Scotsman's post I have implemented the following changes:
Journal Mode set to Persist
DB files stored in C:\Docs + Settings\ApplicationData via System.Windows.Forms.Application.AppData windows call
No inner exception
Witnessed on two distinct machines (albeit very similar hardware and software)
Have been running Process Monitor - no extraneous processes are attaching themselves to the DB files - the problem is definitely in my code...
Does anyone have any idea whats going on here?
I know I just dropped a whole mess of code, but I've been trying to figure this out for way too long. My thanks to anyone who makes it to the end of this question!
brian
UPDATES:
Thanks for the suggestions so far! I've implemented many of the suggested changes. I feel that we are getting closer to the answer...however...
The code above technically works however it is non-deterministic! It is not guaranteed to do anything aside from spin in neutral forever. In practice it seems to work somewhere between the 1st and 10th iteration. If i batch my commits at a reasonable interval damage will be mitigated but I really do not want to leave things in this state...
More suggestions welcome!
It looks like you failed to link the command with the transaction you've created.
Instead of:
using (SQLiteCommand command = conn.CreateCommand())
You should use:
using (SQLiteCommand command = new SQLiteCommand("<INSERT statement here>", conn, trans))
Or you can set its Transaction property after its construction.
While we are at it - your handling of failures is incorrect:
The command's ExecuteNonQuery method can also fail and you are not really protected. You should change the code to something like:
public void Commit()
{
using (SQLiteConnection conn = new SQLiteConnection(this.connString))
{
conn.Open();
SQLiteTransaction trans = conn.BeginTransaction();
try
{
using (SQLiteCommand command = conn.CreateCommand())
{
command.Transaction = trans; // Now the command is linked to the transaction and don't try to create a new one (which is probably why your database gets locked)
command.CommandText = "INSERT OR IGNORE INTO [MY_TABLE] (col1, col2) VALUES (?,?)";
command.Parameters.Add(this.col1Param);
command.Parameters.Add(this.col2Param);
foreach (Data o in this.dataTemp)
{
this.col1Param.Value = o.Col1Prop;
this. col2Param.Value = o.Col2Prop;
command.ExecuteNonQuery();
}
}
trans.Commit();
}
catch (SQLiteException ex)
{
// You need to rollback in case something wrong happened in command.ExecuteNonQuery() ...
trans.Rollback();
throw;
}
}
}
Another thing is that you don't need to cache anything in memory. You can depend on SQLite journaling mechanism for storing incomplete transaction state.
Run Sysinternals Process Monitor and filter on filename while running your program to rule out if any other process does anything to it and to see what exacly your program is doing to the file. Long shot, but might give a clue.
We had a very similar problem using nested Transactions with the TransactionScope class. We thought all database actions occurred on the same thread...however we were caught out by the Transaction mechanism...more specifically the Ambient transaction.
Basically there was a transaction higher up the chain which, by the magic of ado, the connection automatically enlisted in. The result was that, even though we thought we were writing to the database on a single thread, the write didn't really happen until the topmost transaction was committed. At this 'indeterminate' point the database was written to causing it to be locked outside of our control.
The solution was to ensure that the sqlite database did not directly take part in the ambient transaction by ensuring we used something like:
using(TransactionScope scope = new TransactionScope(TransactionScopeOptions.RequiresNew))
{
...
scope.Complete()
}
Things to watch for:
don't use connections across multiple threads/processes.
I've seen it happen when a virus scanner would detect changes to the file and try to scan it. It would lock the file for a short interval and cause havoc.
I started facing this same problem today: I'm studying asp.net mvc, building my first application completely from scratch. Sometimes, when I'd write to the database, I'd get the same exception, saying the database file was locked.
I found it really strange, since I was completely sure that there was just one connection open at that time (based on process explorer's listing of active file handles).
I've also built the whole data access layer from scratch, using System.Data.SQLite .Net provider, and, when I planned it, I took special care with connections and transactions, in order to ensure no connection or transaction was left hanging around.
The tricky part was that setting a breakpoint on ExecuteNonQuery() command and running the application in debug mode would make the error disappear!
Googling, I found something interesting on this site: http://www.softperfect.com/board/read.php?8,5775. There, someone replied the thread suggesting the author to put the database path on the anti-virus ignore list.
I added the database file to the ignore list of my anti-virus (Microsoft Security Essentials) and it solved my problem. No more database locked errors!
Is your database file on the same machine as the app or is it stored on a server?
You should create a new connection in every thread. I would simplefy the creation of a connection, use everywhere: connection = new SQLiteConnection(connString.ToString());
and use a database file on the same machine as the app and test again.
Why the two different ways of creating a connection?
These guys were having similiar problems (mostly, it appears, with the journaling file being locked, maybe TortoiseSVN interactions ... check the referenced articles).
They came up with a set of recommendations (correct directories, changing journaling types from delete to persist, etc). http://sqlite.phxsoftware.com/forums/p/689/5445.aspx#5445
The journal mode options are discussed here: http://www.sqlite.org/pragma.html . You could try TRUNCATE.
Is there a stack trace during the exception into SQL Lite?
You indicate you "batch my commits at a reasonable interval". What is the interval?
I would always use a Connection, Transaction and Command in a using clause. In your first code listing you did, but your third (creating the tables) you didn't. I suggest you do that too, because (who knows?) maybe the commands that create the table somehow continue to lock the file. Long shot... but worth a shot?
Do you have Google Desktop Search (or another file indexer) running? As previously mentioned, Sysinternals Process Monitor can help you track it down.
Also, what is the filename of the database? From PerformanceTuningWindows:
Be VERY, VERY careful what you name your database, especially the extension
For example, if you give all your databases the extension .sdb (SQLite Database, nice name hey? I thought so when I choose it anyway...) you discover that the SDB extension is already associated with APPFIX PACKAGES.
Now, here is the cute part, APPFIX is an executable/package that Windows XP recognizes, and it will, (emphasis mine) ADD THE DATABASE TO THE SYSTEM RESTORE FUNCTIONALITY
This means, stay with me here, every time you write ANYTHING to the database, the Windows XP system thinks a bloody executable has changed and copies your ENTIRE 800 meg database to the system restore directory....
I recommend something like DB or DAT.
While the lock is reported on the COMMIT, the lock is on the INSERT/UPDATE command. Check for record locks not being released earlier in your code.