We have a SQL utility class that takes the name of a stored procedure an its input parameters, and returns the results in datatable. The reasoning behind this is so that we don't have to worry about forgetting to close connections and having connection leaks. Also so that we can reduce code by not having to recreate datadapters and datareaders in our data access layers.
The problem I have with this is that we're populating a datatable so that we can loop through it to create our objects, so we're basically using it like a datareader. I've read about classes that will return a datareader or dataadapter. But the problem with this is either client has to open and close connections, or you have to close the connection in a Finalize method. It seems that you wouldn't want garbage collection being responsible for closing your database connections.
To sum up, we want to have a class so that we can reduce code by not having to create datareaders for every query and so that we can ensure database connections are closed.
What is the best way of handling this?
UPDATE: Still thinking about this, but so far it seems that the best practice is to still return a datareader, use CommandBehavior.CloseConnection, and then trust who ever uses the class to call dr.Close()?
Have you considered the Microsoft Enterprise Library?
public List<User> GetUsers()
{
List<User> result = new List<User>();
Database db = new
Microsoft.Practices.EnterpriseLibrary.Data.Sql.SqlDatabase(this.connectionString);
DbCommand cmd = db.GetStoredProcCommand("GetUsers");
using (IDataReader rdr = db.ExecuteReader(cmd))
{
while (rdr.Read())
{
User user = new User();
FillUser(rdr, user);
result.Add(user);
}
}
return result;
}
We use something like this and it performs very well under high volume.
public SqlDataReader ExecuteReader(string command, SqlParameter[] parameters)
{
SqlDataReader reader = null;
using (SqlConnection conn = new SqlConnection())
using (SqlCommand cmd = conn.CreateCommand())
{
conn.Open();
cmd.CommandText = command;
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.AddRange(parameters);
reader = cmd.ExecuteReader(CommandBehavior.CloseConnection);
}
return reader;
}
DataTables are not considered best practice for several reasons including their bloat and lack of type safety.
I have the same structure - utility classes with methods that fetch the data and return filled DataTables (or fill/update a DataTable passed in to them) - for exactly the same reasons: keeping the database connections separate from the rest of the code and ensuring they are opened only when required and closed asap. Especially since the data is stored in various back-end systems, and I want to present only one interface to my application and not have it worry about the details.
There is one difference to your situation: We don't (in general) create objects from the rows in the DataTables, but rather work directly on the data in the rows. I find working with DataTables simple and efficient.
Other than that, I personally don't see anything wrong with this approach and find that it works very well for our purposes.
Returning a datareader doesn't work in a lot of scenarios. At a lot of places, direct connections to the database from the client machine are not allowed in production (for good reason). So you have to serialize the objects you are retrieving. I can think of designs that would allow you to persist a datareader in whatever class you use for remoting/serialization on the server side but returning items across http or nettcp in row by agonizing row fashion likely does not offer much benefit.
Are you serializing these objects? If so, your choices boil down to Datatable, Dataset, or custom objects. Custom objects, if written well, perform and serialize the best but you have to write concurrency in addition to a bunch of other functionality.
IMO, since ADO.Net 2.0, datatables can perform well even in large scale remoting situations. They provide a special binary remoting format and are simple to work with. Throw in some compression and you're not even using a lot of bandwidth for your large data sets.
well, if you plan to use this class inside of web pages you can register the utility class with the page's unload event. In the event sink you can write your logic to close the database connection. Check out this tip on codeproject for more ideas.
however this solution won't work for use inside web methods (web services). I suppose you'd have to adapt the technique for web service use. Your last line in the web method should should be an event call. So when you write your web service class, define an event called WebMethodCompleted. You'd probably get a reference to the instance of the web service via the technique mentioned in the article. Once you get a reference you can register tthe event in your utility class. Just remember to invoke the event in the web method.
Happy programming.
Related
I have two scenarios (examples below), both are perfectly legitimate methods of making a database request, however I'm not really sure which is best.
Example One - This is the method we generally use when building new applications.
private readonly IInterfaceName _repositoryInterface;
public ControllerName()
{
_repositoryInterface = new Repository(Context);
}
public JsonResult MethodName(string someParameter)
{
var data = _repositoryInterface.ReturnData(someParameter);
return data;
}
protected override void Dispose(bool disposing)
{
Context.Dispose();
base.Dispose(disposing);
}
public IEnumerable<ModelName> ReturnData(filter)
{
Expression<Func<ModelName, bool>> query = q => q.ParameterName.ToUpper().Contains(filter)
return Get(filter);
}
Example Two - I've recently started seeing this more frequently
using (SqlConnection connection = new SqlConnection(
ConfigurationManager.ConnectionStrings["ConnectionName"].ToString()))
{
var storedProcedureName = GetStoredProcedureName();
using (SqlCommand command = new SqlCommand(storedProcedureName, connection))
{
command.CommandType = CommandType.StoredProcedure;
command.Parameters.Add("#Start", SqlDbType.Int).Value = start;
using (SqlDataReader reader = command.ExecuteReader())
{
// DATA IS READ AND PARSED
}
}
}
Both examples use Entity Framework in some form (the first more so than the other), there are Model and Mapping files for every table which could be interrogated. The main thing the second example does over the first (regarding EF) is utilising Migrations as part of the Stored Procedure code generation. In addition, both implement the Repository pattern similar to that which is in the second link below.
Code First - MSDN
Contoso University - Tutorial
My understanding of Example One is that the repository and context are instantiated once the Controller is called. When making the call to the repository it returns the data but leaves the context intact until it is disposed of at the end of the method. Example Two on the other hand will call Dispose as soon as the database call is finished with (unless forced into memory, e.g. using .ToList() on an IEnumerable). If my understanding is not correct, please correct me where appropriate.
So my main question is what are the disadvantages and advantages of using one over the other? Example, is there a larger performance overhead of going with Example 2 compared to Example 1.
FYI: I've tried to search for an answer to the below but have been unsuccessful, so if you are of a similar question please feel free to point me in that direction.
You seem to be making a comparison like this:
Is it better to build a house or to install plumbing in the bathroom?
You can have both. You could have a repository (house) that uses data connections (plumbing) so it's not an "OR" situation.
There is no reason why the call to ReturnData doesn't use a SqlCommand under the hood.
Now, the real important difference that is worth considering is whether or not the repository holds a resource (memory, connection, pipe, file, etc) open for its lifetime, or just per data call.
The advantage of using a using is that resources are only opened for the duration of the call. This helps immensely with scaling of the app.
On the other hand there's an overhead to opening connections, so it's better - particularly for single threaded apps - to open a connection, do several tasks, and then close it.
So it really boils down to what type of app you're writing as to which approach you use.
Your second example isn't using entity framework. It seems you may have two different approaches to data access here although it is hard to tell from the repository snippet as it quite rightly hides the data access implementation. The second example is correctly using a "using" statement as you should on any object that implements IDisposable. It means you don't have to worry about calling dispose. This is using pure ADO.net which is what Entity Framework uses under the hood.
If the first example is using Entity framework you most likely have lazy loading in play in which case you need the DbContext to remain until the query has been executed. Entity Framework is an ORM tool. It too uses ADO.net under the hood to connect to the database but it also offers you alot more on top. A good book on both subjects should help you.
I found learning ADO.net first helps alot in understanding how Entity Framework retrieves info from the Database.
the using statement is good practice where ever you find an object that implements IDisposable. You can read more about that here : IDisposable the right way
In response to the change to the question - the answer still on the whole remains the same. In terms of performance - how fast are the queries returned? Does the performance of one work better than the other? Only your current system and set up can tell you that. Both approaches seem to be doing things the correct way.
I haven't worked with Migrations so not sure why you are getting ADO.net type queries integrating with your EF models but wouldn't be surprised by this functionality. Entity Framework as I have experienced it creates the queries for you and then executes them using the ADO.net objects from your second example. The key point is that you want to have the "using" block for SqlConnection and SqlCommand objects (although I don't think you need to nest them. everything inside the outer "using block will be disposed).
There is nothing stopping you putting a "using" block in your repository around the context but when it comes to lazily load the related Entities you will get an error as the context will have been disposed. If you need to make this change you can include the relevant elements in your query and do away with the lazy loading approach. There are performance gains in certain situations for doing this but again you need to balance this in terms to how your system is performing.
My question might be silly or may not be a question at all, but here it goes..
I am performing quite some database operations in my ASP.net MVC project where I am creating two objects each and every time namely, SqlConnection and SqlCommand.
I have shown an example below
using (SqlConnection connection = new SqlConnection(connectionString))
{
using (SqlCommand command = new SqlCommand("sRegisterUser", connection))
{
I am doing all these different operations inside one class under different methods.
My question is how can I reduce the creation of these objects each and every time? How can I create them globally
PS: sRegisterUser is a stored procedure, like wise other methods use different procedures which gets different values as parameters.
Please help me out.
Thanks for any help in advance.
The answer is, don't. You do not want to share these objects, you are using them appropriately.
You don't. You either keep the object alive, which is bad. Or you dispose it the way you should, and currently do.
So, why do you have use using?
Since using is necessary for disposing the handles in Windows.
You could also write this, which is similar to using using:
SqlConnection connection = new SqlConnection(connectionString);
connection.Close();
connection.Dispose();
using also disposes when the code above throws an error. Using using is actually the shortest way to code this.
So the error-save version should be:
SqlConnection connection;
try
{
connection = new SqlConnection(connectionString);
...
connection.Close();
}
finally
{
connection.Dispose();
}
As others already said, there is nothing wrong with what you're doing right now. ADO.NET objects are meant to be used and disposed of. As a rule of thumb you should create them as late as possible and dispose of them as soon as possible.
With that being said, I understand what you're trying to do here and I do have a few ideas you might be interested in for making your life easier while keeping good programming practices.
If you want to make the code more concise you can chain using statements:
using (var cn = new SqlConnection(yourConnectionString))
using (var cmd = new SqlCommand(yourQuery, cn))
{
// do stuff
}
The IDE will not try to indent the second block, making it easier on you to read.
You can also create an abstraction layer for your procedure:
public static class Procedures
{
public static void RegisterUser() // add whatever parameters you need
{
using (var cn = new SqlConnection(yourConnectionString))
using (var cmd = new SqlCommand(yourQuery, cn))
{
// do stuff
}
}
}
From then on you can execute your procedure simply by calling:
Procedures.RegisterUser();
Another alternative is to introduce a factory pattern to get your objects. This alone won't reduce their number but it might help in getting them all set up already (ie correct connection string).
You also can be creative and combine the two patterns. A custom class implementing IDisposable can take care of creating the necessary ADO objects, open the connection, execute your query, close the connection and dispose of any objects that needs disposal when it is itself disposed of.
Take your pick.
After successfully going through the initial stages of learning C# in tandem with SQL Server, I discovered that the various tutorials that I used simply got it wrong by declaring a global SqlConnection, SqlDataAdapter and even DataSet variables.
As a result, this code that works great in a single threaded application, doesn't work so great in a multi-threaded environment. In my research for a solution, I discovered that both MSDN and this educational answer recommend wrapping the "atomic" parts of a SQL transaction in a using/try method:
private static void CreateCommand(string queryString, string connectionString)
{
using (SqlConnection connection = new SqlConnection(connectionString))
{
try
{
SqlCommand command = new SqlCommand(queryString, connection);
command.Connection.Open();
command.ExecuteNonQuery();
}
catch (InvalidOperationException)
{
//log and/or rethrow or ignore
}
catch (SqlException)
{
//log and/or rethrow or ignore
}
catch (ArgumentException)
{
//log and/or rethrow or ignore
}
}
}
So, what I am going to do now is convert my entire code to using wrappers like this. But before proceeding ahead with this I would like to understand the tradeoffs of this approach. In my experience, there usually is a good reason for a large team of designers/engineers for deciding not to include certain defensive features. This is especially interesting when, from my point of view as a C/C++ programmer, the entire value proposition of C# is "defensiveness" (where the tradeoff is the well known CLR performance hit).
To summarize my question(s):
What are the tradeoffs of encapsulating every transaction in my code as described above?
Are there any caveats I should be looking for?
The reason's down to flexibility. Does the developer want to include the command in a transaction, do they want to retry on a given error, if so how many times, do they want a connection from a thread pool or to create a new connection each time (with a performance overhead), do they want a SQL connection or a more generic DbConnection, etc.
However, MS have provided the Enterprise Library, a suite of functionality which wraps up a lot of common approaches to things in an open source library. Take a look at the Data Access block:
http://msdn.microsoft.com/en-us/library/ff632023.aspx
There is no such method built in because:
Connecting and disconnecting the database for each command is not economical. If you execute more than one command at a given point in the code, you want to use the same connection for them instead of repeatedly opening and closing the connection.
The method can't know what you want to do about each kind of exception, so the only thing that it could do would be to rethrow them, and then there is no point in catching the exceptions in the first place.
So, almost everything that the method does would be specific for each situatuon.
Besides, the method would have to do more to be generally useful. It would have to take parameters for command type and parameters. Otherwise it can only be used for text queries, and would encourage people to create SQL queries dynamically instead of using stored procedures and/or parameterised queries, and that it not something that a general library would want to do.
1 - There are no real tradeoffs, it's pretty standard.
2 - Your code is ok to send commands as strings to be executed as SQL queries, but it lacks quit a bit of flexibility:
You can't use parameterized queries (command.Parameters.AddWithValue(...)) which will be mandatory once you start using stored procedures
You can't use output parameters like this
You can't do anything with whatever would be queried
I prefer to use something like this:
private static void CallProc(string storedProcName, Action<SqlCommand> fillParams, Action postAction, Action onError)
{
using (SqlConnection connection = new SqlConnection(connectionString))
{
using (SqlCommand command = new SqlCommand(String.Format("[dbo].[{0}]", storedProcName), connection))
{
try
{
if(fillParams != null)
fillParams(command);
command.Connection.Open();
command.ExecuteNonQuery();
if(postAction != null)
postAction();
}
catch (InvalidOperationException)
{
//log and/or rethrow or ignore
throw;
}
catch (SqlException)
{
//log and/or rethrow or ignore
throw;
}
catch (ArgumentException)
{
//log and/or rethrow or ignore
throw;
}
catch
{
if(onError != null)
onError();
}
}
}
}
You can then make variants to handle return values, output parameters, etc.
And you call is like:
CallProc("myStoredProc",
command =>
{
command.Parameters.AddWithValue("#paramNameOne", "its value here");
// More parameters for the stored proc...
},
null,
null);
As long as you encapsulate the functionality in a "bottleneck" method like the static method you've posted, so that all your database accesses are implemented in one easy-to-change piece of shared code, there often needs to be no trade-off, because you can change the implementation later without having to rewrite vast tracts of code.
By creating a new connection every time, the risk is that you might incur an expensive overhead for every open/close of the connection. However, the connections should be pooled, in which case the overheads may not be very large and this performance hit may be minimal.
The other approach would be to create a single connection and hold it open, sharing it for all your queries. This is undoubtedly more efficient because you're minimising the overheads per transaction. However, the performance gain may be minimal.
In both cases there will be additional threading (multiple simultaneous queries) issues to resolve unless you make sure that all database queries operate on a single thread. The performance implications all depend on how many queries you're firing off per second - and of course it doesn't matter how efficient your connection approach is if you are using grossly inefficient queries; you need to focus your "optimisation" time on the worst performance issues.
So I'd suggest keeping it simple for now and avoiding premature optimisation, but try to keep the implementation of the database access code in a separate layer, so that your main codebase simply issues commands to the access layer, and has minimal database-specific code in it. The less it "knows" about the database the better. This will make it much easier to change the underlying implementation or port your code to use a different database engine in future.
Another approach that can help with this is to encapsulate queries in stored procedures. This means your program knows the name of the procedure and the parameters for it, but the actual tables/columns that are accessed are hidden inside the database. Your code then knows as little as possible of the low-level structure of the database, which improves its flexibility, maintainability, and portability. Stored procedure calls can also be more efficient than sending generic SQL commands.
I have a c# server that is connecting to a mysql server for data. This c# server is a backend server for a game, that has an open thread for every player currently logged in. How do I go about doing the mysql connection, an open connection for each thread? A single connection using locking for all threads?
I read somewhere that there is a 'threadpool' for this. Is that true? If so, is this the proper way to use it:
using(var conn = new MySqlConnection(DatabaseHelper.getConnectionString()))
using (var cmd = conn.CreateCommand())
{
conn.Open();
cmd.CommandText = "SELECT username FROM characters WHERE id=1";
using (var reader = cmd.ExecuteReader())
{
while (reader.Read())
{
user = reader.GetString("username");
}
}
}
I think you are confusing two different resources here; you may indeed want an open thread (effectively a process) for each player logged on, but this doesn't mean you should have a connection to the database open for each player constantly, as things like database connections, file handles etc are not managed resources and should be freed up as soon as you are finished with them.
There's a good introduction and explanation of C# threading here.
You should only open a connection to the database when you need it. This means your data access classes could be instantiated many times (this is the simplest design) each with their own connection. If you use connection pooling, which I feel you actually might have been asking about, then you'll benefit further. Moving towards a static database connection design (where many people share the same dao classes) can be much more problematic as you may have to synchronize certain resources, ensure certain variables can only be accessed in sequence via locking or similar, and more.
You can read more about this here for example. Microsoft also talk about the benefits of connection pooling here.
The code you have above is a good starting point for connecting to the database, by including the using statement you are closing and disposing of the connection as soon as you are finished with it; you might also consider the following improvements:
using(var conn = new MySqlConnection(DatabaseHelper.getConnectionString()))
{
using (var cmd = conn.CreateCommand())
{
conn.Open();
cmd.CommandType = CommandType.Text;
//Parameterize your queries!
cmd.CommandText = "SELECT username FROM characters WHERE id=?"; //Use ? as placeholder for parameters, and add parameters in the order they appear in your query.
cmd.Parameters.Add(new MySqlParameter("#userid", MySqlDbType.Int, userid));
using(IDataReader reader = cmd.ExecuteReader())
{
//You should only expect one record. You might want to test for more than 1 record.
if (reader.Read())
{
user = reader.GetString(reader.GetOrdinal("username")); //Think also about null value checking.
}
}
}
}
You might have a DAO class, or a method on a user class to do this. For example, if it was a method on user, you might do something like:
User myUser = new User(7);
myUser.Load();
Inside of Load, one of the methods you might call is PopulateFromDB() which would contain the code above, and would load all of the properties of this user. You might also have a DAO class that does the same thing:
UserLoader loader = new UserLoader();
string userName = loader.GetUserName(int userId);
which would return the username using the code in the example above. I'd prefer this method to be on a class like User as it's logically connected to it. However, you then run the risk of mixing the DAO logic with the user business logic, which is a topic all on it's own.
Rather than write a lot of this data access logic, you might consider looking at some form of framework like an ORM or similar - this question has already been answered for MySql on SO. This could also save you a lot of time and effort, and will allow you to concentrate on design.
I am using the .xsd dataset thingies (which I hate) to auto-generate TableAdapter classes for some backend code.
I have not really used these before, tending to favour manual commands and stored procs whenever possible (for various speed-induced reasons: those xsds play hell with dynamic tables and really large amounts of columns), and am finding myself instantiating a TableAdapter in a large number of my methods, so my question is this:
Will the auto-generated code automatically streamline itself so that a full adapter class is not created on an instatiation, and instead share some static data (such as connection information), and if not would it be better for me to have some sort of singleton/static class provider that can give me access to their methods when needed without the overhead of creating a new adapter every time I want to get some information?
Cheers, Ed
If you're concerned about the performance you could always run a benchmark to see what the performance hit, if any, is.
Sorry you didn't find my answer useful.
My point was that while you had received responses they all seemed to be subjective and not based on hard data. So if you had some reason to be concerned that there was a performance hit in your particular application you should measure it.
There is no reason to refactor one area for performance unless there is an actual problem.
I actually tend to instanciate a very low number of adapters (usually only one of each type). I never tried using them as on the stack variables (instantiated when needed), so I never ran into your question, but I understand your concern.
From what I know the aqdapters themselves may be quite heavyweight in instancing, but the real killer is the connection. What I do is I mark the adapter's Connection modifier as Public in the .xsd designer so I can assign the property whatever I need it to use, and maintain a tight grip on the opening and closing of connections:
void Load() {
using (SqlConnection conn = ...) {
conn.Open();
invoicesAdapter.Connection = conn;
customersAdapter.Connection = conn;
invoicesAdapter.Fill(dataSet.Invoices);
customersAdapter.Fill(dataSet.Customers);
}
}
void Save() {
using (SqlConnection conn = ...) {
conn.Open();
invoicesAdapter.Connection = conn;
customersAdapter.Connection = conn;
invoicesAdapter.Update(dataSet);
customersAdapater.Update(dataSet);
}
}
I ommitted transaction control and error handling for brevity.