Based on links around the StackOverflow site (references below), I've come up with this block of code to perform queries from my C# application to a MySQL database.
using (var dbConn = new MySqlConnection(config.DatabaseConnection))
{
using (var cmd = dbConn.CreateCommand())
{
dbConn.Open();
cmd.CommandType = CommandType.Text;
cmd.CommandText = "SELECT version() as Version";
using (IDataReader reader = cmd.ExecuteReader())
{
if (reader.Read())
{
Console.WriteLine("Database Version: " + reader.GetString(reader.GetOrdinal("Version")));
}
}
}
}
The problem I have with this, is that I have to build up this massive block of code every time I have a group of queries to make because I don't (and shouldn't) leave the connection open for the life of the application.
Is there a more efficient way to build the supporting structure (the nested usings, opening the connection, etc), and instead pass my connection string and the query I want to run and get the results back?
Referenced questions:
Use of connections with C# and MySql - Specifically the answer
by tsells
Mysql select where and C#
Update a mysql table using
C#
That is three of the ones I looked at. There were a few more, but my Google-fu can't refind them right now. All of these provide answers for how to perform a single query. I want to perform separate business logic queries - a few of them repeatedly - and don't want to repeat unneeded code.
What I've tried:
Based on the comment from nawfal, I have these two methods:
private MySqlDataReader RunSqlQuery(string query)
{
Dictionary<string, string> queryParms = new Dictionary<string, string>();
MySqlDataReader QueryResult = RunSqlQuery(query, queryParms);
return QueryResult;
}
private MySqlDataReader RunSqlQuery(string query, Dictionary<string, string> queryParms)
{
MySqlDataReader reader = null;
if (queryParms.Count > 0)
{
// Assign parameters
}
try
{
using (var dbConn = new MySqlConnection(config.DatabaseConnection))
{
using (var cmd = dbConn.CreateCommand())
{
dbConn.Open();
cmd.CommandType = CommandType.Text;
cmd.CommandText = query;
using (reader = cmd.ExecuteReader())
{
return reader;
}
}
}
}
catch (MySqlException ex)
{
// Oops.
}
return reader;
}
The problem with this attempt is that the reader closes when it is returned from the method.
Have you considered using an Object Relational Mapper (ORM)? I'm fond of Castle Active Record and NHibernate myself, but there's plenty of others. Entity Framework and Linq to SQL are popular Microsoft solutions too.
With these tools, your queries become pretty simple CRUD method calls that do the connection and session handling for you (mostly).
Instead of creating the reader in a using statement inside your RunSqlQuery method you could return it directly:
return cmd.ExecuteReader();
Then wrap the call to RunSqlQuery in a using statement:
using( var reader = RunSqlQuery(....) )
{
// Do stuff with reader.
}
You could use Actions or Funcs to get what I think you are after.
invoked like this...
RunSqlQuery("SELECT * FROM ...", reader => ReadResult(reader));
private bool ReadResult(MySqlDataReader reader)
{
//Use the reader to read the result
if (!success)
return false;
return true;
}
implemented like this...
private bool RunSqlQuery(string query, Func<MySqlDataReader, bool> readerAction)
{
Dictionary<string, string> queryParms = new Dictionary<string, string>();
return RunSqlQuery(query, readerAction, queryParms);
}
private bool RunSqlQuery(string query, Func<MySqlDataReader, bool> readerAction, Dictionary<string, string> queryParms)
{
MySqlDataReader reader = null;
if (queryParms.Count > 0)
{
// Assign parameters
}
try
{
using (var dbConn = new MySqlConnection(config.DatabaseConnection))
{
using (var cmd = dbConn.CreateCommand())
{
dbConn.Open();
cmd.CommandType = CommandType.Text;
cmd.CommandText = query;
using (reader = cmd.ExecuteReader())
{
return readerAction.Invoke(reader);
}
}
}
}
catch (MySqlException ex)
{
// Oops.
return false;
}
}
Why do you want to return the datareader from the method? It will be closed once u wrap it in inside the using block. Also you can assign parameters only after getting an instance of IDbCommand, so I have moved that part to inside of the using block.
If you strictly want to return the datareader, then better return IEnumerable<IDataRecord> using the yield keyword.
private IEnumerable<IDataRecord> RunSqlQuery(string query,
Dictionary<string, string> queryParms)
{
using (var dbConn = new MySqlConnection(config.DatabaseConnection))
{
using (var cmd = dbConn.CreateCommand())
{
if (queryParms.Count > 0)
{
// Assign parameters
}
cmd.CommandText = query;
cmd.Connection.Open();
using (var reader = cmd.ExecuteReader())
foreach (IDataRecord record in reader as IEnumerable)
yield return record;
}
}
}
Or even better is to read the data there itself and return the data back, as in this question. That way you dont have to rely on classes in db namespaces outside your db class.
I have been down that road. Along the lines of suggesting ORMs, I would recommend EF Code First. Sorry to be a bit off topic, but I have never had a second thought about going back to this pattern after using EF Code First.
Before Code First, EF was quite a pain, but now it has matured and if you had a DB you are potentially modifying structure, i.e. a new app feature requires a new table or column, then EF Code First approach is my recommendation. If it is a third party database or database for another app, that someone else manages its structure, then you only need to refresh your data model whenever they deploy changes, then I would not use Code First, and instead just use traditional EF where you generate/update your model based on some existing database.
Note you could adopt EF and begin using it while you keep your existing code base as-is. This depends on how much of your framework is dependent on using ADO objects though. EF Power Tools extension has a way to generate a Code First model, or you could just use the traditional non-Code First EF to generate a modal from database.
When you want to query, you can get right to the business of what you are trying to query without having alot of infrastructure code or wrappers. The other thing about wrappers like the above, is there are edge cases that you will have to go back to using the ADO API instead of your RunSqlQuery helper.
This is a trivial example, as usually I don't have methods like GetActivePeopleNames, but just put the query where ever it is needed. There is little overhead in terms of fluff code, so it isn't obtrusive to have my query among everything else. Although I do exercise some presenter patterns to abstract the query and data transformation from the business logic.
HREntities db = new HREntities();
private ICollection<string> GetActivePeopleNames()
{
return db.People.Where(p => p.IsActive).Select(p => p.FirstName + " " + p.LastName)
.ToList();
}
I didn't have to create a parameter object. I could have used some variable for Where(p => p.IsActive == someBool) and it would have been safe from SQL injection in that context. The connection is handled automatically. I can use .Include to grab related objects in the same connection if needed.
Related
I'm using a MySQL local database, connecting to the database is not a problem (anymore). I have a small-scale database with around 6 different tables, each with around 4-6 columns, and rows <100 (not working with large data).
I am creating a WPF application that only ever needs to SELECT data from these databases, it never needs to add to them. The database is filled with static data which I will need to run SELECT statements on it and then use the results to display in my WPF app.
I need to make a function in my DBHandler class which can then be called from any other class in my system, to query the database with a specified SELECT statement, and then use the results. The problem is that my queries will vary - sometimes I might be calling for one column, such as;
(SELECT id FROM students WHERE name = 'Conor')
Sometimes I might be calling for multiple rows in a more complex statement.. such as this (pseudo):
(SELECT name, address FROM destinations WHERE long, lat intersects_with (SELECT long, lat FROM trains))
Whenever I call this function with a query, I will always be expecting the format of the data response, so if I just return a List<> or array, it should be no problem accessing the data even though the function is generic and not specific for one query or table.
So far I have tried this:
public static MySqlDataReader Query(string SQLQuery)
{
using (MySqlConnection con = new MySqlConnection(connectionString))
{
con.Open();
MySqlCommand command = new MySqlCommand(SQLQuery, con);
MySqlDataReader reader = command.ExecuteReader();
return reader;
}
}
// Some other class
MySqlDataReader reader = DBHandler.Query("SELECT * FROM destinations");
while (reader.Read())
{
MessageBox.Show(reader[0].ToString());
}
This doesn't work, because it complains the reader is closed. I presume I can't simply return a MySqlDataReader object.
My next thought process would be to do the actual query and return all the data in this Query function, and store all the results which can then be returned. But how I return the data is my main issue, because it needs to be generic for variable SELECT queries, so it can't have a fixed size for number of rows or columns returned. I thought maybe I could store it in a List<>, or a List<> within a List<>, but I'm really not sure on how to lay it out.
I know this is asking a lot but it is boggling my mind - I don't know how to make this generic SELECT function, but I know it will be really helpful as I will just need to call this whenever I need to get data in another part of the system.
Thank you!
You cannot try to use a DataReader when its connection has been closed. So, when your code exits the using block, the connection is closed as well the reader. However, you can pass to your Query method an Action delegate that receives a MySqlDataReader. This function will be defined by the caller of Query so you can customize it for your different tables while keeping a generic approach to the boilerplate code used to open, query and read the database.
public static MySqlDataReader Query(string SQLQuery, Action<MySqlDataReader> loader)
{
using (MySqlConnection con = new MySqlConnection(connectionString))
{
con.Open();
using(MySqlCommand command = new MySqlCommand(SQLQuery, con))
using(MySqlDataReader reader = command.ExecuteReader())
{
// here you can pass the reader, you are still inside the using block
while(reader.Read())
loader.Invoke(reader)
}
}
}
In the caller code you could write
List<Destination> destinations = new List<Destination>();
MySqlDataReader reader = DBHandler.Query("SELECT * FROM destinations", dataLoaderForDestination);
Console.WriteLine("Loaded " + destinations.Count + " destinations");
private void dataLoaderForDestination(MySqlDataReader reader)
{
Destination dest = new Destination();
dest.Address = reader.GetString(0);
dest.Nation = reader.GetInt32(1);
...
destinations.Add(dest);
}
Of course in a different point of your code you could pass the reference to a different Action delegate tailored for a different set of data returned by your query
List<Student> students = new List<Student>();
private void dataLoaderForStudents(MySqlDataReader reader)
{
Student st = new Student();
st.Name = reader.GetString(0);
st.Class = reader.GetInt32(1);
students.Add(st);
}
a reader is online, you need to loop inside (using connection), because if you leave the using, the connction is disposed and closed
I’m calling a stored procedure via ADO.NET as shown (in simplified form) below. The database is a MySQL database.
The stored procedure returns a list of ID values that correspond to calls that need to have their call times rescheduled.
The C# code stores them in a list.
My question is: Is there a more efficient way to get the values into the C# list, instead of using the DataReader as I’m doing?
I don’t know yet if the way I’m doing it is too inefficient for our application (that will be determined during testing), I’m just looking for a faster strategy, if one exists.
I looked at using a DataSet but, from what I’ve read, that could be slower if the list of ID-s is large (which it could be).
Also, from what I’ve read, LINQ might be slower, as well.
I only need to store the list into callsToRescheduleList; ie, I don’t need to do any random access of the ID-s, so those features of the DataSet are not needed.
I’m just looking for the fastest way to get the data into the list.
Any suggestions?
The C# code:
private void GetCallsToRescheduleList()
{
MySqlCommand cmd = new MySqlCommand
(
"`phytel`.`spPhy_GetCallsToRescheduleListPreviousDays`",
(MySqlConnection) DatabaseConnection, workerTransaction
);
cmd.CommandType = CommandType.StoredProcedure;
MySqlDataReader reader = cmd.ExecuteReader();
if (reader.HasRows)
{
while( reader.Read())
{
callsToRescheduleList.Add(reader.GetInt32(0));
}
}
}
The MySQL stored procedure:
CREATE PROCEDURE `spPhy_GetCallsToRescheduleListPreviousDays` ()
BEGIN
SELECT
id
FROM callrequest
WHERE
dialerCampaignId = 'CATH001’
AND
status = 'SCHEDULED’
;
END
AFAIK that's about as good as it gets - the select is narrow, and use of the reader ordinal overload in the tight loop is good.
Some checks:
Ensure that there is an index (key) on (dialerCampaignId, status) in MySql - I'm assuming there is better selectivity on dialerCampaignId here.
If there is a chance that your List<Int> consumer won't iterate the full list every time, as an alternative to rolling the data up into a list, consider also using an enumerable and yield return:
public IEnumerable<int> GetCallsToRescheduleListPreviousDays()
{
using (var cmd = new MySqlCommand
(
"`phytel`.`spPhy_GetCallsToRescheduleListPreviousDays`",
(MySqlConnection) DatabaseConnection, workerTransaction
)
{
cmd.CommandType = CommandType.StoredProcedure;
using (var reader = cmd.ExecuteReader())
{
if (reader.HasRows)
{
while( reader.Read())
{
yield return reader.GetInt32(0);
}
}
}
}
}
Also remember to Dispose of IDisposable resources like Commands and Readers
I am trying to retrieve list of records from one table , and write to another table. I've used a simple query to retrieve the values to SqlDataReader,then load them to a DataTable. Using the DataTableReader , I am going through the entire data set which is Saved in DataTable. The problem is, while reading each and every record I am trying to insert those values to another table using a Stored Procedure.But it only insert the first row of values,and for the second row onward giving some Exception saying."procedure or function has too many arguments specified".
string ConStr = ConfigurationManager.ConnectionStrings["ConString"].ConnectionString;
SqlConnection NewCon = new SqlConnection(ConStr);
NewCon.Open();
SqlCommand NewCmd3 = NewCon.CreateCommand();
NewCmd3.CommandType = CommandType.Text;
NewCmd3.CommandText ="select * from dbo.Request_List where group_no ='" +group_no+ "'";
NewCon.Close();
NewCon.Open();
SqlDataReader dr = (SqlDataReader)NewCmd3.ExecuteReader();
DataTable dt = new DataTable();
dt.Load(dr);
DataTableReader reader = new DataTableReader(dt);
NewCmd.Dispose();
NewCon.Close();
NewCon.Open();
SqlCommand NewCmdGrpReqSer = NewCon.CreateCommand();
NewCmdGrpReqSer.CommandType = CommandType.StoredProcedure;
NewCmdGrpReqSer.CommandText = "Voucher_Request_Connection";
if (reader.HasRows)
{
int request_no = 0;
while (reader.Read())
{
request_no = (int)reader["request_no"];
NewCmdGrpReqSer.Parameters.Add("#serial_no", serial_no);
NewCmdGrpReqSer.Parameters.Add("#request_no", request_no);
try
{
NewCmdGrpReqSer.ExecuteNonQuery();
MessageBox.Show("Connection Updated");//just to check the status.tempory
}
catch (Exception xcep)
{
MessageBox.Show(xcep.Message);
}
MessageBox.Show(request_no.ToString());//
}
NewCmdGrpReqSer.Dispose();
NewCon.Close();
}
Any Solutions ?
As #Sparky suggests, the problem is that you continue to add parameters to the insertion command. There are several other ways in which the code could be improved, however. These improvements would remove the need to clear the parameters and would help to make sure you don't leave disposable resources undisposed.
First - use the using statement for your disposable objects. This removes the need for the explicit Close (btw, only one of Close/Dispose is needed for the connection as I believe Dispose calls Close). Second, simply create a new command for each insertion. This will prevent complex logic around resetting the parameters and, possibly, handling error states for the command. Third, check the results of the insertion to make sure it succeeds. Fourth, explicitly catch a SqlException - you don't want to accidentally hide unexpected errors in your code. If it's necessary to make sure all exceptions don't bubble up, consider using multiple exception handlers and "doing the right thing" for each case - say logging with different error levels or categories, aborting the entire operation rather than just this insert, etc. Lastly, I would use better variable names. In particular, avoid appending numeric identifiers to generic variable names. This makes the code harder to understand, both for others and for yourself after you've let the code sit for awhile.
Here's my version. Note there are several other things that I might do such as make the string literals into appropriately named constants. Introduce a strongly-typed wrapper around the ConfigurationManager object to make testing easier. Remove the underscores from the variable names and use camelCase instead. Though those are more stylistic in nature, you might want to consider them as well.
var connectionString = ConfigurationManager.ConnectionStrings["ConString"].ConnectionString;
using (var newConnection = new SqlConnection(connectionString))
{
newConnection.Open();
using (var selectCommand = newConnection.CreateCommand())
{
selectCommand.CommandType = CommandType.Text;
select.CommandText ="select request_no from dbo.Request_List where group_no = #groupNumber";
selectCommand.Parameters.AddWithValue("groupNumber", group_no);
using (dataReader = (SqlDataReader)newCommand.ExecuteReader())
{
while (reader.HasRows && reader.Read())
{
using (var insertCommand = newConnection.CreateCommand())
{
insertCommand.CommandType = CommandType.StoredProcedure;
insertCommand.CommandText = "Voucher_Request_Connection";
var request_no = (int)reader["request_no"];
insertCommand.Parameters.Add("#serial_no", serial_no);
insertCommand.Parameters.Add("#request_no", request_no);
try
{
if (insertCommand.ExecuteNonQuery() == 1)
{
MessageBox.Show("Connection Updated");//just to check the status.tempory
}
else
{
MessageBox.Show("Connection was not updated " + request_no);
}
}
catch (SqlException xcep)
{
MessageBox.Show(xcep.Message);
}
MessageBox.Show(request_no.ToString());//
}
}
}
}
}
Try clearing your parameters each time...
while (reader.Read())
{
request_no = (int)reader["request_no"];
// Add this line
NewCmdGrpReqSer.Parameters.Clear();
NewCmdGrpReqSer.Parameters.Add("#serial_no", serial_no);
NewCmdGrpReqSer.Parameters.Add("#request_no", request_no);
try
{
I'm trying to find optimal (fast vs easiest) way to access SQL Server code thru code in C#.
As I was learning from books I've encountered multiple suggestions usually telling me to do it via drag and drop. However since I wanted to do it in code first approach was to get data by column numbers, but any reordering in SQL Query (like adding/removing columns) was pain for me to fix.
For example (don't laugh, some code is like 2 years old), I even coded special function to pass sqlQueryResult and check if it's null or not):
public static void exampleByColumnNumber(string varValue) {
string preparedCommand = #"SELECT TOP 1 [SomeColumn],[SomeColumn2]
FROM [Database].[dbo].[Table]
WHERE [SomeOtherColumn] = #varValue";
SqlCommand sqlQuery = new SqlCommand(preparedCommand, Locale.sqlDataConnection);
sqlQuery.Prepare();
sqlQuery.Parameters.AddWithValue("#varValue) ", varValue);
SqlDataReader sqlQueryResult = sqlQuery.ExecuteReader();
if (sqlQueryResult != null) {
while (sqlQueryResult.Read()) {
string var1 = Locale.checkForNullReturnString(sqlQueryResult, 0);
string var2 = Locale.checkForNullReturnString(sqlQueryResult, 1);
}
sqlQueryResult.Close();
}
}
Later on I found out it's possible thru column names (which seems easier to read with multiple columns and a lot of changing order etc):
public static void exampleByColumnNames(string varValue) {
string preparedCommand = #"SELECT TOP 1 [SomeColumn],[SomeColumn2]
FROM [Database].[dbo].[Table]
WHERE [SomeOtherColumn] = #varValue";
SqlCommand sqlQuery = new SqlCommand(preparedCommand, Locale.sqlDataConnection);
sqlQuery.Prepare();
sqlQuery.Parameters.AddWithValue("#varValue) ", varValue);
SqlDataReader sqlQueryResult = sqlQuery.ExecuteReader();
if (sqlQueryResult != null) {
while (sqlQueryResult.Read()) {
string var1 = (string) sqlQueryResult["SomeColumn"];
string var2 = (string) sqlQueryResult["SomeColumn2"];
}
sqlQueryResult.Close();
}
}
And 3rd example is by doing it by column names but using .ToString() to make sure it's not null value, or by doing If/else on the null check.
public static void exampleByColumnNamesAgain(string varValue) {
string preparedCommand = #"SELECT TOP 1 [SomeColumn],[SomeColumn2], [SomeColumn3]
FROM [Database].[dbo].[Table]
WHERE [SomeOtherColumn] = #varValue";
SqlCommand sqlQuery = new SqlCommand(preparedCommand, Locale.sqlDataConnection);
sqlQuery.Prepare();
sqlQuery.Parameters.AddWithValue("#varValue) ", varValue);
SqlDataReader sqlQueryResult = sqlQuery.ExecuteReader();
if (sqlQueryResult != null) {
while (sqlQueryResult.Read()) {
string var1 = (string) sqlQueryResult["SomeColumn"].ToString();
DateTime var2;
DateTime.TryParse(sqlQueryResult["SomeColumn2"].ToString());
int varInt = ((int) sqlQueryResult["SomeColumn3"] == null ? 0 : (int) sqlQueryResult["SomeColumn3"];
}
sqlQueryResult.Close();
}
}
Please bare in mind that I've just created this for sake of this example and there might be some typos or some slight syntax error, but the main question is which approach is best, which is the worst (I know first one is the one that I dislike the most).
I will soon have to start / rewriting some portion of my little 90k lines app which has at least those 3 examples used widely, so i would like to get best method for speed and preferably easiest to maintain (hopefully it will be same approach).
Probably there are some better options out there so please share?
It seems you may be looking at old books. If you're going to do it the "old fashioned way", then you should at least use using blocks. Summary:
using (var connection = new SqlConnection(connectionString))
{
using (var command = new SqlCommand(commandString, connection))
{
using (var reader = command.ExecuteReader())
{
// Use the reader
}
}
}
Better still, look into Entity Framework.
Links: Data Developer Center
If it's easy you're looking for, you can't do any better than Linq-to-SQL:-
http://weblogs.asp.net/scottgu/archive/2007/05/19/using-linq-to-sql-part-1.aspx
If your SQL database already exists, you can be up-and-running in seconds.
Otherwise, I agree with John.
you should have a look into these tutorials,
[http://www.asp.net/learn/data-access/][1]
All the work you are planning is already been done.
have a look at this way of doing same what you are doinng
string preparedCommand =
#"SELECT TOP 1 [SomeColumn],[SomeColumn2], [SomeColumn3]
FROM [Database].[dbo].[Table]
WHERE [SomeOtherColumn] = #varValue";
[1]: http://www.asp.net/learn/data-access/
More better way of doing the same above is by Using LINQ TO SQL
var result = from someObject in SomeTable
where SomeColumnHasValue == ValueToCompare
select new { SomeColumn, SomeColumn1, SomeColumn2};
No Type Safety Issues
Visualise Database in C# while you
work on it
at compile time less errors
less code
more productive
Following are some of the great resources for LINQ if you are interested
http://msdn.microsoft.com/en-us/vcsharp/aa336746.aspx
http://www.hookedonlinq.com/MainPage.ashx
https://stackoverflow.com/questions/47740/what-are-some-good-linq-resouces
Hope it helps
If you're looking into using just straight ADO.net you might want to go out and find Microsoft's Enterprise Library's Data Access Application Block . David Hayden has a decent article that goes into some detail about using it.
Good luck and hope this helps some.
The easiest way to do data access in C#, to my mind, is using typed DataSets. A lot of it really is drag-and-drop, and it's even easier in .NET 2.0+ than in .NET 1.0/1.1.
Have a look at this article, which talks about using typed DataSets and TableAdapters:
Building a DAL using Strongly Typed TableAdapters and DataTables in VS 2005 and ASP.NET 2.0
A typed DataSet is basically a container for your data. You use a TableAdapter to fill it (which happens with SQL or stored procs, whichever you prefer) and to update the data afterwards. The column names in each DataTables in your DataSet are autogenerated from the SQL used to fill them; and relations between database tables are mirrored by relations between DataTables in the DataSet.
Don't convert data to strings only to try to parse it; DataReaders have methods to convert SQL data to .Net data types:
using (var connection = new SqlConnection(Locale.sqlDataConnection))
using (var command = new SqlCommand(preparedCommand, connection))
using (var reader = command.ExecuteReader())
{
int stringColumnOrdinal = reader.GetOrdinal("SomeColumn");
int dateColumnOrdinal = reader.GetOrdinal("SomeColumn2");
int nullableIntColumnOrdinal = reader.GetOrdinal("SomeColumn3");
while (reader.Read())
{
string var1 = reader.GetString(stringColumnOrdinal);
DateTime var2 = reader.GetDateTime(dateColumnOrdinal);
int? var3 = reader.IsDBNull(nullableIntColumnOrdinal) ? null : (int?)reader.GetInt32(nullableIntColumnOrdinal);
}
}
I test the many different ways for get data from sql server database and i faced & found fastest way is following:
First of all create class with "IDataRecord" parameterized method as per your required properties.
class emp
{
public int empid { get; set; }
public string name { get; set; }
public static emp create(IDataRecord record)
{
return new emp
{
empid = Convert.ToInt32(record["Pk_HotelId"]),
name = record["HotelName"].ToString()
};
}
}
Now create method for get data as below:
public List<S> GetData<S>(string query, Func<IDataRecord, S> selector)
{
using (var cmd = conn.CreateCommand())
{
cmd.CommandText = query;
cmd.Connection.Open();
using (var r = cmd.ExecuteReader())
{
var items = new List<S>();
while (r.Read())
items.Add(selector(r));
return items;
}
}
}
And then call function like:
var data = GetData<emp>("select * from employeeMaster", emp.create);
We have a lot of data layer code that follows this very general pattern:
public DataTable GetSomeData(string filter)
{
string sql = "SELECT * FROM [SomeTable] WHERE SomeColumn= #Filter";
DataTable result = new DataTable();
using (SqlConnection cn = new SqlConnection(GetConnectionString()))
using (SqlCommand cmd = new SqlCommand(sql, cn))
{
cmd.Parameters.Add("#Filter", SqlDbType.NVarChar, 255).Value = filter;
result.Load(cmd.ExecuteReader());
}
return result;
}
I think we can do a little better. My main complaint right now is that it forces all the records to be loaded into memory, even for large sets. I'd like to be able to take advantage of a DataReader's ability to only keep one record in ram at a time, but if I return the DataReader directly the connection is cut off when leaving the using block.
How can I improve this to allow returning one row at a time?
Once again, the act of composing my thoughts for the question reveals the answer. Specifically, the last sentence where I wrote "one row at a time". I realized I don't really care that it's a datareader, as long as I can enumerate it row by row. That lead me to this:
public IEnumerable<IDataRecord> GetSomeData(string filter)
{
string sql = "SELECT * FROM [SomeTable] WHERE SomeColumn= #Filter";
using (SqlConnection cn = new SqlConnection(GetConnectionString()))
using (SqlCommand cmd = new SqlCommand(sql, cn))
{
cmd.Parameters.Add("#Filter", SqlDbType.NVarChar, 255).Value = filter;
cn.Open();
using (IDataReader rdr = cmd.ExecuteReader())
{
while (rdr.Read())
{
yield return (IDataRecord)rdr;
}
}
}
}
This will work even better once we move to 3.5 and can start using other linq operators on the results, and I like it because it sets us up to start thinking in terms of a "pipeline" between each layer for queries that return a lot of results.
The down-side is that it will be awkward for readers holding more than one result set, but that is exceedingly rare.
Update
Since I first started playing with this pattern in 2009, I have learned that it's best if I also make it a generic IEnumerable<T> return type and add a Func<IDataRecord, T> parameter to convert the DataReader state to business objects in the loop. Otherwise, there can be issues with the lazy iteration, such that you see the last object in the query every time.
In times like these I find that lambdas can be of great use. Consider this, instead of the data layer giving us the data, let us give the data layer our data processing method:
public void GetSomeData(string filter, Action<IDataReader> processor)
{
...
using (IDataReader reader = cmd.ExecuteReader())
{
processor(reader);
}
}
Then the business layer would call it:
GetSomeData("my filter", (IDataReader reader) =>
{
while (reader.Read())
{
...
}
});
The key is yield keyword.
Similar to Joel's original answer, little more fleshed out:
public IEnumerable<S> Get<S>(string query, Action<IDbCommand> parameterizer,
Func<IDataRecord, S> selector)
{
using (var conn = new T()) //your connection object
{
using (var cmd = conn.CreateCommand())
{
if (parameterizer != null)
parameterizer(cmd);
cmd.CommandText = query;
cmd.Connection.ConnectionString = _connectionString;
cmd.Connection.Open();
using (var r = cmd.ExecuteReader())
while (r.Read())
yield return selector(r);
}
}
}
And I have this extension method:
public static void Parameterize(this IDbCommand command, string name, object value)
{
var parameter = command.CreateParameter();
parameter.ParameterName = name;
parameter.Value = value;
command.Parameters.Add(parameter);
}
So I call:
foreach(var user in Get(query, cmd => cmd.Parameterize("saved", 1), userSelector))
{
}
This is fully generic, fits any model that comply to ado.net interfaces. The connection and reader objects are disposed after the collection is enumerated. Anyway filling a DataTable using IDataAdapter's Fill method can be faster than DataTable.Load
I was never a big fan of having the data layer return a generic data object, since that pretty much dissolves the whole point of having the code seperated into its own layer (how can you switch out data layers if the interface isn't defined?).
I think your best bet is for all functions like this to return a list of custom objects you create yourself, and in your data later, you call your procedure/query into a datareader, and iterate through that creating the list.
This will make it easier to deal with in general (despite the initial time to create the custom classes), makes it easier to handle your connection (since you won't be returning any objects associated with it), and should be quicker. The only downside is everything will be loaded into memory like you mentioned, but I wouldn't think this would be a cause of concern (if it was, I would think the query would need to be adjusted).