I'm working on creating a dashboard. I have started to refactor the application so methods relating to querying the database are generic or dynamic.
I'm fairly new to the concept of generics and still an amateur programmer but I have done some searching and I have tried to come up with a solution. The problem is not really building the query string in a dynamic way. I'm fine with concatenating string literals and variables, I don't really need anything more complex. The bigger issue for me is when I create this query, getting back the data and assigning it to the correct variables in a dynamic way.
Lets say I have a table of defects, another for test cases and another for test runs. I want to create a method that looks something like:
public void QueryDatabase<T>(ref List<T> Entitylist, List<string> Columns, string query) where T: Defect, new()
Now this is not perfect but you get the idea. Not everything about defects, test cases and test runs are the same but, I'm looking for a way to dynamically assign the retrieved columns to its "correct" variable.
If more information is needed I can provide it.
You're re-inventing the wheel. Use an ORM, like Entity Framework or NHibernate. You will find it's much more flexible, and tools like that will continue to evolve over time and add new features and improve performance while you can focus on more important things.
EDIT:
Although I think overall it's important to learn to use tools for something like this (I'm personally a fan of Entity Framework and have used it successfully on several projects now, and used LINQ to SQL before that), it can still be valuable as a learning exercise to understand how to do this. The ORMs I have experience with use XML to define the data model, and use code generation on the XML file to create the class model. LINQ to SQL uses custom attributes on the code-generated classes to define the source table and columns for each class and property, and reflection at run-time to map the data returned from a SqlDataReader to the properties on your class object. Entity Framework can behave differently depending on the version you're using, whether you use the "default" or "POCO" templates, but ultimately does basically the same thing (using reflection to map the database results to properties on your class), it just may or not use custom attributes to determine the mapping. I assume NHibernate does it the same way as well.
You are reinventing the wheel, yes, it's true. You are best advised to use an object-relational mapper off the "shelf". But I think you also deserve an answer to your question: to assign the query results dynamically to the correct properties, you would use reflection. See the documentation for the System.Reflection namespace if you want more information.
Related
Is there a way to automap a query into a class in C# in order to strong type the query results? I don't want to access to query result with a DataTable, specifying manually the Column Name anymore.
There is couple of options:
EntityFramework - powerful and heavyweight. https://msdn.microsoft.com/en-us/library/aa937723(v=vs.113).aspx
Dapper - lightweight - mostly mapper https://www.codeproject.com/Articles/212274/A-Look-at-Dapper-NET
If you need only mapping of stored procedures, queries etc, Dapper is good way to go, as it's basically adding extension methods to connection object. It cannot create database etc as Entity Framework but it all really depends on needs.
My favourite question. Use QueryFirst. Write your SQL in the SQL editor window, syntax validated as you type, then QueryFirst will generate the C# wrapper to execute it, with strongly typed inputs and outputs. And there are numerous other benefits :-) SqlServer, MySql and Postgres support built in. Others easy to add. Disclaimer : I wrote QueryFirst.
You should take a look at AutoMapper. It's an easy and user-friendly way to map a DataTable to a List<object> of your specification.
You can find an article which talks about it (including sample code) here.
If you want to automize the process, you can use dynamic objects which don't require you to create specific classes for every DataTable that you're using.Hope this helped!
There is a helpful library on LINQ to DB
"LINQ to DB is the fastest LINQ database access library offering a simple, light, fast, and type-safe layer between your POCO objects and your database."
https://github.com/linq2db/linq2db
I have a table called 'OrderHeader' in a SQL Server 2008 R2 database.
While I have found tools to generate the POCO class for this table in an automated manner, I cannot find a tool that will also generate the code to populate this class using SqlDataReader. This can help me save a lot of my time since I am writing low level ADO.Net code in my current project that uses SqlDataRader to populate data classes.
My question is: Is there any such tool ?
There are several free libraries that do this sort of mapping from SQL query results to POCO as a part of their broader functionality. AutoMapper for instance can convert your SqlDataReader to an appropriate IList<T> without too much difficulty, as can NHibernate apparently.
Another option is to use a framework like LinqToSQL or the EntityFramework to encapsulate your data, then build your queries against those. There are any number of good reasons to use EF or L2S for this, including such (probably) useful things as lazy loading and such.
Or you could learn to use reflection and build your mappings at runtime. I've used Linq Expression objects to do this in the past, but it's a lot of work to get only a little gain. It's fun to learn though :P
I'd suggest reading this article for more on reflection. It even has a simple example of using reflection to map an IDataReader to any type. It's not a complete solution since it doesn't handle DBNulls or any difference between your POCO and the data returned from the server will throw some exceptions. Make sure you read the whole article, then go investigate anything that still isn't clear.
If you want to go deeper than that, you can use Linq Expressions to build mapping functions for your types, then compile those expressions to Func<IDataReader, T> objects. Much easier than using System.Reflection.Emit... although still not simple.
I want to write a function, which should take SQL query as argument and give data as result.
So, I want that this function will give universal structure as result.
May be the best way to return DataTable or SqlDataReader ?
And after that to extract data from them and put it to the object with special type (different classes of models)?
May be exist some pattern for this?
You can return the values directly to objects. But you will still need to map them eventually to something.
Normally for this sort of 'generic' use, I see everything being transferred as strings. Kind of how JSON gets serialized to some kind of string everywhere until it's ultimately used somewhere as something else.
Whether you use DataTable or something else, it really doesn't matter as it will ultimately end up being completely dependent on how you turn that data into something useful.
you can take a look at the http://en.wikipedia.org/wiki/NoSQL page if you want to see some more info about that method.
Martin Fowler gave a presentation on schemaless and NoSql earlier this year that you may find interesting.
http://www.youtube.com/watch?v=8kotnF6hfd8
I would echo some of the other answers on here though and most likely advocate not doing something 'generic' or 'universal'. Just put in some work up front to make the system conform to what you definitely want to do and change it as needed.
It sounds like you're just starting out. You may want to consider an ORM like Entity Framework Code First or NHibernate, or a MicroORM of which there are too many to list, rather than just running bare SQL against a db.
I find EF Code First to be great for beginners.
http://www.asp.net/mvc/tutorials/getting-started-with-ef-using-mvc/creating-an-entity-framework-data-model-for-an-asp-net-mvc-application
ORMs let you define classes that map to the tables in your database, and ask for lists of those objects explicitly (solving the problem of what to return).
You could use c# dynamic as return type. I would recommend Dapper as ORM, it is fast and easy to use, and it has the functionality to return dynamics from an query, see the section:'Execute a query and map it to a list of dynamic objects'.
In our application we're considering using dynamically generated classes to hold a lot of our data. The reason for doing this is that we have customers with tables that have different structures. So you could have a customer table called "DOG" (just making this up) that contains the columns "DOGID", "DOGNAME", "DOGTYPE", etc. Customer #2 could have the same table "DOG" with the columns "DOGID", "DOG_FIRST_NAME", "DOG_LAST_NAME", "DOG_BREED", and so on. We can't create classes for these at compile time as the customer can change the table schema at any time.
At the moment I have code that can generate a "DOG" class at run-time using reflection. What I'm trying to figure out is how to populate this class from a DataTable (or some other .NET mechanism) without extreme performance penalties. We have one table that contains ~20 columns and ~50k rows. Doing a foreach over all of the rows and columns to create the collection take about 1 minute, which is a little too long.
Am I trying to come up with a solution that's too complex or am I on the right track? Has anyone else experienced a problem like this? Create dynamic classes was the solution that a developer at Microsoft proposed. If we can just populate this collection and use it efficiently I think it could work.
What you're trying to do wil get fairly complicted in .NET 3.5. You're probably better off creating a type with a Dictionary<> of key/value pairs for the data in your model.
If you can use .NET 4.0, you could look at using dynamic and ExpandoObject. The dynamic keyword lets you create reference to truly dynamic objects - ExpandoObject allows you to easily add properties and methods on the fly. All without complicated reflection or codegen logic. The benefit of dynamic types is that the DLR performs some sophisticated caching of the runtime binding information to allow the types to be more performant than regular reflection allows.
If all you need to do is load map the data to existing types, then you should consider using something like EntityFramework or NHibernate to provide ORM behavior for your types.
To deal with the data loading performance, I would suggest you consider bypassing the DataTable altogether and loading your records directly into the types you are generating. I suspect that most of the performance issues are in reading and casting the values from the DataTable where they original reside.
You should use a profiler to figure out, what exactly takes the time, and then optimize on that. If you are using reflection when setting the data, it will be slow. Much can be saved by caching the reflected type data.
I am curious, how will you use these class if the members are not known at compile time ? Perhaps you would be better off with just a plain DataTable for this scenario ?
Check out PicoPoco and Massive. These are Micro ORMS. (light weight opensource ORM frameworks) that you use by simply copying in a single code file into your project.
Massive uses the ExpandoObject and does conversions from IDataRecord to the ExpandoObject wich I think is exactly what you want.
PicoPoco take an existing plain class and dynamically and efficiently generates a method (which it then caches by the way) to load it from a database.
I'm playing around at the start of a personal project in C# and MySQL.
I am familiar with the use of the Gentle Framework (using MyGeneration to generate the classes, based on the data model). Here's what I like about Gentle;
Simple-to-use [class].Retrieve(id) / [object].Persist() semantics with strong-typing of fields;
I start with the DB data model, and choose when to generate new code files;
MyGeneration allows for some 'manual code sections' which are kept across generations...
...and partial classes allow me to add permanent code in parallel files, e.g. simple read-only properties (like 'FullName' from FirstName and Surname members for a Person object) - or I could use inheritance;
I find it a tolerable and quick way to create a DAL, and add certain Business-Object-Layer-like facilities to it.
Unfortunately, to query efficiently, I end up using queries / SqlCommands a fair bit, and relies on weakly typed references to column names etc., and appears to risk sidestepping the object broker and therefore caching advantages. In any event, Gentle is no longer being developed, and it seems like a good time to consider alternatives.
So, what should I consider?
Generation of strongly-typed ADO Datasets is possible, but it seems like it will be difficult to add to it (e.g. that 'FullName' virtual column) in a way that will persist after updates to the table structure with regeneration of the dataset.
NHibernate seems to have lots of fans... but my first looks into it seem to suggest that the XML data definition is king, not the existing data-model in the DB. It also looks quite heavy on dependencies;
The SubSonic demo appears to suggest it generates files, and in the demo of WebAppProjects, looks like it might generate files in a way that I could add to, or inherit from;
The MySql Connector.Net tools appear not to support the dataset generation for Linq (e.g. via drag-and-drop), and I suspect that this is a key need for strongly-typed data access.
Your thoughts will be gratefully appreciated! Thank you in advance...
I had some experience with Gentle and I do have to admit that it was pretty inefficient with queries. I would suggest looking into NHibernate, since it has a rich community. It is true that XML definitions are preferred, but there are ways of doing the mappings using class-level attributes.
SubSonic (especially the 3.0 version) looks very promising with its use of T4 templates. That should give you more control over code generation. It can do LINQ too.
Don't invest in LINQ-to-SQL, since the rumors are that is going to be discontinued.
Assuming that the .Net 3.5 Framework is an option for being used, then you can take a look at Microsoft's Entity Framework (released with .Net 3.5 Service Pack 1).
The Entity Framework allows the generation of DAL classes based on your database schema, but the maintenance of these classes are hidden behind an XML file that can quickly and easily be updated to account for schema changes by a simple command from the Visual Studio IDE.
I am working on a project where we use the Entity Framework with MySQL with few problems.
The main disadvantage to this option is that the official .Net connector provided by MySQL does not yet support the Entity Framework - there is a paid alternative known as MyDirect.Net
link textI would go for Subsonic, mature DAL generator and improves productivity by great margin.
We have used it with both MySQL and SQL Server - no headaches. Generates classes for Tables, Stored procedures, column names. So every time we find ourselves doing Somthing Dot Intellisense Move Arrow keys and semicolon.
Any time your schema changes, you can regenerate those classes and you are home. Also, you can extend them by creating partial classes.
It supports almost all of the SQL Semantics - Joins, loading Collection by primary key, adding WHERE clause, Order by, Count, Top, Calling stored procedures, views and so on and Intuitive syntax is big plus.
To give you some glimpse- For Books table[BookID-PK, title, AuthorID], It generates several types of methods.
Insert method which takes Title, AuthorID
Nullable columns are optional
parameters a.k.a C# Nullable type ?
Update method wich takes BookID, AuthorID, Title
Load Book by Primary key (Useful when displaying detail page)
BookCollection and Book Entities, Just call BookCollection.Load and you have list of books ready to bind to any databound control
Here's quick link.
Thanks,
Maulik Modi
Thanks to both Filip and Snorkpete for your suggestions - your comments and links proved helpful.
I will probably try SubSonic first; it looks like something I will understand and be able to get going with quickly (today should answer that), and I was surprised to see that it is indirectly supported by MS as they employ the guy who writes it. T4 also looks very interesting.
The Entity Relationship Model also looks interesting, and the link to MyDirect may be helpful in the future. The only down side here is one of expectation; MS have screwed-up their approach in the past by making them easy to create the initial design with drag-and-drop, then much harder later to modify or keep up-to-date.
Anyway, thank you both again, and I'll try to keep this question updated.
Nij
I use a bit of SQL to generate strongly typed objects out of tables, it's based on one built by Cade Bryant, but I've made some tweaks. The code it generates is not 100% compilable but it saves a lot of boiler plate work and the gaps are easy to fill (i would make all the properties fully fledged properties if i were you, or bear the wrath of jon skeet!)
http://NotifyURL.com/sql