I've found myself increasingly unsatisfied with the DataSet/DataTable/DataRow paradigm in .Net, mostly because it's often a couple of steps more complicated than what I really want to do. In cases where I'm binding to controls, DataSets are fine. But in other cases, there seems to be a fair amount of mental overhead.
I've played a bit with SqlDataReader, and that seems to be good for simple jaunts through a select, but I feel like there may be some other models lurking in .Net that are useful to learn more about. I feel like all of the help I find on this just uses DataSet by default. Maybe that and DataReader really are the best options.
I'm not looking for a best/worst breakdown, just curious what my options are and what experiences you've had with them. Thanks!
-Eric Sipple
Since .NET 3.5 came out, I've exclusively used LINQ. It's really that good; I don't see any reason to use any of those old crutches any more.
As great as LINQ is, though, I think any ORM system would allow you to do away with that dreck.
We've moved away from datasets and built our own ORM objects loosely based on CSLA. You can get the same job done with either a DataSet or LINQ or ORM but re-using it is (we've found) a lot easier. 'Less code make more happy'.
I was fed up with DataSets in .Net 1.1, at least they optimised it so that it doesn't slow as exponentially for large sets any more.
It was always a rather bloated model - I haven't seen many apps that use most of its features.
SqlDataReader was good, but I used to wrap it in an IEnumerable<T> where the T was some typed representation of my data row.
Linq is a far better replacement in my opinion.
I've been using the Data Transfer Objects pattern (originally from the Java world, I believe), with a SqDataReader to populate collections of DTOs from the data layer for use in other layers of the application.
The DTOs themselves are very lightweight and simple classes composed of properties with gets/sets. They can be easily serialized/deserialized, and used for databinding, making them pretty well suited to most of my development needs.
I'm a huge fan of SubSonic. A well-written batch/CMD file can generate an entire object model for your database in minutes; you can compile it into its own DLL and use it as needed. Wonderful model, wonderful tool. The site makes it sound like an ASP.NET deal, but generally speaking it works wonderfully just about anywhere if you're not trying to use its UI framework (which I'm moderately disappointed in) or its application-level auto-generation tools.
For the record, here is a version of the command I use to work with it (so that you don't have to fight it too hard initially):
sonic.exe generate /server [servername] /db [dbname] /out [outputPathForCSfiles] /generatedNamespace [myNamespace] /useSPs true /removeUnderscores true
That does it every time ... Then build the DLL off that directory -- this is part of an NAnt project, fired off by CruiseControl.NET -- and away we go. I'm using that in WinForms, ASP.NET, even some command-line utils. This generates the fewest dependencies and the greatest "portability" (between related projects, EG).
Note
The above is now well over a year old. While I still hold great fondness in my heart for SubSonic, I have moved on to LINQ-to-SQL when I have the luxury of working in .NET 3.5. In .NET 2.0, I still use SubSonic. So my new official advice is platform version-dependent. In case of .NET 3+, go with the accepted answer. In case of .NET 2.0, go with SubSonic.
I have used typed and untyped DataSets, DataViewManagers, DataViews, DataTables, DataRows, DataRowViews, and just about anything you can do with the stack since it firsts came out in multiple enterprise projects. It took me awhile to get used to how allow of it worked. I have written custom components that leverage the stack as ADO.NETdid not quite give me what I really needed. One such component compares DataSets and then updates backend stores. I really know how all of these items work well and those that have seen what I have done are very impressed that I managed to get beyond there feel that it was only useful for demo use.
I use ADO.NET binding in Winforms and I also use the code in console apps. I most recently have teamed with another developer to create a custom ORM that we used against a crazy datamodel that we were given from contractors that looked nothing like our normal data stores.
I searched today for replacement to ADO.NET and I do not see anything that I should seriously try to learn to replace what I currently use.
DataSets are great for demos.
I wouldn't know what to do with one if you made me use it.
I use ObservableCollection
Then again i'm in the client app space, WPF and Silverlight. So passing a dataset or datatable through a service is ... gross.
DataReaders are fast, since they are a forward only stream of the result set.
I use them extensively but I don't make use of any of the "advanced" features that Microsoft was really pushing when the framework first came out. I'm basically just using them as Lists of Hashtables, which I find perfectly useful.
I have not seen good results when people have tried to make complex typed DataSets, or tried to actually set up the foreign key relationships between tables with DataSets.
Of course, I am one of the weird ones that actually prefers a DataRow to an entity object instance.
Pre linq I used DataReader to fill List of my own custom domain objects, but post linq I have been using L2S to fill L2S entities, or L2S to fill domain objects.
Once I get a bit more time to investigate I suspect that Entity Framework objects will be my new favourite solution!
Selecting a modern, stable, and actively supported ORM tool has to be probably the single biggest boost to productivity just about any project of moderate size and complexity can get. If you're concluding that you absolutely, absolutely, absolutely have to write your own DAL and ORM, you're probably doing it wrong (or you're using the world's most obscure database).
If you're doing raw datasets and rows and what not, spend the day to try an ORM and you'll be amazed at how much more productive you can be w/o all the drudgery of mapping columns to fields or all the time filling Sql command objects and all the other hoop jumping we all once went through.
I love me some Subsonic, though for smaller scale projects along with demos/prototypes, I find Linq to Sql pretty damn useful too. I hate EF with a passion though. :P
I've used typed DataSets for several projects. They model the database well, enforce constraints on the client side, and in general are a solid data access technology, especially with the changes in .NET 2.0 with TableAdapters.
Typed DataSets get a bad rap from people who like to use emotive words like "bloated" to describe them. I'll grant that I like using a good O/R mapper more than using DataSets; it just "feels" better to use objects and collections instead of typed DataTables, DataRows, etc. But what I've found is that if for whatever reason you can't or don't want to use an O/R mapper, typed DataSets are a good solid choice that are easy enough to use and will get you 90% of the benefits of an O/R mapper.
EDIT:
Some here suggest that DataReaders are the "fast" alternative. But if you use Reflector to look at the internals of a DataAdapter (which DataTables are filled by), you'll see that it uses...a DataReader. Typed DataSets may have a larger memory footprint than other options, but I've yet to see the application where this makes a tangible difference.
Use the best tool for the job. Don't make your decision on the basis of emotive words like "gross" or "bloated" which have no factual basis.
I just build my business objects from scratch, and almost never use the DataTable and especially not the DataSet anymore, except to initially populate the business objects. The advantages to building your own are testability, type safety and IntelliSense, extensibility (try adding to a DataSet) and readability (unless you enjoy reading things like Convert.ToDecimal(dt.Rows[i]["blah"].ToString())).
If I were smarter I'd also use an ORM and 3rd party DI framework, but just haven't yet felt the need for those. I'm doing lots of smaller size projects or additions to larger projects.
I NEVER use datasets. They are big heavyweight objects only usable (as someone pointed out here) for "demoware". There are lot's of great alternatives shown here.
Related
I am trying to follow TDD principles in all my code base. The frontend (MVC) and backend part are split, and frontend use their own Model objects, while backed use database objects which are then saved to a document database (RavenDb).
This requires a lot of conversion from say CustomerModel to CustomerData. These are created independently from each other, so the strucutre might not match. For example, the CustomerModel might be flat while CustomerData has a nested object ContactDetails.
Currently, we are implementing two methods, one say ConvertCustomerModelToCustomerData and ConvertCustomerDataToCustomerModel. These are very similar, but inverse of each other. Apart from this, these methods are also unit-tested. Hence, similar code is created in four instances - twice for each conversion, and twice for each unit test.
This is a big headache to maintain, and does not seem right to me. I've tried to use AutoMapper, however I found it quite rigid. Also, I could not find any way how I can unit-test this.
Any ideas would be greatly appreciated.
I think that having well defined boundaries and anti-corruption layers (see this and this) like the one you did is a great way to avoid headache, and bug hunting an highly coupled application is far worse.
Then, these layers are for sure boring to maintain, but I bet that dealing with is is simple, a no-brainer activity.
If you find yourself modifying your entities often (and so having many tests to update), maybe they are not well defined yet, or they have too wide scope. Why do you need to update them? What's the trigger?
Then, AutoMapper can help you a lot, I agree with other comments.
Anyway, without seeing the code it's difficult for me to judge and maybe offer any kind of advice, so feel free to consider this just my humble opinion :)
I tried to google but didn't find a decent tutorial with snippet code.
Does anyone used typed DataSets\DataTable in c# ?
Is it from .net 3.5 and above?
To answer the second parts of the question (not the "how to..." from the title, but the "does anyone..." and "is it...") - the answer would be a yes, but a yes with a pained expression on my face. For new code, I would strongly recommend looking at a class-based model; pick your poison between the many ORMs, micro-ORMs, and raw ADO.NET. DataTable itself does still have a use, in particular for processing and storing unpredictable data (where you have no idea what the schema is in advance). By the time you are talking about typed data-sets, I would suggest you obviously know enough about the type that this no longer applies, and an object-model is a very valid alternative.
It is still a supported part of the framework, and it is still in use as a technology. It has some nice features like the diff-set. However, most (if not all) of that is also available against an object-based design, with classes and properties (without the added overhead of the DataTable abstraction).
MSDN has guidance. It really hasn't changed since typed datasets were first introduced.
http://msdn.microsoft.com/en-us/library/esbykkzb(v=VS.100).aspx
There are tons of videos available here: http://www.learnvisualstudio.net/series/aspdotnet_2_0_data_access_and_databinding/
And I found one more tutorial here: http://www.15seconds.com/issue/031223.htm
Sparingly.... Unless you need to know to maintain legacy software, learn an ORM or two, particularly in conjunction with LINQ.
Some of my colleagues have them, the software I work on doesn't use them at all, on account of some big mouth developer getting his way again...
So, I have this new project. My company uses the SalesForce.com cloud to store information about day-to-day operations. My job is to write a new application that will, among other things, more seamlessly integrate the CRUD operations of this data with existing in-house application functionality.
The heart of the Salesforce WSDL API is a set of "query()" web methods that take the query command as a string. The syntax of the query is SQL-ish, but not quite (they call it SOQL). I'm not a fan of "magic strings", so I'd like to use Linq in the codebase, and parse the IQueryable into the SOQL query I need in the wrapper for the service. It's certainly possible (L2E, L2Sql), but I'd like to know if there's a shortcut, 'cause if I say it'll take more than a day or two to roll my own, I'll be "encouraged" to find another method (most likely a method for each general-use query, which was the method used in older apps). If I succeed in making a general-purpose SOQL parser, we can use it in several other upcoming apps, and I'll be a hero. Even if I make a simple one that only supports certain query structures, it'll go a long way by allowing me to proceed with the current project in a Linq-y way, and I can expand on it in my free time.
Here are the options as I see it:
Look harder for an existing Linq2SOQL provider (My Google-fu is failing me here, or else there simply isn't one; the only .NET wrapper only mentions Linq as a nice-to-have).
Build an expression tree parser. It needs to support, at least, the Select and Where method calls, and needs to either parse lambdas or manipulate their method bodies to get the operations and projections needed. This seems to be a rather massive task, but like I said, it's certainly possible.
Wrap the service in Linq2Sql or a similar existing Linq provider that will allow me to extract a close-enough query string, polish it up and pass it to the service. There must be dozens out there (though none that just drop in, AFAIK).
Call Expression.ToString() (or Expression.DebugView), and manipulate that string to create the SOQL query. It'll be brittle, it'll be ugly (behind the scenes), and it will support only what I'm explicitly looking for, but it will provide a rudimentary translation that will allow me to move on.
What do you guys think? Is building a Linq parser more than a two-day task for one guy? Would a bodged-up solution involving an existing Linq provider possibly do it? Would it be terrible to chop up the expression string and construct my query that way?
EDIT: Thanks to Kirk for the grounding. I took some more looks at what I'd be required to do for even a basic SOQL parser, and it's just beyond the scope of me getting working application code written on any feasible schedule. For instance, I have to build a select list from either the Select() method lambda or a default one from all known columns from my WSDL object, a task I hadn't even thought of (I was focusing more on the Where parsing). I'm sure there are many other "unknown unknowns" that could turn this into a pretty big deal. I found several links which shows the basics of writing a Linq provider, and though they all try to make it simple, it's just not going to be feasible time-wise right now. I'll structure my repository, for now, using named methods that encapsulate named queries (a constant class of formattable query strings should reduce the amount of head-scratching in maintenance). Not perfect, but far more feasible. If and when a Linq2SOQL provider gets off the ground, either in-house or open-source, we can refactor.
For others looking for Linq provider references, here are those helpful links I found:
Building a Linq Provider
Walkthrough: Creating an IQueryable LINQ Provider
Linq: Building an IQueryable provider series - in 17 parts! <-- this one, though long and involved, has a lot of real in-depth explanations and is good for "first-timers".
Let's take them one at a time:
Look harder for an existing Linq2SOQL provider (My Google-fu is failing me here, or else there simply isn't one; the only .NET wrapper only mentions Linq as a nice-to-have).
Yeah, I doubt one exists already, but hopefully you can find one.
Build an expression tree parser. It needs to support, at least, the Select and Where method calls, and needs to either parse lambdas or manipulate their method bodies to get the operations and projections needed. This seems to be a rather massive task, but like I said, it's certainly possible.
This is absolutely the way to go if you really are serious about this in the long-run.
Wrap the service in Linq2Sql or a similar existing Linq provider that will allow me to extract a close-enough query string, polish it up and pass it to the service. There must be dozens out there (though none that just drop in, AFAIK).
What do you mean by "drop in"? You can easily get the SQL straight from L2S.
Call Expression.ToString() (or Expression.DebugView), and manipulate that string to create the SOQL query. It'll be brittle, it'll be ugly (behind the scenes), and it will support only what I'm explicitly looking for, but it will provide a rudimentary translation that will allow me to move on.
I would strongly discourage you from this approach, as, at minimum, it will be at least as difficult as parsing the expression trees properly. If anything, in order to use this, you'd have to first put the parsed strings into a proper object model -- i.e. the existing expression trees you're starting with.
Really, you should think about building a query provider and doing this right. I think two days is a bit of a stretch though to get something working in even a primitive sense, though it might be possible. IMO, you should research it some at home and toy around with it so you have some familiarity with the basic pieces and parts. Then you might barely be able to get some usable queries going after two days.
Honestly though, fully implementing this kind of project is really in the realm of several weeks, if not months -- not days.
If that's too much work, you might consider option 3. I'm no expert on SOQL, so no idea what kind of work would be involved transforming ordinary SQL queries into SOQL queries. If you think it's rather algorithmic and reliable, that might be the way to go.
ActiveRecord is too limiting normally. However, i'm in a difficult situation in terms of the views held by each member of the team in regards to using ORM's. We currently use a very basic ActiveRecord with regret I say is written mostly by hand with some basic generation of code. I would like to build a new DAL for us but avoiding the limitations of ActiveRecord, so DDD more so. The points however I am battling against are (both old skool developers and myself quite young):
Team Lead Developer
Is in favour of stored procedures, but isn't consistant...some just get
a table e.g. SELECT * FROM Company and some get SELECT C.*, O.OtherTableValue FROM Company C...(very frustrating)
Doesn't really know the benefits of some of the latest ORM tools
Won't commit to any tool as its "too restrictive" and if there are
issues what do you do?
DBA
Doesn't like dynamic SQL
Doesn't like SELECT *
I'm not saying the above are off limits, its more convicing them otherwise. I belive we could massively improve our efficent with the use of an ORM but its very difficult to convince them otherwise. If I could give proof to some of these areas I might be able to convince them, even by implementing under the covers without them knowing initally and then seeing the benefits.
What advice can you give to help my situation? I believe many developers come across this and cannot choose which architecture they would like to use.
I think your Team Lead needs to either commit to consistency or spend some time on an ORM research project to make a recommendation to use. In other words, an inconsistent and set-in-ways Team Lead shouldn't be in that role.
Secondly, I tend to agree with your DBA for a number of reasons. As long as he/she is flexible enough to know that there are occasions where dynamic SQL will solve the problem much better.
For your specific situation I'd say ask your DBA to lay down the law that stored procs are to be used every time unless justification is provided on a case-by-case basis and enforce this through policy and monitoring. This will address the consistency issue. Perhaps then, with that policy in hand, an ORM looks more enticing than having to hand-code everything to the Team Lead.
The lead developer should listen to the DBA. "SELECT *" is bad. And from the bullet points about the lead developer it sounds like you've got a familiar uphill battle. Sometimes the path of least resistance in that situation is to implement something using an ORM (such as NHibernate) and schedule some kind of demonstration to the team.
Encourage their input, specially from the lead developer and the DBA and any other nay-sayers. They might have legitimate gripes that can actually be solved with the tool as you learn more about it. On the other hand, they might just be dead-set against it for no good logical reason. If that's the case, it's best to find that out because it could very well mean that there is no winning this argument against them, because they're not really debating.
What are their objections to using an ORM? If they are not just being obstinate and stubborn then if you know what specifically they object to you and address those concerns. Like the others I think your DBA is correct. But if he is concerned about SQL injection from dynamic sql, ORM's alleviate that issue somewhat. Select * should be grounds for a firing.
I'm of the mind where you should just use something, LINQ-to-SQL or subsonic or nhibernate on something small. Show that development can be faster and cleaner using an ORM.
I know that nowadays there is a lot of ready-to-go stuff that you can use, but if you want full control over db requests/queries, and best performance I think that this is the way to go. Also because ADO.NET does the connection pooling automatically for SqlConnection, how do you think ?
Absolutely. If extreme performance is what you want, you can't get much faster than this. I think it's still OK.
In fact, I'm doing a reporting system right now which uses SQL, DataReaders, and SQLConnections because all it does it run SQL. OLAP cubes aren't exactly the best candidates for OR mappers.
If you really need full control over the generated SQL queries and are looking for best performance this is the way to go. But sometimes the best performance comes at the cost of less readable and less maintainable code. If this is the case you might need to make a decision whether this price is worth paying.
If you need best performance it is ok. It gives you more control and better performance. By the way here is an article about things you should take into consideration that affects performance while using ADO.NET: How Data Access Code Affects Database Performance
I think it's perfectly fine. Only you can determine your best toolset for any given project, and as long as you're using them properly, following best practices, use what you think is best.
Personally, I prefer to use the tools you mentioned, because I like the finer control. I only use the drag-and-drop components for very simple tasks, and hey, there's now Dynamic Data for those tasks.
Yes you're right, it has the best performance, but you have to consider all the points before take the decision, there are ORM tools that can control all of your data acces layer, and they are just fine in performance.
Yes, I think it's perfectly OK! I personally like having the full control, to define exactly how db interation is made and giving you absolute control over performance tuning/optimisation.
Just wanted to add that of course SqlDataAdapters and DataTables may have their place too, especially if wanting to pass disconnected data around.
You may also want to look into Enterprise Library. It uses the basic ADO.Net objects while minimizing the code required to make these calls in your code.