I'm trying to reduce the startup time for tests against an EF 6x datastore. The tests are within a transaction and the db gets rolled back once done. I would appreciate any suggestions on how to retain an instance of the DbContext between test sessions so that EF doesn't have to go through the whole view generation process again?
I don't want to use mocks/fakes, non-Microsoft branch of EF and interactive views are already in place. Thank you.
Different options. As you did not mentioned your aim of testing and there is not any code, the options are:
If you are inserting many records into your tables, you can do a bulk insert. The best library for doing this is:EntityFramework.BulkInsert-ef6. You can install it through Nuget console.
If you see slowness while working with data and you have many load/manipulation/save operations, you have to do in-memory operation as Sampath recommends.
If you are loading data, just load the columns that you need. You also should use lazy loading option(which from your post, I think you know it well).
4.Some portion of the slowness could be because of the architecture of your database. The key column types have a considerable effect on Where operations!
I would like to recommend you to use in-memory data for that. I am also used this pattern and it is really well and very fast. This is the pattern where the industry recommended and trouble free in long run. Always try to use best practices when you develop a software app.
When writing tests for your application it is often desirable to avoid
hitting the database. Entity Framework allows you to achieve this by
creating a context – with behavior defined by your tests – that makes
use of in-memory data.
Here is the article about how to do that :Testing with a mocking framework
Another article for you : Unit testing in C# using xUnit, Entity Framework, Effort and ASP.NET Boilerplate
Related
I am currently playing with EntityFramework.BulkInsert.
While it really helps with performance of simple inserts (16 seconds with 1.000.000 rows) I can't find any information about inserting objects mapped over multiple tables. The only thing related to that is old (2014) topic from official website stating that it's not possible. Is this still actual?
If so: are there any good workarounds?
EntityFramework.BulkInsert is a very good library which supports simple scenario. However, the library is limited and not anymore supported.
So far, there are only one good workaround and it's using a library which supports everything!
Disclaimer: I'm the owner of the project Entity Framework Extensions
This library supports everything including all associations and inheritance.
By example, for saving multiple entities in different tables, you can use BulkSaveChanges which work exactly like SaveChanges but way faster!
// Easy to use
context.BulkSaveChanges();
// Easy to customize
context.BulkSaveChanges(bulk => bulk.BatchSize = 100);
The library also do more than only inserting. It support all bulk operations:
BulkInsert
BulkUpdate
BulkDelete
BulkMerge
However unlike EntityFramework.BulkInsert, this library is not free.
EDIT: Answer subquestion
You say way faster - do you have any metrics or a link to metrics
#Mark: You can look on metrics on our website homepage. We report BulkSaveChanges to be at least 15x faster than SaveChanges.
However metric are heavily biased. Too many things may affect it like index, trigger, latency etc!
People usually report us performance improved by 25x, 50x, 80x!
One thing people usually forget when performing benchmarking is calling our library once before the test for JIT compilation! Like Entity Framework, the first hit to the library may take several ms.
If I had a bulk insert problem, I'd not use EF. EF is meant to map objects representing entities during the normal use-cases of your application, where any given transaction should only really touch one entity (assuming your entities are designed around sensible consistency boundaries).
If I was moving lots of data around (imports/exports/transformations etc) then I would use SQL more directly, where I have more control.
I am working on a .NET web api service(with Odata support) to support Mobile client.The service should support both Oracle and SQL server databases, but only one database type will be used at a time, according to which ever database technology client is using.
How to create database agnostic data access layer? Dont want to write code twice - once for SQL server and once for Oracle.
Also it seems like in order to support oracle in EF, 3rd party oracle drivers are required - either from devart or oracle's ODP.NET.
I am debating should I use old style ADO.NET or use EF for building data access layer.
I will appreciate any help on this.
Thanks!
Your question seems to revolve around multiple concerns, i'll give answers based on my views on them:
1.- ¿How can you create a Database (DB Engine) agnostic DAL?
A: One approach for this is to follow the Repository pattern and/or use interfaces to decouple the code that manipulates the data from the code that retrieves/inserts it. The actual implementation of the interfaces used by your code to get the data can also be taylored to be DB Engine agnostic, if you're going to use ADO.NET, you can check out the Enterprise Library for some very useful code which is DB Engine agnostic. Entity Framework is also compatible with different DB engines but, as you mentioned, can only interact with one DB at a time, so whenever you generate the model, you tie it to the specifics of the DB Engine that your DB is hosted in. This is related to another concern in your question:
2.- ¿Should you use plain old ADO.NET or EF?
A: This is a very good question, which i'm sure has been asked before many times and given that both approaches give you the same practical results of being able to retrieve and manipulate data, the resulting question is: ¿what is your personal preference for coding and the time/resources constraints of the project?
IMO, Entity Framework is best suited for Code-First projects and when your business logic doesn't require complex logging, transactions and other security or performance constraints on the DB side, not because EF is not capable of including these requirements, but because it becomes rather convoluted and unpractical to do it and i personally believe that defeats the purpose of EF, to provide you with a tool that allows for rapid development.
So, if the people involved in the project is not very comfortable writing stored procedures in SQL and the data manipulation will revolve mostly around your service without the need for very complex operations on the DB side, then EF is a suitable approach, and you can leverage the Repository pattern as well as interfaces to implement "DBContext" objects that will allow you to create a DB Agnostic DAL.
However, if you are required to implement transactions, security, extensive logging, and are more comfortable writing SQL stored procedures, Entity Framework will often prove to be a burden for you simply because it is not yet suited for advanced tasks, for example:
Imagine you have a User table, with multiple fields (Address, phone, etc) that are not always necessary for all user-related operations (such as authentication); Trying to map an entity to the results of a stored procedure that does not return any of the fields that the entity contains will result in an error, and you will either need to create different models with more or less members or return additional columns in the SP that you might not need for a particular operation, increasing the bandwith consumption unnecessarily.
Another situation is taking advantage of features such as Table Valued Parameters in SQL Server to optimize sending multiple records at once to the DB, in this case Entity Framework does not include anything that will automatically optimize operations with multiple records, so in order to use TVPs you will need to manually define that operation, much like you would if you had gone the ADO.NET route.
Eventually, you will have to weigh the considerations of your project against what your alternatives provide you; ADO.NET gives you the best performance and customization for your DB operations, it is highly scalable and allows optimizations but it takes more time to code, while EF is very straightforward and practical for objects manipulation, and though it is constantly evolving and improving, its performance and capabilities are not quite on pair with ADO.NET yet.
And regarding the drivers issue, it shouldn't weigh too much in the matter since even Oracle encourages you to use their driver instead of the default one provided by Microsoft.
I have been stuck in 2.0 for some time now due to what my job requires and I am trying to now keep up with the latest .net framework. I am begining to build a Mvc application and I am wondering if the Data Access Application Blocks (SqlHelper) is still an acceptable method of writing a data tier.
I know there is a lot of debate over the EF, with it's speed performance, lazy loading and the issue of keeping the database separate from the application. So I hesitate to go there right now.
But I am seeing a lot of code that just uses ADO.Net and the DataAdapter and wondering if there was a shift back to ADO.net without a helping framework. If this is correct then my question would be why? It seems like a step backwards to not use some type of helper class/framework to do data access.
The use of ORMs is seen as best practice these days, and the Data Access Application Block has fallen out of favor as other, easier to work with libraries have come to existence and prominence in the .NET world.
These would include nHibernate, LINQ to SQL, Entity Framework to mention the big, fully featured libraries (some more fully featured than others).
People who do use ADO.Net directly are those who want more control over their SQL than afforded by these (always possible with ORMs, but they tend to be rather cumbersome when it comes to accessing the database not through their idioms).
An example lightweight ORM is Dapper, which is a very thin layer over ADO.Net.
So I have an application which requires very fast access to large volumes of data and we're at the stage where we're undergoing a large re-design of the database, which gives a good opertunity to re-write the data access layer if nessersary!
Currently in our data access layer we use manually created entities along with plain SQL to fill them. This is pretty fast, but this technology is really getting old, and I'm concerned we're missing out on a newer framework or data access method which could be better in terms of neatness and maintainability.
We've seen the Entity Framework, but after some research it just seems that the benefit of the ORM it gives is not enough to justify the lower performance and as some of our queries are getting complex I'm sure performance with the EF would become more of an issue.
So it is a case of sticking with our current methods of data access, or is there something a bit neater than manually creating and maintaining entities?
I guess the thing that's bugging me is just opening our data layer solution and seeing lots of entities, all of which need to be maintained exactly in line with the database, which sometimes can be a lot of work, but then maybe this is the price we pay for performance?
Any ideas, comments and suggestions are very appreciated! :)
Thanks,
Andy.
** Update **
Forgot to mention that we really need to be able to handle using Azure (client requirements), which currently stops us from using stored procedures. ** Update 2 ** Actually we have an interface layer for our DAL which means we can created an Azure implementation which just override data access methods from the Local implementation which aren't suitable for Azure, so I guess we could just use stored procedures for performance sensitive local databases with EF for the cloud.
I would use an ORM layer (Entity Framework, NHibernate etc) for management of individual entities. For example, I would use the ORM / entities layers to allow users to make edits to entities. This is because thinking of your data as entities is conceptually simpler and the ORMs make it pretty easy to code this stuff without ever having to program any SQL.
For the bulk reporting side of things, I would definitely not use an ORM layer. I would probably create a separate class library specifically for standard reports, which creates SQL statements itself or calls sprocs. ORMs are not really for bulk reporting and you'll never get the same flexibility of querying through the ORM as through hand-coded SQL.
Stored procedures for performance. ORMs for ease of development
Do you feel up to troubleshooting some opaque generated SQL when it runs badly...? That generates several round trips where one would do? Or insists on using wrong datatypes?
You could try using mybatis (previously known as ibatis). It allows you to map sql statements to domain objects. This way you keep full control over SQL being executed and get cleanly defined domain model at the same time.
Don't rule out plain old ADO.NET. It may not be as hip as EF4, but it just works.
With ADO.NET you know what your SQL queries are going to look like because you get 100% control over them. ADO.NET forces developers to think about SQL instead of falling back on the ORM to do the magic.
If performance is high on your list, I'd be reluctant to take a dependency on any ORM especially EF which is new on the scene and highly complex. ORM's speed up development (a little) but are going to make your SQL query performance hard to predict, and in most cases slower than hand rolled SQL/Stored Procs.
You can also unit test SQL/Stored Procs independently of the application and therefore isolate performance issues as either DB/query related or application related.
I guess you are using ADO.NET in your DAL already, so I'd suggest investing the time and effort in refactoring it rather than throwing it out.
I would like your opinions regarding "DataSet Designer" and DAL (Data Access Layer) best practices.
I use Visual Studio 2010 Framework .NEt 4.0.
For my understanding "DataSet Designer" allow me to create automatically strictly Typed-DataSet with DataTable and Adapter, this consist in DAL directly in Visual Studio 2010.
I would like to know:
- If in real scenario "DataSet Designer" is working well, or is better write Custom Business Object.
- If exist other new solution introduced in .net 4.0
Thanks for your support! :-)
I have to work with typed datasets and it is a nightmare. If you have an option never use them. Everything is better.
With the advent of the .Net 4.0 framework and the introduction of LINQ to SQL, I've been adopting a customized DAL of strictly written business objects. We experimented with Entity Framework briefly, but ultimately concluded that it is very similar to DataSets in that the auto-generated code, while handy, is just too bloated with extra junk that we ultimately didn't use.
We've found that writing LINQ into our DAL and extracting data pulls into our custom classes, we are able to streamline our data access and control the usage of the data functionally. It has been a very handy process, but it has taken a little bit for the junior developers to grip onto it.
I would suggest a ORM like Entity Framework or Nhibernate.
Data Sets smells too much to database way of thinking and I personally had a lot of problems working with them. They just get broken quite often and throw weird errors that are hard to troubleshoot.
Some other related questions you may find interesting
What are the advantages of using an ORM?
ASP.NET DataSet vs Business Objects / ORM
Use ADO.NET Entity Framework, which is where the future of Microsoft's ORM is going. Or, consider an open-source one like NHibernate...
HTH.
At my company we've been using Typed DataSets for a little while now, and have had a generally positive experience. I understand that many people don't like DataSets, and there are certainly newer data access tools out there, but since you asked about a real-world scenario, here are some of my requirements and findings:
Need to be able to read SQL Server, MS Access, and FoxPro data sources
SQL Server access is only through SPROC calls (not my choice)
Relatively easy to learn, especially to developers new to ASP.NET
I've personally explored low level ado.net access, typed datasets, linq-to-sql, and simply writing custom data access classes. I have not looked at the Entity Framework yet, as the version included in VS2008 seemed to have some mixed reviews, and I did not have access to VS2010 until just recently(I do plan to review EF sometime this year yet).
We chose to use Typed DataSets because they seemed to offer faster development against SPROCS and we found a very comprehensive tutorial by Scott Mitchell on the asp.net site: http://www.asp.net/data-access/tutorials.
As to our experience thus far, it has mostly been good. The DataSet designer generates a huge amount of code even for small number of Tables (<20). Making changes in the SPROCS has caused a few headaches, but I'd like to be shown a tool that would make this easier.
One thing you might try to make your decision easier: Come up with a small domain problem like a customer edit page or order entry page, and implement it multiple times using a variety of technologies. It takes some time to do this, but it is a good way to learn and you can compare the technologies for yourself. We did this and it seemed to help a lot.
I will personally prefer custom business objects with their flexibility but its more work. Also look at with Entity Framework and Linq To Sql. Entity Fx has got a lot more flexibility in .NET 4.0. This article should get you started on Entity Fx.
If anything I think you should look into Entity Framework. There are lots of great tutorials out there to get you started.
I personally agree with Joel Etherton, conditionally.
If you have a small enough project that even with EF's bloat you're still not looking at too much shenanigan-code, I would say the expediency it offers is worthwhile. However in larger codebases, it can become a lot to get your hands around so much bloat.
The other benefit to EF vs older style business objects which goes unmentioned though, is with EF implementation you will probably get easier upgrades to newer .NET versions taking advantage of benefits in the next .NET without having to rewrite a bunch of code by hand. (This can also be a double-edged sword as upgrading to new .NET with EF may affect the behaviour of your dal as opposed to a hand-written dal is less likely to be so affected.)
That said, I agree with Joel Etherton, write the simplest smallest dal you can implementing LINQ, the dal is always too important to make overly-complex whenever it can be avoided.
If you do not want to waste you time do not learn DataSets. Study general concepts of object-relational mapping, their pros and cons. Look at projects like Hibernate for Java or Doctrine for PHP. Approaches behind DataTables and DataSets which provide just wrapping of database objects is over. Your framework should guide you to design you domain model, not the database schema.
NHibernate. Especially if you are using Oracle.