Due to the potential differences between Linq-to-Entities (EF4) and Linq-to-Objects, I need to use an actual database to make sure my query classes retrieve data from EF correctly. Sql CE 4 seems to be the perfect tool for this however I have run into a few hiccups. These tests are using MsTest.
The problem I have is if the database doesn't get recreated (due to model changes), data keeps getting added to the database after each test with nothing getting rid of the data. This can potentially cause conflicts in tests, with more data being returned by queries than intended.
My first idea was to initialize a TransactionScope in the TestInitialize method, and dispose the transaction in TestCleanup. Unfortunately, Sql CE4 does not support transactions.
My next idea was to delete the database in TestCleanup via a File.Delete() call. Unfortunately, this seems to not work after the first test is run, as the first test's TestCleanup seems to delete the database, but every test after the first does not seem to re-create the database, and thus it gives an error that the database file is not found.
I attempted to change TestInitialize and TestCleanup tags to ClassInitialize and ClassCleanup for my testing class, but that errored with a NullReferenceException due to the test running prior to ClassInitialize (or so it appears. ClassInitialize is in the base class so maybe that's causing it).
I have run out of ways to effectively use Sql CE4 for testing. Does anyone have any better ideas?
Edit: I ended up figuring out a solution. In my EF unit test base class I initiate a new instance of my data context and then call context.Database.Delete() and context.Database.Create(). The unit tests run a tad slower, but now I can unit test effectively using a real database
Final Edit: After some emails back and forth with Microsoft, it turns out that TransactionScopes are now allowed in SqlCE with the latest release of SqlCE. However, if you are using EF4 there are some limitations in that you must explicitly open the database connection prior to starting the transaction. The following code shows a sample on how to successfully use Sql CE for unit/functional testing:
[TestMethod]
public void My_SqlCeScenario ()
{
using (var context = new MySQLCeModelContext()) //ß derived from DbContext
{
ObjectContext objctx = ((IObjectContextAdapter)context).ObjectContext;
objctx.Connection.Open(); //ß Open your connection explicitly
using (TransactionScope tx = new TransactionScope())
{
var product = new Product() { Name = "Vegemite" };
context.Products.Add(product);
context.SaveChanges();
}
objctx.Connection.Close(); //ß close it when done!
}
}
In your TestInitialize you should do the following:
System.Data.Entity.Database.DbDatabase.SetInitializer<YourEntityFrameworkClass>(
new System.Data.Entity.Database.DropCreateDatabaseAlways<YourEntityFrameworkClass>());
This will cause entity framework to always recreate the database whenever the test is run.
Incidentally you can create an alternative class that inherits from DropCreateDatabaseAlways. This will allow you to seed your database with set data each time.
public class DataContextInitializer : DropCreateDatabaseAlways<YourEntityFrameworkClass> {
protected override void Seed(DataContext context) {
context.Users.Add(new User() { Name = "Test User 1", Email = "test#test.com" });
context.SaveChanges();
}
}
Then in your Initialize you would change the call to:
System.Data.Entity.Database.DbDatabase.SetInitializer<YourEntityFrameworkClass>(
new DataContextInitializer());
I found the approach in the "final edit" works for me as well. However, it's REALLY annoying. It's not just for testing, but any time you want to use TransactionScope with Entity Framework and SQL CE. I want to code once and have my app support both SQL Server and SQL CE, but anywhere I use transactions I have to do this. Surely the Entity Framework team should have handled this for us!
In the meantime, I took it one step farther to make it a little cleaner in my code. Add this block to your data context (whatever class you derive from DbContext):
public MyDataContext()
{
this.Connection.Open();
}
protected override void Dispose(bool disposing)
{
if (this.Connection.State == ConnectionState.Open)
this.Connection.Close();
base.Dispose(disposing);
}
private DbConnection Connection
{
get
{
var objectContextAdapter = (IObjectContextAdapter) this;
return objectContextAdapter.ObjectContext.Connection;
}
}
This makes it a lot cleaner when you actually use it:
using (var db = new MyDataContext())
{
using (var ts = new TransactionScope())
{
// whatever you need to do
db.SaveChanges();
ts.Complete();
}
}
Although I suppose that if you design your app such that all changes are committed in a single call to SaveChanges(), then the implicit transaction would be good enough. For the testing scenario, we want to roll everything back instead of calling ts.Complete(), so it's certainly required there. I'm sure there are other scenarios where we need the transaction scope available. It's a shame it isn't supported directly by EF/SQLCE.
Related
Background
We are trying to archive old user data to keep our most common tables smaller.
Issue
Normal EF code for removing records works for our custom tables. The AspNetUsers table is a different story. It appears that the way to do it is using _userManager.Delete or _userManager.DeleteAsync. These work without trying to do multiple db calls in one transaction. When I wrap this in a transactionScope, it times out. Here is an example:
public bool DeleteByMultipleIds(List<string> idsToRemove)
{
try
{
using (var scope = new TransactionScope())
{
foreach (var id in idsToRemove)
{
var user = _userManager.FindById(id);
//copy user data to archive table
_userManager.Delete(user);//causes timeout
}
scope.Complete();
}
return true;
}
catch (TransactionAbortedException e)
{
Logger.Publish(e);
return false;
}
catch (Exception e)
{
Logger.Publish(e);
return false;
}
}
Note that while the code is running and I call straight to the DB like:
DELETE
FROM ASPNETUSERS
WHERE Id = 'X'
It will also time out. This SQL works before the the C# code is executed. Therefore, it appears that more than 1 db hit seems to lock the table. How can I find the user(db hit #1) and delete the user (db hit #2) in one transaction?
For me, the problem involved the use of multiple separate DbContexts within the same transaction. The BeginTransaction() approach did not work.
Internally, UserManager.Delete() is calling an async method in a RunSync() wrapper. Therefore, using the TransactionScopeAsyncFlowOption.Enabled parameter for my TransactionScope did work:
using (var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
{
_myContext1.Delete(organisation);
_myContext2.Delete(orders);
_userManager.Delete(user);
scope.Complete();
}
Advice from microsoft is to use a different API when doing transactions with EF. This is due to the interactions between EF and the TransactionScope class. Implicitly transaction scope is forcing things up to serializable, which causes a deadlock.
Good description of an EF internal API is here: MSDN Link
For reference you may need to look into user manager if it exposes the datacontext and replace your Transaction scope with using(var dbContextTransaction = context.Database.BeginTransaction()) { //code }
Alternatively, looking at your scenario, you are actually quite safe in finding the user ID, then trying to delete it and then just catching an error if the user has been deleted in the fraction of a second between finding it and deleting it.
As the title suggest i'm having a problem with the first query against a SQL Server database using the Entity Framework. I have tried looking for an answer but no one seems to actually have a solution to this.
The tests was done in Visual Studio 2012 using Entity Framework 6, I also used the T4 views template to pre-compile the views. The database was on a SQL Server 2008. We have about 400 POCOs (400 mapping files), only have 100 rows data in database table.
Following capture is my test code and result.
static void Main(string[] args){
Stopwatch st=new Stopwatch();
st.Start();
new TestDbContext().Set<Table1>.FirstOrDefault();
st.stop();
Console.WriteLine("First Time "+st.ElapsedMilliseconds+ " milliseconds");
st.Reset();
st.Start();
new TestDbContext().Set<Table1>.FirstOrDefault();
st.stop();
Console.WriteLine("Second Time "+st.ElapsedMilliseconds+ " milliseconds");
}
Test results
First Time 15480 milliseconds
Second Time 10 milliseconds
On the first query EF compiles the model. This can take some serious time for a model this large.
Here are 3 suggestions: http://www.fusonic.net/en/blog/2014/07/09/three-steps-for-fast-entityframework-6.1-first-query-performance/
A summary:
Using a cached db model store
Generate pre-compiled views
Generate pre-compiled version of entityframework using n-gen to avoid jitting
I would also make sure that I compile the application in release mode when doing the benchmarks.
Another solution is to look at splitting the DBContext. 400 entities is a lot and it should be nicer to work with smaller chunks. I haven't tried it but I assume it would be possible to build the models one by one meaning no single load takes 15s. See this post by Julie Lerman https://msdn.microsoft.com/en-us/magazine/jj883952.aspx
With EF Core, you can cheat and load the model early after you call services.AddDbContext (you can probably do something similar with EF6 too, but I haven't tested it).
services.AddDbContext<MyDbContext>(options => ...);
var options = services.BuildServiceProvider()
.GetRequiredService<DbContextOptions<MyDbContext>>();
Task.Run(() =>
{
using(var dbContext = new MyDbContext(options))
{
var model = dbContext.Model; //force the model creation
}
});
This will create the model of the dbcontext in another thread while the rest of the initialization of the application is done (and maybe other warmups) and the beginning of a request. This way, it will be ready sooner. When you need it, EFCore will wait for the Model to be created if it hasn't finished already. The Model is shared across all DbContext instances so it is ok to fire and forget this dummy dbcontext.
You can try something like this: (it worked for me)
protected void Application_Start()
{
Start(() =>
{
using (EF.DMEntities context = new EF.DMEntities())
{
context.DMUsers.FirstOrDefault();
}
});
}
private void Start(Action a)
{
a.BeginInvoke(null, null);
}
Entity Framework - First query slow
this work for me:
using (MyEntities db = new MyEntities())
{
db.Configuration.AutoDetectChangesEnabled = false; // <----- trick
db.Configuration.LazyLoadingEnabled = false; // <----- trick
DateTime Created = DateTime.Now;
var obj = from tbl in db.MyTable
where DateTime.Compare(tbl.Created, Created) == 0
select tbl;
dataGrid1.ItemsSource = obj.ToList();
dataGrid.Items.Refresh();
}
If you have many tables that are not being used on c#, exclude them.
Add a partial class, add the following code and reference this function on OnModelCreating
void ExcludedTables(DbModelBuilder modelBuilder)
{
modelBuilder.Ignore<Table1>();
modelBuilder.Ignore<Table>();
// And so on
}
For me, just using AsParallel() in the first query solved the problem. This runs the query on multiple processor cores (apparently). All my subsequent queries are unchanged, it is only the first one which was causing the delay.
I also tried pre-generated mapping views https://learn.microsoft.com/en-us/ef/ef6/fundamentals/performance/pre-generated-views but this did not improve startup time by much.
I think that is not a very good solution. Ado.net looks like a lot more performance. However, this is my opinion.
Alternatively look at them.
https://msdn.microsoft.com/tr-tr/data/dn582034
https://msdn.microsoft.com/en-us/library/cc853327(v=vs.100).aspx
I'm working on a project built on Asp.Net Boilerplate, and now I have to unit test the services using the real repositories with a real database connection (no mocking). I've been using the last post of BringerOd in https://gist.github.com/hikalkan/1e5d0f0142484da994e0 as a guide for setting up my UnitOfWorkScope instance. So, my code currently looks something like this:
IDisposableDependencyObjectWrapper<IUnitOfWork> _unitOfWork;
[TestInitialize]
public void SetUpService()
{
//initialize service
_unitOfWork = IocManager.Instance.ResolveAsDisposable<IUnitOfWork>();
UnitOfWorkScope.Current = _unitOfWork.Object;
UnitOfWorkScope.Current.Initialize(true);
UnitOfWorkScope.Current.Begin();
}
[TestCleanup]
public void CleanUpService()
{
UnitOfWorkScope.Current.Cancel();
_unitOfWork.Dispose();
UnitOfWorkScope.Current = null;
}
This works like a charm for the first unit test, but when I try to make a repository call in a second test, I get: "The operation cannot be completed because the DbContext has been disposed."
My guess is that when the TestInitialize method runs again, the unit of work scope is getting assigned with the same (disposed) DbContext, rather than a new one. I suppose, inside my actual test methods, I could set up my UnitOfWorkScope inside a using block with the IUnitOfWork. However, I really don't want to repeat that logic inside inside every single test. Does anyone know how to manually get the effect of a using block so that I get a brand new DbContext each time?
Check: http://aspnetboilerplate.com/Pages/Documents/Repositories
You must mark the calling method with [UnitOfWork] attribute.
The reason for this, as explained in the linked document is
When you call GetAll() out of a repository method, there must be an open database connection. This is because of deferred execution of IQueryable<T>. It does not perform database query unless you call ToList() method or use the IQueryable<T> in a foreach loop (or somehow access to queried items). So, when you call ToList() method, database connection must be alive. This can be achieved by marking caller method with the [UnitOfWork] attribute of ASP.NET Boilerplate. Note that Application Service methods are already using [UnitOfWork] as default, so, GetAll() will work without adding the [UnitOfWork] attribute for application service methods.
The previous version and question are provided as an added context below. The improved problem formulation and question could be as follows:
How does one share a transaction between multiple contexts in EF 6.1.0 database first and .NET 4.5.2 without doing a distributed transaction?
For that it looks like I need to share a connection between the multiple contexts, but the code examples and tutorials I've been looking at thus far haven't been that fruitful. The problem looks like is hovering around on how to define a functioning combination of a connection object and transaction object types so that EF database first object metadata is also built and found when constructing the object contexts.
That is, I would like to do akin to what has been described in the EF 6.n tutorials here. Some example code could be
int count1;
int count2;
using (var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
{
//How to define this connection so as not to run into UnintentionalCodeFirstException?
//Creating a dummy context to obtain the connectiong string like so
//dummyContext.Database.Connection.ConnectionString and then using the connection will be greeted with the aforementioned exception.
using(var conn = new SqlConnection("..."))
{
using(var c1 = new SomeEntities(conn, contextOwnsConnection: false))
{
//Use some stored procedures etc.
count1 = await c1.SomeEntity1.CountAsync();
}
using(var c2 = new SomeEntities(conn, contextOwnsConnection: false))
{
//Use some stored procedures etc.
count2 = await c2.SomeEntity21.CountAsync();
}
}
}
int count = count1 + count2;
In the examples there are also other methods as to how to create a shared connection and a transaction, but as written, the culprit seem to be that if, say, I provide the connectiong string in (the "..." part) the previous snippet as dummyContext.Database.Connection.ConnectionString I'll get just an exception.
I'm not sure if I'm just reading the wrong sources or if there's something else that's wrong in my code when I try to share a transaction across multiple EF contexts. How could it be done?
I've read quite a few other SO posts regarding this (e.g. this) and some tutorials. They did not help.
I have a strange problem in that it looks I don't have the constructor overloads defined as in other tutorials and posts. That is, taking the linked tutorial link, I can't write new BloggingContext(conn, contextOwnsConnection: false)) and use a shared connection and an external transaction.
Then if I write
public partial class SomeEntities: DbContext
{
public SomeEntities(DbConnection existingConnection, bool contextOwnsConnection): base(existingConnection, contextOwnsConnection) { }
}
and use it like in the tutorials, I get an exception from the following line from the following T4 template generated code
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
throw new UnintentionalCodeFirstException();
}
I'm using .NET 4.5.2 and EF 6.1.0. I'ved constructed the edmx from an existing database and generated the code from there. In this particular situation I'm using Task Parallel threads to load dozens of SQL Server Master Data Services staging tables (yes, a big model) and to call the associated procedures (provided by the MDS one per table). MDS has its own compensation logic in case staging to some of the tables fails, but rolling back a transaction should be doable too. It just looks like I have a (strange) problem with my EF.
<Addendum: Steve suggested using straight TransactionScope. Without a shared connection that would require a distributed transaction, which isn't an option I can choose. Then if I try to provide a shared connection for the contexts (some options shown in the tutorials, one here I have the problem of "missing constructors". When I define one, I get the exception I refer in the code. All in all, this feels quite strange. Maybe there's something wrong in how I go about generating the DbContext and related classes.
<Note 1: It looks like the root cause is as in this blog post by Arthur (of EF developer team) Don't use Code First by mistake. That is, in database first development the framework seeks for the class-relational mappings as defined in the connection string. Something fishy in my connection string that is..?
Have you tried wrapping the calls in a transaction scope?
using (var scope = new TransactionScope(TransactionScopeOption.Required, new TransactionOptions() { IsolationLevel = IsolationLevel.ReadCommitted }))
{
// Do context work here
context1.Derp();
context2.Derp();
// complete the transaction
scope.Complete();
}
Because of connection pool, using multiple EF DbContext which are using exactly the same connection string, under same transaction scope will usually not result in DTC escalation (unless you disabled pooling in connection string), because the same connection will be reused for them both (from the pool). Anyway, you can reuse the same connection in your case like this (I assume you already added constructor which accepts DbConnection and flag indicating if context owns connection):
using (var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled)) {
// important - use EF connection string here,
// one that starts with "metadata=res://*/..."
var efConnectionString = ConfigurationManager.ConnectionStrings["SomeEntities"].ConnectionString;
// note EntityConnection, not SqlConnection
using (var conn = new EntityConnection(efConnectionString)) {
// important to prevent escalation
await conn.OpenAsync();
using (var c1 = new SomeEntities(conn, contextOwnsConnection: false)) {
//Use some stored procedures etc.
count1 = await c1.SomeEntity1.CountAsync();
}
using (var c2 = new SomeEntities(conn, contextOwnsConnection: false)) {
//Use some stored procedures etc.
count2 = await c2.SomeEntity21.CountAsync();
}
}
scope.Complete();
}
This works and does not throw UnintentionalCodeFirstException because you pass EntityConnection. This connection has information about EDMX metadata and that is what database first needs. When you pass plain SqlConnection - EF has no idea where to look for metadata and actually doesn't even know it should look for it - so it immediately assumes you are doing code-first.
Note that I pass EF connection string in code above. If you have some plain SqlConnection which you obtained by some other means, outside EF, this will not work, because connection string is needed. But, it's still possible because EntityConnection has constructor which accepts plain DbConnection. However, then you should pass reference to metadata yourself. If you are interested in this - I can provide code example of how to do that.
To check that you indeed prevent escalation in all cases - disable pooling (Pooling=false in connection string) and stop DTC service, then run this code - it should run fine. Then run another code which does not share the same connection and you should observe error indicating escalation was about to happen but service is not available.
I'd like to run some tests of stored procedures in my database without actually affecting the data (or, to put it more exactly, without a lasting impact after the test has run).
After some research I came up with the approach of using TransactionScope within my Visual Studio 2010 test project such as
using( new TransactionScope())
{
using( SqlConnection connection = new SqlConnection("someConnectionString"))
{
connection.Open();
using( SqlCommand command = new SqlCommand( "some sql", connection ))
{
// Do some database stuff...
}
}
}
Now this works fine as long as I put all of this within a single test method, i.e. all my changes to the database are automatically rolled back when the using block for TransactionScope is finished.
My problem is now that I'd like to do some database stuff in ClassInitialize so I only have to do it once per test class and not for every test method. When I create a public TransactionScope property and assign it an instance of TransactionScope in the ClassInitialize method, this works okay. As soon as I do any database related stuff in one of my test methods, I run into a TransactionManagerCommunicationException within that method.
I don't quite understand why that is the case, and I'd also like to know whether there is a mistake in my approach or how I can get it to work without having to set up the TransactionScope including all set up stuff for the tests within each test method again.
EDIT
Code excerpt below, I hope this gives enough information:
public TransactionScope Scope { get; set; }
[ClassInitialize]
public static void ClassInitialize( TestContext testContext )
{
Scope = new TransactionScope();
// Do some db set up stuff, e.g. create records used for tests etc.
}
[ClassCleanup]
public static void ClassCleanup()
{
Scope.Dispose();
}
[TestMethod]
public void MyTestMethod()
{
using( SqlConnection connection = new SqlConnection( "someConnectionString" ) )
{
DataTable result = new DataTable();
using( SqlCommand command = new SqlCommand( "spName", connection ) )
{
command.CommandType = CommandType.StoredProcedure;
using( SqlDataAdapter adapter = new SqlDataAdapter() )
{
adapter.SelecteCommand = command;
// The next line causes the exception to be thrown
adapter.Fill( result );
}
}
// Assertions against DataTable result
}
}
The exception is
TransactionManagerCommunicationException was unhandled by user code
Network access for Distributed Transaction Manager (MSDTC) has been disabled. Please enable DTC for network access in the security configuration for MSDTC using the Component Services Administrative tool.
I understand that I could try and change the settings, but I do not understand why I get the exception to begin with - what is different compared to having the code above in a single (test) method?
Thanks in advance and
best regards
G.
Your exception is saying that MSDTC isn't enabled. My guess is that when you were using TransactionScope individually, it was just creating local SQL transactions -- which don't require DTC. However, when you share a TransactionScope over multiple connections, the transaction gets "promoted" to a distributed transaction through the DTC, which you may not have enabled.
Try enabling network access on MSDTC on your local machine and the server. The steps for doing so vary a little depending on your OS. Here's how to do it in Win 2003 Server. Here's a link for Win 2008. Note that you will likely need to enable DTC through your firewalls as well (explained in the last link...)
You could create your setup something like this:
void Main()
{
using(new SetupTransaction())
{
//Your test
}
}
public class SetupTransaction : IDisposable
{
private TransactionScope transaction;
public SetupTransaction()
{
transaction = new TransactionScope();
//Do your stuff here
}
public void Dispose()
{
transaction.Dispose();
}
}
As for the error you're getting, could you post exactly how you're using your implementation?
One approach I have used with success is creating a base class that implements setup and teardown. Within the setup method you create a new transaction scope and store it in a private class variable. Within the teardown method you rollback the transaction scope.
I'm using NUNit, but the principle should be the same for MSTest. The clue here is that SetUp and TearDown executes once before and after each unit test to ensure isolation among unit tests.
Also, as #blech mentions, the Microsoft Distributed Transaction Coordinator (MSDTC) service must be running for this solution to work.