Develop Reference

ServiceStack / FluentNHibernate / MySQL - Same connection used by two concurrent requests

We seem to have come up on a weird issue, where two concurrent requests to our service are actually using the same DB connection.
Our setup is ServiceStack + NHibernate + FluentNHibernate + MySQL. I have set up a small test that recreates the problem:
public class AppHost : AppHostBase
{
private ISessionFactory _sessionFactory;
public AppHost() : base("Lala Service", typeof(AppHost).Assembly)
{
}
public override void Configure(Container container)
{
_sessionFactory = Fluently.Configure()
.Database(MySQLConfiguration.Standard.ConnectionString(conn =>
conn.Server("localhost").Username("lala").Password("lala").Database("lala")))
.Mappings(mappings => mappings.AutoMappings.Add(
AutoMap.Assembly(GetType().Assembly).Where(t => t == typeof(Lala))
.Conventions.Add(DefaultLazy.Never(), DefaultCascade.All())))
.BuildSessionFactory();
container.Register(c => _sessionFactory.OpenSession()).ReusedWithin(ReuseScope.Request);
}
}
public class Lala
{
public int ID { get; set; }
public string Name { get; set; }
}
[Route("/lala")]
public class LalaRequest
{
}
public class LalaReseponse
{
}
public class LalaService : Service
{
private ISession _session;
public ISession Session1
{
get { return _session; }
set { _session = value; }
}
public LalaReseponse Get(LalaRequest request)
{
var lala = new Lala
{
Name = Guid.NewGuid().ToString()
};
_session.Persist(lala);
_session.Flush();
lala.Name += " XXX";
_session.Flush();
return new LalaReseponse();
}
}
The I hit this service 10 times concurrenly via Ajax like so:
<script type="text/javascript">
for (i = 0; i < 10; i++) {
console.log("aa");
$.ajax({
url: '/lala',
dataType: 'json',
cache: false
});
}
</script>
The result is consistenly:
Number of connections open < 10.
Not all records updated.
On occasion - a StaleObjectStateException thrown - if I delete records.
The reason behind this is that the connections are reused by two concurrent requests, and then LAST_INSERT_ID() gives the ID of the wrong row, so two requests are updating the same row.
In short: it's a complete mess and it's clearly sharing the DB connection between requests.
The question is: Why? How should I have configured things so that each request gets its own connection from the connection pool?

Finally solved it, what a day-waster!
The source of the problem is NHibernate's connection release mode:
11.7. Connection Release Modes
The legacy (1.0.x) behavior of NHibernate in regards to ADO.NET
connection management was that a ISession would obtain a connection
when it was first needed and then hold unto that connection until the
session was closed. NHibernate introduced the notion of connection
release modes to tell a session how to handle its ADO.NET connections.
...
The different release modes are identified by the enumerated values of
NHibernate.ConnectionReleaseMode:
OnClose - is essentially the legacy behavior described above. The
NHibernate session obtains a connection when it first needs to perform
some database access and holds unto that connection until the session
is closed.
AfterTransaction - says to release connections after a
NHibernate.ITransaction has completed.
The configuration parameter hibernate.connection.release_mode is used
to specify which release mode to use.
...
after_transaction - says to use
ConnectionReleaseMode.AfterTransaction. Note that with
ConnectionReleaseMode.AfterTransaction, if a session is considered to
be in auto-commit mode (i.e. no transaction was started) connections
will be released after every operation.
This got entangled together with MySQL .NET/Connector's default connection pooling, and effectively meant that the connections were swapped between concurrent requests, as one request released the connection back to the pool and the other acquired it.
However, I think that the fact that NHibernate calls LAST_INSERT_ID() after releasing and re-acquiring the connection is a bug. It should call LAST_INSERT_ID() inside the same "operation".
Anyway, solutions:
Use transactions, which is what we normally do, or
If you can't or don't want to use transactions in a certain context for some reason (which is what happened to use today), set the connection release mode to "on close". With FluentNHibernate that would be:
.ExposeConfiguration(cfg =>
cfg.SetProperty("connection.release_mode", "on_close"));
And from here on the connection is bound to the session even if there is no transaction.

Transaction Escalated to DTC No Multiple Connections

Does anyone know of any cases when using a transaction scope the transaction is escalated to the DTC when multiple connections are NOT open.
I am aware that if I open multiple connections(no matter what connection string) within a transaction scope that the transaction will most likely be promoted to the DTC.
Knowing this I have gone to great lengths to make sure there is only ONE connection opened within my transactions.
However, I have a client where they are getting the exception
An error has occurred. Csla.DataPortalException: DataPortal.Update failed (The underlying provider failed on Open.) ---> Csla.Reflection.CallMethodException: EditableCategory.DataPortal_Update method call failed ---> System.Data.EntityException: The underlying provider failed on Open. ---> System.Transactions.TransactionManagerCommunicationException: Network access for Distributed Transaction Manager (MSDTC) has been disabled. Please enable DTC for network access in the security configuration for MSDTC using the Component Services Administrative tool. ---> System.Runtime.InteropServices.COMException: The transaction manager has disabled its support for remote/network transactions.
Again, I am pretty sure there is only one connection opened within the scope. Take a look.
protected override void DataPortal_Update()
{
using (System.Transactions.TransactionScope ts = new System.Transactions.TransactionScope(System.Transactions.TransactionScopeOption.Required, System.Transactions.TransactionManager.MaximumTimeout))
{
//get the dal manager he knows which dal implementation to use
using (var dalMgr = DataAccess.BusinessObjectsDalFactory.GetManager())
{
//get the category dal implementation
var ecDal = dalMgr.GetProvider<DataAccess.BusinessObjectDalInterfaces.ICategoryDAL>();
//assume all the data is good at this point so use bypassproperty checks
using (BypassPropertyChecks)
{
var catData = new Models.Category { CategoryId = CategoryId, CategoryName = CategoryName, LastChanged = TimeStamp };
ecDal.UpdateCategory(catData);
TimeStamp = catData.LastChanged;
}
}
ts.Complete();
}
base.DataPortal_Update();
}
public class DalManager : Core.Sebring.DataAccess.IBusinessObjectsDalManager {private static string _typeMask = typeof(DalManager).FullName.Replace("DalManager", #"{0}");
public T GetProvider<T>() where T : class
{
var typeName = string.Format(_typeMask, typeof(T).Name.Substring(1));
var type = Type.GetType(typeName);
if (type != null)
return Activator.CreateInstance(type) as T;
else
throw new NotImplementedException(typeName);
}
public Csla.Data.DbContextManager<DataContext> ConnectionManager { get; private set; }
public DalManager()
{
ConnectionManager = Csla.Data.DbContextManager<DataContext>.GetManager();
}
public void Dispose()
{
ConnectionManager.Dispose();
ConnectionManager = null;
}
public void UpdateDataBase()
{
DatabaseUpgrader.PerformUpgrade();
}
}
public void UpdateCategory(Models.Category catData)
{
if (catData == null) return;
using (var cntx = DbContextManager<DataContext>.GetManager())
{
var cat = cntx.DbContext.Set<Category>().FirstOrDefault(c => c.CategoryId == catData.CategoryId);
if (cat == null) return;
if (!cat.LastChanged.Matches(catData.LastChanged))
throw new ConcurrencyException(cat.GetType().ToString());
cat.CategoryName = catData.CategoryName;
//cntx.DbContext.ChangeTracker.DetectChanges();
cntx.DbContext.Entry<Category>(cat).State = System.Data.EntityState.Modified;
cntx.DbContext.SaveChanges();
catData.LastChanged = cat.LastChanged;
}
}
The code for DBContextManager is available, but in short it just makes certain there is only one DBContext, and hence one connection opened. Am I overlooking something? I guess its possible that something is up with DBConextManager, so I have posted in the CSLA forums as well(DBContextManager is a part of CSLA). But has anyone run into scenarios where they are sure one connection is opened within the transaction scope and the transaction is escalated to the DTC?
Of course I cannot reproduce the exception on my local dev machine OR any of our QA machines.
Any help is appreciated.
Thanks.

Entity Framework can randomly try to open a new connection when doing transactions with System.Transactions.TransactionScope
Try adding a finally statement and dispose your transaction, also call your dbContext and manually close the connection , this will lesser the ammount of times the transaction gets escalated but it might still happen:
finally
{
cntx.Database.Connection.Close();
transaction.Dispose();
}
Its a known "bug" you can find more here :
http://petermeinl.wordpress.com/2011/03/13/avoiding-unwanted-escalation-to-distributed-transactions/

Should I keep an instance of DbContext in a separate thread that performs periodic job

I have a class Worker which sends emails periodically,I start in Global.asax.cs on App_start()
public static class Worker
{
public static void Start()
{
ThreadPool.QueueUserWorkItem(o => Work());
}
public static void Work()
{
var r = new DbContext();
var m = new MailSender(new SmtpServerConfig());
while (true)
{
Thread.Sleep(600000);
try
{
var d = DateTime.Now.AddMinutes(-10);
var ns = r.Set<Notification>().Where(o => o.SendEmail && !o.IsRead && o.Date < d);
foreach (var n in ns)
{
m.SendEmailAsync("noreply#example.com", n.Email, NotifyMailTitle(n) + " - forums", NotifyMailBody(n));
n.SendEmail = false;
}
r.SaveChanges();
}
catch (Exception ex)
{
ex.Raize();
}
}
}
}
So I keep this dbcontext alive for the entire lifetime of the application is this a good practice ?

DbContext is a very light-weight object.
It doesn't matter whether your DbContext stays alive or you instantiate it just before making the call because the actual DB Connection only opens when you SubmitChanges or Enumerate the query (in that case it is closed on end of enumeration).
In your specific case. It doesn't matter at all.
Read Linq DataContext and Dispose for details on this.

I would wrap it in a using statement inside of Work and let the database connection pool do it's thing:
using (DbContext r = new DbContext())
{
//working
}
NOTE: I am not 100% sure how DbContext handles the db connections, I am assuming it opens one.
It is not good practice to keep a database connection 'alive' for the lifetime of an application. You should use a connection when needed and close it via the API(using statement will take care of that for you). The database connection pool will actually open and close connections based on connection demands.

I agree with #rick schott that you should instantiate the DbContext when you need to use it rather than keep it around for the lifetime of the application. For more information, see Working with Objects (Entity Framework 4.1), especially the section on Lifetime:
When working with long-running context consider the following:
As you load more objects and their references into memory, the
memory consumption of the context may increase rapidly. This may cause
performance issues.
If an exception causes the context to be in an unrecoverable state,
the whole application may terminate.

Question about Entity Framework and Transactions

public void SomeMethod1()
{
using (TemplateEntities ctx = new TemplateEntities())
{
//do something in this ctx
}
}
public void SomeMethod2()
{
using (TemplateEntities ctx = new TemplateEntities())
{
//do something else in this ctx
}
}
public void SomeMethod()
{
using (TemplateEntities ctx = new TemplateEntities())
{
using (TransactionScope tran = new TransactionScope())
{
SomeMethod1();
SomeMethod2();
var itemToDelete= (from x in ctx.Xxx
where x.Id==1
select x).Single();
ctx.Xxx.DeleteObject(itemToDelete);
ctx.SaveChanges();
tran.Complete();
}
}
}
What happens in SomeMethod is executed in a transaction even if there are more contexts?
I am using POCO.

If you use TransactionScope with multiple ObjectContext instances the transaction will be promoted to distributed and whole operation (SomeMethod) will be handled still as atomic. But distributed transaction requires additional NT service and its dependecies. The service is called Microsoft Distributed Transaction Coordinator (MSDTC). This service has to run on all involved servers (application server and database server). In network scenario service requires some additional configuration. For communication RPC ports have to be opened in firewalls.

Ultimately the database doesn't know about data-contexts, so simply: the rules of transactions apply. Being a serializable transaction, things like read locks and key-range-locks will be issued and honoured. As always, there is a risk of complication from deadlocks, but ultimately it should work. Note that all the contexts involved should enlist as required.

How do you get around multiple database connections inside a TransactionScope if MSDTC is disabled?

I have a web application that issues requests to 3 databases in the DAL. I'm writing some integration tests to make sure that the overall functionality round trip actually does what i expect it to do. This is completely separate from my unit tests, just fyi.
The way I was intending to write these tests were something to the effect of this
[Test]
public void WorkflowExampleTest()
{
(using var transaction = new TransactionScope())
{
Presenter.ProcessWorkflow();
}
}
The Presenter in this case has already been set up. The problem comes into play inside the ProcessWorkflow method because it calls various Repositories which in turn access different databases, and my sql server box does not have MSDTC enabled, so I get an error whenever I try to either create a new sql connection, or try to change a cached connection's database to target a different one.
For brevity the Presenter resembles something like:
public void ProcessWorkflow()
{
LogRepository.LogSomethingInLogDatabase();
var l_results = ProcessRepository.DoSomeWorkOnProcessDatabase();
ResultsRepository.IssueResultstoResultsDatabase(l_results);
}
I've attempted numerous things to solve this problem.
Caching one active connection at all times and changing the target database
Caching one active connection for each target database (this was kind of useless because pooling should do this for me, but I wanted to see if I got different results)
Adding additional TransactionScopes inside each repository so that they have their own transactions using the TransactionScopeOption "RequiresNew"
My 3rd attempt on the list looks something like this:
public void LogSomethingInLogDatabase()
{
using (var transaction =
new TransactionScope(TransactionScopeOption.RequiresNew))
{
//do some database work
transaction.Complete();
}
}
And actually the 3rd thing I tried actually got the unit tests to work, but all the transactions that completed actually HIT my database! So that was an utter failure, since the entire point is to NOT effect my database.
My question therefore is, what other options are out there to accomplish what I'm trying to do given the constraints I've laid out?
EDIT:
This is what "//do some database work" would look like
using (var l_context = new DataContext(TargetDatabaseEnum.SomeDatabase))
{
//use a SqlCommand here
//use a SqlDataAdapter inside the SqlCommand
//etc.
}
and the DataContext itself looks something like this
public class DataContext : IDisposable
{
static int References { get; set; }
static SqlConnection Connection { get; set; }
TargetDatabaseEnum OriginalDatabase { get; set; }
public DataContext(TargetDatabaseEnum database)
{
if (Connection == null)
Connection = new SqlConnection();
if (Connection.Database != DatabaseInfo.GetDatabaseName(database))
{
OriginalDatabase =
DatabaseInfo.GetDatabaseEnum(Connection.Database);
Connection.ChangeDatabase(
DatabaseInfo.GetDatabaseName(database));
}
if (Connection.State == ConnectionState.Closed)
{
Connection.Open() //<- ERROR HAPPENS HERE
}
ConnectionReferences++;
}
public void Dispose()
{
if (Connection.State == ConnectionState.Open)
{
Connection.ChangeDatabase(
DatabaseInfo.GetDatabaseName(OriginalDatabase));
}
if (Connection != null && --ConnectionReferences <= 0)
{
if (Connection.State == ConnectionState.Open)
Connection.Close();
Connection.Dispose();
}
}
}

Set Enlist=false on connection string to avoid auto enlistment on transaction.
Manually enlist connection as participants in transaction scope. (http://msdn.microsoft.com/en-us/library/ms172153%28v=VS.80%29.aspx)

Ok, I found a way around this issue. The only reason I'm doing it this way is because I couldn't find ANY other way to fix this problem, and because it's in my integration tests, so I'm not concerned about this having adverse effects in production code.
I had to add a property to my DataContext to act as a flag to keep track of whether or not to dispose of the connection object when my DataContext is being disposed. This way, the connection is kept alive throughout the entire transaction scope, and therefore no longer bothers DTC
Here's sample of my new Dispose:
internal static bool SupressConnectionDispose { get; set; }
public void Dispose()
{
if (Connection.State == ConnectionState.Open)
{
Connection.ChangeDatabase(
DatabaseInfo.GetDatabaseName(OriginalDatabase));
}
if (Connection != null
&& --ConnectionReferences <= 0
&& !SuppressConnectionDispose)
{
if (Connection.State == ConnectionState.Open)
Connection.Close();
Connection.Dispose();
}
}
this allows my integration tests to take the form of:
[Test]
public void WorkflowExampleTest()
{
(using var transaction = new TransactionScope())
{
DataContext.SuppressConnectionDispose = true;
Presenter.ProcessWorkflow();
}
}
I would not recommend utilizing this in production code, but for integration tests I think it is appropriate. Also keep in mind this only works for connections where the server is always the same, as well as the user.
I hope this helps anyone else who runs into the same problem I had.

If you don't want to use MSDTC you can use SQL transactions directly.
See SqlConnection.BeginTransaction().