I am having a problem in a production environment that I am not getting locally.
I am running some LINQ to SQL code within a TransactionScope as below:
using (var scope = new TransactionScope())
{
uploadRepository.SubmitChanges();
result = SubmitFileResult.Succeed();
ScanForNewData(upload);
scope.Complete();
}
ScanForNewData() calls GetSubmittedData(). If an exception occurs in GetSubmitted() we use Nlog to write the error to file, database and also send an email:
catch (Exception ex)
{
//MT - having to comment this out beause it is causing a problem with transactions on theproduction server
logger.ErrorException(String.Format("Error reading txt file {0} into correct format", upload.DocumentStore.FileName), ex);
return new UploadGetSubmittedDataResult { Exception = ex, Success = false, Message = String.Format("Error reading txt file {0} into correct format", upload.DocumentStore.FileName) };
}
In ScanForNewData we then call repository.SubmitChanges().This then causes:
The operation is not valid for the state of the transaction. System.Transactions.TransactionException TransactionException System.Transactions.TransactionException: The operation is not valid for the state of the transaction.
The best idea I have come up with is that in production this code is running on a web server and calling a separate database server. Both the DataContext and Nlog have the same connectionstring configuration and Sql user, but maybe because the server is remote (whereas locally I am using integrated security) something strange is happening.
Any idea what happens to the transaction in this scenario?
Update - I just tried it with SQL user locally and it still works fine. Must be something to do with the production set up...
Another update - I tell a lie. On the dev maching the Nlog database record is never written, the email is sent, and the TransactionException does not happen.
Hard to guess what is the problem without a full stack trace of the exception, it may depend on multiple things.
For instance, I'm assuming NLog opens a new connection to the db my himself, and that will probably cause the transaction to be promoted to a distributed one, and the Distributed Transaction Coordinator will kick in. This can cause the asymmetry between the behavior of your application in production and locally.
You may be breaking the transaction with some operation inside it, like some unhandled exception or illegal accessing of some data.
Provide full stack trace and more code involved for a deeper analysis.
Without knowing what the inner exceptions off of your TransactionException is it will be difficult to resolve but here is a thought:
If you refactor your code to have the logging occur after the using block around the transaction scope has ended you will likely avoid the issue you are having since the transaction scope will be ended and DTC will roll back the transaction.
I have used and seen this pattern in the past (don't log until after the transaction is ended and rolled back) when dealing with transactions and it has worked well.
Doing logging on a separate database is always advisable to avoid issues like this as well. If you did this the issue would also be avoided.
Have a look at this..seems to be a bug with Nlog.
https://groups.google.com/forum/#!msg/nlog-users/I5UR-bISlTA/6RPtOZhR4NoJ
suggested solution is to use async target for Db logging.
Related
You may think this has been asked before, but hear me out first.
I have a project that runs across different servers with remote connections.
On the development server, all tests pass.
In the QA server that has same configuration, most tests pass but a couple fail and the exception is
The underlying provider failed on Open.
Inner exception:
Network access for Distributed Transaction Manager (MSDTC) has been
disabled.
Please enable DTC for network access in the security configuration for
MSDTC using the Component Services Administrative tool.
Inner inner exception:
The transaction manager has disabled its support for remote/network transactions.
(Exception from HRESULT: 0x8004D024)
As far as I can see the configuration is correct and in fact the tests that pass run the exact same code without issues so there are no configuration issues related to MSDTC.
The piece of code that causes this is an entity framework connection that adds a simple row to a database table:
using (var inbound = new InBound.InBound())
{
var now = DateTime.Now;
var inMessage = inbound.InMessages.Create();
inMessage.Content = message.ToXmlFormattedString();
inMessage.Created = now;
inMessage.SerializedVersion = version;
inMessage.Updated = now;
inbound.InMessages.Add(inMessage);
inbound.SaveChanges(); //Exception here
}
This same piece of code does its job on some tests, but fails in a few others.
This problem only occurs on the QA environment. All MSDTC related configuration has already been checked and appears correct. DEV environment has the same configuration and this error doesn't happen.
The environment is identical. Microsoft Server 2012 R2, MS SQL Server 2008, same codebase, same settings.
The xml in the message is the same in both the test that fails and the test that passes.
I've exhausted the list of places I can look for a solution and there seems no logical explanation at all for this.
Any ideas?
UPDATE
If I add "Enlist=false" in the connection string then the tests pass again.
However I do get an internal exception that gets drowned and that is:
Cannot enlist in the transaction because a local transaction is in
progress on the connection. Finish local transaction and retry.
so the connection must be open somehow, even though during debugging the state of the database is "closed".
I have some business logic that executes within a simple TransactionScope:
using (var scope = new TransactionScope())
{
// does some logic.
// then some more...
scope.Complete();
}
During execution an ADO exception occurs and the transaction is on it's way to being disposed.
In the meantime the exception is captured further upstream, and I try to log it to my database, via another connection.
At this point I get another error saying The operation is not valid for the state of the transaction.
Eeek!
Now I can see I am able to use IsolationLevel, to change the way my transactions interact, but is this the right thing to do? And which should I use?
Since your are mentioning multiple connections and the error you are referring to is something I have seen with DTC transactions I am guessing you are running DTC and that the state of the transaction is faulted due to the AdoException. Try putting a "RequiresNew" transaction scope around your logging code or post more of your code, its hard to see your architecture from that small snippet.
Take a look at this answer showing how two connections can cause DTC to kick in based on which version of sql server you are running:
TransactionScope automatically escalating to MSDTC on some machines?
As your problem occurs because the transaction is being rolled back / disposed, I'd see two options: using a new transaction or no transaction at all to log. I'd probably go with the second option and log without a transaction.
You can pass a parameter of type TransactionScopeOption to the TransactionScope constructor to suppress transactions for your logging statements like this:
using (var scope = new TransactionScope(TransactionScopeOption.Suppress)
{
// .. log here
}
To log within a new transaction pass TransactionScopeOption.RequiresNew.
you can not have two connections open in transaction scope. close first connection, then open connection for logging error in db.
have look at this "The operation is not valid for the state of the transaction" error and transaction scope
I have a service application in C# which queries data from one database and inserts it into another SQL database.
Sometimes, the MSSQLSERVER service crashed for unknown reason and my application will crash as well. I want to do a SQL recovery mechanism that where I check to make sure the sqlconnection state is fine before i write to the database but how i do that?
I tried stopping MSSQLSERVER service and sqlconnection.State is always open even when the MSSQLSERVER service is stopped.
First: Fix your real problem. SQL Server should be very, very stable.
Second: consider using MSMQ (or SQL Service Broker) on both the client application and server to queue updates.
The general strategy of checking the connection state before calling a SQL command fundamentally won't work. What happens if the service crashes after your connection check, but before you call the SQL command?
You probably will need to figure out what exception is thrown when the database is down and recover from that exception at the appropriate layer of code.
I think that the approach you chose is not very good.
If your application is some kind of scheduled job, let it crash. No database - no work can be done. This is ok to crash in this case. Next time it runs and db is up it will do its thing. You can also implement retries.
If your application is a windows service inside and some kind of scheduled timer, you just make sure that your service doesn't crash by handling SqlExcpetion. Retry again until server is up.
Also, you might want to use distributed transactions. To guarantee integrity of the copy procedure, but whether you need it or not, depends on the requirements.
[Edit] In response to retry question.
var attemptNumber = 0;
while (true)
{
try
{
using (var connection = new SqlConnection())
{
connection.Open();
// do the job
}
break;
}
catch (SqlException exception)
{
// log exception
attemptNumber++;
if (attemptNumber > 3)
throw; // let it crash
}
}
When opening a connection to SQL Server 2005 from our web app, we occasionally see this error:
"Impersonate Session Security Context" cannot be called in this batch because a simultaneous batch has called it.
We use MARS and connection pooling.
The exception originates from the following piece of code:
protected SqlConnection Open()
{
SqlConnection connection = new SqlConnection();
connection.ConnectionString = m_ConnectionString;
if (connection != null)
{
try
{
connection.Open();
if (m_ExecuteAsUserName != null)
{
string sql = Format("EXECUTE AS LOGIN = {0};", m_ExecuteAsUserName);
ExecuteCommand(connection, sql);
}
}
catch (Exception exception)
{
connection.Close();
connection = null;
}
}
return connection;
}
I found an MS Connect article which suggests that the error is caused when a previous command has not yet terminated before the EXECUTE AS LOGIN command is sent. Yet how can this be if the connection has only just been opened?
Could this be something to do with connection pooling interacting strangely with MARS?
UPDATE: For the short-term we have implemented a workaround by clearing out the connection pool whenever this happens, to get rid of the bad connection, as it otherwise keeps getting handed back to various users. (This now happens a 5-10 times a day with only a small number of simultaneous users, so it is fairly annoying.) But if anyone has any further ideas, we are still looking out for a real solution...
I would say it's MARS rather then pooling
From "Using Multiple Active Result Sets (MARS)"
Applications can have multiple default
result sets open and can interleave
reading from them.
Applications can
execute other statements (for example,
INSERT, UPDATE, DELETE, and stored
procedure calls) while default result
sets are open.
Connection pooling in it's basic form means the connection open/close overhead is minimised, but any connection (until MARS) has one thing going on at any one time. Pooling has been around for some time and just works out of the box.
MARS (I've not used it BTW) introduces overlapping "stuff" going on for any single connection. So it's probably MARS rather than connection pooling is the bigger culprit of the 2.
From "Extending Database Impersonation by Using EXECUTE AS"
When impersonating a principal by
using the EXECUTE AS LOGIN statement,
or within a server-scoped module by
using the EXECUTE AS clause, the scope
of the impersonation is server-wide.
This may explain why MARS is causing it: the same principal in 2 session both running EXECUTE AS.
There may be something in that article of use, or try this:
IF ORIGINAL_LOGIN() = SUSER_SNAME() EXECUTE AS LOGIN = {0};
On reflection and after reading for this answer, I've not convinced that trying to change execution context for each session (MARS) in one connections is a good idea...
Don't blame connection pooling - MARS is quite notorious for wreaking a havoc. It's not entirely it's blame but it's kind of half and half. The key thing to remember is that MARS is designed, and only works with "normal" DB use (meaning, regular CRUD stuff no admin batches). Any commands that have a wide effect on DB engine can trip MARS even if it's just one connection and single threaded (like running a setup batch to create tables or a nested transaction).
Having said that, one can easily just blame MARS, but it works perfecly fine for normal CRUD scenarios which are like 99% (and things with low efficiencey like ORM-s and LINQ depend on it for life). Meaning that it's important for people to learn that if they want to hack SQL through a connection they can't use MARS. For example I had a setup code that was creating whole DB from scratch, beceuse it's very convenient for deployment, but it was sharing connection sting with web service it was deploying - oops :-) Took me a few days of digging to learn my lesson. So I just maintain the separation of concerns (which is always good) and problems went away.
Have you tried to use a revert at the end of your sql statement?
http://msdn.microsoft.com/en-us/library/ms178632.aspx
I always do this to just make sure the current context is back to normal.
In developing a relatively simple web service, that takes the data provided by a post and records it in a database table, we're getting this error:
Exception caught: The remote server returned an error: (500) Internal Server Er
or.
Stack trace: at System.Net.HttpWebRequest.GetResponse()
on some servers, but no others. The ones that are getting this are the physical machines, the others are virtual, and obviously the physical servers are far more powerful.
As far as we can tell, the problem is that the DB connections aren't being released back to the pools after each query. I'm using the using pattern below:
using (VoteDaoDataContext dao = new VoteDaoDataContext())
{
dao.insert_response_and_update_count(answerVal, swid, agent, geo, DateTime.Now, ip);
dao.SubmitChanges();
msg += "Thank you for your vote.";
dao.Dispose();
}
I added the dao.Dispose() call to ensure that connections are released when the method finishes, but I don't know whether or not it's necessary.
Am I using this pattern correctly? Is there something else I need to do to ensure that connections get returned to the pools correctly?
Thanks!
Your diagnostic information is not good enough. An HTTP/500 isn't enough detail to really tell if your theory is correct. You're going to need to capture a full stack trace in your logging if you want to get to the problem. I think you've jumped to a conclusion here. And no, you do not need that Dispose() before the end of your using{} block. That's what using{} does.
I thought that dispose() call was redundant, but I wanted to be sure.
We're seeing the connection pools saturating in the SQL logs (I can't look at the directly, I'm just a developer, and this stuff's running in a prod environment), and my ops guy said he's seeing connections timing out... and once they time out, the server starts running again, until the next time it saturates the connection pool.
We're going through the process of tweaking the connection pool settings at the moment... I wanted to be certain that I wasn't doing anything wrong, since this is my first time using Linq.
Thanks!