Develop Reference

Replacement for infinite loop

In my console application I have a couple of classes (let's call them MyClass1, MyClass2, ...) having a method that should check the existence of certain records in the database (different classes wait for different records) and return only when the needed records exist. I currently have a simple implementation using an infinite loop and Thread.Sleep. This approach does work, but it tends to cause a high CPU load. What is the way to make these methods more CPU-friendly?
public override void WaitForRecord()
{
MyDatabaseRecord record = null;
while (record == null)
{
Thread.Sleep(500);
using (var dc = new MyDataContext())
{
record = dc.Documents
.Where( /*condition*/)
.SingleOrDefault();
}
}
Logger.Info("Record with ID " + record.Id + " found at " + DateTime.Now)
}
The usage of these methods is pretty straightforward: the calling code creates a bunch of objects, launches each object's WaitForRecord method using Task.Factory.StartNew, periodically checks whether any tasks have finished execution, and prints the results in the console like this:
MyClass1 is still waiting for record...
MyClass2 has found the record...
...

Assuming that you're connecting to a SQL (2005 or greater) database, you could look into SqlDependency. Here is an article on code project about SqlDependency and EF:
http://www.codeproject.com/Articles/496484/SqlDependency-with-Entity-Framework

Using SQL Server application locks to solve locking requirements

I have a large application based on Dynamics CRM 2011 that in various places has code that must query for a record based upon some criteria and create it if it doesn't exist else update it.
An example of the kind of thing I am talking about would be similar to this:
stk_balance record = context.stk_balanceSet.FirstOrDefault(x => x.stk_key == id);
if(record == null)
{
record = new stk_balance();
record.Id = Guid.NewGuid();
record.stk_value = 100;
context.AddObject(record);
}
else
{
record.stk_value += 100;
context.UpdateObject(record);
}
context.SaveChanges();
In terms of CRM 2011 implementation (although not strictly relevant to this question) the code could be triggered from synchronous or asynchronous plugins. The issue is that the code is not thread safe, between checking if the record exists and creating it if it doesn't, another thread could come in and do the same thing first resulting in duplicate records.
Normal locking methods are not reliable due to the architecture of the system, various services using multiple threads could all be using the same code, and these multiple services are also load balanced across multiple machines.
In trying to find a solution to this problem that doesn't add massive amounts of extra complexity and doesn't compromise the idea of not having a single point of failure or a single point where a bottleneck could occur I came across the idea of using SQL Server application locks.
I came up with the following class:
public class SQLLock : IDisposable
{
//Lock constants
private const string _lockMode = "Exclusive";
private const string _lockOwner = "Transaction";
private const string _lockDbPrincipal = "public";
//Variable for storing the connection passed to the constructor
private SqlConnection _connection;
//Variable for storing the name of the Application Lock created in SQL
private string _lockName;
//Variable for storing the timeout value of the lock
private int _lockTimeout;
//Variable for storing the SQL Transaction containing the lock
private SqlTransaction _transaction;
//Variable for storing if the lock was created ok
private bool _lockCreated = false;
public SQLLock (string lockName, int lockTimeout = 180000)
{
_connection = Connection.GetMasterDbConnection();
_lockName = lockName;
_lockTimeout = lockTimeout;
//Create the Application Lock
CreateLock();
}
public void Dispose()
{
//Release the Application Lock if it was created
if (_lockCreated)
{
ReleaseLock();
}
_connection.Close();
_connection.Dispose();
}
private void CreateLock()
{
_transaction = _connection.BeginTransaction();
using (SqlCommand createCmd = _connection.CreateCommand())
{
createCmd.Transaction = _transaction;
createCmd.CommandType = System.Data.CommandType.Text;
StringBuilder sbCreateCommand = new StringBuilder();
sbCreateCommand.AppendLine("DECLARE #res INT");
sbCreateCommand.AppendLine("EXEC #res = sp_getapplock");
sbCreateCommand.Append("#Resource = '").Append(_lockName).AppendLine("',");
sbCreateCommand.Append("#LockMode = '").Append(_lockMode).AppendLine("',");
sbCreateCommand.Append("#LockOwner = '").Append(_lockOwner).AppendLine("',");
sbCreateCommand.Append("#LockTimeout = ").Append(_lockTimeout).AppendLine(",");
sbCreateCommand.Append("#DbPrincipal = '").Append(_lockDbPrincipal).AppendLine("'");
sbCreateCommand.AppendLine("IF #res NOT IN (0, 1)");
sbCreateCommand.AppendLine("BEGIN");
sbCreateCommand.AppendLine("RAISERROR ( 'Unable to acquire Lock', 16, 1 )");
sbCreateCommand.AppendLine("END");
createCmd.CommandText = sbCreateCommand.ToString();
try
{
createCmd.ExecuteNonQuery();
_lockCreated = true;
}
catch (Exception ex)
{
_transaction.Rollback();
throw new Exception(string.Format("Unable to get SQL Application Lock on '{0}'", _lockName), ex);
}
}
}
private void ReleaseLock()
{
using (SqlCommand releaseCmd = _connection.CreateCommand())
{
releaseCmd.Transaction = _transaction;
releaseCmd.CommandType = System.Data.CommandType.StoredProcedure;
releaseCmd.CommandText = "sp_releaseapplock";
releaseCmd.Parameters.AddWithValue("#Resource", _lockName);
releaseCmd.Parameters.AddWithValue("#LockOwner", _lockOwner);
releaseCmd.Parameters.AddWithValue("#DbPrincipal", _lockDbPrincipal);
try
{
releaseCmd.ExecuteNonQuery();
}
catch {}
}
_transaction.Commit();
}
}
I would use this in my code to create a SQL Server application lock using the unique key I am querying for as the lock name like this
using (var sqlLock = new SQLLock(id))
{
//Code to check for and create or update record here
}
Now this approach seems to work, however I am by no means any kind of SQL Server expert and am wary about putting this anywhere near production code.
My question really has 3 parts
1. Is this a really bad idea because of something I haven't considered?
Are SQL Server application locks completely unsuitable for this purpose?
Is there a maximum number of application locks (with different names) you can have at a time?
Are there performance considerations if a potentially large number of locks are created?
What else could be an issue with the general approach?
2. Is the solution actually implemented above any good?
If SQL Server application locks are usable like this, have I actually used them properly?
Is there a better way of using SQL Server to achieve the same result?
In the code above I am getting a connection to the Master database and creating the locks in there. Does that potentially cause other issues? Should I create the locks in a different database?
3. Is there a completely alternative approach that could be used that doesn't use SQL Server application locks?
I can't use stored procedures to create and update the record (unsupported in CRM 2011).
I don't want to add a single point of failure.

You can do this much easier.
//make sure your plugin runs within a transaction, this is the case for stage 20 and 40
//you can check this with IExecutionContext.IsInTransaction
//works not with offline plugins but works within CRM Online (Cloud) and its fully supported
//also works on transaction rollback
var lockUpdateEntity = new dummy_lock_entity(); //simple technical entity with as many rows as different lock barriers you need
lockUpdateEntity.Id = Guid.parse("well known guid"); //well known guid for this barrier
lockUpdateEntity.dummy_field=Guid.NewGuid(); //just update/change a field to create a lock, no matter of its content
//--------------- this is untested by me, i use the next one
context.UpdateObject(lockUpdateEntity);
context.SaveChanges();
//---------------
//OR
//--------------- i use this one, but you need a reference to your OrganizationService
OrganizationService.Update(lockUpdateEntity);
//---------------
//threads wait here if they have no lock for dummy_lock_entity with "well known guid"
stk_balance record = context.stk_balanceSet.FirstOrDefault(x => x.stk_key == id);
if(record == null)
{
record = new stk_balance();
//record.Id = Guid.NewGuid(); //not needed
record.stk_value = 100;
context.AddObject(record);
}
else
{
record.stk_value += 100;
context.UpdateObject(record);
}
context.SaveChanges();
//let the pipeline flow and the transaction complete ...
For more background info refer to http://www.crmsoftwareblog.com/2012/01/implementing-robust-microsoft-dynamics-crm-2011-auto-numbering-using-transactions/

LinQ DataContext - The operation cannot be performed during a call to SubmitChanges

The code which is throwing exception is extremely easy - this is very regular insert and then submit changes statement which looks:
context.tb_dayErrorLog.InsertOnSubmit(data);
context.SubmitChanges();
So really nothing special. This statement is executed about 50 thousands times a day without any problem, but:
about 6 - 10 times a day it finishes with:
The operation cannot be performed during a call to SubmitChanges.
StackTrace: at System.Data.Linq.DataContext.CheckNotInSubmitChanges()
at System.Data.Linq.Table`1.InsertOnSubmit(TEntity entity)
I was trying to find out what that can be but can't find a clue
This behavior is very not deterministic politely saying - how it can finish 50k times correctly and few times not?
DataContext was firstly initialized as a static one, and then reused for all the calls, so I was thinking maybe that's the problem. Then I changed it to be initialized with every call but results are quite similar. Still few exceptions a day.
Any idea?
some additions:
function looks like:
public override bool Log(ErrorLogData logData)
{
try
{
logData.ProcessID = _processID;
//Create new log dataset
var data = new DataRecord
{
application = logData.Application,
date = DateTime.Now,
Other = logData.Other,
process = logData.ProcessName,
processid = logData.ProcessID,
severity = logData.Severity,
username = logData.UserName,
Type = (short)logData.ErrorType
};
var context = new DataContext(ConnectionString);
context.tb_dayErrorLog.InsertOnSubmit(data);
context.SubmitChanges();
}
catch (Exception ex)
{
//log log in eventviewer
LogEvent(logData.ToString(), ex);
return false;
}
return true;
}
so simple record initialization and then insert.
As I wrote in the comment, while making same thing by Ado.Net and SqlCommand this problem is not occuring...
So my curiosity makes me think why?

This sounds like a threading issue where you are calling Log and hence SubmitChanges on one thread when another thread is in the middle of SubmitChanges.
I suspect your DataContext is still a global static variable.
Try changing your Log method to
using (var context = new DataContext(ConnectionString))
{
context.tb_dayErrorLog.InsertOnSubmit(data);
context.SubmitChanges();
}

#SgMoore points to concurrency problem and in my case it really was. If that's the case another approach is to use lock like this:
String lockValue = "";
lock (lockValue)
{
context.tb_dayErrorLog.InsertOnSubmit(data);//UPDATE: concurrency error can occur here too
context.dc.SubmitChanges();
}

Should I keep an instance of DbContext in a separate thread that performs periodic job

I have a class Worker which sends emails periodically,I start in Global.asax.cs on App_start()
public static class Worker
{
public static void Start()
{
ThreadPool.QueueUserWorkItem(o => Work());
}
public static void Work()
{
var r = new DbContext();
var m = new MailSender(new SmtpServerConfig());
while (true)
{
Thread.Sleep(600000);
try
{
var d = DateTime.Now.AddMinutes(-10);
var ns = r.Set<Notification>().Where(o => o.SendEmail && !o.IsRead && o.Date < d);
foreach (var n in ns)
{
m.SendEmailAsync("noreply#example.com", n.Email, NotifyMailTitle(n) + " - forums", NotifyMailBody(n));
n.SendEmail = false;
}
r.SaveChanges();
}
catch (Exception ex)
{
ex.Raize();
}
}
}
}
So I keep this dbcontext alive for the entire lifetime of the application is this a good practice ?

DbContext is a very light-weight object.
It doesn't matter whether your DbContext stays alive or you instantiate it just before making the call because the actual DB Connection only opens when you SubmitChanges or Enumerate the query (in that case it is closed on end of enumeration).
In your specific case. It doesn't matter at all.
Read Linq DataContext and Dispose for details on this.

I would wrap it in a using statement inside of Work and let the database connection pool do it's thing:
using (DbContext r = new DbContext())
{
//working
}
NOTE: I am not 100% sure how DbContext handles the db connections, I am assuming it opens one.
It is not good practice to keep a database connection 'alive' for the lifetime of an application. You should use a connection when needed and close it via the API(using statement will take care of that for you). The database connection pool will actually open and close connections based on connection demands.

I agree with #rick schott that you should instantiate the DbContext when you need to use it rather than keep it around for the lifetime of the application. For more information, see Working with Objects (Entity Framework 4.1), especially the section on Lifetime:
When working with long-running context consider the following:
As you load more objects and their references into memory, the
memory consumption of the context may increase rapidly. This may cause
performance issues.
If an exception causes the context to be in an unrecoverable state,
the whole application may terminate.

Multithreaded linq2sql applications TransactionScope difficulties

I've created a file processing service which reads and imports xml files from a specific directory.
The service starts several workers which will poll a filequeue for new files and uses linq2sql for dataaccess. Each workerthread has its own datacontext.
The files being processed contain several orders and each order contains several addresses (Customer/Contractor/Subcontractor)
I've defined a transactionscope around the handling of each file. This way I want to ensure that the whole file is handled correctly, or that the whole file is rolled back when an exception occurs:
try
{
using (var tx = new TransactionScope(TransactionScopeOption.RequiresNew))
{
foreach (var order in orders)
{
HandleType1Order(order);
}
tx.Complete();
}
}
catch (SqlException ex)
{
if (ex.Number == SqlErrorNumbers.Deadlock)
{
throw new FileHandlerException("File Caused a Deadlock, retrying later", ex, true);
}
else
throw;
}
One of the requirements for the service is that is creates or updates found addresses in the xml files. So I've created an address service which is responsible for address management. The following piece of code gets executed for each order (within the method HandleType1Order()) in the xml importfile (And thus is part of the TransactionScope for the entire file).
using (var tx = new TransactionScope())
{
address = GetAddressByReference(number);
if (address != null) //address is already known
{
Log.Debug("Found address {0} - {1}. Updating...", address.Code, address.Name);
UpdateAddress(address, name, number, isContractor, isSubContractor, isCustomer);
}
else
{
//address not known, so create it
Log.Debug("Address {0} not known, creating address", number);
address = CreateAddress(name, number, sourceSystemId, isContractor, isSubContractor,
isCustomer);
_addressRepository.Save(address);
}
_addressRepository.Flush();
tx.Complete();
}
What I'm trying to do here, is to create or update an address, with the number being unique.
The method GetAddressByReference(string number) returns a known address or null when an address is not found.
public virtual Address GetAddressByReference(string reference)
{
return _addressRepository.GetAll().SingleOrDefault(a=>a.Code==reference);
}
When I run the service it however creates multiple addresses with the same number. The method GetAddressByReference() get's called concurrently and should return a known address when a second thread executes the method with the same addressnumber, however it returns null. There is propably something wrong with my transaction boundaries, or isolationlevel, but I can't seem to get it to work.
Can someone point me in the right direction? Help is much appreciated!!
p.s. I've no problem with the transactions being deadlocked and causing a rollback, the file will just be retried when a deadlock occurs.
Edit 1 Threading code:
public void Work()
{
_isRunning = true;
while (true)
{
ImportFileTask task = _queue.Dequeue(); //dequeue blocks on empty queue
if (task == null)
break; //Shutdown worker when a null task is read from the queue
IFileImporter importer = null;
try
{
using (new LockFile(task.FilePath).Acquire()) //create a filelock to sync access accross all processes to the file
{
importer = _kernel.Resolve<IFileImporter>();
Log.DebugFormat("Processing file {0}", task.FilePath);
importer.Import(task.FilePath);
Log.DebugFormat("Done Processing file {0}", task.FilePath);
}
}
catch(Exception ex)
{
Log.Fatal(
"A Fatal exception occured while handling {0} --> {1}".FormatWith(task.FilePath, ex.Message), ex);
}
finally
{
if (importer != null)
_kernel.ReleaseComponent(importer);
}
}
_isRunning = false;
}
The above method runs in all of our worker threads. It uses Castle Windsor to resolve the FileImporter, which has a transient lifestyle (thus not shared accross threads).

You didn't post your threading code, so its difficult to say what the issue is. I'm assuming you have started DTC (Distributed Transaction Coordinator)?
Are you using a ThreadPool? Are you using the "lock" keyword?
http://msdn.microsoft.com/en-us/library/c5kehkcz.aspx