We seem to have come up on a weird issue, where two concurrent requests to our service are actually using the same DB connection.
Our setup is ServiceStack + NHibernate + FluentNHibernate + MySQL. I have set up a small test that recreates the problem:
public class AppHost : AppHostBase
{
private ISessionFactory _sessionFactory;
public AppHost() : base("Lala Service", typeof(AppHost).Assembly)
{
}
public override void Configure(Container container)
{
_sessionFactory = Fluently.Configure()
.Database(MySQLConfiguration.Standard.ConnectionString(conn =>
conn.Server("localhost").Username("lala").Password("lala").Database("lala")))
.Mappings(mappings => mappings.AutoMappings.Add(
AutoMap.Assembly(GetType().Assembly).Where(t => t == typeof(Lala))
.Conventions.Add(DefaultLazy.Never(), DefaultCascade.All())))
.BuildSessionFactory();
container.Register(c => _sessionFactory.OpenSession()).ReusedWithin(ReuseScope.Request);
}
}
public class Lala
{
public int ID { get; set; }
public string Name { get; set; }
}
[Route("/lala")]
public class LalaRequest
{
}
public class LalaReseponse
{
}
public class LalaService : Service
{
private ISession _session;
public ISession Session1
{
get { return _session; }
set { _session = value; }
}
public LalaReseponse Get(LalaRequest request)
{
var lala = new Lala
{
Name = Guid.NewGuid().ToString()
};
_session.Persist(lala);
_session.Flush();
lala.Name += " XXX";
_session.Flush();
return new LalaReseponse();
}
}
The I hit this service 10 times concurrenly via Ajax like so:
<script type="text/javascript">
for (i = 0; i < 10; i++) {
console.log("aa");
$.ajax({
url: '/lala',
dataType: 'json',
cache: false
});
}
</script>
The result is consistenly:
Number of connections open < 10.
Not all records updated.
On occasion - a StaleObjectStateException thrown - if I delete records.
The reason behind this is that the connections are reused by two concurrent requests, and then LAST_INSERT_ID() gives the ID of the wrong row, so two requests are updating the same row.
In short: it's a complete mess and it's clearly sharing the DB connection between requests.
The question is: Why? How should I have configured things so that each request gets its own connection from the connection pool?
Finally solved it, what a day-waster!
The source of the problem is NHibernate's connection release mode:
11.7. Connection Release Modes
The legacy (1.0.x) behavior of NHibernate in regards to ADO.NET
connection management was that a ISession would obtain a connection
when it was first needed and then hold unto that connection until the
session was closed. NHibernate introduced the notion of connection
release modes to tell a session how to handle its ADO.NET connections.
...
The different release modes are identified by the enumerated values of
NHibernate.ConnectionReleaseMode:
OnClose - is essentially the legacy behavior described above. The
NHibernate session obtains a connection when it first needs to perform
some database access and holds unto that connection until the session
is closed.
AfterTransaction - says to release connections after a
NHibernate.ITransaction has completed.
The configuration parameter hibernate.connection.release_mode is used
to specify which release mode to use.
...
after_transaction - says to use
ConnectionReleaseMode.AfterTransaction. Note that with
ConnectionReleaseMode.AfterTransaction, if a session is considered to
be in auto-commit mode (i.e. no transaction was started) connections
will be released after every operation.
This got entangled together with MySQL .NET/Connector's default connection pooling, and effectively meant that the connections were swapped between concurrent requests, as one request released the connection back to the pool and the other acquired it.
However, I think that the fact that NHibernate calls LAST_INSERT_ID() after releasing and re-acquiring the connection is a bug. It should call LAST_INSERT_ID() inside the same "operation".
Anyway, solutions:
Use transactions, which is what we normally do, or
If you can't or don't want to use transactions in a certain context for some reason (which is what happened to use today), set the connection release mode to "on close". With FluentNHibernate that would be:
.ExposeConfiguration(cfg =>
cfg.SetProperty("connection.release_mode", "on_close"));
And from here on the connection is bound to the session even if there is no transaction.
Related
In the old API (1.X) you could tell whether the server was connected or not by using the State property on the MongoServer instance returned from MongoClient.GetServer:
public bool IsConnceted
{
get
{
return _client.GetServer().State == MongoServerState.Connected;
}
}
However GetServer is not a part of the new API (2.0). How can that be achieved?
The more appropriate way to do that is not by checking the server but rather the cluster (which may contain multiple servers) and you can access it directly from the MongoClient instance:
public bool IsClusterConnceted
{
get
{
return _client.Cluster.Description.State == ClusterState.Connected;
}
}
If you would like to check a specific server that's also possible:
public bool IsServerConnceted
{
get
{
return _client.Cluster.Description.Servers.Single().State == ServerState.Connected;
}
}
Keep in mind that the value is updated by the last operation so it may not be current. The only way to actually make sure there's a valid connection is to execute some kind of operation.
As noted by i3arnon, one has to perform some sort of operation on the database before the state is updated properly.
The act of enumerating the databases is sufficient to update the state.
This worked for me:
var databases = _client.ListDatabasesAsync().Result;
databases.MoveNextAsync(); // Force MongoDB to connect to the database.
if (_client.Cluster.Description.State == ClusterState.Connected)
{
// Database is connected.
}
I'm moving all of my existing Azure In-Role cache use to Redis and decided to use the Azure Redis preview along with the StackExchange.Redis library (https://github.com/StackExchange/StackExchange.Redis). I wrote all the code for it without much problem, but when running it is absolutely unusably slow and constantly throws timeout errors (my timeout period is set to 15 seconds).
Here is the relevant code for how I am setting up the Redis connection and using it for simple operations:
private static ConnectionMultiplexer _cacheService;
private static IDatabase _database;
private static object _lock = new object();
private void Initialize()
{
if (_cacheService == null)
{
lock (_lock)
{
if (_cacheService == null)
{
var options = new ConfigurationOptions();
options.EndPoints.Add("{my url}", 6380);
options.Ssl = true;
options.Password = "my password";
// needed for FLUSHDB command
options.AllowAdmin = true;
// necessary?
options.KeepAlive = 30;
options.ConnectTimeout = 15000;
options.SyncTimeout = 15000;
int database = 0;
_cacheService = ConnectionMultiplexer.Connect(options);
_database = _cacheService.GetDatabase(database);
}
}
}
}
public void Set(string key, object data, TimeSpan? expiry = null)
{
if (_database != null)
{
_database.Set(key, data, expiry: expiry);
}
}
public object Get(string key)
{
if (_database != null)
{
return _database.Get(key);
}
return null;
}
Performing very simple commands like Get and Set often time out or take 5-10 seconds to complete. Seems like it kind of negates the whole purpose of using it as a cache if it's WAY slower than actually fetching the real data from my database :)
Am I doing anything obviously incorrect?
Edit: here are some stats that I pulled from the server (using Redis Desktop Manager) in case that sheds some light on anything.
Server
redis_version:2.8.12
redis_mode:standalone
os:Windows
arch_bits:64
multiplexing_api:winsock_IOCP
gcc_version:0.0.0
process_id:2876
tcp_port:6379
uptime_in_seconds:109909
uptime_in_days:1
hz:10
lru_clock:16072421
config_file:C:\Resources\directory\xxxx.Kernel.localStore\1\redis_2092_port6379.conf
Clients
connected_clients:5
client_longest_output_list:0
client_biggest_input_buf:0
client_total_writes_outstanding:0
client_total_sent_bytes_outstanding:0
blocked_clients:0
Memory
used_memory:4256488
used_memory_human:4.06M
used_memory_rss:67108864
used_memory_rss_human:64.00M
used_memory_peak:5469760
used_memory_peak_human:5.22M
used_memory_lua:33792
mem_fragmentation_ratio:15.77
mem_allocator:dlmalloc-2.8
Persistence
loading:0
rdb_changes_since_last_save:72465
rdb_bgsave_in_progress:0
rdb_last_save_time:1408471440
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
Stats
total_connections_received:25266
total_commands_processed:123389
instantaneous_ops_per_sec:10
bytes_received_per_sec:275
bytes_sent_per_sec:65
bytes_received_per_sec_human:
Edit 2: Here are the extension methods I'm using for Get/Set -- they are very simple methods that just turn an object into JSON and call StringSet.
public static object Get(this IDatabase cache, string key)
{
return DeserializeJson<object>(cache.StringGet(key));
}
public static void Set(this IDatabase cache, string key, object value, TimeSpan? expiry = null)
{
cache.StringSet(key, SerializeJson(value), expiry: expiry);
}
Edit 3: here are a couple example error messages:
A first chance exception of type 'System.TimeoutException' occurred in StackExchange.Redis.dll
Timeout performing GET MyCachedList, inst: 11, queue: 1, qu=1, qs=0, qc=0, wr=0/1, in=0/0
A first chance exception of type 'System.TimeoutException' occurred in StackExchange.Redis.dll
Timeout performing GET MyCachedList, inst: 1, queue: 97, qu=0, qs=97, qc=0, wr=0/0, in=3568/0
Here is the recommended pattern, from the Azure Redis Cache documentation:
private static Lazy<ConnectionMultiplexer> lazyConnection = new Lazy<ConnectionMultiplexer>(() => {
return ConnectionMultiplexer.Connect("mycache.redis.cache.windows.net,abortConnect=false,ssl=true,password=...");
});
public static ConnectionMultiplexer Connection {
get {
return lazyConnection.Value;
}
}
A few important points:
It uses Lazy<T> to handle thread-safe initialization
It sets "abortConnect=false", which means if the initial connect attempt fails, the ConnectionMultiplexer will silently retry in the background rather than throw an exception.
It does not check the IsConnected property, since ConnectionMultiplexer will automatically retry in the background if the connection is dropped.
I was having similar issues. Redis cache was unusually slow but was definitely caching. In some cases, it took 20-40 seconds to load a page.
I realized that the cache server was in a different location than the site's. I updated the cache server to live in the same location as the website and now everything works as expected.
That same page now loads in 4-6 seconds.
Good luck to anyone else who's having these issues.
The problem is how the connection object created and used. we faced exact problem initially and fixed with a single connection object getting used across all web requests. And we check is it null or connected in session start for graceful re creating object. that fixed the issue.
Note: Also check in which Zone of Azure your Redis Cache instance is
running and Which Zone your Web Server exist. It is better to maintain
both in Same Zone
In Global.ascx.cs file
public static ConnectionMultiplexer RedisConnection;
public static IDatabase RedisCacheDb;
protected void Session_Start(object sender, EventArgs e)
{
if (ConfigurationManager.ConnectionStrings["RedisCache"] != null)
{
if (RedisConnection == null || !RedisConnection.IsConnected)
{
RedisConnection = ConnectionMultiplexer.Connect(ConfigurationManager.ConnectionStrings["RedisCache"].ConnectionString);
}
RedisCacheDb = RedisConnection.GetDatabase();
}
}
It worked in my case. Don't forget to increase the SyncTimeout. The default is 1 second.
private static Lazy<ConnectionMultiplexer> ConnectionMultiplexerItem = new Lazy<ConnectionMultiplexer>(() =>
{
var redisConfig = ConfigurationOptions.Parse("mycache.redis.cache.windows.net,abortConnect=false,ssl=true,password=...");
redisConfig.SyncTimeout = 3000;
return ConnectionMultiplexer.Connect(redisConfig);
});
Check if you your Azure Redis Cache and the Client in the same region in Azure. For example, you might be getting timeouts when your cache is in East US but the client is in West US and the request doesn't complete in synctimeout time or you might be getting timeouts when you are debugging from your local development machinex.
It’s highly recommended to have the cache and in the client in the same Azure region. If you have a scenario to do a cross region calls, you would want to set the synctimeout to a higher value.
Read more:
https://azure.microsoft.com/en-us/blog/investigating-timeout-exceptions-in-stackexchange-redis-for-azure-redis-cache/
In our case the issue is when using SSL connection. You're showing that your desktop manager is running on the non-SSL port, but your code is using SSL.
A quick benchmark on our Azure redis without SSL, retrieving around 80k values with an LRANGE command (also with .net and StackExchange.Redis) is basically instand. When SSL is used, the same query takes 27 seconds.
WebApp: Standard S2
Redis: Standard 1 GB
Edit: Checking the SLOWLOG, Redis itself seems to actually hit slowlog with it's 14ms time it takes or so to grab the rows, but this is far from the actual transfer with SSL enabled. We ended up with a premium Redis to have some sort of security between Redis and Web Apps.
I have an old ObjectContext and few new DbContext in my project (i.e. BoundedContext for different purposes).
Some time I need to commit changes from few of them in one transactions. In some cases I need to persist data from ObjectContext and DbContext.
In EF 5.0 to avoid of MSDC I write some wraper
public class ContextUnitOfWork
{
List<IContext> ContextList;
public ContextUnitOfWork()
{
ContextList = new List<IContext>();
}
public void RegisterContext(IContext Context)
{
ContextList.Add(Context);
}
public bool IsDisposed
{
get
{
return ContextList.Any(x => x.IsDisposed);
}
}
public bool HasChangedEntities
{
get
{
return ContextList.Any(x => x.HasChangedEntities);
}
}
public void Commit()
{
bool HasDbContext = ContextList.OfType<System.Data.Entity.DbContext>().Any();
try
{
if (HasDbContext)
{
ContextList.ForEach(x =>
{
if (x is System.Data.Entity.DbContext)
{
(x as System.Data.Entity.DbContext).Database.Connection.Open();
}
else if (x is System.Data.Objects.ObjectContext)
{
((System.Data.Objects.ObjectContext)x).Connection.Open();
}
});
}
using (var scope = new System.Transactions.TransactionScope(System.Transactions.TransactionScopeOption.Required,
new System.Transactions.TransactionOptions { IsolationLevel = System.Transactions.IsolationLevel.ReadCommitted }))
{
ContextList.ForEach(x => x.Commit());
scope.Complete();
}
}
catch (System.Data.UpdateException uex)
{
var ErrorList = uex.StateEntries.Select(x => x.Entity).ToList();
}
finally
{
if (HasDbContext)
{
ContextList.ForEach(x =>
{
if (x is System.Data.Entity.DbContext)
{
(x as System.Data.Entity.DbContext).Database.Connection.Close();
}
else if (x is System.Data.Objects.ObjectContext)
{
((System.Data.Objects.ObjectContext)x).Connection.Close();
}
});
};
}
}
}
But in EntityFramework 6.0.1 it doesn't work. ObjectContext commit successfully, but when DbContext call SaveChanges() an Exception of type EntityException with text
"The underlying provider failed on EnlistTransaction." And Inner Expection contains {"Network access for Distributed Transaction Manager (MSDTC) has been disabled. Please enable DTC for network access in the security configuration for MSDTC using the Component Services Administrative tool."}
Any Idea to commit contexts in one transaction and avoid MDTC exception?
You are attempting to run everything in a local transaction which is very tricky even with multiple contexts of the same type. The reason for this is that you cannot have multiple open connections with the same local transaction. And very often a new connection will be opened for the next context if the previous context is still alive. This will trigger a promotion of the local transaction to a distributed transaction.
My experience with EF is that it only re-uses the current connection when the connectionstring (the normal one inside the entityconnectionstring) is EXACTLY identical. If there is a single difference, the transaction will be promoted to a distributed transaction, which must be enabled by the system, which in your case, it is not.
Also, if you are already executing a query, and are still reading results from that query, that starting another query at the same time, will (of course) require another connection, and therefore, the local transaction will be promoted to a distributed transaction.
Can you check if the connection strings are identical? I would still be surprised if current connection is re-used though.
Does anyone know of any cases when using a transaction scope the transaction is escalated to the DTC when multiple connections are NOT open.
I am aware that if I open multiple connections(no matter what connection string) within a transaction scope that the transaction will most likely be promoted to the DTC.
Knowing this I have gone to great lengths to make sure there is only ONE connection opened within my transactions.
However, I have a client where they are getting the exception
An error has occurred. Csla.DataPortalException: DataPortal.Update failed (The underlying provider failed on Open.) ---> Csla.Reflection.CallMethodException: EditableCategory.DataPortal_Update method call failed ---> System.Data.EntityException: The underlying provider failed on Open. ---> System.Transactions.TransactionManagerCommunicationException: Network access for Distributed Transaction Manager (MSDTC) has been disabled. Please enable DTC for network access in the security configuration for MSDTC using the Component Services Administrative tool. ---> System.Runtime.InteropServices.COMException: The transaction manager has disabled its support for remote/network transactions.
Again, I am pretty sure there is only one connection opened within the scope. Take a look.
protected override void DataPortal_Update()
{
using (System.Transactions.TransactionScope ts = new System.Transactions.TransactionScope(System.Transactions.TransactionScopeOption.Required, System.Transactions.TransactionManager.MaximumTimeout))
{
//get the dal manager he knows which dal implementation to use
using (var dalMgr = DataAccess.BusinessObjectsDalFactory.GetManager())
{
//get the category dal implementation
var ecDal = dalMgr.GetProvider<DataAccess.BusinessObjectDalInterfaces.ICategoryDAL>();
//assume all the data is good at this point so use bypassproperty checks
using (BypassPropertyChecks)
{
var catData = new Models.Category { CategoryId = CategoryId, CategoryName = CategoryName, LastChanged = TimeStamp };
ecDal.UpdateCategory(catData);
TimeStamp = catData.LastChanged;
}
}
ts.Complete();
}
base.DataPortal_Update();
}
public class DalManager : Core.Sebring.DataAccess.IBusinessObjectsDalManager {private static string _typeMask = typeof(DalManager).FullName.Replace("DalManager", #"{0}");
public T GetProvider<T>() where T : class
{
var typeName = string.Format(_typeMask, typeof(T).Name.Substring(1));
var type = Type.GetType(typeName);
if (type != null)
return Activator.CreateInstance(type) as T;
else
throw new NotImplementedException(typeName);
}
public Csla.Data.DbContextManager<DataContext> ConnectionManager { get; private set; }
public DalManager()
{
ConnectionManager = Csla.Data.DbContextManager<DataContext>.GetManager();
}
public void Dispose()
{
ConnectionManager.Dispose();
ConnectionManager = null;
}
public void UpdateDataBase()
{
DatabaseUpgrader.PerformUpgrade();
}
}
public void UpdateCategory(Models.Category catData)
{
if (catData == null) return;
using (var cntx = DbContextManager<DataContext>.GetManager())
{
var cat = cntx.DbContext.Set<Category>().FirstOrDefault(c => c.CategoryId == catData.CategoryId);
if (cat == null) return;
if (!cat.LastChanged.Matches(catData.LastChanged))
throw new ConcurrencyException(cat.GetType().ToString());
cat.CategoryName = catData.CategoryName;
//cntx.DbContext.ChangeTracker.DetectChanges();
cntx.DbContext.Entry<Category>(cat).State = System.Data.EntityState.Modified;
cntx.DbContext.SaveChanges();
catData.LastChanged = cat.LastChanged;
}
}
The code for DBContextManager is available, but in short it just makes certain there is only one DBContext, and hence one connection opened. Am I overlooking something? I guess its possible that something is up with DBConextManager, so I have posted in the CSLA forums as well(DBContextManager is a part of CSLA). But has anyone run into scenarios where they are sure one connection is opened within the transaction scope and the transaction is escalated to the DTC?
Of course I cannot reproduce the exception on my local dev machine OR any of our QA machines.
Any help is appreciated.
Thanks.
Entity Framework can randomly try to open a new connection when doing transactions with System.Transactions.TransactionScope
Try adding a finally statement and dispose your transaction, also call your dbContext and manually close the connection , this will lesser the ammount of times the transaction gets escalated but it might still happen:
finally
{
cntx.Database.Connection.Close();
transaction.Dispose();
}
Its a known "bug" you can find more here :
http://petermeinl.wordpress.com/2011/03/13/avoiding-unwanted-escalation-to-distributed-transactions/
I have a class Worker which sends emails periodically,I start in Global.asax.cs on App_start()
public static class Worker
{
public static void Start()
{
ThreadPool.QueueUserWorkItem(o => Work());
}
public static void Work()
{
var r = new DbContext();
var m = new MailSender(new SmtpServerConfig());
while (true)
{
Thread.Sleep(600000);
try
{
var d = DateTime.Now.AddMinutes(-10);
var ns = r.Set<Notification>().Where(o => o.SendEmail && !o.IsRead && o.Date < d);
foreach (var n in ns)
{
m.SendEmailAsync("noreply#example.com", n.Email, NotifyMailTitle(n) + " - forums", NotifyMailBody(n));
n.SendEmail = false;
}
r.SaveChanges();
}
catch (Exception ex)
{
ex.Raize();
}
}
}
}
So I keep this dbcontext alive for the entire lifetime of the application is this a good practice ?
DbContext is a very light-weight object.
It doesn't matter whether your DbContext stays alive or you instantiate it just before making the call because the actual DB Connection only opens when you SubmitChanges or Enumerate the query (in that case it is closed on end of enumeration).
In your specific case. It doesn't matter at all.
Read Linq DataContext and Dispose for details on this.
I would wrap it in a using statement inside of Work and let the database connection pool do it's thing:
using (DbContext r = new DbContext())
{
//working
}
NOTE: I am not 100% sure how DbContext handles the db connections, I am assuming it opens one.
It is not good practice to keep a database connection 'alive' for the lifetime of an application. You should use a connection when needed and close it via the API(using statement will take care of that for you). The database connection pool will actually open and close connections based on connection demands.
I agree with #rick schott that you should instantiate the DbContext when you need to use it rather than keep it around for the lifetime of the application. For more information, see Working with Objects (Entity Framework 4.1), especially the section on Lifetime:
When working with long-running context consider the following:
As you load more objects and their references into memory, the
memory consumption of the context may increase rapidly. This may cause
performance issues.
If an exception causes the context to be in an unrecoverable state,
the whole application may terminate.