In my tests I am reading a text file line by line and inserting an entity along with other related entities. The problem is when too many are inserted I receive an Out of memory exception.
In my attempt to prevent this I create a new DbContext for every 50 rows and dispose of the old one. It was my understanding that this would free up memory from the earlier entity operations, but the memory continues to climb and if the file is big enough an out of memory exception occurs. This is related to the entity code as if I remove the lines of code that adds the entity the memory stays at a consistent usage.
Below is a simplified version of my code.
public class TestClass
{
public void ImportData(byte[] fileBytes)
{
using (Stream stream = new MemoryStream(fileBytes))
{
TextFieldParser parser = new TextFieldParser(stream);
parser.TextFieldType = FieldType.Delimited;
parser.SetDelimiters(",");
while (!parser.EndOfData)
{
//Processes 50 lines creates a new DbContext each time its called
ImportBatch(parser);
}
}
}
public void ImportBatch(TextFieldParser parser)
{
using(myDbContext context = new myDbContext())
{
context.Configuration.AutoDetectChangesEnabled = false;
int batchCount = 0;
while (!parser.EndOfData && batchCount < 50)
{
string[] fields = parser.ReadFields();
//Here I call some code that will add an entity and add releated entities
//In its navigation properties
MyService.AddMyEntity(fields,myDbContext);
batchCount++;
}
myDbContext.ChangeTracker.DetectChanges();
myDbContext.SaveChanges();
}
}
}
As I am disposing and creating a new context every 50 inserts I would expect the memory usage to stay constant, but it seems to be constant for the first 2 thousands rows but after that the memory constantly climbs, unitl an OutOfMemory exception is hit.
Is there a reason why disposing of a dbContext in the following fashion would not result in the memory being released?
EDIT - added some simplified code of my add entity method
public void AddMyEntity(string[] fields, MyDbContext, myDbContext)
{
MyEntity myEntity = new MyEntity();
newRequest.InsertDate = DateTime.UtcNow;
newRequest.AmendDate = DateTime.UtcNow;
//If I remove this line the memory does not consistently climb
myDbContext.MyEntities.Add(myEntity);
foreach(string item in fields)
{
ReleatedEntity releatedEntity = new ReleatedEntity();
releatedEntity.Value = item;
newRequest.ReleatedEntities.Add(releatedEntity);
}
}
Another Edit
Turns out after more testing it is something to do with Glimpse profiler. I have included Glimpse in my project and the web config has a section similar to below.
<glimpse defaultRuntimePolicy="On" endpointBaseUri="~/Glimpse.axd">
<tabs>
<ignoredTypes>
<add type="Glimpse.Mvc.Tab.ModelBinding, Glimpse.Mvc5"/>
<add type="Glimpse.Mvc.Tab.Metadata, Glimpse.Mvc5"/>
</ignoredTypes>
</tabs>
<inspectors>
<ignoredTypes>
<add type="Glimpse.Mvc.Inspector.ModelBinderInspector, Glimpse.Mvc5"/>
</ignoredTypes>
</inspectors>
Turning defaultRuntimePolicy to Off fixed the memory leak. Still not sure why tho
Calling Dispose on an object does not necessarily free up memory. Objects are removed from memory by the garbage collector when they are no longer referenced by any live object. Calling Dispose might free up other resources (e.g. by closing an open SQL connection in the case of a DbContext), but it's only when the object is no longer referenced that it becomes a candidate for garbage collection.
There's no guarantee that the garbage collector will run at a specific point in time. Calling Dispose certainly doesn't cause it to run. Having said that, I'm surprised that it doesn't run before you run out of memory. You could force it to run with GC.Collect but that really shouldn't be necessary.
It's possible that you still have a reference to your context objects that's causing them not to be considered eligible for garbage collection. For example, you pass myDbContext to your AddEntity method in your service layer - does that store something that includes a back-reference (even indirectly) to the context?
Because whenever you call ImportBatch(parser) it creates a new DbContext. Not 1 DbContext for each 50. You may try a get property to count and give back the context to you. Something like this :
int _batchCount = 0;
public myDbContext _db;
public myDbContext Db
{
get
{
// If batchCount > 50 or _db is not created we need to create _db
if (_db == null || _batchCount > 50)
{
// If db is already created _batchcount > 50
if (_db != null)
{
_db.ChangeTracker.DetectChanges();
_db.SaveChanges();
_db.Dispose();
}
_db = new myDbContext();
_db.Configuration.AutoDetectChangesEnabled = false;
_batchCount = 0;
}
batchCount++;
return _db;
}
}
Additionally in MyService.AddMyEntity(fields); you are using the DbContext from the MyServiceclass, not the one you are creating in using line.
Related
I have code that looks like the example below. There's an explicit transaction involved because of some database tomfoolery that needs to be done via a SP, and a save changes in the middle of it all. (Exception handling, rollbacks, etc.. omitted):
void OuterMethod(MyDatbase context)
{
using(var dbTrans = context.Database.BeginTransaction())
{
// some stuff, the save puts the data where the SP can see it
Stuff(context);
context.SaveChanges();
// now some SP stuff
context.Database.ExecuteSqlCommand(#"spFoo", params);
// more stuff
MoreStuff(context);
AlmostUnrelatedCode(context);
context.SaveChanges();
dbTrans.Commit();
}
}
Right now the method AlmostUnrelatedCode() -- which is only marginally related to the process above -- needs a nice, fast, disposable read-only context 99% of the time. I have a factory that will serve me up the right kind of context when I need it. The 1% of the time it's called from the middle of that block above.
MyDatabase localReadOnlyContext;
void AlmostUnrelatedCode(MyDatabase context)
{
if ( context.Database.CurrentTransaction != null )
{
// Must use the context passed or everything deadlocks :(
localReadOnlyContext = context;
disposeContextLater = false;
}
else
{
// I just want to do this all the time
localReadOnlyContext = _contextFactory.CreateReadOptimized();
disposeContextLater = true;
}
// Do many, many things with my read-optimized context...
// The Dispose() on the class will check for disposeContextLater
}
What I'd like to do is to get rid of that transaction check, and in fact not need to pass the outer context at all if I can help it.
What I've tried:
Just ignoring what's going on in the outer transaction and using the context I generate all the time. Problem: deadlocks.
Trying to get the outermost transaction into the EF context I create with the _contextFactory. Problem: EF context constructors don't allow you to pass an existing transaction; also Database.CurrentTransaction has no setter.
Pulling the whole transaction out into a TransactionScope that wraps everything up. Problem: the method OuterMethod passes in the context, and I don't have control of the caller.
What I can't try:
Dirty reads/nolock. AlmostUnrelatedCode() needs the data as written so far.
I'd rather not:
Just keep using the outer context while inside of AlmostUnrelatedCode. AlmostUnrelatedCode deals with a lot of data trees and that context gets fat and unhappy really fast. It pollutes its context with crap really fast, and I'd rather just dispose of it when I'm done.
you can prevent the deadlocks by using one connection for multiple contexts.
example
var efConnectionString = ConfigurationManager.ConnectionStrings["SomeEntities"].ConnectionString;
// note EntityConnection, not SqlConnection
using (var conn = new EntityConnection(efConnectionString)) {
// important to prevent escalation
await conn.OpenAsync();
using (var c1 = new SomeEntities(conn, contextOwnsConnection: false)) {
//Use some stored procedures etc.
count1 = await c1.SomeEntity1.CountAsync();
}
using (var c2 = new SomeEntities(conn, contextOwnsConnection: false)) {
//Use some stored procedures etc.
count2 = await c2.SomeEntity21.CountAsync();
}
}
in your case just get the connection from the context and reuse it
context.Database.Connection
Can't you separate things done in AlmostUnrelatedCode like this:
void AlmostUnrelatedCode()
{
var context = _contextFactory.CreateReadOptimized();
AlmostUnrelatedCode(context);
context.Dispose();
}
void AlmostUnrelatedCode(MyDatabase context)
{
// Do many, many things with context...
}
Now you can call AlmostUnrelatedCode(with param) from your OuterMethod. And maybe there is even more to be separated. Consider SOLID.
I'm debugging an existing windows service (written in C#) that needs to be manually restarted every few months because it keeps eating memory.
The service is not very complicated. It requests a json file from an external server, which holds products.
Next it parses this json file into a list of products.
For each of these products it is checking if this product already exists in the database. If not it will be added if it does exists the properties will be updated.
The database is a PostgreSQL database and we use NHibernate v3.2.0 as ORM.
I've been using JetBrains DotMemory to profile the service when it runs:
The service starts and after 30s it starts doing its work. SnapShot #1 is made before the first run.
Snapshot #6 was made after the 5th run.
The other snapshots are also made after a run.
As you can see after each run the number of objects increases with approx. 60k and the memory used increases with a few MBs after every run.
Looking closer at Snapshot #6, shows the retained size is mostly used by NHibernate session objects:
Here's my OnStart code:
try
{
// Trying to fix certificate errors:
ServicePointManager.ServerCertificateValidationCallback += delegate
{
_logger.Debug("Cert validation work around");
return true;
};
_timer = new Timer(_interval)
{
AutoReset = false // makes it fire only once, restart when work is done to prevent multiple runs
};
_timer.Elapsed += DoServiceWork;
_timer.Start();
}
catch (Exception ex)
{
_logger.Error("Exception in OnStart: " + ex.Message, ex);
}
And my DoServiceWork:
try
{
// Call execute
var processor = new SAPProductProcessor();
processor.Execute();
}
catch (Exception ex)
{
_logger.Error("Error in DoServiceWork", ex);
}
finally
{
// Next round:
_timer.Start();
}
In SAPProductProcessor I use two db calls. Both in a loop.
I loop through all products from the JSON file and check if the product is already in the table using the product code:
ProductDto dto;
using (var session = SessionFactory.OpenSession())
{
using (var transaction = session.BeginTransaction(IsolationLevel.ReadCommitted))
{
var criteria = session.CreateCriteria<ProductDto>();
criteria.Add(Restrictions.Eq("Code", code));
dto = criteria.UniqueResult<ProductDto>();
transaction.Commit();
}
}
return dto;
And when the productDto is updated I save it using:
using (var session = SessionFactory.OpenSession())
{
using (var transaction = session.BeginTransaction(IsolationLevel.ReadCommitted))
{
session.SaveOrUpdate(item);
transaction.Commit();
}
}
I'm not sure how to change the code above to stop increasing the memory and the number of object.
I already tried using var session = SessionFactory.GetCurrentSession(); instead of using (var session = SessionFactory.OpenSession()) but that didn't stop the increase of memory.
Update
In the constructor of my data access class MultiSessionFactoryProvider sessionFactoryProvider is injected. And the base class is called with : base(sessionFactoryProvider.GetFactory("data")). This base class has a method BeginSession:
ISession session = _sessionFactory.GetCurrentSession();
if (session == null)
{
session = _sessionFactory.OpenSession();
ThreadLocalSessionContext.Bind(session);
}
And a EndSession:
ISession session = ThreadLocalSessionContext.Unbind(_sessionFactory);
if (session != null)
{
session.Close();
}
In my data access class I call base.BeginSession at the start and base.EndSession at then end.
The suggestion about the Singleton made me have a closer look at my data access class.
I thought when creating this class with every run would free the NHibernate memory when it runs out of scope. I even added some dispose call in the class' destructor. But that didn't work, or more likely I'm not doing it correctly.
I now save my data access class in a static field and re-use it. Now my memory doesn't increase anymore and more important the number of open objects stay the same. I just run the service using DotMemory again for over an hour calling the run around 150 times and the memory of the last snapshot is still around 105MB and the number of object is still 117k and my SessionFactory dictionary is now just 4MB instead of 150*4MB.
We are running a very simple function in a console application that loops through databases and lists a table into a variable. Of course originally it did more but we stripped it down to just listing 1 table.
We noticed that on every new creation of ObjectContext the memory grows by about 5MB. We have a using() statement for it and even when doing GC.Collect() the memory doesn't get freed.
When we remove the listing of the table and just create new ClassEntities the memory stays really low.
We tried everything we could to destroy AND collect but to no avail, resulting into a memory use of over 1GB.
Here is the main program:
List < string > databases = (from x in admin_db.tblDbs select x.db_name).ToList();
foreach(var db_name in databases) {
Console.WriteLine("Current db:" + db_name);
var entityString = String.Format("metadata=<here we put the connection string>", db_name);
using(ClassEntities db = new ClassEntities(entityString)) {
try {
List < tblDepartment > departments = db.tblDepartments.ToList();
departments = null;
} catch {}
}
}
Then the ClassEntities (stripped down):
public partial class ClassEntities: ObjectContext {
public ClassEntities(): base("name=ClassEntities", "ClassEntities") {
this.ContextOptions.LazyLoadingEnabled = true;
OnContextCreated();
}
public ClassEntities(string connectionString): base(connectionString, "ClassEntities") {
this.ContextOptions.LazyLoadingEnabled = true;
OnContextCreated();
}
public ClassEntities(EntityConnection connection): base(connection, "ClassEntities") {
this.ContextOptions.LazyLoadingEnabled = true;
OnContextCreated();
}
}
Any help or ideas would be hugely appreciated.
There is 2 conditions for an object to be garbage collected:
It gone out of scope
It is no more referenced elsewhere in the program
As your screencast shows you have circular references that prevent expected garbage collection.
Your list of items is referencing connections, that in return references your items (HorekoEntitiesNoLog)
Try the following:
Identify the complete graph of object references that constitute your circular references objects graph.
Identify and the entry point(s) of this graph.
Clear the reference of any entry point of the graph.
Call GC.Collect().
If the memory don't get freed after GC.Collect call, either your are missing entry points references clear or your graph is not complete.
I am using EF 4.2 and am having an issue that happens quite randomly and without warning. I have a Windows Service which updates the database. In the service I have a timer. When the time of the timer elapses a method gets called. This is the basic structure of the method
IEnumerable<Foo> foos = GetFoosFromDB();
foreach (Foo foo in foos)
{
if (some condition)
{
foo.Bar = 1;
}
if (some other condition)
{
foo.Bar = 2;
}
if (yet some other condition)
{
foo.Bar = 3;
}
else
{
int val = GetSomeValueFromDB();
if (val == something)
{
if(GetSomeOtherValueFromDB())
{
foo.Bar = 4;
}
else
{
CallSomeMethodThatAlsoCallsSaveChanges();
foo.Bat = SomeCalculatedValue();
}
}
}
}
SaveChanges();
Now, the problem is that once we start working with the database for a day and there are a few rows in the tables of that database (we are talking about 100 or 200 rows only), then even though this method is called, SaveChanges doesn't seem to do what it should do. What am I doing wrong?
thanks,
Sachin
Ignoring the other aspects of the code, this line stuck out as a likely problem:
else
{
CallSomeMethodThatAlsoCallsSaveChanges();
foo.Bat = SomeCalculatedValue();
}
// a few }} later...
SaveChanges();
When this logic branch is executed, your context's pending changes are committed to the DB (based on what you've provided). Depending on how you're creating and managing your db context objects, you've either cleared the modified list, or you've introduced potential change conflicts. When SaveChanges() is called after the loop, it may or may not have pending changes to commit (depends on whether the conditional logic called your other method).
Consider what logical unit(s) of work are being performed with this logic and keep those UoW atomically separated. Think about how your DB contexts are being created, managed, and passed around, since those maintain the local state of your objects.
If you are still having trouble, you can post more of your code and we can attempt to troubleshoot futher
I have a class Worker which sends emails periodically,I start in Global.asax.cs on App_start()
public static class Worker
{
public static void Start()
{
ThreadPool.QueueUserWorkItem(o => Work());
}
public static void Work()
{
var r = new DbContext();
var m = new MailSender(new SmtpServerConfig());
while (true)
{
Thread.Sleep(600000);
try
{
var d = DateTime.Now.AddMinutes(-10);
var ns = r.Set<Notification>().Where(o => o.SendEmail && !o.IsRead && o.Date < d);
foreach (var n in ns)
{
m.SendEmailAsync("noreply#example.com", n.Email, NotifyMailTitle(n) + " - forums", NotifyMailBody(n));
n.SendEmail = false;
}
r.SaveChanges();
}
catch (Exception ex)
{
ex.Raize();
}
}
}
}
So I keep this dbcontext alive for the entire lifetime of the application is this a good practice ?
DbContext is a very light-weight object.
It doesn't matter whether your DbContext stays alive or you instantiate it just before making the call because the actual DB Connection only opens when you SubmitChanges or Enumerate the query (in that case it is closed on end of enumeration).
In your specific case. It doesn't matter at all.
Read Linq DataContext and Dispose for details on this.
I would wrap it in a using statement inside of Work and let the database connection pool do it's thing:
using (DbContext r = new DbContext())
{
//working
}
NOTE: I am not 100% sure how DbContext handles the db connections, I am assuming it opens one.
It is not good practice to keep a database connection 'alive' for the lifetime of an application. You should use a connection when needed and close it via the API(using statement will take care of that for you). The database connection pool will actually open and close connections based on connection demands.
I agree with #rick schott that you should instantiate the DbContext when you need to use it rather than keep it around for the lifetime of the application. For more information, see Working with Objects (Entity Framework 4.1), especially the section on Lifetime:
When working with long-running context consider the following:
As you load more objects and their references into memory, the
memory consumption of the context may increase rapidly. This may cause
performance issues.
If an exception causes the context to be in an unrecoverable state,
the whole application may terminate.