I've been working on a C# program that use LINQ to manage a simple SQLite database. The database consists in different tables that in my code are represented by different classes. This is how I create the tables:
public ITable<Doctors> Doctors => GetTable<Doctors>();
public ITable<Patients> Patients=> GetTable<Patients>();
public ITable<Prescriptions> Prescriptions=>GetTable<Prescriptions>();
public ITable<Assistants> Assistants=> GetTable<Assistants>();
public ITable<Medicines> Medicines => GetTable<Medicines>();
public ITable<General_medicines> General_medicines=> GetTable<General_medicines>();
public ITable<Stocks> Stocks=> GetTable<Stocks>();
public ITable<Dosages> Dosages=> GetTable<Dosages>();
public ITable<Recipes> Recipes=> GetTable<Recipes>();
public ITable<Prescriptions_MP> Prescriptions_MP=> GetTable<Prescriptions_MP>();
Now, I want to create a LINQ query (in a separate class) where I get different properties in all these tables and I put them inside an IEnumerable that I can later scan.
To do so, I proceed as following:
public IEnumerable<Therapy> TakePrescriptions()
{
HealthDataContext DbContext = DbFactory.Create();
var dbPrescriptions = DbContext.GetTable<Prescriptions>();
IEnumerable<Prescriptions> prescriptions= dbPrescriptions.AsEnumerable();
var dbPatients= DbContext.GetTable<Patients>();
IEnumerable<Pazienti> patients= dbPatients.AsEnumerable();
var dbPrescrizioniMP = DbContext.GetTable<Prescriptions_MP>();
IEnumerable<Prescriptions_MP> prescriptionsMP = dbPrescriptionsMP .AsEnumerable();
var dbRecipes = DbContext.GetTable<Recipes>();
IEnumerable<Recipes> recipes= dbRecipes .AsEnumerable();
var dbMedicines= DbContext.GetTable<Medicines>();
IEnumerable<Medicines> medicines= dbMedicines.AsEnumerable();
var dbGeneral_medicines = DbContext.GetTable<General_medicines>();
IEnumerable<General_medicines> general_medicines= dbGeneral_medicines.AsEnumerable();
var dbDosages = DbContext.GetTable<Dosages>();
IEnumerable<Dosages> dosages= dbDosages .AsEnumerable();
var query = from p in patients
join pr in prescriptions_MP on p.Id equals pr.Patient
join pre in prescriptions on pr.Prescription equals pre.Id
join fc in medicines on pre.Medicine equals fc.Id
join fg in general_medicines on fc.Medicine equals fg.Id
join ds in dosages on fg.Id equals ds.General_Medicine
where p.Doctor== IdDoctor
select new
{
IdDoctor, //int
p.Name, //string
pr.Prescription, //int
pre.Id, //int
fc.Format, //string
fc.Administration, //string
fc.Downloadable, //boolean
fc.Full_stomach, //boolean
nameM= fg.Name, //string
ds.Quantity, //int
ds.Hour //string
};
List < Therapy> therapy = new List<Therapy>();
foreach(var object in query)
{
Therapy t = new Therapy(IdDoctor, object.Name, object.Prescription, object.Id, object.Format, object .Administration, object.Downloadable, object.Full_stomach, object.nameM, object.Quantity, object.Hour);
therapy.Add(t);
}
return therapy;
}
Now when I try to load the page that should display a list of the results, I get InvalidOperationException: An open reader is associated with this command. Close it before changing the CommandText property. at the foreach operation.
When I try to debug, I can see that the tables I created before the query have items inside, but the result of the query is NULL.
I tried to dispose of the DBContext but then I get this exception: ObjectDisposedException: IDataContext is disposed, see https://github.com/linq2db/linq2db/wiki/Managing-data-connection Object name: 'DataConnection'.
The error you are getting “An open reader is associated with this command. Close it before changing the CommandText property”, suggests that multiple readers are open. However looking at your query it seems like one reader is open for your one query.
The reality however is different. You have 5 tables, such that each has a 1 to many relationship to another table. For example table patient has a 1 to many relationship to prescriptions table. As a patient can have multiple prescriptions.
Thus just considering these two tables, we first have one query to load all the patients, and then another query per patient to load all of its prescriptions, this means if you have N patients, this translates to 1 + N query , 1 to load all the patients, and N to load prescriptions of each of these patients.
Now given the 5- level join that you have in the code, doing the math, you can see how many potential open readers are out there.
The data is loaded on demand, meaning the readers are activated once you iterate through the results of your query, this is to avoid huge memory usage, in the cost of performance, and hence when in your foreach loop you start to iterate through the objects, the data is for real being fetched.
To solve the problem, you can try converting ToList at the end of your query to encourage binding (thus eager loading), or as one of the commenters are suggesting pass MultipleActiveResultSets=true to your connection string.
You should remove AsEnumerable() calls from tables you use in query, because they force linq2db to execute them as separate queries.
This has two consequences:
it attempts to start multiple queries over single db connection, which is not supported and you get InvalidOperationException
let's imagine it will work (e.g. you replaced AsEnumerable() with ToList() to read all data from each query). In this case it will load all data into application and perform joins in C# code - it will lead to really bad performance, especially in cases when you need to discard some data that doesn't meet join conditions.
Related
I have a Entity called "Client", and another Entity called "Card".
A Client may have many Cards.
My Client Entity looks like this:
public class Client{
public virtual ICollection<Card> Cards {get; set;}
}
Now I want to show the Client data in a DataGrid in WPF, and I want to get Cards Count data,so I add a property to Client Entity, which like this:
public class Client{
public virtual ICollection<Card> Cards {get; set;}
public int CardCount
{
return Cards.Count;
}
}
And then I query the data with Linq and Bind to view
var query = from n in db.Clients select n;
When I run the Application, I got a Exception just right on the return Cards.Count; line;
System.ObjectDisposedException
The ObjectContext instance has been disposed and can no longer be used for operations that require a connection.
So how could I correctly get the cards count?
There is a way simpler method than the other answers here show. Please also realize that solutions such as
var client = db.Clients.FirstOrDefault(c=> c.Id = someid); //get a client
if (client != null)
{
cardCount = client.Cards.Count;
}
will cause an issue called Select N+1 problem. Read up on it if interested, but in a nutshell, it means the following:
Because you are not only interested in one exact client, but you want to display N clients, you need to do one (1) query to get just the clients. Then, by doing the FirstOrDefault stuff, you are actually doing one (1) extra db roundtrip to the database per Client record, which results in an additional N * 1 = N roundtrips. What this means that, if you were to just query the Clients without any related data, you could get however many client records you like, in just one query. But by fetching related data to each of them one-by-one, you are doing way too many db roundtrips.
Here is a way to solve this issue, by using joins and projections. You can get all the data you need in a single DB access:
using (var context = GetDbContext())
{
return context.Clients.Select(cli => new YourViewModel
{
Name = cli.FullName,
// Other prop setters go here
CardCount = cli.Cards.Count
}).Skip((page - 1) * pageSize).Take(pageSize).ToList();
}
You might be wondering, what's the difference afterall? Well, here, you are not working with materialized objects, as others call them here, but with a DbContext. By applying the proper LINQ operators to it (note, that this works not just with DbContext, but also with any IQueryable (well obviously not if you call AsQueryable() on an already in-memory collection but whatever)), LINQ to Entities can construct a proper SQL to join the tables and project the results and therefore you fetch all required data in one go. Note that LINQ to Entities IS ABLE to translate the cli.Cards.Count into a proper SQL Count statement.
You can get the count without loading the entities like this:
using (var context = new MyContext())
{
var client = context.Client.Find(clientId);
// Count how many cards the client has
var cardsCount = context.Entry(client)
.Collection(b => b.Cards)
.Query()
.Count();
}
More information on MSDN page.
You get an ObjectDisposedException if you do not materialize the retreived query. In the following case, the query gets executed only when you Access the first time the list from GetNonMaterialized and not before leaving the method. Fact of this the db is disposed because of lost of scope.
public IEnumerable<Client> GetNonMaterialized()
{
return from n in db.Clients select n;
}
In the following case the query is executed before leaving the method.
public IEnumerable<Client> GetMaterialized()
{
return (from n in db.Clients select n).ToList();
}
Always be sure that the query is executed before exiting the scope of a ObjectContext.
If you want to know whether the query is executed and when enalbe Logging of EF.
How can I turn off Entity Framework 6.1 logging?
I've been learning C# / LINQ / ASP.NET / MVC 3 / EF for a few months now comming from Java / Icefaces / Ibatis base (Real world uses .NET D;). I really enjoy LINQ / Entity Framework from the .NET Framework but I'm having a few issues understand what's really happening behind the scenes.
Here's my problem:
I'm using a AJAX / JSON fed jQuery datatable (that I highly recommend to anyone in need of a free web datatable system by the way). I have a method in my MVC3 application that returns a JSON result of the data needed by the table, doing the sorting and all. Everything is working nicely and smoothly. However, I'm having a concern with the "dirty" hack I had to do to make this work.
Here's the complete code:
//inEntities is the Entity Framework Database Context
//It includes the following entities:
// Poincon
// Horaire
// HoraireDetail
//Poincon, Horaire and HoraireDetail are "decorated" using the Metadata technic which
//adds properties methods and such to the Entity (Like getEmploye which you will see in
//the following snippet)
//
//The Entity Employe is not a database data and therefor not handled by the EF.
//Instead, it is a simple object with properties that applies Lazy Loading to get an
//Employe Name based off of his Employe ID in the Active Directory. An employe object
//can be constructed with his Employe ID which will expose the possibility of getting
//the Employe Name from the AD if needed.
[HttpPost]
public JsonResult List(FormCollection form)
{
String sEcho;
int iDisplayStart;
int iDisplayLength;
String sSearch;
int iSortingCols;
Dictionary<String, String> sorting;
try
{
sEcho = form["sEcho"];
iDisplayStart = int.Parse(form["iDisplayStart"]);
iDisplayLength = int.Parse(form["iDisplayLength"]);
sSearch = form["sSearch"];
iSortingCols = int.Parse(form["iSortingCols"]);
sorting = new Dictionary<string,string>();
for (int i = 0; i < iSortingCols; i++)
sorting.Add(form["mDataProp_" + form["iSortCol_" + i]].ToUpper(), form["sSortDir_" + i].ToUpper());
}
catch
{
HttpContext.Response.StatusCode = 500;
return null;
}
var qPoincon = inEntities.Poincons.AsEnumerable();
var lPoincon = qPoincon.Select(o => new
{
o.id,
emp = o.getEmploye(),
o.poinconStart,
o.poinconEnd,
o.commentaire,
o.codeExceptions
}).AsEnumerable();
//Search
lPoincon = lPoincon.Where(p => (p.emp.empNoStr.Contains(sSearch) || p.emp.empNom.Contains(sSearch) || (p.commentaire != null && p.commentaire.Contains(sSearch))));
//Keep count
int iTotalDisplayRecords = lPoincon.Count();
//Sorting
foreach(KeyValuePair<String,String> col in sorting)
{
switch (col.Key)
{
case "EMPNO":
if (col.Value == "ASC")
lPoincon = lPoincon.OrderBy(h => h.emp.empNo);
else
lPoincon = lPoincon.OrderByDescending(h => h.emp.empNo);
break;
case "POINCONSTART":
if (col.Value == "ASC")
lPoincon = lPoincon.OrderBy(h => h.poinconStart);
else
lPoincon = lPoincon.OrderByDescending(h => h.poinconStart);
break;
case "POINCONEND":
if (col.Value == "ASC")
lPoincon = lPoincon.OrderBy(h => h.poinconEnd);
else
lPoincon = lPoincon.OrderByDescending(h => h.poinconEnd);
break;
case "COMMENTAIRE":
if (col.Value == "ASC")
lPoincon = lPoincon.OrderBy(h => h.commentaire);
else
lPoincon = lPoincon.OrderByDescending(h => h.commentaire);
break;
}
}
//Paging
lPoincon = lPoincon.Skip(iDisplayStart).Take(iDisplayLength);
//Building Response
var jdt = new
{
iTotalDisplayRecords = iTotalDisplayRecords,
iTotalRecords = inEntities.Poincons.Count(),
sEcho = sEcho,
aaData = lPoincon
};
return Json(jdt);
}
As you can see, when I'm grabbing the entire list of "Poincons" from the EF and turning it into a Enumerable. From my current understanding, turning the LINQ query into a Enumerable "kills" the link to the EF, or in other words, will generate the SQL required to get that list at that point instead of keeping the LINQ data until the end and execute a percise query that will return only the data you require. After turning this LINQ Query into a Enumerable, I'm heavily filtering the LINQ (since there is paging, sorting, searching in the datatable). This leads me to thinkg that what my code is currently doing is "Grab all the "Poincons" from the database and put it into the web server's memory as a Enumerable, do your work with the Enumerable then serialize the result as a JSON string and send it to the client.
If I'm correct, the performance hit is quite heavy when you hit the couple thousand of entries (which will happen quite fast once in production... everytime an employe comes to work, it will add 1 entry. 100 employes, ~300 work days a year, you get the idea).
The reason for this hack is that the EF does not know what "getEmploye" method of "Poincon" is, therefor throwing an exception at runtime similar to this:
LINQ to Entities ne reconnaît pas la méthode « PortailNorclair.Models.Employe getEmploye() », et cette dernière ne peut pas être traduite en expression de magasin.
Approximated traduction (If anyone can let me know in a comment how to configure IIS / ASP.NET to display errors in english while keeping the globalization in a foreign language, I would be really grateful. French information about error messages is sometimes lacking):
LINQ to Entity does not recognize the method " PortailNorclair.Models.Employe getEmploye()" and the following could not be translated to a SQL expression.
The "getEmploye" method instances and returns a Employe object with the employe id found in the Poincon object. That Employe object has properties that "lazy loads" information like the employe name from the Active Directory.
So the question is: How can I avoid the performance hit from using .AsEnumerable() on the non-filtered list of objects?
Thanks a lot!
The "getEmploye" method instances and returns a Employe object with
the employe id found in the Poincon object. That Employe object has
properties that "lazy loads" information like the employe name from
the Active Directory.
You should be storing the Employee Name in the database, so you can then order, sort, skip and take in your Linq Query without having to load every employee object.
If empNoStr, empNom, and empNo were all in the database, you could retrieve just the records you want, and call getEmploye() (loading whatever else you need from active directory, or wherever) for each of those.
There are some classes on which your program performs its main work.
There are other classes which represent to database rows.
If you keep them separated, you can also separate actions you intend to occur in the database from actions you intend to perform locally. This makes it trivial to avoid loading the full table, when specific rows are required.
I see you're also doing Paging locally, while the database can do that and save your webserver some memory.
I'm using a LinqToSql query to select a list of groups from a database and spew out a table. I've written a custom class to cache the results of this query for better performance. Trouble is, whenever I implement the caching class, I get weird appending behaviour from the outputting statement.
My results are outputted in the format
Test Group (1)
where "Test Group" is the name, and (1) is the number of members within that group. Here's the code that appends the count to the name (from the view)
<td>#group.group_name (#group.num_total)</td>
When I pull this from a live linq query returning groups, everything works as expected.
However, when I use my caching class, every successive page load adds the number on to the end of the group title:
Test Group (1) (1) (1) (1) (1) (1)
This only happens when I use the caching class (included below). I've been over the cache class and there's no reason I can see why this would be happening.
I can think of several workarounds for this issue, so its not a show stopper, but I'm curious as to what the fudge is going on. Any ideas?
Caching Class:
public class Cache
{
public static int user_id {
get { return
Convert.ToInt32(
Membership.GetUser(
HttpContext.Current.User.Identity.Name
).ProviderUserKey
);
}
}
public static void GetGroups_InvalidateCache()
{
if (HttpContext.Current.Cache["GetGroups_" + user_id] != null)
HttpContext.Current.Cache.Remove("GetGroups_" + user_id);
}
public static ICollection<Groups> GetGroups()
{
if (HttpContext.Current.Cache["GetGroups_" + user_id] == null)
{
using(DBContext db = new DBContext())
{
var Groups = (from g in db.Groups
where g.user_id == user_id
select g).ToList();
HttpContext.Current.Cache.Insert(
"GetGroups_" + user_id,
Groups,
null,
DateTime.Now.AddMinutes(5),
TimeSpan.Zero
);
}
}
return HttpContext.Current.Cache["GetGroups_" + user_id]
as ICollection<Groups>;
}
}
UPDATE:
I've now implemented Adam Tuliper and Paul Tyng's suggestions of calling the data context with the using clause, ending the linq statement with ToList() and using ICollection instead of IQueryable. The problem is still occurring.
Another interesting observation: The issue only happens if I navigate away to another page and return. If I simply refresh the page, it doesn't happen (Although any previous number additions still remain when I refresh)
Instead of returning IQueryable, try using simply IEnumerable and also using
using(DBContext db = new DBContext())
{
var Groups =
(from g in db.Groups
where g.user_id == user_id
select g).ToList();
...
}
Also dispose of your context as in the above statement (with the using clause)
The ToList() forces the execution "now" - I think you are potentially having deferred execution issues
I'm having very strange problem in this simple linq query
return (from doc in db.umDocumentActions
where doc.sysDocument.ModuleID == modid
join roleaction in db.umRoleActions on doc.DocumentActionID equals roleaction.DocumentActionID
where roleaction.RoleID == roleID
select new { doc.DocumentID, roleaction.RoleID }).Count() > 0;
When this query is called it gives invalid operation exception telling me that sequence contains no elements. It happens when there is fair amount of traffic on the site. I am using following static method to get instance of datacontext.
public static EvoletDataContext Get()
{
var connection = ProfiledDbConnection.Get(new SqlConnection(ConfigurationManager.ConnectionStrings["cnstring"].ToString()));
return new EvoletDataContext(connection);
//return DataContextUtils.CreateDataContext<EvoletDataContext>(connection);
}
I'm afraid that this method is creating problem as static methods are not thread safe. Any views?
My best guess would be that sysDocument is actually a seperate table with a reference to DocumentID. This would normally mean that there would be a related collection of sysDocuments in the document class, but I'm guessing that you've changed the cardinality to "one to one" in the Linq to SQL designer.
When using a "one to one" cardinality, Linq to SQL uses the Single() method behind the scenes to get the sysDocument. This means that if there are no related sysDocuments you will get an invalidoperation exception.
You can fix this by wither changing the cardinality in you Linq model from "one to one" to "one to many" and use the SingleOrDefault() method to get the related sysDocument from the sysDocuments collection.
If that doesn't sound appealing you can look in the database to find which document doesn't have a related sysDocument and fix it manually.
UPDATE:
Instead of basing the query off the documentActions, try basing it off the sysDocument table instead. I've had to guess at what the table will be called so this might not compile, but hopefully you get the idea:
var query = from sysDocument in db.sysDocuments
where sysDocument.ModuleID == modid
let doc = sysDocument.umDocumentAction
join roleaction in db.umRoleActions on doc.DocumentActionID equals roleaction.DocumentActionID
where roleaction.RoleID == roleID
select new { doc.DocumentID, roleaction.RoleID };
//return true if there are any results (this is more efficient than Count() > 0)
return query.Any();
I have a View Model that has some serious nesting. I need to populate it from Entity Framework 4. I tried creating one big linq statement to populate it, but it says it doesn't recognize the .ToList() methods. It compiles fine. Runtime error is
LINQ to Entities does not recognize the method
'System.Collections.Generic.List`1[ProductDepartment] ToList[ProductDepartment]
(System.Collections.Generic.IEnumerable`1[ProductDepartment])' method,
and this method cannot be translated into a store expression.
What is a more efficient way to populate something like this without doing several thousand database calls?
List<Product> Products {
int ID
string Name
...
List<Department> Departments {
int ID
string Name
}
List<Image> Images {
int ID
string Name
}
List<Price> Prices {
int ID
string Name
List<Version> Versions {
int ID
string Name
List<Pages> Pages {
int ID
string Name
} } } }
The horrible Linq code looks something like this
var myProducts = (from myProduct in DC.MyProducts
where p => p.productGroup == 1
select new Product {
ID = myProduct.ID,
Name = myProduct.Name,
Departments = (from myDept in DC.MyDepartments
where q => q.fkey = myProduct.pkey
select new Department {
ID = myDept.ID,
Name = myDept.Name
}).ToList(),
...
//Same field assignment with each nesting
}).ToList();
Update:
The fix was to remove all the .ToLists(), which worked better anyway.
Now I have to do filtering and sorting on the end product.
Well for starters, that is one crazy model, but i'm assuming you already know this.
Do you really need all that info at once?
I'll play devil's advocate here and assume you do, in which case you have a couple of logical choices:
1) As #xandy mentioned - use .Include to eager load your associations in the one call. This is assuming you have setup navigational properties for your entites in your EDMX.
2) Use a View. Put all that crazy joining logic inside the database, making your EF work a very simple select from the view. The downside of this is your queries to the view basically become read only, as i don't believe you can perform updates to an entity which is mapped to a view.
So it's your choice - if this is a readonly collection for displaying data, use a View, otherwise eager-load your associations in the one hit.
Also, be careful when writing your LINQ queries - i see you have multiple .ToList statements, which will cause the query to be executed.
Build up your query, then perform the .ToList once at the end.
why do you require all this informataion at one go? You can use lazy loading when a nested property is accessed?