I have a list of row IDs being used in a Linq to SQL query.
using (var db = new DataContext(dbConnectionString))
{
ids = new list<long> {"1","2",...};
var data = (from item in db.GetTable<dataTable>().AsEnumerable()
where ids.Contains(item.ID)
select new customDataStructure{}).ToList();
}
In some cases, it passes each of the IDs in the list to SQL as parameters in the format "where ID in (#p0,#p1,...)". In other cases, the query retrieves all records and lets the filter happen in c#. This causes a huge spike in memory usage and possible memory exceptions it the amount of data retrieved is very large.
Is it possible to force the query to execute the where on the SQL server to avoid this issue?
Using .Contains() is not what I think you are wanting to use. I think you want to use .Any() which will see if any of the items in your list match a property or column in your table. See below:
var ids = new List<long> {1,2};
var data = (from item in db.GetTable<dataTable>().AsEnumerable()
where ids.Any(m=> m == item.Property)
select new customDataStructure{}).ToList();
As Evk commented, the AsEnumerable() caused the where to be executed locally, not on the SQL server.
It was being used since I needed lists and dictionaries in the returned type which cause exceptions if you try to build them in an IQueryable.
I altered the query to
using (var db = new DataContext(dbConnectionString))
{
ids = new list<long> {"1","2",...};
var data = (from item in db.GetTable<dataTable>()
where ids.Contains(item.ID)
select new
{
name = item.name,
subIds = item.subitemIDs
...
}).AsEnumerable()
.Select(x=> new customDataStructure
{
itemname = x.name,
subIds = x.subIds.ToList(),
...
}).ToList();
}
It let the data collection be done by SQL and then have the needed structure completed after the SQL executes.
I have the following in my controller get method
private PeopleContext Peopledb = new PeopleContext();
private IARContext db = new IARContext();
public ActionResult OwnerList()
{
var owners = from s in db.Owners
where s.Dormant == false
orderby s.Post.PostName
select s;
var viewModel = owners.Select(t => new OwnerListViewModel
{
Created = t.Created,
Post = Peopledb.Posts.FirstOrDefault(x => x.PostId == t.SelectedPostId).PostName.ToString(),
});
return PartialView("_OwnerList", viewModel);
}
I'm getting this error when I try and load the page:
The specified LINQ expression contains references to queries that are associated with different contexts.
I know the issue is that LINQ cant query two different contexts but having tried several solutions on here I cant seem to fix the issue and sucesfully query the Peopledb.Posts table to find the related PostName to display for each instance in the db.Owners table.
You can try this:
var owners = (from s in db.Owners
where s.Dormant == false
orderby s.Post.PostName
select s).ToList();
This will execute the code in one context and have the List<Owner> in memory for the other context.
Also take a look for the execution of the .Select part, does it execute a separate query for each owner? If so you should optimize it, you can get the posts beforehand using the ids and then build your viewmodel.
Entity Framework context can work only with single database. If you want to get data from another database in this context, you can create proxy View, that will reflect this data in db of your dbcontext.
Just transition from Linq-to-Entities to Linq-to-Objects using AsEnumerable():
var viewModel = owners.AsEnumerable()
.Select(t => new OwnerListViewModel
...
I am using EF6 to develop an application that uses a huge data,so in one of my query
his = db.HISTORies.ToList();
front = db.Fronts.ToList();
warehouses = db.Warehouses.ToList();
//------
var q =
his.GroupBy(j => new { j.Line_Number_PGZ, j.MAT_0_PG,j.JOINT_NO_PGZ }).ToList().Select(m => new gridView()
{
JointNumber = m.Key.JOINT_NO_PGZ,
material0 = m.Key.MAT_0_PGZ,material0Name =his.Where(i=>i.MAT_0_PGZ==m.Key.MAT_0_PGZ).First().Code_0_Object_PGZ,
line = m.Key.Line_Number_PGZ
}).OrderByDescending(i => i.material0Need).ToList();
As you can see i fetch all my database to my application .so in my query i grouped my data based on 2 columns ,but my problem is the his has a lot of records so in this line
his.Where(i=>i.MAT_0_PGZ==m.Key.MAT_0_PGZ).First().Code_0_Object_PGZ
I again search an items in this table and this line makes my application to be so slow how can fix this query?
The Code_0_Object_PGZ is in his list but it can be accessed in the select statement !!!
I want to know how can i access Code_0_Object_PGZ without include it in groupby ?
best regards
This is easy for me to perform in TSQL, but I'm just sitting here banging my head against the desk trying to get it to work in EF4!
I have a table, lets call it TestData. It has fields, say: DataTypeID, Name, DataValue.
DataTypeID, Name, DataValue
1,"Data 1","Value1"
1,"Data 1","Value2"
2,"Data 1","Value3"
3,"Data 1","Value4"
I want to group on DataID/Name, and concatenate DataValue into a CSV string. The desired result should contain -
DataTypeID, Name, DataValues
1,"Data 1","Value1,Value2"
2,"Data 1","Value3"
3,"Data 1","Value4"
Now, here's how I'm trying to do it -
var query = (from t in context.TestData
group h by new { DataTypeID = h.DataTypeID, Name = h.Name } into g
select new
{
DataTypeID = g.Key.DataTypeID,
Name = g.Key.Name,
DataValues = (string)g.Aggregate("", (a, b) => (a != "" ? "," : "") + b.DataValue),
}).ToList()
The problem is that LINQ to Entities does not know how to convert this into SQL. This is part of a union of 3 LINQ queries, and I'd really like it to keep it that way. I imagine that I could retrieve the data and then perform the aggregate later. For performance reasons, that wouldn't work for my app. I also considered using a SQL server function. But that just doesn't seem "right" in the EF4 world.
Anyone care to take a crack at this?
If the ToList() is part of your original query and not just added for this example, then use LINQ to Objects on the resulting list to do the aggregation:
var query = (from t in context.TestData
group t by new { DataTypeID = t.DataTypeID, Name = t.Name } into g
select new { DataTypeID = g.Key.DataTypeID, Name = g.Key.Name, Data = g.AsEnumerable()})
.ToList()
.Select (q => new { DataTypeID = q.DataTypeID, Name = q.Name, DataValues = q.Data.Aggregate ("", (acc, t) => (acc == "" ? "" : acc + ",") + t.DataValue) });
Tested in LINQPad and it produces this result:
Some of the Answers suggest calling ToList() and then perform the calculation as LINQ to OBJECT. That's fine for a little amount of data, but what if I have a huge amount of data that I do not want to load into memory too early, then, ToList() may not be an option.
So, the better idea would be to process/format the data in the presentation layer and let the Data Access layer do only loading or saving raw data that SQL likes.
Moreover, in your presentation layer, most probably you are filtering the data by paging, or maybe you are showing one row in the details page, so, the data you will load into the memory is likely smaller than the data you load from the database. (Your situation/architecture may be different,.. but I am saying, most likely).
I had a similar requirement. My problem was to get the list of items from the Entity Framework object and create a formatted string (comma separated value)
I created a property in my View Model which will hold the raw data from the repository and when populating that property, the LINQ query won't be a problem because you are simply querying what SQL understands.
Then, I created a get-only property in my ViewModel which reads that Raw entity property and formats the data before displaying.
public class MyViewModel
{
public IEnumerable<Entity> RawChildItems { get; set; }
public string FormattedData
{
get
{
if (this.RawChildItems == null)
return string.Empty;
string[] theItems = this.RawChildItems.ToArray();
return theItems.Length > 0
? string.Format("{0} ( {1} )", this.AnotherRegularProperty, String.Join(", ", theItems.Select(z => z.Substring(0, 1))))
: string.Empty;
}
}
}
Ok, in that way, I loaded the Data from LINQ to Entity to this View Model easily without calling.ToList().
Example:
IQueryable<MyEntity> myEntities = _myRepository.GetData();
IQueryable<MyViewModel> viewModels = myEntities.Select(x => new MyViewModel() { RawChildItems = x.MyChildren })
Now, I can call the FormattedData property of MyViewModel anytime when I need and the Getter will be executed only when the property is called, which is another benefit of this pattern (lazy processing).
An architecture recommendation: I strongly recommend to keep the data access layer away from all formatting or view logic or anything that SQL does not understand.
Your Entity Framework classes should be simple POCO that can directly map to a database column without any special mapper. And your Data Access layer (say a Repository that fetches data from your DbContext using LINQ to SQL) should get only the data that is directly stored in your database. No extra logic.
Then, you should have a dedicated set of classes for your Presentation Layer (say ViewModels) which will contain all logic for formatting data that your user likes to see. In that way, you won't have to struggle with the limitation of Entity Framework LINQ. I will never pass my Entity Framework model directly to the View. Nor, I will let my Data Access layer creates the ViewModel for me. Creating ViewModel can be delegated to your domain service layer or application layer, which is an upper layer than your Data Access Layer.
Thanks to moi_meme for the answer. What I was hoping to do is NOT POSSIBLE with LINQ to Entities. As others have suggested, you have to use LINQ to Objects to get access to string manipulation methods.
See the link posted by moi_meme for more info.
Update 8/27/2018 - Updated Link (again) - https://web.archive.org/web/20141106094131/http://www.mythos-rini.com/blog/archives/4510
And since I'm taking flack for a link-only answer from 8 years ago, I'll clarify just in case the archived copy disappears some day. The basic gist of it is that you cannot access string.join in EF queries. You must create the LINQ query, then call ToList() in order to execute the query against the db. Then you have the data in memory (aka LINQ to Objects), so you can access string.join.
The suggested code from the referenced link above is as follows -
var result1 = (from a in users
b in roles
where (a.RoleCollection.Any(x => x.RoleId = b.RoleId))
select new
{
UserName = a.UserName,
RoleNames = b.RoleName)
});
var result2 = (from a in result1.ToList()
group a by a.UserName into userGroup
select new
{
UserName = userGroup.FirstOrDefault().UserName,
RoleNames = String.Join(", ", (userGroup.Select(x => x.RoleNames)).ToArray())
});
The author further suggests replacing string.join with aggregate for better performance, like so -
RoleNames = (userGroup.Select(x => x.RoleNames)).Aggregate((a,b) => (a + ", " + b))
You are so very close already. Try this:
var query = (from t in context.TestData
group h by new { DataTypeID = h.DataTypeID, Name = h.Name } into g
select new
{
DataTypeID = g.Key.DataTypeID,
Name = g.Key.Name,
DataValues = String.Join(",", g),
}).ToList()
Alternatively, you could do this, if EF doesn't allow the String.Join (which Linq-to-SQL does):
var qs = (from t in context.TestData
group h by new { DataTypeID = h.DataTypeID, Name = h.Name } into g
select new
{
DataTypeID = g.Key.DataTypeID,
Name = g.Key.Name,
DataValues = g
}).ToArray();
var query = (from q in qs
select new
{
q.DataTypeID,
q.Name,
DataValues = String.Join(",", q.DataValues),
}).ToList();
Maybe it's a good idea to create a view for this on the database (which concatenates the fields for you) and then make EF use this view instead of the original table?
I'm quite sure it's not possible in a LINQ statement or in the Mapping Details.
How to write LINQ statement that returns ROWNUMBER of element with id == something?
There is no direct way to do this that I'm aware of. You'd have to pull the whole query down to the client, and the from there you could project in the row numbers. As an alternative, you could write a stored procedure that uses ROW_NUMBER, and then hit that proc from Linq to SQL.
In your case, the only way you're going to be able to do this would be client side. Keep in mind that the following statement is NOT going to do this at the server, but will pull down your whole table and get the index at the client...
using (var dc = new DataClasses1DataContext())
{
var result = dc.Users
.AsEnumerable() // select all users from the database and bring them back to the client
.Select((user, index) => new // project in the index
{
user.Username,
index
})
.Where(user => user.Username == "sivey"); // filter for your specific record
foreach (var item in result)
{
Console.WriteLine(string.Format("{0}:{1}", item.index, item.Username));
}
}
You should be able to use the Skip and Take extension methods to accomplish this.
For example, if you want row 10:
from c in customers
where c.Region == "somewhere"
orderby c.CustomerName
select new {c.CustomerID, c.CustomerName}
.Skip(9).Take(1);
How To Project a Line Number Into Linq Query Results
How To Project a Line Number Into Linq Query Results