I was wondering if it was possible to store the results of multiple linq queries in a single IQueryable statement?
I have a query which I use in a foreach:
//Where OnDemandHistory is the table
IOrderedQueryable<OnDemandHistory> A;
foreach (int id in machineID)
{
A = OnDemandHistory.Where(c => c.MachineID == id).OrderByDescending(c => c.ODHisDate);
// I want to Order all results before writing to the table
foreach(var entry in A)
{
// I add to a table based on all entries found in A
}
}
I am trying to get all entries where the machine ID match. The no. of MachineID's is varying (based on the user).
I was wondering if I can do a OrderByDescending after I have stored all the results from the query but before adding to the table.
I know due to the inner foreach loop that it won't happen, however when I try to do this:
foreach (int id in machineID)
{
A = OnDemandHistory.Where(c => c.MachineID == id).OrderByDescending(c => c.ODHisDate);
// I want to Order all results before writing to the table
}
foreach(var entry in A)
{
// I add to a table based on all entries found in A
}
I get a local variable A uninitialized error,
How would I go about solving this?
Thanks in advance
You can do it much simpler by using the Contains statement:
var result = OnDemandHistory.Where(c => machineID.Contains(c.MachineID))
.OrderByDescending(c => c.ODHisDate);
The error is caused because as the final result of your first query produces only the result of the last value of machineID this may result in either a null result or an uninitialisedvalue of A, so A needs to be initialised. Also, I suspect A could be a simple list.
You need something like:
A = new List<OnDemandHistory>();
foreach (int id in machineID)
{
A.AddRange(OnDemandHistory
.Where(c => c.MachineID == id).OrderByDescending(c => c.ODHisDate).ToList());
}
// order A here
Then run your second loop having checked that A has rows. However, I suspect there are smarter ways in LINQ of concatenating the machineID part of the query as a single LINQ statement.
Related
I have a feeling I know that what the reason is for this behavior, but I don't know what the best way of resolving it will be.
I have built a LinqToSQL query:
public IEnumerable<AllConditionByCountry> GenerateConditions(int paramCountryId)
{
var AllConditionsByCountry =
(from cd in db.tblConditionDescriptions...
join...
join...
select new AllConditionByCountry
{
CountryID = cd.CountryID,
ConditionDescription = cd.ConditionDescription,
ConditionID = cd.ConditionID,
...
...
}).OrderBy(x => x.CountryID).AsEnumerable<AllConditionByCountry>();
return AllConditionsByCountry;
}
This query returns about 9500+ rows of data.
I'm calling this from my Controller like so:
svcGenerateConditions generateConditions = new svcGenerateConditions(db);
IEnumerable<AllConditionByCountry> AllConditionsByCountry;
AllConditionsByCountry = generateConditions.GenerateConditions(1);
Which then I'm looping through:
foreach (var record in AllConditionsByCountry)
{
...
...
...
This is where I think the issue is:
var rList = AllConditionsByCountry
.Where(x => x.ConditionID == conditionID)
.Select(x => x)
.AsEnumerable();
I'm doing an nested loop based off the data that I'm gathering from the above query (utilizing the original data I'm getting from AllConditionByCountry. I think this is where my issue lies. When it is doing the filter on the data, it SLOWS down greatly.
Basically this process writes out a bunch of files (.json, .html)
I've tested this at first using just ADO.Net and to run through all of these records it took about 4 seconds. Using EF (stored procedure or LinqToSql) it takes minutes.
Is there anything I should do with my types of lists that I'm using or is that just the price of using LinqToSql?
I've tried to return List<AllConditionByCountry>, IQueryable, IEnumerable from my GenerateConditions method. List took a very long time (similar to what I'm seeing now). IQueryable I got errors when I tried to do the 2nd filter (Query results cannot be enumerated more than once).
I have run this same Linq statement in LinqPad and it returns in less than a second.
I'm happy to add any additional information.
Please let me know.
Edit:
foreach (var record in AllConditionsByCountry)
{
...
...
...
var rList = AllConditionsByCountry
.Where(x => x.ConditionID == conditionID)
.Select(x => x)
.AsEnumerable();
conditionDescriptionTypeID = item.ConditionDescriptionTypeId;
id = conditionDescriptionTypeID + "_" + count.ToString();
...
...
}
TL;DR: You're making 9895 queries against the database instead of one. You need to rewrite your query such that only one is executed. Look into how IEnumerable works for some hints into doing this.
Ah, yeah, that for loop is your problem.
foreach (var record in AllConditionsByCountry)
{
...
...
...
var rList = AllConditionsByCountry.Where(x => x.ConditionID == conditionID).Select(x => x).AsEnumerable();
conditionDescriptionTypeID = item.ConditionDescriptionTypeId;
id = conditionDescriptionTypeID + "_" + count.ToString();
...
...
}
Linq-to-SQL works similarly to Linq in that it (loosely speaking) appends functions to a chain to be executed when the enumerable is iterated - for example,
Enumerable.FromResult(1).Select(x => throw new Exception());
This doesn't actually cause your code to crash because the enumerable is never iterated. Linq-to-SQL operates on a similar principle. So, when you define this:
var AllConditionsByCountry =
(from cd in db.tblConditionDescriptions...
join...
join...
select new AllConditionByCountry
{
CountryID = cd.CountryID,
ConditionDescription = cd.ConditionDescription,
ConditionID = cd.ConditionID,
...
...
}).OrderBy(x => x.CountryID).AsEnumerable<AllConditionByCountry>();
You're not executing anything against a database, you're just instructing C# to build a query that does this when it is iterated. That's why just declaring this query is fast.
Your problem comes when you get to your for loop. When you hit your for loop, you signal that you want to start iterating the AllConditionsByCountry iterator. This causes .NET to go off and execute the initial query, which takes time.
When you call AllConditionsByCountry.Where(x => x.ConditionID == conditionID) in the for loop, you're constructing another iterator that doesn't actually do anything. Presumably you actually use the result of rList within that loop, however, you're essentially constructing N queries to be executed against the database (where N is the size of AllConditionsByCountry).
This leads to a scenario where you are effectively executing approximately 9501 queries against the database - 1 for your initial query and then one query for each element within the original query. The drastic slowdown compared to ADO.NET is because you're probably making 9500 more queries than you were originally.
You ideally should change the code so that there is one and only one query executed against the database. You've a couple of options:
Rewrite the Linq-to-SQL query such that all of the legwork is done by the SQL database
Rewrite the Linq-to-SQL query so it looks like this
var conditions = AllConditionsByCountry.ToList();
foreach (var record in conditions)
{
var rList = conditions.Where(....);
}
Note that in that example I am searching conditions rather than AllConditionsByCountry - .ToList() will return a list that has already been iterated so you do not create any more database queries. This will still be slow (since you're doing O(N^2) over 9500 records), but it will still be faster than creating 9500 queries since it will all be done in memory.
Just rewrite the query in ADO.NET if you're more comfortable with raw SQL than Linq-to-SQL. There's nothing wrong with this.
I think I should point out what methods cause an IEnumerable to be iterated and what ones don't.
Any method named As* (such as AsEnumerable<T>()) do not cause the enumerable to be iterated. It's essentially a way of casting from one type to another.
Any method named To* (such as ToList<T>()) will cause the enumerable to be iterated. In the event of Linq-to-SQL this will also execute the database query. Any method that also results in you getting a value out of the enumerable will also cause iteration. You can use this to your advantage by creating a query and forcing iteration using ToList() and then searching that list - this will cause the comparisons to be done in memory, which is what I demo above
//Firstly: IEnumerable<> should be List<>, because you need to massage result later
public IEnumerable<AllConditionByCountry> GenerateConditions(int paramCountryId)
{
var AllConditionsByCountry =
(from cd in db.tblConditionDescriptions...
join...
join...
select new AllConditionByCountry
{
CountryID = cd.CountryID,
ConditionDescription = cd.ConditionDescription,
ConditionID = cd.ConditionID,
...
...
})
.OrderBy(x => x.CountryID)
.ToList() //return a list, so only 1 query is executed
//.AsEnumerable<AllConditionByCountry>();//it's useless code, anyway.
return AllConditionsByCountry;
}
about this part:
foreach (var record in AllConditionsByCountry) // you can use AllConditionsByCountry.ForEach(record=>{...});
{
...
//AllConditionsByCountry will not query db again, because it's a list, no long a query
var rList = AllConditionsByCountry.Where(x => x.ConditionID == conditionID);//.Select(x => x).AsEnumerable(); //no necessary to use AsXXX if compilation do not require it.
...
}
BTW,
you should have your result paged, no page will need 100+ result. 10K return is the issue itself.
GenerateConditions(int paramCountryId, int page = 0, int pagesize = 50)
it's weird that you have to use a sub-query, usually it means GenerateConditions did not return the data structure you need, you should change it to give right data, no more subquery
use compiled query to improve more: https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/ef/language-reference/compiled-queries-linq-to-entities
we don't see your full query, but usually, it's right the part you should improve, especially when you have many conditions to filter and join and group... a little change could make all differences.
Instead of doing this horrible loop which does achieve the desired result :
foreach (var mealsViewModel in mealsListCollection)
{
foreach (var VARIABLE in mealsViewModel.Items)
{
foreach (var d in VARIABLE.ArticlesAvailable)
{
d.ArticleQty = 0;
}
}
}
I'm trying to achieve the same result but with this linQ statement :
mealsListCollection.ForEach(u =>
u.Items.Select(o => o.ArticlesAvailable.Select(c =>
{
c.ArticleQty = 0;
return c;
})));
But the linQ statement does not reset ArticleQty to zero
What I am doing wrong? and why ?
Change your linq to ForEach cause Select does not iterate through collection in the way you want.
MSDN definition:-
Select Projects each element of a sequence into a new form.
ForEach Performs the specified action on each element of the List.
mealsListCollection.ForEach(u =>
u.Items.ForEach(o =>
o.ArticlesAvailable.ForEach(c =>
{
c.ArticleQty = 0;
})));
Use SelectMany to work through trees of nested lists. Use the ForEach function last to do the work:
mealsListCollection
.SelectMany(m => m.Items)
.SelectMany(i => i.ArticlesAvailable)
.ToList()
.ForEach(a => { a.ArticleQty = 0; });
What you are doing wrong is: select is returning your same collection, but has no effect until the objects are iterated over. Sitting in the foreach call, the selects are outside of the execution path. (Review comments for more information).
.select() in a call by itself does nothing special but determine what the returned list will look like.
.select().ToList() iterates over the collection, applying the projection.
If you were to set a variable equal to the .select call, but never access the data inside it, then the values would essentially still be what they started as. As soon as you iterate over, or select a specific element, it would then apply the projections.
Changing the selects to foreachs per vasily's comments will give you the desired results.
Can I perhaps suggest that you look to set the value equal to 0 further up your stack ( or down)? - Without knowing your use case, maybe there Is a better place to default it back to 0 than where you have chosen?
(automapper, Initializer, etc )
Just some details. Get Records is a variable where it contains the results of my stored procedure. Now, what I want to ask is what if I want to remove the group by function but I still want to get the key and items? Is there a way to do it?
var sortResCinema = GetRecords.Where(x => test2.branch == x.Bbranch && test.movieName == x.MovieName && x.MovieName != null)
.GroupBy(x => x.MovieName,
(key, elements) =>
new
{
Id = key,
Items = elements.ToList()
}).ToList();
There's no need for GroupBy here since you are looking for a specific movieName.
I guess you wanted something like this:
var sortResCinema = GetRecords.Where(x => test2.branch == x.Bbranch && test.movieName == x.MovieName).ToList();
You can replace the GroupBy with a Select. The Select statement can be used to alter the type of the results returned, which is what you appear to want to do. Should work with exactly the same syntax as the second parameter. So replace "GroupBy" with "Select" and remove the first argument. The key and elements properties that are being used in the GroupBy statement are internal to that function so you'd need to work out what function you want to replace these by, for instance the key might be x.MovieName.
I'm trying to emulate a SQL IN statement on a series of SQL LIKE statements using LINQ (C#).
I start off with a IQueryable<user> object named Query which has not yet been 'filtered'.
I'm just starting to post onto Stack Overflow so please be patient with me :)...
ObjectQuery<user> Context = this.Context.users;
IQueryable<user> Query = (IQueryable<user>)Context;
// create a BLANK clone of the FULL list (Query)
var QueryFinal = Query.Where(i => i.ID == 0);
foreach (String Item in userFilter.Name.Contains)
{
// based on the FULL list (Query) return all records that apply to this item & then append results to the final list
QueryFinal = QueryFinal.Concat(Query.Where(i => i.Name.Contains(Item)));
}
return QueryFinal.ToList();
I thought that on each iteration, the result set being returned on the Query.Where statement would be appended to the QueryFinal list, which it does, but for some reason, on each subsequent iteration, it appears to overwrite all the previous records which are supposed to be 'stored for safe-keeping' in the final list. I've have also tried using .Union but still not the results I was hoping for. All it seems to return is the last result set, not all of the appended result sets together. Anyone spot what I'm doing wrong?
Because of deferred execution, when you call QueryFinal.ToList();, it will be using the last value of Item - this is a pretty common problem when doing these sorts of queries in a foreach loop.
In your case, something like this might help:
foreach (String Item in userFilter.Name.Contains)
{
string currentItem = Item;
// based on the FULL list (Query) return all records that apply to this item & then append results to the final list
QueryFinal = QueryFinal.Concat(Query.Where(i => i.Name.Contains(currentItem)));
}
try this:
var query = this.Context.users.Where(u =>
userFilter.Name.Contains.Any(c => u.Name.Contains(c))
);
I get a dictionary and i want to get the matching database entries for the key field in the dictionary.
So the code below gets all database fields that equals the data in the dictionary.keys. So far so good. Now when i loop it and i try to get the field from the fields that matches the key x it fails with an exception:
{"Comparison operators not supported for type 'System.Collections.Generic.Dictionary`2+KeyCollection[System.String,System.String]'"}
var fields = _dc.fieldInfos.Where(x => x.name.Equals(param.MethodData.Keys));
foreach (var entry in param.MethodData)
{
KeyValuePair<string, string> entry1 = entry;
var field = fields.SingleOrDefault(x => x.name.Equals(entry1.Key));
if (field == null)
....
}
The line that fails is this:
var field = fields.SingleOrDefault(x => x.name.Equals(entry1.Key));
Any ideas?
Be mindful that the _dc.fieldInfos.Where(x => x.name.Equals(param.MethodData.Keys)) (line 1) doesn't actually get executed until you try to iterate over fields (line 5).
At that point, x.name.Equals(param.MethodData.Keys) fails because a string (x.name) cannot be compared to KeyCollection<string> (param.MethodData.Keys).
Essentially, you are getting the exception on unexpected line because of the deferred execution.
This is because the LINQ to SQL provider cannot translate the C# expression passed to the SingleOrDefault method into an SQL statement.
You can easily solve this problem by having LINQ to SQL execute the fields query before filtering it further with the SingleOrDefault method. This way the second operation will happen on the collection of objects in memory instead of the database:
var field = fields.ToList().SingleOrDefault(x => x.name.Equals(entry1.Key));
Related resources:
LINQ and Deferred Execution
Are you sure it's not this line that's failing:
var fields = _dc.fieldInfos.Where(x => x.name.Equals(param.MethodData.Keys));
as name isn't going to match a collection of strings.
which if so, I'd rewrite as:
var fields = _dc.fieldInfos.Where(x => param.MethodData.Keys.Contains(x.name));
entry1.key is a string, so I can't see why the line that you highlighted would fail with that message.
var fields = _dc.fieldInfos.Where(x => x.name.Equals(param.MethodData.Keys));
This retrieves (actually filters with deferred execution) fieldInfos by matching their name to a collection of keys. I don't know what fieldInfo.Name and param.MethodData.Keys mean, but i assume it's not what you want.
I'd suggest something along the lines of:
var pairs = param.MethodData.Join(_dc.fieldInfos,
kvp => kvp.Key,
fi => fi.name,
(kvp, fi) => new { Key = kvp.Key, FieldInfo = fi });
foreach (var pair in pairs)
param.MethodData[pair.Key] = pair.FieldInfo;
Im not exactly sure it's gonna work in Linq to SQL though.