I have this function below that takes a list of id's and searches the DB for the matching persons.
public IQueryable<Person> GetPersons(List<int> list)
{
return db.Persons.Where(a => list.Contains(a.person_id));
}
The reason I need to split this into four queries is because the query can't take more than 2100 comma-separated values:
The incoming tabular data stream (TDS) remote procedure call (RPC) protocol stream is incorrect. Too many parameters were provided in this RPC request. The maximum is 2100.
How can I split the list into 4 pieces and make a query for each list. Then join the results into one list of persons?
Solved
I don't want to post it as an own answer and take cred away from #George Duckett's answer, just show the solution:
public IQueryable<Person> GetPersons(List<int> list)
{
var persons = Enumerable.Empty<Person>().AsQueryable<Person>();
var limit = 2000;
var result = list.Select((value, index) => new { Index = index, Value = value })
.GroupBy(x => x.Index / limit)
.Select(g => g.Select(x => x.Value).ToList())
.ToList();
foreach (var r in result)
{
var row = r;
persons = persons.Union(db.Persons.Where(a => row.Contains(a.person_id)));
}
return persons;
}
See this answer for splitting up your list: Divide a large IEnumerable into smaller IEnumerable of a fix amount of item
var result = list.Select((value, index) => new { Index = index, Value = value})
.GroupBy(x => x.Index / 5)
.Select(g => g.Select(x => x.Value).ToList())
.ToList();
Then do a foreach over the result (a list of lists), using the below to combine them.
See this answer for combining the results: How to combine Linq query results
I am not sure why you have a method like this. What exactly are you trying to do. Anyway you can do it with Skip and Take methods that are used for paging.
List<Person> peopleToReturn = new List<Person>();
int pageSize = 100;
var idPage = list.Skip(0).Take(pageSize).ToList();
int index = 1;
while (idPage.Count > 0)
{
peopleToReturn.AddRange(db.Persons.Where(a => idPage.Contains(a.person_id)).ToList());
idPage = list.Skip(index++ * pageSize).Take(pageSize).ToList();
}
Related
How to use OrderBy for shaping output in the same order as per the requested distinct list
public DataCollectionList GetLatestDataCollection(List<string> requestedDataPointList)
{
var dataPoints = _context.DataPoints.Where(c => requestedDataPointList.Contains(c.dataPointName))
.OrderBy(----------) //TODO: RE-ORDER IN THE SAME ORDER AS REQUESTED requestedDataPointList
.ToList();
dataPoints.ForEach(dp =>
{
....
});
}
Do the sorting on the client side:
public DataCollectionList GetLatestDataCollection(List<string> requestedDataPointList)
{
var dataPoints = _context.DataPoints.Where(c => requestedDataPointList.Contains(c.dataPointName))
.AsEnumerable()
.OrderBy(requestedDataPointList.IndexOf(c.dataPointName));
foreach (var dp in dataPoints)
{
....
});
}
NOTE: Also, I don't think ToList().ForEach() is ever better than foreach ().
It think the fastest method is to join the result back with the request list. This makes use of the fact that LINQ's join preserves the sort order of the first list:
var dataPoints = _context.DataPoints
.Where(c => requestedDataPointList.Contains(c.dataPointName))
.ToList();
var ordered = from n in requestedDataPointList
join dp in dataPoints on n equals dp.dataPointName
select dp;
foreach (var dataPoint in ordered)
{
...
}
This doesn't involve any ordering, joining does it all, which will be close to O(n).
Another fast method consists of creating a dictionary of sequence numbers:
var indexes = requestedDataPointList
.Select((n, i) => new { n, i }).ToDictionary(x => x.n, x => x.i);
var ordered = dataPoints.OrderBy(dp => indexes[dp.dataPointName]);
Below i have a snippet of code which outputs a list of Appointments based on clients, some clients can have more than one appointment but the latest one is the one that needs to be outputted for said client
the output is not grouping at all and for some reason i cannot figure why the heck not
foreach (ClientRecord client in clients)
{
List<ReturnRecord> records = db.Appointments
.AsNoTracking()
.Include(rec => rec.Property)
.Include(rec => rec.Property.Address)
.Include(rec => rec.AppointmentType)
.ToList()
.Where(rec => rec.ClientID == client.ID)
.Select(rec => new ReturnRecord
{
ClientName = $"{client.FirstNames} {client.Surnames}",
PropertyAddress = $"{rec.Property.Address.FormattedAddress}",
AppStatus = $"{rec.AppointmentStatus.Name}",
StockStatus = $"{rec.Property.Stocks.FirstOrDefault().StockStatus.Name}",
LastUpdated = rec.LastUpdated
})
.ToList();
returnList.AddRange(records);
}
returnList.GroupBy(rec => rec.PropertyAddress);
return Ok(returnList);
here is an attachment of the screen grab of the output
You need to assign result of GroupBy() to variable:
returnList = returnList.GroupBy(rec => rec.PropertyAddress).ToList();
Make sure to actually use the new IEnumerable that the .GroupBy() Method returned.
If you want to return a List you need to use a workaround:
Get the IEnumerable<IGrouping<int, ReturnRecord>> from the .GroupBy()
Use .SelectMany() to select all elements and save them into an IEnumerable
Now you can convert your IEnumerable into a List with .List()
Example:
// Longer Alternative
IEnumerable<IGrouping<int, ReturnRecord>> groups = resultList
.GroupBy((rec => rec.PropertyAddress);
IEnumerable<ReturnRecord> result = groups.SelectMany(group => group);
List<ReturnRecord> listResult = result.ToList();
return Ok(listResult);
// Shorter Alternative
IEnumerable<IGrouping<int, ReturnRecord>> groups = resultList
.GroupBy((rec => rec.PropertyAddress);
IEnumerable<ReturnRecord> result = groups.SelectMany(group => group);
return Ok(result.ToList());
The below code groups the result (a List of ClassTypeObject with 500,000 items) into List<a> type.
The GroupBy takes around 40 to 50 sec when executed. Is there any way to optimize this?
var groupByTest = result.
GroupBy(g => new
{
First = g.Field1
}).
Select(gp => new
{
gp.Key.Field1,
InnerList = result.Where(x => x.Field1 == gp.Key.Field1).ToList()
}).ToList();
You are selecting InnerList from non-grouped collection i.e. result that's why your query is taking time. You can change the inner query assignment as
InnerList = gp.ToList()
as gp is already grouped based on Field1.
Full Code
var groupByTest = result.
GroupBy(g => new
{
First = g.Field1
}).
Select(gp => new
{
gp.Key.Field1,
InnerList = gp.ToList()
}).ToList();
The way this query is written InnerList ends up containing just the items in the group. In its current form, the original source is scanned once for each group key. The equivalent:
var groupByTest = result.GroupBy(g => g.Field1)
.Select(gp => new {
Field1=gp.Key,
InnerList = gp.ToList()})
.ToList();
Would scan the source only once.
Once this is fixed, the query can be parallelized easily with AsParallel()
var groupByTest = result.AsParallel()
.GroupBy(g => g.Field1)
.Select(gp => new {
Field1=gp.Key,
InnerList = gp.ToList()})
.ToList();
This will use all cores in the machine to partition the data, group them and construct the final list
I'm new to c# so go easy on me. Anyways, I made a list of numbers
List<int> numbers = new List<int>();
and I want to make a list of each number and its count/frequency.
var grouped = numbers
.GroupBy(i => i)
.Select(i => new { Number = i.Key, Count = i.Count() });
In locals, I can see the group, which has an IEnumerator interface with all of the numbers and their count values image of what I'm talking about. So is there a way to make a list with the numbers and their frequency/count?
Thank you.
IEnumerable<T> is a sequence so it doesn't own a count. But Enumerable.Count is an extension method of IEnumerable<T>
That is, you don't necessarily need to convert an IEnumerable<T> into a List<T>:
var grouped = numbers
.GroupBy(i => i)
.Select(i => new { Number = i.Key, Count = i.Count() });
var groupedCount = grouped.Count();
// You may iterate grouped
foreach(var value in grouped)
{
Console.WriteLine($"{value.Number} {value.Count}");
}
If you really need List<T> semantics, you just need to call Enumerable.ToList:
var grouped = numbers
.GroupBy(i => i)
.Select(i => new { Number = i.Key, Count = i.Count() })
.ToList();
In the other hand, you may directly convert everything into a string as follows:
var groupText = string.Join("\n", numbers
.GroupBy(i => i)
.Select(i => $"Number: {i.Key} Count: {i.Count()}"))
To get a list, you just need to call ToList(), for example:
var grouped = numbers
.GroupBy(i => i)
.Select(i => new { Number = i.Key, Count = i.Count() })
.ToList();
However, you really don't need to do that, you can simply loop over the enumerable as it stands:
foreach(var item in grouped)
{
Console.WriteLine($"{item.Number} occurs {item.Count} times");
}
Sounds like you want ToDictionary with the number as key and the frequency as value:
var grouped = numbers
.GroupBy(i => i)
.Select(i => new { Number = i.Key, Count = i.Count() })
.ToDictionary(x => x.Number, x => x.Count);
Now you can easily print every number and its frequency by looping the dictionary.
In fact you don´t even need neither ToDictionary nor your Select, as the IGrouping returned from GroupBy also derives from IEnumerable which is why you can iterate over it.
foreach(var g in grouped = numbers.GroupBy(i => i))
{
var number = g.Key;
var freq = g.Count();
}
I am trying to get the sum of the value from list of list using linq ?my data is as below code
List<List<string>> allData = new List<List<string>>();
using (StreamReader reader = new StreamReader(path))
{
while (!reader.EndOfStream)
{
List<string> dataList;
dataList = reader.ReadLine().Split('|').ToList();
allData.Add(dataList);
}
}
which gives me data in allData as below
allData-->[0]-->[0]-'name1'
[1]-'sub'
[2]-'12'
[1]-->[0]-'name2'
[1]-'sub'
[2]-'15'
[2]-->[0]-'name1'
[1]-'sub2'
[2]-'15'
//and so on ....
i have applied group by that gives me grouping by the name but i am not able to figure out how to get the sum of the marks for each name ?
var grouped = allData.GroupBy(x => x[0]);
after this i get all matching name grouped into one but now how to get sum of the marks for that group ? any help would be great ?
Output should be name1=27 and name2=15 and so on.
Not sure if you want to get the sum of every group or the total. If it's the total then this should do the trick
var sum = allData.Sum(x => Int32.Parse(x[2]));
If it's per key then try the following
var all = allData
.GroupBy(x => x[0])
.Select(x => x.Sum(y => Int32.Parse(y[2]));
var grouped = allData.GroupBy(x => x[0])
.Select(g => new
{
Name = g.Key,
Sum = g.Sum(x => int.Parse(x[2]))
});
It will return an anonymous type instance for each group, with two properties: Name with your grouping key and Sum with sum of marks.
Sticking as much as possible to the LINQ query language:
var grouped = from d in allData
group d by i[0] into g
select new
{
Name = g.Key,
Sum = g.Sum(i => int.Parse(i[2]))
};
This will give you parallel List with each name and the count of how many times each name occurs.
var names = grouped.Select(s => s.Key).ToList();
var nameCount = grouped.Select(s => s.Count()).ToList();
Also... you may want to add this when assigning alldata to grouped. I use this to get a List from greatest to least amount of occurrences.
var grouped = allData.GroupBy(x => x[0]).OrderByDescending(i => i.Count());