Develop Reference

How to merge multiple list by id and get specific data?

i have 3 lists with common IDs. I need to group by object in one list, and extract data from other two. Will give example for more understanding
table for groupNames:
| Id | Name |
|--------------|
| 1 | Hello |
| 2 | Hello |
| 3 | Hey |
| 4 | Dude |
| 5 | Dude |
table for countId:
| Id | whatever |
|---------------|
| 1 | test0 |
| 1 | test1 |
| 2 | test2 |
| 3 | test3 |
| 3 | test4 |
table for lastTime:
| Id | timestamp |
|-----------------|
| 1 | 1636585230 |
| 1 | 1636585250 |
| 2 | 1636585240 |
| 3 | 1636585231 |
| 3 | 1636585230 |
| 5 | 1636585330 |
and I'm expecting result in list like this
| Name | whateverCnt | lastTimestamp |
|---------------------------------------|
| Hello | 3 | 1636585250 |
| Hey | 2 | 1636585231 |
| Dude | 0 | 1636585330 |
for now i had something like this, but it doesnt work
return groupNames
.GroupBy(x => x.Name)
.Select(x =>
{
return new myElem
{
Name = x.Name,
lastTimestamp = new DateTimeOffset(lastTime.Where(a => groupNames.Where(d => d.Name == x.Key).Select(d => d.Id).Contains(a.Id)).Max(m => m.timestamp)).ToUnixTimeMilliseconds(),
whateverCnt = countId.Where(q => (groupNames.Where(d => d.Name == x.Key).Select(d => d.Id)).ToList().Contains(q.Id)).Count()
};
})
.ToList();
Many thanks for any advice.

I think I'd mostly skip LINQ for this
class Thing{
public string Name {get;set;}
public int Count {get;set;}
public long LastTimestamp {get;set;}
}
...
var ids = new Dictionary<int, string>();
var result = new Dictionary<string, Thing>();
foreach(var g in groupNames) {
ids[g.Id] = g.Name;
result[g.Name] = new Whatever { Name = n };
}
foreach(var c in counts)
result[ids[c.Id]].Count++;
foreach(var l in lastTime){
var t = result[ids[l.Id]];
if(t.LastTimeStamp < l.Timestamp) t.LastTimeStamp = l.TimeStamp;
}
We start off making two dictionaries (you could ToDictionary this).. If groupNames is already a dictionary that maps id:name then you can skip making the ids dictionary and just use groupNames directly. This gives us fast lookup from ID to Name, but we actually want to colelct results into a name:something mapping, so we make one of those too. doing result[name] = thing always succeeds, even if we've seen name before. We could skip on some object creation with a ContainsKey check here if you want
Then all we need to do is enumerate our other N collections, building the result. The result we want is accessed from result[ids[some_id_value_here]] and it always exists if groupnames id space is complete (we will never have an id in the counts that we do not have in groupNames)
For counts, we don't care for any of the other data; just the presence of the id is enough to increment the count
For dates, it's a simple max algorithm of "if known max is less than new max make known max = new max". If you know your dates list is sorted ascending you can skip that if too..

In your example, the safest would be a list of the last specified object and just LINQ query the other arrays of objects for the same id.
So something like
public IEnumerable<SomeObject> MergeListsById(
IEnumerable<GroupNames> groupNames,
IEnumerable<CountId> countIds,
IEnumerable<LastTime> lastTimes)
{
IEnumerable<SomeObject> mergedList = new List<SomeObject>();
groupNames.ForEach(gn => {
mergedList.Add(new SomeObject {
Name = gn.Name,
whateverCnt = countIds.FirstOrDefault(ci => ci.Id == gn.Id)?.whatever,
lastTimeStamp = lastTimes.LastOrDefault(lt => lt.Id == gn.Id)?.timestamp
});
});
return mergedList;
}
Try it in a Fiddle or throwaway project and tweak it to your needs. A solution in pure LINQ is probably not desired here, for readability and maintainability sake.
And yes, as the comments say do carefully consider whether LINQ is your best option here. While it works, it does not always do better in performance than a "simple" foreach. LINQ's main selling point is and always has been short, one-line querying statements which maintain readability.

Well, having
List<(int id, string name)> groupNames = new List<(int id, string name)>() {
( 1, "Hello"),
( 2, "Hello"),
( 3, "Hey"),
( 4, "Dude"),
( 5, "Dude"),
};
List<(int id, string comments)> countId = new List<(int id, string comments)>() {
( 1 , "test0"),
( 1 , "test1"),
( 2 , "test2"),
( 3 , "test3"),
( 3 , "test4"),
};
List<(int id, int time)> lastTime = new List<(int id, int time)>() {
( 1 , 1636585230 ),
( 1 , 1636585250 ),
( 2 , 1636585240 ),
( 3 , 1636585231 ),
( 3 , 1636585230 ),
( 5 , 1636585330 ),
};
you can, technically, use the Linq below:
var result = groupNames
.GroupBy(item => item.name, item => item.id)
.Select(group => (Name : group.Key,
whateverCnt : group
.Sum(id => countId.Count(item => item.id == id)),
lastTimestamp : lastTime
.Where(item => group.Any(g => g == item.id))
.Max(item => item.time)));
Let's have a look:
Console.Write(string.Join(Environment.NewLine, result));
Outcome:
(Hello, 3, 1636585250)
(Hey, 2, 1636585231)
(Dude, 0, 1636585330)
But be careful: List<T> (I mean countId and lastTime) are not efficient data structures here. In the Linq query we have to scan them in order to get Sum and Max. If countId and lastTime are long, turn them (by grouping) into Dictionary<int, T> with id being Key

Return certain record based on criteria (2)

I asked this question previously, but missed a vital part of my problem.
Return certain record based on criteria
Take this list of results
Client | Date | YESorNO
-------------------------------
A1 | 01/01/2001 | NO
A1 | 01/01/2002 | NO
A1 | 01/01/2003 | YES
A1 | 01/01/2004 | NO
A1 | 01/01/2005 | NO
A1 | 01/01/2006 | NO
A1 | 01/01/2007 | YES
A1 | 01/01/2008 | YES
A1 | 01/01/2009 | YES
A2 | 01/01/2001 | NO
A2 | 01/01/2002 | NO
A2 | 01/01/2003 | YES
A2 | 01/01/2004 | NO
A2 | 01/01/2005 | YES
A2 | 01/01/2006 | YES
A3 | 01/01/2001 | NO
...etc...
The list is ordered chronologically and I cannot sort this is any other way other than descending / ascending.
I cannot sort for Yes | NO and find the First() or Last() as this won't give me the required value.
I want to be able to return the first 'YES' after all 'NO's have been accounted for, per Client.
In the above example for Client[A1] row 7 is the record I want returned (on 01/01/2007).
Client[A2] - row 5 (01/01/2005) ..etc
My code is as follows
var query =
(
from m in db.MyTable
where m.Criteria == XYZ
select new
{
Client = m.Client,
Date = m.Date,
YESorNO = m.YESorNO
}
).OrderBy(x => x.Date);
Using .FirstOrDefault(x => x.YesOrNO == "YES") returns the 3rd record.
User #RenéVogt advised that
var result = query.AsEnumerable()
.TakeWhile(x => x.YESorNO == "YES")
.LastOrDefault();
would get the job done and it does, but I forgot to add that the query will be returning many Clients and I need the first 'YES' for each Client, therefore the above code won't suffice.
Iterating over my results would be hugely time consuming and whilst that is a solution I would prefer this logic to be within the database query itself (if possible)
Many thanks

What you have to do is grouping by client,and then find the last YES of each one starting from the end. Something like this (ClientList is a List<>, you may have to change it depending on where is your data):
var query = ClientList.OrderBy(x => x.client).ThenBy(x => x.date).GroupBy(x => x.client);
foreach (var client in query)
{
var lastYES=client.Reverse().TakeWhile(x => x.YESorNO == "YES")
.LastOrDefault();
Console.WriteLine(String.Format("{0} {1}",client.Key,lastYES.date));
}
//Output: A1 01/01/2007 0:00:00
// A2 01/01/2005 0:00:00
Edit
Mansur Anorboev rightly suggested ordering by descending date, thus eliminating the need of Reverse, so the code would be:
var query = ClientList.OrderBy(x => x.client).ThenByDescending(x => x.date).GroupBy(x => x.client);
foreach (var client in query)
{
var lastYES=client.TakeWhile(x => x.YESorNO == "YES")
.LastOrDefault();
Console.WriteLine(String.Format("{0} {1}",client.Key,lastYES.date));
}
Edit 2
I still was not completly happy with my solution, as it is using a foreach. This does everything in one Linq command:
var query = ClientList.OrderBy(x => x.client)
.ThenByDescending(x => x.date)
.GroupBy(x => x.client, (key, g) => g.TakeWhile(x => x.YESorNO == "YES").LastOrDefault())
.ToList();
This returns a list with one element per client and with the correct date.

I can provide a little sql query
;WITH cte AS (
SELECT *, ROW_NUMBER() OVER (ORDER BY Client DESC) AS rn
FROM [dbo].[tblSkaterhaz]
)
,gte AS (
SELECT Client,max(rn) mx FROM cte
WHERE YesOrNo = 'NO'
GROUP BY Client
)
SELECT cte.* FROM gte
INNER JOIN cte on cte.Client = gte.Client and cte.rn = gte.mx + 1
Although it is not the required solution, but it yields the required result. You can create a stored proc and use it in your code.
NOTE: This is tested against the same table (and data) mentioned in question above
I hope this will be helpful for you.

Linq Convert or Pivot Rows to Columns [duplicate]

This question already has answers here:
Is it possible to Pivot data using LINQ?
(7 answers)
Closed 7 years ago.
I have the following data returned from a query:
+-------------------------------------+
| **SUM** **Account** |
+-------------------------------------+
| A & E FEE 182396 |
| CM FEE 108569 |
| CONTRACT VALUE 2256044 |
| OVERHEAD 197397 |
+-------------------------------------+
I need to select into object Properties AeFee, CmFee, ContractValue, and Overhead which would essentially make the properties the column headings so I need the data to look like:
+--------+--------+---------------+----------+
| AeFee | CmFee | ContractValue | Overhead |
+--------+--------+---------------+----------+
| 182396 | 108569 | 2256044 | 197397 |
+--------+--------+---------------+----------+
I tried using a subselect to select something like this (pseudo code)
Select(s => new
{
AeFee = Sum if [Account] == "A & E FEE"
}
This link show exactly what I'm trying to do with a SQL pivot. Converts rows into columns. SQL Pivot
Any ideas?

I always love a question where the answer is Aggregate.
var result = data.Aggregate(
// base object to store
new { A = 0, CM = 0, CO = 0, O = 0},
// add in an element
(a,d) => {
switch(d.sum)
{
case "A & E FEE":
a.A += d.Account;
break;
case "CM FEE":
a.CM += d.Account;
break;
//etc
}
return a;
});
Note you can also - not know the possible values for d.sum in this way -- you would need to use and expando object. That would look something like this (NOT TESTED)
var result = data.Aggregate(
// base object to store
new ExpandoObject()
// add in an element
(a,d.sum.Replace(" ","_") => {
a[d.sum.Replace(" ","_")] += d.Account;
return a;
});
This won't work if your sum strings have values which are not valid in property identifiers.

This produced the results I was looking for
var accounts = new List<string>()
{
"A & E FEE",
"cm fee",
"contract value",
"overhead"
};
var commission = PJPTDSUMs
.Where(p => p.Project.StartsWith("b29317")
&& accounts.Contains(p.Acct)
&& !p.Pjt_entity.StartsWith("05"))
.GroupBy(c => c.Project)
.Select(g => new
{
AeFee = g.Where(p => p.Acct == "A & E FEE").Sum(s => s.Eac_amount),
CmFee = g.Where(p => p.Acct == "cm fee").Sum(s => s.Eac_amount),
ContractValue = g.Where(p => p.Acct == "contract value").Sum(s => s.Eac_amount),
Overhead = g.Where(p => p.Acct == "overhead").Sum(s => s.Eac_amount),
});
+--------+--------+---------------+----------+
| AeFee | CmFee | ContractValue | Overhead |
+--------+--------+---------------+----------+
| 182396 | 108569 | 2256044 | 197397 |
+--------+--------+---------------+----------+

Slow Performance of Linq Where statement

I have a List of Objects (roughly 100k) that is must iterate upon in order to produce a Dictionary.
however the code is performing very slowly, specifically on one line
public class Item{
public int ID;
public int Secondary_ID;
public string Text;
public int Number;
}
Data Looks something like (100k lines)
ID | Secondary_ID | Text | Number
1 | 1 | "something" | 3
1 | 1 | "something else"| 7
1 | 1 | "something1" | 4
1 | 2 | "something2" | 344
2 | 3 | "something3" | 74
2 | 3 | "something4" | 1
and i would like it to look like this when finished. (any collection will do to be honest)
Dictionary<int, string>
Key | Value
(secondary_ID) | (Text : Number)
1 | "Something : 3, Something else : 7, Something1 : 4"
2 | "Something2 : 344"
3 | "Something3 : 74, Something4 : 1"
My code currently works like this ListAll contains all data.
var Final=new Dictionary<int, string>();
var id1s=ListAll.Select(x => x.ID).Distinct().ToList();
foreach(var id1 in id1s) {
var shortList=ListAll.Where(x => x.ID==id1).ToList(); //99% of time spent is here
var id2s=shortList.Select(x => x.Secondary_ID).Distinct().ToList();
foreach(var id2 in id2s) {
var s=new StringBuilder();
var items=shortList.Where(x => x.Secondary_ID==id2).ToList();
foreach(var i in items) {
s.Append(String.Format("{0} : {1}", i.Text, i.Number));
}
Final.Add(id2, s.ToString());
}
}
return Final;
now the output is correct however as stated in the above comment, this takes an incredibly long time to process (90 seconds - certainly more than i am comfortable with) and was wondering if there is a faster way of achieving this.
This code is only going to be used once so is not really a normal usage and normally I would ignore it for that reason, but was wondering for learning purposes.

Here's what I would do (untested, but hopefully you get the idea):
var final = ListAll.GroupBy(x => x.Secondary_ID)
.ToDictionary(x => x.Key, x => String.Join(", ",
x.Select(y => String.Format("{0} : {1}",
y.Text, y.Number)))
This first groups by Secondary_ID using GroupBy, then puts the result into a dictionary using ToDictionary.
The GroupBy will group your data into the following groups:
Key = 1:
ID | Secondary_ID | Text | Number
1 | 1 | "something" | 3
1 | 1 | "something else"| 7
1 | 1 | "something1" | 4
Key = 2:
ID | Secondary_ID | Text | Number
1 | 2 | "something2" | 344
Key = 3:
ID | Secondary_ID | Text | Number
2 | 3 | "something3" | 74
2 | 3 | "something4" | 1
Then the .ToDictionary method:
Selects the key as x.Key (the key we grouped on, i.e. Secondary_ID).
Selects the result of a String.Join operation as the value. What is being joined is the collection of "Text : Number" from the elements inside that group - x.Select(y => String.Format("{0} : {1}", y.Text, y.Number).

A much more efficient (and even easier to write) method of grouping the items by ID is to use GroupBy.
var query = ListAll.GroupBy(x => x.Secondary_ID)
.ToDictionary(group => group.Key,
group => string.Join(", ",
group.Select(item => string.Format("{0} : {1}",item.Text , item.Number))),
//consider refactoring part of this line out to another method
});
As for the reason that your code is so slow, you're searching through the entire list for each distinct ID. That's an O(n^2) operation. GroupBy doesn't do that. It uses a hash based structure internally, based on whatever you're grouping on, so that it can quickly (in O(1) time) find the bucket that any given item belongs in, as opposed to the O(n) time it takes your method.

First, remove everywhere ToList(), it should becomes faster; because ToList() performs eager evaluation.
I think what your code expects to do is:
var Final=new Dictionary<int, string>();
foreach(var x in ListAll)
if(Final.ContainsKey(x.Secondary_ID))
Final[x.Secondary_ID]+=String.Format(", {0} : {1}", x.Text, x.Number);
else
Final.Add(x.Secondary_ID, String.Format("{0} : {1}", x.Text, x.Number));
return Final;
A Dictionary cannot contain a duplicate key, so it's no matter here you use ID or Secondary_ID, if your Secondary_ID must be in the range of existing ID; and you even do not need Distinct() in the code.
By doing some simplification, original code would be:
foreach(var id1 in ListAll.Select(x => x.ID).Distinct()) {
foreach(var id2 in ListAll.Where(x => x.ID==id1).Select(x => x.Secondary_ID).Distinct()) {
var s=new StringBuilder();
foreach(var i in ListAll.Where(x => x.ID==id1).Where(x => x.Secondary_ID==id2)) {
s.Append(String.Format("{0} : {1}", i.Text, i.Number));
}
Final.Add(id2, s.ToString());
}
}

How to SUM up results by column value in db query result

My database has a sales table with entries like so:
_____________________________________
| id | title_id | qty |
-------------------------------------
| 0 | 6 | 10 |
-------------------------------------
| 1 | 5 | 5 |
-------------------------------------
| 2 | 6 | 2 |
-------------------------------------
Title_id is Foreign key pointing to Titles table which is as follows:
_____________________________________
| id | title_id | title |
-------------------------------------
| 0 | 5 | Soda |
-------------------------------------
| 1 | 6 | Coffee |
-------------------------------------
I want to find top 5 sold products wich means i need to calculate the qty value for each product for all it's entried in sales table then order the result by qty in descending order and limit the select to 5.
However I'm new to C# ASP.NET and somewhat new to SQL. I dont know how to do this with LINQ.
This is my code so far:
var getIds = (from sale in db.sales
join tit in db.titles on sale.title_id equals tit.title_id
group sale by sale.qty into result
orderby result.Sum(i => i.qty) descending
select new Publication
{
PubID = sales.title_id, Title = tit.title
}
).Take(5);

Assuming you have a navigation property Sale.Title, something like this should do:
var tops =
db.Sales
.GroupBy( o => o.Title )
.Select( o => new { Title = o.Key, Sum = o.Sum( x => x.Quantity ) } )
.OrderByDescending( o => o.Sum )
.Take( 5 )
.ToList();
tops is then a list of an anonymous type with two properties: the Title object and the sum of the quantities.
You can then get the values like this:
foreach( var top in tops )
{
int titleId = top.Title.title_id;
string title = top.Title.title;
int sumOfQuantities = top.Sum;
...
If you just want the top Title objects, can can select them like this:
List<Title> topTitles = tops.Select( o => o.Title ).ToList();

var result= (from p in sales
let k = new
{
Name = p.Name
}
group p by k into t
orderby Name descending
select new
{
Name = t.Name,
Qty = t.Sum(p => p.Qty)
}).Take(5);

If the entries in the Sales table are more than one per item (ie: in your example you have 'Soda' 10 + 'Soda' 2, then you need to GroupBy(), using the name as the key (or it's related id if it's in another table), but not the qty.
var topSales = db.sales.GroupBy(x => x.title)
.Select(g => new
{
Title = g.Key,
Qty = g.Sum(x => x.qty)
})
.OrderByDescending(x => x.Qty)
.Select(x => new Publication
{
PubID = x.Title.title_id,
Title = x.Title.title1
})
.Take(5)
.ToList();
Note that I've omitted the join statement assuming that you have a foreign key between sales.title_id -> title.id, and you are using LINQ to SQL. Also note that I've avoided using the query syntax in favor of the chained method syntax, I think it's much clear in this use case (although not always true, ie: cross-joins).
Also, SQL and LINQ have some similarities but don't let the names of clauses/methods fool you, LINQ is not SQL, IMHO, Microsoft just tried to make people comfortable by making it look similar ;)
EDIT: fixed GroupBy()

var result= (from p in sales
let k = new
{
Name = p.Name
}
group p by k into t
select new
{
Name = t.Name,
Qty = t.Sum(p => p.Qty)
}).OrderByDescending(i => i.Qty).Take(5);

You need to look at GroupBy; this will give you what you need
http://code.msdn.microsoft.com/101-LINQ-Samples-3fb9811b