Develop Reference

Left outer join to include null values linq lambda

I have one table in database named Balance and a list of dates as follows:
List<string> allDates = { "2021-01-02", "2021-01-03", "2021-01-04" }
Balance table:
Id, Amount, BalanceDate
1, 233, "2021-01-02"
2, 442, "2021-01-03
I need to fetch the records in Balance table with amount 0 for the missing dates. For example:
233, "2021-01-02"
442, "2021-01-03"
0, "2021-01-04"
I have tried the following:
balnces.GroupJoin(allDates,
balance => balance.Date,
d => d,
(balance, d) => balance);
But the records are still the same (only the ones in the balance table)

Given a data structure from database:
private class balance
{
public int id { get; set; }
public double amount { get; set; }
public string date { get; set; }
}
You get your data as you want (this is only a mock-up)
List<string> allDates = new List<string> { "2021-01-02", "2021-01-03", "2021-01-04" };
List<balance> balances = new List<balance>();
balances.Add(new balance { id = 1, amount = 233 , date = "2021-01-02" });
balances.Add(new balance { id = 2, amount = 442, date = "2021-01-03" });
you can get your desired result this way:
List<balance> result = allDates.Select(d=>
new balance {
amount =
balances.Any(s=> s.date == d)?
balances.FirstOrDefault(s => s.date == d).amount:0,
date = d
}).ToList();
If your default contains a 0 in amount instead a null, you can skip the .Any check

Assumption
Balance query had been materialized and data are returned from the database.
Solution 1: With .DefaultIfEmpty()
using System.Linq;
var result = (from a in allDates
join b in balances on a equals b.Date.ToString("yyyy-MM-dd") into ab
from b in ab.DefaultIfEmpty()
select new { Date = a, Amount = b != null ? b.Amount : 0 }
).ToList();
Sample Program for Solution 1
Solution 2: With .ToLookup()
var lookup = balances.ToLookup(x => x.Date.ToString("yyyy-MM-dd"));
var result = (from a in allDates
select new
{
Date = a,
Amount = lookup[a] != null && lookup[a].Count() > 0 ? lookup[a].First().Amount : 0
}
).ToList();
Sample Program for Solution 2

EF core query group by date and subquery

I have the following table:
CREATE TABLE "OrderStatusLogs" (
"Id" UNIQUEIDENTIFIER NOT NULL,
"OrderId" UNIQUEIDENTIFIER NOT NULL,
"Status" INT NOT NULL,
"StartDateTime" DATETIMEOFFSET NOT NULL,
"EndDateTime" DATETIMEOFFSET NULL DEFAULT NULL,
PRIMARY KEY ("Id"),
FOREIGN KEY INDEX "FK_OrderStatusLogs_Orders_OrderId" ("OrderId"),
CONSTRAINT "FK_OrderStatusLogs_Orders_OrderId" FOREIGN KEY ("OrderId") REFERENCES "Orders" ("Id") ON UPDATE NO_ACTION ON DELETE CASCADE
)
;
For the following entity:
[DebuggerDisplay(nameof(OrderStatusLog) + " {Status} {StartDateTime} - {EndDateTime}" )]
public class OrderStatusLog
{
public Guid Id { get; set; }
public Guid OrderId { get; set; }
public OrderStatus Status { get; set; }
public DateTimeOffset StartDateTime { get; set; }
public DateTimeOffset? EndDateTime { get; set; }
}
public enum OrderStatus
{
Unknown = 0,
Pending = 1,
Processing = 2,
Shipping = 3,
}
And i'm trying to generate a report which should show how many orders are set to a certain state for a given range.
For example, for the month oktober, we'd have the range 1 to 31 oktober.
The desired output would be something like this:
1/10/2021 Pending 21 orders
1/10/2021 Processing 23 orders
1/10/2021 Shipping 33 orders
1/10/2021 Unknown 0 orders
...
31/10/2021 Pending 1 orders
31/10/2021 Processing 3 orders
31/10/2021 Shipping 44 orders
31/10/2021 Unknown 5 orders
I'm having some difficulties writing a query in EF that would give me the right output. I can get things to work, but only client-side. I'm trying to make this work in the database instead.
So far i tried:
var logsByDayAndOrderId = orderStatusLogs.GroupBy(c => new { c.StartDateTime.Date, c.OrderId }, (key, values) => new
{
key.Date,
key.OrderId,
MaxStartDateTime = values.Max(x => x.StartDateTime)
});
var list = logsByDayAndOrderId.ToList();
var statusByDayAndOrderId = logsByDayAndOrderId.Select(c => new
{
c.Date,
c.OrderId,
orderStatusLogs.FirstOrDefault(x => x.StartDateTime == c.MaxStartDateTime && x.OrderId == c.OrderId).Status
});
//var statusByDayAndOrderId = logsByDayAndOrderId.Join(orderStatusLogs.def, inner => new { inner.OrderId, StartDateTime = inner.MaxStartDateTime }, outer => new { outer.OrderId, outer.StartDateTime }, (inner,outer) => new
//{
// inner.Date,
// inner.OrderId,
// outer.Status
//}); // TODO rem this query gives more results because of the join. we need an Outer join - but i could not get that to work. the version with select above works better, but then it does not use join so it may be slow(er).
var list1 = statusByDayAndOrderId.ToList();
var groupBy = statusByDayAndOrderId
.GroupBy(c => new { c.Date, c.Status })
.Select(c => new { c.Key.Date, c.Key.Status, Count = c.Count() });
var list2 = groupBy.ToList();
Another attempt:
var datesAndOrders = orderStatusLogs
.GroupBy(c => new { c.StartDateTime.Date, c.OrderId }, (key, values) => key);
var ordersByDateAndActiveStatusLog = orderStatusLogs
.Select(c => new
{
c.StartDateTime.Date,
c.OrderId,
ActiveStatusForDate = orderStatusLogs
.OrderByDescending(x => x.StartDateTime)
.FirstOrDefault(x => x.OrderId == c.OrderId && x.StartDateTime.Date == c.StartDateTime.Date)
.Status
});
var list = ordersByDateAndActiveStatusLog.ToList();
var orderCountByDateAndStatus = ordersByDateAndActiveStatusLog
.GroupBy(c => new { c.Date, c.ActiveStatusForDate }, (key, values) => new
{
key, count = values.Count()
});
var list1 = orderCountByDateAndStatus.ToList();
Both of these fail because of Cannot use an aggregate or a subquery in an expression used for the group by list of a GROUP BY clause..
This makes sense.
I'm hoping for someone that could help write a Linq query that generates the right data using ef core.
Notes:
I Solely use the fluent query syntax
I Have more places where i'd like to get data for each day so any other info or tips and tricks are welcome
I use net core 5 with ef core 5.0.11 with a MSSQL database

I would suggest to use EF Core extension linq2db.EntityFrameworkCore which has ability to work with local (in-memory) collections in database queries. Disclaimer: i'm one of the creators.
At first define function which generates days sequence:
public static IEnumerable<DateTime> GenerateDays(int year, int month)
{
var start = new DateTime(year, month, 1);
var endDate = start.AddMonths(1);
while (start < endDate)
{
yield return start;
start = start.AddDays(1);
}
}
Then we can use generated sequence in LINQ Query:
var days = GenerateDays(2021, 10).ToArray();
using var dc = ctx.CreateLinqToDbConnection();
var totalsQuery =
from d in days.AsQueryable(dc)
from l in orderStatusLogs.Where(l =>
(l.EndDateTime == null || l.EndDateTime >= d) && l.StartDateTime < d.AddDays(1))
.DefaultIfEmpty()
group l by new { Date = d, l.Status } into g
into g
select new
{
g.Key.Date,
g.Key.Status,
Count = g.Sum(x => x == null ? 0 : 1),
};
var result = totalsQuery.ToList();
The following SQL should be generated:
SELECT
[d].[item],
[e].[Status],
Sum(IIF([e].[OrderID] IS NULL, 0, 1))
FROM
(VALUES
('2021-05-01T00:00:00'), ('2021-05-02T00:00:00'),
('2021-05-03T00:00:00'), ('2021-05-04T00:00:00'),
('2021-05-05T00:00:00'), ('2021-05-06T00:00:00'),
('2021-05-07T00:00:00'), ('2021-05-08T00:00:00'),
('2021-05-09T00:00:00'), ('2021-05-10T00:00:00'),
('2021-05-11T00:00:00'), ('2021-05-12T00:00:00'),
('2021-05-13T00:00:00'), ('2021-05-14T00:00:00'),
('2021-05-15T00:00:00'), ('2021-05-16T00:00:00'),
('2021-05-17T00:00:00'), ('2021-05-18T00:00:00'),
('2021-05-19T00:00:00'), ('2021-05-20T00:00:00'),
('2021-05-21T00:00:00'), ('2021-05-22T00:00:00'),
('2021-05-23T00:00:00'), ('2021-05-24T00:00:00'),
('2021-05-25T00:00:00'), ('2021-05-26T00:00:00'),
('2021-05-27T00:00:00'), ('2021-05-28T00:00:00'),
('2021-05-29T00:00:00'), ('2021-05-30T00:00:00'),
('2021-05-31T00:00:00')
) [d]([item])
LEFT JOIN [OrderStatusLogs] [e] ON ([e].[EndDateTime] IS NULL OR [e].[EndDateTime] >= [d].[item]) AND [e].[StartDateTime] < DateAdd(day, 1, [d].[item])
GROUP BY
[d].[item],
[e].[Status]

LINQ Join usage data from two data sets into one

What I have:
Two lists of the following model:
int SubscriptionId
int ItemId
double Usage
double EffectiveRate
string ResourceName
string UnitOfMeasure
The first contains usage data of the last month like this:
SubscriptionId ItemId Usage EffectiveRate ResourceName UnitOfMesaure
_________________________________________________________________________
1 1 2 2,75 R1 U1
1 2 3 1,50 R2 U2
The seconds contains usage data of the current month like this:
SubscriptionId ItemId Usage EffectiveRate ResourceName UnitOfMesaure
_________________________________________________________________________
1 1 5 2,75 R1 U1
1 3 2 1,50 R3 U3
What I want:
This should be merge in a list like this:
SubscriptionId ItemId UsageThis UsageLast EffRate ResName UOM
_________________________________________________________________________
1 1 5 2 2,75 R1 U1
1 2 0 3 1,50 R2 U2
1 3 2 0 1,50 R3 U3
What I have:
//data for both months available
if (resourcesThisMonth.Any() && resourcesLastMonth.Any())
{
//join both months
resources = from resourceLastMonth in resourcesLastMonth
join resourceThisMonth in resourcesThisMonth
on new { resourceLastMonth.SubscriptionId, resourceLastMonth.ItemId } equals new { resourceThisMonth.SubscriptionId, resourceThisMonth.ItemId }
select new Resource
{
SubscriptionId = resourceThisMonth.SubscriptionId,
ItemId = resourceThisMonth.ItemId,
UsageThisMonth = resourceThisMonth.Usage,
UsageLastMonth = resourceLastMonth.Usage,
EffectiveRate = resourceThisMonth.EffectiveRate,
ResourceName = resourceThisMonth.ResourceName,
UnitOfMeasure = resourceThisMonth.UnitOfMeasure
};
//resources only last month available
var resourcesOnlyLastMonth = resourcesLastMonth.Where(r => !resourcesThisMonth.Where(s => s.ItemId == r.ItemId && s.SubscriptionId == r.SubscriptionId).Any())
.Select(r => new Resource
{
SubscriptionId = r.SubscriptionId,
ItemId = r.ItemId,
UsageThisMonth = 0.0,
UsageLastMonth = r.Units,
EffectiveRate = r.EffectiveRate,
ResourceName = r.ResourceName,
UnitOfMeasure = r.UnitOfMeasure
});
//resources only this month available
var resourcesOnlyThisMonth = resourcesThisMonth.Where(r => !resourcesLastMonth.Where(s => s.ItemId == r.ItemId && s.SubscriptionId == r.SubscriptionId).Any())
.Select(r => new Resource
{
SubscriptionId = r.SubscriptionId,
ItemId = r.ItemId,
UsageThisMonth = r.Usage,
UsageLastMonth = 0.0,
EffectiveRate = r.EffectiveRate,
ResourceName = r.ResourceName,
UnitOfMeasure = r.UnitOfMeasure
});
//union data
resources = resources.Union(resourcesOnlyLastMonth);
resources = resources.Union(resourcesOnlyThisMonth);
}
//data for last month available
else if (resourcesLastMonth.Any())
{
resources = from resource in resourcesLastMonth
select new Resource
{
SubscriptionId = resource.SubscriptionId,
ItemId = resource.ItemId,
UsageThisMonth = 0.0,
UsageLastMonth = resource.Usage,
EffectiveRate = resource.EffectiveRate,
ResourceName = resource.ResourceName,
UnitOfMeasure = resource.UnitOfMeasure
};
}
//data for this month available
else if (resourcesThisMonth.Any())
{
resources = from resource in resourcesThisMonth
select new Resource
{
SubscriptionId = resource.SubscriptionId,
ItemId = resource.ItemId,
UsageThisMonth = resource.Usage,
UsageLastMonth = 0.0,
EffectiveRate = resource.EffectiveRate,
ResourceName = resource.ResourceName,
UnitOfMeasure = resource.UnitOfMeasure
};
}
//no data available
else
{
resources = new List<Resource>();
}
Problem:
This is very much code - should be less, any possible solutions failed so far
Thanks for helping!

public class ExampleClass
{
public int Id1 { get; set; }
public int Id2 { get; set; }
public int Usage { get; set; }
public int UsageThis { get; set; }
public int UsageLast { get; set; }
}
List<ExampleClass> listThisMonth = new List<ExampleClass>
{
new ExampleClass{Id1=1, Id2=1,Usage=7, UsageThis=1, UsageLast=0},
new ExampleClass{Id1=2, Id2=2,Usage=4, UsageThis=2, UsageLast=0},
new ExampleClass{Id1=3, Id2=3,Usage=1, UsageThis=3, UsageLast=0},
};
List<ExampleClass> listLastMonth = new List<ExampleClass>
{
new ExampleClass{Id1=1, Id2=1,Usage=3, UsageThis=1, UsageLast=1},
new ExampleClass{Id1=4, Id2=4,Usage=3, UsageThis=4, UsageLast=3},
new ExampleClass{Id1=2, Id2=2,Usage=1, UsageThis=8, UsageLast=6},
};
var result = listThisMonth.Select(a=>new {value=a, list=1})
.Union(listLastMonth.Select(a => new { value = a, list = 2 }))
.GroupBy(a => new { Id1 = a.value.Id1, Id2 = a.value.Id2 })
.Select(x => new ExampleClass
{
Id1 = x.Key.Id1,
Id2 = x.Key.Id2,
UsageThis = x.Any(o => o.list == 1) ? x.First(o => o.list == 1).value.Usage : 0,
UsageLast = x.Any(o => o.list == 2) ? x.First(o => o.list == 2).value.Usage : 0,
Usage = x.Sum(o=>o.value.Usage)
}).ToList();
//id1 id2 current last sum
//1 1 7 3 10
//2 2 4 1 5
//3 3 1 0 1
//4 4 0 3 3

It looks to me that what you're looking for is a full outer join. Unfortunately, it looks like LINQ doesn't have a construct for that. So, there are a few options: LINQ - Full Outer Join
For your scenario, it looks like you have some redundant code. You should be able to do the union using two outer joins to get the correct result set. For example:
// Left join the current month with the last month
var currentMonth =
from current in resourcesThisMonth
join last in resourcesLastMonth on new { current.SubscriptionId, current.ItemId } equals new { last.SubscriptionId, last.ItemId } into outer
from o in outer.DefaultIfEmpty()
select new Resource
{
SubscriptionId = current.SubscriptionId,
ItemId = current.ItemId,
UnitsThisMonth = current.Units,
UnitsLastMonth = o?.Units ?? 0, // Replace NULL with 0
EffectiveRate = current.EffectiveRate,
ResourceName = current.ResourceName,
UnitOfMeasure = current.UnitOfMeasure
};
// Reverse of the first join. Last month LEFT JOIN Current month
var lastMonth =
from last in resourcesLastMonth
join current in resourcesThisMonth on new { last.SubscriptionId, last.ItemId } equals new { current.SubscriptionId, current.ItemId } into outer
from o in outer.DefaultIfEmpty()
select new Resource
{
SubscriptionId = last.SubscriptionId,
ItemId = last.ItemId,
UnitsThisMonth = o?.Units ?? 0, // Replace NULL with 0
UnitsLastMonth = last.Units,
EffectiveRate = o?.EffectiveRate ?? last.EffectiveRate,
ResourceName = o?.ResourceName ?? last.ResourceName,
UnitOfMeasure = o?.UnitOfMeasure ?? last.UnitOfMeasure
};
// Union them together to get a full join
var resources = currentMonth.Union(lastMonth);

Tricky combination of two lists using Linq

I hope somebody will be able to guide me in right direction here...
public class SubmissionLog
{
public int PKId {get;set;}
public int SubmissionId {get;set;}
public DateTime Created {get;set;}
public int StatusId {get;set;}
}
And this is the data:
1, 123, '1/24/2013 01:00:00', 1
2, 456, '1/24/2013 01:30:00', 1
3, 123, '1/25/2013 21:00:00', 2
4, 456, '1/25/2013 21:30:00', 2
5, 123, '2/25/2013 22:00:00', 1
6, 123, '2/26/2013 21:00:00', 2
7, 123, '2/16/2013 21:30:00', 1
What I am trying to is following:
I'd like to know the the average time span from StatusId 1 to StatusId 2 on a given day.
So, let's say date is 2/26/2013, then what I thought would make sense if first get the list like this:
var endlingList = (from sl in db.SubmissionLogs
where (DateTime.Now.AddDays(days).Date == sl.Created.Date) // days = passed number of days to make it 2/26/2013
&& (sl.StatusId == 2)
select sl).ToList();
var endingLookup = endlingList.ToLookup(a => a.SubmissionId, a => a.Created); // thought of using lookup because Dictionary doesn't allow duplicates
After that I thought I'd figure out starting points
var startingList = (from sl in db.SubmissionLogs
where endingList.Select(a => a.SubmissionId).ToArray().Contains(sl.QuoteId)
&& sl.StatusId == 1
select sl).ToList();
And then what I did was following:
var revisedList = endingLookup.Select(a =>
new SubmissionInterval {
SubmissionId = a.Key,
EndDateTime = endingLookup[a.Key].FirstOrDefault(), //This is where the problem is. This will only grab the first occurance.
StartDateTime = startLookup[a.Key].FirstOrDefault() //This is where the problem is. This will only grab the first occurance.
});
And then what I do to get average is following (again, this will only include the initial or first ocurances of status 1 and status 2 of some submission id Submission Log):
return revisedList.Count() > 0 ? revisedList.Select(a=> a.EndDateTime.Subtract(a.StartDateTime).TotalHours).Average() : 0;
So, I hope somebody will understand what my problem here is first of all... To re-cap, I want to get timespan between each status 1 and 2. I pass the date in, and then I have to look up 2's as that ensures me that I will find 1's. If I went the other way around and looked for 1's, then 2's may not exist (don't want that anyway).
At the end I wanna be able to average stuff out...
So let's say if some submission first went from 1 to 2 in a time span of 5h (the code that I left, will get me up to this point), then let's say it got reassigned to 1 and then it went back to 2 in a new time span of 6h, I wanna be able to get both and do the average, so (5+6)/2.
Thanks

I think I understand what you're trying to do. Does thishelp
void Main()
{
var list = new List<SubmissionLog>
{
new SubmissionLog(1, 123, "1/24/2013 01:00:00", 1),
new SubmissionLog(2, 456, "1/24/2013 01:30:00", 1),
new SubmissionLog(3, 123, "1/25/2013 21:00:00", 2),
new SubmissionLog(4, 456, "1/25/2013 21:30:00", 2),
new SubmissionLog(5, 123, "2/25/2013 22:00:00", 1),
new SubmissionLog(6, 123, "2/26/2013 21:00:00", 2),
new SubmissionLog(7, 123, "2/16/2013 21:30:00", 1),
};
// split out status 1 and 2
var s1s = list.Where (l => l.StatusId == 1).OrderBy (l => l.Created);
var s2s = list.Where (l => l.StatusId == 2).OrderBy (l => l.Created);
// use a sub-query to get the first s2 after each s1
var q = s1s.Select (s1 => new
{
s1,
s2 = s2s.FirstOrDefault (s2 =>
s1.SubmissionId == s2.SubmissionId &&
s2.Created >= s1.Created
)
}
).Where (s => s.s1.PKId < s.s2.PKId && s.s2 != null);
// extract the info we need
// note that TotalSecond is ok in Linq to Object but you'll
// probably need to use SqlFunctions or equivalent if this is to
// run against a DB.
var q1 = q.Select (x => new
{
Start=x.s1.Created,
End=x.s2.Created,
SubmissionId=x.s1.SubmissionId,
Seconds=(x.s2.Created - x.s1.Created).TotalSeconds
}
);
// group by submissionId and average the time
var q2 = q1.GroupBy (x => x.SubmissionId).Select (x => new {
x.Key,
Count=x.Count (),
Start=x.Min (y => y.Start),
End=x.Max (y => y.End),
Average=x.Average (y => y.Seconds)});
}
public class SubmissionLog
{
public SubmissionLog(int id, int submissionId, string date, int statusId)
{
PKId = id;
SubmissionId = submissionId;
Created = DateTime.Parse(date, CultureInfo.CreateSpecificCulture("en-US"));
StatusId = statusId;
}
public int PKId {get;set;}
public int SubmissionId {get;set;}
public DateTime Created {get;set;}
public int StatusId {get;set;}
}

How to merge two lists based on a property?

I have two lists, one fake and one real, like:
BEFORE
// fake (list 1)
{ ID = 1, Year = 2011, X = "" }
, { ID = 2, Year = 2012, X = "" }
, { ID = 3, Year = 2013, X = "" }
// real (list 2)
{ ID = 35, Year = 2011, X = "Information" }
, { ID = 77, Year = 2013, X = "Important" }
I want to merge them looking for the Year, the result should be:
AFTER
{ ID = 35, Year = 2011, X = "Information" }
, { ID = 2, Year = 2012, X = "" }
, { ID = 77, Year = 2013, X = "Important" }
It must remove elements with the same year on the first list and add the element with the equivalent Year on the second list to the first list, keeping the order.
How can I do it using Linq?

You should be able to do that using a "left join":
from f in fake
join r in real
on f.Year equals r.Year
into joinResult
from r in joinResult.DefaultIfEmpty()
select new
{
ID = r == null ? f.ID : r.ID,
Year = f.Year,
X = r == null ? f.X : r.X
};

Justin's query is the most efficient way to do it, but if you're concerned with keeping identical objects (and not creating new records from the query) you could do it like this:
var combined = from f in fake
let r = (from r1 in real
where r1.Year == f.Year
select r1).SingleOrDefault()
select r ?? f;

Using IEnumerable.Union and IEqualityComparer.
P.S. This would result in a different result when compared to left join if the real list had more elements (years that are not present in fake list). The left join would not return those results which could be a desired result (not clear from OP).
public class MyClass
{
public int ID {get; set;}
public int Year {get; set;}
public string X {get; set;}
}
public class MyClassEqualityComparer : IEqualityComparer<MyClass>
{
public bool Equals(MyClass x, MyClass y)
{
return x.Year == y.Year;
}
public int GetHashCode(MyClass obj)
{
return obj.ToString().ToLower().GetHashCode();
}
}
void Main()
{
var fake = new List<MyClass> {
new MyClass { ID = 1, Year = 2011, X = "" }
, new MyClass { ID = 2, Year = 2012, X = "" }
, new MyClass { ID = 3, Year = 2013, X = "" }
};
var real = new List<MyClass> {
new MyClass { ID = 35, Year = 2011, X = "Information" }
, new MyClass { ID = 77, Year = 2013, X = "Important" }
};
var merged = real.Union(fake, new MyClassEqualityComparer());
}

Instead of defining the fake list yourself, try having Linq do it for you:
Enumerable.Range(2011,3) //2011, 2012, 2013
//use the overload that provides a 0-based ordinal position of each element
.Select(x,i=> new {ID = i+1, Year = x, X = String.Empty)
//now you have your fake list; join with the "real" list based on Year fields,
//taking the real element wherever it exists and the fake one otherwise
.Join(real, l=>l.Year, r=>r.Year, (l,r) => r == null ? l : r);
This will produce exactly the result set you want. You will likely need to define a named type for the list items, though, as two separately-defined anonymous types cannot be implicitly converted even if they have all the same member types/names.