Develop Reference

EF Core Distinct + OrderBy Throws Translation Error

I have been dealing with a really frustrating EF Core (newest version) error. I'm not sure at this point if I am doing something wrong or if it's a bug. Any help the community can provide would be appreciated.
The error is in regards to Entity Framework Core and translating a LINQ expression to SQL. The below code translates to SQL properly. The query variable below could potentially have a variety of Where expressions and Includes applied to it with no issue.
// This works fine
query.Select(price => new Customer {
Name = price.Payer.Name,
Code = price.Payer.Code,
City = price.Payer.City,
ParentCode = price.Payer.ParentCode,
ParentLevel = CustomerLevel.Corporate,
CustomerLevel = CustomerLevel.Payer
}).Distinct().ToListAsync();
As soon as I add a call to OrderBy, it will not evaluate. If I remove the call to Distinct, it once again works, but I can't have both. I've tried several different ways to build the expression and several workarounds that I've found around the interwebz, and nothing seems to resolve it.
// This throws error
// query is of type IQueryable<Price>
query.Select(price => new Customer {
Name = price.Payer.Name,
Code = price.Payer.Code,
City = price.Payer.City,
ParentCode = price.Payer.ParentCode,
ParentLevel = CustomerLevel.Corporate,
CustomerLevel = CustomerLevel.Payer
}).Distinct().OrderBy(cust => cust.Name).ToListAsync();
Also, placement of the OrderBy does not seem to matter. Based on what I've read, the call to Distinct removes all prior ordering, so this one is not too surprising.
// This also throws error
// query is of type IQueryable<Price>
query
.OrderBy(price => price.payer.Name)
.Select(price => new Customer {
Name = price.Payer.Name,
Code = price.Payer.Code,
City = price.Payer.City,
ParentCode = price.Payer.ParentCode,
ParentLevel = CustomerLevel.Corporate,
CustomerLevel = CustomerLevel.Payer
}).Distinct().ToListAsync();

I actually found the cause of the issue today. The ParentCode property (used at line ParentCode = price.Payer.ParentCode) is actually a property on the base class that is not mapped to the table, so EF apparently did not know what to do with it. The very strange part of this is that it worked without the OrderBy. Changing that line to ParentCode = price.Payer.CorporateCode resolved the issue.
Thanks for anyone that took a look at this.

Generate Dynamic Linq query using outerIt

I am using Microsoft's Dynamic Linq (System.Linq.Dynamic) library to generate some queries at run time. This has worked great for me, but for one specific scenario.
Simplified scenario - I am trying to query all claims which have some specific tags that the user has selected and whose Balance is greater than some number.
static void Main(string[] args)
{
var claims = new List<Claim>();
claims.Add(new Claim { Balance = 100, Tags = new List<string> { "Blah", "Blah Blah" } });
claims.Add(new Claim { Balance = 500, Tags = new List<string> { "Dummy Tag", "Dummy tag 1" } });
// tags to be searched for
var tags = new List<string> { "New", "Blah" };
var parameters = new List<object>();
parameters.Add(tags);
var query = claims.AsQueryable().Where("Tags.Any(#0.Contains(outerIt)) AND Balance > 100", parameters.ToArray());
}
public class Claim
{
public decimal? Balance { get; set; }
public List<string> Tags { get; set; }
}
This query throws an error:
An unhandled exception of type 'System.Linq.Dynamic.ParseException' occurred in System.Linq.Dynamic.dll
Additional information: No property or field 'Balance' exists in type 'String'
Dynamic linq parser seems to try to find the Balance property on the Tag and not on the Claim object.
I have tried to play around with outerIt, innerIt, It keywords in Dynamic Linq but none of it seems to work.
Changing the sequence works, but that's not an option for me, since in the real application the filters, operators and patterns will be dynamic (configured by end user).
Boxing the conditions in brackets (), also doesn't help.
Workaround - create a simple contains condition for every Tag selected e.g. Tags.Contains("New") OR Tags.Contains("Blah") etc.. But in the real application it results in a really complex / bad query for each condition and kills the performance.
I might be missing something or this could be a bug in the library.
I would really appreciate if someone could help me with this.

Found a/the bug in ParseAggregate... The pushing of it→outerIt and back doesn't work if there are multiple levels. The code supposes that the it and outerIt won't be changed by a third party before being reset (technically the code isn't reentrant). You can try with other variants of System.Linq.Dynamic (there are like two or three variants out of there). Probably some variants have already fixed it.
Or you can take the code from the linked site and recompile it inside your code (in the end the "original" System.Linq.Dynamic is a single cs file) and you can patch it like this:
Expression ParseAggregate(Expression instance, Type elementType, string methodName, int errorPos)
{
// Change starts here
var originalIt = it;
var originalOuterIt = outerIt;
// Change ends here
outerIt = it;
ParameterExpression innerIt = Expression.Parameter(elementType, elementType.Name);
it = innerIt;
Expression[] args = ParseArgumentList();
// Change starts here
it = originalIt;
outerIt = originalOuterIt;
// Change ends here
MethodBase signature;
if (FindMethod(typeof(IEnumerableSignatures), methodName, false, args, out signature) != 1)
I've already opened an Issue with the suggested bug fix in the github of the project.

This seems to be working correctly in my version: System.Linq.Dynamic.Core
See the test here:
https://github.com/StefH/System.Linq.Dynamic.Core/blob/master/test/System.Linq.Dynamic.Core.Tests/ComplexTests.cs#L19

Libgit2Sharp: get files in all commits between two tags

I can do this in GitBash:
$ git diff --name-only v01...HEAD -- *.sql
which gives:
Components/1/Database/Stored Procedures/spDC1.sql
Components/1/Database/Stored Procedures/spDC2.sql
I can't see how I would do this in LibGit2Sharp.
Any ideas?
Thanks

Here is an example from one of my projects that get a ICommitLog collection between two commits (current HEAD vs. the master branch):
// git log HEAD..master --reverse
public ICommitLog StalkerList {
get {
var filter = new CommitFilter {
SortBy = CommitSortStrategies.Reverse | CommitSortStrategies.Time,
Since = master,
Until = head.Tip,
};
return repo.Commits.QueryBy (filter);
}
}
Once you have your ICommitLog collection of all the commits within the range that you need, you can cycle through each commit to get a list of the files that were effected within that commit (of course you would need to add filtering of the filename via your "*.sql" requirements):
public String[] FilesToMerge (Commit commit)
{
var fileList = new List<String> ();
foreach (var parent in commit.Parents) {
foreach (TreeEntryChanges change in repo.Diff.Compare<TreeChanges>(parent.Tree, commit.Tree)) {
fileList.Add (change.Path);
}
}
return fileList.ToArray ();
}

I think SushiHangover's answer to this is pretty correct. Just a couple of amendments / updates. (P.S. Yeah I know this question's relatively old, but a complete answer would have helped me if I'd found it today, so putting one here for future peeps).
This bit should be an amendment comment, but I can't comment yet (low rep):
First, I think that in Sushi's example, master and head.Tip are the wrong way around. Until excludes a commit from the results and excludes its ancestors. So if you put head.Tip in there it'll exclude basically the entire history tree.
So AFAIK it should read more like this:
// git log HEAD..master --reverse
public ICommitLog StalkerList {
get {
var filter = new CommitFilter {
SortBy = CommitSortStrategies.Reverse | CommitSortStrategies.Time,
Since = head.Tip,
Until = master
};
return repo.Commits.QueryBy (filter);
}
}
It's also SUPER important to realise that the order you give them in matters. If you just switch them around you'll get nothing back.
(Also note that Sushi's original example had a bad ',' after "head.Tip".
This bit's the update:
Also worth noting that the libgit2sharp library has been updated recently. Replacing "Since" and "Until" with "IncludeReachableFrom" and "ExcludeReachableFrom" respectively.
The names aren't particularly helpful until you realise that they're just way more verbose about what they're doing.
The comment for Exclude, for example, reads:
/// A pointer to a commit object or a list of pointers which will be excluded (along with ancestors) from the enumeration.
So the latest implementation would look more like:
using (Repository r = new Repository(#"[repo_location]"))
{
CommitFilter cf = new CommitFilter
{
SortBy = CommitSortStrategies.Reverse | CommitSortStrategies.Time,
ExcludeReachableFrom = r.Branches["master"].Tip,
IncludeReachableFrom = r.Head.Tip
};
var results = r.Commits.QueryBy(cf);
foreach (var result in results)
{
//Process commits here.
}
}
Tested this out in LINQPad and it seems to work. Might have missed something though as it was a quick draft. If so let me know.
Things to note: master and Head are actually properties of a Repository, I couldn't see that coming from anywhere in the old example but it may just be a version difference with the old version.

How to select nested attribute in scan in DynamoDB?

I realize this is answered in the documentation (basically, "use the dot syntax"), but I'm still missing something. I'm using the .NET SDK and need to be able to select just a few attributes from a scan, one of which is a boolean inside a Map attribute. Note that I'm not trying to filter the results. I want all of the items, but I only want some of the attributes returned to me.
var config = new ScanOperationConfig
{
AttributesToGet = new List<string> { "foo.bar" },
Select = SelectValues.SpecificAttributes
};
var search = Table.Scan(config);
var documents = await search.GetRemainingAsync();
This code gets me the items I expect, but it's missing the "foo.bar" attribute. I know I can select the entire foo object, but I'm trying to minimize the amount of data handed back. I don't want the other attributes inside the foo object.
The relevant attribute of the item has the following JSON format:
{
"foo": {
"bar": true
}
}
I checked spelling, case sensitivity, etc. to no avail. Any idea what's wrong?

Instead of using Table.Scan, use the AmazonDynamoDBClient and you get more options.
The Client's ScanAsync method takes a ScanRequest which has a ProjectionExpression string. This is not present on the ScanOperationConfig class and that was the source of confusion.
Use the ProjectionExpression like so:
var scanRequest = new ScanRequest(Table.TableName)
{
ProjectionExpression = "foo.bar"
};
According to the documentation on ProjectionExpression:
ProjectionExpression replaces the legacy AttributesToGet parameter.
I didn't realize AttributesToGet was legacy until finally looking at the client for a totally unrelated problem and happened to find my answer to this problem.

Running identical code on 2 objects

I have a test script that does something to one object then the same thing to a second object. This continues for quite a while. With so much predictable repetition that it seems ripe for automation but I can't figure out how. I wouldn't care so much except with so much repetition, it's easy to overlook using the wrong variable (ie: stagingXyz when prodXyz was intended.)
The details below are irrelevant. What's important is the pattern.
var stagingDbs = cleanupDbs(stagingServer.Databases);
var prodDbs = cleanupDbs(prodServer.Databases);
printDiff(stagingDbs, prodDbs, "Databases mis-matched");
foreach (var db in stagingDbs.Intersect(prodDbs)) {
var stagingDb = stagingServer.Databases[db];
var prodDb = prodServer.Databases[db];
var stagingTables = cleanupTables(stagingDb.Tables);
var prodTables = cleanupTables(prodDb.Tables);
printDiff(stagingTables, prodTables, "Tables mis-matched on " + db);
foreach (var table in stagingTables.Intersect(prodTables)) {
var stagingTable = stagingDb.Tables[table];
var prodTable = prodDb.Tables[table];
var matchedColumns = stagingColumns.Intersect(prodColumns);
var stagingTableColumns = stagingTable.Columns
.Cast<Column>()
.Where(c => matchedColumns.Contains(c.Name))
.Select(c => formatColumn(c));
var prodTableColumns = prodTable.Columns
.Cast<Column>()
.Where(c => matchedColumns.Contains(c.Name))
.Select(c => formatColumn(c));
printDiff(stagingTableColumns, prodTableColumns,
"Columns mis-matched");
}
}
I don't want to go through, for instance, replacing this
var stagingTableColumns = stagingTable.Columns
.Cast<Column>()
.Where(c => matchedColumns.Contains(c.Name))
.Select(c => formatColumn(c));
var prodTableColumns = prodTable.Columns
.Cast<Column>()
.Where(c => matchedColumns.Contains(c.Name))
.Select(c => formatColumn(c));
with this
var stagingTableColumns = doStuff(stagingTable, matchedColumns);
var prodTableColumns = doStuff(prodTable, matchedColumns);
because I have to make sure everything in the 1st line is stagingXyz and the 2nd line is prodXyz. Not so bad for 1 line but the test script is huge and only ever does one of these 2 things:
foo(stagingXyz); foo(prodXyz);
bar(stagingXyz, prodXyz);
Similarly, wrapping with these items in an array and having doStuff[0]; doStuff[1]; is subject to the same easy typo error only a typo with 0 vs. 1 will be even harder to spot at a glance.
I thought about making 2 container objects (one for staging, one for prod) and putting these 2 objects in a collection but I fear this will lead to a bazillion tiny loops that will be very hard to maintain.
Is there anyway to simplify this and still have it be readable and maintainable?

Could you generate your test script? The input might read something like
var %%AB%%Dbs = cleanupDbs(%%AB%%Server.Databases);
printDiff(%%A%%Dbs, %%B%%Dbs, "Databases mis-matched");
foreach (var db in %%A%%Dbs.Intersect(%%B%%Dbs)) {
var %%AB%%Db = %%AB%%Server.Databases[db];
var %%AB%%Tables = cleanupTables(%%AB%%Db.Tables);
printDiff(%%A%%Tables, %%B%%Tables, "Tables mis-matched on " + db);
...
}
A line containing %%AB%% might expand to two copies of the same line, one with the "A" replacement and one with the "B" replacement, where %%A%% or %%B%% by itself might just get replaced.

Edit - After reading your comments I see the problem a bit clearer now. I think the problem is more the clarity of the one big function vs. coming up with a funky way to solve the readability problem. I think the more you broke it up into smaller functions, the clearer it would get.
If the main function was broken up into something like this:
public void mainMethod(DB prodDB, DB stagingDB)
{
doPart1(prodDB, stagingDB);
doPart2(prodDB, stagingDB);
}
...and each part had well named inputs like so:
public void doPart1(DB prodDB, DB stagingDB)
{
// Code...
}
Things would clear themselves up as you made things work at a more and more granular level. Anyone working in the doPart1 method only has to be concerned with it's small amount of code, and anyone working in the main section shouldn't have a million things to look over. I understand if this may sound like an oversimplified response, but it sounds like you're trying to solve a problem that shouldn't exist if the code is properly broken up.
If there is a method that is so huge and unreadable that another developer couldn't figure out what's going on with only TWO variables, then there is a different problem.