I have a table which contains columns among others like Id as Primary Key and LastMeterReadingId (Foreign Key which references to same table) - something like Parent Meter Reading.
I would like to get all rows which are not already used like Parent. I would like to avoid situation when meter reading is parent for more than one meter reading.
I know how to join to same table, but I have no idea how to choose only those records which aren't parent already. That's how query looks like without condition statement.
return (from m in uow.MeterReadingReadWriteRepository.Query()
join parent in uow.MeterReadingReadWriteRepository.Query() on m.Id equals parent.LastMeterReadingId
select new MeterReadingDto()
{
(...)
}).ToList();
Do you have any idea how to achieve it in efficient way?
Regards.
I would like to get all rows which are not already used like Parent
In other words, you want all rows that have no children. Note that the variable name parent in your query is misleading - when you do a join b on a.Id equals b.ParentId, a is the parent and b is the child.
Anyway, there are at least 3 ways to achieve your goal, IMO being equivalent from nowadays database query optimizers point of view (i.e. should be equally efficient):
(1) Using !Any(...) which is equivalent to SQL NOT EXISTS(...):
from m in uow.MeterReadingReadWriteRepository.Query()
where !uow.MeterReadingReadWriteRepository.Query().Any(child => m.Id == child.LastMeterReadingId)
select ...
(2) Using group join:
from m in uow.MeterReadingReadWriteRepository.Query()
join child in uow.MeterReadingReadWriteRepository.Query()
on m.Id equals child.LastMeterReadingId into children
where !children.Any()
select ...
(3) Using left outer antijoin:
from m in uow.MeterReadingReadWriteRepository.Query()
join child in uow.MeterReadingReadWriteRepository.Query()
on m.Id equals child.LastMeterReadingId into children
from child in children.DefaultIfEmpty()
where child == null
select ...
If this is EF (LINQ to Entities), the first two are translated to one and the same SQL NOT EXISTS based query. While the last is translated to the "traditional" SQL LEFT JOIN ... WHERE right.PK IS NULL based query.
You could just add
where !(from child in uow.MeterReadingReadWriteRepository.Query() where child.Id == m.LastMeterReadingId select child).Any()
Not sure how intelligently this would be optimised though. It would also be better to factor out uow.MeterReadingReadWriteRepository.Query().
Do you not have a Child relationship/collection in your Meter Reading entity from the foreign key constraint? - this would make the query much more straightforward.
var readings = uow.MeterReadingReadWriteRepository.Query();
var parents = readings
.Join(readings, child => child.Id, parent => parent.LastMeterReadingId,
(child, parent) => new {parent.Id})
.Distinct()
.ToDictionary(a => a.Id);
var result = (from m in readings
where !parents.Contains(m.Id)
select new
{
Id = m.Id
}).ToList();
Thanks #Ben Jackson
public class MeterReading : EntityBase
{
public long PropertyId { get; set; }
public long? LastMeterReadingId { get; set; }
public long? PaymentId { get; set; }
public Property Property { get; set; }
public MeterReading LastReading { get; set; }
public Payment Payment { get; set; }
}
That's how most value properties looks like. Maybe should I use T-SQL query with JOIN to CTE which mentioned before condition statement? I'll try your solution ASAP.
I have a database that is mapped using the Entity Framework. Entity Framework is generating the C# code of the database objects in the similar manner as shown below. For simplicity I have created Parent, Child, GrandChild hierarchy but the actual db contains much longer hierarchies and many other fields.
class Parent
{
string name;
int id;
datetime DateOfBirth;
}
class Child
{
string name;
int id;
int ParentId; ( FK reference to Parent Child )
}
class GrandChild
{
string name;
int id;
int ChildId; (FK reference to Child )
}
Now, I am building an api where the filters will be provided at runtime. I mean, some of the queries could be
Give all GrandChild rows for ParentId =1
Give All Grandchild rows for ChildName = "x"
Give all GrandChild rows for Parent with DateOfbirth = "x/y/z"
So, how can I build in C# code, using LINQ or Expression Trees to create predicates and join filters dynamically/runtime.
Following URL:
https://msdn.microsoft.com/library/bb882637.aspx
shows how to create dynamic query, but not how to INNER JOIN multiple such queries. Does anyone know how to do that?
This Stackoverflow answer:
The parameter '***' was not bound in the specified LINQ to Entities query expression
also highlights how to create filters dynamically but not how to join them.
Does anyone know how to create dynamic queries to filter rows and join the queries? Let me know if you need more information. Thanks
Well, I'm going to suppose that is not your real model because you should have properties instead fields and your entities must be public.
If you have represented your db relationships using navigation properties, you could create a extension method like this:
static IQueryable<TEntity> Select<TEntity>(this IQueryable<TEntity> query, List<Expression<Func<TEntity, bool>>> filters = null,
List<Expression<Func<TEntity, object>>> includes = null)
{
if (includes != null)
{
query = includes.Aggregate(query, (current, include) => current.Include(include));
}
if (filters != null)
{
query = filters.Aggregate(query, (current, filter) => current.Where(filter)); //at the end this is going to be translated to condition1 && condition2 ...
}
return query;
}
In the first list you pass all the conditions you want to apply to your query and the second helps you to load the related entities that you need in your query:
var conditions = new List<Expression<Func<GrandChild, bool>>>() { (t) => t.Child.Parent.ParentId==1 };
var includes = new List<Expression<Func<GrandChild, object>>>() { (t) => t.Child.Parent };
var query= yourContext.GrandChilds.Select(filters,includes);
1 . Give all GrandChild rows for ParentId =1
var grandChildData=from grand in GrandChild
join ch in Child on grand.ChildId equals ch.Id
where ch.ParentId==1
select grand;
Give All Grandchild rows for ChildName = "x"
var grandChildData=from grand in GrandChild
join ch in Child on grand.ChildId equals ch.Id
where ch.name = "x"
select grand;
Give all GrandChild rows for Parent with DateOfbirth = "x/y/z"
var grandChildData=from grand in GrandChild
join ch in Child on grand.ChildId equals ch.Id
join p in Parent on ch.ParentId equals p.id
where p.DateOfbirth==Convert.ToDateTime("2016/01/01")
select grand;
I have a table of elements that can be associated to each other (within the same table).
So for example a structure like:
[Id] Int
[ParentWithinThisTableId] Int
[AFieldToQueryAgainst] NVarCharMax
At the beginning of the search, I will be given a string query and a single Id (call it StartId that points to an element in this table. What I want to do is compare the string query to the [AFieldToQueryAgainst] on the element at the Id given, but also query against the same column for all rows that have [ParentWithinThisTableId] == StartId and then all of the rows that have these ids as [ParentWithinThisTableId].
The only way I can think to do it is recursively like:
var forms = db.Forms.Where(m => m.ParentWithinThisTableId == this.Id);
var searchMatches = new List<Form>();
foreach (var form in forms)
{
forms = forms.Concat(AddSearches(query, form.Id));
}
foreach (var form in forms)
{
if (form.AFieldToQueryAgainst.Contains(query))
{
searchMatches.Add(form);
}
}
With AddSearches being recursive like:
private IQueryable<Form> AddSearches(string query, int startId)
{
var items = RepositoryProxy.Context.Forms.Where(m => m.ParentWithinThisTableId == startId);
foreach (var item in items)
{
items = items.Concat(AddSearches(query, item.Id));
}
return items;
}
But this takes about 1.8 seconds run time in my testing on a relatively shallow tree. Is there a more efficient way to do this? Possibly avoid iterating each of the IQueryables until they're all compiled somehow?
why can't you just use a self join ?
var these = (from p in items
join x in items on p.Id equals x.ParentId
where x != null
select x).ToList();
In oracle I can do the following query:
SELECT *
FROM Tabl Tabb
WHERE (tabb.Col1, tabb.Col2) IN ( (1,2), (3,4))
Consider I 've following entity:
public class Tabb
{
public int Col1 {get; set; }
public int Col2 {get; set; }
// other props
}
and criteria class
public class Search
{
public int Col1 {get; set; }
public int Col2 {get; set; }
}
I need to write:
public IEnumerable<Tabb> Select(IEnumerable<Search> s)
{
var queryable = this.context.Tabbs;
return queryable.Where(\* some *\).ToList();
}
How can I select entities, that search collection contain instance of search that has the same value of Col1 and Col2?
EDIT:
var result = from x in entity
join y in entity2
on new { x.field1, x.field2 } equals new { y.field1, y.field2 }
It doesn't work (As I expected) - in may case entity2 is not a entity table, it is static collection, so EF throws exception (sth like: cannot find mapping layer to type Search[]);
There's a few ways, which all have pros and cons, and are sometimes a little bit tricky...
Solution 1
You enumerate the ef part first (of course, depending on the size of your data, this might be a very bad idea)
Solution 2
You concatenate your fields with an element you're sure (hum) you won't find in your fields, and use a Contains on concatenated EF data.
var joinedCollection =entity2.Select(m => m.field1 + "~" + m.field2);
var result = entity.Where(m => joinedCollection.Contains(m.field1 + "~" + m.field2));
of course, this would be a little bit more complicated if field1 and field2 are not string, you'll have to use something like that
SqlFunctions.StringConvert((double)m.field1) + "~" + //etc.
Solution 3
you do this in two step, assuming you will have "not too much result" with a partial match (on only one field)
var field1Collection = joinedCollection.Select(m => m.field1);
var result = entity.Where(m => joinedCollection.Contains(m.field1)).ToList();
then you make the "complete join" on the two enumerated lists...
Solution 4
use a stored procedure / generated raw sql...
Just understood the problem better. You want all rows where the columns match, may be this will help:
myDBTable.Where(x =>
myStaticCollection.Any(y => y.Col2 == x.Col2) &&
myStaticCollection.Any(y => y.Col1 == x.Col1))
.ToList()
.Select(x => new Search { Col1 = x.Col1, Col2 = x.Col2 });
This is saying, I want each row where any Col2 in my static collection matches this database Col2 AND where any Col1 matches this database Col1
this.context.Searches.Join(
this.context.Tabbs,
s => s.Col2,
t => t.Col2,
(search, tab) => new {
search,
tab
});
This will bring back IEnumerable<'a> containing a search and a tab
This guy is doing something similar LINK
var result = from x in entity
join y in entity2
on new { x.field1, x.field2 } equals new { y.field1, y.field2 }
Once you have your result then you want to enumerate that to make sure you're hitting the database and getting all your values back. Once they're in memory, then you can project them into objects.
result.ToList().Select(a => new MyEntity { MyProperty = a.Property });
I have an Item. Item has a Category.
Category has ID, Name, Parent and Children. Parent and Children are of Category too.
When I do a LINQ to Entities query for a specific Item, it doesn't return the related Category, unless I use the Include("Category") method. But it doesn't bring the full category, with its parent and children. I could do Include("Category.Parent"), but this object is something like a tree, I have a recursive hierarchy and I don't know where it ends.
How can I make EF fully load the Category, with parent and children, and the parent with their parent and children, and so on?
This is not something for the whole application, for performance considerations it would be needed only for this specific entity, the Category.
Instead of using the Include method you could use Load.
You could then do a for each and loop through all the children, loading their children. Then do a for each through their children, and so on.
The number of levels down you go will be hard coded in the number of for each loops you have.
Here is an example of using Load: http://msdn.microsoft.com/en-us/library/bb896249.aspx
If you definitely want the whole hierarchy loaded, then if it was me I'd try writing a stored procedure who's job it is to return all the items in a hierarchy, returning the one you ask for first (and its children subsequently).
And then let the EF's relationship fixup ensure that they are all hooked up.
i.e. something like:
// the GetCategoryAndHierarchyById method is an enum
Category c = ctx.GetCategoryAndHierarchyById(1).ToList().First();
If you've written your stored procedure correctly, materializing all the items in the hierarchy (i.e. ToList()) should make EF relationship fixup kicks in.
And then the item you want (First()) should have all its children loaded and they should have their children loaded etc. All be populated from that one stored procedure call, so no MARS problems either.
Hope this helps
Alex
Use this extension method which calls the hard-coded version of Include, to achieve a dynamic depth level of inclusion, it works great.
namespace System.Data.Entity
{
using Linq;
using Linq.Expressions;
using Text;
public static class QueryableExtensions
{
public static IQueryable<TEntity> Include<TEntity>(this IQueryable<TEntity> source,
int levelIndex, Expression<Func<TEntity, TEntity>> expression)
{
if (levelIndex < 0)
throw new ArgumentOutOfRangeException(nameof(levelIndex));
var member = (MemberExpression)expression.Body;
var property = member.Member.Name;
var sb = new StringBuilder();
for (int i = 0; i < levelIndex; i++)
{
if (i > 0)
sb.Append(Type.Delimiter);
sb.Append(property);
}
return source.Include(sb.ToString());
}
}
}
Usage:
var affiliate = await DbContext.Affiliates
.Include(3, a => a.Referrer)
.SingleOrDefaultAsync(a => a.Id == affiliateId);
Anyway, meanwhile, join the discussion about it on the EF repo.
It could be dangerous if you did happen to load all recursive entities, especially on category, you could end up with WAY more than you bargained for:
Category > Item > OrderLine > Item
OrderHeader > OrderLine > Item
> Item > ...
All of a sudden you've loaded most of your database, you could have also loaded invoices lines, then customers, then all their other invoices.
What you should do is something like the following:
var qryCategories = from q in ctx.Categories
where q.Status == "Open"
select q;
foreach (Category cat in qryCategories) {
if (!cat.Items.IsLoaded)
cat.Items.Load();
// This will only load product groups "once" if need be.
if (!cat.ProductGroupReference.IsLoaded)
cat.ProductGroupReference.Load();
foreach (Item item in cat.Items) {
// product group and items are guaranteed
// to be loaded if you use them here.
}
}
A better solution however is to construct your query to build an anonymous class with the results so you only need to hit your datastore once.
var qryCategories = from q in ctx.Categories
where q.Status == "Open"
select new {
Category = q,
ProductGroup = q.ProductGroup,
Items = q.Items
};
This way you could return a dictionary result if required.
Remember, your contexts should be as short lived as possible.
You don't want to do recursive loading of the hierarchy, unless you are allowing a user to iteratively drill down/up the tree: Every level of recursion is another trip to the database. Similarly, you'll want lazy loading off to prevent further DB trips as you're traversing the hierarchy when rendering to a page or sending over a webservice.
Instead, flip your query: Get Catalog, and Include the items in it. This will get you all items both hierarchically (navigation properties) and flattened, so now you just need to exclude the non-root elements present at the root, which should be pretty trivial.
I had this problem and provided a detailed example of this solution to another, here
You chould rather introduce a mapping table that maps each Category a parent and a child, instead of adding the parent and child property to the cargo itself.
Depending on how often you need that information it can be queried on demand. Via unique constraints in the db you can avoid an infinite amount of relationships beeing possible.
I found out that if you include "two parent levels", you will get the whole parent hierarchy, like that:
var query = Context.Items
.Include(i => i.Category)
.Include(i => i.Category.Parent.Parent)
And now for a completely different approach to hierarchical data, for example populating a treeview.
First, do a flat query for all data, and then build the object graph in memory:
var items = this.DbContext.Items.Where(i=> i.EntityStatusId == entityStatusId).Select(a=> new ItemInfo() {
Id = a.Id,
ParentId = a.ParentId,
Name = a.Name,
ItemTypeId = a.ItemTypeId
}).ToList();
Get the root item:
parent = items.FirstOrDefault(a => a.ItemTypeId == (int)Enums.ItemTypes.Root);
Now build your graph:
this.GetDecendantsFromList(parent, items);
private void GetDecendantsFromList(ItemInfo parent, List<ItemInfo> items)
{
parent.Children = items.Where(a => a.ParentId == parent.Id).ToList();
foreach (var child in parent.Children)
{
this.GetDecendantsFromList(child,items);
}
}
Here is a clever recursive function I found here that would work for this:
public partial class Category
{
public IEnumerable<Category> AllSubcategories()
{
yield return this;
foreach (var directSubcategory in Subcategories)
foreach (var subcategory in directSubcategory.AllSubcategories())
{
yield return subcategory;
}
}
}
You could also create a tablevalued function in the database and add that to your DBContext. Then you can call that from your code.
This example requires that you import EntityFramework.Functions from
nuget.
public class FunctionReturnType
{
public Guid Id { get; set; }
public Guid AnchorId { get; set; } //the zeroPoint for the recursion
// Add other fields as you want (add them to your tablevalued function also).
// I noticed that nextParentId and depth are useful
}
public class _YourDatabaseContextName_ : DbContext
{
[TableValuedFunction("RecursiveQueryFunction", "_YourDatabaseContextName_")]
public IQueryable<FunctionReturnType> RecursiveQueryFunction(
[Parameter(DbType = "boolean")] bool param1 = true
)
{
//Example how to add parameters to your function
//TODO: Ask how to make recursive queries with SQL
var param1 = new ObjectParameter("param1", param1);
return this.ObjectContext().CreateQuery<FunctionReturnType>(
$"RecursiveQueryFunction(#{nameof(param1)})", param1);
}
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
//add both (Function returntype and the actual function) to your modelbuilder.
modelBuilder.ComplexType<FunctionReturnType>();
modelBuilder.AddFunctions(typeof(_YourDatabaseContextName_), false);
base.OnModelCreating(modelBuilder);
}
public IEnumerable<Category> GetParents(Guid id)
{
//this = dbContext
return from hierarchyRow in this.RecursiveQueryFunction(true)
join yourClass from this.Set<YourClassThatHasHierarchy>()
on hierarchyRow.Id equals yourClass.Id
where hierarchyRow.AnchorId == id
select yourClass;
}
}
try this
List<SiteActionMap> list = this.GetQuery<SiteActionMap>()
.Where(m => m.Parent == null && m.Active == true)
.Include(m => m.Action)
.Include(m => m.Parent).ToList();
if (list == null)
return null;
this.GetQuery<SiteActionMap>()
.OrderBy(m => m.SortOrder)
.Where(m => m.Active == true)
.Include(m => m.Action)
.Include(m => m.Parent)
.ToList();
return list;
#parliament gave me an idea for EF6. Example for Category with Methods to load all parents up to root node and all children.
NOTE: Use this only for non performance critical operation. Example with 1000 nodes performance from http://nosalan.blogspot.se/2012/09/hierarchical-data-and-entity-framework-4.html.
Loading 1000 cat. with navigation properties took 15259 ms
Loading 1000 cat. with stored procedure took 169 ms
Code:
public class Category
{
[Key, DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public int Id { get; set; }
public string Name { get; set; }
public int? ParentId { get; set; }
public virtual Category Parent { get; set; }
public virtual ICollection<Category> Children { get; set; }
private IList<Category> allParentsList = new List<Category>();
public IEnumerable<Category> AllParents()
{
var parent = Parent;
while (!(parent is null))
{
allParentsList.Add(parent);
parent = parent.Parent;
}
return allParentsList;
}
public IEnumerable<Category> AllChildren()
{
yield return this;
foreach (var child in Children)
foreach (var granChild in child.AllChildren())
{
yield return granChild;
}
}
}
My suggestion would be
var query = CreateQuery()
.Where(entity => entity.Id == Id)
.Include(entity => entity.Parent);
var result = await FindAsync(query);
return result.FirstOrDefault();
and it means it will load single entity and all this entity.Parent entities recursive.
entity is same as entity.Parent
public static class EntityFrameworkExtensions
{
public static ObjectContext GetObjectContext(this DbContext context)
{
ObjectContext objectContext = ((IObjectContextAdapter)context).ObjectContext;
return objectContext;
}
public static string GetTableName<T>(this ObjectSet<T> objectSet)
where T : class
{
string sql = objectSet.ToTraceString();
Regex regex = new Regex("FROM (?<table>.*) AS");
Match match = regex.Match(sql);
string table = match.Groups["table"].Value;
return table;
}
public static IQueryable<T> RecursiveInclude<T>(this IQueryable<T> query, Expression<Func<T, T>> navigationPropertyExpression, DbContext context)
where T : class
{
var objectContext = context.GetObjectContext();
var entityObjectSet = objectContext.CreateObjectSet<T>();
var entityTableName = entityObjectSet.GetTableName();
var navigationPropertyName = ((MemberExpression)navigationPropertyExpression.Body).Member.Name;
var navigationProperty = entityObjectSet
.EntitySet
.ElementType
.DeclaredNavigationProperties
.Where(w => w.Name.Equals(navigationPropertyName))
.FirstOrDefault();
var association = objectContext.MetadataWorkspace
.GetItems<AssociationType>(DataSpace.SSpace)
.Single(a => a.Name == navigationProperty.RelationshipType.Name);
var pkName = association.ReferentialConstraints[0].FromProperties[0].Name;
var fkName = association.ReferentialConstraints[0].ToProperties[0].Name;
var sqlQuery = #"
EXEC ('
;WITH CTE AS
(
SELECT
[cte1].' + #TABLE_PK + '
, Level = 1
FROM ' + #TABLE_NAME + ' [cte1]
WHERE [cte1].' + #TABLE_FK + ' IS NULL
UNION ALL
SELECT
[cte2].' + #TABLE_PK + '
, Level = CTE.Level + 1
FROM ' + #TABLE_NAME + ' [cte2]
INNER JOIN CTE ON CTE.' + #TABLE_PK + ' = [cte2].' + #TABLE_FK + '
)
SELECT
MAX(CTE.Level)
FROM CTE
')
";
var rawSqlQuery = context.Database.SqlQuery<int>(sqlQuery, new SqlParameter[]
{
new SqlParameter("TABLE_NAME", entityTableName),
new SqlParameter("TABLE_PK", pkName),
new SqlParameter("TABLE_FK", fkName)
});
var includeCount = rawSqlQuery.FirstOrDefault();
var include = string.Empty;
for (var i = 0; i < (includeCount - 1); i++)
{
if (i > 0)
include += ".";
include += navigationPropertyName;
}
return query.Include(include);
}
}
Let me offer my simple solution that fits needs to enable/disable the branch of hierarchical data of the selected department's structure of an organization.
The table Departments looks according this SQL
CREATE TABLE [dbo].[Departments](
[ID] [int] IDENTITY(1,1) NOT NULL,
[Name] [nvarchar](1000) NOT NULL,
[OrganizationID] [int] NOT NULL,
[ParentID] [int] NULL,
[IsEnabled] [bit] NOT NULL,
CONSTRAINT [PK_Departments] PRIMARY KEY CLUSTERED
(
[ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
C# code provides a very simple approach that work fine for me.
1. It returns the complete table asynchronously.
2. It changes property for the linked rows.
public async Task<bool> RemoveDepartmentAsync(int orgID, int depID)
{
try
{
using (var db = new GJobEntities())
{
var org = await db.Organizations.FirstOrDefaultAsync(x => x.ID == orgID); // Check if the organization exists
if (org != null)
{
var allDepartments = await db.Departments.ToListAsync(); // get all table items
var isExisting = allDepartments.FirstOrDefault(x => x.OrganizationID == orgID && x.ID == depID);
if (isExisting != null) // Check if the department exists
{
isExisting.IsEnabled = false; // Change the property of visibility of the department
var all = allDepartments.Where(x => x.OrganizationID == orgID && x.ID == isExisting.ID).ToList();
foreach (var item in all)
{
item.IsEnabled = false;
RecursiveRemoveDepartment(orgID, item.ID, ref allDepartments); // Loop over table data set to change property of the linked items
}
await db.SaveChangesAsync();
}
return true;
}
}
}
catch (Exception ex)
{
logger.Error(ex);
}
return false;
}
private void RecursiveRemoveDepartment(int orgID, int? parentID, ref List<Department> items)
{
var all = items.Where(x => x.OrganizationID == orgID && x.ParentID == parentID);
foreach (var item in all)
{
item.IsEnabled = false;
RecursiveRemoveDepartment(orgID, item.ID, ref items);
}
}
This approach works very fast for relative small amount of records I guess less 100000.
Probably for big set of data you have to implement server side stored function.
Enjoy!