I have an Entity called Task. A task can have child tasks. In my dal I am trying to automatically evaluate the child count every time I fetch a task from my entity model. The child count is not stored in the database.
For example, let's look at my Fetch method:
public TaskDto Fetch(Guid id)
{
using (var ctx = ObjectContextManager<MyDataContext>.GetManager("MyDataContext"))
{
var data = (from t in ctx.ObjectContext.Tasks
join tty in ctx.ObjectContext.TaskTypes on t.Id equals tty.Id
where t.EntityGuid == id
select new
{
Task = t,
ChildCount = TaskChildCount(t.EntityGuid)
}).FirstOrDefault();
if (data == null)
{
throw new RecordNotFoundException("Task");
}
return ReadData(data.Task, data.ChildCount);
}
}
But instead of calling ReadData with the 2 params, I just want to call it with a Task param: ReadData(data.Task) and the ChildCount to automatically be there. Is there some way I can bake this into my Task entity? It is a very simple function:
public int TaskChildCount(Guid currentTaskId)
{
var ret = 0;
using (var ctx = ObjectContextManager<MyDataContext>.GetManager("MyDataContext"))
{
ret = ctx.ObjectContext.Tasks.Count(x => x.ParentId != currentTaskId);
}
return ret;
}
There is a way to do this, but Task will always need some help. You could Include the child records of Task:
from t in ctx.ObjectContext.Tasks.Include(t => t.Tasks)
Now Task can have a property TaskCount that returns this.Tasks.Count.
You'd have to access this property outside the scope of a LINQ query, otherwise EF will try to translate it into SQL and fail.
The downside is that it populates all Tasks contained by a parent Task, which is a lot heavier than just getting a count from the database. Therefore a better way could be to use AutoMappers capability to project nested properties by name convention. Suppose your TaskDto would have a property TasksCount, then you could do:
Mapper.CreateMap<Task, TaskDto>();
var data = ctx.ObjectContext.Tasks.Project().To<TaskDto>();
(Project is an extension method in AutoMapper.QueryableExtensions, AutoMapper can be obtained through Nuget).
I leave out the join and where for simplicity, but these would not change the concept. AutoMapper does not find TasksCount in Task itself, so by convention it tries to find a property Tasks to which it can apply Count(). It finds the child collection Tasks (well, maybe you named it differently) and there you go. You will see that the count is neatly integrated in the SQL query that is generated.
The difference is that the dto is created by the Fetch method itself, not by ReadData. I can't judge whether this causes any problems. Maybe you have to shuffle some code in order to use this AutoMapper feature.
Related
I only need it to work for SQL Server. This is an example. The question is about a general approach.
There is a nice extension method from https://entityframework-extensions.net called WhereBulkContains. It is, sort of, great, except that the code of the methods in this library is obfuscated and they do not produce valid SQL when .ToQueryString() is called on IQueryable<T> with these extension methods applied.
Subsequently, I can't use such methods in production code as I am not "allowed" to trust such code due to business reasons. Sure, I can write tons of tests to ensure that WhereBulkContains works as expected, except that there are some complicated cases where the performance of WhereBulkContains is well below stellar, whereas properly written SQL works in a blink of an eye. And (read above), since the code of this library is obfuscated, there is no way to figure out what's wrong there without spending a large amount of time. We would've bought the license (as this is not a freeware) if the library weren't obfuscated. All together that basically kills the library for our purposes.
This is where it gets interesting. I can easily create and populate a temporary table, e.g. (I have a table called EFAgents with an int PK called AgentId in the database):
private string GetTmpAgentSql(IEnumerable<int> agentIds) => #$"
drop table if exists #tmp_Agents;
create table #tmp_Agents (AgentId int not null, primary key clustered (AgentId asc));
{(agentIds
.Chunk(1_000)
.Select(e => $#"
insert into #tmp_Agents (AgentId)
values
({e.JoinStrings("), (")});
")
.JoinStrings(""))}
select 0 as Result
";
private const string AgentSql = #"
select a.* from EFAgents a inner join #tmp_Agents t on a.AgentID = t.AgentId";
where GetContext returns EF Core database context and JoinStrings comes from Unity.Interception.Utilities and then use it as follows:
private async Task<List<EFAgent>> GetAgents(List<int> agentIds)
{
var tmpSql = GetTmpAgentSql(agentIds);
using var ctx = GetContext();
// This creates a temporary table and populates it with the ids.
// This is a proprietary port of EF SqlQuery code, but I can post the whole thing if necessary.
var _ = await ctx.GetDatabase().SqlQuery<int>(tmpSql).FirstOrDefaultAsync();
// There is a DbSet<EFAgent> called Agents.
var query = ctx.Agents
.FromSqlRaw(AgentSql)
.Join(ctx.Agents, t => t.AgentId, a => a.AgentId, (t, a) => a);
var sql = query.ToQueryString() + Environment.NewLine;
// This should provide a valid SQL; https://entityframework-extensions.net does NOT!
// WriteLine - writes to console or as requested. This is irrelevant to the question.
WriteLine(sql);
var result = await query.ToListAsync();
return result;
}
So, basically, I can do what I need in two steps:
using var ctx = GetContext();
// 1. Create a temp table and populate it - call GetTmpAgentSql.
...
// 2. Build the join starting from `FromSqlRaw` as in example above.
This is doable, half-manual, and it is going to work.
The question is how to do that in one step, e.g., call:
.WhereMyBulkContains(aListOfIdConstraints, whateverElseIsneeded, ...)
and that's all.
I am fine if I need to pass more than one parameter in each case in order to specify the constraints.
To clarify the reasons why do I need to go into all these troubles. We have to interact with a third party database. We don't have any control of the schema and data there. The database is large and poorly designed. That resulted in some ugly EFC LINQ queries. To remedy that, some of that ugliness was encapsulated into a method, which takes IQueryable<T> (and some more parameters) and returns IQueryable<T>. Under the hood this method calls WhereBulkContains. I need to replace this WhereBulkContains by, call it, WhereMyBulkContains, which would be able to provide correct ToQueryString representation (for debugging purposes) and be performant. The latter means that SQL should not contain in clause with hundreds (and even sometimes thousands) of elements. Using inner join with a [temp] table with a PK and having an index on the FK field seem to do the trick if I do that in pure SQL. But, ... I need to do that in C# and effectively in between two LINQ method calls. Refactoring everything is also not an option because that method is used in many places.
Thanks a lot!
I think you really want to use a Table Valued Parameter.
Creating an SqlParameter from an enumeration is a little fiddly, but not too difficult to get right;
CREATE TYPE [IntValue] AS TABLE (
Id int NULL
)
private IEnumerable<SqlDataRecord> FromValues(IEnumerable<int> values)
{
var meta = new SqlMetaData(
"Id",
SqlDbType.Int
);
foreach(var value in values)
{
var record = new SqlDataRecord(
meta
);
record.SetInt32(0, value);
yield return record;
}
}
public SqlParameter ToIntTVP(IEnumerable<int> values){
return new SqlParameter()
{
TypeName = "IntValue",
SqlDbType = SqlDbType.Structured,
Value = FromValues(values)
};
}
Personally I would define a query type in EF Core to represent the TVP. Then you can use raw sql to return an IQueryable.
public class IntValue
{
public int Id { get; set; }
}
modelBuilder.Entity<IntValue>(e =>
{
e.HasNoKey();
e.ToView("IntValue");
});
IQueryable<IntValue> ToIntQueryable(DbContext ctx, IEnumerable<int> values)
{
return ctx.Set<IntValue>()
.FromSqlInterpolated($"select * from {ToIntTVP(values)}");
}
Now you can compose the rest of your query using Linq.
var ids = ToIntQueryable(ctx, agentIds);
var query = ctx.Agents
.Where(a => ids.Any(i => i.Id == a.Id));
I would propose to use linq2db.EntityFrameworkCore (note that I'm one of the creators). It has built-in temporary tables support.
We can create simple and reusable function which filters records of any type:
public static class HelperMethods
{
private class KeyHolder<T>
{
[PrimaryKey]
public T Key { get; set; } = default!;
}
public static async Task<List<TEntity>> GetRecordsByIds<TEntity, TKey>(this IQueryable<TEntity> query, IEnumerable<TKey> ids, Expression<Func<TEntity, TKey>> keyFunc)
{
var ctx = LinqToDBForEFTools.GetCurrentContext(query) ??
throw new InvalidOperationException("Query should be EF Core query");
// based on DbContext options, extension retrieves connection information
using var db = ctx.CreateLinqToDbConnection();
// create temporary table and BulkCopy records into that table
using var tempTable = await db.CreateTempTableAsync(ids.Select(id => new KeyHolder<TKey> { Key = id }), tableName: "temporaryIds");
var resultQuery = query.Join(tempTable, keyFunc, t => t.Key, (q, t) => q);
// we use ToListAsyncLinqToDB to avoid collission with EF Core async methods.
return await resultQuery.ToListAsyncLinqToDB();
}
}
Then we can rewrite your function GetAgents to the following:
private async Task<List<EFAgent>> GetAgents(List<int> agentIds)
{
using var ctx = GetContext();
var result = await ctx.Agents.GetRecordsByIds(agentIds, a => a.AgentId);
return result;
}
First, I apologise if this is a dupe, finding the right search terms seemed impossible...
We are trying to adopt some best practice and looking at refactoring duplicate code in our projects. On a number of occasions we have something like;
public List<EventModel> GetEvents(bool showInactive, bool showPastEvents)
{
return eventRepository
.GetEvents(_customerId, showInactive, showPastEvents)
.Select(e => New EventModel() { Id = e.EventId, Name = e.EventName, Capacity = e.EventCapacity, Active = e.EventActive })
.ToList();
}
So we tried doing something like this instead;
public List<EventModel> GetEvents(bool showInactive, bool showPastEvents)
{
return eventRepository
.GetEvents(_customerId, showInactive, showPastEvents)
.Select(e => ConvertPocoToModel(e))
.ToList();
}
private EventModel ConvertPocoToModel(TsrEvent tsrEvent)
{
EventModel eventModel = new EventModel()
{
Id = tsrEvent.EventId,
Name = tsrEvent.EventName,
Capacity = tsrEvent.EventCapacity,
Active = tsrEvent.EventActive
};
return eventModel;
}
Sometimes this works, but intermittently we get;
System.NotSupportedException: 'LINQ to Entities does not recognize the
method 'Bll.Models.EventModel ConvertPocoToModel(Dal.Pocos.TsrEvent)'
method, and this method cannot be translated into a store expression.'
I am aware we could add .ToList() or similar to force the conversion to happen in C# but I believe that means SQL will execute SELECT * instead of SELECT EVentId, EventName, EventCapacity, EventActive
Can anyone explain;
Why EF is having issues trying to understand how to handle this simple mapping?
why it work intermittently?
How we should be doing it?
Entity framework doesnt know how to translate your method. You have to use method which returns Expression<Func<TsrEvent,EventModel>> or an property which stores it.
public List<EventModel> GetEvents(bool showInactive, bool showPastEvents)
{
return eventRepository
.GetEvents(_customerId, showInactive, showPastEvents)
.Select(ConvertPocoToModelExpr)
.ToList();
}
private static Expression<Func<TsrEvent,EventModel>> ConvertPocoToModelExpr => (x)=>new EventModel()
{
Id = x.EventId,
Name = x.EventName,
Capacity = x.EventCapacity,
Active = x.EventActive
};
You have to be aware about the differences between an IEnumerable and an IQueryable.
An IEnumerable object holds everything to enumerate over the sequence. You can ask for the first element, and once you've got an element you can ask for the next one, as long as there is a next one. The IEnumerable is meant to be processes locally by your process.
Enumeration at its lowest level is done by asking for the Enumerator and repeatedly calling MoveNext, until you don't need anymore elements. Like this:
IEnumerable<Student> students = ...
IEnumerator<Student> studentEnumerator = students.GetEnumerator();
while (studentEnumerator.MoveNext())
{
// there is still a Student to process:
Student student = studentEnumerator.Current;
ProcessStudent(student);
}
You can do this explicitly, or call it implicitly using foreach or one of the LINQ functions.
On the other hand, an IQueryable is meant to be processed by a different process, usually a database management system. The IQueryable holds an Expression and a Provider. The Expression expresses the query that must be performed in some generic format. The Provider knows who must execute the query (usually a database management system), and the language that this process uses (usually something SQL like).
As soon as you start enumerating by calling GetEnumerator, the Expression is sent to the Provider, who tries to translate the Expression into SQL and executes the query. The fetched data is put into an enumerable sequence, and the enumerator is returned.
Back to your question
The problem is, that SQL does not know ConvertPocoToModel. Hence your provider can't convert the Expression. The compiler can't detect this, because it does not know how smart your Provider is. That is why you don't get this error until you call GetEnumerator, in your case by calling ToList.
Solution
The solution is to make a function that changes the expression. The easiest method would be an extension function. See extension methods demystified. This way you can use it like any other LINQ method:
public static IQueryable<EventModel> ToEventModels(this IQueryable<TsrEvent> tsrEvents)
{
return tsrEvent.Select(tsrEvent => new EventModel
{
Id = tsrEvent.EventId,
Name = tsrEvent.EventName,
Capacity = tsrEvent.EventCapacity,
Active = tsrEvent.EventActive
};
}
Note that I omit the () in the constructor: SQL can't call constructors!
Usage:
var result = dbContext.TsrEvents
.Where(tsrEvent => tsrEvent.Active && tsrEvent.Date == Today)
.ToEventModels()
.GroupBy(...)
... etc
Or, if your GetEvents returns an IQueryable<TsrEvents>
return eventRepository.GetEvents(_customerId, showInactive, showPastEvents)
.ToEventModels();
Final Remark
It is better to let your data-fetch-functions return IQueryable<...> and IEnumerable<...> as long as possible. Let only the end-user materialize the query. It would be a waste of processing power if you do the ToList() and your caller only wants to do FirstOrDefault()
I have a Entity called "Client", and another Entity called "Card".
A Client may have many Cards.
My Client Entity looks like this:
public class Client{
public virtual ICollection<Card> Cards {get; set;}
}
Now I want to show the Client data in a DataGrid in WPF, and I want to get Cards Count data,so I add a property to Client Entity, which like this:
public class Client{
public virtual ICollection<Card> Cards {get; set;}
public int CardCount
{
return Cards.Count;
}
}
And then I query the data with Linq and Bind to view
var query = from n in db.Clients select n;
When I run the Application, I got a Exception just right on the return Cards.Count; line;
System.ObjectDisposedException
The ObjectContext instance has been disposed and can no longer be used for operations that require a connection.
So how could I correctly get the cards count?
There is a way simpler method than the other answers here show. Please also realize that solutions such as
var client = db.Clients.FirstOrDefault(c=> c.Id = someid); //get a client
if (client != null)
{
cardCount = client.Cards.Count;
}
will cause an issue called Select N+1 problem. Read up on it if interested, but in a nutshell, it means the following:
Because you are not only interested in one exact client, but you want to display N clients, you need to do one (1) query to get just the clients. Then, by doing the FirstOrDefault stuff, you are actually doing one (1) extra db roundtrip to the database per Client record, which results in an additional N * 1 = N roundtrips. What this means that, if you were to just query the Clients without any related data, you could get however many client records you like, in just one query. But by fetching related data to each of them one-by-one, you are doing way too many db roundtrips.
Here is a way to solve this issue, by using joins and projections. You can get all the data you need in a single DB access:
using (var context = GetDbContext())
{
return context.Clients.Select(cli => new YourViewModel
{
Name = cli.FullName,
// Other prop setters go here
CardCount = cli.Cards.Count
}).Skip((page - 1) * pageSize).Take(pageSize).ToList();
}
You might be wondering, what's the difference afterall? Well, here, you are not working with materialized objects, as others call them here, but with a DbContext. By applying the proper LINQ operators to it (note, that this works not just with DbContext, but also with any IQueryable (well obviously not if you call AsQueryable() on an already in-memory collection but whatever)), LINQ to Entities can construct a proper SQL to join the tables and project the results and therefore you fetch all required data in one go. Note that LINQ to Entities IS ABLE to translate the cli.Cards.Count into a proper SQL Count statement.
You can get the count without loading the entities like this:
using (var context = new MyContext())
{
var client = context.Client.Find(clientId);
// Count how many cards the client has
var cardsCount = context.Entry(client)
.Collection(b => b.Cards)
.Query()
.Count();
}
More information on MSDN page.
You get an ObjectDisposedException if you do not materialize the retreived query. In the following case, the query gets executed only when you Access the first time the list from GetNonMaterialized and not before leaving the method. Fact of this the db is disposed because of lost of scope.
public IEnumerable<Client> GetNonMaterialized()
{
return from n in db.Clients select n;
}
In the following case the query is executed before leaving the method.
public IEnumerable<Client> GetMaterialized()
{
return (from n in db.Clients select n).ToList();
}
Always be sure that the query is executed before exiting the scope of a ObjectContext.
If you want to know whether the query is executed and when enalbe Logging of EF.
How can I turn off Entity Framework 6.1 logging?
I would love a solution to my current problem, but I would love EVEN MORE if I can understand what that error actually means.
I have LINQ to SQL classes for two tables in my DB: Providers and Assignments. I added the following member to the Provider class:
public IEnumerable<Assignment> Assignments
{
get
{
return (new linqDataContext())
.Assignments
.Where(a => a.ProviderID == this.ProviderID);
}
}
Then, I bind a GridView using a query that pulls from the parent Provider and uses the child member Assignments, like this:
protected void PopulateProviders()
{
linqDataContext context = new linqDataContext();
var list = from p in context.Providers
where (p.Assignments.Count(a => a.Category == ddlCategory.SelectedValue) > 0)
select p;
lvProviders.DataSource = list;
lvProviders.DataBind();
}
At .DataBind(), when it actually runs the query, it throws the following error:
Member access 'System.String Category' of 'Namespace.Assignment' not legal on type 'System.Collections.Generic.IEnumerable`1[Namespace.Assignment].
I've tried checking for nulls (a => a != null && a.Category ...) but that hasn't worked. I'm not sure what to try next. I think that, if I knew what the error is trying to tell me, I could find the solution. As it stands, I don't know why member access would be illegal.
That Assignments property is all wrong. First of all, property getters should not have side-effects, and more importantly, entity classes should never have reverse dependencies on the DataContext. Linq to SQL has no way to decipher this query; it's relying on a property that does all sorts of crazy stuff that Linq to SQL can't hope to understand.
Get rid of that Assignments property now. Instead of doing that, you need to write this query as a join:
int category = (int)ddlCategory.SelectedValue;
var providers =
from p in context.Providers
join a in context.Assignments
on p.ProviderID equals a.ProviderID
into g
where g.Count(ga => ga.Category == category) > 0
select p;
That should do what you're trying to do if I understood the intent of your code correctly.
One last side note: You never dispose properly of the DataContext in any of your methods. You should wrap it like so:
using (var context = new linqDataContext())
{
// Get the data here
}
I think somewhere it doesn't know that type that is in the IEnumerable. You are trying to call a method that is not part of the IEnumerable inteface.
Why don't you just move the query from the property out to the PopulateProviders() method?
Remove your custom-defined Assignments property. In the your Linq-To-SQL dbml file, create an association between Providers and Assignments, with Providers as the parent property, and ProviderID as the Participating Property for both entities. LINQ will generate a property "IEnumerable Assignments" based on matches between ProviderID using a consistent DataContext.
Here is the code I'm working with, I'm still a bit new to LINQ, so this is a work in progress. Specifically, I'd like to get my results from this query (about 7 columns of strings, ints, and datetime), and return them to the method that called the method containing this LINQ to SQL query. A simple code example would be super helpful.
using (ormDataContext context = new ormDataContext(connStr))
{
var electionInfo = from t1 in context.elections
join t2 in context.election_status
on t1.statusID equals t2.statusID
select new { t1, t2 };
}
(In this case, my query is returning all the contents of 2 tables, election and election_status.)
Specifically, I'd like to get my
results from this query (about 7
columns of strings, ints, and
datetime), and return them
Hi, the problem you've got with your query is that you're creating an anonymous type. You cannot return an anonymous type from a method, so this is where you're going to have trouble.
What you will need to do is to create a "wrapper" type that can take an election and an election_status and then return those.
Here's a little sample of what I'm talking about; as you can see I declare a Tuple class. The method that you will wrap your query in returns an IEnumerable.
I hope this helps :-)
class Tuple
{
Election election;
Election_status election_status;
public Tuple(Election election, Election_status election_status)
{
this.election = election;
this.election_status = election_status;
}
}
public IEnumerable<Tuple> getElections()
{
IEnumerable<Tuple> result = null;
using (ormDataContext context = new ormDataContext(connStr))
{
result = from t1 in context.elections
join t2 in context.election_status
on t1.statusID equals t2.statusID
select new Tuple(t1, t2);
}
}
UPDATE
Following from NagaMensch's comments, a better way to achieve the desired result would be to use the built in LINQ to SQL associations.
If you go to your entity diagram and click on toolbox, you will see 3 options. Class, Association and Inheritance. We want to use Association.
Click on Association and click on the ElectionStatus entity, hold the mouse button down and it will allow you to draw a line to the Election entity.
Once you've drawn the line it will ask you which properties are involved in the association. You want to select the StatusId column from the Election entity, and the StatusId column from the ElectionStatus entity.
Now that you've completed your mapping you will be able to simplify your query greatly because the join will not be necessary. You can just access the election status via a brand new property that LINQ to SQL will have added to the Election entity.
Your code can now look like this:
//context has to be moved outside the function
static ExampleDataContext context = new ExampleDataContext();
//Here we can return an IEnumerable of Election now, instead of using the Tuple class
public static IEnumerable<Election> getElections()
{
return from election in context.Elections
select election;
}
static void Main(string[] args)
{
//get the elections
var elections = getElections();
//lets go through the elections
foreach (var election in elections)
{
//here we can access election status via the ElectionStatus property
Console.WriteLine("Election name: {0}; Election status: {1}", election.ElectionName, election.ElectionStatus.StatusDescription);
}
}
You can also find a "how to" on LINQ to SQL associations here.
Note: It's worth mentioning that if you have an FK relationship set up between your tables in the database; LINQ to SQL will automatically pick the relationship up and map the association for you (therefore creating the properties).
You'll need to create classes that have the same structure as the anonymous types. Then, instead of "new { t1, t2 }", you use "new MyClass(t1, t2)".
Once you have a named class, you can pass it all over the place as you were hoping.
The problem is, that you are creating a anonymous type, hence there is no way to declare a method with this return type. You have to create a new type that will hold your query result and return this type.
But I suggest not to return the result in a new type but return just a colection of election objects and access the election_status objects through the relation properties assuming you included them in your model. The data load options cause the query to include the related election status objects in the query result.
public IList<election> GetElections()
{
using (ormDataContext context = new ormDataContext(connStr))
{
DataLoadOptions dlo = new DataLoadOptions();
dlo.LoadWith<election>(e => e.election_status);
context.DeferredLoadingEnabled = false;
context.LoadOptions = dlo;
return context.elections.ToList();
}
}
Now you can do the following.
IList<election> elections = GetElections();
// Access the election status.
Console.WriteLin(elections[0].election_status);
I general LINQ to SQL could just retrieve the related entities on demand - that is called deferred loading.
ormDataContext context = new ormDataContext(connStr));
IList<election> elections = context.elections.ToList();
// This will trigger a query that loads the election
// status of the first election object.
Console.WriteLine(elections[0].election_status);
But this requires you not to close the data context until you finished using the retrieved objects, hence cannot be used with a using statement encapsulated in a method.
IEnumerable<object> getRecordsList()
{
using (var dbObj = new OrderEntryDbEntities())
{
return (from orderRec in dbObj.structSTIOrderUpdateTbls
select orderRec).ToList<object>();
}
}
maybe it can help :)
using (ormDataContext context = new ormDataContext(connStr))
{
var electionInfo = from t1 in context.elections
join t2 in context.election_status
on t1.statusID equals t2.statusID
select new {
t1.column1,
t1.column2,
t1.column3,
t1.column4,
t2.column5,
t2.column6,
t2.column7
};
}
foreach (var ei in electionInfo){
//write what do u want
}
return electionInfo.ToList();
You cannot return an anonymous type from a method. It is only available within the scope in which it is created. You'll have to create a class instead.