Develop Reference

EntityFrameworkCore reading data using reflection - how to create DbSet when type unknown

I am working with a .Net 6 Console application where I need to read data from tables in a custom DbContext using Microsoft.EntityFrameworkCore
I have added the entities to the model in OnModelCreating() and can get them back using a call to
var entity = ctx.Model.GetEntityTypes().FirstOrDefault(e => e.FullName().InfexOf(tableName) >= 0);
Given that, how to I retrieve a list of data, for example entity.ToList() - the type returned for entity is IEntityType?.
As an alternate (and my preferred way if possible), I have created an array of tables using reflection (they all inherit from BaseTable), they are stored as a list.
I would like to create a DbSet<> using DbContext.Set() so that I can use Find(), AsNoTracking() and other such commands (including write operations).
I have the following:-
IQueryable<Object>dbSet = (IQueryable<Object>)ctx
.GetType()
.GetMethod("Set",1,Type.EmptyTypes)
.MakeGenericMethod(t)
.Invoke(ctx, null);
Which allows me to do something like dbSet.ToList(), but I would really like to cast it to a DbSet.
Does anyone know if it is possible to make such a conversion?
(I am reading only a few records from sets of tables and then writing data back to a different database (with the same tables).
Update: *Another way of thinking about this: I am iterating across a collection of tables. I need to pull out the PK and two other columns (for which I have the name of at runtime) - if the value of column 1 contains a specific value, I need to update the value of column 2 *

You can't really do that. However, you can cast the database value to specific types based on discriminators. Take a look at this https://learn.microsoft.com/en-us/ef/core/modeling/value-conversions?tabs=data-annotations

EF Core 5 - How can I use EF.Functions.Like with a custom property that maps to a JSON string?

In one of my DB models I have a property with a custom type (Dictionary<string, string>), which is automatically converted to/from JSON with a custom converter, and is stored as a text field in the database. I would like to be able to use MySQL's LIKE comparer to search for a string within this JSON field, but I am getting an exception. I don't care about the JSON's structure, I'm fine with treating it as a simple text field and searching in it that way.
Here's how I try to do it (Sku is the complex type):
var responseSet = database.Products.Where(p => EF.Functions.Like(p.Sku, "%query%"));
And this is the exception I get:
An unhandled exception has occurred while executing the request.
System.InvalidOperationException: The LINQ expression 'DbSet<ProductObject>()
.Where(p => __Functions_0
.Like(
matchExpression: p.Sku, pattern: __Format_1))' could not be translated.
Additional information:
Translation of method 'Microsoft.EntityFrameworkCore.MySqlDbFunctionsExtensions.Like'
failed. If this method can be mapped to your custom function,
see https://go.microsoft.com/fwlink/?linkid=2132413 for more information.
The link in the exception points to a long git issue with tons of cross-references, but I haven't been able to find anything useful in it.
Is there a way I could prevent this error from happening and search within the complex field?

The fundamental problem with the EF Core value converters is that the LINQ query is build against client type, which then is translated behind the scenes to provider type, and there is no standard way to specify provider type conversion inside the query.
However there is a simple trick with casting (presented in some my answers to other conversion related issues like How can a JSON_VALUE be converted to a DateTime with EF Core 2.2?, Expression tree to SQL with EF Core or Comparing strings as dates using EF core 3), which works for most of the EF Core relational database providers.
In this case, you know that the CLR equivalent of the provider type is string, so you can use the following cast
p => EF.Functions.Like((string)(object)p.Sku, "%query%")
The intermediate cast to object is needed to fool C# compiler to accept the actual cast. EF Core translator is smart enough to remove it (as well as others when not needed like here).

This isn't a supported feature yet, but there is a hacky workaround that might work for you. Define a custom database function to explicitly cast the expression to the underlying db type.
public static string Cast(YourComplexType t) => throw new NotSupportedException();
modelBuilder.HasDbFunction(() => Cast(default))
.HasTranslation(args =>
{
var a = args.First();
return new SqlUnaryExpression(
ExpressionType.Convert,
a,
typeof(string),
new StringTypeMapping(a.TypeMapping.StoreType, DbType.String));
});
EF.Functions.Like(Cast(p.Sku), "%query%")
Unfortunately you can't define generic db functions. The arguments all have to be types the provider can store, which rules out object or an interface. So you may have to define overloaded methods for every possible type.

oData EndPoint: Does it affect the SQL query?

I’ve been using OData for my apis.
While I generally like what is has to offer, it only uses the data Post my query, which forces me to to construct all the relationships ahead of time.
Does an oData EndPoint with EntityFramework pass my oData parameters to be execute pre my SQL query?
Right now if I plan to possibly use oData Syntaxes like $Expand, I have to use EF Include ahead of time. Once again, the issue being that EF must build all of the potential relationships that I may use $Expand with...even if I don’t $expand anything.
Another example is if I am to use the $top(100) syntax. Say I had 10000 results, EF will download all 10000 from the DB, and then OData will select the Top 100.
Would an oData endpoint inject itself between EF and the DB and only select 100 results from the DB in this case?

In general OData and EF go hand in hand, OData translates the incoming HTTP request into a Linq-to-Entities expression that EF then translates to a SQL expression.
tl;dr
All of your comments and observations point to incorrect implementation within your controller, it sounds suspiciously as if you have followed a Repository Pattern based example instead of an EF based examples.
Does an oData EndPoint with EntityFramework pass my oData parameters to be execute pre my SQL query?
That is exactly what the OData framework was designed to enable, but there is 1 caveat, you must configure your controllers in a way that the parameters can be passed through.
There are two mechanisms that allow this to happen, the first is that your controller needs to return an IQueryable<T> result (or you must pass an IQueryable<T> to one of the negotiated response handlers). The other is that you must not apply your own filter expressions that might contradict the parameters, otherwise you may result in no records being returned.
The following is an example of the two standard Get endpoints on an OData controller that will return a Vehicle query that will allow the $expand,$select and $filter expressions to be passed through:
[ODataRoute]
[HttpGet]
[OData.EnableQuery]
public IHttpActionResult Get(ODataQueryOptions<Vehicle> queryOptions)
{
return Ok(db.Vehicles);
}
[ODataRoute]
[HttpGet]
[OData.EnableQuery]
public IHttpActionResult Get([FromOdataUri] int key, ODataQueryOptions<Vehicle> queryOptions)
{
return SingleResultAction(db.Vehicles.Where(x => x.Id == key));
}
If I call this with the following query options:
$filter=Make eq Holden and Model eq Commodore&$orderby=Year desc
then that will translate into a SQL query similar to this:
(SELECT * would be expanded in full)
DECLARE #make varchar(20) = 'Holden';
DECLARE #model varchar(20) = 'Commodore';
SELECT *
FROM Vehicle
WHERE Make = #make
AND Model = #model
ORDER BY Year DESC
Yes the parameters will be properly parameterised as well!
This simple controller will also pass through the $expand request and automatically join the necessary tables, there is no need for us to know or think about the potential includes up front at all.
In fact, if I add $skip=5 and $top=2 those clauses will also be applied directly to the SQL statement, and at most only 2 rows will be returned from the database.
There are two common scenarios where all this auto-magical query translation and pass through mumbo jumbo gets broken.
In the controller method, (for either of the collection or item) you do not return an IQueryable result, or you have otherwise executed the query and resolved it to an IEnumerable and then you have returned that, perhaps by casting it back to an IQueryable.
This tends to happen because when we start out we tend follow simple non EF based OData examples, they usually follow a repository pattern because it is simple to explain the model, example data and implementation in a single page of code.
Public Service Announcement : Do not try to use a Repository Pattern to wrap an EF model and context, almost all the benefits from EF will be lost. EF is a Unit of work pattern, OData is designed to work with it directly.
The following controller implementation will NOT pass through the query options to the database, $top,$skip,$filter,$select,$orderby will still be applied, if it can, but only after retrieving all the records into memory first:
return Ok(db.Vehicles.ToList());
$expand will not be applied, or rather it will result in NULL related records.
The other common issue is after Actions where a data record (or records) have been processed if we want to support the entire range of query options automatically, we again need to make sure we return an IQueryable expression that queries from the DbContext. If the expression is IQueryable, but is only querying against what is already in memory, then $expand and $filter for instance can only be applied to that data which is already loaded.
SingleResultAction is a good helper method for returning an IQueryable that has been scoped toa single item, but it will still allow the full range of QueryOptions to be applied.

Welcome to a world of pain - Odata is a great idea with a pretty bad implementation. But yes, it is a amazing - doing it myself.
it only uses the data Post my query, which forces me to to construct all the
relationships ahead of time.
If this asks what I think it does (your english is VERY unclear) then no, it does not - your fault is exposing ef objects directly. I have separate api objects and expose them using AutoMapper ProjectTo. Yes, I need to define relationships ahead of time, but not on ef level.
I have to use EF Include ahead of time.
That is because you decide to. I acutally use, as I said, Automapper ProjectTo - and get the necessary expands from the OdataOptions SelectExpand information dynamically. No need to send the whole object graph to EF (which is what happens if you expand all possible expansions with includes), this will result in really bad performance. Just a page of programming to get the relevant includes.
Would an oData endpoint inject itself between EF and the DB and only select 100 results
from the DB in this case?
if it is programmed like that, yes. If someone challenged with LINQ programs it and packs ef in one of the ridiculous inefficient repository patterns that return IENumerable - then no, it will pull it possibly all. Been there seen that. But normally TOP(1) results in sql selecting only the first item.

Entity Framework for querying JSON strings in SQL Server

I'm looking for anyone who's done anything along the lines of querying JSON strings with the Entity Framework.
I should give a little background about what I'm trying to do here. The database I'm using is for a workflow engine that I'm working on. It handles all the workflow data, and also allows you to store some custom data as a JSON string. The workflow engine I'm using handles the serializing and de-serializing of the JSON strings on a per request basis, but in the event I would want to do a query and filter based on values in the JSON string, I would have to pull the entire table into memory and de-serialize all the entries and then filter. This is, for obvious reasons, not acceptable. The reason for doing this, is we want a single workflow database that can be used for all applications that use this workflow engine, and we are trying to avoid having to do cross database views to seperate application specific databases to get the custom data. Since in most cases, the custom request data that is being stored as JSON strings is relatively simple, and in most cases isn't needed when querying, which is by design. But in the event that we do need to do custom searches, we need a way to be able to parse these custom JSON objects. And I'm hoping this can be done dynamically with Entity so I don't have to write extra stored procs for querying specific types of objects. Ideally I would just have one library that uses entity to allow querying of any JSON data structure.
I started with a database function that I found that parses JSON and returns a flattened table that contains the values (parent object id, name, value, and type). Then imported that function into my entity model. Here's a link to where I got the code from. Pretty interesting article.
Consuming JSON Strings in SQL Server
Here's the basics of where I'm at.
using (var db = new WorkflowEntities()) {
var objects = db.Requests.RequestData();
}
In the above code example, Request object is my base workflow Request object. RequestData() is an extension method on type
DbSet<Request>
and parseJSON is the name of my database function.
My plan is to write a series of extension methods that will filter the Queryables
IQueryable<parseJSON_result>
So for example, if I have an object that looks like this.
RequestDetail : {
RequestNumber: '123',
RequestType: 1,
CustomerId: 1
}
I would be able to do something like
db.Request.RequestData().Where("RequestType", 1);
or something along those lines. The .Where method would Take RequestData(), which is an IQueryable that contains the parsed JSON, it would filter and return the new IQueryable result.
So my question really is, has anyone done anything like this? If so, what kind of approach have you taken? My original intent was to do something dictionary style, but it seemed too difficult. Any thoughts, ideas, suggestions, wisdom, would be much appreciated. I worked on this for awhile, and I feel like I really didn't get that far. Which is mostly just because I can't decide how I want the syntax to look, and I'm not sure if I should be doing more work database side.
This was my original idea for syntax, but I couldn't run the [] operator without hydrating the object.
db.Request.Where(req => req.RequestData()["RequestType"] == 1).Select(req => req.RequestData()["CustomerInfo"]);
I know this is a pretty long post, so if you've read this far, thanks for just taking the time to read the whole thing.

As of SQL Server 2016, FOR JSON and OPENJSON exist, equivalent to FOR XML and OPENXML. You can Index on expressions that reference JSON stored in NVARCHAR columns.

This is a very late answer, but for anyone who is still searching...
As #Emyr says, SQL 2016 supports querying inside JSON columns using JSON_VALUE or OPENJSON statements.
Entity Framework still does not support this directly, but you can use the SqlQuery method to run a raw SQL command directly against the database which can query inside JSON columns and saves querying and deserializing every row in order to run a simple query.

What you could do is create a CLR SQL Server User-Defined Function then use it from your query.
See this link https://msdn.microsoft.com/en-us/library/ms131077.aspx
i would think that a table-valued functions is more suited for your situation.

Casting to and from Derived Types using LINQ

I'm somewhat new to using LINQ and Entity Framework, and am running into a snag when casting Entity Framework object types to/from objects of a class derived from that object type.
To provide context, I am selecting Survey objects from my Entity Framework DB (records from a Surveys table), for which I have created a derived class that I will actually cast these entity objects to before using them in my application - such that the derived class's signature looks something like:
public sealed class SurveyExtended : Survey
{
public SurveyExtended() : base()
{
// non-base class members initialized here
}
}
And when using LINQ to populate a collection of these objects, I am casting them to the SurveyExtended type using code similar to:
var listOfSurveyExtendedObjects = ( from record in contextFactory.SurveysDbContext.Surveys
select new SurveyExtended()
{
Name = record.Name,
Data = record.Data,
Date = record.Date
}
);
Please note, I know I could use lambda to do the same thing, but I'm just trying to illustrate a concept.
All is well and good, until I actually try and execute DML against SurveysDbContext to do things like UPDATE or DELETE the original record after processing it in my application, such as: contextFactory.SurveysDbContext.Surveys.DeleteObject( surveyExtendedObject );.
Of course this isn't going to work, because I'm manipulating SurveyExtended objects, not the original Survey entity objects, and the ObjectStateManager will throw an InvalidOperationException because the object itself cannot be found. That is to be expected.
I guess what I'm looking for are suggestions and alternative approaches to this scenario. Should I attempt to cast back to Survey objects before trying to DbContext.DeleteObject( record );, or change my approach to this problem entirely? In similar situations, what methods did/do you use, and what benefits/drawbacks do they offer?

The options that come to mind are either cast it back before saving / interacting with EF, or switch to using something like a decorator pattern where the Extended object encapsulates the Survey. Though the second option either means you need to mimic the encapsulated object exposing pass-through accessors (double the code, double the fun) or change referencing code to access SurveyExtended.Survey.Property.