Develop Reference

EntityFrameworkCore reading data using reflection - how to create DbSet when type unknown

I am working with a .Net 6 Console application where I need to read data from tables in a custom DbContext using Microsoft.EntityFrameworkCore
I have added the entities to the model in OnModelCreating() and can get them back using a call to
var entity = ctx.Model.GetEntityTypes().FirstOrDefault(e => e.FullName().InfexOf(tableName) >= 0);
Given that, how to I retrieve a list of data, for example entity.ToList() - the type returned for entity is IEntityType?.
As an alternate (and my preferred way if possible), I have created an array of tables using reflection (they all inherit from BaseTable), they are stored as a list.
I would like to create a DbSet<> using DbContext.Set() so that I can use Find(), AsNoTracking() and other such commands (including write operations).
I have the following:-
IQueryable<Object>dbSet = (IQueryable<Object>)ctx
.GetType()
.GetMethod("Set",1,Type.EmptyTypes)
.MakeGenericMethod(t)
.Invoke(ctx, null);
Which allows me to do something like dbSet.ToList(), but I would really like to cast it to a DbSet.
Does anyone know if it is possible to make such a conversion?
(I am reading only a few records from sets of tables and then writing data back to a different database (with the same tables).
Update: *Another way of thinking about this: I am iterating across a collection of tables. I need to pull out the PK and two other columns (for which I have the name of at runtime) - if the value of column 1 contains a specific value, I need to update the value of column 2 *

You can't really do that. However, you can cast the database value to specific types based on discriminators. Take a look at this https://learn.microsoft.com/en-us/ef/core/modeling/value-conversions?tabs=data-annotations

Combining C# Expressions with ADO.NET Datatables

I have generic queries to which the return type can be different. Because of this I cannot use TVFs and thus I am using Datatables.
I also want to extend the query of the datatable.
I am trying to do this in the following way:
var data = GetDataTable($"SELECT * FROM {tablename}").AsEnumerable().AsQueryable().GetFilteredList(filters);
The following is the function definition of GetFilteredList:
IQueryable<T> GetFilteredList<T>(this IQueryable<T> items, List<PostedFilter> filters)
The logic within the functions GetDataTable and GetFilteredList is correct as they are already in use for mutltiple years. They are however used seperately, as they come from different libraries. The filters parameter contains strings which are the names of properties of the queries. This way the query can be expanded before being executed. This works perfectly fine with static mvc queries on typed EDMX objects.
This code however, does not work for my Datatable. It does not generate any errors, but the filters do not reduce the data either. (I assume this is, among other reasons, because the query is materialized before the AsQueryable function is called)
Does anyone know a way in which I can create the logic which I am trying to implement? (By this I mean creating one large query that is only materialized after the query has been built up completely)

Could it be that GetDataTable().AsEnumarable() returns a list of DataRow's, while the GetFilteredList is typed to an entity class of some sort?
If that's the case then you should somehow convert your 'filters' to a string in the form of WHERE (Filter1 = 'value1') AND ... and put it in your SQL string you're passing to GetDataTable().

LINQ to Entities - Entity Framework

I'm looking to get a better understanding on when we should look to use IEnumerable over IQueryablewith LINQ to Entities.
With really basic calls to the database, IQueryable is way quicker, but when do i need to think about using an IEnumerable in its place?
Where is an IEnumerable optimal over an IQueryable??

Basically, IQueryables are executed by a query provider (for example a database) and some operations cannot be or should not be done by the database. For example, if you want to call a C# function (here as an example, capitalize a name correctly) using a value you got from the database you may try something like;
db.Users.Select(x => Capitalize(x.Name)) // Tries to make the db call Capitalize.
.ToList();
Since the Select is executed on an IQueryable, and the underlying database has no idea about your Capitalize function, the query will fail. What you can do instead is to get the correct data from the database and convert the IQueryable to an IEnumerable (which is basically just a way to iterate through collections in-memory) to do the rest of the operation in local memory, as in;
db.Users.Select(x => x.Name) // Gets only the name from the database
.AsEnumerable() // Do the rest of the operations in memory
.Select(x => Capitalize(x)) // Capitalize in memory
.ToList();
The most important thing when it comes to performance of IQueryable vs. IEnumerable from the side of EF, is that you should always try to filter the data using an IQueryable to get as little data as possible to convert to an IEnumerable. What the AsEnumerable call basically does is to tell the database "give me the data as it is filtered now", and if you didn't filter it, you'll get everything fetched to memory, even data you may not need.

IEnumerable represents a sequence of elements which you enumerate one by one until you find the answer you need, so for example if I wanted all entities that had some property greater than 10, I'd need to go through each one in turn and return only those that matched. Pulling every row of a database table into memory in order to do this would not maybe be a great idea.
IQueryable on the other hand represents a set of elements on which operations like filtering can be deferred to the underlying data source, so in the filtering case, if I were to implement IQueryable on top of a custom data source (or use LINQ to Entities!) then I could give the hard work of filtering / grouping etc to the data source (e.g. a database).
The major downside of IQueryable is that implementing it is pretty hard - queries are constructed as Expression trees which as the implementer you then have to parse in order to resolve the query. If you're not planning to write a provider though then this isn't going to hurt you.
Another aspect of IQueryable that it's worth being aware of (although this is really just a generic caveat about passing processing off to another system that may make different assumptions about the world) is that you may find things like string comparison work in the manner they are supported in the source system, not in the manner they are implemented by the consumer, e.g. if your source database is case-insensitive but your default comparison in .NET is case-sensitive.

Is IEnumerable a good return type for data that allows filtering?

I just read this question and it got me thinking of why I would need to use the IEnumerable when retrieving data. I understand the differences between IQueryable and IEnumerable, but would an IEnumerable object be better for data that allows filtering? for example, a table with the records which contain a date, so I can sort on the date.

If you have the objects in memory - not from another data source such as a database - use IEnumerable<T>. That way the built in LINQ to Objects will work automatically.
You could even extend LINQ to Objects by writing custom extension methods and using yield return and yield break.
If you are using Entity Framework or some other system that is using IQueryable<T>, I would keep it as IQueryable<T> until you need it as objects.

It depends.
If you want to First recieve all Data and afterwards filter the returned Set of Objects use IEnumerable.
If you want to enable Filtering on the Database (e.g. Linq-to-sql / Entity Framework) better use IQueryable.

How do I sort a gridview of Linq objects based on a derived field?

I have written a page which uses Linq to query the database and bind the resulting IQueryable to a datagrid. I have a partial class which contains extra properties which derive their values based on other values brought in from the database.
Sorting works fine on fields that are actually in the database but not for the derived fields. When I attempt to sort on such a field I get an error saying "The member 'Trip.Difference' has no supported translation to SQL.
Any idea how to allow sorting on these derived fields?

The problem is that you are binding to an IQueryable, so every time you enumerate it, you are translating the LINQ expression on the IQueryable to a SQL statement and going back to the database to execute it.
If you are trying to sort on properties that are not bound to the database model then you will get the error mentioned, as those properties only exist once an object has been created from a data row.
The simplest solution is to call ToList() on the IQueryable before using it for sorting and data-binding, so that you sort on the in-memory objects where the properties are actually available. ToList() converts your IQueryable into an IEnumerable (via List<T>), and stops it from going to the database again via LINQ to SQL.
This is generally a good design pattern to follow - the last thing you want is consumers of your business layer being able to unwittingly execute arbitrary queries against your database, simply because you returned IQueryable where you should have returned IEnumerable.

Call ToEnumerable() first, and then add the OrderBy:
var q = (from a in mDataContext.Somethings()
select a).ToEnumerable().OrderBy...