Develop Reference

How to implement for a Web API project field level permission with EF Core 6 and value objects

I'm quite frustrated trying to figure out a possible implementation.
Pursuant DDD and CQS concepts the write side of my application uses various aggregates and associated repositories that are implemented with EF Core.
On the read side I want to use queries that in some cases link related data. This could be just the resolution of an id (i.e. the name of the user that made the last edit) or for performance reasons a list of child objects (i.e. address book of a person).
Therefore the structure of the returned object (GetPersonDTO) including resolutions or children is different of the write object (Person). I use value objects as types for all properties in the entities to keep validation in one spot (always valid objects).
My problems are on the read side. The returned resource representation from a GET request is JSON. Permissions associated with the requesting subject decide if a field is included in the JSON.
My idea was that I use EF Core to return a query object and a permission object that holds the field permissions of that object for the current subject (user). If the subject has read permission for a certain field it will be mapped to the DTO. The DTO uses Optional<T> and a custom JsonConverter as shown here. As a result all Optional<T> fields that are not set will not be included in the JSON response, but it preserves fields that are set to NULL.
I write the queries in EF Core using raw SQL, because I didn't manage to write more complex queries using LINQ. EF Core requires keyless entities for raw SQL queries. I expected EF Core to convert the read fields back into value objects using the converters that have been created for the write side.
But keyless entities cannot be principal end of relationship hence they cannot have owned entities. As various GitHub issues show it is not yet possible that EF Core recreates the object graph from a raw SQL query. It is stated that
In EF each entityType maps to one database object. Same database
object can be used for multiple entityTypes but you cannot specify a
database object to materialize a graph of object.
If I understand correctly it is also not possible to achieve this with a view or stored procedure. It sounds to me that it is also not possible to define another fully connected GetPerson object that uses existing DbSet objects.
How can this be implemented? What are the alternatives?
I can think of
a) using a flat object with primitive types for the raw SQL result and then use it to map to the DTO. A side effect of creating the object graph with the original is that creating the value objects validate this data from the DB. So I either have to trust the database or I need to call the validation methods that are public in my value objects manually.
b) forget EF Core and use ADO.NET. The return object is then the ADO.NET record. Considering the permissions the fields of the record will then be mapped to the DTO. This is simple with less overhead, but requires another part of the framework.
Are there any other options? How have you solved returning a combined object considering field permissions?

EF6 core does not support persisting value objects, this is a feature planned in EF7: https://learn.microsoft.com/en-us/ef/core/what-is-new/ef-core-7.0/plan#value-objects
The purpose of an ORM such as EF, is to allow programmers to manipulate the RDBMS through objects rather than a text based language such as SQL. This object model is not a business model, but symbols of RDBMS concepts (class = table, row = object, column = property, ...). For trivial applications, you can confuse your models, but you will quickly find yourself limited because a business model has different constraints than a database schema. For larger applications, you write a persistence model, consisting of DPO that match your database structure, and translate that model to other models in the infrastructure layer. Decoupling your domain model from your persistence model allows more flexibility in your application, and re-hydrating your domain objects as a polysemic model, limiting side effects of independent use cases.
An example here would be to have a normalized RDBMS with tables including surrogate keys that are hidden from the domain model in the projection process done by the repository. This allows your database to resolve the complexity of relational mapping, while re-hydrating value objects without identities in the domain layer.
For queries, you should not publish your entities with a GetPerson model. The domain layer's purpose is to protect your application from violating any business rules. When querying your application state, you do not modify the state of the application, and cannot violate any rule. A domain model is only useful for state changing use cases. Therefore, when handling a query, you should directly map your DTO from the DPO. You will save performance and allow your DTO to project directly sort/filter/paging features to the database through libraries such as AutoMapper, as long as the projection is in this library's translation capacity. Also your business rules implementation will not impact / be impacted by large and complex query models, which is the initial purpose of a CQS architecture.
Whether you manipulate your database through an ORM such as EF, or as raw SQL queries manipulated directly at ADO.NET level is an implementation detail of your infrastructure layer. Choice depends on whether you think you can write "better" queries than the ORM, "better" being a subjective matter depending on your project constraints.
Update 1: Regarding mapping to your DTO with Optional<T>, EF core has limited ability to map relational data into a model that does not simply represent the database schema. This is by design and you shouldn't force EF to try to restore data into the DTO directly. Using a single model adds adherence between your API interface and database persistence scheme. You would risk interface breaking changes each time you update the persistence schema, and vice-versa. You should have two different models to decouple presentation from persistence.
Whether you use EF core or ADO.NET to read the database does not change much conceptually. In both cases you read database information into a memory model, then translate that model into a DTO. The difference is whether this in-memory model is based on OOP (EF + DPO model) or a key-value table (ADO.NET + datarow).
Pros of using EF core, is that it is less prone to SQL injection as queries are generated and values are always escaped for you. Also, the persistence model can be translated into DTO through a mapping library, such as AutoMapper. AutoMapper makes the translation easier and cheaper to write and maintain. Also, it can project some of the translation into the relational model.
If you manage to model your security into the map profile, your database could only select columns required for data exposition in the DTO. In other words, if the user is not allowed to expose DTO.Name, then the database would not include Table.Name into the select statement. This is not always possible to do, though, but it is much easier done this way than writing "clever" queries in SQL.
One downside of EF, however, is that it is slower than ADO.NET.
If you really need to split your query into a two phase transformation (mapping and security), you should put the security layer closer to the database, unless the translation logic requires that data to map accordingly.

This is a bit of subjective and best-practice question, but I'll answer with how I've solved a similar problem - given that I actually understand your question properly.
As long as you've mapped the database model fully using navigation properties, it is possible to generate very complex queries without resorting to raw queries.
var dto = await context.Persons
.Where(p => p.Id == id)
.Select(p => new GetPersonDTO
{
Id = p.Id,
InternallyVerifiedField = !p.UsersWithAccess.Contains(currentUser) ? new Optional<int>(p.InternallyVerifiedField) : new Optional<int>(),
ExternallyVerifiedField = permissions.Contains(nameof(GetPersonDTO.ExternallyVerifiedField)) ? new Optional<int>(p.ExternallyVerifiedField) : new Optional<int>()
})
.SingleOrDefaultAsync();
In this example the InternallyVerifiedField will depend on some query inline, and ExternallyVerifiedField will depend on some external permission object. The benefit of ExternallyVerifiedField is that it might be optimized out from the expression before even reaching the sql server, if the user does not have permission.
If you want to build the dto object from a fully connected object it can still be done in one query similar to
var dto = await context.Persons
.Where(p => p.Id == id)
.Select(p => new
{
permissions = new GetPersonDTOPermissions
{
FieldA = context.Permissions.Where(...)
},
person = p
})
.SingleOrDefaultAsync();
But with this solution you need to manually craft the dto from the graph object person given the resulting permissions, as long as you start with context and add a filter using Where the query will be inlined.

How to retrieve ID generated by DB after saveChanges using DbContextScope

I plan to create an application having the following layers, and use Entity Framework as my ORM:
Presentation: not relevant for my question.
Business: in this layer I plan to only use DTO objects. I want to have my business layer separated from any database implementation details and therefor all interaction with entities is done in the DAL layer.
DAL: in this layer I plan to have all Entity Framework code and the entities. I will introduce repositories which can be called by the Business layer. These repositories expect DTO objects as input, and return DTO objects as output. Mapping between DTO and entity is done in this layer.
I have looked at quite some online tutorials, and articles related to Entity Framework and came across the DbContextScope project which seems like a very nice solution to control a 'business transaction' and ensure all related changes are committed or rolled back. See GitHub: https://github.com/mehdime/DbContextScope
The demo in that GitHub repository contains a scenario where a new entity is created in the database. When I try to map that scenario to my layers, it seems to go like this:
Business: create DTO with property values for the entity to be stored. Create new DbContextScope and call repository in the DAL layer passing the DTO.
DAL: the repository maps the DTO to an entity and add its to the DbContext of Entity Framework.
Business: call the SaveChanges() method on DbContextScope which in its turn calls the SaveChanges() on the DbContext of Entity Framework.
In the demo the ID of the entity being stored is already known when the DTO is created. However I am looking for a way to determine the ID automatically assigned by EF once the SaveChanges() method on the DbContextScope is called in the business layer. Since I am in the business layer at this point, I do not have access to the entity anymore, hence I cannot reach the ID property of that entity anymore.
I guess I can only determine the ID by querying the database for the record just created, but this is only possible if the original DTO contains some unique identifier I could use to query the database. But what if I do not have a unique value in my DTO I can use to query?
Any suggestions on how to solve this, or do you recommend an alternative approach to my layers? (e.g. use entities in business layer as well - despite this sounds like the wrong thing to do)

I use Mehdime's context scope wherever I can, as I've found it to be an exceptional implementation of a unit of work. I agree with Camilo's comment about the unnecessary separation. If EF is trusted to serve as your DAL then it should be trusted to work as designed so that you can leverage it completely.
In my case, my controllers manage the DbContextScope and I utilize a repository pattern in combination with a DDD design for my entities. The repository serves as the gate keeper for the interactions with the context scoped and located with the DbContextLocator. When it comes to creating entities, the repository serves as the factory with a "Create{X}" method where {X} represents the entity. This ensures that all required information needed to create the entity is provided, and the entity is associated with the DbContext before being returned so that the entity is guaranteed to always be in a valid state. This means that ones the context scope SaveChanges call is made, the bounding service has the entity with it's assigned ID automatically. ViewModels / DTOs are what the controller returns to the consumer. You also do have the option to call the DbContext's SaveChanges within the boundary of the DbContextScope which will also reveal IDs prior to the context scope SaveChanges. This is more of a very edge-case scenario for when you want to fetch an ID for loosely coupled entities. (No FK/mapped relationship) The repository also services "Delete" code to ensure all related entities, rules, and such are managed. While editing entities falls under DDD methods on the entity itself.
There may be a more purist argument that this "leaks" details of the domain or EF specific concerns into the controller, but my personal opinion is that the benefits of "trusting" entities and EF within the scope of the bounded context within the service layer far, far, outweighs anything else. It's simpler, and allows you a lot of flexibility in your code without the need for near-duplicate methods propagating to supply consumers with filtered data, or complex filtering logic to "hide" EF from the service layer. The basic rule I follow is entities are never returned outside of the boundary of their context scope. (No detach/reattach, just Select into ViewModels, and managing Create/Update/Delete on entities based on passed in view models/parameters.)
If there are more specific concerns / examples you can provide, please feel free to add some code outlining where you see those issues.

Using a POCO class as a domain class, but with custom link getters and setters?

I recently learned that it's a good idea (when working with Entity Framework) to design it code-first, with POCO classes that are also mixed with original domain class logic.
So I decided to use this new idea. Before (as an example) I had a POCO class called DatabasePerson and a domain class called Person. I am now trying to merge these into one, so that I can let Entity Framework manage my repository better, and manage changes to the domain layer easier.
Now, in my DatabasePerson POCO class, I have a link to a DatabaseAccount POCO class. Similarly, my Person domain class has a link to a Account domain class.
In Entity Framework, to allow for lazy loading of these types of links, I declare the link properties virtual (as in the DatabasePerson class), like so:
public virtual DatabaseAccount Account { get; set; }
However, what if I want to change how the account is set or get, and how exceptions are handled when attempting to set it to null? How can I make sure that this won't conflict with anything that Entity Framework adds to the table?
Here's my domain class' link:
public Account Account {
get {
//maybe do some other stuff here.
return account;
}
set {
if (value == null) throw new ArgumentNullException("value");
account = value;
//maybe do some other stuff here.
}
}
I want to somehow keep this form of customizability, but also have lazy loading. Is that possible?

Lazy loading conflicts with the concepts of Domain Driven Design, especially that of Aggregates. Retrieving and updating an aggregate from a repository should be a single operation. The aggregate needs to be complete as per the specification in the ubiquitous language. Introducing lazy loading breaks this rule as the aggregate isn't complete (or perhaps this is an indication that you haven't defined your aggregates properly).
In addition to the above, you are also violating one of the core principles in DDD; you are designing your domain with a strong influence of technical concerns (Entity Framework, databases, lazy loading, etc). Introducing these infrastructural 'leaks' will constrain the way in which you make design decisions. Entities and Value Objects form the absolute core of your domain. They are real-world objects that interact with each other. Persistence ignorance is key to designing a good domain model.
I will give you a short example of an aggregate, but you need to do more reading if you wish to better grasp this concept.
Let's say that you have decided that the Order entity is the root of an aggregate which includes Order and OrderLine (an Order can have 1 or many OrderLine entities). This decision can be based off many reasons, some being:
An OrderLine has no need to be retrieved or referenced independently from its Order
An Order should be responsible for changes to its OrderLine 'collection'
An Order and its OrderLines should be transactionally consistent
When fetching Order from a repository, this aggregate will be formed in a single unit of work. All OrderLines will be fetching with their Order and returned. When saving or updating, the aggregate is also persisted in a single unit of work. This ensures that all of the entities (and their relationships) remain consistent and that no 'business rules' are violated.
In your case, a Person and their Accounts should most likely not belong in a single aggregate. I assume that you would need to access a person's accounts without needing to retrieve the person itself (perhaps using identity). I assume that you will want to reference to a specific account from outside of the aggregate (only aggregate roots can be referenced from outside of an aggregate). I also assume that an Account can change independently of Person. Perhaps another reason why you wouldn't want it to belong in the Person aggregate is due to performance reasons (yes, sometimes we have to be pragmatic and not purist!). All of the above entirely depends on your requirements.
Personally I believe in separating your data entities (generally a direct mapping from your database using Entity Framework or some other persistence tool) and your domain entities/value objects. This lets you design your domain in complete isolation of database-related structures, frameworks and constraints.

EntityFramework: To Slice or Not To Slice?

So I'm just getting started with Entity Framework. I'm working with a very large, existing database. I find myself wanting to use EF to create models that are "slices" of the whole database. These slices corresponde to 1 aspect of the application. Is that the right way to look at it, or should I try to model the whole database in 1 EDMX?
Let me give you a fictional example:
Suppose that 1 of the many things that this database contains is customer billing information. I feel like I want to create an EF model that just focuses on the tables that the Customer Billing module needs to interact with. (so then that model would NOT be used for other modules in the app, rather, those same tables might appear in other small EF models). This would allow me to leverage EF's conceptual model features (inheritance, etc) to build a view that is correct for Customer Billing, without worrying about that model's effects, on say Customer Support (even though the 2 modules share some tables)
Does that sound right?

It sounds right to me. The point of an Entity Model, after all, is to provide a set of persistence-capable business objects at a level of abstraction that's appropriate to the required business logic.
You should absolutely create entity models that support modules of the application, not models that copy the underlying database schema. As the link above describes, separating logic from persistence is one of the primary purposes of EF.

I would prefer to use a slice approach, based of following reasons:
If you have a massive database with loads of tables, then it would be difficult to manage massive Entity Model.
It is easier to maintain application / domain specific entities, as entity framework is not a table to entity mapping, you can create custom entities and also combine and split tables across entities.

Application Design - Database Tables and Interfaces

I have a database with tables for each entity in the system. e.g. PersonTable has columns PersonId, Name, HomeStateId. There is also a table for 'reference data' (i.e. states, countries, all currencies, etc.) data that will be used to fill drop down list boxes. This reference table will also be used so that PersonTable's HomeStateId will be a foreign key to the reference table.
In the C# application we have interfaces and classes defined for the entity.
e.g. PersonImplementationClass : IPersonInterface. The reason for having the interfaces for each entity is because the actual entity class will store data differently depending on a 3rd party product that may change.
The question is, should the interface have properties for Name, HomeStateId, and HomeStateName (which will be retrieved from the reference table). OR should the interface not expose the structure of the database, i.e. NOT have HomeStateId, and just have Name, HomeStateName?

I'd say you're on the right track when thinking about property names!
Model your classes as you would in the real world.
Forget the database patterns and naming conventions of StateID and foreign keys in general. A person has a city, not a cityID.
It'll be up to your data layer to map and populate the properties of those objects at run time. You should have the freedom to express your intent and the representation of 'real world' objects in your code, and not be stuck to your DB implementation.

Either way is acceptable, but they both have their pros and cons.
The first way (entities have IDs) is analagous to the ActiveRecord pattern, where your entities are thin wrappers over the database structure. This is often a flexible and fast way of structuring your data layer, because your entities have freedom to work directly with the database to accomplish domain operations. The drawback is that when the data model changes, your app is likely to need maintenance.
The second way (entities reflect more of a real-world structure) is more analagous to a heavier ORM like Entity Framework or Hibernate. In this type of data access layer, your entity management framework would take care of automatically mapping the entities back and forth into the database. This more cleanly separates the application from the data, but can be a lot more plumbing to deal with.
This is a big choice, and shouldn't be taken lightly. It really depends on your project requirements and size, who will be consuming it.

It may help to separate the design a little bit.
For each entity, use two classes:
One that deals with database operations on the entity (where you would put IDs)
One that is a simple data object (where you would have standard fields that actually mean something)
As #womp mentioned, if your entity persistence is only going to be to databases, strongly consider the use of an ORM so you don't end up rolling your own.