I wonder if there is an easy way using Linq to SQL with Entity Framework Core to query check if a given list of ids exist in the database and which returns the list of ids that do not exist.
The use case I come across this is if the user can do something with a list of object (represented through the list of their ids) I want to check if these ids exist or not.
Of course I could query all objects/object ids that exist in the database and cross check in a second step.
Just wondering if it would be possible in one step.
What I mean in code:
public class MyDbObject
{
public int Id { get; set; }
public string Name { get; set; }
}
public IActionResult DoSomethingWithObjects([FromQuery]List<int> ids}
{
List<int> idsThatDoNotExistInTheDb = DbContext.MyDbObject.Where(???)
return NotFound("those ids do not exist: " + string.Join(", ", idsThatDoNotExist));
}
You can obtain the list of IDs that match, then remove them from the original list, like this:
var validIds = DbContext
.MyDbObject
.Where(obj => ids.Contains(obj.Id))
.Select(obj => obj.Id);
var idsThatDoNotExistInTheDb = ids.Except(validIds);
This approach may be slow, though, so you may be better off doing it in a stored procedure that takes a table-valued parameter (how?)
Note: Pre-checks of this kind are not bullet-proof, because a change may happen between the moment when you validate IDs and the moment when you start the operation. It is better to structure your APIs in a way that it validates and then does whatever it needs to do right away. If validation fails, the API returns a list of errors.
I am trying to use Dapper in my project to speed up data loading (currently using EF6)
Here is my SQL
String SQL = #"select vwArtikli_Grid_V2.ArtikalID
,vwArtikli_Grid_V2.ArtikalNaziv
,Artikli_TagLista.ArtikalTagListaID
,Artikli_TagLista.ArtikalTagID
,Artikli_Stanje.ArtikalStanjeID
,Artikli_Stanje.ObjekatID
,Artikli_Stanje.Stanje
,Artikli_Tagovi.GrupaID
,Artikli_Tagovi.ArtikalTagGrupaID
,Artikli_Tagovi.ArtikalTagNaziv
,Artikli_Tagovi.ArtikalTagPrint
,Artikli_Tagovi.ArtikalTagSlika
,Artikli_Tagovi.ArtikalTagID
,vwArtikli_Grid_V2.ArtikalID
from Artikli_Tagovi
inner join Artikli_TagLista on Artikli_Tagovi.ArtikalTagID = Artikli_TagLista.ArtikalTagID
right outer join vwArtikli_Grid_V2 on Artikli_TagLista.ArtikalID = vwArtikli_Grid_V2.ArtikalID
left outer join Artikli_Stanje on vwArtikli_Grid_V2.ArtikalID = Artikli_Stanje.ArtikalID;
I am using my Entity Framework Entities as POCOs and they are
VwArtikliGridV2, Artikli_TagLista, Artikli_Tagovi, Artikli_Stanje
VwArtikliGridV2 has two properties
public virtual ICollection<Artikli_TagLista> Artikli_TagLista { get; set; }
public virtual ICollection<Artikli_Stanje> Artikli_Stanje { get; set; }
and Artikli_TagLista has
public virtual Artikli_Tagovi Artikli_Tagovi { get; set; }
What is the easiest way to execute the query and map my data to the POCOs or Entities ?
I tried
Dapper.Mapper
var Artikli = cn.Query<VwArtikliGridV2, Artikli_TagLista, Artikli_Stanje, Artikli_Tagovi> (SQL);
but it didnot work
I also tried Slapper.AutoMapper
List<dynamic> ArtikliUM = cn.Query<dynamic>(SQL).ToList();
Slapper.AutoMapper.Configuration.AddIdentifiers(typeof(Artikli_Tagovi), new List<string> { "ArtikalTagID" });
Slapper.AutoMapper.Configuration.AddIdentifiers(typeof(Artikli_TagLista), new List<string> { "ArtikalTagListaID" });
Slapper.AutoMapper.Configuration.AddIdentifiers(typeof(Artikli_Stanje), new List<string> { "ArtikalStanjeID" });
Artikli = (Slapper.AutoMapper.MapDynamic<VwArtikliGridV2>(ArtikliUM) as IEnumerable<VwArtikliGridV2>).ToList();
But it also did not work.
I can map VwArtikliGridV2 but i cant map any of the nested objects. They are always null.
What can I try ?
Dapper maps things flat. At first this feels like a big pain because EF nests things so easily. But once you get over the initial pain you realize how simple, predictable, and performance-oriented it is. Maybe an extra 15 min. of query-crafting for untouchable performance.
I answered a similar question here return a list of data via stored proc to dapper.
You should be able to return multiple datasets (one for the parent item, one for the underlying tags, etc.) and combine them in your app tier.
As BlackjacketMack said in his answer, I think multiple data sets are the way to go.
You might want to look in the Dapper documentation at the features Multi Mapping to split single rows into multiple objects, and QueryMultiple to read multiple result sets from a single query.
Obviously, it would mean modifying your query to return multiple result sets, but could achieve what you are looking for.
Examples can be found here: https://github.com/StackExchange/Dapper/blob/master/Dapper.Tests/MultiMapTests.cs
https://github.com/StackExchange/Dapper/blob/master/Dapper.Tests/QueryMultipleTests.cs
In particular, you might want to take a look at method public void TestMultiMapThreeTypesWithGridReader() in MultiMapTests.cs. I found that very useful to help me understand a similar problem of parent with child collections, where the child contains different object types returned in a single row.
I have a Entity called "Client", and another Entity called "Card".
A Client may have many Cards.
My Client Entity looks like this:
public class Client{
public virtual ICollection<Card> Cards {get; set;}
}
Now I want to show the Client data in a DataGrid in WPF, and I want to get Cards Count data,so I add a property to Client Entity, which like this:
public class Client{
public virtual ICollection<Card> Cards {get; set;}
public int CardCount
{
return Cards.Count;
}
}
And then I query the data with Linq and Bind to view
var query = from n in db.Clients select n;
When I run the Application, I got a Exception just right on the return Cards.Count; line;
System.ObjectDisposedException
The ObjectContext instance has been disposed and can no longer be used for operations that require a connection.
So how could I correctly get the cards count?
There is a way simpler method than the other answers here show. Please also realize that solutions such as
var client = db.Clients.FirstOrDefault(c=> c.Id = someid); //get a client
if (client != null)
{
cardCount = client.Cards.Count;
}
will cause an issue called Select N+1 problem. Read up on it if interested, but in a nutshell, it means the following:
Because you are not only interested in one exact client, but you want to display N clients, you need to do one (1) query to get just the clients. Then, by doing the FirstOrDefault stuff, you are actually doing one (1) extra db roundtrip to the database per Client record, which results in an additional N * 1 = N roundtrips. What this means that, if you were to just query the Clients without any related data, you could get however many client records you like, in just one query. But by fetching related data to each of them one-by-one, you are doing way too many db roundtrips.
Here is a way to solve this issue, by using joins and projections. You can get all the data you need in a single DB access:
using (var context = GetDbContext())
{
return context.Clients.Select(cli => new YourViewModel
{
Name = cli.FullName,
// Other prop setters go here
CardCount = cli.Cards.Count
}).Skip((page - 1) * pageSize).Take(pageSize).ToList();
}
You might be wondering, what's the difference afterall? Well, here, you are not working with materialized objects, as others call them here, but with a DbContext. By applying the proper LINQ operators to it (note, that this works not just with DbContext, but also with any IQueryable (well obviously not if you call AsQueryable() on an already in-memory collection but whatever)), LINQ to Entities can construct a proper SQL to join the tables and project the results and therefore you fetch all required data in one go. Note that LINQ to Entities IS ABLE to translate the cli.Cards.Count into a proper SQL Count statement.
You can get the count without loading the entities like this:
using (var context = new MyContext())
{
var client = context.Client.Find(clientId);
// Count how many cards the client has
var cardsCount = context.Entry(client)
.Collection(b => b.Cards)
.Query()
.Count();
}
More information on MSDN page.
You get an ObjectDisposedException if you do not materialize the retreived query. In the following case, the query gets executed only when you Access the first time the list from GetNonMaterialized and not before leaving the method. Fact of this the db is disposed because of lost of scope.
public IEnumerable<Client> GetNonMaterialized()
{
return from n in db.Clients select n;
}
In the following case the query is executed before leaving the method.
public IEnumerable<Client> GetMaterialized()
{
return (from n in db.Clients select n).ToList();
}
Always be sure that the query is executed before exiting the scope of a ObjectContext.
If you want to know whether the query is executed and when enalbe Logging of EF.
How can I turn off Entity Framework 6.1 logging?
I'm using LINQ to SQL and have a database table called Product with 20 columns. The Product table is mapped to the Product class in the LINQ to SQL metadata.
I'd like to use my dbContext and retrieve some product records but only populating 10 columns not all 20 columns.
How would that be possible to specify which columns should be returned/populated with LINQ to SQL (or EF)?
I know one way would be using stored procedures but that's this question is about.
Thanks,
You usually use an anonymous class for that:
db.Products.Where(... filter ...).Select(item => new
{
Field1 = item.Field1,
Field2 = item.Field2,
});
Only the fields you include will be selected. If you intend to pass this data structure to other functions or return it, you need a concrete class definition for sub field set, such as:
class SmallerEntity
{
public something Field1;
public something Field2;
}
And you can initialize this in your Select statement:
db.Products.Where(... filter ...).Select(item => new SmallerEntity
{
Field1 = item.Field1,
Field2 = item.Field2,
});
I don't recommend the practice of half-populating an existing class. That makes your state space unnecessarily complex and allows more bugs in your code. Try to contain smaller subsets of data in their own classes.
I'm using entity framework to connect with the database. I've one little problem:
I've one table which have one varbinary(MAX) column(with filestream).
I'm using SQL request to manage the "Data" part, but EF for the rest(metadata of the file).
I've one code which has to get all files id, filename, guid, modification date, ... of a file. This doesn't need at all the "Data" field.
Is there a way to retrieve a List but without this column filled?
Something like
context.Files.Where(f=>f.xyz).Exclude(f=>f.Data).ToList();
??
I know I can create anonymous objects, but I need to transmit the result to a method, so no anonymous methods. And I don't want to put this in a list of anonymous type, and then create a list of my non-anonymous type(File).
The goal is to avoid this:
using(RsSolutionsEntities context = new RsSolutionsEntities())
{
var file = context.Files
.Where(f => f.Id == idFile)
.Select(f => new {
f.Id, f.MimeType, f.Size, f.FileName, f.DataType,
f.DateModification, f.FileId
}).FirstOrDefault();
return new File() {
DataType = file.DataType, DateModification = file.DateModification,
FileId = file.FileId, FileName = file.FileName, Id = file.Id,
MimeType = file.MimeType, Size = file.Size
};
}
(I'm using here the anonymous type because otherwise you will get a NotSupportedException: The entity or complex type 'ProjectName.File' cannot be constructed in a LINQ to Entities query.)
(e.g. this code throw the previous exception:
File file2 = context.Files.Where(f => f.Id == idFile)
.Select(f => new File() {Id = f.Id, DataType = f.DataType}).FirstOrDefault();
and "File" is the type I get with a context.Files.ToList(). This is the good class:
using File = MyProjectNamespace.Common.Data.DataModel.File;
File is a known class of my EF datacontext:
public ObjectSet<File> Files
{
get { return _files ?? (_files = CreateObjectSet<File>("Files")); }
}
private ObjectSet<File> _files;
Is there a way to retrieve a List but without this column filled?
Not without projection which you want to avoid. If the column is mapped it is natural part of your entity. Entity without this column is not complete - it is different data set = projection.
I'm using here the anonymous type because otherwise you will get a
NotSupportedException: The entity or complex type 'ProjectName.File'
cannot be constructed in a LINQ to Entities query.
As exception says you cannot project to mapped entity. I mentioned reason above - projection make different data set and EF don't like "partial entities".
Error 16 Error 3023: Problem in mapping fragments starting at line
2717:Column Files.Data in table Files must be mapped: It has no
default value and is not nullable.
It is not enough to delete property from designer. You must open EDMX as XML and delete column from SSDL as well which will make your model very fragile (each update from database will put your column back). If you don't want to map the column you should use database view without the column and map the view instead of the table but you will not be able to insert data.
As a workaround to all your problems use table splitting and separate the problematic binary column to another entity with 1 : 1 relation to your main File entity.
I'd do something like this:
var result = from thing in dbContext.Things
select new Thing {
PropertyA = thing.PropertyA,
Another = thing.Another
// and so on, skipping the VarBinary(MAX) property
};
Where Thing is your entity that EF knows how to materialize. The resulting SQL statement shouldn't include the large column in its result set, since it's not needed in the query.
EDIT: From your edits, you get the error NotSupportedException: The entity or complex type 'ProjectName.File' cannot be constructed in a LINQ to Entities query. because you haven't mapped that class as an entity. You can't include objects in LINQ to Entities queries that EF doesn't know about and expect it to generate appropriate SQL statements.
You can map another type that excludes the VarBinary(MAX) column in its definition or use the code above.
you can do this:
var files = dbContext.Database.SqlQuery<File>("select FileId, DataType, MimeType from Files");
or this:
var files = objectContext.ExecuteStoreQuery<File>("select FileId, DataType, MimeType from Files");
depending on your version of EF
I had this requirement because I have a Document entity which has a Content field with the content of the file, i.e. some 100MB in size, and I have a Search function that I wanted to return the rest of the columns.
I chose to use projection:
IQueryable<Document> results = dbContext.Documents.Include(o => o.UploadedBy).Select(o => new {
Content = (string)null,
ContentType = o.ContentType,
DocumentTypeId = o.DocumentTypeId,
FileName = o.FileName,
Id = o.Id,
// etc. even with related entities here like:
UploadedBy = o.UploadedBy
});
Then my WebApi controller passes this results object to a common Pagination function, which applies a .Skip, .Take and a .ToList.
This means that when the query gets executed, it doesn't access the Content column, so the 100MB data is not being touched, and the query is as fast as you'd want/expect it to be.
Next, I cast it back to my DTO class, which in this case is pretty much exactly the same as the entity class, so this might not be a step you need to implement, but it's follows my typical WebApi coding pattern, so:
var dtos = paginated.Select(o => new DocumentDTO
{
Content = o.Content,
ContentType = o.ContentType,
DocumentTypeId = o.DocumentTypeId,
FileName = o.FileName,
Id = o.Id,
UploadedBy = o.UploadedBy == null ? null : ModelFactory.Create(o.UploadedBy)
});
Then I return the DTO list:
return Ok(dtos);
So it uses projection, which might not fit the original poster's requirements, but if you're using DTO classes, you're converting anyway. You could just as easily do the following to return them as your actual entities:
var dtos = paginated.Select(o => new Document
{
Content = o.Content,
ContentType = o.ContentType,
DocumentTypeId = o.DocumentTypeId,
//...
Just a few extra steps but this is working nicely for me.
For EF Core 2
I implemented a solution like this:
var files = context.Files.AsNoTracking()
.IgnoreProperty(f => f.Report)
.ToList();
The base idea is to turn for example this query:
SELECT [f].[Id], [f].[Report], [f].[CreationDate]
FROM [File] AS [f]
into this:
SELECT [f].[Id], '' as [Report], [f].[CreationDate]
FROM [File] AS [f]
you can see the full source code in here:
https://github.com/aspnet/EntityFrameworkCore/issues/1387#issuecomment-495630292
I'd like to share my attempts to workaround this problem in case somebody else is in the same situation.
I started with what Jeremy Danyow suggested, which to me is the less painful option.
// You need to include all fields in the query, just make null the ones you don't want.
var results = context.Database.SqlQuery<myEntity>("SELECT Field1, Field2, Field3, HugeField4 = NULL, Field5 FROM TableName");
In my case, I needed a IQueryable<> result object so I added AsQueryable() at the end. This of course let me add calls to .Where, .Take, and the other commands we all know, and they worked fine. But there's a caveat:
The normal code (basically context.myEntity.AsQueryable()) returned a System.Data.Entity.DbSet<Data.DataModel.myEntity>, while this approach returned System.Linq.EnumerableQuery<Data.DataModel.myEntity>.
Apparently this means that my custom query gets executed "as is" as soon as needed and the filtering I added later is done afterwards and not in the database.
Therefore I tried to mimic Entity Framework's object by using the exact query EF creates, even with those [Extent1] aliases, but it didn't work. When analyzing the resulting object, its query ended like
FROM [dbo].[TableName] AS [Extent1].Where(c => ...
instead of the expected
FROM [dbo].[TableName] AS [Extent1] WHERE ([Extent1]...
Anyway, this works, and as long as the table is not huge, this method will be fast enough. Otherwise you have no option than to manually add the conditions by concatenating strings, like classic dynamic SQL. A very basic example in case you don't know what I'm talking about:
string query = "SELECT Field1, Field2, Field3, HugeField4 = NULL, Field5 FROM TableName";
if (parameterId.HasValue)
query += " WHERE Field1 = " + parameterId.Value.ToString();
var results = context.Database.SqlQuery<myEntity>(query);
In case your method sometimes needs this field, you can add a bool parameter and then do something like this:
IQueryable<myEntity> results;
if (excludeBigData)
results = context.Database.SqlQuery<myEntity>("SELECT Field1, Field2, Field3, HugeField4 = NULL, Field5 FROM TableName").AsQueryable();
else
results = context.myEntity.AsQueryable();
If anyone manages to make the Linq extensions work properly like if it was the original EF object, please comment so I can update the answer.
I'm using here the anonymous type because otherwise you will get a
NotSupportedException: The entity or complex type 'ProjectName.File'
cannot be constructed in a LINQ to Entities query.
var file = context.Files
.Where(f => f.Id == idFile)
.FirstOrDefault() // You need to exeucte the query if you want to reuse the type
.Select(f => new {
f.Id, f.MimeType, f.Size, f.FileName, f.DataType,
f.DateModification, f.FileId
}).FirstOrDefault();
And also its not a bad practice to de-normalize the table into further, i.e one with metadata and one with payload to avoid projection. Projection would work, the only issue is, need to edit any time a new column is added to the table.
I tried this:
From the edmx diagram (EF 6), I clicked the column I wanted to hide from EF and on their properties you can set their getter and setter to private. That way, for me it works.
I return some data which includes a User reference, so I wanted to hide the Password field even though it's encrypted and salted, I just didn't want it on my json, and I didn't want to do a:
Select(col => new {})
because that's a pain to create and maintain, especially for big tables with a lot of relationships.
The downside with this method is that if you ever regenerate your model, you would need to modify their getter and setter again.
Using Entity Framework Power Tools you can do the following in efpt.config.json:
"Tables": [
{
"ExcludedColumns": [
"FileData"
],
"Name": "[dbo].[Attachment]",
"ObjectType": 0
}
]