Develop Reference

Entity Framework Core Linq query returning ids which do not exist in database

I wonder if there is an easy way using Linq to SQL with Entity Framework Core to query check if a given list of ids exist in the database and which returns the list of ids that do not exist.
The use case I come across this is if the user can do something with a list of object (represented through the list of their ids) I want to check if these ids exist or not.
Of course I could query all objects/object ids that exist in the database and cross check in a second step.
Just wondering if it would be possible in one step.
What I mean in code:
public class MyDbObject
{
public int Id { get; set; }
public string Name { get; set; }
}
public IActionResult DoSomethingWithObjects([FromQuery]List<int> ids}
{
List<int> idsThatDoNotExistInTheDb = DbContext.MyDbObject.Where(???)
return NotFound("those ids do not exist: " + string.Join(", ", idsThatDoNotExist));
}

You can obtain the list of IDs that match, then remove them from the original list, like this:
var validIds = DbContext
.MyDbObject
.Where(obj => ids.Contains(obj.Id))
.Select(obj => obj.Id);
var idsThatDoNotExistInTheDb = ids.Except(validIds);
This approach may be slow, though, so you may be better off doing it in a stored procedure that takes a table-valued parameter (how?)
Note: Pre-checks of this kind are not bullet-proof, because a change may happen between the moment when you validate IDs and the moment when you start the operation. It is better to structure your APIs in a way that it validates and then does whatever it needs to do right away. If validation fails, the API returns a list of errors.

Dapper multilevel nesting

I am trying to use Dapper in my project to speed up data loading (currently using EF6)
Here is my SQL
String SQL = #"select vwArtikli_Grid_V2.ArtikalID
,vwArtikli_Grid_V2.ArtikalNaziv
,Artikli_TagLista.ArtikalTagListaID
,Artikli_TagLista.ArtikalTagID
,Artikli_Stanje.ArtikalStanjeID
,Artikli_Stanje.ObjekatID
,Artikli_Stanje.Stanje
,Artikli_Tagovi.GrupaID
,Artikli_Tagovi.ArtikalTagGrupaID
,Artikli_Tagovi.ArtikalTagNaziv
,Artikli_Tagovi.ArtikalTagPrint
,Artikli_Tagovi.ArtikalTagSlika
,Artikli_Tagovi.ArtikalTagID
,vwArtikli_Grid_V2.ArtikalID
from Artikli_Tagovi
inner join Artikli_TagLista on Artikli_Tagovi.ArtikalTagID = Artikli_TagLista.ArtikalTagID
right outer join vwArtikli_Grid_V2 on Artikli_TagLista.ArtikalID = vwArtikli_Grid_V2.ArtikalID
left outer join Artikli_Stanje on vwArtikli_Grid_V2.ArtikalID = Artikli_Stanje.ArtikalID;
I am using my Entity Framework Entities as POCOs and they are
VwArtikliGridV2, Artikli_TagLista, Artikli_Tagovi, Artikli_Stanje
VwArtikliGridV2 has two properties
public virtual ICollection<Artikli_TagLista> Artikli_TagLista { get; set; }
public virtual ICollection<Artikli_Stanje> Artikli_Stanje { get; set; }
and Artikli_TagLista has
public virtual Artikli_Tagovi Artikli_Tagovi { get; set; }
What is the easiest way to execute the query and map my data to the POCOs or Entities ?
I tried
Dapper.Mapper
var Artikli = cn.Query<VwArtikliGridV2, Artikli_TagLista, Artikli_Stanje, Artikli_Tagovi> (SQL);
but it didnot work
I also tried Slapper.AutoMapper
List<dynamic> ArtikliUM = cn.Query<dynamic>(SQL).ToList();
Slapper.AutoMapper.Configuration.AddIdentifiers(typeof(Artikli_Tagovi), new List<string> { "ArtikalTagID" });
Slapper.AutoMapper.Configuration.AddIdentifiers(typeof(Artikli_TagLista), new List<string> { "ArtikalTagListaID" });
Slapper.AutoMapper.Configuration.AddIdentifiers(typeof(Artikli_Stanje), new List<string> { "ArtikalStanjeID" });
Artikli = (Slapper.AutoMapper.MapDynamic<VwArtikliGridV2>(ArtikliUM) as IEnumerable<VwArtikliGridV2>).ToList();
But it also did not work.
I can map VwArtikliGridV2 but i cant map any of the nested objects. They are always null.
What can I try ?

Dapper maps things flat. At first this feels like a big pain because EF nests things so easily. But once you get over the initial pain you realize how simple, predictable, and performance-oriented it is. Maybe an extra 15 min. of query-crafting for untouchable performance.
I answered a similar question here return a list of data via stored proc to dapper.
You should be able to return multiple datasets (one for the parent item, one for the underlying tags, etc.) and combine them in your app tier.

As BlackjacketMack said in his answer, I think multiple data sets are the way to go.
You might want to look in the Dapper documentation at the features Multi Mapping to split single rows into multiple objects, and QueryMultiple to read multiple result sets from a single query.
Obviously, it would mean modifying your query to return multiple result sets, but could achieve what you are looking for.
Examples can be found here: https://github.com/StackExchange/Dapper/blob/master/Dapper.Tests/MultiMapTests.cs
https://github.com/StackExchange/Dapper/blob/master/Dapper.Tests/QueryMultipleTests.cs
In particular, you might want to take a look at method public void TestMultiMapThreeTypesWithGridReader() in MultiMapTests.cs. I found that very useful to help me understand a similar problem of parent with child collections, where the child contains different object types returned in a single row.

SQL query shows good database values, but LINQ to Entity framework brings a null value from nowhere

I'm having a really strange problem here, and i dont have any clue why.
I'm supposed to make small localdb console app in C#. The goal is to enter persons (teachers, actually) in the DB, with a certain amount of information.
I have a few classes, but 2 of them are important here: Certification and Notation.
Certifications are, well, certifications of the professors.
The code for these classes is this:
class Certification
{
public int CertificationID { get; set; }
public virtual Teacher Teacher { get; set; }
public virtual Course Course { get; set; }
public string CertificationName { get; set; }
public virtual Notation Notation { get; set; }
}
class Notation
{
public int NotationID {get;set;}
public string Note {get;set;}
}
Nothing too dangerous. Through migrations i made my database, and they look like they should:
Certification:
CertificationID (PK)
CertificationName
Course_CourseID (FK to another class, course)
Notation_NotationID (FK to notations)
Teacher_TeacherID (FK to the teachers)
Notations:
NotationID (PK)
Note
My program allows me to add teachers, with all the informations i need, and for example, their certifications. Here, i made some dummy teacher, with a dummy certification.
If i call SELECT * FROM Certification , i get exactly what i should get, a single line like this:
CertificationID = 6
CertificationName = placeholder
Course_CourseID = 13
Notation_NotationID = 12
Teacher_TeacherID = 5
Everything is correct in this. CourseID links to an actual course in the database, NotationID in an actual note, and Teacher to an actual teacher too. Everything is fine!
Now, i just want to show the certifications of our teacher:
var certifs = from c in db.Certifications where c.Teacher.TeacherID == item.TeacherID select c;
foreach(var v in certifs )
{
var course = (from c in db.Courses where c.CourseID == v.Course.CourseID select c).First();
var note = (from n in db.Notations where n.NotationID == v.Notation.NotationID select n.NotationID).First();
Console.WriteLine("Name: " + v.CertificationName + ", related to the " + course.CourseName + " course, with a note of " + note);
Console.WriteLine("");
}
And it doesn't work. When my foreach loop starts, my first item in the loop doesn't have any reference to a notation. Everything else is fine: the foreign keys for the course and the teachers are here, and valid, but for the notation, i only get a null value. So my certification item looks more like:
CertificationID = 6
CertificationName = placeholder
Course_CourseID = 13
Notation_NotationID = null
Teacher_TeacherID = 5
Basically, if i do a SQL Query, my row in the database is perfectly fine, but calling it through the entity framework (and LINQ) returns a null value for the notation. (which throws an exception when calling var note etc....
Does anybody have an idea about this? I'm really stuck on this.
I'm sorry if my English isn't good enough. If you guys need more information, just ask.
Anwsered by JC:
Lazy loading isnt working properly. Eager loading solves the problem.

The problem is you aren't populating your navigation properties when you retrieve the Certification entities. Then you try to access them and they're null.
You either need to make sure lazy loading is turned on:
Configuration.LazyLoadingEnabled = true; //In your DbContext's constructor
in which case just accessing the Course and Notification references should cause them to be populated in separate database transactions...
...or you need to employ eager loading when querying against the DbSet:
var certifs = from c in db.Certifications.Include(c=>c.Course).Include(c=>c.Notation) where ...
Which will cause Course and Notation to be loaded at the same time Certifications is loaded all in one database transaction.

In your line
var note = (from n in db.Notations
where n.NotationID == v.Notation.NotationID
select n.NotationID).First();
you are selecting n.NotationID only which would return an integer only. Trying changing the select to select n

Data is Null. This method or property cannot be called on Null values

I'm working on an application where one can get information on movies from a database as well as add, update and delete the movies. In the database I have three tables (Movie, Genre and MovieGenre <- stores the movies and their genre/s). Everything works fine besides one thing, and that's when a movie hasn't got any genres (which should be possible).
The problem occur in the method below, and the following exception is thrown:
Data is Null. This method or property cannot be called on Null values.
The reason (of course) is that the sproc returns null because the movie hasn't got any genres, but I just can't figure out how to prevent this exception being thrown. As I said, it should be possible to store a movie without storing any information of genre/s.
The method:
public List<MovieGenre> GetMovieGenrebyMovieID(int movieID) {
using (SqlConnection conn = CreateConnection()) {
try {
SqlCommand cmd = new SqlCommand("dbo.usp_GetMovieGenreByMovieID", conn);
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.AddWithValue("#MovieID", movieID);
List<MovieGenre> movieGenre = new List<MovieGenre>(10);
conn.Open();
using (SqlDataReader reader = cmd.ExecuteReader()) {
int movieGenreIDIndex = reader.GetOrdinal("MovieGenreID");
int movieIDIndex = reader.GetOrdinal("MovieID");
int genreIDIndex = reader.GetOrdinal("GenreID");
while (reader.Read()) {
movieGenre.Add(new MovieGenre {
MovieID = reader.GetInt32(movieIDIndex),
MovieGenreID = reader.GetInt32(movieGenreIDIndex),
GenreID = reader.GetInt32(genreIDIndex)
});
}
}
movieGenre.TrimExcess();
return movieGenre;
}
catch {
throw new ApplicationException();
}
}
}
The sproc:
ALTER PROCEDURE usp_GetMovieGenreByMovieID
#MovieID int
AS
BEGIN
BEGIN TRY
SELECT m.MovieID, g.GenreID, mg.MovieGenreID, g.Genre
FROM Movie AS m
LEFT JOIN MovieGenre AS mg
ON m.MovieId = mg.MovieID
LEFT JOIN Genre AS g
ON mg.GenreID = g.GenreID
WHERE m.MovieID = #MovieID
END TRY
BEGIN CATCH
RAISERROR ('Error while trying to receive genre(s).',16,1)
END CATCH
END

You shouldn't be trying to convert the null values from the proc into ints - so before you create the MovieGenre instance you need to check the nullable fields using the SqlDataReader.IsDBNull method:
http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqldatareader.isdbnull.aspx
Assuming that the GenreID and MovieGenreID are nullable ints you could do something like:
movieGenre.Add(new MovieGenre {
MovieID = reader.GetInt32(movieIDIndex),
MovieGenreID = reader.IsDBNull(movieGenreIDIndex) ? null : reader.GetInt32(movieGenreIDIndex),
GenreID = reader.IsDBNull(genreIDIndex) ? null : reader.GetInt32(genreIDIndex)
});

This error happens immediately after I enabled C# 8 nullable feature in my Entity Framework Core 3.1 project.
The solution is to change your entity properties to their nullable counterparts. For example,
Change from:
public class Person {
public int Id { get; set; }
public string Name { get;set; }
public string Address { get;set; }
}
To:
public class Person {
public int Id { get; set; }
public string Name { get;set; }
public string? Address { get;set; } //change address to nullable string since it is nullable in database
}

In my case I was using EF Core and the issue was that the field was nullable in the database but in the ModelCreating it was required like that:
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<MyEntity>(entity =>
{
entity.Property(e => e.Details)
.IsRequired()
.HasMaxLength(250);
}
}
I remove the IsRequired() and it worked fine.
Update few days after,
Got same issue, a string field It was not allowing null in the DB.

Edit your select statement as follows to handle null issue.
SELECT ISNULL(m.MovieID,0) AS MovieID,
ISNULL(g.GenreID,0) AS GenreID,
ISNULL(mg.MovieGenreID,0) AS MovieGenreID,
ISNULL(g.Genre,'') AS Genre
FROM --rest of your query...

I realize this is old, but I just had this problem for EF Core in .net 6.
Even though strings are nullable, and Entity Framework even created my objects from the existing table with the string type, I had to change the type of my string columns to the string? type in order to pull back data where the data was null.
wrong:
public string Decision { get; set; }
correct:
public string? Decision { get; set; }
I thought I was having an issue with dependency injection. Turns out it was Entity Framework Core and it was a simple fix.

The simplest answer is to replace the nulls with non-null values. Try:
ALTER PROCEDURE usp_GetMovieGenreByMovieID
#MovieID int
AS
BEGIN
BEGIN TRY
SELECT m.MovieID,
coalesce(g.GenreID,0) GenreID,
coalesce(mg.MovieGenreID,0) MovieGenreID,
coalesce(g.Genre, 'Not Applicable') Genre
FROM Movie AS m
LEFT JOIN MovieGenre AS mg
ON m.MovieId = mg.MovieID
LEFT JOIN Genre AS g
ON mg.GenreID = g.GenreID
WHERE m.MovieID = #MovieID
END TRY
BEGIN CATCH
RAISERROR ('Error while trying to receive genre(s).',16,1)
END CATCH
END

If somebody have faced this issue, here is my case.
I am building WEB Api. Before I put in place [Required] Attribue - https://learn.microsoft.com/en-us/dotnet/api/system.componentmodel.dataannotations.requiredattribute?view=net-5.0, I had some NULL values in my database. Then I added this attribute to my model and was trying to make a GET request, but "System.InvalidOperationException: The data is NULL at ordinal 1. This method can't be called on NULL values. Check using IsDBNull before calling." appeared.
So I deleted NULL values from databases and every request worked fine after that. As far as I understood, an error occurs because of EF Core doesn't allow NULL values in database while [Required] attribute applied.
I hope it will be helpful for someone.

I had this problem in .net5 data first project , because a field in data base was nullable but in c# entity class was required. after recreate entities by ef core power tools extension , the problem resolved.

Today I've faced this issue. But mine has been fixed in another way. If someday anyone stumbled the answer is for them.
As this is a generic C# .NET CLI exception
I've added a new foreign key to one of my DB table with no default value. Thus the value was set to NULLas that column was set to allow null.
And I've been getting this exception while querying on that table.
As solution, I replaced the NULL values with appropriate values (as they are foreign key' they should be appropriate).
That's all.
Thank you.

For me this happened because in the database I had a column 'XYZ' which had a NULL value, but the model's property that mapped to it (bool) wasn't nullable.

I had this problem too, and for me a property in my class was decorated as [Required] but the table had null values in that column. Makes sense. Once I removed Required, the data loaded successfully with null values.

BookingQuantity - column having null value in DB. but actual DB BookingQuantity not null column. Some rare case it happens to enter. In that case below code throw error the same error(Data is Null. This method or property cannot be called on Null values
).
totalBookingQuantity += item.InspPoTransactions.Where(x => x.PoId == inspectionPODetail.PoId && x.ProductId == inspectionPODetail.ProductId).Sum(x => Convert.ToInt32(x.BookingQuantity));

This error might occur due to lack of permissions of the user on the database. Check if the user has at least the EXECUTE, SELECT or SHOW DATABASES permissions

pulling a non-trivial object from a MySQL database

I have a C# .NET 3.5 project using a MySQL database.
I have an object Task which I would like to be able to create by pulling it from a series of database tables.
public class Task
{
public Task()
{
Values = new List<string>();
OtherValues = new List<string>();
Requirement = string.Empty;
Minimum = 1;
Children = new List<Foo>();
}
public IList<string> Values { get; set; }
public IList<string> OtherValues { get; set; }
public string Requirement { get; set; }
public int Minimum { get; set; }
public int Maximum { get; set; }
public IList<Foo> Children { get; set; }
}
I'd like to be able to get the tasks from a TaskList which would lazily read elements of the task as they were accessed by an enumerator.
public class TaskList : IEnumerable<Task>
{
/* ... */
public IEnumerator<Task> GetEnumerator()
{
string query = #"SELECT my_task.*, `Order` FROM my_task ORDER BY `Order` DESC";
using (MySqlConnection connection = new MySqlConnection(connection_string_))
using (MySqlCommand command = connection.CreateCommand())
{
command.CommandText = query;
connection.Open();
using (MySqlDataReader reader = command.ExecuteReader())
{
yeild /* ??? */
}
}
}
}
How is this done?

You can serialize it to XML and store it as a string. Add the following function to Task:
public XElement Serialize()
{
return new XElement("Task",
new XElement("Values",from val in Values select new XElement("Item",val)),
new XElement("OtherValues",from val in OtherValues select new XElement("Item",val)),
new XElement("Requirement",Requirement),
new XElement("Minimum",Minimum),
new XElement("Maximum",Maximum)
);
}
You will need to put using System.Linq; and using System.Xml.Linq; in the top of the .cs file.
I didn't write the code to serialize Children because I don't know what the data type Foo looks like, but you should serialize it in a similar manner. After you've done that, you can easily write the XML to the database, and read it back(write a constructor that parses the Xml into a Task object)
EDIT(addition):
Here is an example to a constructors that receives XML(or parse a string as XML):
public Task(string xmlSourceAsString):
this(XElement.Parse(xmlSourceAsString))
{
}
public Task(XElement xmlSource)
{
Values=(from itm in xmlSource.Element("Values").Elements("Item") select itm.Value).ToList();
OtherValues=(from itm in xmlSource.Element("OtherValues").Elements("Item") select itm.Value).ToList();
Requirement=xmlSource.Element("Requirement").Value;
Minimum=int.Parse(xmlSource.Element("Minimum").Value);
Maximum=int.Parse(xmlSource.Element("Maximum").Value);
}
EDIT(explanation):
You can't store your object as is in the database "as is", because it refers to other objects. For example - the list Values doesn't sit in the same place in memory as the rest of the object, befause it's a ref type - it refers to another object that sits in a different place in the memory. In matter of fact, the only parts of your object that are stored in the same place as the main object are the Minimum and Maximum, which are ref types, so if you could somehow store the object as is(laziest solution possible, if it worked), you would get your Minimum and Maximum fields right, but all other fields will point to the memory addresses where those objects where placed when you stored the Task object, which are now most likely invalid pointers(and I say "most likely" because it is also possible(though rare) that they will point to legitimate objects, maybe event of the same type - but they still won't have your data.
If you want the object with all it's data stored in a database(or in a file. or passed to a proccess that runs on another computer via network) you have to serialize it. Performance-wise the best way is to serialize it to binary(C# have some tools for that, but it's still more complex than XML).
Xml also have the adventage of being easily readable from most modern programming languages and database engines. MySQL has some functions to read and write XML, so you can update the object in the database and access it's fields from MySQL queries.
Conclusion
You asked for a solution that is easy(lazy), efficient, and sql-compatible(access to the object's fields from MySQL queries). I say you can only have two of your three requirements, but you can choose which two:
If you want something easy and efficient, even at the price of loosing compatibility, serialize your objects to binary. True, it's not as easy as XML, but .NET has some tools to help you with that.
If you want something efficient and compatible, and willing to do some work for that, you can put your object in MySQL the way databases are meant to be used - use separate tables for the lists that refers to the objects via OIDs, etc. This will require some work, but after you add the tables and code the MySQL functions and the C# functions that handle everything, you should be able to store, retrieve, and access your objects with ease.
If you want something easy and compatible, and you can afford loosing some efficiency, use my solution and serialize your objects to XML. This is the laziest solution - unless someone knows a library that can automatically serialize any object, LINQ to XML is the easiest way to do it, and requires much less code than any other solution.