Develop Reference

Entity is incorrectly detaching from tracking EF Core

I am trying to query objects from a database, loop through them and check if a column has a value and, if it does not, create a value and assign it to that column and save it to the database. The problem I'm having is that the entity is detaching after the query so I cannot save the changes. Below is the code I am using to query and update the entity.
DateTime runTime = passedDateTime ?? DateTime.Now;
await using DiscordDatabaseContext database = new();
DateTime startOfWeek = exactlyOneWeek ? runTime.OneWeekAgo() : runTime.StartOfWeek(StartOfWeek);
//Add if not in a Weekly Playlist already and if the video was submitted after the start of the week
List<PlaylistData> pld = await database.PlaylistsAdded.Select(playlist => new PlaylistData
{
PlaylistId = playlist.PlaylistId,
WeeklyPlaylistID = playlist.WeeklyPlaylistID,
Videos = playlist.Videos.Where(
video => (video.WeeklyPlaylistItemId == null ||
video.WeeklyPlaylistItemId.Length == 0) &&
startOfWeek <= video.TimeSubmitted)
.Select(video => new VideoData
{
WeeklyPlaylistItemId = video.WeeklyPlaylistItemId,
VideoId = video.VideoId
}).ToList()
}).ToListAsync().ConfigureAwait(false);
int count = 0;
int nRows = 0;
foreach (PlaylistData playlistData in pld)
{
if (string.IsNullOrEmpty(playlistData.WeeklyPlaylistID))
{
playlistData.WeeklyPlaylistID = await YoutubeAPIs.Instance.MakeWeeklyPlaylist().ConfigureAwait(false);
}
foreach (VideoData videoData in playlistData.Videos)
{
PlaylistItem playlistItem = await YoutubeAPIs.Instance.AddToPlaylist(videoData.VideoId, playlistId: playlistData.WeeklyPlaylistID, makeNewPlaylistOnError: false).ConfigureAwait(false);
videoData.WeeklyPlaylistItemId = playlistItem.Id;
++count;
}
}
nRows += await database.SaveChangesAsync().ConfigureAwait(false);
The query works correctly, I get all relevant Playlist and Video Rows to work with, they have the right data in only the specified columns, and the query that is logged looks good, but saves do not work and calling database.Entry() on any of the Playlists or Video objects show that they are all detached. What am I doing wrong? Are collections saved a different way? Should my query be changed? Is there a setting on initialization that should be changed? (The only setting I have set on init that I feel like may affect this is .UseQuerySplittingBehavior(QuerySplittingBehavior.SplitQuery) but the query logged isn't even split as far as I can see)

You work with projected objects
PlaylistData
VideoData
Projected objects does not tracked by EF core as far as I know. So the solution is to select DbSet's entity objects (mean types that specified in database.PlaylistsAdded and playlist.Videos properties) or select those objects before update and then update them.
UPDATE:
Example code for second option:
foreach (PlaylistData playlistData in pld)
{
var playlist = database.PlaylistsAdded
.Include(x=> x.Videos)
.First(x => x.PlaylistId == playlistData.playlistData);
if (string.IsNullOrEmpty(playlistData.WeeklyPlaylistID))
{
playlist.WeeklyPlaylistID = await YoutubeAPIs.Instance.MakeWeeklyPlaylist().ConfigureAwait(false);
}
foreach (VideoData videoData in playlistData.Videos)
{
var video = playlist.Videos.First(x=> x.VideoId == videoData.VideoId);
PlaylistItem playlistItem = await YoutubeAPIs.Instance.AddToPlaylist(videoData.VideoId, playlistId: playlistData.WeeklyPlaylistID, makeNewPlaylistOnError: false).ConfigureAwait(false);
video.WeeklyPlaylistItemId = playlistItem.Id;
++count;
}
}
NOTICE: this would produce double select's so first option is more preferred

C# Mapping a list to numbered variables

Yes, I'm well aware Not to do this, but I have no choice. I'd agree that it's an XYZ issue, but since I can't update the service I have to use, it's out of my hands. I need some help to save some time, maybe learn something handy in the process.
I'm looking to map a list of models (items in this example) to what is essentially numbered variables of a service I'm posting to, in the example, that's the fields a part of new 'newUser'.
Additionally, there may not be always be X amount items in the list (On the right in the example), and yet I have a finite amount (say 10) of numbered variables from 'newUser' to map to (On the left in the example). So I'll have to perform a bunch of checks to avoid indexing a null value as well.
Current example:
if (items.Count >= 1 && !string.IsNullOrWhiteSpace(items[0].id))
{
newUser.itemId1 = items[0].id;
newUser.itemName1 = items[0].name;
newUser.itemDate1 = items[0].date;
newUser.itemBlah1 = items[0].blah;
}
else
{
// This isn't necessary, but this effectively what will happen
newUser.itemId1 = string.Empty;
newUser.itemName1 = string.Empty;
newUser.itemDate1 = string.Empty;
newUser.itemBlah1 = string.Empty;
}
if (items.Count >= 2 && !string.IsNullOrWhiteSpace(items[1].id))
{
newUser.itemId2 = items[1].id;
newUser.itemName2 = items[1].name;
newUser.itemDate2 = items[1].date;
newUser.itemBlah2 = items[1].blah;
}
// Removed the else to clean it up, but you get the idea.
// And so on, repeated many more times..
I looked into an example using Dictionary, but I'm unsure of how to map that to the model without just manually mapping all the variables.
PS: To all who come across this question, if you're implementing numbered variables in your API, please don't- it's wildly unnecessary and time consuming.

As an alternative to fiddling with the JSON, you could get down and dirty and use Reflection.
Given the following test data:
const int maxItemsToSend = 3;
class ItemToSend {
public string
itemId1, itemName1,
itemId2, itemName2,
itemId3, itemName3;
}
ItemToSend newUser = new();
record Item(string id, string name);
Item[] items = { new("1", "A"), new("2", "B") };
Using the rules you set forth in the question, we can loop through the projected fields as so:
// If `itemid1`,`itemId2`, etc are fields:
var fields = typeof(ItemToSend).GetFields();
// If they're properties, replace GetFields() with
// .GetProperties(BindingFlags.Instance | BindingFlags.Public);
for(var i = 1; i <= maxItemsToSend; i++){
// bounds check
var item = (items.Count() >= i && !string.IsNullOrWhiteSpace(items[i-1].id))
? items[i-1] : null;
// Use Reflection to find and set the fields
fields.FirstOrDefault(f => f.Name.Equals($"itemId{i}"))
?.SetValue(newUser, item?.id ?? string.Empty);
fields.FirstOrDefault(f => f.Name.Equals($"itemName{i}"))
?.SetValue(newUser, item?.name ?? string.Empty);
}
It's not pretty, but it works. Here's a fiddle.

Comparing two lists to see if I need to update the database or even add to

I have a simple class which holds a primary key of which I don't know what type it will be before it runs, as i'm getting the data from COM. It will either be an int or string.
I basically just need to fill up my toUpdateList & toAddList. This was working fine below with not too many records to play around with. However now the mongoDBList returns around 65k records and it's all turned very slow and it's taking 15+ minutes to resolve toUpdateList.
I'm pretty new to C# so I'm likely missing something.
I basically just need to compare one list to another and see if the RecordRevision is higher in the toUpdateList. For the toAddList this ones pretty simple as if it doesn't exist it needs to be added.
Thanks for looking I appreciate it!
class KeyRevision
{
public dynamic RecordRevision;
public dynamic PrimaryKey;
}
List<KeyRevision> potentialUpdateList = new List<KeyRevision>();
List<KeyRevision> mongoDBList = new List<KeyRevision>();
List<KeyRevision> toUpdateList = new List<KeyRevision>();
List<KeyRevision> toAddList = new List<KeyRevision>();
var sql = env.ExecuteSQL(sqlQuery);
sql.First();
// Loop over them and add to array
do
{
if (sql.RecordCount > 0)
{
//Console.WriteLine(sql.GetPropertyValue(primaryKey).ToString() + " : " + sql.RecordRevision);
var o = new KeyRevision();
o.PrimaryKey = sql.GetPropertyValue(primaryKey);
o.RecordRevision = sql.RecordRevision;
potentialUpdateList.Add(o);
}
sql.Next();
} while (!sql.EOF);
// Ask mongo for docs
var docs = collection1.Find(_ => true).Project("{RecordRevision: 1}").ToList();
// Get them into our type
mongoDBList = docs.ConvertAll(x => new KeyRevision()
{
PrimaryKey = x.GetValue("_id"),
RecordRevision = x.GetValue("RecordRevision")
});
// Finds which records we need to update
toUpdateList = potentialUpdateList.Where(x =>
mongoDBList.Any(y => y.PrimaryKey == x.PrimaryKey && y.RecordRevision < x.RecordRevision)).ToList();
// Finds the records we need to add
toAddList = potentialUpdateList.Where(x =>
mongoDBList.FindIndex(y => y.PrimaryKey == x.PrimaryKey) < 0).ToList();
Console.WriteLine($"{toUpdateList.Count} need to be updated");
Console.WriteLine($"{toAddList.Count} need to be updated");

Catch read error on EF 6

I have a DB with no constraints (given, not changeable). My model look like
public MyModel
{
public long Id { get; set; }
// Even if database column 'Value' could be NULL,
// the model - from business view - could not.
public long Value { get; set; }
}
My data I'd like to read is
Id Value
1 1
2 2
3 NULL
4 4
When I read with with DBContext.MyModel.ToList() it fails, of course. Is there any possibility to catch the error on 3rd row and return the 3 valid ones?
I don't dependent on EF but I like an automatic mapping between DB an Code.
Update:
It seems I wasn't specific enough. I need the 3 rows AS WELL AS a notification for the error.
Additional I've created a simple case for demo. In real life I have around 800 tables with up to 250 columns. I can't catch anything by model modification like dates out of range, missing relationships and other stuff.
What I really need is a try..catch for every row or an event on row reading failure, something like this.

Ok, solved. Not very elegant, but functional.
var query = _DBContext
.Database
.SqlQuery<MyModel>("SELECT * FROM MyModel");
var result = new List<MyModel>();
var enumerator = query.GetEnumerator();
while (true)
{
try
{
var success = enumerator.MoveNext();
if (!success)
break;
var model = enumerator.Current;
result.Add(model);
}
catch (Exception ex)
{
}
}
return result;

You need to use nullable type
public MyModel
{
public long Id { get; set; }
// Even if database column 'Value' could be NULL,
// the model - from business view - could not.
public long? Value { get; set; }
}
And also, in your select query, you should exlude Value = null
var myModel = models.Where(x => x.Value != null);
Hope it helps.

I'm not entirely sure I understand what you're trying to do, but if you want to get a reflection of what's in the table, then why not make your model match the query? If Value can be NULL, then make Value nullable (i.e. define it as long?).
That way, you can simply do:
var records = DbContext.MyModel.ToList();
If you then want to filter out the NULLs, you can do:
records.Where(r => r.Value.HasValue)
And if you want the ones with NULLs you can do:
records.Where(r => !r.Value.HasValue)
Or if you want to know whether any row had a NULL you could do:
records.Any(r => !r.Value.HasValue)

Use the following code:
var list = from m in DBContext.MyModel
where (m != null)
select m;
And then just convert var list to a List of your choosing.
Edit 1
var myModel = models.Where(x => x.Value != null).ToList();
As kienct89 suggested might also work.
Edit 2
There are multiple options for "catching" the error
If you want to throw an exception just use this:
if(myList.Count() < DBContext.MyModel.Count()){
Exception myException = new Exception("Not all items ware correctly loaded");
throw myException;
}
OR create a seperate array with the faulty ones:
var faultyList = from m in DBContext.MyModel
where (m == null)
select m;
Or:
var faultyList= models.Where(x => x.Value == null).ToList();

Why does this EF upsert logic result in deletions instead of updates?

The following code results in deletions instead of updates.
My question is: is this a bug in the way I'm coding against Entity Framework or should I suspect something else?
Update: I got this working, but I'm leaving the question now with both the original and the working versions in hopes that I can learn something I didn't understand about EF.
In this, the original non working code, when the database is fresh, all the additions of SearchDailySummary object succeed, but on the second time through, when my code was supposedly going to perform the update, the net result is a once again empty table in the database, i.e. this logic manages to be the equiv. of removing each entity.
//Logger.Info("Upserting SearchDailySummaries..");
using (var db = new ClientPortalContext())
{
foreach (var item in items)
{
var campaignName = item["campaign"];
var pk1 = db.SearchCampaigns.Single(c => c.SearchCampaignName == campaignName).SearchCampaignId;
var pk2 = DateTime.Parse(item["day"].Replace('-', '/'));
var source = new SearchDailySummary
{
SearchCampaignId = pk1,
Date = pk2,
Revenue = decimal.Parse(item["totalConvValue"]),
Cost = decimal.Parse(item["cost"]),
Orders = int.Parse(item["conv1PerClick"]),
Clicks = int.Parse(item["clicks"]),
Impressions = int.Parse(item["impressions"]),
CurrencyId = item["currency"] == "USD" ? 1 : -1 // NOTE: non USD (if exists) -1 for now
};
var target = db.Set<SearchDailySummary>().Find(pk1, pk2) ?? new SearchDailySummary();
if (db.Entry(target).State == EntityState.Detached)
{
db.SearchDailySummaries.Add(target);
addedCount++;
}
else
{
// TODO?: compare source and target and change the entity state to unchanged if no diff
updatedCount++;
}
AutoMapper.Mapper.Map(source, target);
itemCount++;
}
Logger.Info("Saving {0} SearchDailySummaries ({1} updates, {2} additions)", itemCount, updatedCount, addedCount);
db.SaveChanges();
}
Here is the working version (although I'm not 100% it's optimized, it's working reliably and performing fine as long as I batch it out in groups of 500 or less items in a shot - after that it slows down exponentially but I think that just may be a different question/subject)...
//Logger.Info("Upserting SearchDailySummaries..");
using (var db = new ClientPortalContext())
{
foreach (var item in items)
{
var campaignName = item["campaign"];
var pk1 = db.SearchCampaigns.Single(c => c.SearchCampaignName == campaignName).SearchCampaignId;
var pk2 = DateTime.Parse(item["day"].Replace('-', '/'));
var source = new SearchDailySummary
{
SearchCampaignId = pk1,
Date = pk2,
Revenue = decimal.Parse(item["totalConvValue"]),
Cost = decimal.Parse(item["cost"]),
Orders = int.Parse(item["conv1PerClick"]),
Clicks = int.Parse(item["clicks"]),
Impressions = int.Parse(item["impressions"]),
CurrencyId = item["currency"] == "USD" ? 1 : -1 // NOTE: non USD (if exists) -1 for now
};
var target = db.Set<SearchDailySummary>().Find(pk1, pk2);
if (target == null)
{
db.SearchDailySummaries.Add(source);
addedCount++;
}
else
{
AutoMapper.Mapper.Map(source, target);
db.Entry(target).State = EntityState.Modified;
updatedCount++;
}
itemCount++;
}
Logger.Info("Saving {0} SearchDailySummaries ({1} updates, {2} additions)", itemCount, updatedCount, addedCount);
db.SaveChanges();
}
The thing that keeps popping up in my mind is that maybe the Entry(entity) or Find(pk) method has some side effects? I should probably be consulting the documentation but any advice is appreciated..

It's a slight assumption on my part (without looking into your models/entities), but have a look at what's going on within this block (see if the objects being attached here are related to the deletions):
if (db.Entry(target).State == EntityState.Detached)
{
db.SearchDailySummaries.Add(target);
addedCount++;
}
Your detached object won't be able to use its navigation properties to locate its related objects; you'll be re-attaching an object in a potentially conflicting state (without the correct relationships).
You haven't mentioned what is being deleted above, so I may be way off. Just off out, so this is a little rushed, hope there's something useful in there.