A beginner in SQL/DB here - I'm designing a universal windows app where a DB might come in handy, so I'm reading on SQL/SQLite and I'm wondering - how does backwards compatibility (classes/tables wise) works? Suppose I add or remove a property from a class that I've been using as a base for a table - will I still be able to interact with the old - class data and cast it into the updated class? How does one go about monitoring the added/ removed properties while querying the db?
Thank you for your time.
Edit: Scenario - if I've got a User with 2 properties - 'Id', 'Name'.
public class User
{
public int Id {get; set; }
public string Name {get; set;}
}
And I've built an application with a 'Users' Db and I'm using that for a while. Then, I would like to add another property to my User- class. Suppose it's 'Age'. Can I add this new property to my original 'User' class and still be able to access the original Users table with:
var user = DbConnection.Table<User>().Where(x => x.Id == tId).FirstOrDefault();
Obviously, one option would be to keep the original User and create another class UserEx which will have the original properties + Age. Then I grab all 'User's, port them to UserEx and save to a new table.
This does seem a bit cumbersome to do for each added/ removed property though
Can I add this new property to my original 'User' class and still be
able to access the original Users table with ?
var user = DbConnection.Table<User>().Where(x => x.Id == tId).FirstOrDefault();
Does adding a property to the user class update the underlying SQLLite
table? You would need to extend the class to maintain compatibility.
public class User
{
public int Id {get; set; }
public string Name {get; set;}
public User getUser(int tId)
{
var user = DbConnection.Table<User>().Where(x => x.Id == tId).FirstOrDefault();
Id = user.Id;
Name = user.Name;
//age = user.age; // not possible
}
}
public class DetailedUser:User
{
public int age { get; set; }
public DetailedUser getUser(int tId)
{
var user = DbConnection.Table<User>().Where(x => x.Id == tId).FirstOrDefault();
base.Id = user.Id;
base.Name = user.Name;
//age = user.age; // not possible
}
public DetailedUser getDetailedUser(int tId)
{
var user = DbConnection.Table<DetailedUser>().Where(x => x.Id == tId).FirstOrDefault();
base.Id = user.Id;
base.Name = user.Name;
age = user.age;
}
}
The obvious 'cumbersome' process you outlined is another valid option. Either way when you are changing the data layer in your application there are consequences.
Two things come to mind, one is that in Android SqlLite has a onUpgrade function whey you can put your "cumbersome" data layer upgrade logic.. or you could use a NoSQL solution like MongoDB where the data layer is much more forgiving about the structure of your underlying records (ymmv).
Finally the better you plan your data layer structure the fewer times you will run into this kind of issue.
I hope that is of some help.
Related
I can add, but not erase any item with the collection - unable to delete.
Found a few partial solutions, but nothing to guide me to a working solution. I can easily add values to the collection; ny help is appreciated.
I have the following:
[HttpPut("updateSOJ4")]
public IActionResult UpdateSOJ4([FromBody] Routing_Tool_SOJ4 Routing_Tool_SOJ4)
{
Routing_Tool_SOJ4 request = new Routing_Tool_SOJ4();
request.Id = Routing_Tool_SOJ4.Id;
request.Routing_Tool_Services = Routing_Tool_SOJ4.Routing_Tool_Services;
request.Routing_ToolId = Routing_Tool_SOJ4.Routing_ToolId;
_repository.UpdateSOJ4(request);
return Ok(request);
}
Here is where I was trying the different solutions, but, I am still stuck:
public void UpdateSOJ4(object routing_Tool_SOJ4)
{
// var missingItem = _context.Routing_Tool_Service.Where(i => i.Routing_Tool_SOJ4Id == _context.Routing_Tool_SOJ4.Id).First(); -- DOES NOT WORK
_context.Update(routing_Tool_SOJ4).State = EntityState.Modified;
_context.SaveChanges();
}
Here is the database structure:
public class Routing_Tool_SOJ4
{
[Key]
[Required]
public int Id { get; set; }
public int Routing_ToolId { get; set; }
[ForeignKey("Routing_ToolId")]
public virtual Routing_Tool Routing_Tool { get; set; }
public virtual ICollection <Routing_Tool_Service> Routing_Tool_Services { get; set; }
}
Collection:
public class Routing_Tool_Service
{
[Key]
[Required]
public int Id { get; set; }
public string ServiceName { get; set; }
[Required]
[ForeignKey("Routing_Tool_SOJ4Id")]
public int Routing_Tool_SOJ4Id { get; set; }
}
What I am deduce from your question is you have a method that accepts an updated Routing Tool object which contains an updated collection of Tool Services. You want to update that tool and it's associated services so that any service within that tool that is new gets added, otherwise updated, and any existing tool in the DB that is no longer in the passed in collection should be deleted..
If this is the case, you need to compare the provided version of the data to the database version of the data. For this example I am not using your Repository instance because I have no idea how it is implemented. Generally this pattern should be avoided unless there is a really good reason to have it.
[HttpPut("updateSOJ4")]
public IActionResult UpdateSOJ4([FromBody] Routing_Tool_SOJ4 updatedRoutingTool)
{
using (var context = new AppDbContext())
{
// Get tool and services from DB.
var existingRoutingTool = context.Routing_Tool_SOJ4s
.Include(x => x.Routing_Tool_Services)
.Single(x => x.Id == updatedRoutingTool.Id);
// Copy values that can be updated from the updatedRoutingTool to existingRoutingTool.
// ...
var updatedServiceIds = updatedRoutingTool.Routing_Tool_Services
.Select(x => x.Id)
.ToList();
var existingServiceIds = existingRoutingTool.Routing_Tool_Services
.Select(x => x.Id)
.ToList();
var serviceIdsToRemove = existingServiceIds
.Except(updatedServiceIds)
.ToList();
foreach (var service in updatedRoutingTool.Routing_Tool_Services)
{
var existingService = existingRoutingTool.Routing_ToolServices
.SingleOrDefault(x => x.Id == service.Id);
if (existingService == null)
existingRoutingTool.Routing_Tool_Services.Add(service);
else
{
// Copy allowed values from service to existingService
}
}
if(serviceIdsToRemove.Any())
{
var servicesToRemove = existingRoutingTool.Routing_Tool_Services
.Where(x => serviceIdsToRemove.Contains(x.Id))
.ToList();
foreach(var serviceToRemove in servicesToRemove)
existingRoutingTool.Routing_Tool_Services.Remove(serviceToRemove);
}
context.SaveChanges();
}
return Ok(request);
}
Normally the DbContext or Unit of Work would be injected into your controller, or the logic would be handed off to a service. This example uses a using block with a DbContext just to outline the minimum viable process flow for the operation.
Essentially load the current data state, compare that with the provided state to determine what needs to be added, updated, or removed.
Generally speaking when it comes to RESTful web services my recommendation is to avoid large update operations like this and instead structure the application to perform more atomic operations such as adding and removing services for a given tool as a distinct operation, working with a persisted copy (i.e. cached instance) of the data if you want the whole related operation to be committed to data state or abandoned at a higher level. This can help keep message sizes small, and server code more compact & worrying about a single responsibility. The risk of performing these large operations is that the passed in data must represent a complete picture of the data state or you could end up deleting/clearing data you don't intend. For example if you later want to optimize your code so that only added and updated services are sent over the wire, not unchanged services (to reduce message size) the above code will not work as it would delete anything not sent.
I am using Entity Framework Core with npgsql postgresql for Entity Framework Core.
and i'm working with .net core 3
My question is, when i try to update a MyTableRelated element from the MyTableClass and saving the context to the database, no changes are detected.
For example, lets suppose we have the following classes:
public class MyTableClass
{
public int Id { get; set; }
[Column(TypeName = "jsonb")]
public virtual List<MyTableRelated> Data { get; set; }
}
public class MyTableRelated
{
public int Id { get; set; }
public string prop1 { get; set; }
public string prop2 { get; set; }
}
and some code like this (this is not actual code, its just to get the ideia):
var context = dbContext;
var newMyTableClass = new MyTableClass() {
Id = 1;
};
var newMyTableRelated = new MyTableRelated(){
Id=1;
prop1 = "";
prop2 = "";
}
newMyTableClass.Data.Add(newMyTableRelated);
context.SaveChanges();
This works, and the entry is saved on the database.
Now somewhere on the application, i want to access that entry and change values on Data:
var context = dbContext;
var updateMyTableClass = context.MyTableClass.FirstOrDefault(x => x.Id == 1);
var tableRelated = updateMyTableClass.Data.FirstOrDefault(y => y.Id == 1);
tableRelated.prop1 = "prop1";
tableRelated.prop2 = "prop2";
context.SaveChanges();
I would suppose this would change values on database, like it does for other types of properties. But nothing happens.
A solution i found, was using this:
var entry = context.Entry(updateMyTableClass);
if (entry.State == EntityState.Unchanged)
{
entry.State = EntityState.Modified;
}
This is more of a temporary solution for that case.
How can we then make the EF automatically detect changes on jsonb properties?
Someone pointed to me that i should look at coase grained lock.
https://www.martinfowler.com/eaaCatalog/coarseGrainedLock.html
How can something like that be implemented?
Automatic change detection would mean that EF Core would take a snapshot of the JSON document when it loads the property (duplicating the entire tree), and then do a complete structural comparison of the original and current tree whenever SaveChanges is called. As this can be very heavy perf-wise, it is not done by default.
However, if you wish to do so, you can create a value comparer to implement precisely this - see the EF docs on how to do that. I've opened an issue on the Npgsql provider repo in case someone wishes to contribute this.
For perf reasons, I'd recommend manually flagging properties when they change, similar to what you have done. Note that you're marking the entire entity instance as changed - so all properties will be saved. You can use the following to only mark the JSON property:
ctx.Entry(entry).Property(e => e.SomeJsonProperty).IsModified = true;
This should be really simple but I think I'm having possible issues with my model. I have been working with linq over a year and I should have this simple remove easily done. Please help! It's removing both records from the database when I only want one deleted
I have a database table with these properties.
Email, EmployeeName, StoreId
jsch#m.com,Joe Schneider,9
jsch#m.com,Joe Schneider,8
I need to delete Joe Schneider with storeId 9
So I run this simple query and remove process.
var temp2 = difference[i];
var PersonToRemove = db.Permissions.SingleOrDefault(s => s.EmployeeName == temp2 && s.StoreId == Persons.StoreId);
if (PersonToRemove.EmployeeName != null)
{
db.Permissions.Remove(PersonToRemove);
db.SaveChanges();
}
I am assuming one is going to say, hey your model is not right and don't put the name as a key, but I can't just be changing the model because other parts of the app are based on this model and would cause huge breaks. Could you give me advise how to edit the linq query to not delete both records?
model
[Table("Permissions")]
public class Permissions
{
[Key]
public String EmployeeName { get; set; }
public string Department { get; set; }
public int? StoreId { get; set; }
public String Email { get; set; }
}
[Table("Permissions")]
public class Permissions
{
[Key]
public String EmployeeName { get; set; }
}
The problem is here you are defining a primary key which has no length constraint on it. (MaxLength). This leads to EF generate a column with NVARCHAR(MAX). As mentioned here VARCHAR(MAX) columns are not allowed to be primary key. So correct definition should be like below
[Table("Permissions")]
public class Permissions
{
[Key]
[DatabaseGenerated(DatabaseGeneratedOption.None)] <--
[MaxLength(255)] // <---
public String EmployeeName { get; set; }
}
Edit: You need to recreate the database in order to associated tables initialized with correct settings.
Edit 2 : Also you may need a DatabaseGenerated(DatabaseGeneratedOption.None) since its not identity column.
you can set Deleted state on individual entity like so:
var temp2 = difference[i];
var PersonToRemove = db.Permissions.SingleOrDefault(s => s.EmployeeName == temp2 && s.StoreId == Persons.StoreId);
if (PersonToRemove.EmployeeName != null)
{
db.Entry(PersonToRemove).State = EntityState.Deleted; // do this instead
db.SaveChanges();
}
EF should then figure out which entity you wanted to delete
UPD
I am assuming you are using EF6 and DB-first approach. I am also assuming you've got your DB context class set up with default convention model builder. It seems EF's default object tracking based on Key will not work as your key is not unique (this is a bigger problem, but I understand you're already aware of that).
You might try circumvent that convention by adding custom model builder configuration like so:
class MyDbContext : DbContext {
public virtual DbSet<Permissions> Permissions {get;set;}
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
modelBuilder.Entity<Permissions>().HasKey(p => new { p.EmployeeName, p.StoreId});
}
}
since you didn't share your DbContext definition this is just a snippet but hopefully gives you some ideas to explore.
this is the API reference: https://learn.microsoft.com/en-us/dotnet/api/system.data.entity.modelconfiguration.entitytypeconfiguration-1?view=entity-framework-6.2.0
i am struggeling for a while now to understand how EF loads / updates entities.
First of all i wanna explain what my app (WPF) is about. I am developing
an application where users can store Todo Items in Categories, these categories are predefined by the application. Each user can read all items but can only delete / update his own items. It's a multiuser system, means the application is running multiple times in the network accessing the same sql server database.
When a user is adding/deleting/updating items the UI on all the other running apps has to update.
My model looks like this:
public class Category
{
public int Id { get; set; }
public string Name { get; set; }
public List<Todo> Todos { get; set; }
}
public class Todo
{
public int Id { get; set; }
public string Content { get; set; }
public DateTime LastUpdate { get; set; }
public string Owner { get; set; }
public Category Category { get; set; }
public List<Info> Infos { get; set; }
}
public class Info
{
public int Id { get; set; }
public string Value { get; set; }
public Todo Todo { get; set; }
}
I am making the inital load like this, which works fine:
Context.dbsCategories.Where(c => c.Id == id).Include(c => c.Todos.Select(t => t.Infos)).FirstOrDefault();
Now i was trying to load only the Todos which are from the current user therefore i tried this:
Context.dbsCategories.Where(c => c.Id == id).Include(c => c.Todos.Where(t => t.Owner == Settings.User).Select(t => t.Infos)).FirstOrDefault();
This does not work because it's not possible to filter within include, so I tried this:
var cat = Context.dbsCategories.Where(c => c.Id == id).FirstOrDefault();
Context.dbsTodos.Where(t => t.Category.Id == cat.Id && t.Owner == Settings.User).Include(t=>t.Infos);
After executing the second line where i look for the Todo Items, these Items were automatically added to cat's Todos collection. Why? I would have expected that i have to add them manually to cat's Todos collection.
Just for my understanding what is EF doing here exactly?
Now to my main problem -> the synchronization of the data between database and client. I am using a long running Context which lives as long as the application is running to save changes to the database which are made on owned items. The user does not have the possibility to manipulate / delete data from other users this is guarantee by the user interface.
To synchronize the data i build this Synch Method which will run every 10 second, right now it's triggere manually.
Thats my synchronization Code, which only synchronizes Items to the client that do not belong to it.
private async Task Synchronize()
{
using (var ctx = new Context())
{
var database = ctx.dbsTodos().Where(x => x.Owner != Settings.User).Select(t => t.Infos).AsNoTracking();
var loaded = Context.dbsTodos.Local.Where(x => x.Owner != Settings.User);
//In local context but not in database anymore -> Detachen
foreach (var detach in loaded.Except(database, new TodoIdComparer()).ToList())
{
Context.ObjectContext.Detach(detach);
Log.Debug(this, $"Item {detach} detached");
}
//In database and local context -> Check Timestamp -> Update
foreach (var update in loaded.Intersect(database, new TodoIdTimeStampComparer()))
{
await Context.Entry(update).ReloadAsync();
Log.Debug(this, $"Item {update} updated");
}
//In database but not in local context -> Attach
foreach (var attach in database.ToList().Except(loaded, new TodoIdComparer()))
{
Context.dbsTodos().Attach(attach);
Log.Debug(this, $"Item {attach} attached");
}
}
}
I am having following problems / issues of unknow origin with it:
Detaching deleted Items seems to work, right now i am not sure if only the Todo Items are detached or also the Infos.
Updating Items works only for the TodoItem itsself, its not reloading the Infos within? How can i reload the whole entity with all it's relations?
I am thankful for every help on this, even if you are saying it's all wrong what i am doing here!
Attaching new Items and Infos does not work so far? What am i doing wrong here?
Is this the right approach to synchronize data between client and database?
What am i doing wrong here? Is there any "How to Sync" Tutorial? I have not found anything helpful so far?
Thanks!
My, you do like to deviate from entity framework code-first conventions, do you?
(1) Incorrect class definitions
The relations between your tables are Lists, instead of ICollections, they are not declared virtual and you forgot to declare the foreign key
There is a one-to-many relation between Todo and Category: every Todo belongs to exactly one Category (using a foreign key), every Category has zero or more Todos.
You choose to give Category a property:
List<Todo> Todos {get; set;}
Are you sure that category.Todos[4] has a defined meaning?
What would category.Todos.Insert(4, new Todo()) mean?
Better stick to an interface where you can't use functions that have no proper meaning in your database: use ICollection<Todo> Todos {get; set;}. This way you'll have only access to functions that Entity Framework can translate to SQL.
Besides, a query will probably be faster: you give entity framework the possibility to query the data in its most efficient way, instead of forcing it to put the result into a List.
In entity framework the columns of a table are represented by non-virtual properties; the virtual properties represent the relations between the tables (one-to-many, many-to-many)
public class Category
{
public int Id { get; set; }
public string Name { get; set; }
... // other properties
// every Category has zero or more Todos (one-to-many)
public virtual ICollection<Todo> Todos { get; set; }
}
public class Todo
{
public int Id { get; set; }
public string Content { get; set; }
... // other properties
// every Todo belongs to exactly one Category, using foreign key
public int CategoryId { get; set }
public virtual Category Category { get; set; }
// every Todo has zero or more Infos:
public virtual ICollection<Info> Infos { get; set; }
}
You'll probably guess Info by now:
public class Info
{
public int Id { get; set; }
public string Value { get; set; }
... // other properties
// every info belongs to exactly one Todo, using foreign key
public int TodoId {get; set;}
public virtual Todo Todo { get; set; }
}
Three major improvements:
ICollections instead of Lists
ICollections are virtual, because it is not a real column in your table,
foreign key definitions non-virtual: they are real columns in your tables.
(2) Use Select instead of Include
One of the slower parts of a database query is the transport of the selected data from the Database Management System to your local process. Hence it is wise to limit the amount of transported data.
Suppose Category with Id [4] has a thousand Todos. Every Todo of this Category will have a foreign key with a value 4. So this same value 4 will be transported 1001 times. What a waste of processing power!
In entity framework use Select instead of Include to query data and select only the properties you actually plan to use. Only use Include if you plan to update the Selected data.
Give me all Categories that ... with their Todos that ...
var results = dbContext.Categories
.Where(category => ...)
.Select(category => new
{
// only select properties that you plan to use
Id = category.Id,
Name = category.Name,
...
Todos = category.Todos
.Where(todo => ...) // only if you don't want all Todos
.Select(todo => new
{
// again, select only the properties you'll plan to use
Id = todo.Id,
...
// not needed, you know the value:
// CategoryId = todo.CategoryId,
// only if you also want some infos:
Infos = todo.Infos
.Select(info => ....) // you know the drill by now
.ToList(),
})
.ToList(),
});
(3) Don't keep DbContext alive for such a long time!
Another problem is that you keep your DbContext open for quite some time. This is not how a dbContext was meant. If your database changes between your query and your update, you'll have troubles. I can hardly imagine that you query so much data that you need to optimize it by keeping your dbContext alive. Even if you query a lot of data, the display of this huge amount of data would be the bottle-neck, not the database query.
Better fetch the data once, dispose the DbContext, and when updating fetch the data again, update the changed properties and SaveChanges.
fetch data:
RepositoryCategory FetchCategory(int categoryId)
{
using (var dbContext = new MyDbContext())
{
return dbContext.Categories.Where(category => category.Id == categoryId)
.Select(category => new RepositoryCategory
{
... // see above
})
.FirstOrDefault();
}
}
Yes, you'll need an extra class RepositoryCategory for this. The advantage is, that you hide that you fetched your data from a database. Your code would hardly change if you'd fetch your data from a CSV-file, or from the internet. This is way better testable, and also way better maintainable: if the Category table in your database changes, users of your RepositoryCategory won't notice it.
Consider creating a special namespace for the data you fetch from your database. This way you can name the fetched Category still Category, instead of RepositoryCategory. You even hide better where you fetched your data from.
Back to your question
You wrote:
Now i was trying to load only the Todos which are from the current user
After the previous improvements, this will be easy:
string owner = Settings.User; // or something similar
var result = dbContext.Todos.Where(todo => todo.Owner == owner)
.Select(todo => new
{
// properties you need
})
I have a Customer class that has a relationship to an Address class:
public class Customer
{
public int Id { get; set; }
public string Name { get; set; }
public virtual ICollection<Address> Addresses { get; set; }
}
public class Address
{
public int Id { get; set; }
public string Street1 { get; set; }
//Snip a bunch of properties
public virtual Customer Customer { get; set; }
}
I have an edit form which displays all the fields for both the customer and address. When this form is submitted, it calls the Edit method in the controller:
public ActionResult Save(Customer customer)
{
if (!ModelState.IsValid)
{
var viewModel = new CustomerFormViewModel
{
Customer = customer,
CustomerTypes = _context.CustomerTypes.ToList()
};
return View("CustomerForm", viewModel);
}
if (customer.Id == 0)
_context.Customers.Add(customer);
else
{
var existingCustomer = _context.Customers
.Include(c => c.Addresses)
.Single(c => c.Id == customer.Id);
existingCustomer.Name = customer.Name;
existingCustomer.TaxId = customer.TaxId;
existingCustomer.CustomerTypeId = customer.CustomerTypeId;
existingCustomer.CreditLimit = customer.CreditLimit;
existingCustomer.Exempt = customer.Exempt;
existingCustomer.Addresses = customer.Addresses;
}
_context.SaveChanges();
return RedirectToAction("Index", "Customers");
}
This doesn't work and creates duplicate entries in the Addresses table in the DB. I think I understand why (EF isn't smart enough to know the Addresses inside the collection need to be added/modified/deleted as the case may be). So, what is the best way to fix this?
My instinct is that I need to iterate over the Addresses collections and compare them manually, adding any new ones from the form that don't exist for the customer, updating ones that do exist, and deleting ones that were not sent by the form but exist in the DB for the customer. Something like (ignoring the delete functionality for now):
foreach(Address address in customer.Addresses)
{
if (address.Id == 0)
// Add record
else
// Fetch address record from DB
// Update data
}
// Save context
Is this the best way to go about this, or are there any EF tricks to iterating and syncing a child collection to the DB?
Oh, and one question which has me scratching my head - I can sort of understand how a new address record is getting created in the DB, but what I don't get is the existing address record is also updated to have its customer_id set to NULL...how the heck does that happen? That leads me to believe that EF does see the original address record is somehow linked (as it is modifying it) but it's not smart enough to realize the record I'm passing in should replace it?
Thanks -- also, this is EF6 and MVC5
The problem comes from the line
existingCustomer.Addresses = customer.Addresses;
in your code. This like assigns field Addresses from customer coming from the model. So far ok. The point is that customer does not have any relation to the database model at this point (it's not coming from the database but from the view).
If you would like to update existingCustomer.Addresses with the data coming from the model, you need to merge the data instead of replacing it. The following "pseudo code" might give you a direction:
void MergeAddresses(var existingAddresses, var newAddresses) {
foreach(var address in newAddresses) {
if (existingAddresses.Contains(newAddress)) {
// merge fields if applicable
}
else {
// add field to existingAddresses - be ware to use a "cloned" list
}
}
// now delete items from existing list
foreach (var address in existingAddresses.CloneList()) {
if (!newAddresses.Contains(address)) {
// remove from existingAddresses
}
}
}
Is this the best way to go about this, or are there any EF tricks to iterating and syncing a child collection to the DB?
No, there aren't such tricks. EF designers left saving detached entities totally up to us - the developers.
However there is a package called GraphDiff which is addressing that, so you could give it a try. Here is how your code would look like using it:
using RefactorThis.GraphDiff;
...
_context.UpdateGraph(customer, map => map.OwnedCollection(
e => e.Addresses, with => with.AssociatedEntity(e => e.Customer)));
_context.SaveChanges();