I wonder what you are using for updating a client database when your program is patched?
Let's take a look at this scenario:
You have a desktop application (.net, entity framework) which is using sql server compact database.
You release a new version of your application which is using extended database.
The user downloads a patch with modified files
How do you update the database?
I wonder how you are doing this process. I have some conception but I think more experienced people can give me better and tried solutions or advice.
You need a migration framework.
There are existing OSS libraries like FluentMigrator
project page
wiki
long "Getting started" blogpost
Entity Framework Code First will also get its own migration framework, but it's still in beta:
Code First Migrations: Beta 1 Released
Code First Migrations: Beta 1 ‘No-Magic’ Walkthrough
Code First Migrations: Beta 1 ‘With-Magic’ Walkthrough (Automatic Migrations)
You need to provide explicitly or hidden in your code DB upgrade mechanism, and - thus implement something like DB versioning chain
There are a couple of aspects to it.
First is versioning. You need some way of tying teh version of teeh db to the version of the program, could be something as simple as table with a version number in it. You need to check it on executing the application as well.
One fun scenario is you 'update' application and db successfully, and then for some operational reason the customer restores a previous version of the db, or if you are on a frequent patch cycle, do you have to do each patch in order or can thay catch up. Do you want to deal with application only or database only upgrades differently?
There's no one right way for this, you have to look at what sort of changes you make, and what level of complexity you are prepared to maintain in order to cope with everything that could go wrong.
A couple a of things worth looking at.
Two databases, one for static 'read-only' data, and one for more dynamic stuff. Upgrading the static data, can then simply be a restore from a resource within the upgrade package.
The other is how much can you do with meta-data, stored in db tables. For instance a version based xsd to describe your objects instead of a concrete class. That's goes in your read only db, now you've updated code and application with a restore and possibly some transforms.
Lots of ways to go, just remember
'users' will always find some way of making you look like an eejit, by doing something you never thought they would.
The more complex you make the system, the more chance of the above.
And last but not least, don't take short cuts on data version conversions, if you lose data integrity, everything else you do will be wasted.
Related
We are building an Asp.Net core web API that must use MongoDb on the backend. According to what I found, it is recommended to use directly MongoDb.Driver because it does already most of the job and it makes less sense to use an ORM(EF Core) with a NoSql DB.
One thing that I'm not sure:
Is there a way to make "migration" as we would with Entity Framework? Same thing for the data seeding? I could imagine some way to doing it myself, but it feels like re-inventing the wheel.
So, how should we handle the potentials data migrations?
Ps, I understand that if we just add a property, we might not do an update, but there might be some occasion where you have a real structure changes
Some of the changes can be handled without explicit migrations; for some examples, see this link.
For others, we executed some code at startup that created indexes, set up collections or performed migrations. In the best case, this code is idempotent, so that it can be run multiple times. Otherwise we stored a migration marker in the database so that only the necessary migrations are run and parallel executions of the code are avoided.
If you need a more sophisticated approach with documents storing their version and the option to perform an on-the-fly-migration on a per-document-basis, you could check out this package.
Up to now, it was sufficient for us to perform the migrations at startup so that we haven't used this package yet.
In my Application I have used Entity Framework Database First approach.
Currently my application is in Dev Environment, now it need to be moved into Test Environment and later into Production Environment.
So is there anyway that I can use .net feature or Entity framework feature to migrate/create database in Test environment. Other than using SQL feature of restoring the database.
Also note that if any enhancement comes then Database structure can change, table schema can change.
So can you suggest me the best way to easily migrate database schema in different environment without losing existing Data.
If you want to take advantage of EF-Migrations feature, you must convert your application to Code First with Existing Database http://msdn.microsoft.com/en-us/data/jj200620.aspx
If you are unable to convert to code first then you must create the update script by hand.
Use a schema compare tool, compare the development and production server.
For each difference found, create an update query.
Once the entire script is finished, test it on the staging server.
Automating the migration is very risky, it depends on the type and size changes you made to the schema. You can't trust any single feature or tool specially if the changes requires data motion (moving data around).
The following links might help you:
How to do Migrations in DB first approach
EF Migrations for Database-first approach?
With Database First, the easiest way to copy a schema is to extract a data tier application in management studio, create an empty database on the target, register it as a data tier application with the same name, and upgrade the empty database using the upgraded file. You can repeat this step to manage schema changes.
Having said that, going forward you're really better off switching your Database First to Code First as it will make change management across your deployments much easier.
Migrations are best way to deal with it
Preferred way to update production db is to first generate sql file and then run the sql file in production environment.
MS had a very good artical on this
http://msdn.microsoft.com/en-in/data/jj591621.aspx#script
We want to progress towards being able to do continuous delivery of of our application into production. We currently deploy to azure and use table/blob storage and have a azure sql database, which we access with the entity.
As the database schema changes we want to be able to automatically apply the schema changes to the production database, but as this will happen whilst the application is live and the code changes are being deployed to many nodes at the same time we are not sure what the correct approach is.
After some reading it seems (and this makes sense) that the application needs to be tolerant of the 2 different database schema versions, so that it doesn't matter if its an old version of the code or a new version of the code which sees the database, however I'm not sure what the best way to approach handling this in the application is, using the entity framework.
Should we have versioned instances of the EF generated classes in the code which know how to access a specific version of the schema? What happens when the schema is updated and an old version of the code is running against the database?
Our entity framework classes are mapped to views in specific schemas in the db and nothing is mapped to the underlying tables, so potentially this could allow us to create v1 views which the old code uses and v2 views which the new code uses, but maintaining this feels like it would be a bit of a nightmare (its already enough of a pain simply maintaining the EF mappings to views rather than tables)
So what are best practices in this area? What do others do to solve this problem?
Whether you use EF or not, maintaining the code's ability to work with 2 consecutive versions of the database is a good (and perhaps the only viable) approach here.
Here are some ways we handle specific types of migrations:
When adding a column, we can typically just add the column (with a default constraint if non-nullable) and not worry about the code. EF will never issue a "SELECT *", so it will be able to continue to function properly while ignoring the new column. Similarly, adding a table is easy.
When removing a column or table, simply keep that column around 1 version longer than you would have otherwise.
For more complex migrations (e. g. completely changing the structure for a table or segment of the data model), deploy the new model alongside backwards-compatibility views (or tables with triggers to keep them in-sync), which will live as long as does the code that references them. As you say, this can a lot of work depending on the complexity of the migration, but it sounds like you are already well-positioned to do this because your EF entities point to views anyway. On the other hand, the benefit of this work is that you have more time to do the code migration. If you have a large codebase, this could be really beneficial in allowing you to migrate the data model to fit the needs of new features while still supporting old features without major code changes.
As a side-note, the difficulty of data migration often makes us push developing a finalized data model as far back as possible in the development schedule. With EF, you can write and test a lot of code before the data model is finalized (we use code-first to generate a sample SQLExpress database in a unit tests, even though our production database is not maintained by code-first). That way, we make fewer incremental changes to the production data model once a new feature is released.
I'm just starting out an extremely large web project, and really trying to do everything right.
My tools using so far are
ASP.NET MVC 3
Entity Framework 4.3
Ninject 3
All is going well, but I'm finding a few things with Entity Framework CodeFirst a bit sketchy.
For example, I had to use http://codefirstmembership.codeplex.com/ to setup membership information as part of the code first setup. This feels a bit ropey to have to use something third party of this. Obviously I should be 1337 enough to "roll my own" but I don't want to bite off too much from the get go. Running aspnet_regsql feels horrible, and will get lost with each db update. Anyway, got it all working with the above library and it's not too bad. Scaffolding seems to have broken however.
Now beyond all this, it now seems that this stuff is going to become probamatic when I am running in the live environment. Any schema changes I'm going to want to make between the dev db and live db will have to be manually managed with scripts anyway, so at that point am I not losing the point of code first?
I've been working with Google App Engine for the last year, and was hoping that code first would essentially work in the same way? Ie, make changes and they modify the live data. Now I assume, due to not having done severe refactoring in app engine, that it basically doesn't harm anything in production. So you could never rename a table with AppEngine. It would always create a new table, and leave the old one. You would have to manually port data.
So I'm now thinking. Why not just go Database first? I've been working with linq2sql for 3 years and very comfortable with going db first. Although TBH my db source control stratergy has been a little....lacking. So I was hoping code first would enforce that situation to improve, but it actually makes me feel that I should go DB first, and just be strict about keeping it under control.
I would really appreciate any thoughts on this kind of situation, and also, how does this compare to using Nhibinate?
The upgrade scenario's you're describing are being added in EF-Migrations. The go-live release of this is already available and it should become available as a officially supported release version soon.
Check out: https://blogs.msdn.com/b/adonet/archive/2012/02/09/ef-4-3-automatic-migrations-walkthrough.aspx?Redirected=true
Check out: http://coding.abel.nu/tag/ef-migrations/
Just my take on this...
You don't need to make a choice for good -
you can use Seed, custom initializer (and migrations hand in hand - with migrations you can actually manually change them, e.g. add indexes there etc. - though carefully), to 'inject' the raw SQL (and most of the needs could be covered by that). As far as I know you could run a custom batch file form initializer (but haven't tried, though I don't see why not). Migrations are done through PS so I'm guessing there is space to integrate on that side too.
You can use it to prototype things and startup - get your C# structure in place and in sync fast - then shape it up later - or once fixed use some 'use existing initializer' which does nothing from the code and move everything to Db side and your DBA.
IMO, you can go 'very far' in this sense, before actually needing to turn CF off and just work from Db.
For larger databases, or corporate environments, it's definitely not the right tool for the job.
I think it's now safe for production - but depends on what you consider safe, few things...
Also, EF/CF has gone a long way since the beginnings really - I used it since the start - and it had big problems initially. The latest versions are now pretty solid on all sides, so it's getting there I think.
...on the downside - I had worked out some pretty complex scenarios with code first approach (EF and otherwise). However, few things held me back with EF (on top of what you mentioned)...
UDF support, views (with linq - as if you're not to use LINQ one of the big 'pros' is gone) - which should come up in EF 5.0 (I did not check but it's what they're promissing) - as for more complex scenarios you need to optimize on the SQL side (or to be able to execute custom queries on the back and then map back, i.e. more flexibility on that side). So that's a promising thing if it comes (though won't be perfect I expect). (on the plus side, you can actually run custom queries and then map to your objects - but that sort of works, and you cannot do everything so it's limited in a way)
Performance with queries and again for more demanding heavy duty scenarios - that was my main problem with it. That's also promised to come up better in EF 5.0 - so to wait and see.
Having said that -
my favorite is a custom or open source code first like solution, ORM - where you can have more things under control. Still having many providers support etc. is what makes 'official' or MS implementations more viable on a long run.
hope this helps any
I have a system (using Entity Framework) that is deployed in various production systems and also on a quality control system. My problem is that data entry is often done only on one of those occurrences of my system (different databases).
I want to find the best way to transfer my data from one database to another database. Ids can change, as long as the relations between my objects are maintained. 98% of my data in in DB, some of it is external files, I can manage those separately, manually.
Currently we use a xml structure as a transition file. The file is then imported in the destination system, and code manually imports the entities and re-creates the data.
I'm looking for a more generic way to do this, with less code. Since all my data in stored in Entities couldn't I simply create a big List and throw all my objects in there, then serialize that in some matter into an external file and finally generically import all the entities in there in my destination system? (I'll probably have to be careful in maintaining relation ids, but should be ok...)
Anyways I'm wondering if anyone would have smart approaches, I'm pretty sure I,m not the first with a similar problem.
Thanks!
You need to get some process around this. If all environments contain the same data (unlikely) you can replicate. It is the most automatic. But a QA environ should not update production, so you have to really think this through.
If semi-automated is okay, there are tools out there you can use from a variety of vendors. I use Red Gate tools, personally, but others are also fine.
Can you set up a more automated push with EF? Sure, but the amount of time you spend is really not worth it.
In my opinion you can check some of the following approaches:
1) Use Sql Compare or Sql Data Compare. Those tools are from Red Gate and can be found here
2) Regular backups and restores of the databases. You could, if it is an option regularly backup your most up-to-date database and restore it on the destination systems. I have no experience in automatizing this but here is a link to do that through .net.
3) You could always give it a go creating a version control system of your own. I would picture one such system selecting all records from a certain table (or all of them), deleting all records in the target database and inserting them. This seems pretty complex though, as you have to worry about relationships, data dependencies, etc.
Hope this helps in some way.
Regards
If you for some reason will not be satisfied with existing tools may be you'll want take a look at the Sync Framework and implement this functionality yourself for your very particular data bases.
Given what you described, pushing data from One SQL Server to another for demo purposes, you should consider SQL Server Integration Services.
If you're got a simple scenario where you just move the data and objects from DB to the next you can use their built-in Wizards. If you need to do custom stuff you can build complex workflows using C# and SQL (tools you already know). Note: most of what you're going to want comes with the standard edition so if you're using express this is less interesting.
The story for Red Gate products is more compelling when you don't have SQL Server (So you have to go out and buy something) and if you are interested in finding out what the changes are between DB's (like viewing code changes in a .cs file in a source control product)