I realize this question has been asked before, but nothing I've read really answers my question.
I have a table with millions of rows of data that is used in multiple queries a day. I want to move the majority of the data to another table with the same schema. The second table will be an "archive" table.
I would like a list of options to archive data, so I can present them to my boss. So far I'm considering an insert into select statement, SQLBulkCopy in a C# console application, and I'm starting to dig in to SSIS to see what it can do. I plan on doing this over a weekend or multiple weekends.
The table has an ID as the primary key
The table also has a few foreign key constraints
Thanks for any help.
I assume that this is for SQL Server. In that case, partitioned tables might be an additional option. Otherwise I'd always go for a INSERT ... SELECT run by a job in SQL Server, or - if you can't run it directly in SQL Server - create a stored procedure and run it through a little C# tool that you schedule.
Try execute something like
CREATE TABLE mynewtable as select * from myoldtable where any_filter..;
You could create new table with data copy with one instruction on most database engines.
Use this, in case of SQL Server 2008
Select * into new_table from old_table
In the event that you have a set data archive interval, you may be able to leverage the partition-to-archive solution described in the following article.
http://blogs.msdn.com/b/felixmar/archive/2011/02/14/partitioning-amp-archiving-tables-in-sql-server-part-1-the-basics.aspx
Our team has leveraged a similar partition / archive solution in the past with good success.
Regards,
We use LINQ to SQL in our project. One of the tables is "Users" used in every action in the project.
Recently we were said to add "IsDeleted" column to the table and consider that column in every data fetching in LINQ to SQL queries.
We wouldn't want to add "WHERE IsDeleted = Fasle" to all queries.
Is it possible "to interrupt" to LINQ after the data was fetched but before sending further to code in the project?
This can be solved by C# but it would really be the wrong tool for the job.
Create a view in the database that includes this statement and only work with the view from now on. You can even enforce this by not granting privileges on the table any more.
I am using Entity Framework, C# 4.0 and Visual Studio 2010. I need to execure a simple SQL query to delete the content of four tables. The query contains:
DELETE FROM dbo.tblMFile
DELETE FROM dbo.tblMContacts
DELETE FROM dbo.tblPersonDetails
DELETE FROM dbo.tblAddresses
There is a foreign key contraint between some of the tables.
There seems to be no simple way to do this.
With reference to my comments on the first respondant's answer:
Because of the highly confidential nature of the data I NEED a way to quickly delete all content (a requirement and a security issues)
I am new to EF and keen to learn
I am using delete instead of truncate because of the foreign key constraints mentioned above (also the SQL above is illustrative, not definitive).
I am a firm and long time believer in strong typing and prefixes all my objects with a type indicator. It has saved me hours (or days even) of debugging.
Humans are polite, sensitive and informative (among many other attributes). I aspire to be human.
No, and that is good so. ORM's are not for bulk deletions. If you really want to use LINQ to issue a DELETE, use BlToolkit that allows you to express arbitrary pretty much standard DML non bulk (no truncate) in LINQ.
Delete ALL records?
TRUNCATE TABLE, NOT DELETE.
TRUNCATE TABLE tblMFile
Depending how much the table contains that can be thousands of times faster. Mostly because it does also not log the deleted data - just the fact that a truncate occurred.
To add to the answer from #TomTom. You can't do that directly using Entity Framework. But you can:
create a stored procedure to delete the records (it can also take the table name as parameter)
use ExecuteStoreQuery see see and write the SQL command directly
Try This
_db.ExecuteStoreCommand("delete from MachineMaster where IdMachine =" + obj_Machine.IdMachine + " or IdMasterMachine=" + obj_Machine.IdMachine );
I have a table with a lot of rows that is the products table, when an user searches the website, I make a select from this table and I also use the Include method in the select, I used profiler and I noticed that EF generates a query with a lot of inner joins, left joins etc.
What I wanted to do is to use this select query and insert the result in a temporary cache table in the same database and then I can create a service to update this table every x minutes.
The problem is, how do I make EF use this cache table to select the rows so I could just make a select * instead of every time query the products table with the joins?
thanks!
I'm pretty certain EF doesn't have temp tables support - at least out of the box - but it's constantly changing.
Your best bet is to do something like this...
dbcontext.Database.ExecuteSqlCommand("...")
...and I'm guessing there you could run an arbitrary SQL (I know most things can be passed in but I'm not sure about the limitations, but you could run a SP, create indexes etc.) - to set up a temp table.
Then the next step would be to do the opposite side something like this..
dbcontext.MyTable.SqlQuery("...").ToList()
...to map back the sql results into some entity of yours - or for a non-mapped entity to string or something. (dbcontext.MyTable.SqlQuery<T>("...").ToList())
The question is how to do it exactly - not sure of your specifics really. But you could create a temp table before hand and have it mapped - and use it for temp purposes.
Basically, that's a DBA thinking - but EF is not perfect for such things (see something similar here Recommed usage of temp table or table variable in Entity Framework 4. Update Performance Entity framework) but you might be ok with a custom ran queries like the above.
hope it helps
EDIT: this might also help from EF forums but it's more involving.
Every change of data in some row in database should save the previous row data in some kind of history so user can rollback to previous row data state. Is there any good practice for that approach? Tried with DataContract and serializing and deserializing data objects but it becomes little messy with complex objects.
So to be more clear:
I am using NHibernate for data access and want to stay out off database dependency (For testing using SQL server 2005)
What is my intention is to provide data history so every time user can rollback to some previous versions.
An example of usage would be the following:
I have a news article
Somebody make some changes to that article
Main editor see that this news has some typos
It decides to rollback to previous valid version (until the newest version is corrected)
I hope I gave you valid info.
Tables that store changes when the main table changes are called audit tables. You can do this multiple ways:
In the database using triggers: I would recommend this approach because then there is no way that data can change without a record being made. You have to account for 3 types of changes when you do this: Add, Delete, Update. Therefore you need trigger functionality that will work on all three.
Also remember that a transaction can modify multiple records at the same time, so you should work with the full set of modified records, not just the last record (as most people belatedly realize they did).
Control will not be returned to the calling program until the trigger execution is completed. So you should keep the code as light and as fast as possible.
In the middle layer using code: This approach will let you save changes to a different database and possibly take some load off the database. However, a SQL programmer running an UPDATE statement will completely bypass your middle layer and you will not have an audit trail.
Structure of the Audit Table
You will have the following columns:
Autonumber PK, TimeStamp, ActionType + All columns from your original table
and I have done this in the following ways in the past:
Table Structure:
Autonumber PK, TimeStamp, ActionType, TableName, OriginalTableStructureColumns
This structure will mean that you create one audit table per data table saved. The data save and reconstruction is fairly easy to do. I would recommend this approach.
Name Value Pair:
Autonumber PK, TimeStamp, ActionType, TableName, PKColumns, ColumnName, OldValue, NewValue
This structure will let you save any table, but you will have to create name value pairs for each column in your trigger. This is very generic, but expensive. You will also need to write some views to recreate the actual rows by unpivoting the data. This gets to be tedious and is not generally the method followed.
Microsoft have introduced new auditing capabilities into SQL Server 2008. Here's an article describing some of the capabilities and design goals which might help in whichever approach you choose.
MSDN - Auditing in SQL Server 2008
You can use triggers for that.
Here is one example.
AutoAudit is a SQL Server (2005, 2008)
Code-Gen utility that creates Audit
Trail Triggers with:
* Created, Modified, and RowVerwsion (incrementing INT) columns to table
* view to reconstruct deleted rows
* UDF to reconstruct Row History
* Schema Audit Trigger to track schema changes
* Re-code-gens triggers when Alter Table changes the table
http://autoaudit.codeplex.com/
Saving serialized data always gets messy in the end, you're right to stay away from that. The best thing to do is to create a parallel "version" table with the same columns as your main table.
For instance, if you have a table named "book", with columns "id", "name", "author", you could add a table named "book_version" with columns "id", "name", "author", "version_date", "version_user"
Each time you insert or update a record on table "book", your application will also insert into "book_version".
Depending on your database system and the way you database access from your application, you may be able to completely automate this (cfr the Versionable plugin in Doctrine)
One way is to use a DB which supports this natively, like HBase. I wouldn't normally suggest "Change your DB server to get this one feature," but since you don't specify a DB server in your question I'm presuming you mean this as open-ended, and native support in the server is one of the best implementations of this feature.
What database system are you using? If you're using an ACID (atomicity, consistency, isolation, durability) compliant database, can't you just use the inbuilt rollback facility to go back to a previous transaction?
I solved this problem very nice by using NHibernate.Enverse
For those intersted read this:
http://nhforge.org/blogs/nhibernate/archive/2010/07/05/nhibernate-auditing-v3-poor-man-s-envers.aspx