What is the easiest and most efficient way to insert data from a DataTable into a SQL Server database table? For what it's worth, the DataTable and the database table have matching columns.
What I am doing is transferring data from one database to another (separate instances, so I will be using my application as the intermediate "receiver/sender" of data). I can easily populate a DataTable with a SqlDataAdapater.Fill() call from the source database table. But I'm trying to find the most proficient way to send that DataTable's data to the final destination database table.
Any suggestions/advice/opinions are much appreciated.
EDIT: the destination database table already exists.
You should take a look at the SqlBulkCopy class, particularly the overload of the WriteToServer method that takes a DataTable as a parameter.
If you want to be even more efficient, and you don't have the requirement to materialize the entire table into a DataSet (or, you can process the contents as you move them in a forward-only manner), then use the overload of WriteToServer that takes an IDataReader run your query using the ExecuteReader method on the SqlCommand class instead of using a SqlDataAdapter to load the entire table into memory.
Related
I have json which the converted to Dataset. And this data I wanna written to table in SQl. But I don't know how this doing. I did tried doing through SqlBulkCopy but me this don't likes because that needed create other table. And data always different.Thanks.
Im trying to figure out how I can do bulk operations on a table in SQL datbase where I want to be able to handle failed rows.
Let's say I do a bulk insert of 10 rows into a table and row number 5 has some faulty data and can't be inserted. I then want the 9 other rows to be inserted but in some way be able to get the 1 row that coulden't be inserted in return so I can store this row in some place for later error handling.
It would be great if I could use some nuget package for C# programming.
Typically the most optimal solution is to validate bad data as early in the process as possible. Consider the following strategy.
Create a datatable in your c# code where the datacolumns precisely match the datatype, length, nullable, etc. of the target tables where the needs to be persisted. If any data does not conform it will throw an exception when attempting to insert into the datatable.
Populate the datatable from your source data, one data row at a time. If a row fails to insert into the datatable, your exception handling could catch that row and log that data as needed.
Create a custom table-valued type in SQL Server to match the datatable. Make sure everything matches including all of the meta data of each datacolumn.
Create a stored procedure on SQL Server that contains the table-values input parameter. This parameter will be the new custom type created in step 3. The table valued parameter can then be used to quickly insert the data into the database tables.
Pass your datatable as a parameter value to the stored procedure. The data type should be SqlDbType.Structured.
This design adds speed and efficiency with early input validation, while allowing the process to pass data in bulk, allowing for fast inserts.
I guess you can do a trigger instead of insert, check if the data is valid. If it is not you can save it in another table from where you will retrieve the rows with errors, if there are not errors in the row you can insert it in your original table.
I am using a huge dataset (>10M records * ~16k) on a locally-stored MySQL database.
The user will filter by, say, fields A/B/C, returning between 1-200k records. THIS query takes up to a minute
With THIS set of data, I want to do further analysis with SQL; i.e. dynamically change a further set of fields, say D/E/F, depending on settings in the UI, running these further queries only on the smaller dataset.
My question is - conceptually - how is it best in C#/MySQL to approach this;
Can I keep the original query 'open' on the MySQL server, and dynamically adjust that to suit?
Do I need to take the whole dataset from the original query into memory and then filter it further in C#?
Should I copy the relevant data into a temporary table, and perform queries on that table?
I have created one DataTable in C# dynamically. I want to insert this whole DataTable in MySql database table.
I am using MySql hence SqlBulkCopy is not an option for me.
I have tried MySqlBulkLoader but it seems it only insert data from file.
I can do it via iterating through every datarow of datatable and insert it to database.
Is there any another way I can achieve multiple inserts into database?
Thanks for any help!
If the number of rows are not much you can simply create an insert statement
INSERT INTO table VALUES (1,2,3), (4,5,6), (7,8,9);
Another article at http://www.codeproject.com/Articles/19727/ADO-NET-Generic-Copy-Table-Data-Function may help.
Though, if the number of rows are high, you can probably create a temporary text file and then use BulkLoader !
I know SQL Server 2000 has a bulk insert. Does it support bulk insert from a C# collection, such as a dataset?
I need to insert 30 rows at a time, fairly regularly. I don't want to create 30 DB connections for this if I don't have to.
Have a look at SqlBulkCopy (has according to forums SQL 2000 support). It's easy to use. Basically just provide it with a data table (or data reader) and it will copy the rows from that source to your destination table.
You can insert using a DataSet in SQL 2000, I've never tried because I never use DataSets.
http://www.dotnet247.com/247reference/msgs/3/16570.aspx has a good post on it:
(From the article)
Steps involved
1.Create SqlDataAdapter with proper select statement
2.Create dataset and fill dataset with SqlDataAdapter
3.Add rows to the table in the dataset (for all the actions you said above, like radio
button selected, check box enabled)
4.Use SqlCommandBuilder helper object to generate the
UpdateStatements. Its very easy to use command builder. Just a one
call to the SqlCommandBuilder constructor.
5.Once your are done adding rows to the datatable int the dataset call
SqlDataAdapter.update and pass the modified dataset as a parameter.
This should automatically add the rows from dataset to the
database.(if no database error occurs)
Have you considered XML?
Working with XML in SQL 2000 isn't as nice as in 2008, but it is still doable:
http://www.codeproject.com/KB/database/insxmldatasqlsvr.aspx
http://www.codeproject.com/KB/database/generic_OpenXml.aspx
http://support.microsoft.com/default.aspx?scid=kb;en-us;315968
Another option you could look at would be to:
Open Connection.
Iterate through the inserts
Close Connection.