What is the fastest way of transferring few thousand rows of data from one DataTable to another? Would be great to see some sample code snippets.
Edit: I need to explain a bit more. There is a filtering condition for copying the rows. So, a plain Copy() will not work.
You can't copy the whole table, you need to copy one rows. From http://support.microsoft.com/kb/308909 (sample code if you follow the link)
"How to Copy DataRows Between DataTables
Before you use the ImportRow method, you must ensure that the target table has the identical structure as the source table. This sample uses the Clone method of DataTable class to copy the structure of the DataTable, including all DataTable schemas, relations, and constraints.
This sample uses the Products table that is included with the Microsoft SQL Server Northwind database. The first five rows are copied from the Products table to another table that is created in memory."
What is wrong with DataTable.Copy?
Copying rows to a table throws some flags at me. I've seen people try this before, and in every single case what they really wanted was a System.Data.DataView. You really should check to see if the RowFilter property will do what you need it to do.
Related
I need to handle very large datatables (2 million rows+) that comes from databases (SQL, Oracle, Access, MySQL, Sharepoint etc) outside of my control: Currently I loop through every row and column building a string object, but I run out of memory at about 100k rows.
The only solution I may take is to break the datatable into smaller pieces and persisting each block before starting on the next block of rows.
Since I cannot add ROW_NUMBER() or anything similar, I have to handle the populated datatable.
How can I easily (keep performance in mind) break the populated datatable into smaller datatables like paging?
PS there is no visual component to this functionality.
Are you using string concatenation? like this string += string.
Change that to StringBuilder and you should not have problems, at least not for 20k rows.
If you are talking about filling a DataTable object (which loads the results of your calls into memory before processing), you will likely be better off using a datareader for each of the mentioned providers so then you can process each row as it is read from the database instead of storing the DataTable in memory...
A great answer to another question lists the pro/cons of datareaders/datatables
If you're already using datareaders- ignore this. But your memory problem might be from also storing the retrieved results...
I am stucked up at this point. I searched alot on gooogle but didnot find anything.
My problem is:
I have an Excel file which i want to export to datatable and from datatable i want to save it to oracle DB.
Excel file contains multiple columns and each column consists of large data(approx 20000characters/numbers).
using oledbconnection,excel columns with such large data are not copied to datatble.(Small data columns gets copied).
Can anyone suggest workaround to my problem???
Thanks in advance.
Check the dataypes and their length i.e nvarchar(3000)
If that doesnt work, test with a small set of data maybe 5 rows, you should be able to see a trend there.
Also check the data type of your application, sometimes for large number you may use long, or if they are large strings maybe use a stringbuilder to pass the data instead of just a string....
Can someone explain me DataSet.Copy() vs Dataset.Clone()
Also let me know some scenario's where we can use these
Clone will create a new, empty DataSet with the same schema (tables and columns) as the old one. The new DataSet will not have any data.
Copy does the same thing, but also copies the rows in the tables.
Clone copies the only the structure of the dataset where as Copy copies the data as well.
http://www.dotnetspider.com/forum/ViewForum.aspx?ForumId=52327
The Clone() method creates a new DataSet with the same structure. The Copy() method does too, and copies the data.
DataSet.Copy copies both the structure (tables, relations etc) and the data.
DataSet.Clone copies only the structure of the DataSet.
.Clone returns a data set with exactly the same schema but without any rows, .Copy returns both the schema and the rows.
What is the best method for saving thousands of rows and after doing something, updating them.
Currently, I use a datatable, filling it, when done inserting by
MyDataAdapter.Update(MyDataTable)
After doing some change on MyDataTable, I again use MyDataAdapter.Update(MyDataTable) method.
Edit:
I am sorry for not providing more info.
There may be up to 200.000 rows which will be created from an XML file. There rows will be saved to the database. After than there will be some process for each row. And I will need to update each row in database.
Instead of updating row by row, I decided to update the datatable and using the same dataadapter to update the rows.
This is the best of me.
I think that there may be a smarter approach.
In Reacting to your comments:
An DataAdapter.Update() will Udate (and Insert/Delete) row by row. If you have individual changes there really is no faster way. If you have systematic changes, like SET Price = Price+ 2 WHERE SelByDate < '1/1/2010' you are better of by running a DbCommand against the database.
But maybe you should worry about transactions and error handling before performance.
If I understand correctly you are doing two separate operations: loading rows to a database, and then updating those rows.
If the rows you are inserting come from another ADO.NET supported datasource then you can use SqlBulkCopy to insert the rows in batches, which will be more efficient than using a datatable.
Once the rows are in the database I would assume you would be better off executing a SQLCommand to modify their values.
If you can provide more details about what--and why--you're asking the question then perhaps we can better tailor an answer for it.
Suppose I have an ADO.NET DataTable that I was to 'persist' by saving it to a new table in a SQL Server database - is there a fast way of doing this?
I realise I could write code generating the DDL for the 'CREATE TABLE' statement by looping through the DataColumns collection and working out the right type mappings and so on ... but I'm wondering if there is an existing method to do this, or a framework someone has written?
(NB: I need to be able to handle arbitrary columns, nothing too fancy like blobs; just common column types like strings, numbers, guids and dates. The program won't know what the columns in the DataTable are until run-time so they can't be hard-coded.)
ADO.net cannot create tables in SQL Server directly, however, SMO can do this with the .Create method of the Table class. Unfortunately, there is no built-in way to use a DataTable to define an SMO Table object.
Fortunately, Nick Tompson wrote just such a DataTable-to-SMO.Table routine back in 2006. It is posted as one of the replies to this MSDN forums topic http://social.msdn.microsoft.com/forums/en-US/adodotnetdataproviders/thread/4929a0a8-0137-45f6-86e8-d11e220048c3/ (edit: I can make hyperlinks now).
Note also, the reply post that shows how to add SQLBulkCopy to it.
If the table exists, you can use SqlBulkCopy (which will accept a DataTable) to get the data into the table in the fastest possible way (much faster than via an adapter). I don't think it will create the table though. You might have to write the DDL yourself, or find some existing code to loop over the DataTable.Columns to do it.
I think this post can help
I might be easier to store your DataTable as XML. just create a table with an XML column.