What is the best method for saving thousands of rows and after doing something, updating them.
Currently, I use a datatable, filling it, when done inserting by
MyDataAdapter.Update(MyDataTable)
After doing some change on MyDataTable, I again use MyDataAdapter.Update(MyDataTable) method.
Edit:
I am sorry for not providing more info.
There may be up to 200.000 rows which will be created from an XML file. There rows will be saved to the database. After than there will be some process for each row. And I will need to update each row in database.
Instead of updating row by row, I decided to update the datatable and using the same dataadapter to update the rows.
This is the best of me.
I think that there may be a smarter approach.
In Reacting to your comments:
An DataAdapter.Update() will Udate (and Insert/Delete) row by row. If you have individual changes there really is no faster way. If you have systematic changes, like SET Price = Price+ 2 WHERE SelByDate < '1/1/2010' you are better of by running a DbCommand against the database.
But maybe you should worry about transactions and error handling before performance.
If I understand correctly you are doing two separate operations: loading rows to a database, and then updating those rows.
If the rows you are inserting come from another ADO.NET supported datasource then you can use SqlBulkCopy to insert the rows in batches, which will be more efficient than using a datatable.
Once the rows are in the database I would assume you would be better off executing a SQLCommand to modify their values.
If you can provide more details about what--and why--you're asking the question then perhaps we can better tailor an answer for it.
Related
Good day.
I am asking for a bit of advise on what other experience has been and pitfalls ect. I am a SQL developer but needing to write a front end using c#.
I am returning a query from a MSSQL database via a stored procedure and putting it into a Datatable. There are about 140k rows in the result set. I am using standard calls with a datareader to return the resultset. No binding.
What I would like to do is return parts of the datatable to datagrid on a form and allow a user to manipulate the data in the grid and save back to the datatable then collect the next part of the datatable and manipulate that. I don't want to to pull things into a datatable in segments as I need to update calculations on the entire datatable when a change is made.
And then finally save the changes back to the database, when done.
If anyone can point me to a the best and most efficient way it would be greatly appreciated.
Thank you in advance
Scott
I am working with a webservice that accepts a ADO.NET DataSet. The webservice will reject a submission if over 1000 rows are changed across all of the ten or so tables in the dataset.
I need to take my dataset and break it apart into chunks of less than 1000 changed rows. I can use DataSet.GetChanges() to produce a reduced dataset, but that still may exceed the changed row limit. Often a single table will have more than 1000 changes.
Right now, I think I need to: Create an empty copy of the dataset
Iterate over the DataTableCollection and .Add rows individually to the new tables until I get to the limit. Start a new dataset, and repeat until I've gone through everything.
Am I missing a simpler approach to this?
This is asking for trouble. Often changes to one table are dependent on changes to another. You don't want to split those up or bad things which may be difficult to debug problems will occur unless you are very very careful about this. Most likely, the "right" thing to do here is to submit changes to the webservice on a more frequent basis instead of batching them up so much.
I need to handle very large datatables (2 million rows+) that comes from databases (SQL, Oracle, Access, MySQL, Sharepoint etc) outside of my control: Currently I loop through every row and column building a string object, but I run out of memory at about 100k rows.
The only solution I may take is to break the datatable into smaller pieces and persisting each block before starting on the next block of rows.
Since I cannot add ROW_NUMBER() or anything similar, I have to handle the populated datatable.
How can I easily (keep performance in mind) break the populated datatable into smaller datatables like paging?
PS there is no visual component to this functionality.
Are you using string concatenation? like this string += string.
Change that to StringBuilder and you should not have problems, at least not for 20k rows.
If you are talking about filling a DataTable object (which loads the results of your calls into memory before processing), you will likely be better off using a datareader for each of the mentioned providers so then you can process each row as it is read from the database instead of storing the DataTable in memory...
A great answer to another question lists the pro/cons of datareaders/datatables
If you're already using datareaders- ignore this. But your memory problem might be from also storing the retrieved results...
I have two mirrored datatables (same structure with two primary keys) :
DataTable_A ---> bound to a datagridView
DataTable_B ---> filled from a database
Since DataTable_B is filled by a query into database every 2 seconds, I need to mirror the DataTable_A like DataTable_B avoiding filling directly DataTable_A. When a record disappears from DataTable_B i need to delete the record also from DataTable_A. What is the best way to do this ?
Right now I am doing a "for cycle" on each row of DataTable_B and if the row doesn't exist on DataTable_A, I delete it.
Is there a better way to do it ?
The best way may be not to have a TableA at all but use a DataView on TableB. That would solve all problems at once. Can you elaborate on why you need the copy?
But otherwise you would want to handle the RowChanged and TableNewRow RowDeleted event of TableB
A more general idea, after seeing your comments: If it is possible to add a Timestamp column to the table in the database you can run a much more efficient query. And the DataTable.Merge method would do the rest.
My goal is to maximise performance. The basics of the scenario are:
I read some data from SQL Server 2005 into a DataTable (1000 records x 10 columns)
I do some processing in .NET of the data, all records have at least 1 field changed in the DataTable, but potentially all 10 fields could be changed
I also add some new records in to the DataTable
I do a SqlDataAdapter.Update(myDataTable.GetChanges()) to persist the updates (an inserts) back to the db using a InsertCommand and UpdateCommand I defined at the start
Assume table being updated contains 10s of millions of records
This is fine. However, if a row has changed in the DataTable then ALL columns for that record are updated in the database even if only 1 out of 9 columns has actually changed value. This means unnecessary work, particularly if indexes are involved. I don't believe SQL Server optimises this scenario?
I think, if I was able to only update the columns that had actually changed for any given record, that I should see a noticeable performance improvement (esp. as cumulatively I will be dealing with millions of rows).
I found this article: http://netcode.ru/dotnet/?lang=&katID=30&skatID=253&artID=6635
But don't like the idea of doing multiple UPDATEs within the sproc.
Short of creating individual UPDATE statements for each changed DataRow and then firing them in somehow in a batch, I'm looking for other people's experiences/suggestions.
(Please assume I can't use triggers)
Thanks in advance
Edit: Any way to get SqlDataAdapter to send UPDATE statements specific to each changed DataRow (only to update the actual changed columns in that row) rather than giving a general .UpdateCommand that updates all columns?
Isn't it possible to implement your own IDataAdapter where you implement this functionality ?
Offcourse, the DataAdapter only fires the correct SqlCommand, which is determined by the RowState of each DataRow.
So, this means that you would have to generate the SQL command that has to be executed for each situation ...
But, I wonder if it is worth the effort. How much performance will you gain ?
I think that - if it is really necessary - I would disable all my indexes and constraints, do the update using the regular SqlDataAdapter, and afterwards enable the indexes and constraints.
you might try is do create an XML of your changed dataset, pass it as a parameter ot a sproc and the do a single update by using sql nodes() function to translate the xml into a tabular form.
you should never try to update a clustered index. if you do it's time to rethink your db schema.
I would VERY much suggest that you do this with a stored procedure.
Lets say that you have 10 million records you have to update. And lets say that each record has 100 bytes (for 10 columns this could be too small, but lets be conservative). This amounts to cca 100 MB of data that must be transferred from database (network traffic), stored in memory and than returned to database in form of UPDATE or INSERT that are much more verbose for transfer to database.
I expect that SP would perform much better.
Than again you could divide you work into smaller SP (that are called from main SP) that would update just the necessary fields and that way gain additional performance.
Disabling indexes/constraints is also an option.
EDIT:
Another thing you must consider is potential number of different update statements. In case of 10 fields per row any field could stay the same or change. So if you construct your UPDATE statement to reflect this you could potentially get 10^2 = 1024 different UPDATE statements and any of those must be parsed by SQL Server, execution plan calculated and parsed statement stored in some area. There is a price to do this.