Good day.
I am asking for a bit of advise on what other experience has been and pitfalls ect. I am a SQL developer but needing to write a front end using c#.
I am returning a query from a MSSQL database via a stored procedure and putting it into a Datatable. There are about 140k rows in the result set. I am using standard calls with a datareader to return the resultset. No binding.
What I would like to do is return parts of the datatable to datagrid on a form and allow a user to manipulate the data in the grid and save back to the datatable then collect the next part of the datatable and manipulate that. I don't want to to pull things into a datatable in segments as I need to update calculations on the entire datatable when a change is made.
And then finally save the changes back to the database, when done.
If anyone can point me to a the best and most efficient way it would be greatly appreciated.
Thank you in advance
Scott
Related
I'm new to n-tier enterprise development. I just got quite a tutorial just reading threw the 'questions that may already have your answer' but didn't find what I was looking for. I'm doing a geneology site that starts off with the first guy that came over on the boat, you click on his name and the grid gets populated with all his children, then click on one of his kids that has kids and the grid gets populated with his kids and so forth. Each record has an ID and a ParentID. When you choose any given person, the ID is stored and then used in a search for all records that match the ParentID which returns all the kids. The data is never changed (at least by the user) so I want to just do one database access, fill all fields into one datatable and then do a requery of it each time to get the records to display. In the DAL I put all the records into a List which, in the ObjectDataSource the function that fills the GridView just returns the List of all entries. What I want to do is requery the datatable, fill the list back up with the new query and display in the GridView. My code is in 3 files here
(I can't get the backticks to show my code in this window) All I need is to figure out how to make a new query on the existing DataTable and copy it to a new DataTable. Hope this explains it well enough.
[edit: It would be easier to just do a new query from the database each time and it would be less resource intensive (in the future if the database gets too large) to store in memory, but I just want to know if I can do it this way - that is, working from 1 copy of the entire table] Any ideas...
Your data represents a tree structure by nature.
A grid to display it may not be my first choice...
Querying all data in one query can be done by using a complex SP.
But you are already considering performance. Thats always a good thing to keep in mind when coming up with a design. But creating something, improve it and only then start to optimize seems a better to go.
Since relational databases are not real good on hierarchical data, consider a nosql (graph)database. As you mentioned there are almost no writes to the DB, nosql shines here.
I need to handle very large datatables (2 million rows+) that comes from databases (SQL, Oracle, Access, MySQL, Sharepoint etc) outside of my control: Currently I loop through every row and column building a string object, but I run out of memory at about 100k rows.
The only solution I may take is to break the datatable into smaller pieces and persisting each block before starting on the next block of rows.
Since I cannot add ROW_NUMBER() or anything similar, I have to handle the populated datatable.
How can I easily (keep performance in mind) break the populated datatable into smaller datatables like paging?
PS there is no visual component to this functionality.
Are you using string concatenation? like this string += string.
Change that to StringBuilder and you should not have problems, at least not for 20k rows.
If you are talking about filling a DataTable object (which loads the results of your calls into memory before processing), you will likely be better off using a datareader for each of the mentioned providers so then you can process each row as it is read from the database instead of storing the DataTable in memory...
A great answer to another question lists the pro/cons of datareaders/datatables
If you're already using datareaders- ignore this. But your memory problem might be from also storing the retrieved results...
What is the best method for saving thousands of rows and after doing something, updating them.
Currently, I use a datatable, filling it, when done inserting by
MyDataAdapter.Update(MyDataTable)
After doing some change on MyDataTable, I again use MyDataAdapter.Update(MyDataTable) method.
Edit:
I am sorry for not providing more info.
There may be up to 200.000 rows which will be created from an XML file. There rows will be saved to the database. After than there will be some process for each row. And I will need to update each row in database.
Instead of updating row by row, I decided to update the datatable and using the same dataadapter to update the rows.
This is the best of me.
I think that there may be a smarter approach.
In Reacting to your comments:
An DataAdapter.Update() will Udate (and Insert/Delete) row by row. If you have individual changes there really is no faster way. If you have systematic changes, like SET Price = Price+ 2 WHERE SelByDate < '1/1/2010' you are better of by running a DbCommand against the database.
But maybe you should worry about transactions and error handling before performance.
If I understand correctly you are doing two separate operations: loading rows to a database, and then updating those rows.
If the rows you are inserting come from another ADO.NET supported datasource then you can use SqlBulkCopy to insert the rows in batches, which will be more efficient than using a datatable.
Once the rows are in the database I would assume you would be better off executing a SQLCommand to modify their values.
If you can provide more details about what--and why--you're asking the question then perhaps we can better tailor an answer for it.
Suppose I have an ADO.NET DataTable that I was to 'persist' by saving it to a new table in a SQL Server database - is there a fast way of doing this?
I realise I could write code generating the DDL for the 'CREATE TABLE' statement by looping through the DataColumns collection and working out the right type mappings and so on ... but I'm wondering if there is an existing method to do this, or a framework someone has written?
(NB: I need to be able to handle arbitrary columns, nothing too fancy like blobs; just common column types like strings, numbers, guids and dates. The program won't know what the columns in the DataTable are until run-time so they can't be hard-coded.)
ADO.net cannot create tables in SQL Server directly, however, SMO can do this with the .Create method of the Table class. Unfortunately, there is no built-in way to use a DataTable to define an SMO Table object.
Fortunately, Nick Tompson wrote just such a DataTable-to-SMO.Table routine back in 2006. It is posted as one of the replies to this MSDN forums topic http://social.msdn.microsoft.com/forums/en-US/adodotnetdataproviders/thread/4929a0a8-0137-45f6-86e8-d11e220048c3/ (edit: I can make hyperlinks now).
Note also, the reply post that shows how to add SQLBulkCopy to it.
If the table exists, you can use SqlBulkCopy (which will accept a DataTable) to get the data into the table in the fastest possible way (much faster than via an adapter). I don't think it will create the table though. You might have to write the DDL yourself, or find some existing code to loop over the DataTable.Columns to do it.
I think this post can help
I might be easier to store your DataTable as XML. just create a table with an XML column.
Is there any way I can push a new record to SqlDataReader after i pull a table down? I have this piece of trash code that I need to modify, and this seems like the easiest way to do what I need. I understand that this should not be done, and if you have to do it there is something seriously wrong with your logic, but is there a way?
Easiest way from that point is to just manually create a command with a command string of an insert(parametized if not sanitized/clean data, best to do that anyways, but could make code bulkier). Code for that should be quite small, considering you already have everything else setup.
When you say "push a new record to"... do you mean you want to add a record to the results? Or do you mean you want to do an INSERT?
The INSERT cannot be done with a reader; however, you can do things with readers. Of course, it would be simpler to update the original query so that you UNION the data.
In particular: you can't create your own SqlDataReader, but you can create your own bespoke IDataReader implementation; this could wrap the SqlDataReader, simply proxying data from the inner SqlDataReader until the SqlDataReader.Read() method returns false - then you could swap to returning you own data, returning true until you have run out of data. Not trivial to implement (mainly because you need to implement a lot of methods to write your own IDataReader), but certainly not impossible.
SqlDataReaders are forward read only so I doubt you can add a record in (regardless of whether you have pulled the whole table down). In fact anything that inherits DbDataReader is forward read only.
I'm guessing you need to do some manipulation with the records. Maybe what you can do instead is use the SqlDataReader to fill a DataTable and put a new record into the DataTable. But then you'd need to change your code to juggle a DataTable.
You need to expand your question a bit,
If, for example, you need to walk through a million records and update a field on the same table while walking through the data.
You can create a second SqlConnection to your db and execute update statements on the table (prone to locking issues), or better still insert all your changes into a temp table and merge the changes back into the original table after you are done with the reader.
There is little question I am tempted to ask, can this piece of logic be replaced with a single SQL UPDATE statement?