Develop Reference

DataTable update() inserts duplicate new rows without checking if it exists

I'm trying to use the update() method, but it is inserting my datatable data into my database without checking if the row exists, so it is inserting duplicate data. It is also not deleting rows that don't exist in datatable. How to resolve this? I want to synchronize my datatable with server table.
private void Form1_Load(object sender, EventArgs e)
{
// TODO: This line of code loads data into the 'MyDatabaseDataSet11.Vendor_GUI_Test_Data' table. You can move, or remove it, as needed.
this.vendor_GUI_Test_DataTableAdapter.Fill(this.MyDatabaseDataSet11.Vendor_GUI_Test_Data);
// read target table on SQL Server and store in a tabledata var
this.ServerDataTable = this.MyDatabaseDataSet11.Vendor_GUI_Test_Data;
}
Insertion
private void convertGUIToTableFormat()
{
ServerDataTable.Rows.Clear();
// loop through GUIDataTable rows
for (int i = 0; i < GUIDataTable.Rows.Count; i++)
{
String guiKEY = (String)GUIDataTable.Rows[i][0] + "," + (String)GUIDataTable.Rows[i][8] + "," + (String)GUIDataTable.Rows[i][9];
//Console.WriteLine("guiKey: " + guiKEY);
// loop through every DOW value, make a new row for every true
for(int d = 1; d < 8; d++)
{
if ((bool)GUIDataTable.Rows[i][d] == true)
{
DataRow toInsert = ServerDataTable.NewRow();
toInsert[0] = GUIDataTable.Rows[i][0];
toInsert[1] = d + "";
toInsert[2] = GUIDataTable.Rows[i][8];
toInsert[3] = GUIDataTable.Rows[i][9];
ServerDataTable.Rows.InsertAt(toInsert, 0);
//printDataRow(toInsert);
//Console.WriteLine("---------------");
}
}
}
Trying to update
// I got this adapter from datagridview, casting my datatable to their format
CSharpFirstGUIWinForms.MyDatabaseDataSet1.Vendor_GUI_Test_DataDataTable DT = (CSharpFirstGUIWinForms.MyDatabaseDataSet1.Vendor_GUI_Test_DataDataTable)ServerDataTable;
DT.PrimaryKey = new DataColumn[] { DT.Columns["Vendor"], DT.Columns["DOW"], DT.Columns["LeadTime"], DT.Columns["DemandPeriod"] };
this.vendor_GUI_Test_DataTableAdapter.Update(DT);

Let's look at what happens in the code posted.
First this line:
this.ServerDataTable = this.MyDatabaseDataSet11.Vendor_GUI_Test_Data;
This is not a copy, but just an assignment between two variables. The assigned one (ServerDataTable) receives the 'reference' to the memory area where the data coming from the database has been stored. So these two variables 'point' to the same memory area. Whatever you do with one affects what the other sees.
Now look at this line:
ServerDataTable.Rows.Clear();
Uh! Why? You are clearing the memory area where the data loaded from the database were. Now the Datatable is empty and no records (DataRow) are present there.
Let's look at what happen inside the loop
DataRow toInsert = ServerDataTable.NewRow();
A new DataRow has been created, now every DataRow has a property called RowState and when you create a new row this property has the default value of DataRowState.Detached, but when you add the row inside the DataRow collection with
ServerDataTable.Rows.InsertAt(toInsert, 0);
then the DataRow.RowState property becomes DataRowState.Added.
At this point the missing information is how a TableAdapter behaves when you call Update. The adapter needs to build the appropriate INSERT/UPDATE/DELETE sql command to update the database. And what is the information used to choose the proper sql command? Indeed, it looks at the RowState property and it sees that all your rows are in the Added state. So it chooses the INSERT command for your table and barring any duplicate key violation you will end in your table with duplicate records.
What should you do to resolve the problem? Well the first thing is to remove the line that clears the memory from the data loaded, then, instead of calling always InsertAt you should first look if you have already the row in memory. You could do this using the DataTable.Select method. This method requires a string like it is a WHERE statement and you should use some value for the primarykey of your table
var rows = ServerDataTable.Select("PrimaryKeyFieldName = " + valueToSearchFor);
if you get a rows count bigger than zero then you can use the first row returned and update the existing values with your changes, if there is no row matching the condition then you can use the InsertAt like you are doing it now.

You're trying too hard, I think, and you're unfortunately getting nearly everything wrong
// read target table on SQL Server and store in a tabledata var
this.ServerDataTable = this.MyDatabaseDataSet11.Vendor_GUI_Test_Data;
No, this line of code doesn't do anything at all with the database, it just assigns an existing datatable to a property called ServerDataTable.
for (int i = 0; i < GUIDataTable.Rows.Count; i++)
It isn't clear if GUIDataTable is strongly or weakly typed, but if it's strong (I.e. it lives in your dataset, or is of a type that is a part of your dataset) you will do yourself massive favors if you do not access it's Rows collection at all. The way to access a strongly typed datatable is as if it were an array
myStronglyTypedTable[2] //yes, third row
myStronglyTypedTable.Rows[2] //no, do not do this- you end up with a base type DataRow that is massively harder to work with
Then we have..
DataRow toInsert = ServerDataTable.NewRow();
Again, don't do this.. you're working with strongly typed datatables. This makes your life easy:
var r = MyDatabaseDataSet11.Vendor_GUI_Test_Data.NewVendor_GUI_Test_DataRow();
Because now you can refer to everything by name and type, not numerical index and object:
r.Total = r.Quantity * r.Price; //yes
toInsert["Ttoal"] = (int)toInsert["Quantity"] * (double)toInsert["Price"]; //no. Messy, hard work, "stringly" typed, casting galore, no intellisense.. The typo was deliberate btw
You can also easily add data to a typed datatable like:
MyPersonDatatable.AddPersonRow("John, "smith", 29, "New York");
Next up..
// I got this adapter from datagridview, casting my datatable to their format
CSharpFirstGUIWinForms.MyDatabaseDataSet1.Vendor_GUI_Test_DataDataTable DT = (CSharpFirstGUIWinForms.MyDatabaseDataSet1.Vendor_GUI_Test_DataDataTable)ServerDataTable;
DT.PrimaryKey = new DataColumn[] { DT.Columns["Vendor"], DT.Columns["DOW"], DT.Columns["LeadTime"], DT.Columns["DemandPeriod"] };
this.vendor_GUI_Test_DataTableAdapter.Update(DT);
Need to straighten out the concepts and terminology in your mind here.. that is not an adapter, it didn't come from a datagridview, grid views never provide adapters, your datatable variable was always their format and if you typed it as DataTable ServerDataTable then that just makes it massively harder to work with, in the same way that saying object o = new Person() - now you have to cast o every time you want to do nearly anything Person specific with it. You could always declare all your variables in every program, as type object.. but you don't.. Hence don't do the equivalent by putting your strongly typed datatables inside DataTable typed variables because you're just hiding away the very things that make them useful and easy to work with
If you download rows from a database into a datatable, and you want to...
... delete them from the db, then call Delete on them in the datatable
... update them in the db, then set new values on the existing rows in the datatable
... insert more rows into the db alongside the existing rows, then add more rows to the datatable
Datatables track what you do to their rows. If you clear a datatable it doesn't mark every row as deleted, it just jettisons the rows. No db side rows will be affected. If you delete rows then they gain a rowstate of deleted and a delete query will fire when you call adapter.Update
Modify rows to cause an update to fire. Add new rows for insert
As Steve noted, you jettisoned all the rows, added new ones, added (probably uselessly) a primary key(the strongly typed table will likely have already had this key) which doesn't mean that the new rows are automatically associated to the old/doesn't cause them to be updated, hen inserted a load of new rows and wrote them to the db. This process was never going to update or delete anything
The way this is supposed to work is, you download rows, you see them in the grid, you add some, you change some, you delete some, you hit the save button. Behind the scenes the grid just poked some new rows into the datatable, marked some as deleted, changed others. It didn't go to the huge (and unfortunately incorrect) lengths your code went to. If you want your code to behave the same you follow the same idea:
var pta = new PersonTableAdapter();
var pdt = pta.GetData(); //query that returns all rows
pta.Fill(somedataset.Person); //or can do this
pdt = somedataset.Person; //alias of Person table
var p = pdt.FindByPersonId(123); //PersonId is the primary key in the datatable
p.Delete(); //mark person 123 as deleted
p = pdt.First(r => r.Name = "Joe"); //LINQ just works on strongly typed datatables, out of the box, no messing
p.Name = "John"; //modify joes name to John
pdt.AddPersonRow("Jane", 22);
pta.Update(pdt); //saves changes(delete 123, rename joe, add Jane) to db
What you need to appreciate is that all these commands are just finding or creating datarow obj3cts, that live inside a table.. the table tracks what you do and the adapter uses appropriate sql to send changes to the db.. if you wanted to mark all rows in a datatable as deleted you can visit each of them and call Delete() on it, then update the datatable to save the changes to the db

Optimizing query that uses AsEnumerable and SingleOrDefault

Not long ago there was a feature request in the program I am maintaining. Basically it has to fill up a table in the database with info from a text file. These files can be pretty big, but it was fairly easy to do because these files were defined as the complete list of user data. Therefore the table could be truncated and the just filled up again with data from the text file.
But then a week ago it was decided that these files are actually updates of current user info, so now I have to retrieve the correct MeteringPointId (which only exist once if it does exist) and then update info on it. If it doesn't exist, just insert data as before.
The way I do this is retrieving the complete database table with data from the database into memory and then just updating on that info before finally saving the changes by calling the datatables update function. It works fine, except that finding the row with the MeteringPointId is slow:
DataRow row = MeteringPointsDataTable.NewRow();
// this is called for each line in the text file to find the corresponding MeteringPointId. It can be 300.000 times.
row = MeteringPointsDataTable.AsEnumerable().SingleOrDefault(r => r.Field<string>("MeteringPointId").ToString() == MeteringPointId);
Is there a way to retrieve a DataRow from a DataTable that is faster than this?

If you are sure that only one item con fullfil the condition use FirstOrDefault instead of Single. Thus you won´t collect the whole table but only the first entry you´ve found.

You can use Select method of DataTable.
var expression = "[MeteringPointId] = '" + MeteringPointId + "'";
DataRow[] result = MeteringPointsDataTable.Select(expression);
Also you can create an expression like,
var idList = new []{"id1", "id2", "id3", ...};
var expression = "[MeteringPointId] in " + string.Format("({0})", string.Join(",", idList.Select(i=> "'"+i+"'")));
Similar usage is here
Hope it helps..

You could put the whole table in a dictionary:
//At the start
var meteringPoints = MeteringPointsDataTable.AsEnumerable().ToDictionary(r => r.Field<string>("MeteringPointId").ToString());
//For each row of the text file:
DataRow row;
if (!meteringPoints.TryGetValue(MeteringPointId, out row))
{
row = MeteringPointsDataTable.NewRow();
meteringPoints[MeteringPointId] = row;
}

Querying data from SQL

I need to query a table from database which has 400 rows and 24 columns. I need to query this table so that on each row and then on each column of row I can perform some C# code ( I can use column information to execute some code).
Now at the moment I am querying each row again and again from table using select statement and storing to a custom list and performing custom operations on it.
Is it best and fastest way of doing it ? or should I just query the whole table one time and store somewhere ? not sure where in a dataset and then run throw custom code to do some operation using information in each row ?

You can fetch the table from database once and store it in datatable and then just use linq to select the column something like this
var data = dt.AsEnumerable().Select(s => s.Field<string>("myColumnName")).ToArray<string>();
and If you don't want to use other columns anywhere in your code then you should select only useful column from the database.
You can also select multiple columns of a database using linq. The values will be stored in anonymous type of object.
var mutipleData = from row
in dt.AsEnumerable()
select new
{ Value1 = row["Column1"].ToString(),
Value2 = row["Column2"].ToString()
};

Assuming that each field is 1000 bytes, the total memory to hold your 400 rows would be 9.6MB. Peanuts! Just read the whole table in a DataTable and process it as you wish.

Select all the records from DB table which are f your concern
Copy the select records into DataTable.
Pseudo Code:
--dt is datatable
foreach(datarow dr in dt.rows)
{
--perform operation
string str=dr["columnname"].tostring
}

400 rows isn't a massive amount, however it depends on the data in each column and how often you are likely to run the query. If all you are going to do is run the query and manipulate the output, use a DataReader instead.

If it's only 400 records, I'd fetch them all at once, store them in a class and iterate over each instance.
Something like:
Class
public class MyTableClass(){
string property1 {get; set;}
string property2 {get; set;}
int property3 {get; set;}
// etc.
}
Logic:
ICollection<MyTableClass> lstMyTableClass = (
from m in db.mytableclass
select m
).ToList();
return lstMyTableClass;
And then a loop:
foreach(var myTableInstance in lstMyTableClass){
myTableInstance.DoMyStuff();
}

If the number of records will always be below thoussand, just query all the records and keep it in a List.
Once the data is in the List you can query the List n number of times using LINQ without hitting the database for each request.

Querying inside a Dataset C#

I have an ADO.NET dataset which is set by a certain query,
say
SELECT ID,USER,PRODUCT,COUNT FROM PRODUCTION
Without using a where clause I need to derive some results from the dataset. Say I want to get the User and Product count of the user who has the maximum product count. (And I want to do it by using the existing dataset. I can't derive this from dataset.)
Any idea of a way to query inside the dataset? Since there are Datatables my thought was there is some way to query it.

Traditional SQL queries cannot be applied to the DataSet. The following is possible, however:
Filter rows using DataTable.Select. See here for detailed information about expressions in DataTables.
Calculate totals etc. using DataTable.Compute.
If these two don't do the trick, there's always LINQ.
Quick-and-dirty LINQ example: (which doesn't return a DataTable, but a list containing an anonymous type):
var joinedResult = dataTable1
// filtering:
.Select("MyColumn = 'value'")
// joining tables:
.Join(
dataTable2.AsEnumerable(),
row => row.Field<long>("PrimaryKeyField"),
row => row.Field<long?>("ForeignKeyField"),
// selecting a custom result:
(row1, row2) => new { AnotherColumn = row1.Field<string>("AnotherColumn") });
AsEnumerable converts a DataTable into an IEnumerable on which LINQ queries can be performed. If you are new to LINQ, check out this introduction.

Yes, you can use DataTable.Select method.
DataTable table = DataSet1.Tables["Orders"];
// Presuming the DataTable has a column named Date.
string expression;
expression = "Date > #1/1/00#";
DataRow[] foundRows;
// Use the Select method to find all rows matching the filter.
foundRows = table.Select(expression);
// Print column 0 of each returned row.
for(int i = 0; i < foundRows.Length; i ++)
{
Console.WriteLine(foundRows[i][0]);
}
Also see this link.

You can do cross-table queries of a Dataset object using LINQ to DataSet:
msdn.microsoft.com/en-us/library/bb386969.aspx

What is the best way to fast insert SQL data and dependant rows?

I need to write some code to insert around 3 million rows of data.
At the same time I need to insert the same number of companion rows.
I.e. schema looks like this:
Item
- Id
- Title
Property
- Id
- FK_Item
- Value
My first attempt was something vaguely like this:
BaseDataContext db = new BaseDataContext();
foreach (var value in values)
{
Item i = new Item() { Title = value["title"]};
ItemProperty ip = new ItemProperty() { Item = i, Value = value["value"]};
db.Items.InsertOnSubmit(i);
db.ItemProperties.InsertOnSubmit(ip);
}
db.SubmitChanges();
Obviously this was terribly slow so I'm now using something like this:
BaseDataContext db = new BaseDataContext();
DataTable dt = new DataTable("Item");
dt.Columns.Add("Title", typeof(string));
foreach (var value in values)
{
DataRow item = dt.NewRow();
item["Title"] = value["title"];
dt.Rows.Add(item);
}
using (System.Data.SqlClient.SqlBulkCopy sb = new System.Data.SqlClient.SqlBulkCopy(db.Connection.ConnectionString))
{
sb.DestinationTableName = "dbo.Item";
sb.ColumnMappings.Add(new SqlBulkCopyColumnMapping("Title", "Title"));
sb.WriteToServer(dt);
}
But this doesn't allow me to add the corresponding 'Property' rows.
I'm thinking the best solution might be to add a Stored Procedure like this one that generically lets me do a bulk insert (or at least multiple inserts, but I can probably disable logging in the stored procedure somehow for performance) and then returns the corresponding ids.
Can anyone think of a better (i.e. more succinct, near equal performance) solution?

To combine the previous best two answers and add in the missing piece for the IDs:
1) Use BCP to Load the data into a temporary "staging" table defined like this
CREATE TABLE stage(Title AS VARCHAR(??), value AS {whatever});
and you'll need the appropriate index for performance later:
CREATE INDEX ix_stage ON stage(Title);
2) Use SQL INSERT to load the Item table:
INSERT INTO Item(Title) SELECT Title FROM stage;
3) Finally load the Property table by joining stage with Item:
INSERT INTO Property(FK_ItemID, Value)
SELECT id, Value
FROM stage
JOIN Item ON Item.Title = stage.Title

The best way to move that much data into SQL Server is bcp. Assuming that the data starts in some sort of file, you'll need to write a small script to funnel the data into the two tables. Alternately you could use bcp to funnel the data into a single table and then use an SP to INSERT the data into the two tables.

Bulk copy the data into a temporary table, and then call a stored proc that splits the data into the two tables you need to populate.

You can bulk copy in code as well, using the .NET SqlBulkCopy class.