I currently have a data table which adds new rows based on the Access table rows. In the data table, I change the value of certain columns and update them back to the database. I want to write a line of code that overwrites the data if the primary key already exists. I already looked into a query INSERT INTO table (id, name, age) VALUES(1, "A", 19) ON DUPLICATE KEY UPDATE name="A", age=19 but I can't find a way to implement this since i'm using a data adapter.
I have made the following code:
for (int i = 1; i < 19; i++)
{
Normen.Clear();
Normen.Add(-1);
for (int j = 9; j < 18; j++)
{
Normen.Add(Convert.ToInt32(NormDb.Rows[j].Field<double>(i)));
}
newRow[i] = Convert.ToInt32(GlobalMethods.LeftSegmentIndex(Normen, Convert.ToInt32(newRow[i + 18])));
}
schoolDb.Rows.Add(newRow);
Normen is in this case a list with integers i use to calculate test scores which are then inserted into the empty columns i fetched from the database table.
when this code runs, i have a datatable with rows and columns which are identical to the database columns. When this code is done, the other method is triggered to update the data into the datatable like this:
using (var dbcommand = new OleDbCommand(ZoekQuery, connection))
{
using (var SchoolAdapter = new OleDbDataAdapter(ZoekQuery, connection))
{
OleDbCommandBuilder cb = new OleDbCommandBuilder(SchoolAdapter);
cb.GetInsertCommand();
SchoolAdapter.FillSchema(schoolDb, SchemaType.Source);
SchoolAdapter.Fill(schoolDb);
Normeringen.Normeer(exportDb.Rows.Count, schoolDb, exportDb, NormDb, normcode);
SchoolAdapter.Update(schoolDb);
MessageBox.Show(Success);
}
}
But when the primary key already exists i want to overwrite the record because in this use case it is possible to grade a test with 2 different values. Is there any way i can implement this?
I have a problem and I don't know why it's happening!
I am trying to find Max value in a column inside the database table, I use this code:
private void FrmCustomer_New_Load(object sender, EventArgs e)
{
int NewID;
DataTable _Dt = CusVar.GetCustomersIDs();
if (_Dt.Rows.Count == 0)
{
NewID = 1;
this.txt1.Text = NewID.ToString();
DataRow _Row = CusVar.GetCustomersIDs().Rows[0];
MessageBox.Show(Convert.ToString(_Dt.Rows[0]["MaxID"]));
}
}
the code is working but it gives 1 although there are no records in the table?
I use C# and Access database ACCDB.
I use this function in Cls_Customers:
public DataTable GetCustomersIDs()
{
DAL.DataAccessLayer DAL = new DAL.DataAccessLayer();
DataTable Dt = new DataTable();
Dt = DAL.DataSelect("Select Max(CustomerID) AS MaxID From TblCustomers", null);
DAL.CloseConn();
return Dt;
}
what is the problem, please?
This is your query:
Select Max(CustomerID) AS MaxID
From TblCustomers
It is an aggregation query. Aggregation queries with no group by always return one row -- regardless of whether any rows match.
The value returned in the single row is NULL for MaxId.
I am highly suspicious of what you want to do. If you want the maximum id -- and no rows if the table is empty -- then do:
select c.CustomerID
from TblCustomers c
order by c.CustomerID desc
fetch first 1 row only;
(This uses ANSI/ISO standard syntax so the exact logic might depend on your database.)
My suspicion is that you then want to use this id for an insert -- and that is a bad approach. Almost all databases support some sort of auto-incremented column (with syntax elements such as auto_increment, serial, or identity). That is the right way to assign a unique incrementing id to a column in a table.
The Sql query might be returning null and if there are no rows that match the criteria.
If the intention is to find the number of rows returned.Change the sql query to return the count instead of using aggregate function.
Try with this:
SELECT MAX(T1.CustomerID) AS MaxID From TblCustomers T1
I have an ID list of 75000 records.Based on this I need to fetch corresponding items from another table which have 1700000+ records in c#. I tried with foreach and it taking 1+ hours.Both tables are in different servers. I tried contains in linq and it's not working. Please suggest me some better options.
foreach(var item in cotext1.Table1)
{
var Employee=Context2.table2.where(x=>x.EmpId==item.Id);
}
I tried to join, but getting out of memory exception.
To clarify what was discussed in comments:
Create user defined type which corresponds to your list with IDs (since it doesn't make sense for ids to repeat in your list - I made it primary key):
CREATE TYPE dbo.IntHashSet AS TABLE
(
ID int NOT NULL PRIMARY KEY
)
Query:
using (var ctx = new TestDBEntities()) {
// fill data table with your ids
var dt = new DataTable();
dt.Columns.Add("ID", typeof(int));
for (int i = 0; i < 75000; i++) {
dt.Rows.Add(i);
}
// make a query
var result = ctx.Database.SqlQuery<BigTable>("select BT.* from BigTable BT inner join #IDS I on BT.CodeID = I.ID",
new SqlParameter("IDS", SqlDbType.Structured)
{
// name of type you created in step 1
TypeName = "dbo.IntHashSet",
Value = dt
}).ToArray();
}
This query with 75.000 ids against table with 1.700.000 rows takes about 600ms on my (non-server, consumer grade) machine. Which is several orders of magnituge faster than what it takes now with your query (more than one hour).
You can use a join statement in linq
Server1Entities dc1 = new Server1Entities ();
List<IDTable> ids = (from ro in dc1.IDTable select ro).ToList();
Server2Entities dc2 = new Server2Entities();
var list = (from firstTable in ids
join secondTable in dc.YourSecondTable
on firstTable.ID equals secondTable.ID
select new {
field1 = firstTable.Field1,
field2 = secondTableField1 // You can define fieldnames as you want.
}).ToList();
Another option is using the morelinq library. Really extraordinary.
you can take a look :
https://github.com/morelinq/MoreLINQ
I am using a XML file and the data from the XML is set into a dataset and user selected table is stored as a datatable.A query is been generated with filter criteria, grouping, aggregate function, expressions etc.Is it possible to query the datatable?
I did come accross table.Select(filter criteria, sort) method.but kindly let me know how grouping, aggregate function and expression (eg: Column1 + Column2 as SumColumn) can be got.
You could query the data using LINQ - assuming you are using a .Net Framework version that supports it. Check out LINQ To Dataset
Unfortunately, table.Select(filterCriteria, sort) is your only option without LINQ (I'm not a LINQ guru, so don't ask me what it can do).
Anytime I need something specific, I create/add that column to the DataTable.
DataTable table = new DataTable();
// code that populates the table
DataColumn c = table.Columns.Add("Column1 + Column2", typeof(int));
int Sum = 0;
for (int i = 0; i < table.Rows.Count; i++) {
r = table.Rows[i];
int col1 = (int)r["Column1"];
int col2 = (int)r["Column2"];
int both = col1 + col2;
Sum += both;
r[c] = string.Format("{0}", both);
}
DataRow summaryRow = table.NewRow();
summaryRow[c] = (int)((float)Sum / table.Rows.Count + 0.5); // add 0.5 to round
table.Rows.Add(summaryRow);
HTH.
I have two DataTables, A and B, produced from CSV files. I need to be able to check which rows exist in B that do not exist in A.
Is there a way to do some sort of query to show the different rows or would I have to iterate through each row on each DataTable to check if they are the same? The latter option seems to be very intensive if the tables become large.
Assuming you have an ID column which is of an appropriate type (i.e. gives a hashcode and implements equality) - string in this example, which is slightly pseudocode because I'm not that familiar with DataTables and don't have time to look it all up just now :)
IEnumerable<string> idsInA = tableA.AsEnumerable().Select(row => (string)row["ID"]);
IEnumerable<string> idsInB = tableB.AsEnumerable().Select(row => (string)row["ID"]);
IEnumerable<string> bNotA = idsInB.Except(idsInA);
would I have to iterate through each row on each DataTable to check if they are the same.
Seeing as you've loaded the data from a CSV file, you're not going to have any indexes or anything, so at some point, something is going to have to iterate through every row, whether it be your code, or a library, or whatever.
Anyway, this is an algorithms question, which is not my specialty, but my naive approach would be as follows:
1: Can you exploit any properties of the data? Are all the rows in each table unique, and can you sort them both by the same criteria? If so, you can do this:
Sort both tables by their ID (using some useful thing like a quicksort). If they're already sorted then you win big.
Step through both tables at once, skipping over any gaps in ID's in either table. Matched ID's mean duplicated records.
This allows you to do it in (sort time * 2 ) + one pass, so if my big-O-notation is correct, it'd be (whatever-sort-time) + O(m+n) which is pretty good.
(Revision: this is the approach that ΤΖΩΤΖΙΟΥ describes )
2: An alternative approach, which may be more or less efficient depending on how big your data is:
Run through table 1, and for each row, stick it's ID (or computed hashcode, or some other unique ID for that row) into a dictionary (or hashtable if you prefer to call it that).
Run through table 2, and for each row, see if the ID (or hashcode etc) is present in the dictionary. You're exploiting the fact that dictionaries have really fast - O(1) I think? lookup. This step will be really fast, but you'll have paid the price doing all those dictionary inserts.
I'd be really interested to see what people with better knowledge of algorithms than myself come up with for this one :-)
You can use the Merge and GetChanges methods on the DataTable to do this:
A.Merge(B); // this will add to A any records that are in B but not A
return A.GetChanges(); // returns records originally only in B
The answers so far assume that you're simply looking for duplicate primary keys. That's a pretty easy problem - you can use the Merge() method, for instance.
But I understand your question to mean that you're looking for duplicate DataRows. (From your description of the problem, with both tables being imported from CSV files, I'd even assume that the original rows didn't have primary key values, and that any primary keys are being assigned via AutoNumber during the import.)
The naive implementation (for each row in A, compare its ItemArray with that of each row in B) is indeed going to be computationally expensive.
A much less expensive way to do this is with a hashing algorithm. For each DataRow, concatenate the string values of its columns into a single string, and then call GetHashCode() on that string to get an int value. Create a Dictionary<int, DataRow> that contains an entry, keyed on the hash code, for each DataRow in DataTable B. Then, for each DataRow in DataTable A, calculate the hash code, and see if it's contained in the dictionary. If it's not, you know that the DataRow doesn't exist in DataTable B.
This approach has two weaknesses that both emerge from the fact that two strings can be unequal but produce the same hash code. If you find a row in A whose hash is in the dictionary, you then need to check the DataRow in the dictionary to verify that the two rows are really equal.
The second weakness is more serious: it's unlikely, but possible, that two different DataRows in B could hash to the same key value. For this reason, the dictionary should really be a Dictionary<int, List<DataRow>>, and you should perform the check described in the previous paragraph against each DataRow in the list.
It takes a fair amount of work to get this working, but it's an O(m+n) algorithm, which I think is going to be as good as it gets.
Just FYI:
Generally speaking about algorithms, comparing two sets of sortable (as ids typically are) is not an O(M*N/2) operation, but O(M+N) if the two sets are ordered. So you scan one table with a pointer to the start of the other, and:
other_item= A.first()
only_in_B= empty_list()
for item in B:
while other_item > item:
other_item= A.next()
if A.eof():
only_in_B.add( all the remaining B items)
return only_in_B
if item < other_item:
empty_list.append(item)
return only_in_B
The code above is obviously pseudocode, but should give you the general gist if you decide to code it yourself.
Thanks for all the feedback.
I do not have any index's unfortunately. I will give a little more information about my situation.
We have a reporting program (replaced Crystal reports) that is installed in 7 Servers across EU. These servers have many reports on them (not all the same for each country). They are invoked by a commandline application that uses XML files for their configuration. So One XML file can call multiple reports.
The commandline application is scheduled and controlled by our overnight process. So the XML file could be called from multiple places.
The goal of the CSV is to produce a list of all the reports that are being used and where they are being called from.
I am going through the XML files for all references, querying the scheduling program and producing a list of all the reports. (this is not too bad).
The problem I have is I have to keep a list of all the reports that might have been removed from production. So I need to compare the old CSV with the new data. For this I thought it best to put it into DataTables and compare the information, (this could be the wrong approach. I suppose I could create an object that holds it and compares the difference then create iterate through them).
The data I have about each report is as follows:
String - Task Name
String - Action Name
Int - ActionID (the Action ID can be in multiple records as a single action can call many reports, i.e. an XML file).
String - XML File called
String - Report Name
I will try the Merge idea given by MusiGenesis (thanks). (rereading some of the posts not sure if the Merge will work, but worth trying as I have not heard about it before so something new to learn).
The HashCode Idea sounds interesting as well.
Thanks for all the advice.
I found an easy way to solve this. Unlike previous "except method" answers, I use the except method twice. This not only tells you what rows were deleted but what rows were added. If you only use one except method - it will only tell you one difference and not both. This code is tested and works. See below
//Pass in your two datatables into your method
//build the queries based on id.
var qry1 = datatable1.AsEnumerable().Select(a => new { ID = a["ID"].ToString() });
var qry2 = datatable2.AsEnumerable().Select(b => new { ID = b["ID"].ToString() });
//detect row deletes - a row is in datatable1 except missing from datatable2
var exceptAB = qry1.Except(qry2);
//detect row inserts - a row is in datatable2 except missing from datatable1
var exceptAB2 = qry2.Except(qry1);
then execute your code against the results
if (exceptAB.Any())
{
foreach (var id in exceptAB)
{
//execute code here
}
}
if (exceptAB2.Any())
{
foreach (var id in exceptAB2)
{
//execute code here
}
}
Could you not simply compare the CSV files before loading them into DataTables?
string[] a = System.IO.File.ReadAllLines(#"cvs_a.txt");
string[] b = System.IO.File.ReadAllLines(#"csv_b.txt");
// get the lines from b that are not in a
IEnumerable<string> diff = b.Except(a);
//... parse b into DataTable ...
public DataTable compareDataTables(DataTable First, DataTable Second)
{
First.TableName = "FirstTable";
Second.TableName = "SecondTable";
//Create Empty Table
DataTable table = new DataTable("Difference");
DataTable table1 = new DataTable();
try
{
//Must use a Dataset to make use of a DataRelation object
using (DataSet ds4 = new DataSet())
{
//Add tables
ds4.Tables.AddRange(new DataTable[] { First.Copy(), Second.Copy() });
//Get Columns for DataRelation
DataColumn[] firstcolumns = new DataColumn[ds4.Tables[0].Columns.Count];
for (int i = 0; i < firstcolumns.Length; i++)
{
firstcolumns[i] = ds4.Tables[0].Columns[i];
}
DataColumn[] secondcolumns = new DataColumn[ds4.Tables[1].Columns.Count];
for (int i = 0; i < secondcolumns.Length; i++)
{
secondcolumns[i] = ds4.Tables[1].Columns[i];
}
//Create DataRelation
DataRelation r = new DataRelation(string.Empty, firstcolumns, secondcolumns, false);
ds4.Relations.Add(r);
//Create columns for return table
for (int i = 0; i < First.Columns.Count; i++)
{
table.Columns.Add(First.Columns[i].ColumnName, First.Columns[i].DataType);
}
//If First Row not in Second, Add to return table.
table.BeginLoadData();
foreach (DataRow parentrow in ds4.Tables[0].Rows)
{
DataRow[] childrows = parentrow.GetChildRows(r);
if (childrows == null || childrows.Length == 0)
table.LoadDataRow(parentrow.ItemArray, true);
table1.LoadDataRow(childrows, false);
}
table.EndLoadData();
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
return table;
}
try
{
if (ds.Tables[0].Columns.Count == ds1.Tables[0].Columns.Count)
{
for (int i = 0; i < ds.Tables[0].Rows.Count; i++)
{
for (int j = 0; j < ds.Tables[0].Columns.Count; j++)
{
if (ds.Tables[0].Rows[i][j].ToString() == ds1.Tables[0].Rows[i][j].ToString())
{
}
else
{
MessageBox.Show(i.ToString() + "," + j.ToString());
}
}
}
}
else
{
MessageBox.Show("Table has different columns ");
}
}
catch (Exception)
{
MessageBox.Show("Please select The Table");
}
I'm continuing tzot's idea ...
If you have two sortable sets, then you can just use：
List<string> diffList = new List<string>(sortedListA.Except(sortedListB));
If you need more complicated objects, you can define a comparator yourself and still use it.
The usual usage scenario considers a user that has a DataTable in hand and changes it by Adding, Deleting or Modifying some of the DataRows.
After the changes are performed, the DataTable is aware of the proper DataRowState for each row, and also keeps track of the Original DataRowVersion for any rows that were changed.
In this usual scenario, one can Merge the changes back into a source table (in which all rows are Unchanged). After merging, one can get a nice summary of only the changed rows with a call to GetChanges().
In a more unusual scenario, a user has two DataTables with the same schema (or perhaps only the same columns and lacking primary keys). These two DataTables consist of only Unchanged rows. The user may want to find out what changes does he need to apply to one of the two tables in order to get to the other one. That is, which rows need to be Added, Deleted, or Modified.
We define here a function called GetDelta() which does the job:
using System;
using System.Data;
using System.Xml;
using System.Linq;
using System.Collections.Generic;
using System.Data.DataSetExtensions;
public class Program
{
private static DataTable GetDelta(DataTable table1, DataTable table2)
{
// Modified2 : row1 keys match rowOther keys AND row1 does not match row2:
IEnumerable<DataRow> modified2 = (
from row1 in table1.AsEnumerable()
from row2 in table2.AsEnumerable()
where table1.PrimaryKey.Aggregate(true, (boolAggregate, keycol) => boolAggregate & row1[keycol].Equals(row2[keycol.Ordinal]))
&& !row1.ItemArray.SequenceEqual(row2.ItemArray)
select row2);
// Modified1 :
IEnumerable<DataRow> modified1 = (
from row1 in table1.AsEnumerable()
from row2 in table2.AsEnumerable()
where table1.PrimaryKey.Aggregate(true, (boolAggregate, keycol) => boolAggregate & row1[keycol].Equals(row2[keycol.Ordinal]))
&& !row1.ItemArray.SequenceEqual(row2.ItemArray)
select row1);
// Added : row2 not in table1 AND row2 not in modified2
IEnumerable<DataRow> added = table2.AsEnumerable().Except(modified2, DataRowComparer.Default).Except(table1.AsEnumerable(), DataRowComparer.Default);
// Deleted : row1 not in row2 AND row1 not in modified1
IEnumerable<DataRow> deleted = table1.AsEnumerable().Except(modified1, DataRowComparer.Default).Except(table2.AsEnumerable(), DataRowComparer.Default);
Console.WriteLine();
Console.WriteLine("modified count =" + modified1.Count());
Console.WriteLine("added count =" + added.Count());
Console.WriteLine("deleted count =" + deleted.Count());
DataTable deltas = table1.Clone();
foreach (DataRow row in modified2)
{
// Match the unmodified version of the row via the PrimaryKey
DataRow matchIn1 = modified1.Where(row1 => table1.PrimaryKey.Aggregate(true, (boolAggregate, keycol) => boolAggregate & row1[keycol].Equals(row[keycol.Ordinal]))).First();
DataRow newRow = deltas.NewRow();
// Set the row with the original values
foreach(DataColumn dc in deltas.Columns)
newRow[dc.ColumnName] = matchIn1[dc.ColumnName];
deltas.Rows.Add(newRow);
newRow.AcceptChanges();
// Set the modified values
foreach (DataColumn dc in deltas.Columns)
newRow[dc.ColumnName] = row[dc.ColumnName];
// At this point newRow.DataRowState should be : Modified
}
foreach (DataRow row in added)
{
DataRow newRow = deltas.NewRow();
foreach (DataColumn dc in deltas.Columns)
newRow[dc.ColumnName] = row[dc.ColumnName];
deltas.Rows.Add(newRow);
// At this point newRow.DataRowState should be : Added
}
foreach (DataRow row in deleted)
{
DataRow newRow = deltas.NewRow();
foreach (DataColumn dc in deltas.Columns)
newRow[dc.ColumnName] = row[dc.ColumnName];
deltas.Rows.Add(newRow);
newRow.AcceptChanges();
newRow.Delete();
// At this point newRow.DataRowState should be : Deleted
}
return deltas;
}
private static void DemonstrateGetDelta()
{
DataTable table1 = new DataTable("Items");
// Add columns
DataColumn column1 = new DataColumn("id1", typeof(System.Int32));
DataColumn column2 = new DataColumn("id2", typeof(System.Int32));
DataColumn column3 = new DataColumn("item", typeof(System.Int32));
table1.Columns.Add(column1);
table1.Columns.Add(column2);
table1.Columns.Add(column3);
// Set the primary key column.
table1.PrimaryKey = new DataColumn[] { column1, column2 };
// Add some rows.
DataRow row;
for (int i = 0; i <= 4; i++)
{
row = table1.NewRow();
row["id1"] = i;
row["id2"] = i*i;
row["item"] = i;
table1.Rows.Add(row);
}
// Accept changes.
table1.AcceptChanges();
PrintValues(table1, "table1:");
// Create a second DataTable identical to the first.
DataTable table2 = table1.Clone();
// Add a row that exists in table1:
row = table2.NewRow();
row["id1"] = 0;
row["id2"] = 0;
row["item"] = 0;
table2.Rows.Add(row);
// Modify the values of a row that exists in table1:
row = table2.NewRow();
row["id1"] = 1;
row["id2"] = 1;
row["item"] = 455;
table2.Rows.Add(row);
// Modify the values of a row that exists in table1:
row = table2.NewRow();
row["id1"] = 2;
row["id2"] = 4;
row["item"] = 555;
table2.Rows.Add(row);
// Add a row that does not exist in table1:
row = table2.NewRow();
row["id1"] = 13;
row["id2"] = 169;
row["item"] = 655;
table2.Rows.Add(row);
table2.AcceptChanges();
Console.WriteLine();
PrintValues(table2, "table2:");
DataTable delta = GetDelta(table1,table2);
Console.WriteLine();
PrintValues(delta,"delta:");
// Verify that the deltas DataTable contains the adequate Original DataRowVersions:
DataTable originals = table1.Clone();
foreach (DataRow drow in delta.Rows)
{
if (drow.RowState != DataRowState.Added)
{
DataRow originalRow = originals.NewRow();
foreach (DataColumn dc in originals.Columns)
originalRow[dc.ColumnName] = drow[dc.ColumnName, DataRowVersion.Original];
originals.Rows.Add(originalRow);
}
}
originals.AcceptChanges();
Console.WriteLine();
PrintValues(originals,"delta original values:");
}
private static void Row_Changed(object sender,
DataRowChangeEventArgs e)
{
Console.WriteLine("Row changed {0}\t{1}",
e.Action, e.Row.ItemArray[0]);
}
private static void PrintValues(DataTable table, string label)
{
// Display the values in the supplied DataTable:
Console.WriteLine(label);
foreach (DataRow row in table.Rows)
{
foreach (DataColumn col in table.Columns)
{
Console.Write("\t " + row[col, row.RowState == DataRowState.Deleted ? DataRowVersion.Original : DataRowVersion.Current].ToString());
}
Console.Write("\t DataRowState =" + row.RowState);
Console.WriteLine();
}
}
public static void Main()
{
DemonstrateGetDelta();
}
}
The code above can be tested in https://dotnetfiddle.net/. The resulting output is shown below:
table1:
0 0 0 DataRowState =Unchanged
1 1 1 DataRowState =Unchanged
2 4 2 DataRowState =Unchanged
3 9 3 DataRowState =Unchanged
4 16 4 DataRowState =Unchanged
table2:
0 0 0 DataRowState =Unchanged
1 1 455 DataRowState =Unchanged
2 4 555 DataRowState =Unchanged
13 169 655 DataRowState =Unchanged
modified count =2
added count =1
deleted count =2
delta:
1 1 455 DataRowState =Modified
2 4 555 DataRowState =Modified
13 169 655 DataRowState =Added
3 9 3 DataRowState =Deleted
4 16 4 DataRowState =Deleted
delta original values:
1 1 1 DataRowState =Unchanged
2 4 2 DataRowState =Unchanged
3 9 3 DataRowState =Unchanged
4 16 4 DataRowState =Unchanged
Note that if your tables don't have a PrimaryKey, the where clause in the LINQ queries gets simplified a little bit. I'll let you figure that out on your own.
Achieve it simply using linq.
private DataTable CompareDT(DataTable TableA, DataTable TableB)
{
DataTable TableC = new DataTable();
try
{
var idsNotInB = TableA.AsEnumerable().Select(r => r.Field<string>(Keyfield))
.Except(TableB.AsEnumerable().Select(r => r.Field<string>(Keyfield)));
TableC = (from row in TableA.AsEnumerable()
join id in idsNotInB
on row.Field<string>(ddlColumn.SelectedItem.ToString()) equals id
select row).CopyToDataTable();
}
catch (Exception ex)
{
lblresult.Text = ex.Message;
ex = null;
}
return TableC;
}