DataTable.Merge Issue: Adds New Column (Due to Case Sensitivity) - c#

I am facing a strange issue with DataTable.Merge in development. I am trying to merge two tables into one. I am using the following code
gridData.Merge(existingGridData);
gridData has columns "PK", "A", "B", "AA", "BB" and existingGridData contains "PK", "A", "B", "C".
The merge result actually contains the following columns "PK", "A", "B", "AA", "BB", "A", "B", "C" (The columns are duplicated).
Any Idea? I also tried with
gridData.Merge(existingGridData, false, MissingSchemaAction.Ignore);
In this case, the primary key is not copied to the merge result and I am getting Constraint Violation exception.
My Table actually has extended properties (for Columns) and I also tried after copying the extended properties, so that two tables have same extended properties. I verified the Column Name, caption, DataType and all are same.
Two tables are constructed in different locations and have no previous relations.
I tried various scenarios in a new solution and it works fine.
Any help? Thanks in Advance.

See below Code it is giving the correct output
DataTable dataTable = new DataTable();
dataTable.Columns.Add("PK");
dataTable.Columns.Add("A");
dataTable.Columns.Add("B");
dataTable.Columns.Add("AA");
dataTable.Columns.Add("BB");
DataRow drRow = dataTable.NewRow();
drRow[0] = 1;
drRow[1] = 2;
drRow[2] = 1;
drRow[3] = 2;
drRow[4] = 1;
dataTable.Rows.Add(drRow);
drRow = dataTable.NewRow();
drRow[0] = 3;
drRow[1] = 4;
drRow[2] = 12;
drRow[3] = 23;
drRow[4] = 14;
dataTable.Rows.Add(drRow);
DataTable newTable = new DataTable();
newTable.Columns.Add("PK");
newTable.Columns.Add("A");
newTable.Columns.Add("B");
newTable.Columns.Add("C");
newTable.ExtendedProperties.Add("TimeStamp",DateTime.Now);
drRow = newTable.NewRow();
drRow[0] = 5;
drRow[1] =6;
drRow[2] = 5;
drRow[3] = 6;
newTable.Rows.Add(drRow);
drRow = newTable.NewRow();
drRow[0] = 7;
drRow[1] = 8;
drRow[2] = 55;
drRow[3] = 66;
newTable.Rows.Add(drRow);
dataTable.Merge(newTable,false);

You have 2 methods which you can follow. Either you create the logic yourself, like what I am doing in the following method, or you can use inbuilt LINQ functions, like my second approach.
///
/// This method is used to merge a set of data tables, based on common columns between them both
///
///
///
///
public static DataTable MergeDataTables(DataTable dt1, DataTable dt2)
{
try
{
// Get common columns
var commonColumns = dt1.Columns.OfType().Intersect(dt2.Columns.OfType(), new DataColumnComparer());
// Create the result which is going to be sent to the user
DataTable result = new DataTable();
// Add all the columns from both tables
result.Columns.AddRange(
dt1.Columns.OfType()
.Union(dt2.Columns.OfType(), new DataColumnComparer())
.Select(c => new DataColumn(c.Caption, c.DataType, c.Expression, c.ColumnMapping))
.ToArray());
// Add the records of each data table to the new data table, based on the columns
var rowData = dt1.AsEnumerable().Join(
dt2.AsEnumerable(),
row => commonColumns.Select(col => row[col.Caption]).ToArray(),
row => commonColumns.Select(col => row[col.Caption]).ToArray(),
(row1, row2) =>
{
var row = result.NewRow();
row.ItemArray = result.Columns.OfType().Select(col => row1.Table.Columns.Contains(col.Caption) ? row1[col.Caption] : row2[col.Caption]).ToArray();
return row;
},
new ObjectArrayComparer());
// Loop and add
foreach (var row in rowData)
result.Rows.Add(row);
// Return result...
return result;
}
catch (Exception ex)
{
throw new Exception("Problem while merging data tables. Check that there are common columns between the 2 data tables. Error : " + ex.Message);
}
}
Or do the following;
// Results
DataTable reportResult1 = new DataTable(); // Your data table 1
DataTable reportResult2 = new DataTable(); // Your data table 2
// Merge tables
var commonColumns = reportResult1.Columns.OfType().Intersect(reportResult2.Columns.OfType(), new DataColumnComparer());
// Remove DB Nulls, replace with empty strings
reportResult1.RemoveColumnNulls(commonColumns.ToList());
reportResult2.RemoveColumnNulls(commonColumns.ToList());
reportResult1.PrimaryKey = commonColumns.ToArray();
result.Merge(reportResult2, false, MissingSchemaAction.AddWithKey);
result.Merge(reportResult1, false, MissingSchemaAction.AddWithKey);
return result;
Let me know if you manage. I would go for option number 2, as it is much more optimised in terms of performance. What is done is the common columns between 2 tables are extracted, another table is created, and you will have a FULL OUTER JOIN effect. The common columns would be used as the JOINed columns.

Root Cause: I found why it happened. I have changed the caption of the gridData to lowercase some where in code. In detail, I fetched data from database. At that time the column name was in ALLCAPS, say "SAMPLEFIELD". Then in the following code, I renamed the column name (not caption) to "SampleField". The existingGridData also contained "SampleField". I expected both fields to be merged but its not.
When debugging through the .Net Framework Code, I found that there is a dictionary with Column Names (Columns.columnFromName), when you add Columns for the first time. But when you change the column name, this dictionary is not updated - if you provide the same column name with CAPS changed (but.. will be updated if its a different name).
This dictionary is used during Merge. Since the cases are changed it fails. Try the following code.
for (int i = 0; i < existingData.Columns.Count; i++)
{
DataColumn src = existingData.Columns[i];
DataColumn dest = (gridData.Columns.Contains(src.ColumnName)) ? gridData.Columns[src.ColumnName] : null;
if (dest == null)
{
//You will get the column here
}
BindingFlags bindingFlags = BindingFlags.Instance | BindingFlags.NonPublic;
MethodInfo minfo = typeof(DataColumnCollection).GetMethod("Contains", bindingFlags); //This override is used internally duting Merge
var result = minfo.Invoke(gridData.Columns, new object[]{src.ColumnName, true});
dest = ((bool)result) ? gridData.Columns[src.ColumnName] : null;
if (dest == null)
{
//You wont get the column here. Its case sensitive
}
}
Some other issues related to case sensitivity.
http://forums.asp.net/t/707552.aspx
Hope this helps some one

I faced the same issue. I changed all column names to upper case but still the issue was there. I took the default view of both tables and it fixed the issue. Anyway this will remove all the table expression. eg: column expressions.
So the steps was
Make sure all the columns are in same case.
Use default view of both table.
Additionally make sure all the columns has same max lengths. It may cause problems if exceeds the length.
dataTable1 = dataTable1.DefaultView.ToTable();
dataTable2 = dataTable2.DefaultView.ToTable();
dataTable1 .Merge(dataTable2 , true, MissingSchemaAction.Add);

Related

SSIS C# scrip modification to get file created time in loop/list

I"m using SSIS package with Script task to get files not older then n days and it's working fine, but now I need to bring into next step CreatedTime for each file. Below I pasted the body of my script. It works partially I just can't pass new var into LastUpdated. Frankly don't know how to deal with this structure, can I add another dimension to into existing list of create another list. I plan to use User:LastUpdated in the same way as FileNameArray.
Tx much !)
DataTable NewList = new DataTable();
DataColumn col = new DataColumn("FileName");
NewList.Columns.Add(col);
DataColumn col2 = new DataColumn("LastUpdated", System.Type.GetType("System.DateTime"));
NewList.Columns.Add(col2);
foreach (string f in MyDirFiles)
{
finf = new System.IO.FileInfo(f);
if (finf.LastWriteTime > DateTime.Now.AddDays(-7) )
)
{
NewList.Rows.Add(System.IO.Path.GetFileName(f) ,
System.IO.File.GetCreationTime(f));
}
}
Dts.Variables["User::FileNameArray"].Value = NewList.Columns["FileName"]; //<--- need convert into object
////**Dts.Variables["User::LastUpdated"].Value = NewList(xxx);
Dts.TaskResult = (int)ScriptResults.Success;
From your code and comments - can conclude the following:
NewList2 variable has DataTable type (not present in code)
User:LastUpdated SSIS package variable has DateTime type
In this case - you are trying to assign a complex structure (DataTable) to single value DateTime variable, which certainly raises an error. To do so, change type of User:LastUpdated to Object.
One can extend NewList table to contain both columns, like in the example below
DataTable NewList = new DataTable();
DataColumn col = new DataColumn("FileName");
NewList.Columns.Add(col);
DataColumn col2 = new DataColumn("LastUpdated", System.Type.GetType("System.DateTime"));
NewList.Columns.Add(col2);
Adding a new row will be more awkward.
DataRow newRow = NewList.NewRow();
newRow["FileName"] = System.IO.Path.GetFileName(f);
newRow["LastUpdated"] = System.IO.File.GetCreationTime(f);
NewList.Rows.Add(newRow);

Linq Query on DataTable and Update Records

I have a datatable in memory and I need to select some records from it, walk through the records making changes to fields and they same the changes back to the datatable. I can do this with filters, views, and sql but I'm trying to do it in Linq.
var results = (from rows in dtTheRows.AsEnumerable()
select new
{
rows.Job,
}).Distinct();
foreach (var row in results)
{
firstRow = true;
thisOnHand = 0;
var here = from thisRow in dtTheRows.AsEnumerable()
orderby thisRow.PromisedDate
select new
{
thisRow.OnHandQuantity,
thisRow.Balance,
thisRow.RemainingQuantity
};
foreach(var theRow in here)
{
// business logic here ...
theRow.OnHandQuantity = 5;
} // foreach ...
The first linq query and foreach are gain the list of subsets of data to be considered. I include it here in case it is relevant. My problem is at this line:
heRow.OnHandQuantity = 5;
My error is:
"Error 19 Property or indexer 'AnonymousType#1.OnHandQuantity' cannot be assigned to -- it is read only"
What am I missing here? Can I update this query back into the original datatable?
var here = from thisRow in dtTheRows.AsEnumerable()
orderby thisRow.PromisedDate
select new
{
thisRow.OnHandQuantity,
thisRow.Balance,
thisRow.RemainingQuantity
};
Instead of passing three variables in select, pass thisRow itself. That may solve error on statement - theRow.OnHandQuantity = 5;
The error is self descriptive, you can't update/modify an anonymous type. You have to return the original entity you want to modify from your query.
select thisRow;
instead of
select new
{
thisRow.OnHandQuantity,
thisRow.Balance,
thisRow.RemainingQuantity
};

Moving the column of RenderTable

I have a renderTable and I am adding rows and columns to the table as follows-
RenderTable renderTable = new RenderTable();
DataTable dt = GetData();
foreach (DataRow row in dt.Rows)
{
var header = renderTable.Rows[renderTable.Rows.Count];
header[0].Text = "Column 1";
header[1].Text = "Column 2";
header[2].Text = "Column 3";
header[1].Text = "Column 4";
var data = renderTable.Rows[renderTable.Rows.Count];
data [0].Text = row["col1"].ToString(); // 10
data [1].Text = row["col2"].ToString(); // 11
data [2].Text = row["col3"].ToString(); // 12
data [3].Text = row["col4"].ToString(); // 13
}
This is working fine and table is rendering as folllows-
Column 1 Column2 Column3 Column4
10 11 12 13
My requirement is, now I want to move the column 4 to another place like 2nd place as follows . (this place can differ depending on condition)
Column 1 Column4 Column2 Column3
10 13 11 12
I tried Insert method but it is not working for me as the insert index may change.
Is there any function of render table to move the column to specified index.
Please suggest any alternative if any.
We regret to mention but there isn't any function that can allow the moving of column of RenderTable to a specified index since the Cols of C1PrintDocument is ReadOnly.
I've done this by creating a new class from System.Web.UI.WebControls.GridView. I override CreateColumns which is used to return an array of the column objects in order. I read a cookie from the page (this allows me to change the columns via a cookie on the page) and create a new column array based on the cookie. This cookie is just a string of the column names in the order required with a | separator. I had another column picker page that would set this cookie. If you don't need to change the columns with a cookie this is not needed -- you could read / create this string from a database or configuration file. I believe the code is well commented and clear -- one note, our application has a requirement to include hidden columns, so I add those to the end of the column list before I return the array.
using System.Collections;
using System.Linq;
using System.Web.UI.WebControls;
public class ChangeColumnGridView : System.Web.UI.WebControls.GridView
{
protected override ICollection CreateColumns(PagedDataSource dataSource, bool useDataSource)
{
// Get the needful from the base class
var baseColList = base.CreateColumns(dataSource, useDataSource);
var inColList = baseColList.OfType<object>();
// Get our column order
string columnOrder;
if (Page.Request.Cookies["colOrder"] != null)
columnOrder = Page.Request.Cookies["colOrder"].Value;
else
return baseColList;
// change it to an array
string[] columnOrderA = columnOrder.Split(new char[] { '|' });
// this is where we will put our results
ArrayList newColumnList = new ArrayList();
// look for each name in the list and add when we find it.
foreach (string name in columnOrderA)
{
var found = inColList.Where((c) => c.ToString() == name).FirstOrDefault();
if (found != null)
newColumnList.Add(found);
}
// look for non-visible items in the list and add them if we don't already have them.
foreach (var a in inColList)
{
if (((System.Web.UI.WebControls.DataControlField)a).Visible == false)
{
var found = newColumnList.Cast<object>().Where((c) => c.ToString() == a.ToString()).FirstOrDefault();
if (found == null)
newColumnList.Add(a);
}
}
return newColumnList;
}
}

Add a second datatable as rowsource of datagridview

I have a datagridview, and I am populating it from a datatable which in turn retrieves it's data from a webservice query (against SalesForce cloud system).
Essentially we want to show the results of attempting to remove attachments from SalesForce cases which have been put on there by the users, currently we query a SalesForce XML webservice called case and we want to add the capability to also query a second new SalesForce object called credit case.
It has been working fine getting its data from an object called cases, and displaying them in a Windows Forms datagridview control.
Now we want to add another object (let's call it creditCases), so I have the query all setup, have added another task object with the fields and datatypes and so on.
Once the dataset is populated we set the data source of the datagridview using gvTaskCases.DataSource = dtCases;
But of course I now have a second datasource (with a different number of columns) that I want to add to the table.
If I do this with my new datatable (gvTaskCases.DataSource = dtcreditCases) how do I stop it replacing the data from the existing datatable?
CODE:
gvTaskCases.DataSource = null;
dtCases.Rows.Clear();
foreach (task_cases item in cases)
{
DataRow drCases = dtCases.NewRow();
// Then add the new row to the collection.
drCases["Case ID"] = item.c_Id;
drCases["Case Number"] = item.c_Number;
drCases["Case Topic"] = item.c_Topic;
drCases["Case SubTopic"] = item.c_Subtopic;
drCases["Account Number"] = item.c_CustomerNumber;
drCases["Additional Info"] = item.c_AdditionalInfo;
drCases["Closed Date"] = Convert.ToDateTime(item.c_ClosedDate).ToString("dd/MM/yyyy");
drCases["Attachment"] = item.c_Attachment;
drCases["Content Type"] = item.c_ContentType;
drCases["Detach Status"] = item.c_Status;
drCases["Document Type"] = item.c_DocumentType;
drCases["Imaging Directory"] = item.c_ImagingDSXDirectory;
drCases["Imaging Document"] = item.c_ImagingDocument;
dtCases.Rows.Add(drCases);
}
gvTaskCases.DataSource = dtCases;
gvTaskCases.ColumnHeadersHeightSizeMode = DataGridViewColumnHeadersHeightSizeMode.EnableResizing;
gvTaskCases.AutoSizeColumnsMode = DataGridViewAutoSizeColumnsMode.DisplayedCells;
foreach (task_creditcases item in creditCases)
{
DataRow drCases = dtCases.NewRow();
// Then add the new row to the collection.
drCases["Case ID"] = item.c_Id;
drCases["Case Number"] = item.c_Name; // SalesForece Credit Case Object CASE NUMBER;
drCases["Account Number"] = item.c_Account__c;
drCases["Closed Date"] = Convert.ToDateTime(item.c_Closed_Date__c).ToString("dd/MM/yyyy");
drCases["Attachment"] = item.c_Imaging_Document_Attached__c;
drCases["Detach Status"] = item.c_Status__c;
drCases["Document Type"] = item.c_Document_Type__c;
drCases["Imaging Directory"] = item.c_Directories__c;
drCases["Imaging Document"] = item.c_Imaging_Document;
dtCases.Rows.Add(drCases);
}
// add the new records:
// Adjust size & hide columns that aren't needed
gvTaskCases.Columns[0].Visible = false;
gvTaskCases.Columns[11].Visible = false;
gvTaskCases.Columns[12].Visible = false;
gvTaskCases.Columns[13].Visible = false;
You can use the Merge method of the DataTable.
((DataTable) gvTaskCases.DataSource).Merge(dtcreditCases);
Another approach is just keep your old DataTable, you have to use a little LINQ to merge the dataTables like this:
gvTaskCases.DataSource = (dtCases.Columns.Count > dtcreditCases.Columns.Count ?
dtCases.AsEnumerable().Concat(dtcreditCases.AsEnumerable()) :
dtcreditCases.AsEnumerable().Concat(dtCases.AsEnumerable())).CopyToDataTable();

Storing references to an object in DataTable using the "this" keyword

I have a collection of objects with no key or order or other obvious index.
I wish to store data regarding each object in a DataTable. I thought an elegant way of doing this would be to store a reference in the owner column, and make that columns type typeof(MyClass).
However, when I try to do this in practice, it doesn't work (it says the primary keys collide). Turns out that putting the instances into a row field just writes "MyProgram.MyClass" into the field - presumably the output of toString even though that row's type was supposed to be MyClass not string.
Here is some sample code which works in LINQPad:
void Main()
{
// Create a table
var table = new DataTable();
var ownerColumn = new DataColumn("Owner", typeof(MyClass));
var primaryKey = new[] { ownerColumn };
table.Columns.AddRange(primaryKey);
table.PrimaryKey = primaryKey;
table.Columns.Add(new DataColumn("Some Data", typeof(int)) { DefaultValue = 0 });
// Create 2 objects
var c1 = new MyClass();
var c2 = new MyClass();
// Store their data in the table
var row = table.NewRow();
row["Owner"] = c1;
row["Some Data"] = 1;
table.Rows.Add(row);
row = table.NewRow();
row["Owner"] = c2;
row["Some Data"] = 2;
table.Rows.Add(row);
}
// Define other methods and classes here
class MyClass {
}
What do I do to solve this? Do I have to make an id field in MyClass, then use id to fill in the owner column, and then make sure each object receives a unique id at creation myself?
You have to implement System.IComparable (non-generic version) interface on MyClass so that DataTable knows how to to compare the value of the column. If this interface is not defined, the code falls back on comparing object.ToString() results.
You can use auto increment column :
DataTable dTable = new DataTable();
DataColumn auto = new DataColumn("AutoID", typeof(System.Int32));
dTable.Columns.Add(auto);
auto.AutoIncrement = true;
auto.AutoIncrementSeed = 1;
auto.ReadOnly = true;

Categories

Resources