I am writing a C# program that will grab some data from a pipe delimited file with 400 columns in it. I'm only required to work with 6 of the columns in each row. The file does not have headers, and the first line is a 5 column row with general description of file (file name, batch date, number of records, total, report id). Before I create a class with 400 fields in it, I was curious if anyone here had a better idea of how to approach this. Thanks for your time.
Well, you don't mention much as to how you're loading the file, but I imagine it is using System.IO and then doing a string split on each line. If so, you need not extract every field in the resulting splitted array.
Imagine you only needed two columns, the second and fourth, and had a class to accept each row as follows:
public class row {
public string field2;
public string field4;
}
Then you would extract your data like this:
IEnumerable<row> parsed =
File.ReadLines(#"path to file")
.Skip(1)
.Select(line => {
var splitted = line.Split('|');
return new row {
field2 = splitted[1],
field4 = splitted[3]
};
});
You could use the Microsoft.VisualBasic.FileIO reference and then do something like this:
using(var parser = new TextFieldParsser(file))
{
Int32 skipHeader = 0;
parser.SetDelimiters("|");
while (!parser.EndOfData)
{
//Processing row
string[] fields = parser.ReadFields();
Int32 x = 0;
if (skipHeader > 0)
{
foreach (var field in fields)
{
if (x == 0)
{
//SAVE STUFF TO VARIABLE
}
else if (x==4)
{
//SAVE MORE STUFF
}
else if (x == 20)
{
//SAVE LAST STUFF
break;//THIS IS THE LAST COLUMN OF DATA NEEDED SO YOU BREAK
}
x++;
}
//DO SOMETHING WITH ALL THE SAVED STUFF AND CLEAR IT OUT
}
else
{
skipHeader++;
}
}}
I have a Datatable in C# with 2 fields and the first fields has the Label and it's value as well in next rows. And the second field has the values only.
Field1 Field2
10th Nick
Nick John
11th Marry
John Andy
Marry Sandy
12th null
Andy null
Sandy null
Is there any way or logic I can apply to structure my data to upload in SQL table in format like below.
Class Name
10th Nick
11th John
11th Marry
12th Andy
12th Sandy
This data is just an example but I need to apply the logic in around million of rows with thousands of class label and looking for something which can pick the field1 value one by one and then search that value in field2
If found - Add that in Name field
if not found - Add that in class field
var list = new List<TempFieldList>();
foreach (DataRow row in ListTable.Rows)
{
list.Add(new TempFieldList{
field1 = row["Field1"].ToString(),
field2 = row["Field2"].ToString()
});
}
foreach (DataRow row in ListTable.Rows)
{
var result = list.FirstOrDefault(x => x.field2 == row["Field1"].ToString());
//row.Dump();
if(result != null )
ResultTable.Rows.Add("");
else
ResultTable.Rows.Add(row["Field1"].ToString(), null);
}
Any better way, or code? As what I was trying is not working unfortunately.
Use the below pseudocode and you'll have a list of custom objects that have class & Names matched up. Then you can loop through that inserting into the database.
List<string> classes = new List();
classes.add("10th");
class.add("11th"); //and so on
string currentClass = new String();
SomeObjectWithClassAndName obj = new SomeObjectWithClassAndName();
List<SomeObjectWithClassAndName> myList = new List();
foreach(DataRow row in yourDataTable)
{
if(classes.Contains(row.Field<string>[1])
{
currentClass = row.Field<string>[1];
}
else{
obj = new SomeObject..;
obj.Class = currentClass;
obj.Name = row.Field<string>[1];
MyList.Add(obj);
}
}
I am facing a strange issue with DataTable.Merge in development. I am trying to merge two tables into one. I am using the following code
gridData.Merge(existingGridData);
gridData has columns "PK", "A", "B", "AA", "BB" and existingGridData contains "PK", "A", "B", "C".
The merge result actually contains the following columns "PK", "A", "B", "AA", "BB", "A", "B", "C" (The columns are duplicated).
Any Idea? I also tried with
gridData.Merge(existingGridData, false, MissingSchemaAction.Ignore);
In this case, the primary key is not copied to the merge result and I am getting Constraint Violation exception.
My Table actually has extended properties (for Columns) and I also tried after copying the extended properties, so that two tables have same extended properties. I verified the Column Name, caption, DataType and all are same.
Two tables are constructed in different locations and have no previous relations.
I tried various scenarios in a new solution and it works fine.
Any help? Thanks in Advance.
See below Code it is giving the correct output
DataTable dataTable = new DataTable();
dataTable.Columns.Add("PK");
dataTable.Columns.Add("A");
dataTable.Columns.Add("B");
dataTable.Columns.Add("AA");
dataTable.Columns.Add("BB");
DataRow drRow = dataTable.NewRow();
drRow[0] = 1;
drRow[1] = 2;
drRow[2] = 1;
drRow[3] = 2;
drRow[4] = 1;
dataTable.Rows.Add(drRow);
drRow = dataTable.NewRow();
drRow[0] = 3;
drRow[1] = 4;
drRow[2] = 12;
drRow[3] = 23;
drRow[4] = 14;
dataTable.Rows.Add(drRow);
DataTable newTable = new DataTable();
newTable.Columns.Add("PK");
newTable.Columns.Add("A");
newTable.Columns.Add("B");
newTable.Columns.Add("C");
newTable.ExtendedProperties.Add("TimeStamp",DateTime.Now);
drRow = newTable.NewRow();
drRow[0] = 5;
drRow[1] =6;
drRow[2] = 5;
drRow[3] = 6;
newTable.Rows.Add(drRow);
drRow = newTable.NewRow();
drRow[0] = 7;
drRow[1] = 8;
drRow[2] = 55;
drRow[3] = 66;
newTable.Rows.Add(drRow);
dataTable.Merge(newTable,false);
You have 2 methods which you can follow. Either you create the logic yourself, like what I am doing in the following method, or you can use inbuilt LINQ functions, like my second approach.
///
/// This method is used to merge a set of data tables, based on common columns between them both
///
///
///
///
public static DataTable MergeDataTables(DataTable dt1, DataTable dt2)
{
try
{
// Get common columns
var commonColumns = dt1.Columns.OfType().Intersect(dt2.Columns.OfType(), new DataColumnComparer());
// Create the result which is going to be sent to the user
DataTable result = new DataTable();
// Add all the columns from both tables
result.Columns.AddRange(
dt1.Columns.OfType()
.Union(dt2.Columns.OfType(), new DataColumnComparer())
.Select(c => new DataColumn(c.Caption, c.DataType, c.Expression, c.ColumnMapping))
.ToArray());
// Add the records of each data table to the new data table, based on the columns
var rowData = dt1.AsEnumerable().Join(
dt2.AsEnumerable(),
row => commonColumns.Select(col => row[col.Caption]).ToArray(),
row => commonColumns.Select(col => row[col.Caption]).ToArray(),
(row1, row2) =>
{
var row = result.NewRow();
row.ItemArray = result.Columns.OfType().Select(col => row1.Table.Columns.Contains(col.Caption) ? row1[col.Caption] : row2[col.Caption]).ToArray();
return row;
},
new ObjectArrayComparer());
// Loop and add
foreach (var row in rowData)
result.Rows.Add(row);
// Return result...
return result;
}
catch (Exception ex)
{
throw new Exception("Problem while merging data tables. Check that there are common columns between the 2 data tables. Error : " + ex.Message);
}
}
Or do the following;
// Results
DataTable reportResult1 = new DataTable(); // Your data table 1
DataTable reportResult2 = new DataTable(); // Your data table 2
// Merge tables
var commonColumns = reportResult1.Columns.OfType().Intersect(reportResult2.Columns.OfType(), new DataColumnComparer());
// Remove DB Nulls, replace with empty strings
reportResult1.RemoveColumnNulls(commonColumns.ToList());
reportResult2.RemoveColumnNulls(commonColumns.ToList());
reportResult1.PrimaryKey = commonColumns.ToArray();
result.Merge(reportResult2, false, MissingSchemaAction.AddWithKey);
result.Merge(reportResult1, false, MissingSchemaAction.AddWithKey);
return result;
Let me know if you manage. I would go for option number 2, as it is much more optimised in terms of performance. What is done is the common columns between 2 tables are extracted, another table is created, and you will have a FULL OUTER JOIN effect. The common columns would be used as the JOINed columns.
Root Cause: I found why it happened. I have changed the caption of the gridData to lowercase some where in code. In detail, I fetched data from database. At that time the column name was in ALLCAPS, say "SAMPLEFIELD". Then in the following code, I renamed the column name (not caption) to "SampleField". The existingGridData also contained "SampleField". I expected both fields to be merged but its not.
When debugging through the .Net Framework Code, I found that there is a dictionary with Column Names (Columns.columnFromName), when you add Columns for the first time. But when you change the column name, this dictionary is not updated - if you provide the same column name with CAPS changed (but.. will be updated if its a different name).
This dictionary is used during Merge. Since the cases are changed it fails. Try the following code.
for (int i = 0; i < existingData.Columns.Count; i++)
{
DataColumn src = existingData.Columns[i];
DataColumn dest = (gridData.Columns.Contains(src.ColumnName)) ? gridData.Columns[src.ColumnName] : null;
if (dest == null)
{
//You will get the column here
}
BindingFlags bindingFlags = BindingFlags.Instance | BindingFlags.NonPublic;
MethodInfo minfo = typeof(DataColumnCollection).GetMethod("Contains", bindingFlags); //This override is used internally duting Merge
var result = minfo.Invoke(gridData.Columns, new object[]{src.ColumnName, true});
dest = ((bool)result) ? gridData.Columns[src.ColumnName] : null;
if (dest == null)
{
//You wont get the column here. Its case sensitive
}
}
Some other issues related to case sensitivity.
http://forums.asp.net/t/707552.aspx
Hope this helps some one
I faced the same issue. I changed all column names to upper case but still the issue was there. I took the default view of both tables and it fixed the issue. Anyway this will remove all the table expression. eg: column expressions.
So the steps was
Make sure all the columns are in same case.
Use default view of both table.
Additionally make sure all the columns has same max lengths. It may cause problems if exceeds the length.
dataTable1 = dataTable1.DefaultView.ToTable();
dataTable2 = dataTable2.DefaultView.ToTable();
dataTable1 .Merge(dataTable2 , true, MissingSchemaAction.Add);
I want to update column B to 1 for the row where the value of column A is 2. In SQL terms, a simple
UPDATE SpreadSheet
SET B = 1
WHERE A = 2
Can this be done in a one or two calls to Google Docs without having to loop over rows/columns?
I wasn't able to use B or A and couldn't find a way to do the update without looping. This example uses the column header (the value of the first row). Replace id and val with your own columns.
WorksheetFeed wsFeed = spreadsheet.Worksheets;
WorksheetEntry worksheet = (WorksheetEntry)wsFeed.Entries[0];
// Define the URL to request the list feed of the worksheet.
AtomLink listFeedLink = worksheet.Links.FindService(GDataSpreadsheetsNameTable.ListRel, null);
// Fetch the list feed of the worksheet.
ListQuery listQuery = new ListQuery(listFeedLink.HRef.ToString());
listQuery.SpreadsheetQuery = "id = 2";
ListFeed listFeed = service.Query(listQuery);
var row = (ListEntry)listFeed.Entries[0];
foreach (ListEntry.Custom element in row.Elements)
{
if (element.LocalName == "val")
{
element.Value = "1";
}
}
row.Update();
My statement:
string sqlCommandText = "SELECT * FROM ForumThread WHERE";
How can I read * from it?
I'm trying like this but its doesnt work:
if (reader["*"] != DBNull.Value)
{
result.TextContent = Convert.ToString(reader["*"]);
}
* is not a column, it's a instruction to SELECT to just grab every column it can find.
The columns will appear individually though, so you need to iterate through the columns one by one and grab their contents.
To get the column contents, you can do this:
StringBuilder sb = new StringBuilder();
for (int index = 0; index < reader.FieldCount; index++)
{
object value = reader[index];
if (value != DBNull.Value)
sb.Append(value.ToString());
}
result.TextContent = sb.ToString();
However, this will result in everything being mashed together.
For instance, if your result set look like this:
A B C D
10 20 30 Some name here
Then your resulting string will look like this:
102030Some name here
Is this what you want?
Perhaps you told us what you want to accomplish, we can think of a better answer?
Additionally, you can use this method to retrieve the associated field name:
reader.GetName( /* index of the column */);
Suppose your ForumThread table contains the ThreadResponse column, and you actually want to read the ThreadResponse column, you can use the next code:
//calculates the column ordinal based on column name
int ThreadResponseColumnIdx = reader.GetOrdinal("ThreadResponse");
//If there is a line in the result
if (reader.Read())
{
//read from the current line the ThreadResponse column
result.TextContent = reader.GetString(ThreadResponseColumnIdx);
}