This code is working but taking too much time. Every data table contains 1000nds of rows and each time I need to filter data from another data tables with respect to a column.
for (int i = 0; i < dsResult.Tables[0].Rows.Count; i++)
{
DataTable dtFiltered = dtWorkExp.Clone();
foreach (DataRow drr in dtWorkExp.Rows)
{
if (drr["UserId"].ToString() == dsResult.Tables[0].Rows[i]["Registration NO."].ToString())
{
dtFiltered.ImportRow(drr);
}
}
DataTable dtFilteredAward= dtAwards.Clone();
foreach (DataRow drr in dtAwards.Rows)
{
if (drr["UserId"].ToString() == dsResult.Tables[0].Rows[i]["Registration NO."].ToString())
{
dtFilteredAward.ImportRow(drr);
}
}
DataTable dtFilteredOtherQual = dtOtherQual.Clone();
foreach (DataRow drr in dtOtherQual.Rows)
{
if (drr["UserId"].ToString() == dsResult.Tables[0].Rows[i]["Registration NO."].ToString())
{
dtFilteredOtherQual.ImportRow(drr);
}
}
//Do some operation with filtered Data Tables
}
You can declare these lines outside the for loop.
DataTable dtFiltered = dtWorkExp.Clone();
And instead of doing accessing dsResult.Table[0] each time, you can assign this to one variable and use it.
You can also replace the foreach loop with LINQ.
What I would do:
All rows of the main datatable as enumerable:
var rows = dsResult.Tables[0].AsEnumerable();
Get the column you're going to filter with:
var filter = rows.Select(r => r.Field<string>("Registration NO."));
Create a method that accepts that filter, a table to filter and a field to compare.
public static DataTable Filter<T>(EnumerableRowCollection<T> filter, DataTable table, string fieldName)
{
return table.AsEnumerable().Where(r => filter.Contains(r.Field<T>(fieldName))).CopyToDataTable();
}
Finally use the method to filter all tables:
var dtFiltered = Filter<string>(filter, dtWorkExp, "UserId");
var dtFilteredAward = Filter<string>(filter, dtAwards, "UserId");
var dtFilteredOtherQual = Filter<string>(filter, dtOtherQual, "UserId");
All together woul be something like this
public void YourMethod()
{
var rows = dsResult.Tables[0].AsEnumerable();
var filter = rows.Select(r => r.Field<string>("Registration NO."));
var dtFiltered = Filter<string>(filter, dtWorkExp, "UserId");
var dtFilteredAward = Filter<string>(filter, dtAwards, "UserId");
var dtFilteredOtherQual = Filter<string>(filter, dtOtherQual, "UserId");
}
public static DataTable Filter<T>(EnumerableRowCollection<T> filter, DataTable table, string fieldName)
{
return table.AsEnumerable().Where(r => filter.Contains(r.Field<T>(fieldName))).CopyToDataTable();
}
Put the value of the expression in a variable.
var regNo = dsResult.Tables[0].Rows[i]["Registration NO."].ToString();
Put the index of column to the variable. Access by index more faster then by column name.
int index = dtWorkExp.Columns["UserId"].Ordinal;
Result code:
int dtWorkIndex = dtWorkExp.Columns["UserId"].Ordinal;
int dtAwardsIndex = dtAwards.Columns["UserId"].Ordinal;
int dtOtherQualIdex = dtOtherQual.Columns["UserId"].Ordinal;
for (int i = 0; i < dsResult.Tables[0].Rows.Count; i++)
{
var regNo = dsResult.Tables[0].Rows[i]["Registration NO."].ToString();
DataTable dtFiltered = dtWorkExp.Clone();
foreach (DataRow drr in dtWorkExp.Rows)
{
if (drr[dtWorkIndex].ToString() == regNo)
{
dtFiltered.ImportRow(drr);
}
}
...
Of course, the column index can be set as a constant if you know it exactly in advance. Also, if the UserId indexes match in all tables, a single variable is sufficient.
You can also try using the BeginLoadData and EndLoadData methods.
DataTable dtFiltered = dtWorkExp.Clone();
dtFiltered.BeginLoadData();
foreach (DataRow drr in dtWorkExp.Rows)
{
if (drr[dtWorkIndex].ToString() == regNo)
{
dtFiltered.ImportRow(drr);
}
}
dtFiltered.EndLoadData();
But I'm not sure if they make sense together with ImportRow.
Finally, parallelization comes to help.
for (int i = 0; i < dsResult.Tables[0].Rows.Count; i++)
{
var regNo = ...;
var workTask = Task.Run(() =>
{
DataTable dtFiltered = dtWorkExp.Clone();
foreach (DataRow drr in dtWorkExp.Rows)
{
if (drr[dtWorkIndex].ToString() == regNo)
{
dtFiltered.ImportRow(drr);
}
}
return dtFiltered;
});
var awardTask = Task.Run(() =>
...
var otherQualTask = Task.Run(() =>
...
//Task.WaitAll(workTask, awardTask, otherQualTask);
await Task.WhenAll(workTask, awardTask, otherQualTask);
//Do some operation with filtered Data Tables
}
Related
I have a project that I am working on, which is related to the work of daily shifts for employees, so that I have made the code for filling in the dates and their days in the relevant table ResultTB as follows and it's work correctly
var dt1 = DateTime.Parse(txtFromDate.Text);
var dt2 = DateTime.Parse(txtToDate.Text);
var dt = DateTime.Parse(txtFromDate.Text);
if (dt <= dt2)
{
dt = dt.AddDays(-1);
while (dt2 >= dt1)
{
List<ResultTB> ResultList = new List<ResultTB>
{
new ResultTB{NameOfDay = dt1.DayOfWeek.ToString(),DateOfDay = dt=dt.AddDays(1) }
};
foreach (var item in ResultList)
{
dt1 = dt1.AddDays(1);
db.ResultTBs.InsertOnSubmit(item);
}
db.SubmitChanges();
DGVResult.DataSource = db.ResultTBs.ToList();
}
}
Now I would like to make an update to the table ResultTB to columns EmpName+EmpID from EmployeeTB, so that the code searches in the table EmployeeTB which have the status of the employee “possible” and assigns it to a date in order, and this is the code that I wrote, but it adds the first person in the EmployeeTB table and assigns it to all the dates in the table ResultTB.
var EmpList = db.EmployeeTBs
.Where(x => x.EmpStatus == "possible").ToList();
foreach (var EmpItem in EmpList)
{
var ListOfResult = db.ResultTBs.Where(r => r.EmpName == null).ToList();
foreach (var itemResult in ListOfResult)
{
itemResult.EmpID = EmpItem.EmpID;
itemResult.EmpName = EmpItem.EmpName;
}
db.SubmitChanges();
}
I tried many solutions but it didn't work with me please help me to solve this problem. Thanks
var EmpList = db.EmployeeTBs.Where(x => x.EmpStatus == "possible").ToList();
for (int i=0; i< EmpList.length; i++)
{
var ListOfResult = db.ResultTBs.Where(r => r.EmpName == null).ToList();
ListOfResult[i].EmpID = EmpList[i].EmpID
ListOfResult[i].EmpName = EmpList[i].EmpName
}
db.SubmitChanges();
If I understood your question correctly this should work.
I have 500 Columns in my DataTable and I want to remove all of them except for 25 columns.
Is there any way to do this faster to save time and lines of code?
This is what I already tried:
private static void DeleteUselessColumns()
{
//This is example data!
List<DataColumn> dataColumnsToDelete = new List<DataColumn>();
DataTable bigData = new DataTable();
bigData.Columns.Add("Harry");
bigData.Columns.Add("Konstantin");
bigData.Columns.Add("George");
bigData.Columns.Add("Gabriel");
bigData.Columns.Add("Oscar");
bigData.Columns.Add("Muhammad");
bigData.Columns.Add("Emily");
bigData.Columns.Add("Olivia");
bigData.Columns.Add("Isla");
List<string> columnsToKeep = new List<string>();
columnsToKeep.Add("Isla");
columnsToKeep.Add("Oscar");
columnsToKeep.Add("Konstantin");
columnsToKeep.Add("Gabriel");
//This is the code i want to optimize------
foreach (DataColumn column in bigData.Columns)
{
bool keepColumn = false;
foreach (string s in columnsToKeep)
{
if (column.ColumnName.Equals(s))
{
keepColumn = true;
}
}
if (!keepColumn)
{
dataColumnsToDelete.Add(column);
}
}
foreach(DataColumn dataColumn in dataColumnsToDelete)
{
bigData.Columns.Remove(dataColumn);
}
//------------------------
}
var columnsToKeep = new List<string>() { "Isla", "Oscar", "Konstantin", "Gabriel"};
var toRemove = new List<DataColumn>();
foreach(DataColumn column in bigData.Columns)
{
if (!columnsToKeep.Any(name => column.ColumnName == name ))
{
toRemove.Add(column);
}
}
toRemove.ForEach(col => bigData.Columns.Remove(col));
Test1...test9 same code could be made a loop. No need to add the columns to delete in a list, just delete them in the first while loop. As for performance, not sure how to improve it.
You could try to use a DataView that selects the desired columns then copy to table. You need to experiment.
if they have different names create an array of string
var columns = new string[] { "Harry", "Konstantin","John"};
var columnsToKeep = new string[] { "John", "Konstantin"};
var columnsToDelete = from item in columns
where !columnsToKeep.Contains(item)
select item;
or using lambda
var columnsToDelete = columns
.Where (i=> !columnsToKeep.Contains(i))
.ToList();
toDelete
Harry
I need to create a DataTable from another DataTable that I have but with only some columns (exactly same rows).
Is there an easy way to do this using LINQ?
I tried:
DataTable dataCopy = dt.AsEnumerable()
.Select(r => new { A = r.Field<string>("Column A"),
B = r.Field<string>("Column B") });
But it doesn't have a "CopyToDataTable" method.
I'm looking for the best performance way to do this because my "source" DataTable is huge!
You can simply copy the dataTable using dataTable.Copy() and remove the unwanted columns from the copied object.
var dt1 = dt.Copy();
dt.Columns.Remove("<columnsToRemove>")
Stopwatch time = new Stopwatch();
time.Start();
//COMPARE YOUR CODE (.Copy, Clone or Develop yourself)
DataTable dtTarget = dtSource.CopyDataTable(new List<string>() { "A", "B" });
foreach (DataColumn column in dtTarget.Columns)
{
Console.WriteLine("ColumnName : {0}", column.ColumnName);
foreach (DataRow row in dtTarget.Rows)
{
Console.WriteLine("Rows : {0}", row[column.ColumnName]);
}
}
time.Stop();
Console.WriteLine("{0}", time.Elapsed);
public static class DataTableHelper
{
public static DataTable CopyDataTable(
this DataTable dtSource,
List<string> columnsName)
{
DataTable dtTarget = new DataTable();
if (dtSource.Columns.Count > 0)
{
foreach (DataColumn columnSource in dtSource.Columns)
{
var columnTargetMapped = columnsName.FirstOrDefault(c => c == columnSource.ColumnName);
if(columnTargetMapped == null)
{
continue;
}
dtTarget.Columns.Add(columnTargetMapped);
foreach (DataRow drSource in dtSource.Rows)
{
var valueColumn = drSource[columnSource];
DataRow drTarget = dtTarget.NewRow();
drTarget[columnTargetMapped] = valueColumn;
dtTarget.Rows.Add(drTarget);
}
}
}
return dtTarget;
}
public static DataTable GetDataTablePart(this DataTable dt, params string[] ColumnNames)
{
var dtPart = new DataTable("TablePart");
var Names = new List<DataColumn>();
foreach (DataColumn Column in dt.Columns)
{
if(ColumnNames.Contains(Column.ColumnName))
{
Names.Add(Column);
}
}
dtPart.Columns.AddRange(Names.ToArray());
foreach(DataRow row in dt.Rows)
{
var NewRow = new object[Names.Count()];
var i = 0;
foreach (var Name in Names)
{
NewRow[i] = row[Name];
i = i + 1;
}
dtPart.LoadDataRow(NewRow, false);
}
return dtPart;
}
linq version....
public static DataTable GetDataTablePart(this DataTable dt, params string[] ColumnNames)
{
var RowCount = 0;
var dtPart = new DataTable("TablePart");
dtPart.Columns.AddRange((from column in dt.Columns.Cast<DataColumn>()
where ColumnNames.Contains(column.ColumnName)
select column).ToArray());
return (from row in dt.AsEnumerable()
let rowCount = RowCount = dt.Rows.Count
let RowValues = (from column in dtPart.Columns.Cast<DataColumn>()
select row[column]).ToArray()
let decCount = RowCount = RowCount - 1
where dtPart.LoadDataRow(RowValues,LoadOption.OverwriteChanges) != default && decCount > 0
select dtPart).FirstOrDefault();
}
I got roped into some old code, that uses loose (untyped) datasets all over the place.
I'm trying to write a helper method to find the DataTable.Name using the names of some columns.....(because the original code has checks for "sometimes we have 2 datatables in a dataset, sometimes 3, sometimes 4)..and its hard to know the order. Basically, the TSQL Select statements conditionally run. (Gaaaaaaaaaaaaaahhh).
Anyway. I wrote the below, and if I give it 2 column names, its matching on "any" columnname, not "all column names".
Its probably my linq skillz (again), and probably a simple fix.
But I've tried to get the syntax sugar down..below is one of the things I wrote, that compiles.
private static void DataTableFindStuff()
{
DataSet ds = new DataSet();
DataTable dt1 = new DataTable("TableOne");
dt1.Columns.Add("Table1Column11");
dt1.Columns.Add("Name");
dt1.Columns.Add("Age");
dt1.Columns.Add("Height");
DataRow row1a = dt1.NewRow();
row1a["Table1Column11"] = "Table1Column11_ValueA";
row1a["Name"] = "Table1_Name_NameA";
row1a["Age"] = "AgeA";
row1a["Height"] = "HeightA";
dt1.Rows.Add(row1a);
DataRow row1b = dt1.NewRow();
row1b["Table1Column11"] = "Table1Column11_ValueB";
row1b["Name"] = "Table1_Name_NameB";
row1b["Age"] = "AgeB";
row1b["Height"] = "HeightB";
dt1.Rows.Add(row1b);
ds.Tables.Add(dt1);
DataTable dt2 = new DataTable("TableTwo");
dt2.Columns.Add("Table2Column21");
dt2.Columns.Add("Name");
dt2.Columns.Add("BirthCity");
dt2.Columns.Add("BirthState");
DataRow row2a = dt2.NewRow();
row2a["Table2Column21"] = "Table2Column1_ValueG";
row2a["Name"] = "Table2_Name_NameG";
row2a["BirthCity"] = "BirthCityA";
row2a["BirthState"] = "BirthStateA";
dt2.Rows.Add(row2a);
DataRow row2b = dt2.NewRow();
row2b["Table2Column21"] = "Table2Column1_ValueH";
row2b["Name"] = "Table2_Name_NameH";
row2b["BirthCity"] = "BirthCityB";
row2b["BirthState"] = "BirthStateB";
dt2.Rows.Add(row2b);
ds.Tables.Add(dt2);
DataTable dt3 = new DataTable("TableThree");
dt3.Columns.Add("Table3Column31");
dt3.Columns.Add("Name");
dt3.Columns.Add("Price");
dt3.Columns.Add("QuantityOnHand");
DataRow row3a = dt3.NewRow();
row3a["Table3Column31"] = "Table3Column31_ValueM";
row3a["Name"] = "Table3_Name_Name00M";
row3a["Price"] = "PriceA";
row3a["QuantityOnHand"] = "QuantityOnHandA";
dt3.Rows.Add(row3a);
DataRow row3b = dt3.NewRow();
row3b["Table3Column31"] = "Table3Column31_ValueN";
row3b["Name"] = "Table3_Name_Name00N";
row3b["Price"] = "PriceB";
row3b["QuantityOnHand"] = "QuantityOnHandB";
dt3.Rows.Add(row3b);
ds.Tables.Add(dt3);
string foundDataTable1Name = FindDataTableName(ds, new List<string> { "Table1Column11", "Name" });
/* foundDataTable1Name should be 'TableOne' */
string foundDataTable2Name = FindDataTableName(ds, new List<string> { "Table2Column21", "Name" });
/* foundDataTable1Name should be 'TableTwo' */
string foundDataTable3Name = FindDataTableName(ds, new List<string> { "Table3Column31", "Name" });
/* foundDataTable1Name should be 'TableThree' */
string foundDataTableThrowsExceptionName = FindDataTableName(ds, new List<string> { "Name" });
/* show throw exception as 'Name' is in multiple (distinct) tables */
}
public static string FindDataTableName(DataSet ds, List<string> columnNames)
{
string returnValue = string.Empty;
DataTable foundDataTable = FindDataTable(ds, columnNames);
if (null != foundDataTable)
{
returnValue = foundDataTable.TableName;
}
return returnValue;
}
public static DataTable FindDataTable(DataSet ds, List<string> columnNames)
{
DataTable returnItem = null;
if (null == ds || null == columnNames)
{
return null;
}
List<DataTable> tables =
ds.Tables
.Cast<DataTable>()
.SelectMany
(t => t.Columns.Cast<DataColumn>()
.Where(c => columnNames.Contains(c.ColumnName))
)
.Select(c => c.Table).Distinct().ToList();
if (null != tables)
{
if (tables.Count <= 1)
{
returnItem = tables.FirstOrDefault();
}
else
{
throw new IndexOutOfRangeException(string.Format("FindDataTable found more than one matching Table based on the input column names. ({0})", String.Join(", ", columnNames.ToArray())));
}
}
return returnItem;
}
I tried this too (to no avail) (always has 0 matches)
List<DataTable> tables =
ds.Tables
.Cast<DataTable>()
.Where
(t => t.Columns.Cast<DataColumn>()
.All(c => columnNames.Contains(c.ColumnName))
)
.Distinct().ToList();
To me sounds like you're trying to see if columnNames passed to the method are contained within Column's name collection of Table. If that's the case, this should do the work.
List<DataTable> tables =
ds.Tables
.Cast<DataTable>()
.Where(dt => !columnNames.Except(dt.Columns.Select(c => c.Name)).Any())
.ToList();
(Below is an append by the asker of the question)
Well, I had to tweak it to make it compile, but you got me there..
Thanks.
Final Answer:
List<DataTable> tables =
ds.Tables.Cast<DataTable>()
.Where
(dt => !columnNames.Except(dt.Columns.Cast<DataColumn>()
.Select(c => c.ColumnName))
.Any()
)
.ToList();
Final Answer (which is not case sensitive):
List<DataTable> tables =
ds.Tables.Cast<DataTable>()
.Where
(dt => !columnNames.Except(dt.Columns.Cast<DataColumn>()
.Select(c => c.ColumnName), StringComparer.OrdinalIgnoreCase)
.Any()
)
.ToList();
Please look at what is wrong? I want to convert datarow to a string list.
public List<string> GetEmailList()
{
// get DataTable dt from somewhere.
List<DataRow> drlist = dt.AsEnumerable().ToList();
List<string> sEmail = new List<string>();
foreach (object str in drlist)
{
sEmail.Add(str.ToString()); // exception
}
return sEmail; // Ultimately to get a string list.
}
Thanks for help.
There's several problems here, but the biggest one is that you're trying to turn an entire row into a string, when really you should be trying to turn just a single cell into a string. You need to reference the first column of that DataRow, which you can do with brackets (like an array).
Try something like this instead:
public List<string> GetEmailList()
{
// get DataTable dt from somewhere.
List<string> sEmail = new List<string>();
foreach (DataRow row in dt.Rows)
{
sEmail.Add(row[0].ToString());
}
return sEmail; // Ultimately to get a string list.
}
Here's a one liner I got from ReSharper on how to do this, not sure of the performance implications, just thought I'd share it.
List<string> companyProxies = Enumerable.Select(
vendors.Tables[0].AsEnumerable(), vendor => vendor["CompanyName"].ToString()).ToList();
Here is how it would be with all additional syntax noise stripped: one-liner.
public List<string> GetEmailList()
{
return dt.AsEnumerable().Select(r => r[0].ToString()).ToList();
}
The Linq way ...
private static void Main(string[] args)
{
var dt = getdt();
var output = dt
.Rows
.Cast<DataRow>()
.ToList();
// or a CSV line
var csv = dt
.Rows
.Cast<DataRow>()
.Aggregate(new StringBuilder(), (sb, dr) => sb.AppendFormat(",{0}", dr[0]))
.Remove(0, 1);
Console.WriteLine(csv);
Console.ReadLine();
}
private static DataTable getdt()
{
var dc = new DataColumn("column1");
var dt = new DataTable("table1");
dt.Columns.Add(dc);
Enumerable.Range(0, 10)
.AsParallel()
.Select(i => string.Format("row {0}", i))
.ToList()
.ForEach(s =>
{
var dr = dt.NewRow();
dr[dc] = s;
dt.Rows.Add(dr);
});
return dt;
}