Develop Reference

Remove duplicates data from a dataset based on 2 column values

Here is my code. I just want to compare two data sets columns and exclude same records. I tried this
code. It exclude the same records but it also generated duplicate data.
for (int i = 0; i < ds2. Tables[0].Rows.Count; i++)
{
for each (Data Row dr2 in ds1. Tables[0].Rows)
{
string table2 = dr2["code"].ToString();
if (i.ToString() != table2.ToString())
{
dsnew. Tables[0].Rows.Add(i);
}
}
}

strange comparison. you don't seem to be using the records in ds2. only the count of ds2.
i think you are getting duplicates because you are
comparing each element in ds2 with 1 element in ds1 before deciding to place it in dsnew
try
comparing each element in ds2 with all elements in ds1 before deciding to place it in dsnew
for (int i = 0; i < ds2.Tables[0].Rows.Count; i++)
{
bool found = false;
foreach (DataRow dr2 in ds1.Tables[0].Rows)
{
string table2 = dr2["code"].ToString();
if (i.ToString() == table2.ToString())
{
found = true;
}
}
if(!found) dsnew.Tables[0].Rows.Add(i);
}

How to check items in CheckedListBox based on DataTable value in WinForms?

I fill my CheckedListBox using DataSource
CheckedListBox.DataSource = this.datatable;
CheckedListBox.DisplayMember = "Name";
CheckedListBox.ValueMember = "Id";
I aim to check items based on datatable values with where clause
I'm trying to use this code
foreach (DataRow dr in datatable.Rows)
{
for (int i = 0; i < CheckedListBox.Items.Count; i++)
{
if ( dr["Id"].ToString().Cast<DataRow>().Where(x => x.Field<int>("State") == 1).ToString() == ((DataRowView)CheckedListBox.Items[i])[0].ToString())
{
checkboxlist.SetItemChecked(i, true);
}
}
}
But it doesn't work.
If i remove this
.Cast<DataRow>().Where(x => x.Field<int>("State") == 1).ToString()
The code is working but with checking all items.
So, how can i check a specific items that refer to "Id" in datatable that equal to value of CheckedListBox and column "state" in datatable == 1?
Any help ..?

I found the solution in 2 ways
The long way: is to create another DataTable and insert the filtered Ids into it, then call the new DataTable in the foreach, Here is the code:
DataTable dtId = datatable.Clone();
var rows = datatable.AsEnumerable()
.Where(x => x.Field<bool>(2) == true).ToList();
foreach (var row in rows)
{
dtId.ImportRow(row);
}
foreach (DataRow dr in dtId.Rows)
{
for (int i = 0; i < CheckedListBox.Items.Count; i++)
{
if ( dr["Id"].ToString() == ((DataRowView)CheckedListBox.Items[i])[0].ToString())
{
CheckedListBox.SetItemChecked(i, true);
}
}
}
And the simple way that i use is to change this line:
if ( dr["Id"].ToString().Cast<DataRow>().Where(x => x.Field<int>("State") == 1).ToString() == ((DataRowView)CheckedListBox.Items[i])[0].ToString())
to this:
if (Convert.ToBoolean(dr[2].ToString()) == true && dr[0].ToString() == ((DataRowView)CheckedListBox.Items[i])[0].ToString())

How to read data fast from an excel and convert it to list from file stream

I am using EPPlus.
The excel I am uploading has column headers in row number 2 . And from row 4 onward it has the data which may vary up to 2k records.
The way I am doing it , it takes a lot of time for reading 2k records and putting to a list .
using (var excel = new ExcelPackage(hpf.InputStream))
{
var ws = excel.Workbook.Worksheets["Sheet1"];
//Read the file into memory
for (int rw = 4; rw <= ws.Dimension.End.Row; rw++)
{
if (!ws.Cells[rw, 1, rw, 24].All(c => c.Value == null))
{
int headerRow = 2;
GroupMembershipUploadInput gm = new GroupMembershipUploadInput();
for (int col = ws.Dimension.Start.Column; col <= ws.Dimension.End.Column; col++)
{
var s = ws.Cells[rw, col].Value;
if (ws.Cells[headerRow, col].Value.ToString().Equals("Existing Constituent Master Id"))
{
gm.cnst_mstr_id = (ws.Cells[rw, col].Value ?? (Object)"").ToString();
}
else if (ws.Cells[headerRow, col].Value.ToString().Equals("Prefix of the constituent(Mr, Mrs etc)"))
{
gm.cnst_prefix_nm = (ws.Cells[rw, col].Value ?? (Object)"").ToString();
}
}
lgl.GroupMembershipUploadInputList.Add(gm);
}
}
GroupMembershipUploadInputList is the list of objects of type GroupMembershipUploadInput that I am adding the excel values to after reading from the cell wise.
Can it be done faster ? What am I missing here ?
Please help to improve the performance.

You are making a lot iterations there, for each row, you visit each column twice. I assume that you only need those two values per row and if so the following code would reduce time drastically:
using (var excel = new ExcelPackage(hpf.InputStream))
{
var ws = excel.Workbook.Worksheets["Sheet1"];
int headerRow = 2;
// hold the colum index based on the value in the header
int col_cnst_mstr_id = 2;
int col_cnst_prefix_nm = 4;
// loop once over the columns to fetch the column index
for (int col = ws.Dimension.Start.Column; col <= ws.Dimension.End.Column; col++)
{
if ("Existing Constituent Master Id".Equals(ws.Cells[headerRow, col].Value))
{
col_cnst_mstr_id = col;
}
if ("Prefix of the constituent(Mr, Mrs etc)".Equals(ws.Cells[headerRow, col].Value))
{
col_cnst_prefix_nm = col;
}
}
//Read the file into memory
// loop over all rows
for (int rw = 4; rw <= ws.Dimension.End.Row; rw++)
{
// check if both values are not null
if (ws.Cells[rw, col_cnst_mstr_id].Value != null &&
ws.Cells[rw, col_cnst_prefix_nm].Value != null)
{
// the correct cell will be selcted based on the column index
var gm = new GroupMembershipUploadInput
{
cnst_mstr_id = (string) ws.Cells[rw, col_cnst_mstr_id].Value ?? String.Empty,
cnst_prefix_nm = (string) ws.Cells[rw, col_cnst_prefix_nm].Value ?? String.Empty
};
lgl.GroupMembershipUploadInputList.Add(gm);
}
}
}
I removed the inner column loop and moved it to the start of the method. There it is used to just get the columnindex for each field you're interested in. The expensive null check can now also be reduced. To fetch the value, all that is now needed is a simple index lookup in the row.

Improve performance in converting large execl file to list of objects

I know this question have been asked multiple times . But I could not find much help from anyone of those.
I don't want to convert the excel into data table but I want it to be converted to a list of objects and sent to server side for processing.
If it has more than 2K rows it should throw an error. Currently what I am doing is something like :
using (var excel = new ExcelPackage(hpf.InputStream))
{
var ws = excel.Workbook.Worksheets["Sheet1"];
for (int rw = 4; rw <= ws.Dimension.End.Row; rw++)
{
if (ws.Cells[rw, 1].Value != null)
{
int headerRow = 2;
GroupMembershipUploadInput gm = new GroupMembershipUploadInput();
for (int col = ws.Dimension.Start.Column; col <= ws.Dimension.End.Column; col++)
{
var s = ws.Cells[rw, col].Value;
if (ws.Cells[headerRow, col].Value.ToString().Equals("Existing Constituent Master Id"))
{
gm.cnst_mstr_id = (ws.Cells[rw, col].Value ?? (Object)"").ToString();
}
else if (ws.Cells[headerRow, col].Value.ToString().Equals("Prefix of the constituent(Mr, Mrs etc)"))
{
gm.cnst_prefix_nm = (ws.Cells[rw, col].Value ?? (Object)"").ToString();
}
else if (ws.Cells[headerRow, col].Value.ToString().Equals("First Name of the constituent(Mike)"))
{
gm.cnst_first_nm = (ws.Cells[rw, col].Value ?? (Object)"").ToString();
}
.....................
.....................
}
}
iUploadedCnt = iUploadedCnt + 1; //Increase the count by 1
}
if (lgl.GroupMembershipUploadInputList.Count < 2003) //Check for the uploaded list count
{
//throw the error
}
But this approach seems slow.
Conversion of the excel to list seems slow to me. For example , when I upload more than 2k records , the list gets converted first to list and then the count is checked if more than 2003 . This process is definitely slower.
How can it be achieved in a faster /better way ?

You do a lot of repeated string processing which is unnecessary. For each row you check the column headers again if they fit some predefined value. (for instance if (ws.Cells[headerRow, col].Value.ToString().Equals("Existing Constituent Master Id")).
You could do this once before you start parsing all rows and create for instance a Dictionary<int, SomeEnum> which maps the column number to a specific enum value. When parsing the rows you then can make a quick lookup in the dictionary, which column maps to which property.
Furthermore, you define a var s = ws.Cells[rw, col].Value; but never use it. Instead, you read this cell value again, when you assign it to a property of your object. You could just make the necessary conversions and checks here, and then use only s;
// define this enum somewhere
enum ColumPropEnum {
cnst_mstr_id, cnst_prefix_nm, ...
}
//define this prop somewhere
Dictionary<int, ColumnPropEnum> colprops = new Dictionary<int, ColumnPropEnum>();
//do this once before processing all rows
for (int col = ws.Dimension.Start.Column; col <= ws.Dimension.End.Column; col++) {
if (ws.Cells[headerRow, col].Value.ToString().Equals("Existing Constituent Master Id"))
colprops.Add(col, ColumnPropEnum.cnst_mstr_id);
else if (ws.Cells[headerRow, col].Value.ToString().Equals(" ..."))
colprops.Add(col, ColumnPropEnum.cnst_prefix_nm);
...
}
//now use this dictionary in each row
for (int rw = 4; rw <= ws.Dimension.End.Row; rw++)
{
....
for (int col = ws.Dimension.Start.Column; col <= ws.Dimension.End.Column; col++) {
//the single ? checks, whether the Value is null, if yes it returns null, otherwise it returns ToString(). Then the double ?? checks whether the result if the operation is null, if yes, it assigns "" to s, otherwise the result of ToString();
var s = ws.Cells[rw, col].Value?.ToString() ?? "";
ColumnPropEnum cp;
if (colpros.TryGetValue(col, out cp)) {
switch (cp) {
case cnst_mstr_id: gm.cnst_mstr_id = s; break;
case cnst_prefix_nm: gm.cnst_prefix_nm = s; break;
...
}
}
}
}
I'm not sure at which position you add this object to a list or upload it to the server, as this is not part of the code. But it could be faster, to first check only the first column of each row if you have the necessary count of non-null values and throw an error if not and do all the other processing only if you didn't throw the error.
int rowcount = 0;
//If you need at minimum 2000 rows, you can stop after you count 2000 valid rows
for (int rw = 4; rw <= ws.Dimension.End.Row && rowcount < 2000; rw++)
{
if (ws.Cells[rw, 1].Value != null) rowcount++
}
if (rowcount < 2000) {
//throw error and return
}
//else do the list building and uploading

Remove blank row from dataset

ds.Tables.Add(dt);
return ds;
In the above code snippet, how can i return my dataset but exclude all blank rows i.e blank meaning rows with null or an empty string in all their columns.

You will have to do that checking before hand and then return the DataTable something like below (an example)
for (int i = dt.Rows.Count - 1; i >= 0; i--)
{
if (dt.Rows[i]["col1"] == DBNull.Value && dt.Rows[i]["col2"] == DBNull.Value)
{
dt.Rows[i].Delete();
}
}
dt.AcceptChanges();
ds.Tables.Add(dt);
return ds;

In case anyone stumbles across this article, this is the solution I came up with:
// REMOVE ALL EMPTY ROWS
dt_Parsed.Rows.Cast<DataRow>().ToList().FindAll(Row =>
{ return String.IsNullOrEmpty(String.Join("", Row.ItemArray)); }).ForEach(Row =>
{ dt_Parsed.Rows.Remove(Row); });

Here had helper function in which pass your table that you want to delete datarow with all empty columns(Here I assumed all string are of type string then it will work)
For other type u can check datacolumn type and then can make relavant checking.
public DataTable DeleteEmptyRows(DataTable dt)
{
DataTable formattedTable = dt.Copy();
List<DataRow> drList = new List<DataRow>();
foreach (DataRow dr in formattedTable.Rows)
{
int count = dr.ItemArray.Length;
int nullcounter=0;
for (int i = 0; i < dr.ItemArray.Length; i++)
{
if (dr.ItemArray[i] == null || string.IsNullOrEmpty(Convert.ToString(dr.ItemArray[i])))
{
nullcounter++;
}
}
if (nullcounter == count)
{
drList.Add(dr);
}
}
for (int i = 0; i < drList.Count; i++)
{
formattedTable.Rows.Remove(drList[i]);
}
formattedTable.AcceptChanges();
return formattedTable;
}

You can try to loop the DataTables in DataSet with this method:
public void Clear_DataTableEmptyRows(DataTable dataTableControl)
{
for (int i = dataTableControl.Rows.Count - 1; i >= 0; i--)
{
DataRow currentRow = dataTableControl.Rows[i];
foreach (var colValue in currentRow.ItemArray)
{
if (!string.IsNullOrEmpty(colValue.ToString()))
break;
dataTableControl.Rows[i].Delete();
break;
}
}
}