Develop Reference

How to use ChoETL to compare two CSV files for ADD, CHANGED or DELETED records (Master vs Detail)?

I've been playing with #Cinchoo's fantastic ETL system for C#. I need to compare two CSV files, where one CSV file is defined as a dynamically growing master table and the other is a feeder "detail" table.
The detail table may have differences in terms of NEW records, CHANGED records, or a record no longer (DELETED) existing in the master CSV file.
The output should be a 3rd table that replaces or updates the master table - so it's a growing CSV file.
Both tables have unique ID columns and a header row.
MASTER CSV
ID,name
1,Danny
2,Fred
3,Sam
DETAIL
ID,name
1,Danny
<-- record no longer exists
3,Pamela <-- name change
4,Fernando <-- new record
So far I've been referring to this fiddle, and the code below:
using System;
using ChoETL;
using System.Linq;
public class Program
{
public static void Main()
{
var input1 = ChoCSVReader.LoadText(csv1).WithFirstLineHeader().ToArray();
var input2 = ChoCSVReader.LoadText(csv2).WithFirstLineHeader().ToArray();
Console.WriteLine("NEW records\n");
using (var output = new ChoCSVWriter(Console.Out).WithFirstLineHeader())
{
output.Write(input2.OfType<ChoDynamicObject>().Except(input1.OfType<ChoDynamicObject>(),
new ChoDynamicObjectEqualityComparer(new string[] { "id" })));
}
Console.WriteLine("\n\nDELETED records\n");
using (var output = new ChoCSVWriter(Console.Out).WithFirstLineHeader())
{
output.Write(input1.OfType<ChoDynamicObject>().Except(input2.OfType<ChoDynamicObject>(),
new ChoDynamicObjectEqualityComparer(new string[] { "id" })));
}
Console.WriteLine("\n\nCHANGED records\n");
using (var output = new ChoCSVWriter(Console.Out).WithFirstLineHeader())
{
output.Write(input1.OfType<ChoDynamicObject>().Except(input2.OfType<ChoDynamicObject>(),
new ChoDynamicObjectEqualityComparer(new string[] { "id", "name" })));
}
}
static string csv1 = #"
ID,name
1,Danny
2,Fred
3,Sam";
static string csv2 = #"
ID,name
1,Danny
3,Pamela
4,Fernando";
}
OUTPUT
NEW records
ID,name
4,Fernando
DELETED records
ID,name
2,Fred
CHANGED records
ID,name
2,Fred
3,Sam
The CHANGED records is not working. As an added extra, I need a status so I want it to look like this:
CHANGED records
ID,name,status
1,Danny,NOCHANGE
2,Fred,DELETED
3,Pamela,CHANGED
4,Fernando,NEW
Thanks

Here is how you can do with Cinchoo ETL
string csv1 = #"ID,name
1,Danny
2,Fred
3,Sam";
string csv2 = #"ID,name
1,Danny
3,Pamela
4,Fernando";
var r1 = ChoCSVReader.LoadText(csv1).WithFirstLineHeader().ToArray();
var r2 = ChoCSVReader.LoadText(csv2).WithFirstLineHeader().ToArray();
using (var w = new ChoCSVWriter(Console.Out).WithFirstLineHeader())
{
var newItems = r2.OfType<ChoDynamicObject>().Except(r1.OfType<ChoDynamicObject>(), new ChoDynamicObjectEqualityComparer(new string[] { "ID" }))
.Select(r =>
{
var dict = r.AsDictionary();
dict["Status"] = "NEW";
return new ChoDynamicObject(dict);
}).ToArray();
var deletedItems = r1.OfType<ChoDynamicObject>().Except(r2.OfType<ChoDynamicObject>(), new ChoDynamicObjectEqualityComparer(new string[] { "ID" }))
.Select(r =>
{
var dict = r.AsDictionary();
dict["Status"] = "DELETED";
return new ChoDynamicObject(dict);
}).ToArray();
var changedItems = r2.OfType<ChoDynamicObject>().Except(r1.OfType<ChoDynamicObject>(), ChoDynamicObjectEqualityComparer.Default)
.Except(newItems.OfType<ChoDynamicObject>(), new ChoDynamicObjectEqualityComparer(new string[] { "ID" }))
.Select(r =>
{
var dict = r.AsDictionary();
dict["Status"] = "CHANGED";
return new ChoDynamicObject(dict);
}).ToArray();
var noChangeItems = r1.OfType<ChoDynamicObject>().Intersect(r2.OfType<ChoDynamicObject>(), ChoDynamicObjectEqualityComparer.Default)
.Select(r =>
{
var dict = r.AsDictionary();
dict["Status"] = "NOCHANGE";
return new ChoDynamicObject(dict);
}).ToArray();
var finalResult = Enumerable.Concat(newItems, deletedItems).Concat(changedItems).Concat(noChangeItems).OfType<dynamic>().OrderBy(r => r.ID);
w.Write(finalResult);
}
Console.WriteLine();
Output:
ID,name,Status
1,Danny,NOCHANGE
2,Fred,DELETED
3,Pamela,CHANGED
4,Fernando,NEW
Sample fiddle: https://dotnetfiddle.net/mrHpFx
UPDATE #1:
Above approach will work for small CSV files. For large CSV files, you must avoid it. Rather approach it in stream manner. Sample fiddle shows how (Not fully tested, but it gives direction to do it.)
Sample fiddle: https://dotnetfiddle.net/mh6w44
UPDATE #2:
Now Cinchoo ETL (v1.2.1.33) comes with built-in API to compare the CSV files in simplified manner
var r1 = ChoCSVReader.LoadText(csv1).WithFirstLineHeader().WithMaxScanRows(1).OfType<ChoDynamicObject>();
var r2 = ChoCSVReader.LoadText(csv2).WithFirstLineHeader().WithMaxScanRows(1).OfType<ChoDynamicObject>();
using (var w = new ChoCSVWriter(Console.Out).WithFirstLineHeader())
{
foreach (var t in r1.Compare(r2, "ID", "name" ))
{
dynamic v1 = t.MasterRecord as dynamic;
dynamic v2 = t.DetailRecord as dynamic;
if (t.Status == CompareStatus.Unchanged || t.Status == CompareStatus.Deleted)
{
v1.Status = t.Status.ToString();
w.Write(v1);
}
else
{
v2.Status = t.Status.ToString();
w.Write(v2);
}
}
}
Sample fiddle: https://dotnetfiddle.net/uPR5Sq

csvhelper: how to write a specific "cell" on an existing csv on C#?

I have a customer list in csv format which I'm using to send out emails. I would like to write to the CSV after each row has been executed in order to place a conditional rule. I'm using csvhelper to manipulate the file. Here's the code:
var scan = new StreamReader(myBlob);
var csvv = new CsvReader(scan, CultureInfo.InvariantCulture);
var records = csvv.GetRecords<Records>().ToList();
var scanwriter = new StreamWriter(myBlob4);
var csvwriter = new CsvWriter(scanwriter, CultureInfo.InvariantCulture);
foreach (Records record in records)
{
var from = new EmailAddress("example.com", "John");
var to = new EmailAddress(record.Email, record.Name);
var subject = "exapmple";
var msg = MailHelper.CreateSingleEmail(from, to, subject, txtf, htmlf);
StringBuilder text = new StringBuilder();
text.AppendFormat("sent", record.EmailSent);
csvwriter.WriteField(record.EmailSent);
csvwriter.NextRecord();
var response = await client.SendEmailAsync(msg);
}
However my csv is not appending the "sent" value to the file under the emailsent column. I'm using StringBuilder which might not be helpful in this scenario.

It seems like you are trying to do something more like this.
void Main()
{
var records = new List<SendEmail>
{
new SendEmail{ Email = "example.com", Name = "John" },
new SendEmail{ Email = "example2.com", Name = "Jenny" }
};
var csvwriter = new CsvWriter(Console.Out, CultureInfo.InvariantCulture);
foreach (var record in records)
{
// var from = new EmailAddress("example.com", "John");
// var to = new EmailAddress(record.Email, record.Name);
//
// var subject = "exapmple";
//
// var msg = MailHelper.CreateSingleEmail(from, to, subject, txtf, htmlf);
record.EmailSent = "sent";
csvwriter.WriteRecord(record);
csvwriter.NextRecord();
//var response = await client.SendEmailAsync(msg);
}
}
public class SendEmail
{
public string Email { get; set; }
public string Name { get; set; }
public string EmailSent { get; set; }
}

//using blocks will make sure the streams and disposed and file handles are closed properly,
// **even if an exception is thrown **
using(var scan = new StreamReader(myBlob))
using (var csvv = new CsvReader(scan, CultureInfo.InvariantCulture))
using (var scanwriter = new StreamWriter(myBlob4))
using (var csvwriter = new CsvWriter(scanwriter, CultureInfo.InvariantCulture))
{
var records = csvv.GetRecords<Records>(); //ToList() was not needed or helpful here
foreach (var record in records)
{
var from = new EmailAddress("example.com", "John");
var to = new EmailAddress(record.Email, record.Name);
var subject = "example";
var msg = MailHelper.CreateSingleEmail(from, to, subject, txtf, htmlf);
csvwriter.WriteField($"sent {record.EmailSent}");
csvwriter.NextRecord();
var response = await client.SendEmailAsync(msg);
}
}

Write Text file with tab-delimited in Asp.Net Core 2.2

Hi do you have any guides, work aid or step by step how to export to text with tab delimited. Im using Asp.Net Core 2.2 MVC EF. I want to export a list from my table.. I want to have a button where the user click in this DownloadFile Action will trigger.
public IActionResult DownloadFile()
{
var payments = new List<BdoPE>
{
new BdoPE
{
DocDateInDoc = "01/01/2019",
DocType = "DZ",
CompanyCode = "3000",
PosDateInDoc = "01/01/2019",
FiscalPeriod = "01",
CurrentKey = "PHP",
RefDocNum = "Over-The-Counter",
DocHeadT = "BDO",
PosKeyInNextLine = "40",
AccMatNextLine = "11231131",
AmountDocCur = "0000000010050",
ValDate = "01/01/2019",
AssignNum = "EEA",
ItemText = "1000136212 ",
PosKeyInNextLine2 = "15",
AccMatNextLine2 = "0115027FF",
AmountDocCur2 = "0000000010050",
BaseDateDueCal = "01/01/2019",
ItemText2 = "1000136212"
},
};
// I want this part to let the user select where they want to save the text file.
using (var writer = new StreamWriter("path\\to\\file.txt")) // not static location like this one.
using (var csv = new CsvWriter(writer))
{
csv.WriteHeader<BdoPE>();
csv.WriteRecord(payments);
}
// where should i put the delimiter part?
return;
}

You will need to setup the CsvWriter with a Configuration.
Thus, your code needs only a slight change:
[...]
var configuration = new CsvHelper.Configuration.Configuration();
configuration.Delimiter = '\t';
using (var csv = new CsvWriter(writer, configuration))
{
csv.WriteHeader<BdoPE>();
csv.WriteRecord(payments);
}
[...]

I use the code below to set the Delimiter using CsvHelper.
var config = new CsvConfiguration(CultureInfo.CurrentCulture)
{
Delimiter = "\t"
};

StringBuilder within IEnumerable

I have a ControlMeasure table that holds information on each control measure and a ControlMeasurepeopleExposed Table that holds a record for each person exposed in the control measure this could be 1 record or many records.
I Have a controller that populates a List view
For each item in the list, Control Measure, I would like to create a string that shows all the People at risk
e.g.
PeopleString = "Employees, Public, Others";
Ive added a foreach in the controller to show what I'm trying to do however I'm aware that this wont work.
The controller is this:
public ActionResult ControlMeasureList(int raId)
{
//Populate the list
var hazards = new List<Hazard>(db.Hazards);
var controlMeasures = new List<ControlMeasure>(db.ControlMeasures).Where(x => x.RiskAssessmentId == raId);
var cmcombined = (
from g in hazards
join f in controlMeasures
on new { g.HazardId } equals new { f.HazardId }
select new CMCombined
{
Activity = f.Activity,
ControlMeasureId = f.ControlMeasureId,
ExistingMeasure = f.ExistingMeasure,
HazardName = g.Name,
LikelihoodId = f.LikelihoodId,
Rating = f.Rating,
RiskAssessmentId = f.RiskAssessmentId,
SeverityId = f.SeverityId,
}).OrderBy(x => x.Activity).ToList();
var cmPeopleExp = new List<ControlMeasurePeopleExposed>(db.ControlMeasurePeopleExposeds).Where(x => x.RiskAssessmentId == raId);
var peopleExp = from c in cmPeopleExp
join d in db.PeopleExposeds
on c.PeopleExposedId equals d.PeopleExposedId
orderby d.Name
select new RAPeopleExp
{
RAPeopleExpId = c.PeopleExposedId,
PeopleExpId = c.PeopleExposedId,
PeopleExpName = d.Name,
RiskAssessmentId = c.RiskAssessmentId,
ControlMeasureId = c.ControlMeasureId
};
var model = cmcombined.Select(t => new FullControlMeasureListViewModel
{
ControlMeasureId = t.ControlMeasureId,
HazardName = t.HazardName,
LikelihoodId = t.LikelihoodId,
Rating = t.Rating,
SeverityId = t.SeverityId,
Activity = t.Activity,
ExCM = t.ExistingMeasure,
//This section here is where I'm struggling
var PeopleString = new StringBuilder();
foreach (var p in peopleExp)
{
PeopleString.AppendLine(p.PeopleName);
{
PeopleExposed = PeopleString,
});
return PartialView("_ControlMeasureList", model);
}
I know I cant directly put this code in the controller but it does represent what I want to do.

You can't foreach within an object initializer (which is what you're trying to do when instantiating FullControlMeasureListViewModel). You can, however, use a combination of string.Join and peopleExp.Select:
var model = cmcombined.Select(t => new FullControlMeasureListViewModel
{
//other props
PeopleExposed = string.Join(",", peopleExp
.Where(p => p.ControlMeasureId == t.ControlMeasureId)
.Select(p => p.PeopleExpName));
//other props
});

Reading the XML values using LINQ

what is the best way of reading xml file using linq and the below code you will see that, I have three different loops and I feel like its not elegant or do I have options to retrofit the below code?
public static void readXMLOutput(Stream stream)
{
XDocument xml = new XDocument();
xml = LoadFromStream(stream);
var header = from p in xml.Elements("App").Elements("Application")
select p;
foreach (var record in header)
{
string noym = record.Element("nomy").Value;
string Description = record.Element("Description").Value;
string Name = record.Element("Name").Value;
string Code = record.Element("Code").Value;
}
var appRoles = from q in xml.Elements("App").Elements("Application").Elements("AppRoles").Elements("Role")
select q;
foreach (var record1 in appRoles)
{
string Name = record1.Element("Name").Value;
string modifiedName = record1.Element("ModifiedName").Value;
}
var memeber = from r in xml.Elements("App").Elements("Application").Elements("AppRoles").Elements("Role").Elements("Members")
select r;
foreach (var record2 in memeber)
{
string ExpirationDate = record2.Element("ExpirationDate").Value;
string FullName = record2.Element("FullName").Value;
}
}
UPDATED:
foreach (var record in headers)
{
..............
string Name1 = record.Attribute("Name").Value;
string UnmodifiedName = record.Attribute("UnmodifiedName").Value;
string ExpirationDate = record.Attribute("ExpirationDate").Value;
string FullName = record.Attribute("FullName").Value;
...............
}

Is that your actual code ? All those string variables you are assigning in the foreach loops only have a scope of one iteration of the loop. They are created and destroyed each time.

This may not work precisely in your case depending on the xml structure. Play around with it. Try it using LinqPad
var applications = from p in xml.Descendants("Application")
select new { Nomy = p.Element("nomy").Value
, Description = p.Element("Description").Value
, Name = p.Element("Name").Value
, Code = p.Element("Code").Value
};
var appRoles = from r in xml.Descendants("Role")
select new { Name = r.Element("Name").Value
, ModifiedName = r.Element("ModifiedName").Value
};

This answer is a hierarchical query.
var headers =
from header in xml.Elements("App").Elements("Application")
select new XElement("Header",
new XAttribute("noym", header.Element("nomy").Value),
new XAttribute("Description", header.Element("Description").Value),
new XAttribute("Name", header.Element("Name").Value),
new XAttribute("Code", header.Element("Code").Value),
from role in header.Elements("AppRoles").Elements("Role")
select new XElement("Role",
new XAttribute("Name", role.Element("Name").Value),
new XAttribute("ModifiedName", role.Element("ModifiedName").Value),
from member in role.Elements("Members")
select new XElement("Member",
new XAttribute("ExpirationDate", member.Element("ExpirationDate").Value),
new XAttribute("FullName", member.Element("FullName").Value)
)
)
);