Related
I've been playing with #Cinchoo's fantastic ETL system for C#. I need to compare two CSV files, where one CSV file is defined as a dynamically growing master table and the other is a feeder "detail" table.
The detail table may have differences in terms of NEW records, CHANGED records, or a record no longer (DELETED) existing in the master CSV file.
The output should be a 3rd table that replaces or updates the master table - so it's a growing CSV file.
Both tables have unique ID columns and a header row.
MASTER CSV
ID,name
1,Danny
2,Fred
3,Sam
DETAIL
ID,name
1,Danny
<-- record no longer exists
3,Pamela <-- name change
4,Fernando <-- new record
So far I've been referring to this fiddle, and the code below:
using System;
using ChoETL;
using System.Linq;
public class Program
{
public static void Main()
{
var input1 = ChoCSVReader.LoadText(csv1).WithFirstLineHeader().ToArray();
var input2 = ChoCSVReader.LoadText(csv2).WithFirstLineHeader().ToArray();
Console.WriteLine("NEW records\n");
using (var output = new ChoCSVWriter(Console.Out).WithFirstLineHeader())
{
output.Write(input2.OfType<ChoDynamicObject>().Except(input1.OfType<ChoDynamicObject>(),
new ChoDynamicObjectEqualityComparer(new string[] { "id" })));
}
Console.WriteLine("\n\nDELETED records\n");
using (var output = new ChoCSVWriter(Console.Out).WithFirstLineHeader())
{
output.Write(input1.OfType<ChoDynamicObject>().Except(input2.OfType<ChoDynamicObject>(),
new ChoDynamicObjectEqualityComparer(new string[] { "id" })));
}
Console.WriteLine("\n\nCHANGED records\n");
using (var output = new ChoCSVWriter(Console.Out).WithFirstLineHeader())
{
output.Write(input1.OfType<ChoDynamicObject>().Except(input2.OfType<ChoDynamicObject>(),
new ChoDynamicObjectEqualityComparer(new string[] { "id", "name" })));
}
}
static string csv1 = #"
ID,name
1,Danny
2,Fred
3,Sam";
static string csv2 = #"
ID,name
1,Danny
3,Pamela
4,Fernando";
}
OUTPUT
NEW records
ID,name
4,Fernando
DELETED records
ID,name
2,Fred
CHANGED records
ID,name
2,Fred
3,Sam
The CHANGED records is not working. As an added extra, I need a status so I want it to look like this:
CHANGED records
ID,name,status
1,Danny,NOCHANGE
2,Fred,DELETED
3,Pamela,CHANGED
4,Fernando,NEW
Thanks
Here is how you can do with Cinchoo ETL
string csv1 = #"ID,name
1,Danny
2,Fred
3,Sam";
string csv2 = #"ID,name
1,Danny
3,Pamela
4,Fernando";
var r1 = ChoCSVReader.LoadText(csv1).WithFirstLineHeader().ToArray();
var r2 = ChoCSVReader.LoadText(csv2).WithFirstLineHeader().ToArray();
using (var w = new ChoCSVWriter(Console.Out).WithFirstLineHeader())
{
var newItems = r2.OfType<ChoDynamicObject>().Except(r1.OfType<ChoDynamicObject>(), new ChoDynamicObjectEqualityComparer(new string[] { "ID" }))
.Select(r =>
{
var dict = r.AsDictionary();
dict["Status"] = "NEW";
return new ChoDynamicObject(dict);
}).ToArray();
var deletedItems = r1.OfType<ChoDynamicObject>().Except(r2.OfType<ChoDynamicObject>(), new ChoDynamicObjectEqualityComparer(new string[] { "ID" }))
.Select(r =>
{
var dict = r.AsDictionary();
dict["Status"] = "DELETED";
return new ChoDynamicObject(dict);
}).ToArray();
var changedItems = r2.OfType<ChoDynamicObject>().Except(r1.OfType<ChoDynamicObject>(), ChoDynamicObjectEqualityComparer.Default)
.Except(newItems.OfType<ChoDynamicObject>(), new ChoDynamicObjectEqualityComparer(new string[] { "ID" }))
.Select(r =>
{
var dict = r.AsDictionary();
dict["Status"] = "CHANGED";
return new ChoDynamicObject(dict);
}).ToArray();
var noChangeItems = r1.OfType<ChoDynamicObject>().Intersect(r2.OfType<ChoDynamicObject>(), ChoDynamicObjectEqualityComparer.Default)
.Select(r =>
{
var dict = r.AsDictionary();
dict["Status"] = "NOCHANGE";
return new ChoDynamicObject(dict);
}).ToArray();
var finalResult = Enumerable.Concat(newItems, deletedItems).Concat(changedItems).Concat(noChangeItems).OfType<dynamic>().OrderBy(r => r.ID);
w.Write(finalResult);
}
Console.WriteLine();
Output:
ID,name,Status
1,Danny,NOCHANGE
2,Fred,DELETED
3,Pamela,CHANGED
4,Fernando,NEW
Sample fiddle: https://dotnetfiddle.net/mrHpFx
UPDATE #1:
Above approach will work for small CSV files. For large CSV files, you must avoid it. Rather approach it in stream manner. Sample fiddle shows how (Not fully tested, but it gives direction to do it.)
Sample fiddle: https://dotnetfiddle.net/mh6w44
UPDATE #2:
Now Cinchoo ETL (v1.2.1.33) comes with built-in API to compare the CSV files in simplified manner
var r1 = ChoCSVReader.LoadText(csv1).WithFirstLineHeader().WithMaxScanRows(1).OfType<ChoDynamicObject>();
var r2 = ChoCSVReader.LoadText(csv2).WithFirstLineHeader().WithMaxScanRows(1).OfType<ChoDynamicObject>();
using (var w = new ChoCSVWriter(Console.Out).WithFirstLineHeader())
{
foreach (var t in r1.Compare(r2, "ID", "name" ))
{
dynamic v1 = t.MasterRecord as dynamic;
dynamic v2 = t.DetailRecord as dynamic;
if (t.Status == CompareStatus.Unchanged || t.Status == CompareStatus.Deleted)
{
v1.Status = t.Status.ToString();
w.Write(v1);
}
else
{
v2.Status = t.Status.ToString();
w.Write(v2);
}
}
}
Sample fiddle: https://dotnetfiddle.net/uPR5Sq
Hi do you have any guides, work aid or step by step how to export to text with tab delimited. Im using Asp.Net Core 2.2 MVC EF. I want to export a list from my table.. I want to have a button where the user click in this DownloadFile Action will trigger.
public IActionResult DownloadFile()
{
var payments = new List<BdoPE>
{
new BdoPE
{
DocDateInDoc = "01/01/2019",
DocType = "DZ",
CompanyCode = "3000",
PosDateInDoc = "01/01/2019",
FiscalPeriod = "01",
CurrentKey = "PHP",
RefDocNum = "Over-The-Counter",
DocHeadT = "BDO",
PosKeyInNextLine = "40",
AccMatNextLine = "11231131",
AmountDocCur = "0000000010050",
ValDate = "01/01/2019",
AssignNum = "EEA",
ItemText = "1000136212 ",
PosKeyInNextLine2 = "15",
AccMatNextLine2 = "0115027FF",
AmountDocCur2 = "0000000010050",
BaseDateDueCal = "01/01/2019",
ItemText2 = "1000136212"
},
};
// I want this part to let the user select where they want to save the text file.
using (var writer = new StreamWriter("path\\to\\file.txt")) // not static location like this one.
using (var csv = new CsvWriter(writer))
{
csv.WriteHeader<BdoPE>();
csv.WriteRecord(payments);
}
// where should i put the delimiter part?
return;
}
You will need to setup the CsvWriter with a Configuration.
Thus, your code needs only a slight change:
[...]
var configuration = new CsvHelper.Configuration.Configuration();
configuration.Delimiter = '\t';
using (var csv = new CsvWriter(writer, configuration))
{
csv.WriteHeader<BdoPE>();
csv.WriteRecord(payments);
}
[...]
I use the code below to set the Delimiter using CsvHelper.
var config = new CsvConfiguration(CultureInfo.CurrentCulture)
{
Delimiter = "\t"
};
"0.0.0.0,""0.255.255.255"",""ZZ"""
"1.0.0.0,""1.0.0.255"",""AU"""
"1.0.1.0,""1.0.3.255"",""CN"""
"1.0.4.0,""1.0.7.255"",""AU"""
"1.0.8.0,""1.0.15.255"",""CN"""
"1.0.16.0,""1.0.31.255"",""JP"""
"1.0.32.0,""1.0.63.255"",""CN"""
"1.0.64.0,""1.0.127.255"",""JP"""
"1.0.128.0,""1.0.255.255"",""TH"""
"1.1.0.0,""1.1.0.255"",""CN"""
"1.1.1.0,""1.1.1.255"",""AU"""
"1.1.2.0,""1.1.63.255"",""CN"""
"1.1.64.0,""1.1.127.255"",""JP"""
"1.1.128.0,""1.1.255.255"",""TH"""
İN EXCEL
0.0.0.0,"0.255.255.255","ZZ"
1.0.0.0,"1.0.0.255","AU"
1.0.1.0,"1.0.3.255","CN"
1.0.4.0,"1.0.7.255","AU"
1.0.8.0,"1.0.15.255","CN"
1.0.16.0,"1.0.31.255","JP"
1.0.32.0,"1.0.63.255","CN"
1.0.64.0,"1.0.127.255","JP"
1.0.128.0,"1.0.255.255","TH"
1.1.0.0,"1.1.0.255","CN"
1.1.1.0,"1.1.1.255","AU"
1.1.2.0,"1.1.63.255","CN"
1.1.64.0,"1.1.127.255","JP"
1.1.128.0,"1.1.255.255","TH"
1.2.0.0,"1.2.2.255","CN"
1.2.3.0,"1.2.3.255","AU"
1.2.4.0,"1.2.127.255","CN"
1.2.128.0,"1.2.255.255","TH"
1.3.0.0,"1.3.255.255","CN"
1.4.0.0,"1.4.0.255","AU"
1.4.1.0,"1.4.127.255","CN"
1.4.128.0,"1.4.255.255","TH"
How can split this CSV file.
For example 0.0.0.0 0.255.255.255 ZZ for first row and how can add datagridview with 3columns
You can do it via the following way..
using System.IO;
static void Main(string[] args)
{
using(var reader = new StreamReader(#"C:\test.csv"))
{
List<string> listA = new List<string>();
List<string> listB = new List<string>();
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
var values = line.Split(','); // or whatever yur get by reading that file
listA.Add(values[0]);
listB.Add(values[1]);
}
}
}
A CSV file is either a Tab delimited or a Comma delimited file. That said; you have to read the file line by line and then separate the values available in a line based on the delimiter character. The first line usually appears in a CSV file is usually the headers which you can use in order to produce a KeyValue pair to make your collection more efficient. For example:
Dictionary<int, Dictionary<String, String>> values = new Dictionary<int, Dictionary<String,String>>();
using(FileStream fileStream = new FileStream(#"D:\MyCSV.csv", FileMode.Open, FileAccess.Read, FileShare.Read)) {
using(StreamReader streamReader = new StreamReader(fileStream)){
//You can skip this line if there is no header
// Then instead of Dictionary<String,String> you use List<String>
var headers = streamReader.ReadLine().Split(',');
String line = null;
int lineNumber = 1;
while(!streamReader.EndOfStream){
line = streamReader.ReadLine().split(',');
if(line.Length == headers.Length){
var temp = new Dictionary<String, String>();
for(int i = 0; i < headers.Length; i++){
// You can remove '"' character by line[i].Replace("\"", "") or through using the Substring method
temp.Add(headers[i], line[i]);
}
values.Add(lineNumber, temp);
}
lineNumber++;
}
}
In case the data structure of your CSV is constant and it will not change in the future, you can develop a strongly typed data model and get rid of the Dictionary type. This approach will be more elegant and more efficient.
First of all, your CSV lines are surrounded by quotes. Is it copy/paste mistake? If not, you will need to sanitize the file to a valid CSV file.
You can try Cinchoo ETL - an open source library to load the CSV file to datatable, then you can assign it to your DataGridView source.
I'll show you both approach, how to handle
Valid CSV: (test.csv)
0.0.0.0,"0.255.255.255","ZZ"
1.0.0.0,"1.0.0.255","AU"
1.0.1.0,"1.0.3.255","CN"
1.0.4.0,"1.0.7.255","AU"
1.0.8.0,"1.0.15.255","CN"
1.0.16.0,"1.0.31.255","JP"
1.0.32.0,"1.0.63.255","CN"
1.0.64.0,"1.0.127.255","JP"
1.0.128.0,"1.0.255.255","TH"
1.1.0.0,"1.1.0.255","CN"
1.1.1.0,"1.1.1.255","AU"
1.1.2.0,"1.1.63.255","CN"
1.1.64.0,"1.1.127.255","JP"
1.1.128.0,"1.1.255.255","TH"
Read CSV:
using (var p = new ChoCSVReader("test.csv"))
{
var dt = p.AsDataTable();
//Assign dt to DataGridView
}
Next approach
Invalid CSV: (test.csv)
"0.0.0.0,""0.255.255.255"",""ZZ"""
"1.0.0.0,""1.0.0.255"",""AU"""
"1.0.1.0,""1.0.3.255"",""CN"""
"1.0.4.0,""1.0.7.255"",""AU"""
"1.0.8.0,""1.0.15.255"",""CN"""
"1.0.16.0,""1.0.31.255"",""JP"""
"1.0.32.0,""1.0.63.255"",""CN"""
"1.0.64.0,""1.0.127.255"",""JP"""
"1.0.128.0,""1.0.255.255"",""TH"""
"1.1.0.0,""1.1.0.255"",""CN"""
"1.1.1.0,""1.1.1.255"",""AU"""
"1.1.2.0,""1.1.63.255"",""CN"""
"1.1.64.0,""1.1.127.255"",""JP"""
"1.1.128.0,""1.1.255.255"",""TH"""
Read CSV:
using (var p = new ChoCSVReader("Sample6.csv"))
{
p.SanitizeLine += (o, e) =>
{
string line = e.Line as string;
if (line != null)
{
line = line.Substring(1, line.Length - 2);
line = line.Replace(#"""""", #"""");
}
e.Line - line;
};
var dt = p.AsDataTable();
//Assign dt to DataGridView
}
Hope it helps.
What is the Uniform Type Identifer to provide to a UIDocumentMenuViewController to allow a user to select *.docx files?
The documentation in System-Declared Uniform Type Identifiers does not list a public UTType that allows filtering by .docx. An identifier exists for a standard *.doc file but not *.docx, is there instead an alternate UTType?
This is my current code:
var allowedDocumentTypes = new string[] {
UTType.RTF,
UTType.Text,
UTType.PDF,
UTType.UTF8PlainText,
UTType.RTFD,
UTType.UTF16ExternalPlainText,
UTType.UTF16PlainText,
UTType.UTF8PlainText,
UTType.FlatRTFD,
"com.microsoft.word.doc",
"com.microsoft.word.docx" // An attempt to include docx filtering.
};
var pickerMenu = new UIDocumentMenuViewController(allowedDocumentTypes, UIDocumentPickerMode.Open);
pickerMenu.DidPickDocumentPicker += (sender, args) =>
{
args.DocumentPicker.DidPickDocument += (sndr, pArgs) =>
{
var securityEnabled = pArgs.Url.StartAccessingSecurityScopedResource();
FileInfo fi = new FileInfo(pArgs.Url.Path);
var result = new SelectFileResult();
result.FilePath = fi.FullName;
result.FileName = fi.Name;
NSUrlRequest urlReq = NSUrlRequest.FromUrl(pArgs.Url);
NSUrlResponse response;
NSError error;;
var data = NSUrlConnection.SendSynchronousRequest(urlReq, out response, out error);
result.MimeType = response.MimeType;
Action onFileConsumeDone = () =>
{
pArgs.Url.StopAccessingSecurityScopedResource();
};
onFileSelected(result, onFileConsumeDone);
};
// Display the document picker
AppDelegate.TopViewController.PresentViewController(args.DocumentPicker, true, null);
};
pickerMenu.ModalPresentationStyle = UIModalPresentationStyle.Popover;
AppDelegate.TopViewController.PresentViewController(pickerMenu, true, null);
I had a shot in the dark and included the identifier com.microsoft.word.docx but it does not trigger filtering by docx.
My current solution is C# but objective-c and swift solutions accepted.
Try org.openxmlformats.wordprocessingml.document
Try
let supportedTypes: [UTType] = [UTType.content]
I'm having a question about my CSV.
I export a CSV and read it in C#.
The last colomn of each line in CSV is A,B,C,D,E or G.
Now, I want my CSV to be cut in pieces, like; I want a new CSV with the lines which contain A and D. And another one which contains B and C for example.
Can anyone point me in the right direction? I'm stuck..
This is a part of my code
StreamReader debtors = new StreamReader(#"C:\CSV\Debtors.csv");
StreamWriter debtorsMetaal = new StreamWriter(#"C:\CSV\DebtorsMetaal.csv");
StreamWriter debtorsSystemen = new StreamWriter(#"C:\CSV\DebtorsSystemen.csv");
StreamWriter debtorsHolding = new StreamWriter(#"C:\CSV\DebtorsHolding.csv");
while(debtors.Peek() >=0)
{
string line = debtors.ReadLine();
try
{
string[] rowsArray = line.Split(';');
//..... etc
Now the lines are in pieces, but how can I select the last colomn in my line and create a new CSV file based upon the values of the last colomn?
debtorsMetaal, debtorsSystemen and debtorsHolding will be the new CSV files.
For example;
In a line in the CSV I have the following info
number - name- description - type
Where type can be A, B, C, D , E or G.
Now I want the lines where type = A and the lines where type = D together in one CSV file.
Is this even possible?
The values A,B,C,D,E or G are always in colomn AJ in excel format.
I would use a loop like this:
var adLines = new List<string>();
var bcLines = new List<string>();
var unknownLines = new List<string>();
var adList = new[]{"A", "D"};
var bcList = new[]{"B", "C"};
using(var debtors = new StreamReader(#"C:\CSV\Debtors.csv"))
{
string line = null;
while((line = debtors.ReadLine()) != null)
{
string[] columns = line.Split(';'); // you should check if columns.Length is correct
string lastColumn = columns.Last().Trim();
if(adList.Contains(lastColumn, StringComparer.CurrentCultureIgnoreCase))
adLines.Add(line);
else if(bcList.Contains(lastColumn, StringComparer.CurrentCultureIgnoreCase))
bcLines.Add(line);
else
unknownLines.Add(line);
}
}
File.WriteAllLines(#"C:\CSV\DebtorsSystemen.csv", adLines);
File.WriteAllLines(#"C:\CSV\DebtorsHolding.csv", bcLines);
However, in general you should not reinvent the wheel and use an abvailable CSV-parser like:
http://www.codeproject.com/Articles/9258/A-Fast-CSV-Reader