Thanks in advance, C# newb here having a few issues.
I this CSV file provided daily, large, and has no header. I only need certain items out of this file.
Here is the code I have so far.
var config = new CsvConfiguration(CultureInfo.InvariantCulture)
{
HasHeaderRecord = false,
};
using (var reader = new StreamReader(iFile.FileName))
using (var csv = new CsvReader(reader, config))
{
var records = new List<BQFile>();
csv.Read();
csv.ReadHeader();
while (csv.Read())
{
var record = new BQFile()
{
SNumber = csv.GetField<string>("SNumber"),
FOBPoint = csv.GetField<string>("FOBPoint")
};
}
What I am not understanding since this CSV files 150+ fields, is how do grab the correct data. For example, if SNumber is column 46, FOBPoint is column 123. I am finding the CSVHelper documentation a little limited to me.
Any help is appreciated.
What I am not understanding since this CSV files 150+ fields, is how do grab the correct data
By index, because there is no header
In your BQFile, decorate the properties with an attribute of [Index(NNN)] where N is the column number (0-based). The IndexAttribute is found in CsvHelper.Configuration.Attributes namespace - I mention this because Entity Framework also has an Index attribute; be sure you use the correct one
pubic class BQFile{
[Index(46)]
public string SNumber { get; set;}
...
}
Then do:
var config = new CsvConfiguration(CultureInfo.InvariantCulture)
{
HasHeaderRecord = false,
};
using (var reader = new StreamReader(iFile.FileName))
using (var csv = new CsvReader(reader, config))
{
var records = csv.GetRecords<BQFile>();
...
records is an enumeration on top of the file stream (via CSVHelper, which reads records as it goes and creates instances of BQFile). You can only enumerate it once, and then after you're done enumerating it the filestream will be at the end - if you wanted to re-read the file you'd have to Seek the stream or renew the reader. Also, the file is only read (in chunks, progressively) as you enumerate. If you return records somewhere, so you drop out of the using and you thus dispose the reader, you'll get an error when you try to start reading from records (because it's disposed)
To work with records, you either foreach it, processing the objects you get as you go:
foreach(BQFile bqf in records){
//do stuff with each BQFile here
}
Or if you want to load it all into memory, you can do something like ToList() it so you end up with a bunch of BQFile in a List, and then you can e.g. access them randomly, read them over and over etc..
var bqfs = records.ToList();
ps; I don't know, when you said "it's column 46" if that's counting from 1 or 0.. You might have to adjust your 46.
CSVHelper and FileHelper is not an option
I have a .csv export that I need to check for consistency structured like the below
Reference,Date,EntryID
ABC123,08/09/2015,123
ABD234,08/09/2015,124
XYZ987,07/09/2015,125
QWE456,08/09/2016,126
I can use ReadLine or RealAllLines and .Split which give me entire rows/columns BUT I have need to select each row and then go through each attribute (separated by ',') for format checking
I am running into problems here. I can not single out each value in a row for this check.
It is probably either something simple onto
class Program
{
static void Main(string[] args)
{
string csvFile = #"proof.csv";
string[] lines = File.ReadAllLines(csvFile);
var values = lines.Skip(1).Select(l => new { FirstRow = l.Split('\n').First(), Values = l.Split('\n').Select(v => int.Parse(v)) });
foreach (var value in values)
{
Console.WriteLine(string.Format("{0}", value.FirstRow));
}
}
}
Or I am going down the wrong path, my searches relate to pulling specific rows or columns (as opposed to checking the individual values associated)
The sample of the data above has a highlighted example: The date is next year and I would like to be able to proof that value (just an example as it could be in either column where errors appear)
I can not single out each value in a row
That's because you split on \n twice. The values within a row are separated by comma (,).
I'm not sure what all that LINQ is supposed to do, but it's as simple as this:
string[] lines = File.ReadAllLines(csvFile);
foreach (var line in lines.Skip(1))
{
var values = line.Split(',');
// access values[0], values[1] ...
}
Instead of reading it as text read it by OLEDB object, so data of CSV file will come in datatable and you do not need to spit it.
To Read the csv file you can use these objects of OLEDB
System.Data.OleDb.OleDbCommand
System.Data.OleDb.OleDbDataAdapter
System.Data.OleDb.OleDbConnection
and
System.Data.DataTable
I have 2 csv files, file1.csv and file2.csv. Some lines in each file will be identical. I wish to create a 3rd csv file, based upon file2.csv but with any lines that are present in file1.csv removed from it. Effectively I wish to subtract file1.csv from file2.csv ignoring any lines present in file1 that are not in file2.
I know that I could use streamreader to read each line in file2.csv and search for it in file1.csv. If it does not exist in file1.csv I can write it to file3.csv. However, the files are very large (over 30000 lines) and I believe this would take a lot of processing time.
I suspect there may be a better method of loading each csv to an array and then performing a simple subtraction function on them to obtain the desired result. I would appreciate either some help with the code or on method that I should approach this problem with.
Example content of files:
file1.csv
dt97861.jpg,149954,c1714ee1,\folder1\folderA\,
dt97862.jpg,149955,c1714ee0,\folder1\folderA\,
dt97863.jpg,59368,cd23f223,\folder2\folderA\,
dt97864.jpg,57881,0835be4a,\folder2\folderB\,
dt97865.jpg,57882,0835be4b,\folder2\folderB\,
file2.csv
dt97862.jpg,149955,c1714ee0,\folder1\folderA\,
dt97863.jpg,59368,cd23f223,\folder2\folderA\,
dt97864.jpg,57881,0835be4a,\folder2\folderB\,
dt97865.jpg,57882,0835be4b,\folder2\folderB\,
dt97866.jpg,57883,0835be4c,\folder2\folderB\,
dt97867.jpg,57884,0835be4d,\folder3\folderA\,
dt97868.jpg,57885,0835be4e,\folder3\folderA\,
The results I require is:
file3.csv
dt97866.jpg,57883,0835be4c,\folder2\folderB\,
dt97867.jpg,57884,0835be4d,\folder3\folderA\,
dt97868.jpg,57885,0835be4e,\folder3\folderA\,
EDIT:
With the help below I came to the following solution which I believe to be nice and elegant:
public static IEnumerable<string> ReadFile(string path)
{
string line;
using (var reader = File.OpenText(path))
while ((line = reader.ReadLine()) != null)
yield return line;
}
then:
var file2 = ReadFile(file2FilePath);
var file1 = ReadFile(file1FilePath);
var file3 = file2.Except(file1);
File.WriteAllLines(file3FilePath, file3);
Assume the line is perfectly identical, you can read both file into two IEnumerable<string> and extract with IEnumerable.Except<T>. This will produce the same result regardless of the ordering~
Example :
var file1 = new List<string>{
#"dt97861.jpg,149954,c1714ee1,\folder1\folderA\,",
#"dt97862.jpg,149955,c1714ee0,\folder1\folderA\,",
#"dt97863.jpg,59368,cd23f223,\folder2\folderA\,",
#"dt97864.jpg,57881,0835be4a,\folder2\folderB\,",
#"dt97865.jpg,57882,0835be4b,\folder2\folderB\,",
};
var file2 = new List<string>{
#"dt97862.jpg,149955,c1714ee0,\folder1\folderA\,",
#"dt97863.jpg,59368,cd23f223,\folder2\folderA\,",
#"dt97864.jpg,57881,0835be4a,\folder2\folderB\,",
#"dt97865.jpg,57882,0835be4b,\folder2\folderB\,",
#"dt97866.jpg,57883,0835be4c,\folder2\folderB\,",
#"dt97867.jpg,57884,0835be4d,\folder3\folderA\,",
#"dt97868.jpg,57885,0835be4e,\folder3\folderA\,",
};
file2.Except(file1).Dump();
Output :
dt97866.jpg,57883,0835be4c,\folder2\folderB\,
dt97867.jpg,57884,0835be4d,\folder3\folderA\,
dt97868.jpg,57885,0835be4e,\folder3\folderA\,
Here is the function to load any file into IEnumerable<string>. Just dont forget to using System.IO;.
public static IEnumerable<string> ReadFile(string path)
{
string line;
using(var reader = File.OpenText(path))
while((line = reader.ReadLine()) != null)
yield return line;
}
To write the result to a file :
//using System.IO; is required
File.WriteAllLines("file3.csv", file2.Except(file1))
Remarks : File.WriteAllLines will create or overwrite the file.
While this may not be the best approach, it's the one I've used in the past. It's a bit of a dirty hack, but...
Import both CSV files into a datatable (so you will have two datatables -I personally prefer closed xml if you plan to use an excel type format, otherwise just use a normal file read/write - My example uses regular read/write)
Move data from datatable into a list (my example assumes comma separated values, one per line.)
Find unique values between lists and merge
Export the merged lists to a csv file
*[Edited steps after actually working on the code]
Per request from Bit, I've added an example using sample data from Some Random Website - This was written in VS2008 against .NET 3.5, but it should work on 3.5+. I copied us-500 into 2 versions, the original and modified 1 row to create a unique value to test. This project is targeting x86 platform. I've used a new windows form for testing
using System.Data;
using System.Data.OleDb;
using System.IO;
using System.Linq;
using System.Windows.Forms;
namespace TestSandbox
{
public partial class Form1 : Form
{
public Form1()
{
var file1 = new DataTable();
var file2 = new DataTable();
InitializeComponent();
//Gets data from csv file, select allows for filtering
using (var conn = new OleDbConnection(#"Provider=Microsoft.Jet.OLEDB.4.0;Data Source=c:\;Extended Properties=""text;HDR=Yes;FMT=Delimited"";"))
{
conn.Open();
using (var adapter = new OleDbDataAdapter(#"select * from [us-500.csv]", conn))
{
adapter.Fill(file1);
}
}
using (var conn = new OleDbConnection(#"Provider=Microsoft.Jet.OLEDB.4.0;Data Source=c:\;Extended Properties=""text;HDR=Yes;FMT=Delimited"";"))
{
conn.Open();
using (var adapter = new OleDbDataAdapter(#"select * from [us-500-2.csv]", conn))
{
adapter.Fill(file2);
}
}
//Moves datatable information to lists for comparison
var file1List = (from DataRow row in file1.Rows select row.ItemArray.Select(field => field.ToString()).ToArray() into fields select string.Join(",", fields)).ToList();
var file2List = (from DataRow row in file2.Rows select row.ItemArray.Select(field => field.ToString()).ToArray() into fields select string.Join(",", fields)).ToList();
//Adds all data from file2 into file1 list, except for data that already exists in file1
file1List.AddRange(file2List.Except(file1List));
//Exports all results to c:\results.csv
File.WriteAllLines(#"C:\Results.csv", file1List.ToArray());
}
}
}
*Note: After looking at the code, importing straight to a list looks like it would be more efficient, but I'll leave this as is for now since it's not overly complicated.
Step 1. Using System.IO, we'll read two files using FileStream and create a third file using StreamWriter.
Step 2. Use FileStream to read file #1. e.g.
using (var FS = new System.IO.FileStream(file1, System.IO.FileMode.Open, System.IO.FileAccess.Read)) { ...<insert next steps in here>...}
Step 3. Nest another FileStream to read file #2. This stream will be read multiple times, so it's best if you can put the smaller file in this part of the nest. You can do this by checking the size of the file prior to jumping into these loops.
Step 4. Read in a single line from our biggest file, File#1, then we compare it against ALL lines from File#2 sequentially. If a match is found, set a boolean to TRUE indicating that there is a matching line found in File #2.
Step 5. Once we're at the end of File #2, check for a true/false condition of the boolean. If its false, SAVE the string we read from File #1 into File #3. This is your output file.
Step 6. Reset the stream pointer for File #2 to the beginning of the file e.g. FS.Seek(0, System.IO.SeekOrigin.Begin)
Step 7. Repeat from Step 4 until we've reached the end of File #1. File #3's contents should represent only unique entries from File #1 that are not members of File #2
I am currently working on a school project involving a large number of students where I have to insert a new student alphabetically and do a few other calculations. I am having trouble getting it so that it only adds the new student once. I have an if statement but it doesn't appear to be working properly.
`//this adds the new student
StreamWriter changeFile = new StreamWriter("Students.txt", true);
string newStudent = "(LIST (LIST 'Malachi 'Constant 'A ) '8128675309 'iwishihadjessesgirl#mail.usi.edu 4.0 )";
// this is where I am getting stumped
if (File.Exists(newStudent))
{
changeFile.Close();
}
else
{
changeFile.WriteLine(newStudent);
changeFile.Close();
}`
Whenever I run the code like this it will just add the new student every time I debug the program. How can I make it only add him one time?
File.Exists determines if the file at the given path exists (which, for the record, you should still be doing before trying to read/write to the file). You're trying to find out if the given line of text exists within a given file. That's a very different task.
You'll need to read through the lines in the file and compare them to your given text.
if(!File.ReadLines(filepath).Contains(newStudent))
{
//TODO: Append student to the file
}
File.Exists(string path) returns a bool that determines if a file exists at the specified path.
http://msdn.microsoft.com/en-us/library/system.io.file.exists(v=vs.110).aspx
string newStudent is not a file path, so it will always return false.
I think what you want is something like this: (this is by memory so it likely won't compile as is)
var file = File.Open("students.txt");
var fileContents = file.ReadToEnd();
if (!fileContents.Contains(newStudent))
{
file.WriteLine(newStudent);
}
file.Close();
First read the Existing file data into String variable and then check the given student data is available or not in the received file.if the given student data is not found then write the new student data into file otherwise,if already present then close the opened steream.
String StudentInfo = System.IO.File.ReadAllText("Students.txt");
StreamWriter changeFile = new StreamWriter("Students.txt", true);
string newStudent = "(LIST (LIST 'Malachi 'Constant 'A ) '8128675309 'iwishihadjessesgirl#mail.usi.edu 4.0 )";
// this is where I am getting stumped
if (StudentInfo.Contains(newStudent))
{
changeFile.Close();
}
else
{
changeFile.WriteLine(newStudent);
changeFile.Close();
}
I have this code for parsing a CSV file.
var query = from line in File.ReadAllLines("E:/test/sales/" + filename)
let customerRecord = line.Split(',')
select new FTPSalesDetails
{
retailerName = "Example",
};
foreach (var item in query)
{
//sales details table
ItemSale ts = new ItemSale
{
RetailerID = GetRetailerID(item.retailerName)
};
}
Obviously there will be more data in the above code, I am just awaiting the test information file details/structure.
In the mean time I thought I'd ask if this could me modified to parse TSV files?
All help is appreciated,
thanks :)
assuming tsv is tab separated value, you can use
line.Split('\t')
if you are using .NET 4.0, i would recommend that u use File.ReadLines for large files in order to use LINQ and not to load all the lines in memory at once.