I have a lot of CSV files without header and need to read it in C#. I manually added header to one of these files and with the following code using CSVHelper I can read the files and show them in a GridView.
Now my question is, how can I read these files without a header? Or how can I add a header (a new record) using CSVHelper in the first line?
public Form1()
{
InitializeComponent();
List<Festival> records;
var config = new CsvConfiguration(CultureInfo.InvariantCulture) { Delimiter = ";" };
using (var reader = new StreamReader(#"File8.csv"))
using(var csv = new CsvReader(reader, config))
{
records = csv.GetRecords<Festival>().ToList();
}
dataGridView1.DataSource = records;
}
Class
public class Festival
{
public string Day { get; set; }
public string Start { get; set; }
public int Lenght { get; set; }
public string FilmName { get; set; }
public float Rating { get; set; }
}
csv sample
Mi;22:15;110;A;8
Mi;19:00;106;B;8
Mi;19:15;97;C;8.2
Add column-index mapping attributes to the target members:
public class Festival
{
[Index(0)]
public string Day { get; set; }
[Index(1)]
public string Start { get; set; }
[Index(2)]
public int Lenght { get; set; }
[Index(3)]
public string FilmName { get; set; }
[Index(4)]
public float Rating { get; set; }
}
And specify HasHeaderRecord = false in the config:
var config = new CsvConfiguration(CultureInfo.InvariantCulture) { Delimiter = ";", HasHeaderRecord = false };
If modifying the target model isn't desirable, implement a ClassMap instead:
public sealed class FestivalMap : ClassMap<Festival>
{
public FestivalMap()
{
Map(f => f.Day).Index(0);
Map(f => f.Start).Index(1);
Map(f => f.Lenght).Index(2);
Map(f => f.FilmName).Index(3);
Map(f => f.Rating).Index(4);
}
}
And register it like this before fetching the records (you still need to specify HasHeaderRecord = false in the config):
csv.Context.RegisterClassMap<FestivalMap>();
records = csv.GetRecords<Festival>().ToList();
Related
I am building an API pulling data from Azure SQL would like to create a JSON array.
Currently I have an Azure Function written in C#.
Sample data looks like this:
I would like the output to look like this
My Azure Function is working fine, I just need to create an array. (I think)
await connection.OpenAsync();
SqlDataReader dataReader = await command.ExecuteReaderAsync();
var r = Serialize(dataReader);
json = JsonConvert.SerializeObject(r, Formatting.Indented);
I'm new to .NET and not sure quite where to begin. Thanks!
You could do it this way. Read the data into a Type that you can then use LINQ on to group into the desired shape, then serialize to JSON.
//Start with a list of the raw data by reading the rows into CardData list
List<CardData> cards = new List<CardData>();
while (dataReader.Read())
{
//You should check for DBNull, this example not doing that
cards.Add(new CardData
{
card_key = dataReader.GetString(0),
card_name = dataReader.GetString(1),
card_network = dataReader.GetString(2),
annual_fee = dataReader.GetDecimal(3),
speed_bonus_category = dataReader.GetString(4),
speed_bonus_amount = dataReader.GetInt32(5)
});
}
//Now transform the data into an object graph that will serialize
//to json the way you want. (flattens the redundant data)
var grp = cards.GroupBy(x => new { x.card_key, x.card_name, x.card_network, x.annual_fee });
var groupedData = new List<CardsModel>();
groupedData = grp.Select(g => new CardsModel
{
card_key = g.Key.card_key,
card_name = g.Key.card_name,
card_network = g.Key.card_network,
annual_fee = g.Key.annual_fee,
Bonuses = g.Select(b => new SpeedBonus
{
SpeedBonusCategory = b.speed_bonus_category,
SpeedBonusAmount = b.speed_bonus_amount
}).ToList()
}).ToList();
//Finally you can serialize
var json = JsonConvert.SerializeObject(groupedData, Formatting.Indented);
Here are the supporting classes you could use:
//represents the non-redundant object graph
public class CardsModel
{
public string card_key { get; set; }
public string card_name { get; set; }
public string card_network { get; set; }
public decimal annual_fee { get; set; }
public List<SpeedBonus> Bonuses { get; set; }
}
public class SpeedBonus
{
public string SpeedBonusCategory { get; set; }
public int SpeedBonusAmount { get; set; }
}
//represents raw data, has redundant cc info
public class CardData
{
public string card_key { get; set; }
public string card_name { get; set; }
public string card_network { get; set; }
public decimal annual_fee { get; set; }
public string speed_bonus_category { get; set; }
public int speed_bonus_amount { get; set; }
}
I'm trying to create a windows service that detects if there are new CSV files in a folder and parse those files. CsvHelper seems to not be able to read the CSV file. All the parameters that I try to populate are empty.
Is there something wrong in the code? The GetField method does not return any values and If I print the parameters are all empty.
The path is correct and the csv files paths are also correct.
public class CSVBatch
{
public string MaterialID { get; set; }
public string MaterialName { get; set; }
public string Location { get; set; }
public string Quantity { get; set; }
public string BatchID { get; set; }
public string ProcessOrder { get; set; }
public string Recipe { get; set; }
public List<CSVRawMaterial> CSVRawMaterials { get; set; }
public class CSVRawMaterial
{
public string MaterialID { get; set; }
public string MaterialName { get; set; }
public string Location { get; set; }
public string Quantity { get; set; }
public string BatchID { get; set; }
public string ProcessOrder { get; set; }
public string Recipe { get; set; }
}
}
protected override void OnStart(string[] args)
{
var folder = "C:\\BOM";
FileSystemWatcher fileSystemWatcher = new FileSystemWatcher(folder);
var fw = fileSystemWatcher;
fw.IncludeSubdirectories = true;
fw.EnableRaisingEvents = true;
fw.Created += Newfileevent;
}
static void Newfileevent(object sender, FileSystemEventArgs e)
{
string[] filePaths = Directory.GetFiles("C:\\BOM");
foreach (string s in filePaths)
{
var config = new CsvConfiguration(CultureInfo.InvariantCulture)
{
Delimiter = ",",
MissingFieldFound = null,
TrimOptions = TrimOptions.Trim,
HeaderValidated = null,
HasHeaderRecord = true
};
using (var reader = new StringReader(s))
using (var csv = new CsvReader(reader, config))
{
csv.Read();
var batch = new CSVBatch
{
MaterialID = csv.GetField<string>(0),
MaterialName = csv.GetField<string>(1),
Location = csv.GetField<string>(2),
Quantity = csv.GetField<string>(3),
BatchID = csv.GetField<string>(4),
ProcessOrder = csv.GetField<string>(5),
Recipe = csv.GetField<string>(6)
};
csv.Read();
var rawMaterials = new List<CSVRawMaterial>();
while (csv.Read())
{
var rawmaterial = new CSVRawMaterial
{
MaterialID = csv.GetField<string>(0),
MaterialName = csv.GetField<string>(1),
Location = csv.GetField<string>(2),
Quantity = csv.GetField<string>(3)
};
rawMaterials.Add(rawmaterial);
}
batch.CSVRawMaterials = rawMaterials;
}
}
CSV File:
You have 2 issues.
You are using StringReader instead of StreamReader. It should be:
using (var reader = new StreamReader(s))
If you have a header row, you also have to specifically read the header when reading by hand.
using (var reader = new StreamReader(s))
using (var csv = new CsvReader(reader, config))
{
csv.Read();
csv.ReadHeader();
csv.Read();
batch = new CSVBatch
{
I have a csv consisting of many columns. From that csv I have to select only few required columns.
The code I have written is
for (int i = 0; i < lineCount; i++)
{
var line = str.ReadLine();
if (line != null)
{
var values = line.Split(',');
dataInformation.Add(new DataInformation
{
timestamp_iso = values[3],
last_attributed_touch_data_tilde_campaign = values[9],
last_attributed_touch_data_tilde_channel = values[11],
last_attributed_touch_data_tilde_feature = values[12],
last_attributed_touch_data_tilde_ad_set_name = values[19],
user_data_platform = values[69],
user_data_aaid = values[70],
user_data_idfa = values[71],
user_data_idfv = values[72]
});
}
}
I am getting wrong values while using this. Is there any other approach to retrieve the values using the column names instead of column numbers.
The Data Information is a class
public class DataInformation
{
public string timestamp_iso { get; set; }
public string last_attributed_touch_data_tilde_campaign { get; set; }
public string last_attributed_touch_data_tilde_channel { get; set; }
public string last_attributed_touch_data_tilde_feature { get; set; }
public string last_attributed_touch_data_tilde_ad_set_name { get; set; }
public string user_data_platform { get; set; }
public string user_data_aaid { get; set; }
public string user_data_idfa { get; set; }
public string user_data_idfv { get; set; }
}
Please help me on this.
I recommend using a library to deal with CSV format. CsvHelper is a good one. It allows accessing fields by column name:
csv.Read();
var field = csv["HeaderName"];
CSV format may look simple, but there are a few corner cases (like quotes), so it is better to use an existing solution.
I have used the below code to get all the records of the type DataInformation.
using (TextReader fileReader = File.OpenText(FileName))
{
var csv = new CsvReader(fileReader);
dataInformation = csv.GetRecords<DataInformation>().ToList();
}
And after that I have used the below code to get the required columns.
using (TextWriter writer = new StreamWriter(ConfigurationManager.AppSettings["downloadFilePath"] + ConfigurationManager.AppSettings["fileName"] + date + ConfigurationManager.AppSettings["csvExtension"].ToString()))
{
using (var csv = new CsvWriter(TextWriter.Synchronized(writer)))
{
csv.WriteHeader(typeof(DataInformation));
csv.NextRecord();
csv.WriteRecords(dataInformation);
}
}
It works for me.
I have class with nested list properties, I am trying to write the value to CSV file, but I am getting output appended with [{ }] like shown below:
Client TDeals
ABC [{DealName:59045599,TShape:[{StartDate:"2014-01-
28T23:00:00",EndDate:"2014-01-28T23:30:00",Volume:0.00},
{StartDateTime:"2014-01-
28T23:30:00",EndDateTime:"2014-01-29T00:00:00",Volume:0.00}}]
I want my output in CSV file like shown below:
Client DealNo StartDate EndDate Volume
ABC 59045599 - - -
Class Properties
public class TRoot
{
public string Client { get; set; }
public List<TDeal> Deals { get; set; }
}
public class TDeal
{
public string DealName{get;set;}
public List<TInterval> TShape { get; set; }
}
public class TInterval
{
public string StartDate{ get; set; }
public string EndDate{ get; set; }
public string Volume {get;set;}
}
I am using ServiceStack.Text to create CSV file from object
ServiceStack.Text.CsvSerializer.SerializeToWriter<TRoot>(TRoot, writer);
Reference URL
https://github.com/ServiceStack/ServiceStack.Text
Define a new class for single csv line:
public class CsvLine
{
public string Client { get; set; }
public string DealName { get; set; }
public string StartDate { get; set; }
public string EndDate { get; set; }
public string Volume { get; set; }
}
Now you can transfrom your objects into collection of lines with Linq SelectMany method:
TRoot root = ...
var lines = root.Deals.SelectMany(d => d.TShape.Select(s => new CsvLine
{
Client = root.Client,
DealName = d.DealName,
StartDate = s.StartDate,
EndDate = s.EndDate,
Volume = s.Volume
})).ToArray();
Then call SerializeToWriter on that collection
I would recommend to "flatten" your output to CSV.
Create one more class that will be a mirror of what you would like to have in CSV file. Before writing to the file, convert your TRoot to that new class and write it to CSV.
Quite quick and elegant solution :)
You can try Cinchoo ETL to create the CSV file. First you will have to flatten out root object using Linq and pass them to CSV writer to create file.
Sample below show how to
private static void Test()
{
TRoot root = new TRoot() { Client = "ABC", Deals = new List<TDeal>() };
root.Deals.Add(new TDeal
{
DealName = "59045599",
TShape = new List<TInterval>()
{
new TInterval { StartDate = DateTime.Today.ToString(), EndDate = DateTime.Today.AddDays(2).ToString(), Volume = "100" },
new TInterval { StartDate = DateTime.Today.ToString(), EndDate = DateTime.Today.AddDays(2).ToString(), Volume = "200" }
}
});
using (var w = new ChoCSVWriter("nestedObjects.csv").WithFirstLineHeader())
{
w.Write(root.Deals.SelectMany(d => d.TShape.Select(s => new { ClientName = root.Client, DealNo = d.DealName, StartDate = s.StartDate, EndDate = s.EndDate, Volume = s.Volume })));
}
}
The output is:
ClientName,DealNo,StartDate,EndDate,Volume
ABC,59045599,1/17/2018 12:00:00 AM,1/19/2018 12:00:00 AM,100
ABC,59045599,1/17/2018 12:00:00 AM,1/19/2018 12:00:00 AM,200
For more information about it, visit the codeproject article at
https://www.codeproject.com/Articles/1155891/Cinchoo-ETL-CSVWriter
Disclaimer: I'm the author of this library.
I am using elasticsearc 2.3.3 and Nest 2.3.2, I need to create index. Need to map the properties with attributes added in the class file.
public class IndexDocument
{
[Number(Store = true)]
public long Id { get; set; }
[String(Store = true, Index = FieldIndexOption.Analyzed, TermVector = TermVectorOption.WithPositionsOffsets)]
public string Title { get; set; }
public Attachment File { get; set; }
[String(Store = true, Index = FieldIndexOption.Analyzed)]
public string DocumentType { get; set; }
[String(Store = true, Index = FieldIndexOption.NotAnalyzed)]
public string DocLocation { get; set; }
[String(Store = true, Index = FieldIndexOption.Analyzed)]
public DateTime LastModifiedDate { get; set; }
}
public class Attachment
{
public Attachment()
{
}
[String(Name = "_content_length", Store = true, Index = FieldIndexOption.Analyzed)]
public long ContentLength { get; set; }
[String(Store = true, Index = FieldIndexOption.Analyzed, TermVector = TermVectorOption.WithPositionsOffsets, Name = "_content")]
public string Content { get; set; }
}
Also I would like to add a completion suggest to the File field.
I am new in this elastic search. Can anyone please help?
I have created my attachment index as below. How to append completion suggest and stemmer code with this?
this.client.CreateIndex("mydocs", c => c.Mappings(mp => mp.Map<IndexDocument>
(m => m.Properties(ps => ps.Attachment
(a => a.Name(o => o.File)
.TitleField(t => t.Name(x => x.Title).TermVector(TermVectorOption.WithPositionsOffsets))
)))));