When I try to use CsvHelper to parse the below CSV, I get a "conversion cannot be performed" error (full error is below). It looks like I'm missing something about how to deal with reading values as decimals. Have seen some other answers relating to setting the culture, but that doesn't seem to have helped.
The CSV data is:
Title,Amount,NHS,Reference,GoCardless ID,email,surname,firstname,Full Name,DOB,Age,Right Lens,Left Lens,RightLensMonthlyAmount,LeftLensMonthlyAmount,LensMonthlyAmount,FeeMonthlyAmount,VAT Basis,LensBespokePrice,CareOnly,Notes
Mrs,24.3,N,100247,CUXXX,email#gmail.com,User,Test,Test User,17/09/1957,64,DAILIES® AquaComfort PLUS 30 Pack,DAILIES® AquaComfort PLUS 30 Pack,16.5,16.5,33,6.35,,,,
My class to map this data to properties is:
public class Payer
{
public string Title { get; set; }
public decimal Amount { get; set; }
[BooleanTrueValues("Y")]
[BooleanFalseValues("N")]
public bool NHS { get; set; }
public string Reference { get; set; }
[Name("GoCardless ID")]
public string GoCardless_ID { get; set; }
public string email { get; set; }
public string surname { get; set; }
public string firstname { get; set; }
[Name("Full Name")]
public string Fullname { get; set; }
[Name("DOB")]
public string Dob { get; set; }
public int Age { get; set; }
[Name("Right Lens")]
public string RightLens { get; set; }
[Name("Left Lens")]
public string LeftLens { get; set; }
public decimal RightLensMonthlyAmount { get; set; }
public decimal LeftLensMonthlyAmount { get; set; }
public decimal LensMonthlyAmount { get; set; }
public decimal FeeMonthlyAmount { get; set; }
[Name("VAT Basis")]
public string VATBasis { get; set; }
public decimal LensBespokePrice { get; set; }
[BooleanTrueValues("Y")]
public bool CareOnly { get; set; }
}
My code related to parsing the CSV is:
static void Main(string[] args)
{
var culture = new CultureInfo("en-GB");
var config = new CsvHelper.Configuration.CsvConfiguration(culture);
using (var reader = new StreamReader("test.csv"))
using (var csv = new CsvReader(reader, config))
{
var records = csv.GetRecords<Payer>();
Console.WriteLine("Got records"); //this prints on the console
foreach (var payer in records)
{
Console.WriteLine(payer);
}
}
}
The error only happens with the foreach loop, not the actual GetRecords() method.
Full error:
CsvHelper.TypeConversion.TypeConverterException: "The conversion cannot be performed.\n Text: ''\n MemberType: System.Decimal\n TypeConverter: 'CsvHelper.TypeConversion.DecimalConverter'\nIReader state:\n ColumnCount: 0\n CurrentIndex: 18\n HeaderRecord:\n["Title","Amount","NHS","Reference","GoCardless ID","email","surname","firstname","Full Name","DOB","Age","Right Lens","Left Lens","RightLensMonthlyAmount","LeftLensMonthlyAmount","LensMonthlyAmount","FeeMonthlyAmount","VAT Basis","LensBespokePrice","CareOnly","Notes"]\nIParser state:\n ByteCount: 0\n CharCount: 392\n Row: 2\n RawRow: 2\n Count: 21\n RawRecord:\nMrs,24.3,N,100247,CUXXX,email#gmail.com,User,Test,Test User,17/09/1957,64,DAILIES® AquaComfort PLUS 30 Pack,DAILIES® AquaComfort PLUS 30 Pack,16.5,16.5,33,6.35,,,,\r\n\n"
at CsvHelper.TypeConversion.DefaultTypeConverter.ConvertFromString(String text, IReaderRow row, MemberMapData memberMapData)\n at CsvHelper.TypeConversion.DecimalConverter.ConvertFromString(String text, IReaderRow row, MemberMapData memberMapData)\n at CsvHelper.Expressions.RecordCreator.CreateT\n at CsvHelper.Expressions.RecordManager.CreateT\n at CsvHelper.CsvReader.d__87`1.MoveNext()\n at dd_journal.Program.Main(String[] args) in /Users/abhi/Documents/Practice/dd-journal/Program.cs:22
So the issue is that CSVHelper doesn't understand how to convert an empty field for LensBespokePrice to a decimal value. There are two options you can use here:
Update the CSV file to add a default value to the empty fields (i.e. 0 for LensBespokePrice).
Create a Type Conversion to handle an empty cell to a decimal.
Do you have the ability to modify the CSV file? If so, then #1 works by changing your CSV to be (note the change for LensBespokePrice and CareOnly):
Title,Amount,NHS,Reference,GoCardless ID,email,surname,firstname,Full Name,DOB,Age,Right Lens,Left Lens,RightLensMonthlyAmount,LeftLensMonthlyAmount,LensMonthlyAmount,FeeMonthlyAmount,VAT Basis,LensBespokePrice,CareOnly,Notes
Mrs,24.3,N,100247,CUXXX,email#gmail.com,User,Test,Test User,17/09/1957,64,DAILIES® AquaComfort PLUS 30 Pack,DAILIES® AquaComfort PLUS 30 Pack,16.5,16.5,33,6.35,,0,N,
If not, you'll need a type converter for both the empty decimal and empty boolean. For example, with all your code in a single file, that may look like:
using CsvHelper;
using CsvHelper.Configuration;
using CsvHelper.Configuration.Attributes;
using CsvHelper.TypeConversion;
using System;
using System.Globalization;
using System.IO;
using System.Text.Json;
namespace ConsoleApp1
{
class Program
{
static void Main(string[] args)
{
var culture = new CultureInfo("en-GB");
var config = new CsvConfiguration(culture);
using (var reader = new StreamReader("test.csv"))
using (var csv = new CsvReader(reader, config))
{
var records = csv.GetRecords<Payer>();
Console.WriteLine("Got records"); //this prints on the console
foreach (var payer in records)
{
var j = JsonSerializer.Serialize(payer);
Console.WriteLine(j);
}
}
}
}
public class CustomDecimalConverter : DecimalConverter
{
public override object ConvertFromString(string text, IReaderRow row, MemberMapData memberMapData)
{
if(decimal.TryParse(text, out var result))
{
return result;
} else
{
return decimal.Zero;
}
}
}
public class CustomBooleanConverter : BooleanConverter
{
public override object ConvertFromString(string text, IReaderRow row, MemberMapData memberMapData)
{
if (bool.TryParse(text, out var result))
{
return result;
}
else
{
return false;
}
}
}
public class Payer
{
public string Title { get; set; }
public decimal Amount { get; set; }
[BooleanTrueValues("Y")]
[BooleanFalseValues("N")]
public bool NHS { get; set; }
public string Reference { get; set; }
[Name("GoCardless ID")]
public string GoCardless_ID { get; set; }
public string email { get; set; }
public string surname { get; set; }
public string firstname { get; set; }
[Name("Full Name")]
public string Fullname { get; set; }
[Name("DOB")]
public string Dob { get; set; }
public int Age { get; set; }
[Name("Right Lens")]
public string RightLens { get; set; }
[Name("Left Lens")]
public string LeftLens { get; set; }
public decimal RightLensMonthlyAmount { get; set; }
public decimal LeftLensMonthlyAmount { get; set; }
public decimal LensMonthlyAmount { get; set; }
public decimal FeeMonthlyAmount { get; set; }
[Name("VAT Basis")]
public string VATBasis { get; set; }
[TypeConverter(typeof(CustomDecimalConverter))]
public decimal LensBespokePrice { get; set; }
[BooleanTrueValues("Y")]
[BooleanFalseValues("N")]
[TypeConverter(typeof(CustomBooleanConverter))]
public bool CareOnly { get; set; }
}
}
Please note that I added the JsonSerializer.Serialize(payer); in the foreach loop within the Main method so that you can view the JSON result from the console.
I added in two custom converters (CustomBooleanConverter and CustomDecimalConverter). The payer class is then updated to have attributes added to the LensBespokePrice and CareOnly properties. Additionally, you didn't have an attribute on CareOnly for false values and while not required is a good practice.
To clarify why this is happening only in your foreach loop and not var records = csv.GetRecords<Payer>(); is because the values aren't actually converted into your payer class until the records are enumerated.
Payer is a whole class of values. It looks like it is trying to convert when you write to the console. I believe you will need to tell it what part of payer you are wanting to print like:
Console.WriteLine(payer.surname);
Related
I have a problem with regex. The variable media.Value may contain the characters "() +" which causes an error in the regex and prevents it from working.
My code
Match renderMatch = Regex.Match(mediaMatch.Groups[0].Value, "(?<=\"name\":\"" + media.Value + "\",\"render\":).*?(?=,)");
Match mutedMatch = Regex.Match(mediaMatch.Groups[0].Value, "(?<=\"muted\":).*?(?=,\"name\":\"" + media.Value + "\")");
json I work with
[{"alignment":5,"cx":844.0,"cy":264.0,"id":4,"locked":false,"muted":false,"name":"Text (GDI +)","render":true,"source_cx":844,"source_cy":264,"type":"text_gdiplus_v2","volume":1.0,"x":549.0,"y":383.0},{"alignment":5,"cx":1920.0,"cy":1080.0,"id":3,"locked":false,"muted":false,"name":"Color","render":true,"source_cx":1920,"source_cy":1080,"type":"color_source_v3","volume":1.0,"x":0.0,"y":0.0}]
As long as there are no "()" in the name field everything works. For example:
Working
"muted":false,"name":"Color","render":true
Not working
"muted":false,"name":"Text (GDI +)","render":true
The question is.
Is there any regex option that would ignore the () in string, or how else could I get an output like this:
"Text \(GDI \+\)"
You can deserialize your JSON string to strongly typed models and then gather your required fields
An example with your JSON string is: https://dotnetfiddle.net/cqkOdu
using System;
using System.Collections.Generic;
using Newtonsoft.Json;
public class Program
{
public static void Main()
{
var myJsonResponse= #"[{'alignment':5,'cx':844.0,'cy':264.0,'id':4,'locked':false,'muted':false,'name':'Text (GDI +)','render':true,'source_cx':844,'source_cy':264,'type':'text_gdiplus_v2','volume':1.0,'x':549.0,'y':383.0},{'alignment':5,'cx':1920.0,'cy':1080.0,'id':3,'locked':false,'muted':false,'name':'Color','render':true,'source_cx':1920,'source_cy':1080,'type':'color_source_v3','volume':1.0,'x':0.0,'y':0.0}]";
List<Root> myDeserializedClass = JsonConvert.DeserializeObject<List<Root>>(myJsonResponse);
foreach(var item in myDeserializedClass)
{
Console.WriteLine(item.name);
}
}
}
public class Root
{
public int alignment { get; set; }
public double cx { get; set; }
public double cy { get; set; }
public int id { get; set; }
public bool locked { get; set; }
public bool muted { get; set; }
public string name { get; set; }
public bool render { get; set; }
public int source_cx { get; set; }
public int source_cy { get; set; }
public string type { get; set; }
public double volume { get; set; }
public double x { get; set; }
public double y { get; set; }
}
I'm looking to parse this JSON and I've had nothing but problems. The link to the JSON is here. I'm trying to access the "href" field. While writing this up, I realized that that the data field is actually an array so that is part of my problem.
class Program
{
static void Main(string[] args)
{
var json = System.IO.File.ReadAllText(#"C:\Users\...\file.json");
Root myDeserializedClass = JsonConvert.DeserializeObject<Root>(json);
var x = myDeserializedClass.result.extractorData.data;
Console.Write(x.ToString());
}
public class Newcolumn
{
public string text { get; set; }
public string xpath { get; set; }
public string href { get; set; }
public string filepath { get; set; }
public string fileMimeType { get; set; }
public int fileTotalBytes { get; set; }
public string fileLastModifiedTime { get; set; }
}
public class Group
{
public List<Newcolumn> Newcolumn { get; set; }
}
public class Datum
{
public List<Group> group { get; set; }
}
public class ExtractorData
{
public string url { get; set; }
public List<Datum> data { get; set; }
}
public class PageData
{
public int statusCode { get; set; }
public long timestamp { get; set; }
}
public class Inputs
{
public string _url { get; set; }
}
public class Result
{
public ExtractorData extractorData { get; set; }
public PageData pageData { get; set; }
public Inputs inputs { get; set; }
public string taskId { get; set; }
public long timestamp { get; set; }
public int sequenceNumber { get; set; }
}
public class Root
{
public string url { get; set; }
public Result result { get; set; }
}
}
This ends up returning: System.Collections.Generic.List`1[ConsoleApp3.Datum]
I notice that the field name data actually turns into an array though I'm not sure how to structure that. data.[0].new Column.[0].group.etc... does not work obviously. The space in the "new Column" field is also problematic. Additionally, when I debug and look at the JSON viewer, the "new column field is null. I also tried this code:
public static void Main()
{
var json = System.IO.File.ReadAllText(#"C:\Users\...\file.json");
dynamic stuff = JsonConvert.DeserializeObject(json);
var a = stuff.result.extractorData.data;
string b = a.ToString();
Console.WriteLine(b);
Console.WriteLine("Press any key to exit.");
System.Console.ReadKey();
}
This actually does return the data field object however, if I do stuff.result.extractorData.data.group; I get this:
Microsoft.CSharp.RuntimeBinder.RuntimeBinderException
HResult=0x80131500
Message='Newtonsoft.Json.Linq.JArray' does not contain a definition for 'group'
Source=<Cannot evaluate the exception source>
StackTrace:
<Cannot evaluate the exception stack trace>
I assume that this is probably because of the array contained within the data field, regardless the "new Column' field is also an issue with this method due to the space.
in your above code
public class ExtractorData
{
public string url { get; set; }
public List<Datum> data { get; set; }
}
where data is List and you are trying to access as string
Console.Write(x.ToString());
in data variable Datum is List ( where your variable have multiple Data) and in every element there is a List. (Nestest List Concept Applied in This JSON)
Try to Add Break Point on below Line and check the line by line Excution of Code.
var a = stuff.result.extractorData.data;
After Looking Your JSON File Image Try This Code
Console.Write(a.FirstOrDefault()?.group.FirstOrDefault()?.Newcolumn.FirstOrDefault()?.href);
I am currently setting some strings via this method:
string marketlabel = allmarketdata.#return.markets.COLXPM.label.ToString();
I would like to set the market label dynamically by having a string for the actual market choice.
string currentMarketSelected= this.marketTextBox.Text; // Specific market: COLXPM
string marketlabel=allmarketdata.#return.markets.currentMarketSelected.label.ToString();
I have been searching for a few hours and probably am not explaining correctly. I tried some stuff with reflections with no success. Basically what I want to do is have a textbox or list which contains all the market names and based on which one is selected start setting the data.
Above is the best type of example of what I want to do even though it is not syntactically possible to use a variable in place.
public class Markets
{
public COLXPM COLXPM { get; set; }
//Lots of markets below here
}
public class COLXPM
{
public string marketid { get; set; }
public string label { get; set; }
public string lasttradeprice { get; set; }
public string volume { get; set; }
public string lasttradetime { get; set; }
public string primaryname { get; set; }
public string primarycode { get; set; }
public string secondaryname { get; set; }
public string secondarycode { get; set; }
public List<Recenttrade> recenttrades { get; set; }
public List<Sellorder> sellorders { get; set; }
public List<Buyorder> buyorders { get; set; }
}
public class Return
{
public Markets markets { get; set; }
}
public class RootObject
{
public int success { get; set; }
public Return #return { get; set; }
}
The proposed solution below that worked
string currentMarketSelected = "DOGEBTC"; // Just selecting one of the markets to test it works
var property = allmarketdata.#return.markets.GetType().GetProperty(currentMarketSelected);
dynamic market = property.GetMethod.Invoke(allmarketdata.#return.markets, null);
string marketlabel = market.label.ToString(); //Gets all my selected market data
Here is a solution using reflection.
string currentMarketSelected= this.marketTextBox.Text; // Specific market: COLXPM
var property = allmarketdata.#return.markets.GetType().GetProperty(currentMarketSelected);
dynamic market = property.GetGetMethod().Invoke(allmarketdata.#return.markets, null);
string marketlabel=market.label.ToString();
You need something like this:
public class Markets
{
public COLXPM this[string key]
{
get
{
COLXPM colxpm;
switch (key)
{
// TODO : use "key" to select instance of COLXPM;
case "example1":
colxpm = ...;
break;
default:
throw new NotSupportedException();
}
return colxpm;
}
}
}
Then you can do something like:
string marketlabel=allmarketdata.#return.markets[currentMarketSelected]label.ToString();
This is an indexed property.
I've got a problem regarding Json.NET and the omdbapi. I'm trying to retrieve information from the omdbapi and some properties are giving me headaches, particularly the "imdbVotes" one since it's written, in example, as "321,364" so I can't get an integer from it.
I'm betting that I need a custom converter, but I'm afraid that, at the moment, I don't really understand how to create one for my particular problem.
All other properties work well (I'm not using all of them at the moment).
This is the response for, lets say Snatch : http://www.omdbapi.com/?i=&t=snatch
This is my class :
public class MovieJSON
{
[JsonProperty(PropertyName = "Title")]
public String Title { get; set; }
[JsonProperty(PropertyName = "Year")]
public int Year { get; set; }
[JsonProperty(PropertyName = "Genre")]
public String Genre { get; set; }
[JsonProperty(PropertyName = "Director")]
public String Director { get; set; }
[JsonProperty(PropertyName = "Actors")]
public String Actors { get; set; }
[JsonProperty(PropertyName = "Plot")]
public String Plot { get; set; }
[JsonProperty(PropertyName = "Poster")]
public String Poster { get; set; }
[JsonProperty(PropertyName = "Metascore")]
public int Metascore { get; set; }
[JsonProperty(PropertyName = "imdbRating")]
public decimal ImdbRating { get; set; }
[JsonProperty(PropertyName = "imdbVotes")]
public int ImdbVotes { get; set; }
}
UPDATE #1 :
How can I handle the response when the property has the value "N/A"?. That happens for some movies (ie. http://www.omdbapi.com/?i=&t=four+rooms has it's Metascore set to N/A).
UPDATE #2 :
Another related inquiry. I'm using EF6 with MySQL and the idea's to populate the database with movies created through JSON parsing.
This is my Movie class :
[JsonObject(MemberSerialization.OptIn)]
[Table("movies")]
public class MovieJSON
{
[Key]
public int Id { get; set; }
[JsonProperty(PropertyName = "Title")]
[Column("title")]
public String Title { get; set; }
[JsonProperty(PropertyName = "Year")]
[Column("year")]
public int Year { get; set; }
[JsonProperty(PropertyName = "Genre")]
[Column("genre")]
public String Genre { get; set; }
[JsonProperty(PropertyName = "Director")]
[Column("director")]
public String Director { get; set; }
[JsonProperty(PropertyName = "Actors")]
[Column("actors")]
public String Actors { get; set; }
[JsonProperty(PropertyName = "Plot")]
[Column("plot")]
public String Plot { get; set; }
[JsonProperty(PropertyName = "Poster")]
[Column("poster")]
public String Poster { get; set; }
[JsonProperty(PropertyName = "Metascore")]
public String Metascore { get; set; }
[Column("metascore")]
public int MetascoreInt
{
get
{
int result;
if (int.TryParse(Metascore, NumberStyles.AllowThousands, CultureInfo.InvariantCulture, out result))
return result;
return 0;
}
}
[JsonProperty(PropertyName = "imdbRating")]
public String ImdbRating { get; set; }
[Column("imdb_rating")]
public Decimal ImdbRatingDecimal
{
get
{
Decimal result;
if (Decimal.TryParse(ImdbRating, out result))
return result;
return 0;
}
}
[JsonProperty(PropertyName = "imdbVotes")]
public String ImdbVotes { get; set; }
[Column("imdb_votes")]
public long ImdbVotesLong
{
get
{
long result;
String stringToParse = ImdbVotes.Remove(ImdbVotes.IndexOf(','), 1);
if (long.TryParse(stringToParse, out result))
return result;
return 0;
}
}
[JsonProperty(PropertyName = "imdbID")]
[Column("imdb_id")]
public String ImdbID { get; set; }
[JsonProperty(PropertyName = "type")]
[Column("type")]
public String Type { get; set; }
public override string ToString()
{
String[] propertiesToIgnore = {"MetascoreInt", "ImdbRatingDecimal", "ImdbVotesLong"};
var sb = new StringBuilder();
PropertyInfo[] properties = GetType().GetProperties();
foreach (PropertyInfo propertyInfo in properties)
{
if (propertiesToIgnore.Contains(propertyInfo.Name))
continue;
sb.AppendLine(String.Format("{0} : {1} ",
propertyInfo.Name, propertyInfo.GetValue(this, null)));
}
return sb.ToString();
}
}
This is my EF6 configuration-context class (I'm ignoring the String fields and instead, using the Helper ones since the database is configured to accept int for Metascore and so on) :
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
modelBuilder.Entity<MovieJSON>().Ignore(e => e.Metascore).Ignore(e => e.ImdbRating).Ignore(e => e.ImdbVotes);
base.OnModelCreating(modelBuilder);
}
Additional image info :
Object values before insertion into the database (all values are properly set)
Valid XHTML http://imagizer.imageshack.us/v2/800x600q90/689/8x5m.png
Values in the database :
Valid XHTML http://imagizer.imageshack.us/v2/800x600q90/844/nvc5.png
The helper fields (MetascoreInt, ImdbRatingDecimal, ImdbVotesLong) are returning zero, I can't figure out why.
Any help would be mucho appreciated! :)
All the best
You could have two properties: one would be the string property as it comes from IMDB, and the other would be the int property that converts the string one. To convert, you can use the nifty NumberStyles.AllowThousands flag. So you would have
[JsonProperty(PropertyName = "imdbVotes")]
public string ImdbVotes { get; set; }
public int ImdbVotesInt
{
get
{
int result;
if (int.TryParse(ImdbVotes,
NumberStyles.AllowThousands,
CultureInfo.InvariantCulture,
out result))
return result; // parse is successful, use 'result'
else
return 0; // parse is unsuccessful, return whatever default value works for you
}
}
I'm trying to learn how to use LINQTOExcel to query a CSV file. Following the tutorial on the site I adapted their example to work with my data (filename is passed to it via an OpenDialog component):
var csv = new ExcelQueryFactory(filename);
var test = from c in csv.Worksheet<TestData>()
select c;
foreach(var t in test)
{
Console.WriteLine(t.Contract_Id);
}
I've got a separate TestData class/model which looks like this:
class TestData
{
public string Transaction_Id { get; set; }
public string Value_Date { get; set; }
public string Transmit_Date { get; set; }
public string Transmit_Time { get; set; }
public string Contract_Id { get; set; }
public string Contract_Amount { get; set; }
public string Contract_Rage { get; set; }
public string TestAmount { get; set; }
public string Employer_Code { get; set; }
public string Test_Acceptor { get; set; }
public string Institution_Id { get; set; }
}
But when I loop through, all of the values for each item are 'null'. Am I missing a step somewhere?
Example CSV Data:
transaction_id,value_date,transmit_date,transmit_time,contract_no,contract_amount,instalment,test_amount,employer_code,test_acceptor,institution_id
35454521,20111230,20120102,2:23:12,1442,1714.56,1,285.76,0,643650,a
The CSV file needs a header row that matches the property names:
Transaction_Id,Value_Date,Transmit_Date,Transmit_Time,Contract_Id,Contract_Amount,Contract_RageTestAmount,Employer_Code,Test_Acceptor,Institution_Id
35454521,20111230,20120102,2:23:12,1442,1714.56,1,285.76,0,643650