I'm attempting to read a big list of data into a flat CSV. One field of each row is XML data - with the XML dictating additional Properties in each row. Therefore each row could have slightly different Properties.
I've got my list in an initial structured class:
private class Response {
public Response() {
Values = new Dictionary<string, string>();
}
public string Owner { get; set; }
public string Author { get; set; }
public string Email { get; set; }
public string Data { get; set; } // XML data in here
public Dictionary<string, string> Values { get; set; }
public DateTime Created { get; set; }
}
I then thought to make my list dynamic to handle the varied Properties:
var responses = new List<dynamic>();
foreach (var response in query) {
if (!string.IsNullOrEmpty(response.Data))
{
var doc = new XmlDocument();
doc.LoadXml(response.Data);
foreach (XmlNode node in doc.GetElementsByTagName("field")) {
try
{
response.Values.Add(node.FirstChild.InnerText, node.LastChild.InnerText);
}
catch (Exception ex) {
// log
}
}
}
dynamic fullResponse = new ExpandoObject();
fullResponse.Owner = response.Owner;
fullResponse.Author = response.Author;
fullResponse.Email = response.Email;
fullResponse.Created = response.Created;
IDictionary<string, object> map = fullResponse;
foreach (var value in response.Values) {
if (map.ContainsKey(value.Key)) {
map[value.Key] = value.Value;
}
else {
map.Add(value.Key, value.Value);
}
}
responses.Add(fullResponse);
}
... which also works fine. But now I want to basically flatten this list and export it as a CSV, knowing that each row of my dynamic list might have slightly different properties.
Grabbing the PropertyInfo, a dynamic type, obviously comes back with nothing. Am I just going to have to loop through my list to extract the unique properties manually into something like a DataTable? Or are there some calls I can make with Reflection to get this done more efficiently?
I ended up using a plain old DataTable to get it sorted:
using (var dt = new DataTable())
{
var keys = responses.SelectMany(x => ((IDictionary<string, object>)x).Keys).Distinct();
var columns = keys.Select(x => new DataColumn(x)).ToArray();
dt.Columns.AddRange(columns);
foreach(IDictionary<string, object> response in responses)
{
var row = dt.NewRow();
foreach (var kvp in response)
{
row[kvp.Key] = kvp.Value;
}
dt.Rows.Add(row);
}
// write to CSV
}
Though I feel like I could've done something a lot cooler with Reflection to achieve the same result.
Related
I have a model class
public class Model
{
public string name { get; set; }
public string files { get; set; }
}
I have a controller class that populates the data from database into this model
List<Model> model = new List<Model>();
while (reader.Read()){
var om = new Model();
if (reader != null)
{
om.name = reader.GetString(0);
om.files = reader.GetString(1);
model.Add(om)
}
How can I filter and combine all files that have the similar first names?
I read about linq and tried this
var jm = om.name
.Where(o => om.name.Contains(o))
.Select() ;
This might work for you:
var grouped = model.GroupBy(m => m.name).ToArray();
This will create an object grouped of type IGrouping<string, Model>[] (array of IGrouping...)
.ToArray() is optional and if you remove it, you get IEnumerable<IGrouping<string, Model>>, but don't let the nested <> scare you.
The items in your original list are grouped by name, so the group will have Key property that will be the same for each element in the group.
To print the results, for example:
foreach(var group in grouped)
{
Console.WriteLine($"Printing group: {group.Key}");
foreach(var model in group)
{
Console.WriteLine($"\tName: {model.name}, File: {model.file}");
}
}
Note that each element in a group is a collection of your model objects.
Use debugger to see what that structure looks like and if you need more help - ask...
Change your Model
public class Model
{
public string name { get; set; }
public IList<string> files { get; set; }
}
Use a Dictionary.
var models = new Dictionary<string, Model>();
while (reader.Read())
{
var om = new Model();
if (reader != null)
{
var name = reader.GetString(0);
var file = reader.GetString(1);
if (models.Contains(name))
models[name].files.Add(file);
else
{
models.Add(name, new Model
{
name = name,
files = new List<string>{files}
});
}
}
}
Then you can pull out a combined list with LINQ:
var peopleFileLists = models.Select(x => x.Value).ToList();
foreach(var person in peopleFileLists)
{
foreach(var file in person.files)
Console.WriteLine(person.name + "-" + file);
}
I trying to get the id and email list from the JSON. How can i achieve this?
My JSON string is
{
"name":"name1",
"username":"name1",
"id":505,
"state":"active",
"email":"name1#mail.com",
},
{
"name":"name2",
"username":"name2",
"id":504,
"state":"active",
"email":"name2#mail.com",
}
My code is
Dictionary<string, string> engineers = new Dictionary<string, string>();
using (StreamReader r = new StreamReader(#"D:\project\Gitlap\EngineerEmail\jsonlist5.json"))
{
using (JsonTextReader reader = new JsonTextReader(r))
{
JObject o2 = (JObject)JToken.ReadFrom(reader);
string id = o2["id"].ToString();
string email = o2["email"].ToString();
engineers.Add(email, id);
}
}
class UserItems
{
public string id { get; set; }
public string email { get; set; }
}
I can able to get the first person`s mail ID and ID details. I need to iterate this JSON and get all the mail ID and ID.
I don`t know that how to iterate this JSON. I tried some method from the internet but that was not succeeded.
How can I do?
First thing is your JSON input is not valid json, you need to fix it. There are two issues in it. Its not collection of json objects and comma is missing between two objects.
Valid json should look like below.
[{
"name":"name1",
"username":"name1",
"id":505,
"state":"active",
"email":"name1#mail.com",
},
{
"name":"name2",
"username":"name2",
"id":504,
"state":"active",
"email":"name2#mail.com",
}]
Now define a c# class representing your json object.
public class User
{
public string name { get; set; }
public string username { get; set; }
public int id { get; set; }
public string state { get; set; }
public string email { get; set; }
}
Use JSON.Net library to deserialize it as shown below.
private void button1_Click(object sender, EventArgs e)
{
if(File.Exists("json1.json"))
{
string inputJSON = File.ReadAllText("json1.json");
if(!string.IsNullOrWhiteSpace(inputJSON))
{
var userList = JsonConvert.DeserializeObject<List<User>>(inputJSON);
}
}
}
JObject o2 = (JObject)JToken.ReadFrom(reader);
foreach(var obj in o2)
{
string id = obj["id"].ToString();
string Email= obj["Email"].ToString();
engineers.Add(email, id);
}
I would recommend using the Json.NET NuGet package to accomplish this.
Firstly, create a model to represent your JSON data. Typically I would capitalize the first letter of the property names here, but to keep it consistent with the JSON, they are lower case.
public class UserData
{
public string name { get; set; }
public string username { get; set; }
public int id { get; set; }
public string state { get; set; }
public string email { get; set; }
}
You will need to add a using for Json.NET
using Newtonsoft.Json;
Finally, you can load, and deserialize your data into a strongly typed list, which you can then use to populate your engineers dictionary.
string datapath = #"D:\project\Gitlap\EngineerEmail\jsonlist5.json";
Dictionary<string, string> engineers = new Dictionary<string, string>();
List<UserData> data = new List<UserData>();
using (StreamReader r = new StreamReader(datapath))
{
string json = r.ReadToEnd();
data = JsonConvert.DeserializeObject<List<UserData>>(json);
data.ForEach(engineer => engineers.Add(engineer.email, engineer.id.ToString()));
}
As mentioned in another answer, your JSON is also badly formed. This will need correcting before it will deserialize correctly. We just need to add a comma to separate the two objects, and wrap them both in a JSON array, with []
[
{
"name":"name1",
"username":"name1",
"id":505,
"state":"active",
"email":"name1#mail.com"
},
{
"name":"name2",
"username":"name2",
"id":504,
"state":"active",
"email":"name2#mail.com"
}
]
Improvements
As your Id field is an integer, it would be better to change your dictionary from
Dictionary<string, string> engineers = new Dictionary<string, string>();
into
Dictionary<string, int> engineers = new Dictionary<int, string>();
You will then be able to simplify your ForEach query slightly. The ForEach can also be moved outside of the using() block.
data.ForEach(engineer =>
engineers.Add(engineer.email, engineer.id));
Improved solution
This includes the improvements above, I've used var for brevity.
var datapath = #"D:\project\Gitlap\EngineerEmail\jsonlist5.json";
var engineers = new Dictionary<string, int>();
var data = new List<UserData>();
using (var r = new StreamReader(datapath))
{
var json = r.ReadToEnd();
data = JsonConvert.DeserializeObject<List<UserData>>(json);
}
data.ForEach(engineer =>
engineers.Add(engineer.email, engineer.id));
try to create class that represent the data in json object for example
Class obj
{
public int Id { get ; set; }
public string email { get ; set; }
public string username { get ; set; }
public string state { get ; set; }
public string email { get ; set; }
}
then
using System.Web.Script.Serialization;
var js = new JavaScriptSerializer();
List<obj> list = js.Deserialize<List<obj>>(jsonString);
after that you can access all list items id and email by using foreach
I have a JSON file that I need to deserialize using the class System.Web.Script.Serialization. The following code:
string json = File.ReadAllText(#"C:\file.json");
JavaScriptSerializer ser = new JavaScriptSerializer();
var dict = ser.Deserialize<Dictionary<string, object>>(json);
creates a nested dictionary, something like this:
Dictionary<string, Object> outerDictionary;
Dictionary<string, Object> middleDictionary;
Dictionary<string, string> innerDictionary;
I now need to grab some of the values from the innerDictionary into a C# object class, like:
public class Location
{
public string Id { get; set; }
public string Name { get; set; }
public string Street { get; set; }
public string City { get; set; }
public string State { get; set; }
public string Zip { get; set; }
}
So my question is, how can I iterate over the nested disctionary to get the values I need at the innermost level?
UPDATE
With the help of Jonesopolis I now have the correct loop structure, but I am still unable to get the values from the innermost dictionary?
foreach (var outer in dict)
{
foreach (var middle in (Dictionary<string, object>)outer.Value)
{
foreach (var inner in (ArrayList)middle.Value)
{
foreach (var loc in (Dictionary<string, object>)inner)
{
}
}
}
}
If you don't like the nested foreach in Jonesopolis's answer, then you can use LINQ this way:
var data = from outer in dict
from middle in (Dictionary<string, object>) outer.Value
from inner in (Dictionary<string, string>) middle.Value
select new { outer, middle, inner };
foreach (var item in data) {
// do things with item.inner
}
if you really want to work with those objects:
foreach(var outer in dict)
{
foreach (var middle in (Dictionary<string, object>) outer.Value)
{
foreach (var inner in (Dictionary<string, string>) middle.Value)
{
var location = new Location();
location.Id = inner["Id"];
//etc..
}
}
}
though, it would be much better to create a class structure that you can deserialize your json to, that accurately represents your data, instead of casting Dictionaries.
I am writing a class that reads different kinds of CSV files. It picks out the important information based on Model classes, where the properties of the model class are the column names that I want to grab. For example, I could have an OutlookModel with columns FromAddress and ToAddress. Or I could have a SalesforceModel with totally different columns.
When the reader class parses through the rows and columns, it loads up the cells into an instance of the model class. In the code below, the argument className = OutlookModel. The most relevant lines of code here are the signature and the return...
protected void MapColumns(string row, string className, List<OutlookModel> list)
{
string[] cols = row.Split(',');
// create a model to save the important columns
var model = Activator.CreateInstance(nameSpace, nameSpace + className);
int j = 0;
if (cols.Length > 0)
{
foreach (var c in cols)
{
// is this column index one of our important columns?
if (Ordinals.ContainsKey(j))
{
// this is a column we care about, so set the model property
model.GetType().GetProperty(Ordinals[j]).SetValue(model, c);
}
j++;
}
}
list.Add(model);
}
The problem I am having is the collection of model objects. If I define the object as List< OutlookModel > in the arguments, then the method is not extensible. If I define it as List< object >, then (i think) I have to cast the inside list to use my properties which are all different between the models.
I am fairly new to C#. Is there a better way to capture these different model types into a list/array/collection/whatever so that I can then apply logic to the lists?
So first of all i suggest to add a custom attribute to mark the properties you want to read from the csv, so you don't run into any problem when you have to add something later and you don't have to rely on too many magic strings. Here is my test setup:
class ReadFromCsvAttribute : Attribute { }
class OutlookModel
{
public int DontSetThisValueFromCsv { get; set; }
[ReadFromCsv]
public string FromAddress { get; set; }
[ReadFromCsv]
public string ToAddress { get; set; }
}
class SalesForceModel
{
[ReadFromCsv]
public string Name { get; set; }
[ReadFromCsv]
public string Age { get; set; }
}
static void Main(string[] args)
{
string outlookSample = "Id,FromAddress,ToAddress,Useless\r\n" +
"1,a#b.com,c#d.com,asdf\r\n" +
"3,y#z.com,foo#bar.com,baz";
string salesForceSample = "Id,Name,Age\r\n" +
"1,John,30\r\n" +
"2,Doe,100";
var outlook = ReadFromCsv<OutlookModel>(outlookSample);
var salesForce = ReadFromCsv<SalesForceModel>(salesForceSample);
}
I put together this generic method to read whatever model you want from the data:
static List<T> ReadFromCsv<T>(string data)
{
var objs = new List<T>();
var rows = data.Split(new[] {"\r\n"}, StringSplitOptions.None);
//create index, header dict
var headers = rows[0].Split(',').Select((value, index) => new {value, index})
.ToDictionary(pair => pair.index, pair => pair.value);
//get properties to find and cache them for the moment
var propertiesToFind = typeof (T).GetProperties().Where(x => x.GetCustomAttributes<ReadFromCsvAttribute>().Any());
//create index, propertyinfo dict
var indexToPropertyDict =
headers.Where(kv => propertiesToFind.Select(x => x.Name).Contains(kv.Value))
.ToDictionary(x => x.Key, x => propertiesToFind.Single(p => p.Name == x.Value));
foreach (var row in rows.Skip(1))
{
var obj = (T)Activator.CreateInstance(typeof(T));
var cells = row.Split(',');
for (int i = 0; i < cells.Length; i++)
{
if (indexToPropertyDict.ContainsKey(i))
{
//set data
indexToPropertyDict[i].SetValue(obj, cells[i]);
}
}
objs.Add(obj);
}
return objs;
}
Here's another sample. Since you're new to c#, I've avoided linq and extension methods as much as possible. Just copy it into a console app and run.
Also, I like theHennyy recommendation of using .net attributes to describe a class but only if you have full control of your ecosystem.
public class Account
{
public string FirstName { get; set; }
public string LastName { get; set; }
}
public class LastNameAccount
{
public string LastName { get; set; }
public string Address { get; set; }
}
public class Program
{
public static void Main(string[] args)
{
Test1();
}
private static void Test1()
{
/*
* defines the result of your CSV parsing.
*/
List<string> csvColumns = new List<string> { "FirstName", "LastName" };
List<List<string>> csvRows = new List<List<string>>() {
new List<string>(){"John","Doe"},
new List<string>(){"Bill", "Nie"}
};
//Map the CSV files to Account type and output it
var accounts = Map<Account>(csvColumns, csvRows);
if (accounts != null)
{
foreach (var a in accounts)
{
Console.WriteLine("Account: {0} {1}", a.FirstName, a.LastName);
}
}
//Map the CSV files to LastNameAccount type and output it
var accounts2 = Map<LastNameAccount>(csvColumns, csvRows);
if (accounts2 != null)
{
foreach (var a in accounts2)
{
Console.WriteLine("Last Name Account: {0} {1}", a.LastName, a.Address);
}
}
}
private static List<T> Map<T>(List<string> columns, List<List<string>> rows)
where T : class, new()
{
//reflect the type once and get valid columns
Type typeT = typeof(T);
Dictionary<int, PropertyInfo> validColumns = new Dictionary<int, PropertyInfo>();
for (int columnIndex = 0; columnIndex < columns.Count; columnIndex++)
{
var propertyInfo = typeT.GetProperty(columns[columnIndex]);
if (propertyInfo != null)
{
validColumns.Add(columnIndex, propertyInfo);
}
}
//start mapping to T
List<T> output = null;
if (validColumns.Count > 0)
{
output = new List<T>();
foreach (var row in rows)
{
//create new T
var tempT = new T();
//populate T's properties
foreach (var col in validColumns)
{
var propertyInfo = col.Value;
var columnIndex = col.Key;
propertyInfo.SetValue(tempT, row[columnIndex]);
}
//add it
output.Add(tempT);
}
}
return output;
}
}
This is basically a follow up to a previous question I posted of Deserializing JSON to a DataTable. Well the process is nearly finished, this is the code:
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
result = streamReader.ReadToEnd();
dynamic d = JObject.Parse(result);
}
var root = JsonConvert.DeserializeObject<RootObject>(result);
public static DataTable ToDataTable<T>(this IList<T> data)
{
PropertyDescriptorCollection props =
TypeDescriptor.GetProperties(typeof(T));
DataTable table = new DataTable();
for (int i = 0; i < props.Count; i++)
{
PropertyDescriptor prop = props[i];
table.Columns.Add(prop.Name, prop.PropertyType);
}
object[] values = new object[props.Count];
foreach (T item in data)
{
for (int i = 0; i < values.Length; i++)
{
values[i] = props[i].GetValue(item);
}
table.Rows.Add(values);
}
return table;
}
public class Record
{
public int StatusID { get; set; }
public string Identifier { get; set; }
public string Status { get; set; }
public string StatusDate { get; set; }
public string WorkedBy { get; set; }
public string ContactedOn { get; set; }
public string Email { get; set; }
}
public class RootObject
{
public List<Record> Record { get; set; }
}
Now, the deserialization is working perfectly, but I cannot convert to DataTable. I am using the ToDataTable extension I found in SO, and this snippet is supposed to turn my list into a DataTable:
RootObject.Record.ToDataTable<Record>();
Now of course I cannot do this, since Record is not a static member, but if I make it static, like this:
public static List<Record> record { get; set; }
and change the ToDataTable extension call to
RootObject.record.ToDataTable<Record>();
It breaks the conversion from JSON to List. Using breakpoints if I verify the "root" var its null and has no data, so when it tries to turn it into a DataTable the whole thing crashes since it only has null values.
Your RootObject contains the Record property so you need to use it to create the DataTable like this:
var root = JsonConvert.DeserializeObject<RootObject>(result);
root.Record.ToDataTable<Record>();
What you have tried is to access it as a static memeber of the RootObject class which is in this context wrong becasue the deserialization creates already an instance of RootObject. You now just have to use the Record property that you want to convert to a DataTable.
That's why I think it's not always good to use the var keyword. Sometimes you forget or oversee what type it is. If you wrote:
RootObject root = JsonConvert.DeserializeObject<RootObject>(result);
you might have found the solution by yourself.