MongoDb: documents with variable structure using and C#

MongoDb: documents with variable structure using and C# - c#

I have a HTML form that creates documents with a dynamic structure.
Here below some samples of the data inserted by the users.
A very simple document
{
"name" : "Simple element",
"notes" : "Lorem ipsum rocks",
"values" : [
{
"name" : "An array with 2 values",
"value" : [ 100,200],
"editable" : true
}
]
}
And more complex document
{
"name" : "Complex element",
"notes" : "Lorem ipsum rocks",
"values" : [
{
"name" : "A text value",
"value" : "ABCDEF",
"editable" : true
},
{
"name" : "A numeric value",
"value" : 100,
"editable" : false
},
{
"name" : "A array of 4 values",
"value" : [1,2,3,4],
"editable" : false
},
{
"name" : "A matrix 2x4",
"value" : [[1,2,3,4],[5,6,7,8]],
"editable" : false
}
]
}
The documents must be saved in MongoDB using C# MongoCharp driver and NancyFX.
At the moment the POST is implemented in this way but I'm not sure if this the correct way to handle object with a dynamic structure
Post["/api/docs"] = _ =>
{
//looking for better solution
var json = Request.Body.AsString();
var item = BsonDocument.Parse(json);
database.GetCollection("docs").Insert(item);
return new Response { StatusCode = HttpStatusCode.Created };
};
but can't find a good solution for the GET method
Get["/api/docs"] = _ =>
{
//looking for solution
};
What do you think would be the best solution for this scenario?

There's also another way to solve the problem. Let's call it "the strongly typed solution". I've created two POCO objects
public class DocumentItem
{
[BsonId]
[BsonRepresentation(BsonType.ObjectId)]
public String Id { get; set; }
public String Name { get; set; }
public String Notes { get; set; }
public SubItem[] Values { get; set; }
}
public class SubItem
{
public String Name { get; set; }
public Boolean Editable { get; set; }
public Object Value { get; set; }
}
than in the module the read data is implemented as shown below
Get["/api/docs/{id}"] = p => database.GetCollection<DocumentItem>("docs")
.FindOne(Query<DocumentItem>.EQ(x => x.Id, (string)p.id));
Get["/api/docs"] = _ => database.GetCollection<DocumentItem>("docs")
.FindAll()
.ToList();
and I can use binding for the insert in this way
Post["/api/docs"] = _ =>
{
var item = this.Bind<DocumentItem>();
database.GetCollection("docs").Insert(item);
return item;
};

If you are just wanting to return the document from MongoDB as json try something like this
Get["/api/docs/{category}"] = _ =>
{
var filterValue = _.category;
//Search the DB for one record where the category property matches the filterValue
var item = database.GetCollection("docs").FindOne(Query.EQ("category", filterValue))
var json = item.ToJson();
var jsonBytes = Encoding.UTF8.GetBytes(json );
return new Response
{
ContentType = "application/json",
Contents = s => s.Write(jsonBytes, 0, jsonBytes.Length)
};
};

Related

How can I deserialize complex json repsonse into easy-to-deserialize structure using Newtonsoft.Json?

Having this strange-looking json response:
{
"invoices":{
"0":{
"invoice":{
"id":"420",
"invoicecontents":{
"0":{
"invoicecontent":{
"name":"Here's the name of the content 0"
}
},
"1":{
"invoicecontent":{
"name":"Here's the name of the content 1"
}
}
}
}
},
"1":{
"invoice":{
"id":"420",
"invoicecontents":{
"0":{
"invoicecontent":{
"name":"Here's the name of the content 0"
}
}
}
}
},
"parameters":{
"limit":"3",
"page":"1",
"total":"420"
}
},
"status":{
"code":"OK"
}
}
How do I change the structure into this easy-to-deserialize one?
{
"invoices":[
{
"id":"420",
"invoicecontents":[
{
"name":"Here's the name of the content 0"
},
{
"name":"Here's the name of the content 1"
}
]
},
{
"id":"420",
"invoicecontents":[
{
"name":"Here's the name of the content 0"
}
]
}
]
}
I'd like to deserialize into List of Invoices as below
class Invoice {
public string Id { get; set; }
[JsonProperty("invoicecontents")]
public InvoiceContent[] Contents { get; set; }
class InvoiceContent {
public string Name { get; set; }
}
}
There's no problem with getting the status code or parameters, I simply do this:
var parsed = JObject.Parse(jsonInvoices);
var statusCode = parsed?["status"]?["code"]?.ToString();
var parameters = parsed?["invoices"]?["parameters"]?;
The real problem starts when I'm trying to achieve easy-to-deserialize json structure I've mentioned before. I've tried something like this:
var testInvoices = parsed?["invoices"]?
.SkipLast(1)
.Select(x => x.First?["invoice"]);
But I can't manage to "repair" invoicecontents/invoicecontent parts.
My goal is to deserialize and store the data.

This isn't JSON.
JavaScript Object Notation literally describes Objects. What you have here is the punchline of a joke that starts out with: "A SQL JOIN, a StringBuilder, and a couple for loops walk into a bar..."
As others have demonstrated, JObject is great for working with JSON that would be impractical to define as classes. You can use Linq to navigate through it, but JSONPath Expressions can be much simpler.
Example 1: Let's get the status code.
var status = parsed.SelectToken(".status.code").Value<string>();
Example 2: Let's 'deserialize' our invoices.
public string Invoice
{
public string Id { get; set; }
public List<string> Content { get; set; }
}
public List<Invoice> GetInvoices(string badJson)
{
var invoices = JObject.Parse(badJson).SelectTokens(".invoices.*.invoice");
var results = new List<Invoice>();
foreach (var invoice in invoices)
{
results.Add(new Invoice()
{
Id = invoice.Value<string>("id"),
Contents = invoice.SelectTokens(".invoicecontents.*.invoicecontent.name")
.Values<string>().ToList()
// Note: JToken.Value<T> & .Values<T>() return nullable types
});
}
return results;
}

try this
var jsonParsed = JObject.Parse(json);
var invoices = ((JObject) jsonParsed["invoices"]).Properties()
.Select(x => (JObject) x.Value["invoice"] ).Where(x => x != null)
.Select(s => new Invoice
{
Id = s.Properties().First(x => x.Name == "id").Value.ToString(),
Contents = ((JObject)s.Properties().First(x => x.Name == "invoicecontents").Value).Properties()
.Select(x => (string) x.Value["invoicecontent"]["name"]).ToList()
}).ToList();
and I simplified your class too, I don't see any sense to keep array of classes with one string, I think it should be just array of strings
public class Invoice
{
public string Id { get; set; }
public List<string> Contents { get; set; }
}
result
[
{
"Id": "420",
"Contents": [
"Here's the name of the content 0",
"Here's the name of the content 1"
]
},
{
"Id": "420",
"Contents": [
"Here's the name of the content 0"
]
}
]

Extract a particular record from MongoDB where the records use Object GUID in C#

One of the Mongo objects I use (referred to as 'Admin' here) uses GUID as the primary key and has a list of 'DailyActivities' which contains datevalue and another list called 'Subactivities'. The Admin object looks like something below. I am struggling to find resources in C# that will help extract DailyActivities that only corresponds to a particular date with Subactivities that has a category of 'Power Consumption'.
{
"_id" : ObjectId("5a2b7b887df7ce464404dc7d"),
"DailyActivities" : [
{
"datetime" : ISODate("2017-12-09T16:29:00.916Z"),
"Subactivities" : [
{
"entryDate" : ISODate("2017-12-09T06:30:26.658Z"),
"category" : "Power Consumption"
},
{
"entryDate" : ISODate("2017-12-09T06:30:26.658Z"),
"category" : "Machinery"
}
]
},
{
"datetime" : ISODate("2017-12-13T00:00:00.916Z"),
"Subactivities" : [
{
"entryDate" : ISODate("2017-12-13T06:30:26.658Z"),
"category" : "Lamination"
}
]
}
]
}
The result I would like to receive should be:
{
"_id" : ObjectId("5c7044f07ef75175b2b8efd6"),
"entryDate" : ISODate("2017-12-09T06:30:26.658Z"),
"category" : "Power Consumption"
}

Right now, I don't have the time to complete this exercise but here is something to get you going. I shall edit and improve this (as in convert it into some c# a lot of which won't be possible using a typed approach...) next week.
db.collection.aggregate([{
$project: {
"DailyActivities": {
$filter: {
input: "$DailyActivities",
cond: {
$eq: [ "$$this.datetime", ISODate("2017-12-09T16:29:00.916Z") ]
}
}
}
}
}, {
$unwind: "$DailyActivities"
}, {
$unwind: "$DailyActivities.Subactivities"
}, {
$replaceRoot: {
"newRoot": "$DailyActivities.Subactivities"
}
}, {
$match: {
"category": "Power Consumption"
}
}])

let me offer you a solution which uses MongoDAL as the data access layer. it's a wrapper around the c# driver so you get all the features of the driver plus a highly typed api.
using System;
using System.Linq;
using MongoDAL;
namespace AdminActs
{
class Admin : Entity
{
public DailyActivity[] DailyActivities { get; set; }
}
class DailyActivity
{
public DateTime Time { get; set; }
public SubActivity[] SubActivities { get; set; }
}
class SubActivity
{
public DateTime EntryDate { get; set; }
public string Category { get; set; }
}
class Program
{
static void Main(string[] args)
{
new DB("activities");
var now = DateTime.Now;
var admin = new Admin
{
DailyActivities = new DailyActivity[]
{
new DailyActivity{
Time = now,
SubActivities = new SubActivity[]
{
new SubActivity{
Category ="Power Consumption",
EntryDate = DateTime.Now}
}
}
}
};
admin.Save();
var subActivities = admin.Collection()
.SelectMany(a => a.DailyActivities)
.Where(da => da.Time == now)
.SelectMany(da => da.SubActivities)
.Where(sa => sa.Category == "Power Consumption");
var res = subActivities.ToArray();
Console.ReadKey();
}
}
}

Update a JSON file with a model object in C#

I have a json file that looks something like :
[
{
"picklist_typ": "Address Assessment Code",
"picklist_typ_key": null,
"picklist_typ_cd": "DELIVERABLE",
"picklist_typ_dsc": "Address is deliverable : Deliverable",
"ref_order": null,
"dw_trans_ts": "2016-07-17T12:59:15"
},
{
"picklist_typ": "Address Assessment Code",
"picklist_typ_key": null,
"picklist_typ_cd": "NOT-DELIVERABLE",
"picklist_typ_dsc": "Address is not deliverable : Undeliverable",
"ref_order": null,
"dw_trans_ts": "2016-07-17T12:59:15"
},
{
"picklist_typ": "Address Type",
"picklist_typ_key": null,
"picklist_typ_cd": "B",
"picklist_typ_dsc": "Billing Address",
"ref_order": null,
"dw_trans_ts": "2016-07-17T12:59:15"
},
....
Now , I have an object that looks like :
public class GroupMembershipWriteOutput
{
public long? groupKey { get; set; }
public string transOutput { get; set; }
public long? transKey { get; set; }
public string groupCode {get;set;}
public string groupName {get;set;}
}
And it has a value accordingly :
groupKey:121
transOutput: "Success"
transKey:998546
groupCode:"My Group Test"
groupName: "My Created Group Test"
What I want to is ..
I want to read the JSON file and if any entry's picklist_typ_key matches the incoming object's groupKey , I want to update only that object's picklist_typ_cd with groupCode
and picklist_typ_dsc with groupName.
I could read the data from JSON as ...
if(gmwo.transOutput.ToUpper() == "SUCCESS")
{
string json = System.Configuration.ConfigurationManager.AppSettings["PicklistDataPath"];
List<PicklistData> deserializedPicklistData = JsonConvert.DeserializeObject<List<PicklistData>>(System.IO.File.ReadAllText(json));
//find the object that matches
IEnumerable<PicklistData> results = deserializedPicklistData.Where(item => item.picklist_typ_key == gmwo.groupKey.ToString());
if(results != null)
{
**//What is the logic to update only that entry in the json file**
}
Please help me do it .

Could you try the LINQ "All" chain? Something like this:
IEnumerable<PicklistData> results = deserializedPicklistData
.Where(item => item.picklist_typ_key == gmwo.groupKey.ToString())
.All(selectedItem => someFunction(selectedItem));
:
:
someFunction(PicklistData myPick) {
:
}

LINQ to JSON - Query for object or an array

I'm trying to get a list of SEDOL's & ADP values. Below is my json text:
{
"DataFeed" : {
"#FeedName" : "AdminData",
"Issuer" : [{
"id" : "1528",
"name" : "ZYZ.A a Test Company",
"clientCode" : "ZYZ.A",
"securities" : {
"Security" : {
"id" : "1537",
"sedol" : "SEDOL111",
"coverage" : {
"Coverage" : [{
"analyst" : {
"#id" : "164",
"#clientCode" : "SJ",
"#firstName" : "Steve",
"#lastName" : "Jobs",
"#rank" : "1"
}
}, {
"analyst" : {
"#id" : "261",
"#clientCode" : "BG",
"#firstName" : "Bill",
"#lastName" : "Gates",
"#rank" : "2"
}
}
]
},
"customFields" : {
"customField" : [{
"#name" : "ADP Security Code",
"#type" : "Textbox",
"values" : {
"value" : "ADPSC1111"
}
}, {
"#name" : "Top 10 - Select one or many",
"#type" : "Dropdown, multiple choice",
"values" : {
"value" : ["Large Cap", "Cdn Small Cap", "Income"]
}
}
]
}
}
}
}, {
"id" : "1519",
"name" : "ZVV Test",
"clientCode" : "ZVV=US",
"securities" : {
"Security" : [{
"id" : "1522",
"sedol" : "SEDOL112",
"coverage" : {
"Coverage" : {
"analyst" : {
"#id" : "79",
"#clientCode" : "MJ",
"#firstName" : "Michael",
"#lastName" : "Jordan",
"#rank" : "1"
}
}
},
"customFields" : {
"customField" : [{
"#name" : "ADP Security Code",
"#type" : "Textbox",
"values" : {
"value" : "ADPS1133"
}
}, {
"#name" : "Top 10 - Select one or many",
"#type" : "Dropdown, multiple choice",
"values" : {
"value" : ["Large Cap", "Cdn Small Cap", "Income"]
}
}
]
}
}, {
"id" : "1542",
"sedol" : "SEDOL112",
"customFields" : {
"customField" : [{
"#name" : "ADP Security Code",
"#type" : "Textbox",
"values" : {
"value" : "ADPS1133"
}
}, {
"#name" : "Top 10 - Select one or many",
"#type" : "Dropdown, multiple choice",
"values" : {
"value" : ["Large Cap", "Cdn Small Cap", "Income"]
}
}
]
}
}
]
}
}
]
}
}
Here's the code that I have so far:
var compInfo = feed["DataFeed"]["Issuer"]
.Select(p => new {
Id = p["id"],
CompName = p["name"],
SEDOL = p["securities"]["Security"].OfType<JArray>() ?
p["securities"]["Security"][0]["sedol"] :
p["securities"]["Security"]["sedol"]
ADP = p["securities"]["Security"].OfType<JArray>() ?
p["securities"]["Security"][0]["customFields"]["customField"][0]["values"]["value"] :
p["securities"]["Security"]["customFields"]["customField"][0]["values"]["value"]
});
The error I get is:
Accessed JArray values with invalid key value: "sedol". Int32 array
index expected
I think I'm really close to figuring this out. What should I do to fix the code? If there is an alternative to get the SEDOL and ADP value, please do let me know?
[UPDATE1] I've started working with dynamic ExpandoObject. Here's the code that I've used so far:
dynamic obj = JsonConvert.DeserializeObject<ExpandoObject>(json, new ExpandoObjectConverter());
foreach (dynamic element in obj)
{
Console.WriteLine(element.DataFeed.Issuer[0].id);
Console.WriteLine(element.DataFeed.Issuer[0].securities.Security.sedol);
Console.ReadLine();
}
But I'm now getting the error 'ExpandoObject' does not contain a definition for 'DataFeed' and no extension method 'DataFeed' accepting a first argument of type 'ExpandoObject' could be found. NOTE: I understand that this json text is malformed. One instance has an array & the other is an object. I want the code to be agile enough to handle both instances.
[UPDATE2] Thanks to #dbc for helping me with my code so far. I've updated the json text above to closely match my current environment. I'm now able to get the SEDOLs & ADP codes. However, when I'm trying to get the 1st analyst, my code only works on objects and produces nulls for the analysts that are part of an array. Here's my current code:
var compInfo = from issuer in feed.SelectTokens("DataFeed.Issuer").SelectMany(i => i.ObjectsOrSelf())
let security = issuer.SelectTokens("securities.Security").SelectMany(s => s.ObjectsOrSelf()).FirstOrDefault()
where security != null
select new
{
Id = (string)issuer["id"], // Change to (string)issuer["id"] if id is not necessarily numeric.
CompName = (string)issuer["name"],
SEDOL = (string)security["sedol"],
ADP = security["customFields"]
.DescendantsAndSelf()
.OfType<JObject>()
.Where(o => (string)o["#name"] == "ADP Security Code")
.Select(o => (string)o.SelectToken("values.value"))
.FirstOrDefault(),
Analyst = security["coverage"]
.DescendantsAndSelf()
.OfType<JObject>()
.Select(jo => (string)jo.SelectToken("Coverage.analyst.#lastName"))
.FirstOrDefault(),
};
What do I need to change to always select the 1st analyst?

If you want all SEDOL & ADP values with the associated issuer Id and CompName for each, you can do:
var compInfo = from issuer in feed.SelectTokens("DataFeed.Issuer").SelectMany(i => i.ObjectsOrSelf())
from security in issuer.SelectTokens("securities.Security").SelectMany(s => s.ObjectsOrSelf())
select new
{
Id = (long)issuer["id"], // Change to (string)issuer["id"] if id is not necessarily numeric.
CompName = (string)issuer["name"],
SEDOL = (string)security["sedol"],
ADP = security["customFields"]
.DescendantsAndSelf()
.OfType<JObject>()
.Where(o => (string)o["#name"] == "ADP Security Code")
.Select(o => (string)o.SelectToken("values.value"))
.FirstOrDefault(),
};
Using the extension methods:
public static class JsonExtensions
{
public static IEnumerable<JToken> DescendantsAndSelf(this JToken node)
{
if (node == null)
return Enumerable.Empty<JToken>();
var container = node as JContainer;
if (container != null)
return container.DescendantsAndSelf();
else
return new[] { node };
}
public static IEnumerable<JObject> ObjectsOrSelf(this JToken root)
{
if (root is JObject)
yield return (JObject)root;
else if (root is JContainer)
foreach (var item in ((JContainer)root).Children())
foreach (var child in item.ObjectsOrSelf())
yield return child;
else
yield break;
}
}
Then
Console.WriteLine(JsonConvert.SerializeObject(compInfo, Formatting.Indented));
Produces:
[
{
"Id": 1528,
"CompName": "ZYZ.A a Test Company",
"SEDOL": "SEDOL111",
"ADP": "ADPSC1111"
},
{
"Id": 1519,
"CompName": "ZVV Test",
"SEDOL": "SEDOL112",
"ADP": "ADPS1133"
},
{
"Id": 1519,
"CompName": "ZVV Test",
"SEDOL": "SEDOL112",
"ADP": "ADPS1133"
}
]
However, in the query you have written so far, you seem to be trying to return only the first SEDOL & ADP for each issuer. If that is really what you want, do:
var compInfo = from issuer in feed.SelectTokens("DataFeed.Issuer").SelectMany(i => i.ObjectsOrSelf())
let security = issuer.SelectTokens("securities.Security").SelectMany(s => s.ObjectsOrSelf()).FirstOrDefault()
where security != null
select new
{
Id = (long)issuer["id"], // Change to (string)issuer["id"] if id is not necessarily numeric.
CompName = (string)issuer["name"],
SEDOL = (string)security["sedol"],
ADP = security["customFields"]
.DescendantsAndSelf()
.OfType<JObject>()
.Where(o => (string)o["#name"] == "ADP Security Code")
.Select(o => (string)o.SelectToken("values.value"))
.FirstOrDefault(),
};
Which results in:
[
{
"Id": 1528,
"CompName": "ZYZ.A a Test Company",
"SEDOL": "SEDOL111",
"ADP": "ADPSC1111"
},
{
"Id": 1519,
"CompName": "ZVV Test",
"SEDOL": "SEDOL112",
"ADP": "ADPS1133"
}
]
As an aside, since your JSON is rather polymorphic (properties are sometimes arrays of objects and sometimes just objects) I don't think deserializing to a class hierarchy or ExpandoObject will be easier.
Update
Given your updated JSON, you can use SelectTokens() with the JSONPath recursive search operator .. to find the first analyst's last name, where the recursive search operator handles the fact that the analysts might or might not be contained in an array:
var compInfo = from issuer in feed.SelectTokens("DataFeed.Issuer").SelectMany(i => i.ObjectsOrSelf())
let security = issuer.SelectTokens("securities.Security").SelectMany(s => s.ObjectsOrSelf()).FirstOrDefault()
where security != null
select new
{
Id = (string)issuer["id"], // Change to (string)issuer["id"] if id is not necessarily numeric.
CompName = (string)issuer["name"],
SEDOL = (string)security["sedol"],
ADP = (string)security["customFields"]
.DescendantsAndSelf()
.OfType<JObject>()
.Where(o => (string)o["#name"] == "ADP Security Code")
.Select(o => o.SelectToken("values.value"))
.FirstOrDefault(),
Analyst = (string)security.SelectTokens("coverage.Coverage..analyst.#lastName").FirstOrDefault(),
};

Also note your JSON is malformed.
The proxy class created can't figure out the type of Security because in one instance it's an array, and in another it's a simple object.
Original answer:
I used json2csharp to help me get some classes, and now I can use the types:
var obj = JsonConvert.DeserializeObject<RootObject>(json);
var result = from a in obj.DataFeed.Issuer
select new
{
Sedol = a.securities.Security.sedol,
Name = a.name
};
Classes:
public class Values
{
public object value { get; set; }
}
public class CustomField
{
public string name { get; set; }
public string type { get; set; }
public Values values { get; set; }
}
public class CustomFields
{
public List<CustomField> customField { get; set; }
}
public class Security
{
public string id { get; set; }
public string sedol { get; set; }
public CustomFields customFields { get; set; }
}
public class Securities
{
public Security Security { get; set; }
}
public class Issuer
{
public string id { get; set; }
public string name { get; set; }
public string clientCode { get; set; }
public Securities securities { get; set; }
}
public class DataFeed
{
public string FeedName { get; set; }
public List<Issuer> Issuer { get; set; }
}
public class RootObject
{
public DataFeed DataFeed { get; set; }
}

Using newtonsoft.json
dynamic obj = JsonConvert.DeserializeObject<ExpandoObject>(json);
I dont have my ide in front of me but the value should be at:
obj.DataFeed.Issuer[0].securities.Security.sedol

MongoDB Csharp: Dictionary of Lists "must implement IBsonArraySerializer and provide item serialization info"

This is simplified code of implementation.
The Document
public class Document
{
public Dictionary<string,List<Information>> Data { get; set; }
public class Information
{
public object Value { get; set; }
}
}
My Insert Action
var doc = new Document
{
Data = new Dictionary<string, List<Document.Information>>
{
{ "Some Key", new List<Document.Information>
{
new Document.Information
{
Value = "Some Value",
},
}}
}
};
myCollection.InsertOneAsync(doc);
Now i want to query the data using
var builder = MongoDB.Driver.Builders<Document>.Filter;
var filter = builder.ElemMatch(u => u.Data, w => w.Key.Equals("Some Key"));
var docs = myCollection.Find(filter).ToListAsync().Result;
But iam getting the follwing exception
System.AggregateException The serializer for field 'Data' must implement IBsonArraySerializer and provide item serialization info.
Question
How i can get serialization of Dictionaries of Lists to work?
Update
This is how the data looks like in the database.
{
"_id" : ObjectId("560e44fffa67693da7db224f"),
"Data" : {
"Some Key" : [
{
"Value" : "Some Value"
}
]
}
}

Develop Reference

C# (C-Sharp) is a programming language developed by Microsoft that runs on the .NET Framework.

MongoDb: documents with variable structure using and C# - c#

Related

How can I deserialize complex json repsonse into easy-to-deserialize structure using Newtonsoft.Json?

Extract a particular record from MongoDB where the records use Object GUID in C#

Update a JSON file with a model object in C#

LINQ to JSON - Query for object or an array

MongoDB Csharp: Dictionary of Lists "must implement IBsonArraySerializer and provide item serialization info"

Categories

Resources