I have 1 json file and these lines of code:
Here's my code:
using (var r = new ChoJSONReader("data.json")
.Configure(c => c.ThrowAndStopOnMissingField = true)
.Configure(c => c.DefaultArrayHandling = true)
.Configure(c => c.FlattenNode = true)
.Configure(c => c.IgnoreArrayIndex = false)
.Configure(c => c.NestedKeySeparator = '.')
.Configure(c => c.NestedColumnSeparator = '.')
)
{
var dt = r.AsDataTable();
Console.WriteLine(dt.DumpAsJson());
}
My data.json file:
{
"BrandId": "998877665544332211",
"Categories": [
"112233445566778899"
],
"Contact": {
"Phone": [
{
"Value": "12346789",
"Description": {
"vi": "Phone"
},
"Type": 1
},
{
"Value": "987654321",
"Description": {
"vi": "Phone"
},
"Type": 1
}
]
}
}
After running this code, I got the output like this:
[
{
"BrandId": "998877665544332211",
"Contact.Phone.0.Value": "12346789",
"Contact.Phone.0.Description.vi": "Phone",
"Contact.Phone.0.Type": 1,
"Contact.Phone.1.Value": "987654321",
"Contact.Phone.1.Description.vi": "Phone",
"Contact.Phone.1.Type": 1,
"Category0": "112233445566778899"
}
]
The question here is how can I get some kind of output json without "0" at the flattened key node
Expected output:
[
{
"BrandId": "998877665544332211",
"Contact.Phone.Value": "12346789",
"Contact.Phone.Description.vi": "Phone",
"Contact.Phone.Type": 1,
"Category": "112233445566778899"
},
{
"BrandId": "998877665544332211",
"Contact.Phone.Value": "987654321",
"Contact.Phone.Description.vi": "Phone",
"Contact.Phone.Type": 1,
"Category": "112233445566778899"
}
]
I've research by many ways but it's results doesn't as same as my expected result.
Thanks for any kind of help
As your json is nested/complex in nature, you need to unpack and flatten into multiple simple data element rows using ChoETL/Linq as below.
ChoETLSettings.KeySeparator = '-';
using (var r = ChoJSONReader.LoadText(json)
.WithField("BrandId")
.WithField("Category", jsonPath: "Categories[0]", isArray: false)
.WithField("Phone", jsonPath: "Contact.Phone[*]")
)
{
var dt = r.SelectMany(rec => ((IList)rec.Phone).OfType<dynamic>().Select(rec1 =>
{
dynamic ret = new ChoDynamicObject();
ret["BrandId"] = rec.BrandId;
ret["Contact.Phone.Value"] = rec1.Value;
ret["Contact.Phone.Description.vi"] = rec1.Description.vi;
ret["Contact.Phone.Type"] = rec1.Type;
ret["Category"] = rec.Category;
return ret;
})).AsDataTable();
dt.DumpAsJson().Print();
}
Sample fiddle: https://dotnetfiddle.net/PHK8LO
Related
I have multiple document like below in mongo collection(giving just one sample).
{
"_id": "5fdb",
"createddate": "2020-12-17",
"orders": [
{
"_id": "4c65",
"sourcesystemrecordid": null,
"accepteddate": "2020-12-19",
"fulfillment": [
{
"_id": "7d3ceb",
"createdby": "Azekry",
"systemid": "123",
"systemrecordname": "source1"
}
]
}
]
}
Under fulfilment, I want to update "systemid" value from "123" to "789" for all the documents where "systemrecordname" = "source1" and for that I have written below piece of code but the data is not updated. Can you help me on this?
foreach (var item in resultFromDatabse)
{
var ordr = item.orders;
foreach (var e in ordr)
{
var actualval = "789";
foreach (var ef in e.fulfillment.Where(x => x.systemrecordname == "source1"))
{
filter = builder.Eq("systemrecordname", ef.systemrecordname);
var update = Builders<EquipmentDemandPlan>.Update.Set("orders.$[e].fulfillment.$[ef].systemid", actualval);
UpdateOneModel<EquipmentDemandPlan> updateOne = new UpdateOneModel<EquipmentDemandPlan>(filter, update)
{
ArrayFilters = new List<ArrayFilterDefinition> {
new BsonDocumentArrayFilterDefinition<BsonDocument>(new BsonDocument("ef.systemrecordname", ef.systemrecordname))
}
};
bulkupdate.Add(updateOne);
}
}
}
collection.BulkWriteAsync(bulkupdate);
you only need arrayFilters in $set
db.collection.update({
"orders.fulfillment.systemrecordname": "source1"
},
{
$set: {
"orders.$[o].fulfillment.$[f].systemid": "789"
}
},
{
arrayFilters: [
{
"o._id": {
$exists: true
}
},
{
"f.systemrecordname": "source1"
}
],
multi: true
})
mongoplayground
Edit 8 Sept
I'm having trouble parsing a configuration spreadsheet into JSON (generated from a multitude of other spreadsheets)
Initially I'm loading a spreadsheet from Excel using ClosedXML into a Datatable
After much rewriting, I'm pretty close. The issue I've got is that I need to exclude any options under pages where there's no But_Type_P1 [aka type] value - the button_num serves no useful purpose if it doesn't have a button type associated with it - it just indicates it's position on an X-Y grid on a hardware device.
I need to work out how I filter out any children if specific ones are null.
If the alias name is empty (or null) we can generate a placeholder for that downstream. ... I haven't added those to the export yet as they are just cosmetic.
[
{
"type": "function",
"label": "VM",
"Location": "lhs_front_pc",
"options": [
{
"type": "page",
"label": "VM|PCs",
"options": [
{
"type": "Source",
"index": "1",
"value": "45401"
},
{
"type": "Source",
"index": "2",
"value": "45402"
},
...continues...
{
"type": "page",
"label": "HW|PCs",
"options": [
{
"type": "Source",
"index": "1",
"value": "45751"
},
{
"type": "Source",
"index": "2",
"value": "45752"
},
{
"type": "Source",
"index": "3",
"value": "45753"
},
{
"type": "Source",
"index": "4",
"value": "45754"
},
{
"type": "",
"index": "5",
"value": ""
},
{
"type": "",
"index": "6",
"value": ""
},
I'm at the point now where I've got a filtered datatable for the function level,
and another one filtered for the pages for each function (up to 5 pages per function)
Code Snippet
public void dtToFunction(DataTable dt)
{
var functionTable = from instance in dt.AsEnumerable()
group instance by instance.Field<string>("Panel_Instance") into panel_instance
select new
{
Instance = panel_instance.Key,
List = panel_instance.ToList(),
};
foreach (var row in functionTable)
{
logger.Info($"{row.Instance.ToString()}");
var obj = new JArray();
var pageTable = from page in row.List.AsEnumerable()
group page by page.Field<string>("Page_P1") into page
select new
{
Page = page.Key,
List = page.ToList(),
};
var pageObj = new JArray();
try {
pageObj = new JArray(
row.List
.GroupBy(r => new { Page = r["Page_P1"] })
.Select(g => new JObject(
new JProperty("type", "page"),
new JProperty("label", g.Key.Page.ToString()),
new JProperty("options",g.Select(r =>
new JObject(
new JProperty("type", r["But_Type_P1"].ToString()),
new JProperty("index", r["Button_Num"]),
new JProperty("value", r["But_Idx_P1"])
)
)
)
)
)
);
}
catch (Exception exp)
{ logger.Error($"Exception: {exp.Message}"); }
try
{
obj = new JArray(
row.List
.GroupBy(r => new { Function = r["Function_Name"], Location = r["Location"] })
.Select(g => new JObject(
new JProperty("type", "function"),
new JProperty("label", g.Key.Function.ToString()),
new JProperty("Location", g.Key.Location.ToString()),
new JProperty("options",
pageObj
)
)
)
);
logger.Info($"{obj.ToString()}");
}
catch (Exception exp)
{
logger.Error($"Exception: {exp.Message}");
}
I've been through
DataTable to JSON
Convert DataTable to JSON with key per row
How to [group by] the datatable and put the result in another datatable?
How to convert datatable to json string using json.net?
For example I have a document below for collection = delivery:
{
"doc": [
{
"docid": "15",
"deliverynum": "123",
"text": "txxxxxx",
"date": "2019-07-18T12:37:58Z"
},
{
"docid": "16",
"deliverynum": "456",
"text": "txxxxxx",
"date": "2019-07-18T12:37:58Z"
},
{
"docid": "17",
"deliverynum": "999",
"text": "txxxxxx",
"date": "2019-07-18T12:37:58Z"
}
],
"id": "123",
"cancelled": false
}
is it possible to do a search with "deliverynum" = 999 and the output would be like below?
{
"doc": [
{
"docid": "17",
"deliverynum": "999",
"text": "txxxxxx",
"date": "2019-07-18T12:37:58Z"
}
],
"id": "123",
"cancelled": false
}
or should I make another Collection just for the Doc part?
I am having trouble making a query in C# for this kind of scenario.
In Mongo shell you can use the $(projection) operator:
db.collection.find({ "doc.deliverynum": "999" }, { "doc.$": 1 })
Corresponding C# code can look like below:
var q = Builders<Model>.Filter.ElemMatch(x => x.doc, d => d.deliverynum == "999");
var p = Builders<Model>.Projection.ElemMatch(x => x.doc, d => d.deliverynum == "999");
var data = Col.Find(q).Project(p).ToList();
You can also use q = Builders<Model>.Filter.Empty if you want to get all documents even if the don't contain deliverynum =``999
Consider I have a JSON data as:
{
"entities":[
{
"republish": false,
"OrgID": "",
"createdby": "730",
"questions": [
{
"sequence": "5",
"QuestionId": "57BB6DDC-A90A-10EE-E224-EC658A825871",
"metadata": [
{
"key": "Group",
"value": 0
},
{
"key": "Part",
"value": "0"
}
]
},
{
"sequence": "4",
"QuestionId": "57BB6DDC-A90A-10EE-E224-EC658A825871",
"metadata": [
{
"key": "Group",
"value": 1
},
{
"key": "Part",
"value": "A"
}
]
},
{
"sequence": "3",
"QuestionId": "57BB6DDC-A90A-10EE-E224-EC658A825871",
"metadata": [
{
"key": "Group",
"value": 1
},
{
"key": "Part",
"value": "B"
}
]
}
]
}
]
}
As you can see I have a list of questions available and in each question, I have a metadata which holds Key-Value pair.
Above example demonstrates, I have 3 questions and out of it 2 question, metadata key-value is "Group 1".
Now I want to do is combine the questions with a same key-value pair and treat it as one.
so in my final case, I will have 2 questions instead of 3. And out of that one question will have two separate questions inside.
And I want to achieve this using Linq query. And if possible please use Newtonsoft for parse if needed. I have been stuck for long onto this.
Things I have done:
public virtual HttpResponseMessage AddQuestionsToStandardMaster(TaxonomyMetaData objQuestion)
{
List<ResponseEntity> objResponseList = new List<ResponseEntity>();
try
{
if (ModelState.IsValid)
{
foreach (var objEntity in objQuestion.Entities)
{
EntityResponse objentityresponse = new EntityResponse();
ResponseEntity objResponse = new ResponseEntity();
}
List<Question> objQuestionList = new List<Question>();
if (objEntity.Questions.Length > 0)
{
foreach (var item in objEntity.Questions)
{
int questionTypeid = 0;
dynamic objQuestionJson = JObject.Parse(item.ToString())
}
}
}
Question objCurrentQuestion = new Question();
Question objQuestionforDelete = new Question();
JObject itemToParese = new JObject();
string SingleQuestionJson = objQuestionJson.GetValue("QuestionData").ToString();
string questionstem = "";
Regex rgx = new Regex("/\'");
objCurrentQuestion.Sequence = Convert.ToInt32(objQuestionJson.GetValue("sequence"));
objCurrentQuestion.tag = objQuestionJson.tag.ToObject<JToken[]>(); ;
objCurrentQuestion.metadata = objQuestionJson.metadata.ToObject<JToken[]>();
objCurrentQuestion.SingleQuestionJson = rgx.Replace(SingleQuestionJson, "'");
objCurrentQuestion.QuestionsType = questionTypeid;
objCurrentQuestion.QuestionsId = new Guid(objQuestionJson.GetValue("QuestionId").ToString());
objCurrentQuestion.VersionNo = Convert.ToInt32(objQuestionJson.GetValue("VersionNo"));
objCurrentQuestion.DisplayQuestionId = Convert.ToString(objQuestionJson.GetValue("DisplayQuestionId"));
objCurrentQuestion.OriginalQuestionId = Convert.ToString(objQuestionJson.GetValue("OriginalQuestionId"));
objCurrentQuestion.PassageText = Convert.ToString(objQuestionJson.GetValue("passage_text"));
objCurrentQuestion.PassageCode = Convert.ToString(objQuestionJson.GetValue("passage_id"));
objCurrentQuestion.PassageTitle = Convert.ToString(objQuestionJson.GetValue("passage_title"));
objCurrentQuestion.IsPublished = Convert.ToByte(true);
objCurrentQuestion.ProductId = objEntity.ProductID;
foreach (var metadata in objCurrentQuestion.metadata)
{
switch (metadata["key"].ToString())
{
case "Group":
objCurrentQuestion.Group = Convert.ToInt32(metadata["value"].ToString());
break;
case "Part":
objCurrentQuestion.Part = metadata["value"].ToString();
break;
}
}
objQuestionList.Add(objCurrentQuestion);
int counter = 1;
//Here I get the data in a group which needs to coverted to JSOn and then replace the original JSON data with this. But I know this is irrelevant to what we need to achieve.
var yui = objQuestionList.Where(tma => tma.Group == counter).Select(t => t).GroupBy(s => new { s.Group }).Where(p => p.Count() > 1).ToList();
//After proper conversion I need to enter this data to a database.
I am a bit unclear on what do you mean by make it
combine the questions with a same key-value pair and treat it as one
But here is how you can group by the JSON into groups in metadata you can do a custom select to what ever you want.
var text = #"{
""entities"":[
{
""republish"": false,
""OrgID"": """",
""createdby"": ""730"",
""questions"": [
{
""sequence"": ""5"",
""QuestionId"": ""57BB6DDC-A90A-10EE-E224-EC658A825871"",
""metadata"": [
{
""key"": ""Group"",
""value"": 0
},
{
""key"": ""Part"",
""value"": ""0""
}
]
},
{
""sequence"": ""4"",
""QuestionId"": ""57BB6DDC-A90A-10EE-E224-EC658A825871"",
""metadata"": [
{
""key"": ""Group"",
""value"": 1
},
{
""key"": ""Part"",
""value"": ""A""
}
]
},
{
""sequence"": ""3"",
""QuestionId"": ""57BB6DDC-A90A-10EE-E224-EC658A825871"",
""metadata"": [
{
""key"": ""Group"",
""value"": 1
},
{
""key"": ""Part"",
""value"": ""B""
}
]
}
]
}
]
}";
var json = JObject.Parse(text);
var groupedData = from entity in json["entities"]
from question in entity["questions"]
group question by question["metadata"][0]["value"] into questionGroup
select questionGroup;
foreach (var data in groupedData)
{
Console.WriteLine("_____________________");
Console.WriteLine("Group");
Console.WriteLine(data.Key);
foreach (var question in data)
{
Console.WriteLine(question["QuestionId"]);
}
Console.WriteLine("_____________________");
}
If "Group" is not always the fist item in the array you can do
question["metadata"].First(md => md.Value<string>("key") == "Group")["value"]
to get it.
I'm trying to do a SuggestCompletion query for a location (countries and cities), I'd like to perform the query over those two fields.
my mapping so far is the following:
var response = _client.CreateIndex(PlatformConfiguration.LocationIndexName,
descriptor => descriptor.AddMapping<LocationInfo>(
m => m.Properties(
p => p.Completion(s => s
.Name(n=>n.CountryName)
.IndexAnalyzer("simple")
.SearchAnalyzer("simple")
.MaxInputLength(50)
.Payloads()
.PreserveSeparators()
.PreservePositionIncrements()).
Completion(s=>s.Name(n => n.City)
.IndexAnalyzer("simple")
.SearchAnalyzer("simple")
.MaxInputLength(50)
.Payloads()
.PreserveSeparators()
.PreservePositionIncrements())
)));
Edit:
How I'm indexing the elements:
public bool IndexLocations(IList<LocationInfo> locations)
{
var bulkParams = locations.Select(p => new BulkParameters<LocationInfo>(p){
Id = p.Id,
Timestamp = DateTime.Now.ToTimeStamp()
});
var response = _client.IndexMany(bulkParams, PlatformConfiguration.LocationIndexName);
return response.IsValid;
}
Edit
After viewing the mappings I changed my query to the following:
var response = _client.Search<LocationInfo>(location =>
location.Index(PlatformConfiguration.LocationIndexName).
SuggestCompletion("locationinfo", f => f.OnField("countryName").Text(text).Size(1)));
and I also I tried:
var response = _client.Search<LocationInfo>(location =>
location.Index(PlatformConfiguration.LocationIndexName).
SuggestCompletion("countryName", f => f.OnField("countryName").Text(text).Size(1)));
.....And I still get an empty result
the mapping
{
"locationindex": {
"mappings": {
"locationinfo": {
"properties": {
"countryName": {
"type": "completion",
"analyzer": "simple",
"payloads": true,
"preserve_separators": true,
"preserve_position_increments": true,
"max_input_length": 50
}
}
},
"bulkparameters`1": {
"properties": {
"document": {
"properties": {
"city": {
"type": "string"
},
"countryName": {
"type": "string"
},
"countryTwoDigitCode": {
"type": "string"
},
"id": {
"type": "string"
},
"latitude": {
"type": "string"
},
"longitude": {
"type": "string"
}
}
},
"id": {
"type": "string"
},
"timestamp": {
"type": "long"
},
"versionType": {
"type": "long"
}
}
}
}
}
}
The support for IndexMany() with wrapped BulkParameters has been removed in NEST 1.0.0 beta 1
If you want to use a bulk with more advanced parameters you now have to use the Bulk() command.
The beta sadly still shipped with the BulkParameters class in the assembly
This has since been removed in the develop branch.
So what happens now is that you are actually indexing "bulkparameters``1``" type documents and not "locationinfo". So the mapping specified for "locationinfo" does not come into play.
See here for an example on how to use Bulk() to index many objects at once while configuring advanced parameters for individual items.