Develop Reference

Mongo query a log collection with C# Api

To contextualize:
A Batch file has BachItems
Each BatchItem is a row and has a line number.
Each row is processed in order.
I'm new to NoSQL and mongo and I'd like to know how to query the last processing step executed (most recent EventType column) for each BatchItem (line number), filtering by BatchId?
For example, it should return the following result for BatchId "102030":
I believe I can achieve this using Aggregate and Group functions but don't know how.
Thanks.

You can do it as below:
db.batch.aggregate([
{
$match: {
"BatchId": 102030
}
},
{ $sort: { "Date": -1 } },
{
$group: {
_id: "$BatchItemId",
"doc": { "$push": { lastEventName: "$EventType" } },
}
},
{
$replaceRoot: {
newRoot: { $arrayElemAt: ["$doc", 0] }
}}
])

MongoDB grouping by month using aggregation timezone issue

I have the following aggregation structure where it groups the data by month and sums it up on a monthly basis starting from the beginning of the year. At the moment my documents are starting at month 9 and nothing before that.When I run the below aggregation, result shows that data starts from the month 8. I guess it is related to the timezone. Because only 3 hours of data (my timezone is +3) is listed in the month 8.
db.collectedData.aggregate(
// Pipeline
[
// Stage 1
{
$match: {
_Timestamp: {
$gte: ISODate("2017-01-01T21:00:00.000Z"),
$lte: ISODate("2019-04-19T01:30:00.000Z")
},
"Registers.totalconsumedactiveenergy" : {"$exists" : true}
}
},
// Stage 2
{
$project: {
"Datem" : "$_Timestamp",
"month" : {"$month" : "$_Timestamp" },
"device" : "$_DeviceID",
"datam" : "$$ROOT.Registers.totalconsumedactiveenergy"
}
},
// Stage 3
{
$group: {
"_id": {
"Month": "$month",
"Device" : "$device"
},
first:{$first:"$$ROOT.datam"},
last: { $last: "$$ROOT.datam" }
}
},
// Stage 4
{
$project: {
totalConsumption: { $sum: { $subtract: [ "$last", "$first" ] } },
month: "$_id.Month"
}
},
// Stage 5
{
$group: {
"_id": "$month",
"total": { "$sum": "$totalConsumption" }
}
},
// Stage 6
{
$sort: {
"_id" : 1
}
},
]
);
This works and gives me results like this
From my application I am sending localtimezone to the aggregation but still no success.
Earliest date in my documents is
2017-08-31T21:00:00.000Z (UTC)
How can I solve this?

Selecting value from a JSON string

I am having a heck of a time getting a particular value from json and am hoping someone can help. The json I'm using comes from a form post, so will typically be different each time, but the IDs will always be the same (just like every form!)...and I do not have control over the format of the json. I am able to get the name value from this just fine (using Newtonsoft.Json), but am stuck on how to get the "More then 5 years" value...
Here is what I am using to get the "My name is" value, which (obviously) doesn't work for "Years of Employment".
C#:
obj["form_response"]["answers"].SelectToken("$.[?(#field.id == '26')]..text").ToString();
JSON:
{
"event_type":"form_response",
"form_response":{
"definition":{
"id":"c33500e5",
"title":"Employee Satisfaction",
"fields":[
{
"id":"33",
"title":"Years of employment:",
"type":"multiple_choice"
},
{
"id":"26",
"title":"My name is:",
"type":"short_text"
}
]
},
"answers":[
{
"type":"choice",
"choice":{
"label":"More than 5 years"
},
"field":{
"id":"33",
"type":"multiple_choice"
}
},
{
"type":"text",
"text":"Bill Dunn",
"field":{
"id":"26",
"type":"short_text"
}
}
]
}
}

The value you want is nested as follows: { "choice": { "label": "More than 5 years", ... } } so you can do:
var result = obj["form_response"]["answers"].SelectToken("$.[?(#field.id == '33')].choice.label");
var resultJson = result.ToString();

Elasticsearch - Rolling up "other" results

I'm trying to rollup some of my 'other' results using Elasticsearch. Ideally, I'd like my query to return the top N hits and then roll the rest of the data up into an N+1 hit titled "Other".
So for example, if I'm trying to aggregate "Institutions by Total Value", I'd get back 10 Institutions with the most value and then the total aggregated value of the other institutions as another record. The purpose is that I'd like to see the total value aggregated across all institutions but not have to list thousands.
An example search I've been using is:
GET my_index/institution/_search?pretty=true
{
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
... terms queries ...
]
}
}
}
},
"aggs": {
"dimension_type_name_agg": {
"terms": {
"field": "institution_name",
"order": {
"metric_sum_total_value_agg": "desc"
},
"size": 0
},
"aggs": {
"metric_sum_total_value_agg": {
"sum": {
"field": "total_value"
}
},
"metric_count_account_id_agg": {
"value_count": {
"field": "institution_id"
}
}
}
}
}
}
I'm curious as to if this can be done by modifying a query like the one given above. Also, I'm using C# and Nest/Elasticsearch.NET so any tips on how this translates to that side is appreciated as well.

Generate lightweight JSON using DataContractJsonSerializer

I'm trying to generate JSON using C# and DataContractJsonSerializer in .Net 3.5. The problem is that I can't figure out how to build the structure correct for the result I need.
I've tried to reproduce PHP's associative arrays using both hashtables, list objects and arraylists but can't figure out how to generate my result in the best way using DataContractJsonSerializer without having to create my own recursive loop for building JSON.
The closest approach is using the a Dictionary < string, Dictionary < string, string>> aproach but the result is too big as I can't "name" the keys.
This is what I get:
[
{
"Key":"status",
"Value":[
{
"Key":"value",
"Value":"ok"
}
]
},
{
"Key":"1",
"Value":[
{
"Key":"name",
"Value":"Result1"
},
{
"Key":"details",
"Value":"Result 1 details"
}
]
},
{
"Key":"2",
"Value":[
{
"Key":"name",
"Value":"Result2"
},
{
"Key":"details",
"Value":"Result 2 details"
}
]
},
{
"Key":"caller",
"Value":[
{
"Key":"value",
"Value":"1135345"
}
]
}
]
This is what I want:
{
"status":"ok",
"response":[
{
"id":1,
"name":"Result1"
"details":"Result 1 details"
},
{
"id":2,
"name":"Result2"
"details":"Result 2 details"
},
{
"id":3,
"name":"Result3"
"details":"Result 3 details"
],
"caller":"1135345"
}
Does anybody have a clue how I can generate this piece of JSON using C# without having to load the entire Json.NET framework? I need this to be generated as fast as possible as this project aims to become an site search engine.

You should look at using the JavaScriptSerializer. It's part of the .NET framework (distributed in System.Web.Extensions). To get the result you want you can do this:
var results = new[]
{
new{id=1,name="Result 1"},
new{id=2,name="Result 2"},
new{id=3,name="Result 3"}
};
var js = new JavaScriptSerializer();
var result = js.Serialize(new
{
status = "ok",
response = results,
caller = 1135345
});
You can either use anonymous classes, or any existing ones. Works perfectly fine :)
The return value of this call would be:
{"status":"ok","response":[{"id":1,"name":"Result 1"},{"id":2,"name":"Result 2"},{"id":3,"name":"Result 3"}],"caller":1135345}

Using the JavaScriptSerializer rather than DataContractJsonSerializer on a Dictionary will remove the Key/Value json attributes and make them into a "keyed" array.
http://msdn.microsoft.com/en-us/library/bb412168.aspx
Have you attributed your classes with the [DataContract] and [DataMember] decorators?
http://msdn.microsoft.com/en-us/library/bb412179.aspx