Develop Reference

ElasticSearch's dynamic index mapping dropping logs due to poor field names

I have a semi-large project that has been using nlog, and throughout I re-used alot of field names for different datatypes. I started to send my logs (including all log properties/fields) to ElasticSearch, and now its starting to haunt me. I noticed if ElasticSearch is unable to convert a field to it's dataType it will just drop the log entirely. The dynamic index mapping is deciding the dataType depending on what it's seen first.
Is there anyway I can tell the index to use a default dataType like string?
PS; The Instance is cloud hosted, I access it through Kibana, and I have no Idea where to find a log that tells me if/when it drops logs for parsing errors.
Edit
Index Mapping
PUT /indexName
{
"mappings": {
"properties": {
"#domain": {
"type": "keyword"
},
"#logTarget": {
"type": "keyword"
},
"#logger": {
"type": "keyword"
},
"#memUsage": {
"type": "long"
},
"#processID": {
"type": "integer"
},
"#serviceGUID": {
"type": "keyword"
},
"#timestamp": {
"type": "date"
},
"level": {
"type": "keyword"
},
"message": {
"type": "text"
}
}
}
}
Unfortunately I don't know the api for pushing logs to elastic but here are some examples of what they might look like. Theses aren't the best examples but they show the overlapping/re-use of field names.
Example 1:
//NLog structured log
logger.LogInfo("That Lady has {count} cats", 5);
//JSON Object
{
"#domain": "Service1",
"#logTarget": "None",
"#logger": "AppName.Program.Main",
"#memUsage": 100,
"#processID": 17000,
"#serviceGUID": 0,
"level": "INFO",
"message": "That Lady has 5 cats",
"count": 5,
}
Example 2:
//NLog structured log
int count = 3;
logger.LogInfo("Loaded {count}", count + " dogs");
//JSON Object
{
"#domain": "Service1",
"#logTarget": "None",
"#logger": "AppName.Program.Main",
"#memUsage": 100,
"#processID": 17000,
"#serviceGUID": 0,
"level": "INFO",
"message": "Loaded "5 dogs"",
"count": 5,
}
Example 3:
//NLog structured log
object count = nil;
logger.LogInfo("Value is {count}", count);
//JSON Object
{
"#domain": "Service1",
"#logTarget": "None",
"#logger": "AppName.Program.Main",
"#memUsage": 100,
"#processID": 17000,
"#serviceGUID": 0,
"level": "INFO",
"message": "Loaded "5 dogs"",
"count": 5,
}

Getting values from nested object with C#

I have this object and I am trying to get the lat and lon values in the location field within telemetry. I am new to c#, but know python. What I would have done in python was:
item.details["telemetry"].location.lat
Can someone please show me how this is supposed to be done in c#?
"details": {
"asset": {
"id": "5ca12266-35fe-4f75-8593-588fba777d6d",
"name": "ZS-FOO"
},
"assetType": {
"id": "87bc0a83-045d-4810-888c-237b5ef17ea4",
"name": "FOO"
},
"telemetry": {
"flags": 0,
"ownerId": "4adc68e4-7113-4b0f-8aba-dea213e8a948",
"originId": "09e0021f-9c54-425b-ae23-cbfe3c786a66",
"type": "telemetry",
"linked": ["5ca12266-35fe-4f75-8593-588fba777d6d"],
"date": "2017/01/20 13:46:01",
"received": "2017/01/20 13:46:21",
"active": true,
"location": {
"lon": 116072,
"lat": -87448,
"speed": 74,
"altitude": 98.228,
"heading": 56,
"accuracy": 5,
"age": 0
},
"zones": [],
"routes": null,
"state": null,
"telemetry": {
"msg_type": 0,
"vert_speed": 1.2,
"hdop": 1.65,
"vdop": 3.51,
"movement": 1,
"odo_counter": 162704.12317,
"hours_00_counter": 1027.885442,
"idle_counter": 0
},
"io": null,
"spd": null,
}
}

As i said in comment download Newtonsoft.Json
JObject data= JObject.Parse(data);
JToken details = data["details"];
string lat = details["telemetry"]["location"]["lat"].ToString();

Firstly, generate models for your data. You can do it manually by creating classes or if you are in Visual Studio, this would be helpful:
Edit=> Paste Special=> Paste JSON as class. This will generate the classes for you.
Afterwords, go with Json.NET or any other library to parse your file:
public class Program{
static void Main (string[] args){
var client = new WebClient();
var jsonString = client.DownloadString("Your link to Json file");
var response= JsonConvert.DeserializeObject</*Json Class*/>(jsonString);
//Now you can retrieve the data like this:
var lat = response.details.telemetry.location.lat;
}
}
Don't forget to import:
using System.Net;
using Newtonsoft.Json;

how to build a good c# data structure

my aim is to develop a custom search based on goeuro.com(overview) in my spare time.
I simplyfied the search parameters to the minimum.
For example(you can try this, as loong as the search_id is valid):
http://www.goeuro.com/GoEuroAPI/rest/api/v5/results?&search_id=428558909
The search_id will be generated when you visit http://www.goeuro.com and enter the first time your search parameters.
This is the simplified data structure I generated from http://json2csharp.com/ using this as my input JSON:
{
"serviceVersion": "v1.0.0",
"sortVariants": null,
"companies": {
"1007": {
"name": "Eurolines Germany",
"logoUrl": "http://cdn-goeuro.com/static_content/web/logos/{size}/eurolines_germany.png",
"id": "1007"
}
},
"outbounds": {
"3624107261930718525-38-flight-1-27": {
"companyId": "38",
"mode": "flight",
"duration": "873",
"outboundId": "3624107261930718525",
"journeyId": "27",
"departureTime": "2017-01-15T19:12:00.000+01:00",
"arrivalTime": "2017-01-16T09:45:00.000+01:00",
"stops": "1",
"price": 16209,
"updatedAt": "1",
"segments": [ 1344486303, 574447503, 689435565, 833161604 ],
"arrivalOvernightOffset": 1,
"overnightOffset": 1
}
},
"query": {
"roundTrip": false,
"airportToAirport": false,
"locationsOutsideEurope": false,
"searchId": "428558909",
"departurePosition": "377001",
"arrivalPosition": "398532",
"departureDate": "2017-01-15T00:00:00.000+01:00",
"passengers": {
"adults": 1,
"children": 0,
"infants": 0
},
"userInfo": {
"userLocale": "en",
"userCurrency": "EUR"
},
"searchModes": {
"bus": {
"status": "done",
"resultsQty": 13,
"filteredResultsQty": 13
},
"flight": {
"status": "done",
"resultsQty": 276,
"filteredResultsQty": 276
},
"train": {
"status": "done",
"resultsQty": 4,
"filteredResultsQty": 4
}
}
},
"itineraries": [
{ "outboundLegId": "3624107261930718525-38-flight-1-27" }
],
"segmentDetails": {
"1002857016": {
"type": "flight",
"departureTime": "2017-01-16T08:35:00.000+01:00",
"arrivalTime": "2017-01-16T12:05:00.000+02:00",
"departurePosition": "313870",
"arrivalPosition": "314920",
"duration": "150",
"company": "190",
"transportId": "ca6199",
"direction": "OUTBOUND",
"overnightOffset": 1,
"departureOvernightOffset": 1,
"arrivalOvernightOffset": 1
}
},
"positions": {
"2091737": {
"positionType": "busstation",
"name": "Warszawa, Warszawa Centralny",
"cityName": "Warsaw",
"latitude": 52.22782,
"longitude": 21.00224
}
},
"currencies": {
"EUR": {
"name": "Euro",
"symbol": "€"
}
}
}
The result from http://json2csharp.com/ is pretty good, but on the other side it generates something like this
public class Outbounds
{
public __invalid_type__362410726193071852538Flight127 __invalid_name__3624107261930718525-38-flight-1-27 { get; set; }
}
I see two problems here:
using __invalid_name__3624107261930718525-38-flight-1-27 is not a valid identifier in c#
and the above mentioned name is some random name, which I can not rely on my data structure.
So, my actually questions are:
How can I handle the request ?
How shall the data structure look like ?
By the way this is the code I am using(plus the generated results from http://json2csharp.com/):
static void Main()
{
var client = new RestClient("https://www.goeuro.com");
var request = new RestRequest("/GoEuroAPI/rest/api/v5/results?&search_id=428558909", Method.GET);
request.RequestFormat = DataFormat.Json;
// contentType: "application/json; charset=utf-8",
var response = client.Execute<Response>(request).Data;
}
Actually I found an existing solution(partly) for my problem:
https://github.com/evgenTraytyak/goeuro-api
, which is actually written in node.js, but I need c#, but by the way this example does not quite work(only if you have an existing search_id) and at the end I want to get the search_id from goeuro, without manually type it to my code.
Maybe the reason that using this is that the JSON-Format changed...

Which JSON adapter enables Ember to propagate all necessary requests to the server to maintain model consistence?

I am using Andy Crum's EmberDataModelMaker.
Having punched in the following two classes
// app/models/server-item.js
export default DS.Model.extend({
hostName: DS.attr('string'),
syncServers: DS.hasMany('string'),
subscribers: DS.hasMany('string'),
mailHost: DS.attr('string'),
mailHostLogin: DS.hasMany('credentials')
});
// app/models/credentials.js
export default DS.Model.extend({
user: DS.attr('string'),
password: DS.attr('string'),
server: DS.belongsTo('serverItem')
});
It's showing the following three different expected JSON formats (a very nice feature btw.):
DS.RESTAdapter
"serverItems": [
{
"id": 1,
"hostName": "foo",
"syncServers": [
<stringids>
],
"subscribers": [
<stringids>
],
"mailHost": "foo",
"mailHostLogin": [
<Credentialsids>
]
}
],
"credentials": [
{
"id": 1,
"user": "foo",
"password": "foo",
"server": <ServerItemid>
}
]
DS.ActiveModelAdapter
"serverItems": [
{
"id": 1,
"host_name": "foo",
"sync_server_ids": [
<stringids>
],
"subscriber_ids": [
<stringids>
],
"mail_host": "foo",
"mail_host_login_ids": [
<Credentialsids>
]
}
],
"credentials": [
{
"id": 1,
"user": "foo",
"password": "foo",
"server_id": <ServerItemid>
}
]
DS.JSONAPIAdapter
{
"data": {
"type": "server-items",
"id": "1",
"attributes": {
"HostName": "foo",
"MailHost": "foo",
},
"relationships": {
"SyncServers": {
"data": {
"type": "SyncServers",
"id": <SyncServersid>
}
},
"Subscribers": {
"data": {
"type": "Subscribers",
"id": <Subscribersid>
}
},
"MailHostLogin": {
"data": {
"type": "MailHostLogin",
"id": <MailHostLoginid>
}
}
},
"included": [
{
<sideloadedrelationships>
]
}
}
}
{
"data": {
"type": "credentials",
"id": "1",
"attributes": {
"User": "foo",
"Password": "foo",
},
"relationships": {
"Server": {
"data": {
"type": "Server",
"id": <Serverid>
}
}
},
"included": [
{
<sideloadedrelationships>
]
}
}
}
I am going to implement (or rather change) some WebServices on the Server side (using C#, ASP.NET Web API). Currently, the WebService already creates a result that is pretty similar to the format expected with DS.RESTAdapter - obviously, it would be ideal if I could use it without compromising the Data Integrity - can I?
If yes, would it empower Ember Data to send all the requests necessary to maintain the data consistency on the server? Meaning, would the client send a DELETE request to the server not only for the ServerItem but also for the Credentials item that is referenced via the mailHostLogin property when the user wants to delete a ServerItem?
If not: are both of the other two adapters fulfilling the above mentioned consistency requirement? Which of the other two should I implement - any experiences/recommendations out there?

You should choose whichever Adapter closest fits your API data structure as a basis(sounds like DS.RESTAdapter in this case). You can extend the adapters and serializers that are a closest fit to make any necessary adjustments(this can be done both application wide or on a per model basis).
However, I don't think that the Ember Data model relationships(i.e. belongsTo and hasMany) are binding in such a way that will automatically result in the "data consistency" you are looking for. If your application requirements are to delete all associated Credentials records when a ServerItem is deleted, I would recommend doing that server side when handling the DELETE ServerItem API request. That would result in better performance(1 HTTP call instead of 2 or N depending if credentials can be deleted in bulk) and be much less error prone due to potential network or other failure of calls to delete Credentials after a ServerItem is deleted.
After a successful ServerItem delete, you could loop through it's credentials and unload the records from the client side store to keep it in sync with the new state on the server. Something like:
serverItemCredentials.forEach(function(id) {
if (this.store.recordIsLoaded('credential', id)) {
this.store.unloadRecord(this.store.peekRecord('credential', id));
}
});

Deserializing Chrome Bookmark JSON Data in C#

In response to a question I asked a few days ago, I'm attempting to stretch myself a little, and do something that I've not really focussed on much before. I've done some searching (both here, and in general), but can't find the answers (or even reasonable hints) to what I want to achieve (though, a few things come close-ish).
Basically, I'm trying to deserialize the data for the Google Chrome bookmarks file using the Json.NET library (though, if there's a better alternative, I'm all for that - the documentation for this library is a little confusing in places). I'm a little confused as to the next step to take, due primarily to being used to PHP's fantastic handling of JSON data (using json_decode()), allowing for a single function call, and then simple associative-array access.
The library (Json.NET) wants me to specify an Object type that it can deserialize the JSON data into, but I'm not really sure how to go about structuring such an Object, given the format of the Bookmarks file itself. The format is something along the lines of:
{
"roots": {
"bookmark_bar": {
"children": [ {
"children": [ {
"date_added": "12880758517186875",
"name": "Example URL",
"type": "url",
"url": "http://example.com"
}, {
"date_added": "12880290253039500",
"name": "Another URL",
"type": "url",
"url": "http://example.org"
} ],
"date_added": "12880772259603750",
"date_modified": "12880772452901500",
"name": "Sample Folder",
"type": "folder"
}, {
"date_added": "12880823826333250",
"name": "Json.NET",
"type": "url",
"url": "http://james.newtonking.com/pages/json-net.aspx";
} ],
"date_added": "0",
"date_modified": "12880823831234250",
"name": "Bookmarks bar",
"type": "folder"
},
"other": {
"children": [ ],
"date_added": "0",
"date_modified": "0",
"name": "Other bookmarks",
"type": "folder"
}
},
"version": 1
}
Now, in PHP, I'd be far more used to doing something along the lines of the following, to get the data I wanted, and ending up with 'Json.NET':
$data['roots']['bookmark_bar']['children'][0]['name'];
I can work out, simply enough, what objects to create to represent the data (something like a root object, then a bookmark list object, and finally an individual bookmark object) - but I'm really not sure as to how to implement them, and then get the library to deserialize into the relevant objects correctly.
Any advice that can be offered would be greatly appreciated.

It is not necessary to declare a type that reflects the json structure:
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using System.IO;
using System;
class Program
{
static void Main(string[] args)
{
string json =
#"
{
""roots"": {
""bookmark_bar"": {
""children"": [ {
""children"": [ {
""date_added"": ""12880758517186875"",
""name"": ""Example URL"",
""type"": ""url"",
""url"": ""http://example.com""
}, {
""date_added"": ""12880290253039500"",
""name"": ""Another URL"",
""type"": ""url"",
""url"": ""http://example.org""
} ],
""date_added"": ""12880772259603750"",
""date_modified"": ""12880772452901500"",
""name"": ""Sample Folder"",
""type"": ""folder""
}, {
""date_added"": ""12880823826333250"",
""name"": ""Json.NET"",
""type"": ""url"",
""url"": ""http://james.newtonking.com/pages/json-net.aspx""
} ],
""date_added"": ""0"",
""date_modified"": ""12880823831234250"",
""name"": ""Bookmarks bar"",
""type"": ""folder""
},
""other"": {
""children"": [ ],
""date_added"": ""0"",
""date_modified"": ""0"",
""name"": ""Other bookmarks"",
""type"": ""folder""
}
},
""version"": 1
}
";
using (StringReader reader = new StringReader(json))
using (JsonReader jsonReader = new JsonTextReader(reader))
{
JsonSerializer serializer = new JsonSerializer();
var o = (JToken)serializer.Deserialize(jsonReader);
var date_added = o["roots"]["bookmark_bar"]["children"][0]["date_added"];
Console.WriteLine(date_added);
}
}