I want to index a collection of dynamic objects using the ElasticSearch Nest Client.
The objects are first materialized by NewtonsoftJson.NET from a JSON file into dynamic objects and then manipulated by the program. All objects get a property "Id". This should serve as "_id" field for ElasticSearch.
The "_id" field must be the same for identical data records in order to be able to update data later.
Because attributes "IdProperty" cannot be added to a dynamic object and a
mapping can also not be used, I was forced to the following solution.
I would like to keep the dynamic objects because I manipulate only a few properties and the other properties are of no interest to me.
var values = new List<dynamic>();
dynamic obj = new System.Dynamic.ExpandoObject();
obj.Id = "ABC";
obj.SomeValue0 = "12";
obj.SomeValue1 = 99;
values.Add(obj);
var descriptor = new BulkDescriptor();
// Now i want to Index this List
foreach (var doc in values) {
// Here the StackOverflowException will be thrown
descriptor.Index<object>(i => i
.Index("abc")
.Id(doc.Id)
.Document(doc));
}
client.Bulk(descriptor);
(Index a dynamic object using NEST - This was my inspiration)
This example raises a StackOverflow exception during indexing. It does not matter whether one or more objects are in the list.
Interestingly, the following method works without problems. The only thing that doesn't work is the
ElasticSearch "_id" field was generated automatically and therefore does not correspond to the "Id" field.
client.IndexMany(value, index);
What am I doing wrong with the first possibility and is there a possibility to set a "_id" on a dynamic object?
You need to
cast the Id property to string (or Nest.Id if it could be another type with an implicit conversion to Nest.Id, for example, Guid
cast the object to object
Here's an example
var client = new ElasticClient();
var values = new List<dynamic>();
dynamic obj = new System.Dynamic.ExpandoObject();
obj.Id = "ABC";
obj.SomeValue0 = "12";
obj.SomeValue1 = 99;
values.Add(obj);
var descriptor = new BulkDescriptor();
// Now i want to Index this List
foreach (var doc in values)
{
descriptor.Index<object>(i => i
.Index("abc")
.Id((Id)doc.Id)
.Document((object)doc));
}
client.Bulk(descriptor);
which will send a request like the following
POST http://localhost:9200/_bulk
{"index":{"_index":"abc","_type":"object","_id":"ABC"}}
{"Id":"ABC","SomeValue0":"12","SomeValue1":99}
Dynamic types don't play nicely with generic types and member access expressions, which looks to be related to runtime type resolution. In the example above, I would recommend using anonymous types and a List<object>
Related
I'd like to store several different object types in a single Cosmos DB container, as they are all logically grouped and make sense to read together by timestamp to avoid extra HTTP calls.
However, the Cosmos DB client API doesn't seem to provide an easy way of doing the reads with multiple types. The best solution I've found so far is to write your own CosmosSerializer and JsonConverter, but that feels clunky: https://thomaslevesque.com/2019/10/15/handling-type-hierarchies-in-cosmos-db-part-2/
Is there a more graceful way to read items of different types to a shared base class so I can cast them later, or do I have to take the hit?
Thanks!
The way I do this is to create the ItemQueryIterator and FeedResponse objects as dynamic and initially read them untyped so I can inspect a "type" property that tells me what type of object to deserialize into.
In this example I have a single container that contains both my customer data as well as all their sales orders. The code looks like this.
string sql = "SELECT * from c WHERE c.customerId = #customerId";
FeedIterator<dynamic> resultSet = container.GetItemQueryIterator<dynamic>(
new QueryDefinition(sql)
.WithParameter("#customerId", customerId),
requestOptions: new QueryRequestOptions
{
PartitionKey = new PartitionKey(customerId)
});
CustomerV4 customer = new CustomerV4();
List<SalesOrder> orders = new List<SalesOrder>();
while (resultSet.HasMoreResults)
{
//dynamic response. Deserialize into POCO's based upon "type" property
FeedResponse<dynamic> response = await resultSet.ReadNextAsync();
foreach (var item in response)
{
if (item.type == "customer")
{
customer = JsonConvert.DeserializeObject<CustomerV4>(item.ToString());
}
else if (item.type == "salesOrder")
{
orders.Add(JsonConvert.DeserializeObject<SalesOrder>(item.ToString()));
}
}
}
Update:
You do not have to use dynamic types if want to create a "base document" class and then derive from that. Deserialize into the documentBase class, then check the type property check which class to deserialize the payload into.
You can also extend this pattern when you evolve your data models over time with a docVersion property.
Consider the following code:
foreach (Type formType in allFormsToLoopThrough)
{
var nonPriorityForm = _context.Query(formType);
foreach (var nonpriority in nonPriorityForm)
{
var name = nonpriority.GetType().Name;
MyWorkAssignmentDTO form = new MyWorkAssignmentDTO
{
FormName = formType.Name
Id = nonpriority.Id
};
}
}
This snippet is looping thought a list of types.
Each type is taken from the list and passed to a Query function that returns an IQueryable - basically a list of records in a given table in a database that matches the type.
Then for each of the record sets that come back, I want to loop through those and from each create a new instance of MyWorkAssignmentDTO. I am only interested in a form name (which I can get from formType) but I cannot get nonpriority.Id
I know for sure that every nonpriority will have an Id once it is resolved in the loop.
What I can't do is implement this to work at run time.
Can anyone help?
I was able to use the dynamic keyword instead of var. While I lose compile time validation, this gets me over the line when I know for sure there will be an Id.
dynamic nonPriorityForm = _context.Query(formType);
Currently I am receiving an array of objects from a database.
object [] sqlResultData = DatabaseCall.Result();
This array of objects needs to be matched to class variables like this
CClassOfVars classVar = new CClassOfVars();
classVar.myProperty = sqlResultData[0];
classVar.myProperty1 = sqlResultData[1];
What i wish to do is pass the list of propertys on the class in order to a function and have the mapping from the object array occur automatically based on the order.
For example:
Method defined like this
FillData(object [] databaseValues, IList<object>())
Called like this
CClassOfVars classVar = new CClassOfVars();
object [] sqlResultData = DatabaseCall.Result();
FillData(sqlResultData, new List<object>(){classVar.myProperty,classVar.myProperty1});
The FillData function would hopefully type cast and set the values of myProperty and myProperty1 to the values in array locations of 0,1 etc...
Something like this
FillData(object [] databaseValues, IList<object> mapMe)
{
for (int i = 0; i < mapMe.Count; i++)
{
mapMe[i] = CastToTheCorrectType(mapMe[i], databaseValues[i]);
}
}
Cast to the correct type could look like this?? I took from here: cast object with a Type variable
public T CastToTheCorrectType<T>(T hackToInferNeededType, object givenObject) where T : class
{
return givenObject as T;
}
How can i pass a list of different object types to all have there values modified and assigned within a different function?
The matter you asking about is dark and difficult to be implemented through just a function. There are frameworks out there dealing with object relational mapping. If it is an option, install and learn some OR/M. If not ... well, there might be some dirty way.
You can use the JSON.NET library to do the heavy lifting for you. It's super easy to use and install through Nuget. My point is as follows.
Construct an anonymous object. Use the property names of the original object.
Fill it with the data from the object array. Spin a loop over the object array...
Serialize the anonymous object.
Deserialize the JSON string into the target type.
At this point, JSON.NET will handle property mapping for you.
List item
E.g. if your target type is Person you might do this:
var x = new
{
FirstName = String.Empty,
LastName = String.Empty
};
var persons = new List<Person>(sqlResultData.Length);
foreach (var record in sqlResultData)
{
x.FirstName = record[0];
x.LastName = record[1];
var s = JsonConvert.SerializeObject(x)`
var personX = JsonConvert.Deserialize<Person>(s);
persons.Add(person);
}
I'm using TweetInvi to grab a bunch of tweets that match a specified hashtag. I do this with the following:
var matchingTweets = Search.SearchTweets(hashtag);
This returns an IEnumerable (named ITweet, interface of Tweet), however I cannot create a List<> of Tweets, because Tweet is a static type.
I made, instead, a list of objects, using:
List<object> matches = matchingTweets.Cast<object>().ToList();
However, although each member of the matchingTweets IEnumerable has a number of properties, I cannot access them using:
long tweetID = matches[i].<property>;
Using matches[i].ToString() returns the tweet content, so how can I effectively cast the results in matchingTweets to a list, and subsequently access the properties of those list members? I would ideally like to avoid using dynamic.
In your example above you were trying to grab the ID from the tweet. ITweet implements ITweetIdentifier which contains the Id property. You can literally just access it by:
var matchingTweets = Search.SearchTweets(hashtag);
//Grab the first 5 tweets from the results.
var firstFiveTweets = matchingTweets.Take(5).ToList();
//if you only want the ids and not the entire object
var firstFiveTweetIds = matchingTweets.Take(5).Select(t => t.Id).ToList();
//Iterate through and do stuff
foreach (var tweet in matchingTweets)
{
//These are just examples of the properties accessible to you...
if(tweet.Favorited)
{
var text = tweet.FullText;
}
if(tweet.RetweetCount > 100)
{
//TODO: Handle popular tweets...
}
}
//Get item at specific index
matchingTweets.ElementAt(index);
I don't know exactly what you want to do with all the info, but since the SearchTweets returns a IEnumerable of ITweets you have access to anything an ITweet has defined.
I highly recommend looking through their wiki. It's pretty well organized and gives you clear examples of some basic tasks.
It makes sense you cannot access the properties. You cast it into object so you can only access the objects properties and methods (that like you said might have been overridden).
It should be fine to just access it like this:
List<ITweet> tweets = matchingTweets.Take(5).ToList();
What you can do is project it to a new object of yours:
var tweets = matchingTweets.Select(item => new {
property1 = item.property1,
property2 = item.property2
})
.Take(5).ToList();
Then you will be able to access what you need. Now, if you need to share this data outside the scope of that function create a DTO object and initialize it instead of the anonymous type.
Depending on the size of the project and amount of effort usually it is in any case a good practice to create a layer of DTO objects when you interact with an external service like this. Then if their models changed you can contain your changes only to the DTOs.
If all you want are the ids of the first 5 then:
var ids = matchingTweets.Take(5).Select(item => item.id).ToList();
I have a dynamic object (it's actually json) that I pass into my MVC WebApi controller.
The json object contains multiple lists within an anonymous object that are submitted to the controller from another application via client.PostAsJsonAsync("myapiurl", objectGraph).
What I need to do to validate the object on the MVC side, is to get the count of objects in each list. I can access the lists dynamically via mydynamicobject.mylist and individual items via mydynamicobject.mylist[index] but I can't seem to be able to get a count of mydynamicobject.mylist.
What I've tried so far:
LINQ extension methods - doesn't work on dynamic
Enumerable.Count(mydynamicobject.mylist) - can't infer type
Any other ideas? The count is actually correctly available in the dynamic object's base but obviously not accessible as a property. Help!
This works now:
// This is a MVC/WebApi method
public dynamic Post(dynamic mydynamicobject)
if (((ICollection)mydynamicobject.mylist).Count == 0)
{
// do something
}
The code that sends the dynamic object (different app):
HttpClient client = new HttpClient();
client.DefaultRequestHeaders.Accept.Add
(new MediaTypeWithQualityHeaderValue("application/json"));
var objectGraph = new { mylist = new { Id = 1 }, mylist2 = new { Name = "ABC" } };
var r = client.PostAsJsonAsync("api/mycontroller", objectGraph).Result;
If they are arrays, I believe you're looking for their Length property.
mydynamicobject.mylist.Length
Alternatively, I think you might be able to get away with casting mydynamicobject.mylist to an IEnumerable and then hand it to IEnueramble.Count like so:
IEnumerable.Count((IEnumerable)mydynamicobject.mylist);
you could also do as Paolo mentioned:
((ICollection)mydynamicobject.mylist).Count
Although I can't take credit for that one.