Develop Reference

Filter multiple Event Types on Pulumi AzureNative Event Grid Topic Subscription

I create Event Grid Topic Subscriptions to Service Bus Queues:
var args = new EventSubscriptionArgs
{
EventSubscriptionName = topic.SubscriptionName,
Scope = topic.Id,
Destination = new Pulumi.AzureNative.EventGrid.Inputs.ServiceBusQueueEventSubscriptionDestinationArgs
{
EndpointType = "ServiceBusQueue",
ResourceId = queue.Id
},
};
var sub=new EventSubscription(
topic.SubscriptionName,
args
);
This works fine. Now I want to add a filter to only allow specific Event Types. According to the documentation I can add Filters like those:
args.Filter=new Pulumi.AzureNative.EventGrid.Inputs.EventSubscriptionFilterArgs
{
SubjectBeginsWith="the_topic",
SubjectEndsWith="the_topic"
}
But this is just one Topic, obviously. I need to be able to add an array of filters.
I expect that this can be done by EnableAdvancedFilteringOnArrays and AdvancedFilters somehow. But those properties aren't described anywhere (or I just wasn't able to find that information)
according to the documentation, AdvancedFilters is of type List<object> and I did not find any information what kind of object is expected there to be able to add multiple filters.

Haven’t used Pulumi, but I would suggest looking into the MSFT docs for Advanced filters (link) together with the docs on the operator you want to use, in your case stringBeginsWith.
As you can see the filtering should be applied in a key-value manner together with the operator to be used.

The Pulumi docs were half helpful. They showed the object you need to use but forget to mention the class is in the Pulumi.AzureNative.EventGrid.Inputs namespace. Below is a code snippet for creating the Pulumi.AzureNative.EventGrid.EventSubscription object. The Azure docs list the different type of filters you can use.
Filter = new AzureNative.EventGrid.Inputs.EventSubscriptionFilterArgs
{
IncludedEventTypes = new[]
{
"Microsoft.Devices.DeviceTelemetry"
},
AdvancedFilters = new[]
{
new Pulumi.AzureNative.EventGrid.Inputs.StringInAdvancedFilterArgs
{
OperatorType = Pulumi.AzureNative.EventGrid.AdvancedFilterOperatorType.StringIn.ToString(),
Key= "data.MyProperty",
Values= new[] { "MyValue" }
}
}
},

MongoDB: Combine Aggregation and Filter

Please see the following post for some background: MongoDB C# Driver - Return last modified rows only
After almost two years of running this code, we've been experiencing performance problems lately and as much as I keep on saying that the code is not the issue, Infrastructure are insisting it's because I'm doing full table scans.
The thing is that the problem is environment specific. Our QA environment runs like a dream all the time but Dev and Prod are very slow at times and fine at other - it's very erratic. They have the same data and code on but Dev and Prod have another app that is also running on the database.
My data has an Id as well as an _id (or AuditId) - I group the data by Id and then return the last _id for that record where it was not deleted. We have multiple historic records for the same ID and I would like to return the last one (see original post).
So I have the following method:
private static FilterDefinition<T> ForLastAuditIds<T>(IMongoCollection<T> collection) where T : Auditable, IMongoAuditable
{
var pipeline = new[] { new BsonDocument { { "$group", new BsonDocument { { "_id", "$Id" }, { "LastAuditId", new BsonDocument { { "$max", "$_id" } } } } } } };
var lastAuditIds = collection.Aggregate<Audit>(pipeline).ToListAsync().Result.ToList().Select(_ => _.LastAuditId);
var forLastAuditIds = Builders<T>.Filter.Where(_ => lastAuditIds.Contains(_.AuditId) && _.Status != "DELETE");
return forLastAuditIds;
}
This method is called by the one below, which accepts an Expression that it appends to the FilterDefinition created by ForLastAuditIds.
protected List<T> GetLatest<T>(IMongoCollection<T> collection,
Expression<Func<T, bool>> filter, ProjectionDefinition<T, T> projection = null,
bool disableRoleCheck = false) where T : Auditable, IMongoAuditable
{
var forLastAuditIds = ForLastAuditIds(collection);
var limitedList = (
projection != null
? collection.Find(forLastAuditIds & filter, new FindOptions()).Project(projection)
: collection.Find(forLastAuditIds & filter, new FindOptions())
).ToListAsync().Result.ToList();
return limitedList;
}
Now, all of this works really well and is re-used by all of my code that calls Collections, but this specific collection is a lot bigger than the others and we are getting slowdowns just on that one.
My question is: Is there a way for me to take the aggregate and Filter Builder and combine them to return a single FilterDefinition that I could use without running the full table scan first?
I really hope I am making sense.

Assuming I fully understand what you want, this should be as easy as this:
First, put a descending index on the LastAuditId field:
db.collection.createIndex{ "LastAuditId": -1 /* for sorting */ }
Or even extend the index to cover for other fields that you have in your filter:
db.collection.createIndex{ "Status": 1, "LastAuditId": -1 /* for sorting */ }
Make sure, however, that you understand how indexes can/cannot support certain queries. And always use explain() to see what's really going on.
The next step is to realize that you must always filter as much as possible as the very first step to reduce the amount of sorting required.
So, if you need to e.g. filter by Name then by all means do it as the very first step if your business requirements permit it. Be careful, however, that filtering at the start changes your semantics in the sense that you will get the last modified documents per each Id that passed the preceeding $match stage as opposed to the last documents per each Id that happen to also pass the following $match stage.
Anyway, most importantly, once you've got a sorted set, you can easily and quickly get the latest full document by using $group with $first which - with the right index in place - will not do a collection scan anymore (it'll be an index scan for now and hence way faster).
Finally, you want to run the equivalent of the following MongoDB query through C# leveraging the $$ROOT variable in order to avoid a second query (I can put the required code together for you once you post your Audit, Auditable and IMongoAuditable types as well as any potential serializers/conventions):
db.getCollection('collection').aggregate({
$match: {
/* some criteria that you currently get in the "Expression<Func<BsonDocument, bool>> filter" */
}
}, {
$sort: {
"ModifiedDate": -1 // this will use the index!
}
}, {
$group: {
"_id": "$Id",
"document": { $first: "$$ROOT" } // no need to do a separate subsequent query or a $max/$min across the entire group because we're sorted!
}
}, {
$match: { // some additional filtering depending on your needs
"document.Status": { $ne: "Delete" }
}
})
Lastly, kindly note that it might be a good idea to move to the latest version of MongoDB because they are currently putting a lot of effort into optimizing aggregation cases like yours, e.g. this one: https://jira.mongodb.org/browse/SERVER-9507

Accessing the properties of an IEnumerable

I'm using TweetInvi to grab a bunch of tweets that match a specified hashtag. I do this with the following:
var matchingTweets = Search.SearchTweets(hashtag);
This returns an IEnumerable (named ITweet, interface of Tweet), however I cannot create a List<> of Tweets, because Tweet is a static type.
I made, instead, a list of objects, using:
List<object> matches = matchingTweets.Cast<object>().ToList();
However, although each member of the matchingTweets IEnumerable has a number of properties, I cannot access them using:
long tweetID = matches[i].<property>;
Using matches[i].ToString() returns the tweet content, so how can I effectively cast the results in matchingTweets to a list, and subsequently access the properties of those list members? I would ideally like to avoid using dynamic.

In your example above you were trying to grab the ID from the tweet. ITweet implements ITweetIdentifier which contains the Id property. You can literally just access it by:
var matchingTweets = Search.SearchTweets(hashtag);
//Grab the first 5 tweets from the results.
var firstFiveTweets = matchingTweets.Take(5).ToList();
//if you only want the ids and not the entire object
var firstFiveTweetIds = matchingTweets.Take(5).Select(t => t.Id).ToList();
//Iterate through and do stuff
foreach (var tweet in matchingTweets)
{
//These are just examples of the properties accessible to you...
if(tweet.Favorited)
{
var text = tweet.FullText;
}
if(tweet.RetweetCount > 100)
{
//TODO: Handle popular tweets...
}
}
//Get item at specific index
matchingTweets.ElementAt(index);
I don't know exactly what you want to do with all the info, but since the SearchTweets returns a IEnumerable of ITweets you have access to anything an ITweet has defined.
I highly recommend looking through their wiki. It's pretty well organized and gives you clear examples of some basic tasks.

It makes sense you cannot access the properties. You cast it into object so you can only access the objects properties and methods (that like you said might have been overridden).
It should be fine to just access it like this:
List<ITweet> tweets = matchingTweets.Take(5).ToList();
What you can do is project it to a new object of yours:
var tweets = matchingTweets.Select(item => new {
property1 = item.property1,
property2 = item.property2
})
.Take(5).ToList();
Then you will be able to access what you need. Now, if you need to share this data outside the scope of that function create a DTO object and initialize it instead of the anonymous type.
Depending on the size of the project and amount of effort usually it is in any case a good practice to create a layer of DTO objects when you interact with an external service like this. Then if their models changed you can contain your changes only to the DTOs.
If all you want are the ids of the first 5 then:
var ids = matchingTweets.Take(5).Select(item => item.id).ToList();

is filter is replaceable of query in mongo queries using c#?

I'm new to mongo in c#
I found tow ways to find documents based on search critiria:
the first using filter:
var collection = _database.GetCollection<BsonDocument>("restaurants");
var filter = Builders<BsonDocument>.Filter.Eq("address.zipcode", "10075");
var result = await collection.Find(filter).ToListAsync();
The second using query:
MongoCollection<BsonDocument> books;
var query = Query.EQ("author", "Kurt Vonnegut");
foreach (BsonDocument book in books.Find(query)) {
// do something with book
}
What's the best way to find documents based to what MongoDB recommendation?

As far I know query builders (like your second example using Query.EQ) belong to old versions of C# drivers (1.X) (see Query class). Also I suggest you to see Builder section in this link that confirm query builders is the old way to consult data.
After the release of the 2.0 version of the .NET driver, a lot of changes were made, including the way of consult the data (you can read more about that in this link). If you are using the last C# driver version you should use your first approach.

The first way you mentioned is quite fine. You might want to incorporate using a mongo cursor as well to allow you to iterate through results
var collection = _database.GetCollection<BsonDocument>("restaurants");
var filter = Builders<BsonDocument>.Filter.Eq("address.zipcode", "10075");
using(var _cursor = await collection.Find(filter).ToCursorAsync())
{
while(await _cursor.MoveNextAsync())
{
foreach(var _document in _cursor.Current) //gets current document
{
//you can do things like get the _id of the document
var _id = _document["_id"];
}
}
}

Retrieve only a particular number of child records

If I write
db.Topics.Include("ChildTopics")
then it gives all the child comments for that. But what I want is only the top two child topics ordered by "childtopic created date".

This is not possible to include only the first two via Include method.
In your case, you can do this :
var firstTwo = from topic in datacontext.Topics
select new { Topic= topic, ChildTopic= topic.ChildTopics.OrderBy(c => c.childtopic_created_date).Take(2) };

Include does an early load of all dependencies, but I am not aware of a way to apply restrictions on this implementation.
An alternate way would be to shape the results like:
var comments = db.Topics.Select(_x=>
new{
Topic = _x,
TopTwoChildTopics = _x.ChildTopics.Top(2)
});