I'm trying to implement paging using the SDK v3 CosmosClient instead of the old DocumentClient.
Reason for this is it seems DocumentClient doesn't translate LINQ queries that contains spatial functions very well (ie: When using Within() I'll get an error from DocumentClient stating the methods is not implemented).
Paging works well with DocumentClient.CreateDocumentQuery<T> as such:
var query = DocumentClient.CreateDocumentQuery<T>(UriFactory.CreateDocumentCollectionUri("master", "features"), feedOptions)
.Where(t => t.Type == typeof(T).Name)
.Where(pred)
.AsDocumentQuery();
string queryContinuationToken = null;
var page = await query.ExecuteNextAsync<T>();
if (query.HasMoreResults)
queryContinuationToken = page.ResponseContinuation;
I'm at a little loss as to where to gather a continuation token using CosmosClient and its Container class:
QueryRequestOptions options = new QueryRequestOptions();
options.MaxItemCount = maxRecords;
FeedIterator<T> feed;
if (continuationToken == "")
feed = Container.GetItemLinqQueryable<T>(true, null, options).Where(x => x.Type == typeof(T).Name).Where(pred).ToFeedIterator();
else
feed = Container.GetItemLinqQueryable<T>(true, continuationToken, options).Where(x => x.Type == typeof(T).Name).Where(pred).ToFeedIterator();
FeedIterator seems to have some of the members IDocumentQuery has (like HasMoreResults) but I can't find a continuation token anywhere.
What am I missing?
Alright, here's a Where method I implemented. Seems to work at first glance.
If you do var f = feed.ReadNextAsync() you won't get an object that's of type FeedResponse, preventing you access to the token. You need to declare f explicitly of type FeedResponse<T>
public async Task<(IEnumerable<T> Results, string ContinuationToken)> Where<T>(Expression<Func<T, bool>> pred, int maxRecords = 0, string partitionKey = "", string continuationToken = "") where T : IDocumentModel
{
QueryRequestOptions options = new QueryRequestOptions();
if (partitionKey != "")
options.PartitionKey = new PartitionKey(partitionKey);
if (maxRecords == 0)
{
return (Container.GetItemLinqQueryable<T>(true, null, options).Where(x => x.Type == typeof(T).Name).Where(pred), "");
}
else
{
options.MaxItemCount = maxRecords;
string token = "";
FeedIterator<T> feed;
List<T> res = new List<T>();
if (continuationToken == "")
feed = Container.GetItemLinqQueryable<T>(true, null, options).Where(x => x.Type == typeof(T).Name).Where(pred).ToFeedIterator();
else
feed = Container.GetItemLinqQueryable<T>(true, continuationToken, options).Where(x => x.Type == typeof(T).Name).Where(pred).ToFeedIterator();
Microsoft.Azure.Cosmos.FeedResponse<T> f = await feed.ReadNextAsync();
token = f.ContinuationToken;
foreach (var item in f)
{
res.Add(item);
}
return (res, token);
}
}
With the version 3.12.0 of the Cosmos SDK the following works as expected as a pretty much in place replacement for the older DocumentQuery.
Original DocumentClient method for comparison:
IDocumentQuery<ToDoItem> query = client.CreateDocumentQuery<ToDoItem>(collectionUri)
.Where(t => t.Description.Contains(searchterm))
.AsDocumentQuery();
while (query.HasMoreResults)
{
foreach (ToDoItem result in await query.ExecuteNextAsync())
{
log.LogInformation(result.Description);
}
}
Using a CosmosClient this becomes:
var database = client.GetDatabase("ToDoItems");
var container = database.GetContainer("Items");
var query = container.GetItemLinqQueryable<ToDoItem>()
.Where(t => t.Description.Contains(searchTerm))
.ToFeedIterator();
while (query.HasMoreResults)
{
foreach (ToDoItem result in await query.ReadNextAsync())
{
log.LogInformation(result.Description);
}
}
So your query is now a FeedIterator, and you can call HasMoreResults and ReadNextAsync on it.
Admittedly this won't get you access to the diagnostics, request charge, etc. that comes on the FeedIterator, but it will page through the results cleanly.
IQueryable<returnVModel> query;
var requestOptions = new QueryRequestOptions
{
MaxItemCount = 20
};
if (Token == "" || Token == null)
{
query = Container.GetItemLinqQueryable<returnVModel>(false, null, requestOptions).Where(x => x.id == id);
}
else
{
query = Container.GetItemLinqQueryable<returnVModel>(false, Token, requestOptions).Where(x => x.id == id);
}
var ct = new CancellationTokenSource();
var totalCount = await query.CountAsync(ct.Token); //Total Count
var feedIterator = query.ToFeedIterator();
var queryResults = new List<returnVModel>();
FeedResponse<returnVModel> feedResults = await feedIterator.ReadNextAsync(ct.Token);
queryResults.AddRange(feedResults); // Output
var PaginationToken = feedResults.ContinuationToken //Token
First time we need to pass token as null, from next page onwards pass the token which we received in previous output.
Pagination was working fine in v3.
Related
A bit of background, I am looking to replace existing code in a C# App from the existing Microsoft.Azure.Management.Fluent (now deprecated) to the newer Azure.ResourceManager components.
Existing code to copy a database:
public async Task<bool> CopyDb(string? server, string? fromName, string? toName)
{
_log.LogInformation("Connecting to Azure");
var azure = GetAzureObject();
var servers = await azure.SqlServers.ListAsync();
var fromServer = servers.FirstOrDefault(f => server != null && server.Contains(f.Name));
if (fromServer == null)
{
throw new InvalidOperationException("Unable to find original database server");
}
var toNameBackup = $"{toName}-Old";
var existingDbs = await fromServer.Databases.ListAsync();
var fromDB = existingDbs.FirstOrDefault(f => f.Name.Equals(fromName));
if (fromDB == null)
{
throw new InvalidOperationException("Unable to find original database");
}
if (existingDbs.Any(a => a.Name.Equals(toNameBackup, StringComparison.OrdinalIgnoreCase))
&& existingDbs.Any(a => a.Name.Equals(toName, StringComparison.OrdinalIgnoreCase)))
{
_log.LogInformation("Deleting any existing backup called {0}", toNameBackup);
await fromServer.Databases.DeleteAsync(toNameBackup);
}
if (existingDbs.Any(a => a.Name.Equals(toName, StringComparison.OrdinalIgnoreCase)))
{
_log.LogInformation("Renaming target database from {0} to {1} (if exists)", toName, toNameBackup);
await (await fromServer.Databases.GetAsync(toName)).RenameAsync(toNameBackup);
}
_log.LogInformation("Copying database from from {0} to {1}", fromName, toName);
var result = await fromServer.Databases.
Define(toName).
WithSourceDatabase(fromDB).
WithMode(Microsoft.Azure.Management.Sql.Fluent.Models.CreateMode.Copy).CreateAsync();
return result != null;
}
private Microsoft.Azure.Management.Fluent.IAzure GetAzureObject()
{
var clientId = _configuration["AzureClientId"];
var clientSecret = _configuration["AzureClientSecret"];
var tenantId = _configuration["AzureTenantId"];
var subscriptionId = _configuration["AzureSubscriptionId"];
var credentials = Microsoft.Azure.Management.ResourceManager.Fluent.SdkContext.AzureCredentialsFactory.FromServicePrincipal(
clientId: clientId,
clientSecret: clientSecret,
tenantId: tenantId,
environment: Microsoft.Azure.Management.ResourceManager.Fluent.AzureEnvironment.AzureGlobalCloud);
return Microsoft.Azure.Management.Fluent.Azure.Configure().Authenticate(credentials).WithSubscription(subscriptionId);
}
The newer components all work with resources and I've been struggling how to do a couple operations with the newer Azure.ArmClient. I've been able to query with it finding my SQL server and databases. I can even delete some DBs, but I'm unable to work out how to rename or copy databases like the above code. I know there are alternative ways to do this directly in SQL, but I'd prefer to see how to do it in code.
I have had a look around MS docs, I can only find information on the object definitions but no examples.
I have managed to get down to the point of renaming:-
var backupDb = fromServer.GetSqlDatabase(toName);
if (backupDb != null && backupDb.Value != null)
{
// What do I pass in to the definition?
var moveDefinition = new SqlResourceMoveDefinition()
{
// What to set here?
};
await (await backupDb.Value.GetAsync()).Value.RenameAsync(moveDefinition);
}
I'm not sure on how to define the SqlResourceMoveDefinition. I also can't work out at all how to perform the copy like in the older SDK.
Anyone have any guides on how to achieve these operations in C#?
Managed to work it out after eventually working from https://learn.microsoft.com/en-us/dotnet/azure/sdk/resource-management?tabs=PowerShell. There may be better ways to do this, and I'll edit the answer when I find them if others don't by then!
public async Task<bool> CopyDb(string? server, string? fromName, string? toName)
{
_log.LogInformation("Connecting to Azure");
var azure = GetAzureSubscription();
var servers = azure.GetSqlServers().ToList();
var fromServer = servers.SingleOrDefault(f => server != null && f.Data != null && server.Contains(f.Data.Name));
if (fromServer == null)
{
throw new InvalidOperationException("Unable to find original database server");
}
var oldName = $"{toName}-Old";
var databases = fromServer.GetSqlDatabases();
_log.LogInformation("Check for any existing backup called {0}", oldName);
if (await databases.ExistsAsync(oldName))
{
_log.LogInformation("Deleting for any existing backup called {0}", oldName);
var oldBackup = await databases.GetAsync(oldName);
await oldBackup.Value.DeleteAsync(WaitUntil.Completed);
}
_log.LogInformation("Check target database {0} exists", toName, oldName);
if (await databases.ExistsAsync(toName))
{
_log.LogInformation("Renaming target database from {0} to {1}", toName, oldName);
var toDbBackup = await databases.GetAsync(toName);
var resourceIdString = toDbBackup.Value.Data.Id.Parent?.ToString();
var newResourceId = new ResourceIdentifier($"{resourceIdString}/databases/{oldName}");
var moveDefinition = new SqlResourceMoveDefinition(newResourceId);
var toDb = await toDbBackup.Value.GetAsync();
await toDb.Value.RenameAsync(moveDefinition);
}
_log.LogInformation("Copying database from from {0} to {1}", fromName, toName);
var fromDb = await databases.GetAsync(fromName);
var result = await databases.CreateOrUpdateAsync(WaitUntil.Completed, toName, fromDb.Value.Data);
_log.LogInformation("Operation completed!");
return result.HasValue;
}
private SubscriptionResource GetAzureSubscription()
{
var configValue = _configuration["AzureSubscriptionId"];
var subscriptionId = new ResourceIdentifier($"/subscriptions/{configValue}");
return GetAzureArmClient().GetSubscriptionResource(subscriptionId);
}
private ArmClient GetAzureArmClient()
{
var clientId = _configuration["AzureClientId"];
var clientSecret = _configuration["AzureClientSecret"];
var tenantId = _configuration["AzureTenantId"];
var credentials = new ClientSecretCredential(
clientId: clientId,
clientSecret: clientSecret,
tenantId: tenantId);
return new ArmClient(credentials);
}
I have following service method
public async Task<IResultList<IEnrichedContentEntity>> QueryAsync(Context context, string schemaIdOrName, Q q)
{
Guard.NotNull(context, nameof(context));
if (q == null)
{
return EmptyContents;
}
var schema = await GetSchemaOrThrowAsync(context, schemaIdOrName);
var permissionReadOwn = Permissions.ForApp(Permissions.AppContentReadOwn, context.App.Name, schemaIdOrName);
if (context.Permissions.Allows(permissionReadOwn))
{
q.CreatedBy = context.User.Token();
}
using (Profiler.TraceMethod<ContentQueryService>())
{
q = await queryParser.ParseAsync(context, q, schema);
var contents = await contentRepository.QueryAsync(context.App, schema, q, context.Scope());
if (q.Ids != null && q.Ids.Count > 0)
{
contents = contents.SortSet(x => x.Id, q.Ids);
}
return await TransformAsync(context, contents);
}
}
q.CreatedBy must set value if permission is correct.
How can I test if q.CreatedBy is not empty and has correct value.
I implemented following test, but no idea how check this params?
public async Task QueryAll_should_return_own_user_contents(int isFrontend, int unpublished, SearchScope scope)
{
var ctx = CreateContextWithOwnReadPermission(isFrontend: isFrontend == 1, allowSchema: true)
.WithUnpublished(unpublished == 1);
var content = CreateContent(contentId);
var q = Q.Empty.WithReference(DomainId.NewGuid());
A.CallTo(() => contentRepository.QueryAsync(ctx.App, schema, q, scope))
.Returns(ResultList.CreateFrom(5, content));
//A.CallTo(() => contentRepository.QueryAsync(ctx.App, schema, A<Q>.That.Matches(x => x.CreatedBy == null), scope))
// .MustHaveHappened();
//A.CallTo(() => contentRepository.QueryAsync(ctx.App, schema, A<Q>.That.Matches(x => x.CreatedBy == ctx.User.Token()), scope))
// .MustHaveHappened();
//q.CreatedBy = ctx.User.Token();
var result = await sut.QueryAsync(ctx, schemaId.Name, q);
Assert.Equal(contentData, result[0].Data);
Assert.Equal(contentId, result[0].Id);
Assert.Equal(5, result.Total);
}
I am getting Azure AD users into a list to be used in a dropdown, but it takes about at least 8/9 seconds to do the call... I know this can probably be reduced... So I will place my code here, hoping that someone can give me a better idea of how to change the code to a better one.
public async Task<List<Microsoft.Graph.User>> getAzureUsers()
{
IConfidentialClientApplication confidentialClientApplication = ConfidentialClientApplicationBuilder
.Create("***")
.WithTenantId("***")
.WithClientSecret("***")
.Build();
ClientCredentialProvider authProvider = new ClientCredentialProvider(confidentialClientApplication);
GraphServiceClient graphClient = new GraphServiceClient(authProvider);
List<Microsoft.Graph.User> utilizadores = new List<Microsoft.Graph.User>();
var user = await graphClient.Users
.Request()
.Select(p => new {
p.DisplayName,
p.UserPrincipalName,
p.Id
})
.GetAsync();
utilizadores.AddRange(user.CurrentPage);
while (user.NextPageRequest != null)
{
user = await user.NextPageRequest.GetAsync();
utilizadores.AddRange(user.CurrentPage);
}
return utilizadores;
}
public async Task<ViewModelColaboradores> GetVM()
{
bool colExist = false;
List<Microsoft.Graph.User> utilizadores = new List<Microsoft.Graph.User>();
List<UserAD> nomes = new List<UserAD>();
utilizadores = await getAzureUsers();
ViewModelColaboradores vm = new ViewModelColaboradores();
foreach (var n in utilizadores)
{
foreach (var col in _context.RH_Colaboradores)
{
if (n.DisplayName == col.Nome)
{
colExist = true;
}
}
if (!colExist)
{
nomes.Add(new UserAD { DisplayName = n.DisplayName, Id = n.Id, Email = n.UserPrincipalName });
}
colExist = false;
}
vm.usersAd = nomes;
return vm;
}
and then I just call GETVM to get the viewmodel with the users inside. Any ideas?
You can use pagination to improve the performance.
You can have a search-based dropdown(non-paginated) or scroll based dropdown (paginated - that fetches new items on scrolling down and fetches scroll up results from cache).
C# sample code:
public async Task<AzureUserDto> GetUsersPage(int pageSize, string skipToken)
{
var filterString = $"startswith(givenName, '{firstName}')";
var queryOptions = new List<QueryOption>
{
new QueryOption("$top", pageSize)
};
if (!string.IsNullOrEmpty(skipToken))
{
queryOptions.Add(new QueryOption("$skiptoken", skipToken));
}
var azureUsers = await GraphServiceClient.Users
.Request(queryOptions)
.Filter(filterString)
.Select(x => new
{
x.Id,
x.DisplayName,
x.GivenName,
x.Surname,
x.UserPrincipalName,
x.AccountEnabled,
x.Identities,
x.BusinessPhones,
x.JobTitle,
x.MobilePhone,
x.OfficeLocation,
x.PreferredLanguage,
x.Mail
})
.GetAsync();
// Get SkipToken, if exists
var skipToken = azureUsers
.NextPageRequest?
.QueryOptions?
.FirstOrDefault(
x => string.Equals("$skiptoken", x.Name, StringComparison.InvariantCultureIgnoreCase))?
.Value;
var azureUserDto = new AzureUserDto
{
// Map the azureUsers to AzureUsersDto or something similar and return
// don't forget to include the SkipToken
};
return azureUsers;
}
For more details on how to implement the latter method, you can visit Paging Microsoft Graph data in your app and Microsoft Graph API: C# / Filtering / Pagination.
I have this block of code. My data is around 20000 tokens. And I am sending them 1000 and 1000. I am haing performance issues here. What can I do here to make it perform better? Should I try cutting back on the use of .ToList()
var badgewiseGroup = tokensWithoutOnbehlfOf.GroupBy(x => x.BadgeCount);
foreach (var bGroup in badgewiseGroup)
{
var badgeTokensWithoutOnbehlfOf = bGroup.Select(x => x).ToList();
while (badgeTokensWithoutOnbehlfOf.Any())
{
var tokens = badgeTokensWithoutOnbehlfOf.Take(numberToTake).ToList();
Log.Log(EventSeverity.Informational, $"Processing Push notification for MessageId:{pushContent.MessageId}, Username:{tokens.FirstOrDefault().Username}," +
$" OnBehalfOf:{tokens.FirstOrDefault().OnBehalfOf}, Id:{tokens.FirstOrDefault().Id}, DeviceId:{tokens.FirstOrDefault().DeviceId}, " +
$"Token:{tokens.FirstOrDefault().Token}, ");
pushContent.Title = msgTitle;
pushContent.Body = msgBody;
var message = new Messaging.Message();
var androidToken = tokens.Where(x => x.OsVersion != null && x.OsVersion.ToLower().Contains("android")).ToList();
if (androidToken != null && androidToken.Any())
{
message = GetAndroidMessageNotifictationObject(androidToken, pushContent);
SendMessage(message, pushContent.MessageId);
}
var iosToken = tokens.Where(x => x.OsVersion != null && x.OsVersion.ToLower().Contains("ios")).ToList();
if (iosToken != null && iosToken.Any())
{
message = GetIOSMessageNotifictationObject(iosToken, pushContent);
SendMessage(message, pushContent.MessageId);
}
var oldTokens = tokens.Where(x => string.IsNullOrEmpty(x.OsVersion)).ToList();
if (oldTokens != null && oldTokens.Any())
{
message = GetOldAppUsersMessageNotifictationObject(oldTokens, pushContent);
SendMessage(message, pushContent.MessageId);
}
badgeTokensWithoutOnbehlfOf = badgeTokensWithoutOnbehlfOf.Skip(numberToTake).ToList();
}
}
If execute order is not important and u use .Net Core u can try use action block for this.
var blockOptions = new ExecutionDataflowBlockOptions()
{
MaxDegreeOfParallelism = 48,
SingleProducerConstrained = true,
EnsureOrdered = true
};
ActionBlock<TypeInList> block = null;
block = new ActionBlock<TypeInList>(bGroup =>
{
//your bGroup code
}, blockOptions);
tokensWithoutOnbehlfOf.ForEach(bGroup => block.Post(bGroup));
More info about ActionBlock u can find here: https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.dataflow.actionblock-1?view=netcore-1.0
Just trying to do normal query on ef but getting below error message
"A second operation started on this context before a previous asynchronous operation completed"
I am using await for all aync call but not sure what could be causing this issue.
here' code
var sids = await context.Sites.Where(s => slist.Contains(s.JVSiteID)).ToListAsync();
var header = await context.GenericCsvHeaders.FirstOrDefaultAsync(x => x.Header == csvh) ?? new GenericCsvHeader { Header = csvh };
context.Entry<GenericCsvHeader>(header).State = (header.GenericCsvHeaderID == 0) ? EntityState.Added : EntityState.Unchanged;
var sa = await context.SAs.Where(x => x.StatusID == Data.Enum.Status.Active && mapkeys.Contains(x.SAName)).Select(x => new { x.SAName, x.SACode, x.SAID }).ToListAsync();
if (sa.Count > 0)
sasitelist = await context.Site_SA.Where(x => x.StatusID == Data.Enum.Status.Active && siteids.Contains(x.SiteID ?? 0)).ToListAsync();
var az = await context.Azimuths.Where(x => x.StatusID == Data.Enum.Status.Active && mapkeys.Contains(x.AzimuthName)).Select(x => new { x.AzimuthName, x.AzimuthID }).ToListAsync();
if (az.Count > 0)
azsitelist = await context.Site_Azimuth.Where(x => x.StatusID == Data.Enum.Status.Active && siteids.Contains(x.SiteID ?? 0)).ToListAsync();
var rows = new List<dynamic>(); //getting this list from csv file via csvHelper
foreach (var r in rows)
{
var s = sids.FirstOrDefault(x => x.JVSiteID == (((IDictionary<String, Object>)r)[siteid]).ToString()) ?? new Site();
UpdateSite(s, r, map);
}
private async void UpdateSite(Site site, dynamic csvSite, IDictionary<string, string> map)
{
context.Entry(site).State = (site.StateID == 0) ? EntityState.Added : EntityState.Modified;
site.SiteForAuditTrail = site;
if (map.ContainsKey("SiteName") && !String.IsNullOrWhiteSpace(GetStringOfDynamic(map,csvSite,"SiteName")))
site.SiteName = GetStringOfDynamic(map,csvSite, "SiteName");
if (map.ContainsKey("State") && !String.IsNullOrWhiteSpace(GetStringOfDynamic(map,csvSite, "State")))
{
//getting exception at below line
var state = (await GetRefTypeList<State>(x => x.StateCode == GetStringOfDynamic(map,csvSite, "State"))) ?? new State { StateCode = GetStringOfDynamic(map,csvSite, "State") };
context.Entry(state).State = (state.StateID == 0) ? EntityState.Added : EntityState.Unchanged;
site.State = state;
state.SiteForAuditTrail = site;
}
}
private async Task<T> GetRefTypeList<T>(Func<T, bool> expression) where T : EntityBase
{
if (!refTypes.ContainsKey(typeof(T).Name))
refTypes[typeof(T).Name] = (await context.Set<T>().Where(x => x.StatusID == Data.Enum.Status.Active).ToListAsync());
return (refTypes[typeof(T).Name] as List<T>)?.FirstOrDefault(expression);
}
private string GetStringOfDynamic(IDictionary<string, string> map,dynamic obj, string propertyName)
{
if (((IDictionary<String, Object>)obj).ContainsKey(map[propertyName]))
return (((IDictionary<String, Object>)obj)[map[propertyName]]??"").ToString();
return "";
}
found problem, was missing await when calling UpdateSite
foreach (var r in rows)
{
var s = sids.FirstOrDefault(x => x.JVSiteID == (((IDictionary<String, Object>)r)[siteid]).ToString()) ?? new Site();
**await** UpdateSite(s, r, map);
}
You are having a race condition here. While you are using await you have other code executing potentially before the variable sids, header, sa and az, have been returned.
You should refactor your code so that the remaining code is not executed until these variables are returned. You can do this by using a task and then using the extension WhenAll which will ensure each await has been completed before proceeding. Here is a link to MS docs on Task and WhenAll implementation