Develop Reference

Microsoft Graph API Delta Calendar query sometimes returns deleted event if it has been changed

I am integrating a planner application with Microsoft Graph using the C# nuget package 4.36.
Sometimes when I change an event from the Outlook.com online calendar, my Delta query will return the event as deleted, with End and Start properties == null:
AdditionalData[0] = {[#removed, ValueKind = Object : "{"reason":"deleted"}"]}
This only happens some times, I am often allowed to move an event backwards beyond "today". But if it is moved forward again, moving it up to, or beyond "today", the delta will return a deleted event. Often moving it forwards in time will return a deleted event, as long as it is dropped beyond "today". If I omit providing a deltaToken to the query, all events are returned in valid state even though they have been altered. But trying again to add my deltaToken, will return some deleted events, that are in fact never deleted.
What is happening and how do I avoid it?
My steps to reproduce the issue:
A test environment in Azure AD is needed and proper set up of App Only authentication (what I am using).
Having this, I create a calendar class with an active AppClient wrapping the Delta query like this:
public async Task<IEventDeltaCollectionPage> GetChangedEventsByUserId(string userId, int maxPageCount = 50,
string deltaToken = "")
{
var queryOptions = new List<QueryOption>();
queryOptions.Add(new QueryOption("startdatetime", "2010-08-01T00:00:00"));
queryOptions.Add(new QueryOption("enddatetime",
DateTime.Now.ToString("o")));
// Check for skipToken -> Start from where we left of last sync:
if (!string.IsNullOrWhiteSpace(deltaToken))
queryOptions.Add(new QueryOption("$deltatoken", deltaToken));
try
{
return await AppClient.Users[userId].Calendar.CalendarView
.Delta()
.Request(queryOptions)
.Header("Prefer", "outlook.body-content-type='text'")
.Header("Prefer", "outlook.timezone=\"W. Europe Standard Time\"")
.Header("Prefer", $"odata.maxpagesize={maxPageCount}")
.GetAsync();
}
catch (Exception e)
{
if (!string.IsNullOrWhiteSpace(deltaToken))
{
// If the request fails and has been given a deltaToken, try without the token:
return await GetChangedEventsByUserId(userId, maxPageCount);
}
}
return null;
}
I then created an "Explicit" nUnit-test that will create 3 events for me in a given users calendar. Breaking in the middle of this test, I change one event in outlook.com and observe that the delta query returns a deleted event Start and End == null. If however, I break at the first delta query where no deltaToken is given, I recieve all events as expected without any null values for Start/End.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.Graph;
using NUnit.Framework;
[TestFixture]
public class GraphClientTests
{
[TestCase, Explicit]
public void CheckDeltaQueryOnManualUpdates()
{
var timeout = new TimeSpan(0, 2, 0);
var idList = new List<string>();
var userId = "8078e9b9-088e-45ad-8c0b-ccf2e78b32d9";// Insert your own user ID
// Create new events:
var appointment = new Event
{
Subject = "Let's go for lunch",
Body = new ItemBody
{
ContentType = BodyType.Html,
Content = "Forget your worries!"
},
Start = new DateTimeTimeZone
{
DateTime = (DateTime.Now.Date + TimeSpan.FromHours(12) + TimeSpan.FromDays(1)).ToString("o"),
TimeZone = "W. Europe Standard Time"
},
End = new DateTimeTimeZone
{
DateTime = (DateTime.Now.Date + TimeSpan.FromHours(12+8) + TimeSpan.FromDays(1)).ToString("o"),
TimeZone = "W. Europe Standard Time"
},
Location = new Location
{
DisplayName = "Blue Diner"
},
};
for (int i = 0; i < 3; i++)
{
idList.Add(_calendar.AddEvent(userId, appointment).Result.Id);
}
/// BREAK or step to this place to test without DeltaToken - this works
var newEvents = _calendar.GetChangedEventsByUserId(userId, 7).Result;
calendarList.AddRange(newEvents.ToList());
while (newEvents.NextPageRequest != null)
{
newEvents = newEvents.NextPageRequest.GetAsync().Result;
calendarList.AddRange(newEvents.ToList());
}
foreach (var happening in calendarList)
{
Assert.NotNull(happening.Start);
Assert.NotNull(happening.End);
}
string deltaTokenName = "deltatoken=";
// Get delta link to be used on next call:
var deltaToken = newEvents.AdditionalData["#odata.deltaLink"].ToString();
deltaToken = deltaToken.Substring(deltaToken.IndexOf(deltaTokenName, StringComparison.Ordinal) +
deltaTokenName.Length);
Assert.IsFalse(string.IsNullOrWhiteSpace(deltaToken));
/// BREAK or step to this place to test with DeltaToken - This fails
newEvents = _calendar.GetChangedEventsByUserId(userId, 7, deltaToken).Result;
foreach (var happening in newEvents)
{
Assert.NotNull(happening.Start);
Assert.NotNull(happening.End);
}
deltaToken = newEvents.AdditionalData["#odata.deltaLink"].ToString();
deltaToken = deltaToken.Substring(deltaToken.IndexOf(deltaTokenName, StringComparison.Ordinal) +
deltaTokenName.Length);
/// BREAK here to restart from earlier break-points.
Assert.IsFalse(string.IsNullOrWhiteSpace(deltaToken));
// Cleanup after test is done:
foreach (var deletableId in idList)
{
Task.WaitAll(new Task[]{
_calendar.DeleteEvent(userId, deletableId)
}, timeout);
}
}
}
Update:
This question points in the direction of a hack to get around the issue: Since the "Deleted" event contains an Id, it is possible to look up the failing event directly via:
return await AppClient.Users[userId].Calendar.Events[eventId]
.Request()
.Header("Prefer", "outlook.body-content-type='text'")
.Header("Prefer", "outlook.timezone=\"W. Europe Standard Time\"")
.GetAsync();
Hence, by looping over all events from the delta query and asking directly for those with Start == null the desired set can be obtained. This however, still keeps me wondering why the Delta query does not give me edited events correctly. Any thoughts?

Get data from document DB using multithreading/parallel

I have the JSON documents in the document DB (~30k documents) where each document has a unique ID something like AA123, AA124. There is a tool we use to pull those documents from the document DB where it has a restriction of 500 documents per GET request call. So this has to go through 60 times GET requests to fetch the result which takes sometime. I am looking to get this optimized to run this in quick time(run threads parallely), so that I can get the data quickly. Below is the sample code on how I am pulling the data from the DB as of now.
private int maxItemsPerCall = 500;
public override async Task<IEnumerable<docClass>> Getdocuments()
{
string accessToken = "token";
SearchResponse<docClass> docs = await db.SearchDocuments<docClass>(initialload, accessToken); //Gets top 500
List<docClass> routeRules = new List<docClass>();
routeRules.AddRange(docs.Documents);
var remainingCalls = (docs.TotalDocuments / maxItemsPerCall);
while (remainingCalls > 0 && docs.TotalDocuments > maxItemsPerSearch)
{
docs = await db.SearchDocuments<docClass>(GetFollowUp(docs.Documents.LastOrDefault().Id.Id), requestOptions);
routeRules.AddRange(docs.Documents);
remainingCalls--;
}
return routeRules;
}
private static SearchRequest initialload = new SearchRequest()
{
Filter = new SearchFilterGroup(
new[]
{
new SearchFilter(Field.Type, FilterOperation.Equal, "documentRule")
},
GroupOperator.And),
OrderBy = Field.Id,
Top = maxItemsPerCall,
Descending = false
};
private static SearchRequest GetFollowUp(string lastId)
{
SearchRequest followUpRequest = new SearchRequest()
{
Filter = new SearchFilterGroup(
new[] {
new SearchFilter(Field.Type, FilterOperation.Equal, "documentRule"),
new SearchFilter(Field.Id, FilterOperation.GreaterThan, lastId)
},
GroupOperator.And),
OrderBy = Field.Id,
Top = maxItemsPerCall,
};
return followUpRequest;
}
Help needed: Since I am using the each GET request(500 documents based on IDs depending on the ID of the previous run), how can I use to run this parallely (atleast 5 parallel threads at a time) fetching 500 records per thread (i.e. 2500 parallely in total for 5 threads at a time). I am not familiar with threading, so it would be helpful if someone can suggest how to do this.

Idempotency for BigQuery load jobs using Google.Cloud.BigQuery.V2

You are able to create a csv load job to load data from a csv file in Google Cloud Storage by using the BigQueryClient in Google.Cloud.BigQuery.V2 which has a CreateLoadJob method.
How can you guarantee idempotency with this API to ensure that say the network dropped before getting a response and you kicked off a retry you would not end up with the same data being loaded into BigQuery multiple times?
Example API usage
private void LoadCsv(string sourceUri, string tableId, string timePartitionField)
{
var tableReference = new TableReference()
{
DatasetId = _dataSetId,
ProjectId = _projectId,
TableId = tableId
};
var options = new CreateLoadJobOptions
{
WriteDisposition = WriteDisposition.WriteAppend,
CreateDisposition = CreateDisposition.CreateNever,
SkipLeadingRows = 1,
SourceFormat = FileFormat.Csv,
TimePartitioning = new TimePartitioning
{
Type = _partitionByDayType,
Field = timePartitionField
}
};
BigQueryJob loadJob = _bigQueryClient.CreateLoadJob(sourceUri: sourceUri,
destination: tableReference,
schema: null,
options: options);
loadJob.PollUntilCompletedAsync().Wait();
if (loadJob.Status.Errors == null || !loadJob.Status.Errors.Any())
{
//Log success
return;
}
//Log error
}

You can achieve idempotency by generating your own jobid based on e.g. file location you loaded and target table.
job_id = 'my_load_job_{}'.format(hashlib.md5(sourceUri+_projectId+_datasetId+tableId).hexdigest())
var options = new CreateLoadJobOptions
{
WriteDisposition = WriteDisposition.WriteAppend,
CreateDisposition = CreateDisposition.CreateNever,
SkipLeadingRows = 1,
JobId = job_id, #add this
SourceFormat = FileFormat.Csv,
TimePartitioning = new TimePartitioning
{
Type = _partitionByDayType,
Field = timePartitionField
}
};
In this case if you try reinsert the same job_id you got error.
You can also easily generate this job_id for check in case if pooling failed.

There are two places you could end up losing the response:
When creating the job to start with
When polling for completion
The first one is relatively tricky to recover from without a job ID; you could list all the jobs in the project and try to find one that looks like the one you'd otherwise create.
However, the C# client library generates a job ID so that it can retry, or you can specify your own job ID via CreateLoadJobOptions.
The second failure time is much simpler: keep the returned BigQueryJob so you can retry the polling if that fails. (You could store the job name so that you can recover even if your process dies while waiting for it to complete, for example.)

Insufficient system resources exist to complete the requested service when using GeneticSharp

Long story short, I'm using GeneticSharp for an iterative/conditional reinforcement learning algorithm. This means that I'm making a bunch of different GeneticAlgorithm instances, each using a shared SmartThreadPool. Only one GA is running at a time though.
After a few iterations of my algorithm, I run into this error, which happens when attempting to start the SmartThreadPool.
Is there any obvious reason this should be happening? I've tried using a different STPE and disposing of it each time, but that didn't seem to help either. Is there some manual cleanup I need to be doing in between each GA run? Should I be using one shared GA instance?
Edit: Quick code sample
static readonly SmartThreadPoolTaskExecutor Executor = new SmartThreadPoolTaskExecutor() { MinThreads = 2, MaxThreads = 8 };
public static void Main(string[] args)
{
var achromosome = new AChromosome();
var bchromosome = new BChromosome();
while(true)
{
achromosome = FindBestAChromosome(bchromosome);
bchromosome = FindBestBChromosome(achromosome);
// Log results;
}
}
public static AChromosome FindBestAChromosome(BChromosome chromosome)
{
AChromosome result;
var selection = new EliteSelection();
var crossover = new UniformCrossover();
var mutation = new UniformMutation(true);
using (var fitness = new AChromosomeFitness(chromosome))
{
var population = new Population(50, 70, chromosome);
var ga = new GeneticAlgorithm(population, fitness, selection, crossover, mutation);
ga.Termination = new GenerationNumberTermination(100);
ga.GenerationRan += LogGeneration;
ga.TaskExecutor = Executor;
ga.Start();
LogResults();
result = ga.BestChromosome as AChromosome;
ga.GenerationRan -= LogGeneration;
}
return result;
}
public static BChromosome FindBestBChromosome(AChromosome chromosome)
{
BChromosome result;
var selection = new EliteSelection();
var crossover = new UniformCrossover();
var mutation = new UniformMutation(true);
using (var fitness = new BChromosomeFitness(chromosome))
{
var population = new Population(50, 70, chromosome);
var ga = new GeneticAlgorithm(population, fitness, selection, crossover, mutation);
ga.Termination = new GenerationNumberTermination(100);
ga.GenerationRan += LogGeneration;
ga.TaskExecutor = Executor;
ga.Start();
LogResults();
result = ga.BestChromosome as BChromosome;
ga.GenerationRan -= LogGeneration;
}
return result;
}
AChromosome and BChromosome are each fairly simple, a couple doubles and ints and maybe a function pointer (to a static function).
Edit2: Full call stack with replaced bottom two entries
Unhandled Exception: System.IO.IOException: Insufficient system resources exist to complete the requested service.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.Threading.EventWaitHandle..ctor(Boolean initialState, eventResetMode mode, string name)
at Amib.Threading.SmartThreadPool..ctor()
at GeneticSharp.Infrastructure.Threading.SmartThreadPoolTaskExecutor.Start()
at GeneticSharp.Domain.GeneticAlgorithm.EvaluateFitness()
at GeneticSharp.Domain.GeneticAlgorithm.EndCurrentGeneration()
at GeneticSharp.Domain.GeneticAlgorithm.EvolveOneGeneration()
at GeneticSharp.Domain.GeneticAlgorithm.Resume()
at GeneticSharp.Domain.GeneticAlgorithm.Start()
at MyProject.Program.FindBestAChromosome(BChromosome chromosome)
at MyProject.Program.Main(String[] args)
Edit3: One last thing to note is that my fitness functions are pretty processing-intensive and one run can take almost 2g of ram (running on a machine with 16g, so no worries there). I've seen no problems with garbage collection though.
So far, this only happens after about 5 iterations (which takes multiple hours).

It turns out it was my antivirus preventing the threads from finalizing. Now I'm running it on a machine with a different antivirus and it's running just fine. If I come up with a better answer for how to handle this in the future, I'll update here.

System.DirectoryServices.Protocol search question

I'm trying to re write a search from System.DirectoryServices to System.DirectoryServices.Protocol
In S.DS I get all the requested attributes back, but in S.DS.P, I don't get the GUID, or the HomePhone...
The rest of it works for one user.
Any Ideas?
public static List<AllAdStudentsCV> GetUsersDistinguishedName( string domain, string distinguishedName )
{
try
{
NetworkCredential credentials = new NetworkCredential( ConfigurationManager.AppSettings[ "AD_User" ], ConfigurationManager.AppSettings[ "AD_Pass" ] );
LdapDirectoryIdentifier directoryIdentifier = new LdapDirectoryIdentifier( domain+":389" );
using ( LdapConnection connection = new LdapConnection( directoryIdentifier, credentials ) )
{
SearchRequest searchRequest = new SearchRequest( );
searchRequest.DistinguishedName = distinguishedName;
searchRequest.Filter = "(&(objectCategory=person)(objectClass=user)(sn=Afcan))";//"(&(objectClass=user))";
searchRequest.Scope = SearchScope.Subtree;
searchRequest.Attributes.Add("name");
searchRequest.Attributes.Add("sAMAccountName");
searchRequest.Attributes.Add("uid");
searchRequest.Attributes.Add("telexNumber"); // studId
searchRequest.Attributes.Add("HomePhone"); //ctrId
searchRequest.SizeLimit = Int32.MaxValue;
searchRequest.TimeLimit = new TimeSpan(0, 0, 45, 0);// 45 min - EWB
SearchResponse searchResponse = connection.SendRequest(searchRequest) as SearchResponse;
if (searchResponse == null) return null;
List<AllAdStudentsCV> users = new List<AllAdStudentsCV>();
foreach (SearchResultEntry entry in searchResponse.Entries)
{
AllAdStudentsCV user = new AllAdStudentsCV();
user.Active = "Y";
user.CenterName = "";
user.StudId = GetstringAttributeValue(entry.Attributes, "telexNumber");
user.CtrId = GetstringAttributeValue(entry.Attributes, "HomePhone");
user.Guid = GetstringAttributeValue(entry.Attributes, "uid");
user.Username = GetstringAttributeValue(entry.Attributes, "sAMAccountName");
users.Add(user);
}
return users;
}
}
catch (Exception ex)
{
throw;
}
}
Also, if I want to fetch EVERY user in AD, so I can synch data with my SQL DB, how do I do that, I Kept getting max size exceeded, errors. I set the size to maxInt32... is there an "ignore size" option?
Thanks,
Eric-

I think that the standard way is to use System.DirectoryServices, not System.DirectoryServices.Protocol. Why do you want to user the later ?
Concerning your second question about the error message "max sized exceeded", it may be because you try to fetch too many entries at once.
Active Directory limits the number of objects returned by query, in order to not overload the directory (the limit is something like 1000 objects). The standard way to fetch all the users is using paging searchs.
The algorithm is like this:
You construct the query that will fetch all the users
You specify a specific control (Paged Result Control) in this query indicating that this is
a paged search, with 500 users per page
You launch the query, fetch the first page and parse the first 500 entries in
that page
You ask AD for the next page, parse the next 500 entries
Repeat until there are no pages left