I have an Azure WebKob with blob and queue triggers to save data to Azure DocumentDb.
From time to time I'm getting an error:
Microsoft.Azure.Documents.RequestRateTooLargeException: Message: {"Errors":["Request rate is large"]}
Currently I throttle requests using this code. A WebJob function:
public async Task ParseCategoriesFromCsv(...)
{
double find = 2.23, add = 5.9, replace = 10.67;
double requestCharge = Math.Round(find + Math.Max(add, replace));
await categoryProvider.SaveCategories(requestCharge , categories);
}
Category provider to manipulate document db client:
public async Task<ResourceResponse<Document>[]> SaveCategories(double requestCharge, Category[] categories)
{
var requestDelay = TimeSpan.FromSeconds(60.0 / (collectionOptions.RequestUnits / requestCharge));
var scheduler = new IntervalTaskScheduler(requestDelay, Scheduler.Default); // Rx
var client = new DocumentClient(endpoint, authorizationKey,
new ConnectionPolicy
{
ConnectionMode = documentDbOptions.ConnectionMode,
ConnectionProtocol = documentDbOptions.ConnectionProtocol
});
return await Task.WhenAll(documents.Select(async d =>
await scheduler.ScheduleTask(
() => client.PutDocumentToDb(collectionOptions.CollectionLink, d.SearchIndex, d))));
}
Task scheduler to throttle/measure/synchronize requests:
private readonly Subject<Action> _requests = new Subject<Action>();
private readonly IDisposable _observable;
public IntervalTaskScheduler(TimeSpan requestDelay, IScheduler scheduler)
{
_observable = _requests.Select(i => Observable.Empty<Action>()
.Delay(requestDelay)
.StartWith(i))
.Concat()
.ObserveOn(scheduler)
.Subscribe(action => action());
}
public Task<T> ScheduleTask<T>(Func<Task<T>> request)
{
var tcs = new TaskCompletionSource<T>();
_requests.OnNext(async () =>
{
try
{
T result = await request();
tcs.SetResult(result);
}
catch (Exception ex)
{
tcs.SetException(ex);
}
});
return tcs.Task;
}
So it's basically a number of constants from ResourceResponse<Document>.RequestCharge but:
When I have 1 queue triggered it works fine but when 8 queue it throws an error.
If increase request charge in 8 times then 8 queues work fine but just 1 works 8 times slower than it could.
What a throttling/measuring/synchronization mechanism could work here well?
Starting .NET SDK 1.8.0, we automatically handle the Request Rate too large exceptions to a reasonable extent(will retry 9 times by default and honor the retry after returned from server for next retry).
In case you need greater control, you can configure the RetryOptions on the ConnectionPolicy instance that you pass in to the DocumentClient objectand we would override the default retry policy with it.
So you no longer need to add any custom logic for handling 429 exceptions in your application code like above.
When getting a 429 (Request rate too large) the response tells you how long to wait. There is a header x-ms-retry-after. This has a value. Wait for that time period in ms.
catch (AggregateException ex) when (ex.InnerException is DocumentClientException)
{
DocumentClientException dce = (DocumentClientException)ex.InnerException;
switch ((int)dce.StatusCode)
{
case 429:
Thread.Sleep(dce.RetryAfter);
break;
default:
Console.WriteLine(" Failed: {0}", ex.InnerException.Message);
throw;
}
}
It seems to me that you should be able to do this with your SaveCategories method to make it work nicely with Rx:
public IObservable<ResourceResponse<Document>[]> SaveCategories(double requestCharge, Category[] categories)
{
var requestDelay = TimeSpan.FromSeconds(60.0 / (collectionOptions.RequestUnits / requestCharge));
var client = new DocumentClient(endpoint, authorizationKey,
new ConnectionPolicy
{
ConnectionMode = documentDbOptions.ConnectionMode,
ConnectionProtocol = documentDbOptions.ConnectionProtocol
});
return
Observable.Interval(requestDelay)
.Zip(documents, (delay, doc) => doc)
.SelectMany(doc => Observable.FromAsync(() => client.PutDocumentToDb(collectionOptions.CollectionLink, doc.SearchIndex, doc)))
.ToArray();
}
This totally gets rid of your IntervalTaskScheduler class and ensures that you limit the request rate to one request per the requestDelay time span, but allows the response to take as long as needed. To .ToArray() call turns the IObservable<ResourceResponse<Document>> that returns many values into an IObservable<ResourceResponse<Document>[]> that returns a single array of values when the observable completes.
I couldn't test your code, so I tested a sample which I think simulates your code:
var r = new Random();
var a = Enumerable.Range(0, 1000);
var i = Observable.Interval(TimeSpan.FromSeconds(2.0));
var sw = Stopwatch.StartNew();
var query =
i.Zip(a, (ii, aa) => aa)
.SelectMany(aa => Observable.Start(() =>
{
var x = sw.Elapsed.TotalMilliseconds;
Thread.Sleep(r.Next(0, 5000));
return x;
}))
.Select(x => new
{
started = x,
ended = sw.Elapsed.TotalMilliseconds
});
I got this kind of result which shows that the requests were throttled:
4026.2983 5259.7043
2030.1287 6940.2326
6027.0439 9664.1045
8027.9993 10207.0579
10028.1762 12301.4746
12028.3190 12711.4440
14040.7972 17433.1964
16040.9267 17574.5924
18041.0529 19077.5545
Related
If I query items from CosmosDB with SkipToken,
Like Pseudo code:
do{
var page = Query();
foreach(var item in page)
{
Update(item);
}
}while(HasNextPage());
The page I get may not be complete, which means I will miss some item.
But if I wait a moment after Update
Like:
do{
var page = Query();
foreach(var item in page)
{
Update(item);
}
// difference here:
WaitAMoment();
}while(HasNextPage());
, the error will not happen, and I will get the complete page with all I need.
So what happened to such a process?
You don't have to wait in code as such, this functionality is handled by CosmosDB internally. Check out Pagination in SDKs of Cosmos DB and, for example sake, I am adding code of handling server-side pagination in C# below (to get a gist of how it works):
private static async Task QueryPartitionedContainerInParallelAsync(Container container)
{
List<Family> familiesSerial = new List<Family>();
string queryText = "SELECT * FROM Families";
// 0 maximum parallel tasks, effectively serial execution
QueryRequestOptions options = new QueryRequestOptions() { MaxBufferedItemCount = 100 };
options.MaxConcurrency = 0;
using (FeedIterator<Family> query = container.GetItemQueryIterator<Family>(
queryText,
requestOptions: options))
{
while (query.HasMoreResults)
{
foreach (Family family in await query.ReadNextAsync())
{
familiesSerial.Add(family);
}
}
}
Assert("Parallel Query expected two families", familiesSerial.ToList().Count == 2);
// 1 maximum parallel tasks, 1 dedicated asynchronous task to continuously make REST calls
List<Family> familiesParallel1 = new List<Family>();
options.MaxConcurrency = 1;
using (FeedIterator<Family> query = container.GetItemQueryIterator<Family>(
queryText,
requestOptions: options))
{
while (query.HasMoreResults)
{
foreach (Family family in await query.ReadNextAsync())
{
familiesParallel1.Add(family);
}
}
}
Assert("Parallel Query expected two families", familiesParallel1.ToList().Count == 2);
AssertSequenceEqual("Parallel query returns result out of order compared to serial execution", familiesSerial, familiesParallel1);
// 10 maximum parallel tasks, a maximum of 10 dedicated asynchronous tasks to continuously make REST calls
List<Family> familiesParallel10 = new List<Family>();
options.MaxConcurrency = 10;
using (FeedIterator<Family> query = container.GetItemQueryIterator<Family>(
queryText,
requestOptions: options))
{
while (query.HasMoreResults)
{
foreach (Family family in await query.ReadNextAsync())
{
familiesParallel10.Add(family);
}
}
}
Assert("Parallel Query expected two families", familiesParallel10.ToList().Count == 2);
AssertSequenceEqual("Parallel query returns result out of order compared to serial execution", familiesSerial, familiesParallel10);
}
I'm creating a BLE application using xamarin forms. Everything is working fine in Android I'm able to read and write GATT characteristics. In iOS I'm able to read successfully but the write operation doesn't update the characteristics value. There is no error in the write operation as well it is executing but the characteristics value is not changing. I tried iOS native application called light blue there its working fine the characteristic value is updated I'm facing issue only in Xamarin forms app. This is my code
private async Task<string> ProcessDeviceInformationService(IService deviceInfoService)
{
try
{
await adapter.ConnectToDeviceAsync(device);
var sb = new StringBuilder("Getting information from Device Information service: \n");
var characteristics = deviceInfoService.GetCharacteristicsAsync();
var characteristic = await deviceInfoService.GetCharacteristicAsync(Guid.Parse("00002a2b-0000-1000-8000-00805F9B34FB"));
try
{
if (characteristic != null)
{
var sbnew = new StringBuilder("BLE Characteristics\n");
byte[] senddata = Encoding.UTF8.GetBytes(string.IsNullOrEmpty(SendMessageLabel.Text) ? "12" : SendMessageLabel.Text);
characteristic.ValueUpdated += (o, args) =>
{
var bytes = characteristic.Value;
};
await characteristic.WriteAsync(senddata);
string str = Encoding.UTF8.GetString(senddata);
sbnew.AppendLine($"Characteristics found on this device: {string.Join(", ", str.ToString())}");
CharactericsLabel.Text = sbnew.ToString();
}
}
catch (Exception ex)
{
//return ex.Message;
DisplayAlert("Notice", ex.Message.ToString(), "OK");
}
I tried delay and I also tried to get write without response from peripheral but it doesn't work. This is my peripheral code
// Current Time characteristic
BluetoothGattCharacteristic currentTime = new BluetoothGattCharacteristic(CURRENT_TIME,
//Read-only characteristic, supports notifications
BluetoothGattCharacteristic.PROPERTY_READ | BluetoothGattCharacteristic.PROPERTY_NOTIFY | BluetoothGattCharacteristic.PROPERTY_WRITE_NO_RESPONSE,
BluetoothGattCharacteristic.PERMISSION_READ | BluetoothGattCharacteristic.PROPERTY_WRITE);
BluetoothGattDescriptor configDescriptor = new BluetoothGattDescriptor(CLIENT_CONFIG,
//Read/write descriptor
BluetoothGattDescriptor.PERMISSION_READ | BluetoothGattDescriptor.PERMISSION_WRITE);
currentTime.addDescriptor(configDescriptor);
// Local Time Information characteristic
BluetoothGattCharacteristic localTime = new BluetoothGattCharacteristic(LOCAL_TIME_INFO,
//Read-only characteristic
BluetoothGattCharacteristic.PROPERTY_READ,
BluetoothGattCharacteristic.PERMISSION_READ);
BluetoothGattCharacteristic sampleText = new BluetoothGattCharacteristic sampleText = new BluetoothGattCharacteristic(SAMPLE_TEXT,
//Read-only characteristic
BluetoothGattCharacteristic.PROPERTY_WRITE_NO_RESPONSE | BluetoothGattCharacteristic.PROPERTY_READ | BluetoothGattCharacteristic.PROPERTY_NOTIFY,
BluetoothGattCharacteristic.PERMISSION_WRITE | BluetoothGattCharacteristic.PERMISSION_READ);
I have no clue how to fix this any suggestions.I tried even Semaphore but it didn't help as you can see in my code
private static async Task<string> WriteAndWaitForResponseAsync(
ICharacteristic characteristic,
byte[] senddata)
{
var semaphore = new SemaphoreSlim(0, 1);
string result = null;
characteristic.ValueUpdated += (o, args) =>
{
var bytes = characteristic.Value;
result = Encoding.UTF8.GetString(bytes); // Note I don't know if this is your intended behaviour with the values you get back, you can decide what to actually do with the response.
// Notify a value has been received.
semaphore.Release();
};
await characteristic.WriteAsync(senddata,new CancellationToken(true)).ConfigureAwait(false);
// Wait until we receive a notification.
await semaphore.WaitAsync(); // I strongly suggest you look in to CancellationTokens but I am not going in to that now.
return result;
}
I suspect that threading is part of the problem here and that you subscribe for the event, call WriteAsync but then you ultimately leave the method before the data/event has been received. I suggest you need to try blocking the method from leaving before you receive your data.
Try something like:
private static async Task<string> WriteAndWaitForResponseAsync(
ICharacteristic characteristic,
byte[] senddata)
{
var semaphore = new SemaphoreSlim(0, 1);
string result = null;
characteristic.ValueUpdated += (o, args) =>
{
// Make sure you read from the new value in event not the existing characteristic.
var bytes = args.Characteristic.Value;
result = Encoding.UTF8.GetString(bytes); // Note I don't know if this is your intended behaviour with the values you get back, you can decide what to actually do with the response.
// Notify a value has been received.
semaphore.Release();
};
await characteristic.WriteAsync(senddata);
// Wait until we receive a notification.
await semaphore.WaitAsync(); // I strongly suggest you look in to CancellationTokens but I am not going in to that now.
return result;
}
Then you can call this method from within your current one like:
string str = await WriteAndWaitForResponseAsync(characteristic, senddata);
I'm trying to use a dataflowblock and I need to spy the items passing through for unit testing.
In order to do this, I'm using the AsObservable() method on ISourceBlock<T> of my TransformBlock<Tinput, T>,
so I can check after execution that each block of my pipeline have generated the expected values.
Pipeline
{
...
var observer = new MyObserver<string>();
_block = new TransformManyBlock<string, string>(MyHandler, options);
_block.LinkTo(_nextBlock);
_block.AsObservable().Subscribe(observer);
_block.Post("Test");
...
}
MyObserver
public class MyObserver<T> : IObserver<T>
{
public List<Exception> Errors = new List<Exception>();
public bool IsComplete = false;
public List<T> Values = new List<T>();
public void OnCompleted()
{
IsComplete = true;
}
public void OnNext(T value)
{
Values.Add(value);
}
public void OnError(Exception e)
{
Errors.Add(e);
}
}
So basically I subscribe my observer to the transformblock, and I expect that each value passing through get registered in my observer "values" list.
But, while the IsComplete is set to true, and the OnError() successfully register exception,
the OnNext() method never get called unless it is the last block of the pipeline...
I can't figure out why, because the "nextblock" linked to this sourceBlock successfully receive the data, proving that some data are exiting the block.
From what I understand, the AsObservable is supposed to report every values exiting the block and not only the values that have not been consumed by other linked blocks...
What am I doing wrong ?
Your messages are being consumed by _nextBlock before you get a chance to read them.
If you comment out this line _block.LinkTo(_nextBlock); it would likely work.
AsObservable sole purpose is just to allow a block to be consumed from RX. It doesn't change the internal working of the block to broadcast messages to multiple targets. You need a special block for that BroadcastBlock
I would suggest broadcasting to another block and using that to Subscribe
BroadcastBlock’s mission in life is to enable all targets linked from
the block to get a copy of every element published
var options = new DataflowLinkOptions {PropagateCompletion = true};
var broadcastBlock = new BroadcastBlock<string>(x => x);
var bufferBlock = new BufferBlock<string>();
var actionBlock = new ActionBlock<string>(s => Console.WriteLine("Action " + s));
broadcastBlock.LinkTo(bufferBlock, options);
broadcastBlock.LinkTo(actionBlock, options);
bufferBlock.AsObservable().Subscribe(s => Console.WriteLine("peek " + s));
for (var i = 0; i < 5; i++)
await broadcastBlock.SendAsync(i.ToString());
broadcastBlock.Complete();
await actionBlock.Completion;
Output
peek 0
Action 0
Action 1
Action 2
Action 3
Action 4
peek 1
peek 2
peek 3
peek 4
I have a an aync method that is looped:
private Task<HttpResponseMessage> GetResponseMessage(Region region, DateTime startDate, DateTime endDate)
{
var longLatString = $"q={region.LongLat.Lat},{region.LongLat.Long}";
var startDateString = $"{startDateQueryParam}={ConvertDateTimeToApixuQueryString(startDate)}";
var endDateString = $"{endDateQueryParam}={ConvertDateTimeToApixuQueryString(endDate)}";
var url = $"http://api?key={Config.Key}&{longLatString}&{startDateString}&{endDateString}";
return Client.GetAsync(url);
}
I then take the response and save it to my ef core database, however in some instances I get this Exception message: The Operaiton was canceled
I really dont understand that. This is a TCP handshake issue?
Edit:
For context I am making many of these calls, passing response to the method that writes to db (which is also so slow Its unbelievable):
private async Task<int> WriteResult(Response apiResponse, Region region)
{
// since context is not thread safe we ensure we have a new one for each insert
// since a .net core app can insert data at the same time from different users different instances of context
// must be thread safe
using (var context = new DalContext(ContextOptions))
{
var batch = new List<HistoricalWeather>();
foreach (var forecast in apiResponse.Forecast.Forecastday)
{
// avoid inserting duplicates
var existingRecord = context.HistoricalWeather
.FirstOrDefault(x => x.RegionId == region.Id &&
IsOnSameDate(x.Date.UtcDateTime, forecast.Date));
if (existingRecord != null)
{
continue;
}
var newHistoricalWeather = new HistoricalWeather
{
RegionId = region.Id,
CelsiusMin = forecast.Day.Mintemp_c,
CelsiusMax = forecast.Day.Maxtemp_c,
CelsiusAverage = forecast.Day.Avgtemp_c,
MaxWindMph = forecast.Day.Maxwind_mph,
PrecipitationMillimeters = forecast.Day.Totalprecip_mm,
AverageHumidity = forecast.Day.Avghumidity,
AverageVisibilityMph = forecast.Day.Avgvis_miles,
UvIndex = forecast.Day.Uv,
Date = new DateTimeOffset(forecast.Date),
Condition = forecast.Day.Condition.Text
};
batch.Add(newHistoricalWeather);
}
context.HistoricalWeather.AddRange(batch);
var inserts = await context.SaveChangesAsync();
return inserts;
}
Edit: I am making 150,000 calls. I know this is questionable since It all goes in memory I guess before even doing a save but this is where I got to in trying to make this run faster... only I guess my actual writing code is blocking :/
var dbInserts = await Task.WhenAll(
getTasks // the list of all api get requests
.Select(async x => {
// parsed can be null if get failed
var parsed = await ParseApixuResponse(x.Item1); // readcontentasync and just return the deserialized json
return new Tuple<ApiResult, Region>(parsed, x.Item2);
})
.Select(async x => {
var finishedGet = await x;
if(finishedGet.Item1 == null)
{
return 0;
}
return await writeResult(finishedGet.Item1, finishedGet.Item2);
})
);
.net core has a DefaultConnectionLimit setting as answered in comments.
this limits outgoing connections to specific domains to ensure all ports are not taken etc.
i did my parallel work incorrectly causing it to go over the limit - which everything i read says should not be 2 on .net core but it was - and that caused connections to close before receiving responses.
I made it greater, did parallel work correctly, lowered it again.
Here is my method for requesting information from Azures FaceAPI.
I finally realized that in order to make my application work best, a bunch of my security camera frames must be grabbed in parallel, then sent away to the Neural Network to be analyzed.
(Note: it's an Alexa custom skill, so it times out around 8-10 seconds)
Because I grab a bunch of frames in parallel, and asynchronously, there is no way to know which of the images will return a decent face detection. However, when a good detection is found, and the request return Face Data, there is no way to stop the rest of the information from returning.
This happens because the Security camera images were sent in Parallel, they are gone, the info is coming back no matter what.
You'll see that I'm able to use "thread local variables" to capture information and send it back to the function scoped "imageAnalysis" variable to serialize and allow Alexa to describe people in the security image. BUT, because the loop is in Parallel, it doesn't break right away.
It may only take a second or two, but I'm on a time limit thanks to Alexas strict time-out policies.
There doesn't seem to be a way to break the parallel loop immediately...
Or is there?
The more time is spent collecting the "imageAnalysis" Data, the longer Alexa has to wait for a response. She doesn't wait long, and it's important to try and send as many possible images for analysis as possible before Alexa times-out, and also keeping under the Azure FaceAPI limits.
public static async Task<List<Person>> DetectPersonAsync()
{
ConfigurationDto config = Configuration.Configuration.GetSettings();
string imageAnalysis;
using (var cameraApi = new SecurityCameraApi())
{
byte[] imageData = cameraApi.GetImageAsByte(config.SecurityCameraUrl +
config.SecurityCameraStaticImage +
DateTime.Now);
//Unable to get an image from the Security Camera
if (!imageData.Any())
{
Logger.LogInfo(Logger.LogType.Info, "Unable to aquire image from Security Camera \n");
return null;
}
Logger.LogInfo(Logger.LogType.Info, "Attempting Image Face Detection...\n");
Func<string, bool> isEmptyOrErrorAnalysis = s => s.Equals("[]") || s.Contains("error");
imageAnalysis = "[]";
List<byte[]> savedImageList = cameraApi.GetListOfSavedImagesAsByte();
if (savedImageList.Any())
{
Parallel.ForEach(savedImageList, new ParallelOptions
{
MaxDegreeOfParallelism = 50
},
async (image, loopState) =>
{
string threadLocalImageAnalysis = "[]";
if (!loopState.IsStopped)
threadLocalImageAnalysis = await GetImageAnalysisAsync(image, config);
if (!isEmptyOrErrorAnalysis(threadLocalImageAnalysis))
{
imageAnalysis = threadLocalImageAnalysis;
loopState.Break();
}
});
}
// Don't do to many image analysis - or Alexa will time-out.
Func<List<byte[]>, int> detectionCount =
savedImageListDetections => savedImageListDetections.Count > 5 ? 0 : 16;
//Continue with detections of current Camera image frames
if (isEmptyOrErrorAnalysis(imageAnalysis))
{
Parallel.For(0, detectionCount(savedImageList), new ParallelOptions
{
MaxDegreeOfParallelism = 50
},
async (i, loopState) =>
{
imageData = cameraApi.GetImageAsByte(config.SecurityCameraUrl +
config.SecurityCameraStaticImage +
DateTime.Now);
string threadLocalImageAnalysis = "[]";
if (!loopState.IsStopped)
threadLocalImageAnalysis = await GetImageAnalysisAsync(imageData, config);
if (!isEmptyOrErrorAnalysis(threadLocalImageAnalysis))
{
imageAnalysis = threadLocalImageAnalysis;
loopState.Break();
}
});
}
}
try
{
//Cognitive sense has found elements(people) in the image
return new NewtonsoftJsonSerializer().DeserializeFromString<List<Person>>(imageAnalysis);
}
catch (Exception ex)
{
//No elements(people) detected from the camera stream
Logger.LogInfo(Logger.LogType.Info,
string.Format(
"Microsoft Cognitive Sense Face Api Reports: \n{0} \nNo people in the CCTV Camera Image.\n",
ex.Message));
return new List<Person>(); //Empty
}
}
private static async Task<string> GetImageAnalysisAsync(byte[] image, ConfigurationDto config)
{
string json;
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key",
config.MicrosoftCognitiveSenseApiKey);
// Request parameters.
const string queryString =
"returnFaceId=true" +
"&returnFaceLandmarks=false" +
"&returnFaceAttributes=age,gender,accessories,hair";
const string uri =
"https://westus.api.cognitive.microsoft.com/face/v1.0/detect?" + queryString;
using (var content = new ByteArrayContent(image))
{
content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
HttpResponseMessage response = await client.PostAsync(uri, content);
json = await response.Content.ReadAsStringAsync();
try
{
Logger.LogInfo(Logger.LogType.Info, json);
}
catch
{
}
}
}
return json;
}