I am using Polly Circuit Breaker for an API which throws a user defined Exception if service is down for maintenance.
The Exception will contain a value how long the service will be down.
Is it possible to configure/update the circuit breaker policy after first user exception is handled ?
e.g.
CircuitBreakerPolicy breaker = Policy.Handle<UserException>()
.CircuitBreaker(
exceptionsAllowedBeforeBreaking: 1,
durationOfBreak: TimeSpan.FromMinutes(1));
Example()
{
try
{
string response = await breaker.ExecuteAsync<String>(() =>
{
return client.GetStringAsync("/api/values/");
});
}
catch (UserException ex)
{
var downtime = GetDowntime(ex);
//how to update the duration of break ?
breaker.durationOfBreak = downtime;
}
catch (Exception ex)
{
…
}
}
It is not possible to change the durationOfBreak on an existing CircuitBreakerPolicy instance, after creation.
For general dynamic reconfiguration during running, the Polly team recommends atomically updating a policy instance stored in the PolicyRegistry.
For the specific scenario:
[the] API throws a user defined Exception if service is down for maintenance [which] will contain a value how long the service will be down
you could, more simply, use the circuit-breaker's manual controls:
.Isolate() to manually force the circuit open;
.Reset() to manually reset the circuit.
You could .Isolate() the circuit when the given exception is thrown; and set a Timer to call Reset() on it at the time that the exception indicates the service should be available again.
Related
I have read the Microsoft Documentation but it doesn't mention CallActivityWithRetryAsync.
My IDurableOrchestrationContext is mocked for the call I'm making:
mockContext.Setup(c => c.CallActivityWithRetryAsync(nameof(SerialiseXml), It.IsAny<RetryOptions>(),
It.IsAny<InboundOrchestrationData>()))
.ThrowsAsync(new IOException());
and I can put a breakpoint at the calling point:
try
{
await context.CallActivityWithRetryAsync("SerialiseXml",
new RetryOptions(TimeSpan.FromMinutes(1), 3), data); // breakpoint
}
catch (Exception e)
{
log.LogError("Problem serialising xml.");
}
but my code only ever breaks here once when I would expect it to break three times according to the RetryOptions.
If I pass in a Mock<ILogger<Class>> and check the invocations there is also only one. I have also configured my mock using SetupSequence() and multiple .ThrowsAsync.
Do mind that if you start mocking
mockContext.Setup(c => c.CallActivityWithRetryAsync(nameof(SerialiseXml), It.IsAny<RetryOptions>(),
It.IsAny<InboundOrchestrationData>()))
.ThrowsAsync(new IOException());
there won't be any retries performed at all since it is mocked. There is no real implementation that is executed.
Inside the implementation of CallActivityWithRetryAsync there will be a retry of the specified activity call in case of a retryable failure. Only after all retry attempts have failed an exception will bubble up to your code. So I ever expect just one call to CallActivityWithRetryAsync.
Your exception handling code will also be just called once, and only when all retries have failed or there is another reason for the call to fail.
The implementation of CallActivityWithRetryAsync will look like this (simplified):
async Task<int> CallActivityWithRetryAsync(string activity, RetryOptions retryOptions)
{
int attempts = 1;
while (true)
{
try
{
return await CallActivity(activity);
}
catch
{
++attempts;
if (attempts > retryOptions.RetryCount)
{
throw;
}
}
}
}
Given this pseudocode you can easily see why the breakpoint hits only one.
I am developing NET Core web app and I am using blob to store some objects what could be modified during request. I had to prevent multiple paralel access to one object so I add 'lease' into my storage integration.
In prctice when I receive request, one object is fetched from blob with lease for some period. On the end of request this object is updated in storage and lease is removed - pretty simple.
But what is correct exception handling?
I am faicing agains the problem when some exception occured in the middle of request, lease is not released. I tried to implement release into dispose (in some class where I control fetching and leasing from blob). But this is not executed when unhandled exception is thrown.
Add try/catch/finally seems not clean for me. My question is do you know some best common approach how release lease on the end request? Thank you
Per your description, I write a simple demo for you about lease and break lease, just try the code below:
using System;
using Azure.Storage.Blobs;
using Microsoft.AspNetCore.Mvc;
using Azure.Storage.Blobs.Specialized;
using System.Threading;
namespace getSasTest.Controllers
{
[ApiController]
[Route("[controller]")]
public class editBlob : ControllerBase
{
[HttpGet]
public string get()
{
var connstr = "";
var container = "";
var blob = "";
var blobClient = new BlobContainerClient(connstr,container).GetBlobClient(blob);
var leaseClient = new BlobLeaseClient(blobClient);
try
{
//auto break lease after 15s
var duration = new TimeSpan(0, 0, 15);
leaseClient.Acquire(duration, null);
}
//if some error occurs, request ends here
catch (Azure.RequestFailedException e)
{
if (e.ErrorCode.Equals("LeaseAlreadyPresent"))
{
return "Blob is under process,it will take some time,please try again later";
}
else
{
return "some other Azure request errors:"+ e.Message;
}
}
catch (Exception e) {
return "some other errors:" + e.Message;
}
//mock time consumption to process blob
Thread.Sleep(10000);
//break relase first if process finishs in 15s.
leaseClient.Break();
return "Done";
}
}
}
So, based on your requirements, can't you authorize a party/cron/event (register event/ttl handlers?) to break a lease when some weirdness (whatever that means for you) is detected. It looks like you are really worried on the "pattern" over correctness?.
This should complement the correctness.
In practice, an exception handling strategy should lead to sufficient actionable information.
For some, that means:
E -> E - 1 (digest or no digest) -> E - 2 (digest or no digest) ..
such that:
E - n: Exception, at a certain level of nesting
digest: would you propagate this or digest and move forward?
Breaking the lease (essentially correctness of program) should not mean you hamper your version of the elegance.
Every service is usually a pair of services:
service itself
service clean-up handler 1 through n
service edge case handler 1 through n
primer for service
so on...
With no luck, I tried configuring my ServiceBusClient to retry a message with a Fixed Delay of 10 seconds. I also tried Exponential retries configuration. However, the code is always retrying the message within a second or 2 and completely ignoring the configuration. It even ignores the MaxRetries and only retries 10 times, the value configured in Azure Portal for the queue. What am I doing wrong?
I am using The Azure.Messaging.ServiceBus library, NuGet package 7.0.0.
The code:
ServiceBusClient client = new ServiceBusClient(serviceBusConnectionString, new ServiceBusClientOptions()
{
RetryOptions = new ServiceBusRetryOptions()
{
Mode = ServiceBusRetryMode.Fixed,
Delay = TimeSpan.FromSeconds(10),
MaxDelay = TimeSpan.FromMinutes(3),
MaxRetries = 30
}
});
ServiceBusProcessor processor = client.CreateProcessor(queueName, new ServiceBusProcessorOptions());
// throwing an exception in MyMessageHandlerAsync on purpose
// to test out the retries configuration
processor.ProcessMessageAsync += MyMessageHandlerAsync;
// The uncaught exception causes this method to execute.
// Processing is attempted 10 times with
// virtually no delay between each attempt.
// After the 10th attempt, the message goes to deadletter,
// which is expected.
processor.ProcessErrorAsync += MyErrorHandler;
I'm adding more to this question after receiving the 1st response:
Currently, MyMessageHandlerAsync is:
private async Task MyMessageHandlerAsync(EventArgs eventArgs)
{
var args = (ProcessMessageEventArgs)eventArgs;
var body = args.Message.Body.ToString();
// ...
// process body
// ...
await args.CompleteMessageAsync(args.Message);
}
How should I change the method's contents to retry a non-transient ServiceBusException? Please help provide the code where the TODOs are below:
private async Task MyMessageHandlerAsync(EventArgs eventArgs)
{
var args = (ProcessMessageEventArgs)eventArgs;
try
{
var body = args.Message.Body.ToString();
// ...
// process body
// ...
await args.CompleteMessageAsync(args.Message);
}
catch (ServiceBusException sbe)
{
if (sbe.IsTransiet)
{
// TODO: Is it correct that the exponential retry will work
// here? The one defined in the ServiceBusClient.
// So, no code is needed here, just throw.
throw;
}
else
{
// TODO: for non-transient, this is where the
// options in the ServiceBusClient don't apply.
// Is that correct? How do I do an
// exponential retry here?
}
}
catch (Exception e)
{
// TODO: same problem as else in first catch.
}
}
ServiceBusRetryOptions is intended to be used by the ASB client when there are transient errors that are not bubbled up to your code right away, i.e. an internal retry mechanism built into the client to perform retries on your behalf before exception is raised.
Use retry policy to specify to the ASB client how to deal with transient errors prior to giving up, not how many times a message handler throws error:
In my Azure Functions 2.x Project, i have a part of an Function, a try-catch block without finally, that more or less look like this.
Dictionary<string, int> varDict = null;
Tuple<string, DateTime> varTupl = null;
try
{
varDict = await Core.GetDict(lcLat.Value, lcLong.Value);
varTupl = await Core.GetTupl(lcLat.Value, lcLong.Value);
}
catch (AggregateException ae)
{
ae.Flatten().Handle(ex =>
{
// `log` is an ILogger, the standard Azure Functions passed param
log.LogError(ex, ""); // Writes the ex's error
Debug.WriteLine(""); // Writes the ex's error
// the written content is ommited for readability sake
// But will be shown below
return true;
});
}
catch (Exception ex)
{
// Does exactly like Handle() Does
}
if(varDict != null && varTupl != null)
{
// The Code won't go here, and always return HTTP 500 Instead
}
else
{
// Here neither
}
The Run method itself is an async Task<IActionResult>, with Core as a static public class containing GetDict() and GetTupl() methods, each of them are also an static async Task<T> with their respective T return type and both doesn't have any try-catch block, only using (which are not supposed to throw any exceptions, right ?)
The problem is, even though (i assume) the exceptions raised then bubbled up into my try-catch block, even with my catch block running printing the exception with my formatting from catch block, as shown in the screenshot ,my Azure Functions keep returning HTTP Error 500, skipping the rest of the code after the try-catch block
What i have tried
Disable 'Just My Code' debugging options in my Visual Stuido 2017
Adding AggregateExceptions, before this it's only catching for Exception
Flatten the AggregateException before Handle() it
Is this common on local development environment, or it's just me handling everything incorectly ?
Also, the output window keep printing out something like this
and this
even in idle state (while the HTTP endpoint isn't being invoked, just run in debug mode, idly waiting for invocation)
are these something that i have to concerned about ? are those even related with my problem
I execute calls to an external service that is not very stable, and thus throws WebExceptions.
I want to retry a few times and after the last attempt I want to throw the last error received.
This is my attempt with Polly (v6.1.1):
public static Policy WaitAndRetryPolicy<T>(short nrOfRetryAttempts = 5) where T : Exception
{
var waitAndRetry = Policy
.Handle<T>()
.WaitAndRetry(nrOfRetryAttempts, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));
var fallbackForLastError = Policy
.Handle<T>()
.Fallback(
fallbackAction: () => { },
onFallback: (ex) => { throw ex; });
return Policy.Wrap(fallbackForLastError, waitAndRetry);
}
Caller, legacy VB.Net:
Dim retryPolicy = Policies.WaitAndRetryPolicy(Of WebException)()
Dim theResult = retryPolicy.
ExecuteAndCapture(Function()
Return aProxy.GetSomething(a, b)
End Function).Result
When I run the code as depicted above, theResult stays null and it seems like the service is not called.
If I just use the WaitAndRetryPolicy without the Fallback function, the service is called and the retry mechanism works as expected (without throwing the exception of course).
How can I achieve my goal, without having to check PolicyResult.FinalException in the caller code?
To have Polly rethrow any final exception, rather than capture it into PolicyResult.FinalException, simply execute the policy with the .Execute(...) or .ExecuteAsync(...) overloads, rather than .ExecuteAndCapture(...) or .ExecuteAndCaptureAsync(...)
I don´t know about the last exception but i have implemented a very similar behaviour with Retry and CircuitBreakerException(with Wrapping). So you can try 3 times and throw a circuitbreakerexception after 2 failures. Then you are able to react on the last exception.
Policy
.Handle<SomeExceptionType>()
.CircuitBreaker(2, TimeSpan.FromMinutes(x));