Exception retry attempts are higher than expected

Exception retry attempts are higher than expected - c#

I'm counting my Exceptions for a basic test application. I'm expecting exceptions, we are testing the application's behavior during SQL High Availability failovers. There is a blink period while the server fails over where some exceptions are expected, briefly. I have some basic retry logic in my catch to count and handle/ retry the action that had the exceptions. Here is my code.
My retry class:
public static class Retry
{
public static void DoWithRetry(Action action, TimeSpan sleepPeriod, int retryCount = 3)
{
while (true)
{
try
{
action();
break; // success!
}
catch
{
if (--retryCount == 0) throw;
else Thread.Sleep(sleepPeriod);
}
}
}
}
Here is where I'm referencing it in my catch:
catch (Exception ex)
{
ExCounter++;
Retry.DoWithRetry(DbInsertTestTable, TimeSpan.FromSeconds(1), retryCount: 3);
//AppCount.Text = ex.Message;
}
The retry works but my exception count seems off as I'm getting over 100. If I turn off the retry I only get an avg of about 5 exceptions. With the retry logic I'm expecting it to retry the action after a 1 second timespan sleep with a maximum of 3 attempts. How am I getting over 100 exceptions? Is there a postback here that is resetting my timespan or retry count?

I thought Timespan wasn't being passed correctly but I was wrong. I needed to break the initial SQL insert into it's own method. The problem was my retry which is a try catch was retrying an action() that had the original try/catch in it. This was causing a loop of sorts in try catches which is why I had so many exceptions. I separated the insert itself into a method and then referenced that in first try, if there's an exception, it's gets sent to my retry which uses that same SQL method (instead of using that same SQL insert with Try/Catch). Here is what it looks like now.
private void DbInsertTestTable()
{
string User = System.Web.HttpContext.Current.User.Identity.Name.ToString();
string ServName = "";
string SqlStatement = "INSERT INTO [myDB].[dbo].[mytable] ([AppCounterData],[User],[DateTime],[ServName]) VALUES (#COLUMN1,#COLUMN2,#COLUMN3,##ServerName);";
using (SqlConnection c = new SqlConnection(System.Configuration.ConfigurationManager.ConnectionStrings["webconfconstringname"].ConnectionString))
{
SqlCommand cmd = SqlCmdQuery(c, SqlStatement);
cmd.Parameters.AddWithValue("#COLUMN1", AppCounter);
cmd.Parameters.AddWithValue("#COLUMN2", User);
cmd.Parameters.AddWithValue("#COLUMN3", DateTime.Now);
cmd.Parameters.AddWithValue("#COLUMN4", ServName);
SqlCmdOpExCl(c, cmd);
}
}
private void SqlInsertFirstAttempt()
{
try
{
DbInsertTestTable();
}
catch (SqlException sqlEx)
{
DoWithRetry(DbInsertTestTable, TimeSpan.FromSeconds(1), retryCount: 30);
}
}
Still pretty new to the game. I'm open to comments but this code is working as it should. My exception count is normal and my retries are decrementing.

Related

What is the best way to catch "Operation cancelled by user" exception

i have following peace of code:
IAsyncResult beginExecuteReader = command.BeginExecuteNonQuery();
while (!beginExecuteReader.IsCompleted)
{
if (controllerTask.CancellationTokenSource.IsCancellationRequested)
{
command.Cancel();
}
Thread.Sleep(100);
}
try
{
result = command.EndExecuteNonQuery(beginExecuteReader);
}
catch (SqlException exception)
{
if (exception.ErrorCode == OperationCanceled)
{
throw new OperationCanceledException();
}
throw;
}
How can i identify, that catched exception is caused by operation cancelation. In this case ExecuteNonQuery throws exception with error code 0x80131904, but it's very general exception which can be caused by many reasons. Error message looks like this: {"A severe error occurred on the current command. The results, if any, should be discarded.\r\nOperation cancelled by user."}
I don't see any options except of parsing of error message... Any ideas?
Thanks
PS. Yeah, i know that Cancel command for asyncronyc operation probably is not the best idea, because for .NET 2.0 there was warning on MSDN, but for .NET 4.0 this warning is removed. And i also don't like another implementations when cancel method is called from another thread, as for me it makes code more difficult

There doesn't seem to be a locale insensitive mechanism to catch just this error. The HResult 0x80131904 is just COR_E_SqlException. The error is initiated at TdsParser.cs:2332 without any unique properties. It is almost the exact same code as :2759 - Unknown Error and :3850 - Unexpected Collation.
Here are the bad solutions I have come up with:
Option 1: Break the good advice of "don't make logic locale sensitive"
using (var con = new SqlConnection("Server=(local);Integrated Security=True;"))
{
con.Open();
try
{
var sqc = new SqlCommand("WAITFOR DELAY '1:00:00'", con);
var readThread = Task.Run(() => sqc.ExecuteNonQuery());
// cancel after 5 seconds
Thread.Sleep(5000);
sqc.Cancel();
// this should throw
await readThread;
// unreachable
Console.WriteLine("Succeeded");
}
catch (SqlException ex) when (ex.Number == 0 && ex.State == 0 && ex.Class == 11
&& ex.Message.Contains("Operation cancelled by user."))
{
Console.WriteLine("Cancelled");
}
catch (Exception ex)
{
Console.WriteLine("Error");
}
}
Option 2: Assume that no other severe locally generated error matters after a cancel has been issued
using (var con = new SqlConnection("Server=(local);Integrated Security=True;"))
{
con.Open();
bool isCancelled = false;
try
{
var sqc = new SqlCommand("WAITFOR DELAY '1:00:00'", con);
var readThread = Task.Run(() => sqc.ExecuteNonQuery());
// cancel after 5 seconds
Thread.Sleep(5000);
isCancelled = true;
sqc.Cancel();
// this should throw
await readThread;
// unreachable
Console.WriteLine("Succeeded");
}
catch (SqlException ex) when (isCancelled && ex.Number == 0 && ex.State == 0 && ex.Class == 11)
{
Console.WriteLine("Cancelled");
}
catch (Exception ex)
{
Console.WriteLine("Error");
}
}

So imho you should to do next:
Make a thread where you will use ado (read this Thread example)
Delete Thread.Sleep(100); //Imho never use it
Add to your class static bool muststop=false;
Add to your class public static function to change "muststop"
Change you thread's function to stop it if muststop==true
I hope this help you

You can exam exception message in catch block to find which operation was cancelled by user:
try
{
//your code
}
catch (SqlException ex)
{
if (ex.Message.Contain("Operation cancelled by user"))
{
//Do something here
}
}

Retries in Catch block?

How can I implement the code in the catch block?
try
{
// Call a MS SQL stored procedure (MS SQL 2000)
// Stored Procedure may deadlock
}
catch
{
// if deadlocked Call a MS SQL stored procedure (may deadlock again)
// If deadlocked, keep trying until stored procedure executes
}
finally
{
}

Doing this isn't recommended and could cause serious problems in your program. For example, what if the database was down?
But, here's how to do it in a loop:
for(int attempts = 0; attempts < 5; attempts++)
// if you really want to keep going until it works, use for(;;)
{
try
{
DoWork();
break;
}
catch { }
Thread.Sleep(50); // Possibly a good idea to pause here, explanation below
}
Update: As Mr. Disappointment mentioned in a comment below: The Thread.Sleep method pauses the execution for the specified number of milliseconds. No error is completely random, most that would work simply by trying again only work because something has changed in the time it took between the tries. Pausing the execution of the thread will give a much bigger window of opportunity for this to happen (for example, more time for the database engine to start up).

What about something like this
bool retry = true;
while( retry ){
try{
...
retry = false;
}
catch
{
...
}
finally
{
...
}
}
As long as the last line of the try block gets run ( retry = false ), it will carry on. If some exception occurs, it will run the catch and finally block, and then loop back up and run the try block again.
If you want to only try x times, you can replace the retry with a int with a startvalue of number of tries first. Then check if it equals 0 in the while loop, decrement it in the start of the loop, and set it to 0 as the last line of the try block.
And you should of course do something to that empty catch block so it catches the exceptions you anticipate, and not one that catches everything.

Copied verbatim from a Microsoft Developer Network page on what they call the Retry Pattern:
private int retryCount = 3;
...
public async Task OperationWithBasicRetryAsync()
{
int currentRetry = 0;
for (; ;)
{
try
{
// Calling external service.
await TransientOperationAsync();
// Return or break.
break;
}
catch (Exception ex)
{
Trace.TraceError("Operation Exception");
currentRetry++;
// Check if the exception thrown was a transient exception
// based on the logic in the error detection strategy.
// Determine whether to retry the operation, as well as how
// long to wait, based on the retry strategy.
if (currentRetry > this.retryCount || !IsTransient(ex))
{
// If this is not a transient error
// or we should not retry re-throw the exception.
throw;
}
}
// Wait to retry the operation.
// Consider calculating an exponential delay here and
// using a strategy best suited for the operation and fault.
Await.Task.Delay();
}
}
// Async method that wraps a call to a remote service (details not shown).
private async Task TransientOperationAsync()
{
...
}
They go into more detail, explaining appropriate uses, and non-appropriate uses of this pattern. For example, if you expect the errors you're running into are transient, and that retrying again in a moment will likely succeed, this may be for you. If this is to help you deal with some scaling problems, this is not for you.
You may also be interested in their Circuit Breaker Pattern which they describe as being able to, "Handle faults that may take a variable amount of time to rectify when connecting to a remote service or resource."

Don't implement it in the catch block. Instead write a loop around it that repeats until either it was successful or some limit is reached.
Something like:
bool quit = false;
int loopcount = 0;
while(!quit )
{
try
{
// execute the command, might throw an exception)
quit = true; // no exception if you got here
}
catch(Exception ex)
{
if (ex != deadlock) // doesn't work like this :-(
quit = true;
}
finally
{
// etc.
}
loopcount++;
if (loopcount > 3)
quit = true;
}

It may be as simple as wrapping the whole try/catch in a while loop:
while (!success) {
try
{
// Call a MS SQL stored procedure (MS SQL 2000)
// Stored Procedure may deadlock
success = true;
}
catch
{
// if deadlocked Call a MS SQL stored procedure (may deadlock again)
// If deadlocked, keep trying until stored procedure executes
success = false;
}
}

You really shouldn't just hammer the database until it succeeds in executing your SP, but that's another story.
You could do it like this:
Boolean succeeded = false;
while (!succeeded)
{
try
{
// Call a MS SQL stored procedure (MS SQL 2000)
// Stored Procedure may deadlock
succeeded = true;
}
catch (Exception ex)
{
// Log
}
}

You can implement Timers to check the healthy of your store procedures, and throw answers based on that, inside a loop as the colleagues said.

Repeating a function in C# until it no longer throws an exception

I've got a class that calls a SOAP interface, and gets an array of data back. However, if this request times out, it throws an exception. This is good. However, I want my program to attempt to make this call again. If it times out, I'd like it to keep making this call until it succeeds. How can I accomplish this?
For example:
try
{
salesOrdersArray = MagServ.salesOrderList(sessID, filter);
}
catch
{
?? What Goes Here to FORCE the above line of code to rerun until it succeeds.
}

You just need to loop forever:
while (true)
{
try
{
salesOrdersArray = MagServ.salesOrderList(sessID, filter);
break; // Exit the loop. Could return from the method, depending
// on what it does...
}
catch
{
// Log, I suspect...
}
}
Note that you should almost certainly not actually loop forever. You should almost certainly have a maximum number of attempts, and probably only catch specific exceptions. Catching all exceptions forever could be appalling... imagine if salesOrderList (unconventional method name, btw) throws ArgumentNullException because you've got a bug and filter is null... do you really want to tie up 100% of your CPU forever?

You must place the try/catch block inside a loop construct. If you wish not to consume 100% of your processor place a Thread.Sleep in the catch block, so everytime an exception occurs, it will wait some time, freeing the processor to do other things.
// iterate 100 times... not forever!
for (int i = 0; i < 100; i++)
{
try {
// do your work here;
break; // break the loop if everything is fine
} catch {
Thread.Sleep(1000);
}
}
You could also specify exception type, so that only the timeout exception is handled, and other kinds of exceptions pass-through.
// iterate 100 times... not forever!
for (int i = 0; i < 100; i++)
{
try {
// do your work here;
break; // break the loop if everything is fine
} catch (TimeOutException) {
Thread.Sleep(1000);
}
}
Note that, TimeOutException should be replaced by the real name of the exception... I don't know if that is the real name.
Also adjust the sleep time, given in millisecs and the amount of repeats, in the case I presented, 100 repeats of 1000ms yields a maximum wait of 1 minute and 40 seconds, plus the operation time itself.

If you can't change the timeout, the below should work. salesOrdersArray should be initialized to null.
while(salesOrdersArray == null)
{
try
{
salesOrdersArray = MagServ.salesOrderList(sessID, filter);
}
catch
{
// Log failure
}
}

It its not gernally a good idead to use exceptions as control flow, but this will do what you requested.
bool Caught = true;
while (Caught)
try
{
salesOrdersArray = MagServ.salesOrderList(sessID, filter);
Caught = false;
}
catch
{
Caught = true;
}

I will use a transactional queue (MSMQ) to store the service call. A loop will dequeue messages and call the service in a TransactionScope, if the call fails the message appear to be still in the queue. An ov erall timeout can be specified by adding a time to expire in the message. This solution is good if you really want a reliable solution since I guessed that calling that operation is critical.

Try
bool failed = false;
do {
try
{
salesOrdersArray = MagServ.salesOrderList(sessID, filter);
}
catch
{
failed = true;
}
} while(failed);
The behavior you are after might cause an endless loop if this never succeeds though...

Try something like this:
var failed = true;
while (failed)
{
try
{
salesOrdersArray = MagServ.salesOrderList(sessID, filter);
failed = false;
}
catch
{
}
}
Edit: Wow! Great minds think alike! :)

Although I would NOT recommend you to do this for an infinite number of times, you could make a separate function out of that one sentence:
void GoConnect()
{
try
{
salesOrdersArray = MagServ.salesOrderList(sessID, filter);
}
catch
{
GoConnect();
}
}

while(salesOrdersArray == null){
try
{
salesOrdersArray = MagServ.salesOrderList(sessID, filter);
}
catch(salesOrderException e)
{
log(e.message);
}
}
This will run forever, and is using exceptions as a loop which is slow. Is there a way you can modify your function that it returns null, instead of throwing an exception? If you're expecting that this call will fail regularly, don't use a try/catch block.

I follow this pattern in order to solve this problem:
public void Send(String data, Int32 attemptNumber)
{
try
{
yourCodeHere(data);
}
catch (WebException ex)
{
if (attemptNumber > 0)
Send(data, --attemptNumber);
else
throw new AttemptNumberExceededException("Attempt number exceeded!", ex);
}
catch (Exception ex)
{
//Log pourpose code goes here!
throw;
}
}
Trying forever seems not to be a good idea as you may end up having an infinite process. If you think you need many attempts to achieve your goal just set huge number here.
I personally think its wise to wait some milliseconds, or seconds after eac attempt Thread.Sleep(1000); before callig Send(data); --- you could for example, use the attempNumber variable to increse or decrease this waiting time if you think its wise for your scenario.

bool repeat = true;
while (repeat)
{
try
{
salesOrdersArray = MagServ.salesOrderList(sessID, filter);
repeat = false;
}
catch
{
}
}

What is good C# coding style for catching SQLException and retrying

I have a method that calls a SQLServer function to perform a free text search against a table. That function will occasionally on the first call result in a SQLException: "Word breaking timed out for the full-text query string". So typically I want to retry that request because it will succeed on subsequent requests. What is good style for structuring the retry logic. At the moment I have the following:
var retryCount = 0;
var results = new List<UserSummaryDto>();
using (var ctx = new UsersDataContext(ConfigurationManager.ConnectionStrings[CONNECTION_STRING_KEY].ConnectionString))
{
for (; ; )
{
try
{
results = ctx.SearchPhoneList(value, maxRows)
.Select(user => user.ToDto())
.ToList();
break;
}
catch (SqlException)
{
retryCount++;
if (retryCount > MAX_RETRY) throw;
}
}
}
return results;

I'd change the exception handling to only retry on certain errors:
1204, 1205 deadlocks
-2 timeout
-1 connection broken
These are the basic "retryable" errors
catch (SqlException ex)
{
if !(ex.Number == 1205 || ex.Number == 1204 || ... )
{
throw
}
retryCount++;
if (retryCount > MAX_RETRY) throw;
}
Edit, I clean forgot about waits so you don't hammer the SQL box:
Add a 500 ms wait on deadlock
Add a 5 sec delay on timeout
Edit 2:
I'm a Developer DBA, don't do much C#.
My answer was to correct exception processing for the calls...

Thanks for all the feedback. I'm answering this myself so I can incorporate elements from the answers given. Please let me know if I've missed something. My method becomes:
var results = new List<UserSummaryDto>();
Retry<UsersDataContext>(ctx => results = ctx.SearchPhoneList(value, maxRows)
.Select(user => user.ToDto())
.ToList());
return results;
And I've refactored the original method for reuse. Still lots of levels of nesting. It also relies on there being a default constructor for the data context which may be too restrictive. #Martin, I considered including your PreserveStackTrace method but in this case I don't think it really adds enough value - good to know for future reference thanks:
private const int MAX_RETRY = 2;
private const double LONG_WAIT_SECONDS = 5;
private const double SHORT_WAIT_SECONDS = 0.5;
private static readonly TimeSpan longWait = TimeSpan.FromSeconds(LONG_WAIT_SECONDS);
private static readonly TimeSpan shortWait = TimeSpan.FromSeconds(SHORT_WAIT_SECONDS);
private enum RetryableSqlErrors
{
Timeout = -2,
NoLock = 1204,
Deadlock = 1205,
WordbreakerTimeout = 30053,
}
private void Retry<T>(Action<T> retryAction) where T : DataContext, new()
{
var retryCount = 0;
using (var ctx = new T())
{
for (;;)
{
try
{
retryAction(ctx);
break;
}
catch (SqlException ex)
when (ex.Number == (int) RetryableSqlErrors.Timeout &&
retryCount < MAX_RETRY)
{
Thread.Sleep(longWait);
}
catch (SqlException ex)
when (Enum.IsDefined(typeof(RetryableSqlErrors), ex.Number) &&
retryCount < MAX_RETRY)
{
Thread.Sleep(shortWait);
}
retryCount++;
}
}
}

My enum of retryables for sql looks like this:
SqlConnectionBroken = -1,
SqlTimeout = -2,
SqlOutOfMemory = 701,
SqlOutOfLocks = 1204,
SqlDeadlockVictim = 1205,
SqlLockRequestTimeout = 1222,
SqlTimeoutWaitingForMemoryResource = 8645,
SqlLowMemoryCondition = 8651,
SqlWordbreakerTimeout = 30053

It's not good style, but sometimes you have to do it, because you simply can't change existing code and have to deal with it.
I am using the following generic method for this scenario. Note the PreserveStackTrace() method, which can sometimes be very helpful in a re-throw scenario.
public static void RetryBeforeThrow<T>(Action action, int retries, int timeout) where T : Exception
{
if (action == null)
throw new ArgumentNullException("action", string.Format("Argument '{0}' cannot be null.", "action"));
int tries = 1;
do
{
try
{
action();
return;
}
catch (T ex)
{
if (retries <= 0)
{
PreserveStackTrace(ex);
throw;
}
Thread.Sleep(timeout);
}
}
while (tries++ < retries);
}
/// <summary>
/// Sets a flag on an <see cref="T:System.Exception"/> so that all the stack trace information is preserved
/// when the exception is re-thrown.
/// </summary>
/// <remarks>This is useful because "throw" removes information, such as the original stack frame.</remarks>
/// <see href="http://weblogs.asp.net/fmarguerie/archive/2008/01/02/rethrowing-exceptions-and-preserving-the-full-call-stack-trace.aspx"/>
public static void PreserveStackTrace(Exception ex)
{
MethodInfo preserveStackTrace = typeof(Exception).GetMethod("InternalPreserveStackTrace", BindingFlags.Instance | BindingFlags.NonPublic);
preserveStackTrace.Invoke(ex, null);
}
You would call it like that:
RetryBeforeThrow<SqlException>(() => MethodWhichFails(), 3, 100);

There is no good style for doing something like this. You'd be better off figuring out why the request fails the first time but succeeds the second time.
It seems possible that Sql Server has to initially compile an execution plan and then execute the query. So the first call fails because the combined times exceed your timeout property, and succeeds the second time because the execution plan is already compiled and saved.
I don't know how UsersDataContext works, but it may be the case that you have the option to Prepare the query before actually executing it.
Real Answer: If I had to do this, I would retry just once and not again, like this:
var results = new List<UserSummaryDto>();
using (var ctx = new
UsersDataContext(ConfigurationManager.ConnectionStrings[CONNECTION_STRING_KEY].ConnectionString))
{
try
{
results = ctx.SearchPhoneList(value, maxRows)
.Select(user => user.ToDto())
.ToList();
break;
}
catch (SqlException)
{
try
{
results = ctx.SearchPhoneList(value, maxRows)
.Select(user => user.ToDto())
.ToList();
break;
}
catch (SqlException)
{
// set return value, or indicate failure to user however
}
}
}
}
return results;
While I might trust you to not abuse the retry process, you'd be tempting your successor to increase the retry count as a quick fix.

I think annotating a method with an aspect specifying the retry count would result in more structured code, although it needs some infrastructure coding.

You can simply use SqlConnectionStringBuilder properties to sql connection retry.
var conBuilder = new SqlConnectionStringBuilder("Server=.;Database=xxxx;Trusted_Connection=True;MultipleActiveResultSets=true");
conBuilder.ConnectTimeout = 90;
conBuilder.ConnectRetryInterval = 15;
conBuilder.ConnectRetryCount = 6;
Note:- Required .Net 4.5 or later.

Pull the relevant code out into its own method, then use recursion.
Pseudo-code:
try
{
doDatabaseCall();
}
catch (exception e)
{
//Check exception object to confirm its the error you've been experiencing as opposed to the server being offline.
doDatabaseCall();
}

How can I improve this exception retry scenario?

I have a web service method I am calling which is 3rd party and outside of my domain. For some reason every now and again the web service fails with a gateway timeout. Its intermittent and a call to it directly after a failed attempt can succeed.
Now I am left with a coding dilemma, I have code that should do the trick, but the code looks like amateur hour, as you'll see below.
Is this really bad code, or acceptable given the usage? If its not acceptable, how can I improve it?
Please try hard to keep a straight face while looking at it.
try
{
MDO = OperationsWebService.MessageDownload(MI);
}
catch
{
try
{
MDO = OperationsWebService.MessageDownload(MI);
}
catch
{
try
{
MDO = OperationsWebService.MessageDownload(MI);
}
catch
{
try
{
MDO = OperationsWebService.MessageDownload(MI);
}
catch
{
try
{
MDO = OperationsWebService.MessageDownload(MI);
}
catch (Exception ex)
{
// 5 retries, ok now log and deal with the error.
}
}
}
}
}

You can do it in a loop.
Exception firstEx = null;
for(int i=0; i<5; i++)
{
try
{
MDO = OperationsWebService.MessageDownload(MI);
firstEx = null;
break;
}
catch(Exception ex)
{
if (firstEx == null)
{
firstEx = ex;
}
Thread.Sleep(100 * (i + 1));
}
}
if (firstEx != null)
{
throw new Exception("WebService call failed after 5 retries.", firstEx);
}

Here's another way you might try:
// Easier to change if you decide that 5 retries isn't right for you
Exception exceptionKeeper = null;
for (int i = 0; i < MAX_RETRIES; ++i)
{
try
{
MDO = OperationsWebService.MessageDownload(MI);
break; // correct point from Joe - thanks.
}
catch (Exception ex)
{
exceptionKeeper = ex;
// 5 retries, ok now log and deal with the error.
}
}
I think it documents the intent better. It's less code as well; easier to maintain.

All of the answers so far assume that the reaction to any exception should be to retry the operation. This is a good assumption right up until it's a false assumption. You could easily be retrying an operation that is damaging your system, all because you didn't check the exception type.
You should almost never use a bare "catch", nor "catch (Exception ex). Catch a more-specific exception - one you know you can safely recover from.

Try a loop, with some kind of limit:
int retryCount = 5;
var done = false;
Exception error = null;
while (!done && retryCount > 0)
{
try
{
MDO = OperationsWebService.MessageDownload(MI);
done = true;
}
catch (Exception ex)
{
error = ex;
}
if (done)
break;
retryCount--;
}

You should use recursion (or a loop), and should only retry if you got the error you expected.
For example:
static void TryExecute<TException>(Action method, Func<TException, bool> retryFilter, int maxRetries) where TException : Exception {
try {
method();
} catch(TException ex) {
if (maxRetries > 0 && retryFilter(ex))
TryExecute(method, retryFilter, maxRetries - 1);
else
throw;
}
}
EDIT: With a loop:
static void TryExecute<TException>(Action method, Func<TException, bool> retryFilter, int maxRetries) where TException : Exception {
while (true) {
try {
method();
return;
} catch(TException ex) {
if (maxRetries > 0 && retryFilter(ex))
maxRetries--;
else
throw;
}
}
}
You can try to prevent future errors in retryFilter, perhaps by Thread.Sleep.
If the last retry fails, this will throw the last exception.

Here is some retry logic we are using. We don't do this a lot and I was going to pull it out and document it as our Retry Pattern/Standard. I had to wing it when I first wrote it so I came here to see if I was doing it correctly. Looks like I was. The version below is fully commented. See below that for an uncommented version.
#region Retry logic for SomeWebService.MyMethod
// The following code wraps SomeWebService.MyMethod in retry logic
// in an attempt to account for network failures, timeouts, etc.
// Declare the return object for SomeWebService.MyMethod outside of
// the following for{} and try{} code so that we have it afterwards.
MyMethodResult result = null;
// This logic will attempt to retry the call to SomeWebService.MyMethod
for (int retryAttempt = 1; retryAttempt <= Config.MaxRetryAttempts; retryAttempt++)
{
try
{
result = SomeWebService.MyMethod(myId);
// If we didn't get an exception, then that (most likely) means that the
// call was successful so we can break out of the retry logic.
break;
}
catch (Exception ex)
{
// Ideally we want to only catch and act on specific
// exceptions related to the failure. However, in our
// testing, we found that the exception could be any type
// (service unavailable, timeout, database failure, etc.)
// and attempting to trap every exception that was retryable
// was burdensome. It was easier to just retry everything
// regardless of the cause of the exception. YMMV. Do what is
// appropriate for your scenario.
// Need to check to see if there will be another retry attempt allowed.
if (retryAttempt < Config.MaxRetryAttempts)
{
// Log that we are re-trying
Logger.LogEvent(string.Format("Retry attempt #{0} for SomeWebService.MyMethod({1})", retryAttempt, myId);
// Put the thread to sleep. Rather than using a straight time value for each
// iteration, we are going to multiply the sleep time by how many times we
// have currently tried to call the method. This will allow for an easy way to
// cover a broader range of time without having to use higher retry counts or timeouts.
// For example, if MaxRetryAttempts = 10 and RetrySleepSeconds = 60, the coverage will
// be as follows:
// - Retry #1 - Sleep for 1 minute
// - Retry #2 - Sleep for 2 minutes (covering three minutes total)
// - Retry #10 - Sleep for 10 minutes (and will have covered almost an hour of downtime)
Thread.Sleep(retryAttempt * Config.RetrySleepSeconds * 1000);
}
else
{
// If we made it here, we have tried to call the method several
// times without any luck. Time to give up and move on.
// Moving on could either mean:
// A) Logging the exception and moving on to the next item.
Logger.LogError(string.Format("Max Retry Attempts Exceeded for SomeWebService.MyMethod({0})", MyId), ex);
// B) Throwing the exception for the program to deal with.
throw new Exception(string.Format("Max Retry Attempts Exceeded for SomeWebService.MyMethod({0})", myId), ex);
// Or both. Your code, your call.
}
}
}
#endregion
I like Samuel Neff's example of using an exception variable to see if it completely failed or not. That would have made some of the evaluations in my logic a little simpler. I could go either way. Not sure that either way has a significant advantage over the other. However, at this point in time, I'm not going to change how we do it. The important thing is to document what you are doing and why so that some idiot doesn't come through behind you and muck with everything.
Just for kicks though, to get a better idea if the code is any shorter or cleaner one way or the other, I pulled out all the comments. They came out exactly the same number of lines. I went ahead and compiled the two versions and ran them through Reflector Code Metrics and got the following:
Metric: Inside-Catch / Outside-For
CodeSize: 197 / 185
CyclomaticComplexity: 3 / 3
Instructions: 79 / 80
Locals: 6 / 7
Final exception logic inside the catch (22 lines):
MyMethodResult result = null;
for (int retryAttempt = 1; retryAttempt <= Config.MaxRetryAttempts; retryAttempt++)
{
try
{
result = SomeWebService.MyMethod(myId);
break;
}
catch (Exception ex)
{
if (retryAttempt < Config.MaxRetryAttempts)
{
Logger.LogEvent(string.Format("Retry attempt #{0} for SomeWebService.MyMethod({1})", retryAttempt, myId);
Thread.Sleep(retryAttempt * Config.RetrySleepSeconds * 1000);
}
else
{
Logger.LogError(string.Format("Max Retry Attempts Exceeded for SomeWebService.MyMethod({0})", MyId), ex);
throw new Exception(string.Format("Max Retry Attempts Exceeded for SomeWebService.MyMethod({0})", myId), ex);
}
}
}
Final exception logic after the for-loop (22 lines):
MyMethodResult result = null;
Exception retryException = null;
for (int retryAttempt = 1; retryAttempt <= Config.MaxRetryAttempts; retryAttempt++)
{
try
{
result = SomeWebService.MyMethod(myId);
retryException = null;
break;
}
catch (Exception ex)
{
retryException = ex;
Logger.LogEvent(string.Format("Retry attempt #{0} for SomeWebService.MyMethod({1})", retryAttempt, myId);
Thread.Sleep(retryAttempt * Config.RetrySleepSeconds * 1000);
}
}
if (retryException != null)
{
Logger.LogError(string.Format("Max Retry Attempts Exceeded for SomeWebService.MyMethod({0})", MyId), ex);
throw new Exception(string.Format("Max Retry Attempts Exceeded for SomeWebService.MyMethod({0})", myId), ex);
}

I'm using the following generic method for a retry scenario. I especially want to draw attention to the PreserveStackTrace method which helps to preserve the full call stack trace, because (as I learned the hard way) neither throw or throw ex yields the complete call stack trace information.
public static void RetryBeforeThrow<T>(Action action, int retries, int timeout) where T : Exception
{
int tries = 1;
do
{
try
{
action();
return;
}
catch (T ex)
{
if (retries <= 0)
{
PreserveStackTrace(ex);
throw;
}
Thread.Sleep(timeout);
}
}
while (tries++ < retries);
}
/// <summary>
/// Sets a flag on an <see cref="T:System.Exception"/> so that all the stack trace information is preserved
/// when the exception is re-thrown.
/// </summary>
/// <remarks>This is useful because "throw" removes information, such as the original stack frame.</remarks>
/// <see href="http://weblogs.asp.net/fmarguerie/archive/2008/01/02/rethrowing-exceptions-and-preserving-the-full-call-stack-trace.aspx"/>
public static void PreserveStackTrace(Exception ex)
{
MethodInfo preserveStackTrace = typeof(Exception).GetMethod("InternalPreserveStackTrace", BindingFlags.Instance | BindingFlags.NonPublic);
preserveStackTrace.Invoke(ex, null);
}

As everyone else has pointed out the correct approach is to wrap your try/catch inside some loop with a MAX_RETRY of some sort.
You might also consider adding a timeout between each loop iteration. Otherwise you're likely to burn through your retry counter before the transient issue has had a chance to resolve itself.

It seems you have the answers you need, but I thought I'd post this link, What is an Action Policy?, that I found to provide a much more elegant solution. Lokad has some rather labyrinthine implementations, but the guy's logic is pretty solid, and the end code you'd end up writing is pretty and simple.

int cnt=0;
bool cont = true;
while (cont)
{
try
{
MDO = OperationsWebService.MessageDownload(MI);
cont = false;
}
catch (Exception ex)
{
++cnt;
if (cnt == 5)
{
// 5 retries, ok now log and deal with the error.
cont = false;
}
}
}
UPDATED : Fixed code based on comments.

Develop Reference

C# (C-Sharp) is a programming language developed by Microsoft that runs on the .NET Framework.

Exception retry attempts are higher than expected - c#

Related

What is the best way to catch "Operation cancelled by user" exception

Retries in Catch block?

Repeating a function in C# until it no longer throws an exception

What is good C# coding style for catching SQLException and retrying

How can I improve this exception retry scenario?

Categories

Resources