Develop Reference

Best practice for working with CouchbaseClient

We are working with couchbase for more than a year and want to improve our work with their .Net sdk.
In their website i could not find the recommendation for the best way to use get/set.
I know that when i am trying to use : CouchbaseClient.Get(key) it can fail,
and than we should wait for X time and try again.
I want to understand what is the best practice for the waiting times when doing a get/set process.
Can anyone who encounter this issue advise me about it please?

In general, I suggest you use the ExecuteXXX where XXX is an operation methods as opposed to the regular/legacy Get or Set methods. The reason is that these methods return additional information about the operation and this info can be used to tailor your retry strategy.
For more information, read this portion of the documentation: http://docs.couchbase.com/couchbase-sdk-net-1.3/#working-with-operation-results-and-error-codes

I haven't used the .net client as I only work with Java and Ruby with Couchbase but the high level concepts are going to be the same.
You say that the get operation can fail, do you find it fails a lot? You have to handle several exceptions, operation timeout when the smart client fails to communicate in a timely manner with the cluster or the queue is too full to accept new requests (meant to be very rare).
You should wrap your calls (get/set/add) and catch the particular exception and then it all depends on what the requirements of your system are. If you are reaching a full queue retrying requests multiple times is going to make the situation worse. Is there not a default you can return in the case of an exception?
In our application (REST API) if we catch a timeout issue then we throw a custom exception which we return to the user notifying them to retry in the future.
IF we had the business requirement that we had to retry then we'd need to catch the timeout and either place the request onto a queue or invoke a specific retry strategy.
If you are catching either exception then it could be a sign that your system is or is about to go down, so monitor it carefully. I think this page from the Couchbase documentation covers the issues quite well.
https://www.couchbase.com/docs/couchbase-devguide-2.0/about-client-timeouts.html
And documentation on the C# specific sdk http://www.couchbase.com/docs/couchbase-sdk-net-1.2/couchbase-sdk-net-configuration.html
I hope this helps, if not feel free to expand on your original question with more details.

Thanks for the answers here.
By those answers and good post i found in this forum:
Retry process
I wrote some code and here is an example for one of its methods:
public static TResult Do<T,TResult>(Func<T,TResult> action, T param, TimeSpan retryInterval, int retryCount = 3)
{
for (int retry = 0; retry < retryCount; retry++)
{
try
{
return action(param);
}
catch (RetryOperationException)
{
Log.SaveFormat("Retry:{0} for action:{1}", "RetryProcessLog",retry, action.Method.Name);
Thread.Sleep(retryInterval);
}
catch (Exception ex)
{
Log.SaveFormat("Exception when retry:{0} error:{1}","RetryProcessLog",action.Method.Name,ex.Message);
return default(TResult);
}
}
return default(TResult);
}
public static T ExecuteGet<T>(string key)
{
var defaultReturnValue = default(T);
cacheHit("ExecuteGet", key);
var result = CacheClient.ExecuteGet<T>(CachePrefix + key);
if (result.Success)
{
return result.Value;
}
else if (result.StatusCode ==
CouchbaseStatusCode.StatusCodeExtension.ToInt(CouchbaseStatusCode.StatusCode.Busy)
||
result.StatusCode ==
CouchbaseStatusCode.StatusCodeExtension.ToInt(CouchbaseStatusCode.StatusCode.TemporaryFailure)
||
result.StatusCode ==
CouchbaseStatusCode.StatusCodeExtension.ToInt(CouchbaseStatusCode.StatusCode.SocketPoolTimeout)
||
result.StatusCode ==
CouchbaseStatusCode.StatusCodeExtension.ToInt(CouchbaseStatusCode.StatusCode.UnableToLocateNode)
||
result.StatusCode ==
CouchbaseStatusCode.StatusCodeExtension.ToInt(CouchbaseStatusCode.StatusCode.NodeShutdown)
||
result.StatusCode ==
CouchbaseStatusCode.StatusCodeExtension.ToInt(CouchbaseStatusCode.StatusCode.OperationTimeout))
{
Log.SaveFormat("Error:{0}:{2} in ExecuteGet for key:{1}. Going to throw RetryOperationException",
"CacheManager", result.StatusCode, key,result.Message);
throw new RetryOperationException();
}
Log.SaveFormat("Error:{0}:{2} in ExecuteGet for key:{1}", "CacheManager", result.StatusCode, key,result.Message);
return defaultReturnValue;
}
This is just an example and you can add more method overloads that gets more than one param or other method without a return value.
The second method is in different class and uses the ExecuteGet.
And here is an example of usage:
var result = RetryProcess.Do(CacheManager.ExecuteGet<long>, keyOfValue, TimeSpan.FromMilliseconds(10));
Hope it helps.

When to Throw an Exception? When to Handle an Exception (Service Layer, Controller) where?

I rewritten my question as I think it was too wordy and maybe what I am trying to achieve was lost.
I written this code in notepad so it may have mistakes and some stuff maybe not well thoughout but it is to illustrate what I see my options are.
// I wrap all code send back from service layer to controller in this class.
public class ResponseResult
{
public ResponseResult()
{
Errors = new Dictionary<string, string>();
Status = new ResponseBase();
}
public void AddError(string key, string errorMessage)
{
if (!Errors.ContainsKey(key))
{
Errors.Add(key, errorMessage);
}
}
public bool IsValid()
{
if (Errors.Count > 0)
{
return false;
}
return true;
}
public Dictionary<string, string> Errors { get; private set; }
public ResponseBase Status { get; set; }
}
public class ResponseResult<T> : ResponseResult
{
public T Response { get; set; }
}
public class ResponseBase
{
public HttpStatusCode Code { get; set; }
public string Message { get; set; }
}
Option 1 (what I am using now)
//controller
public HttpResponseMessage GetVenue(int venueId)
{
if (venueId == 0)
{
ModelState.AddModelError("badVenueId", "venue id must be greater than 0");
if (ModelState.IsValid)
{
var venue = venueService.FindVenue(venueId);
return Request.CreateResponse<ResponseResult<Venue>>(venue.Status.Code, venue);
}
// a wrapper that I made to extract the model state and try to make all my request have same layout.
var responseResult = new ResponseResultWrapper();
responseResult.Status.Code = HttpStatusCode.BadRequest;
responseResult.Status.Message = GenericErrors.InvalidRequest;
responseResult.ModelStateToResponseResult(ModelState);
return Request.CreateResponse<ResponseResult>(responseResult.Status.Code, responseResult);
}
// service layer
public ResponseResult<Venue> FindVenue(int venueId)
{
ResponseResult<Venue> responseResult = new ResponseResult<Venue>();
try
{
// I know this check was done in the controller but pretend this is some more advanced business logic validation.
if(venueId == 0)
{
// this is like Model State Error in MVC and mostly likely would with some sort of field.
responseResult.Errors.Add("badVenueId", "venue id must be greater than 0");
responseResult.Status.Code = HttpStatusCode.BadRequest;
}
var venue = context.Venues.Where(x => x.Id == venueId).FirstOrDefault();
if(venue == null)
{
var foundVenue = thirdPartyService.GetVenue(venueId);
if(foundVenue == null)
{
responseResult.Status.Code = HttpStatusCode.NotFound;
responseResult.Status.Message = "Oops could not find Venue";
return responseResult;
}
else
{
var city = cityService.FindCity(foundVenue.CityName);
if(city == null)
{
city = cityService.CreateCity(foundVenue.CityName);
if(city.Response == null)
{
responseResult.Status.Code = city.Status.Code;
responseResult.Status.Message = city.Status.Message;
return responseResult;
}
CreateVenue(VenueId, city.Response, foundVenue.Name);
responseResult.Status.Code = HttpStatusCode.Ok;
// I don't think I would return a success message here as the venue being displayed back to the user should be good enough.
responseResult.Status.Message = "";
reponseResult.Response = foundVenue;
}
}
return responseResult;
}
}
catch (SqlException ex)
{
ErrorSignal.FromCurrentContext().Raise(ex);
responseResult.Status.Code = HttpStatusCode.InternalServerError;
responseResult.Status.Message = GenericErrors.InternalError;
// maybe roll back statement here depending on the method and what it is doing.
}
// should I catch this, I know it should be if you handle it but you don't want nasty messages going back to the user.
catch (InvalidOperationException ex)
{
ErrorSignal.FromCurrentContext().Raise(ex);
responseResult.Status.Code = HttpStatusCode.InternalServerError;
responseResult.Status.Message = GenericErrors.InternalError;
}
// should I catch this, I know it should be if you handle it but you don't want nasty messages going back to the user.
catch (Exception ex)
{
ErrorSignal.FromCurrentContext().Raise(ex);
responseResult.Status.Code = HttpStatusCode.InternalServerError;
responseResult.Status.Message = GenericErrors.InternalError;
}
return responseResult;
}
// another service layer.
// it is ResponseResult<City> and not city because I could have a controller method that directly calls this method.
// but I also have a case where my other method in another service needs this as well.
public ResponseResult<City> CreateCity(string CityName)
{
ResponseResult<City> responseResult = new ResponseResult<City>();
try
{
City newCity = new City { Name = "N" };
context.Cities.Add(newCity);
context.SaveChanges();
responseResult.Status.Code = HttpStatusCode.Ok;
responseResult.Status.Message = "City was succesfully added";
}
// same catch statmens like above
catch (SqlException ex)
{
ErrorSignal.FromCurrentContext().Raise(ex);
responseResult.Status.Code = HttpStatusCode.InternalServerError;
responseResult.Status.Message = GenericErrors.InternalError;
// maybe roll back statement here depending on the method and what it is doing.
}
return responseResult;
}
As you can see the methods are all wrapped in the status codes as they could be directly called by the controller being public. FindCity() and CreateVenue() could also have this wrapping.
Option 2
public HttpResponseMessage GetVenue(int venueId)
{
try
{
if (venueId == 0)
{
ModelState.AddModelError("badVenueId", "venue id must be greater than 0");
if (ModelState.IsValid)
{
var venue = venueService.FindVenue(venueId);
return Request.CreateResponse<ResponseResult<Venue>>(HttpSatusCode.Ok, venue);
}
// a wrapper that I made to extract the model state and try to make all my request have same layout.
var responseResult = new ResponseResultWrapper();
responseResult.Status.Code = HttpStatusCode.BadRequest;
responseResult.Status.Message = GenericErrors.InvalidRequest;
responseResult.ModelStateToResponseResult(ModelState);
return Request.CreateResponse<ResponseResult>(responseResult.Status.Code, responseResult);
}
catchcatch (SqlException ex)
{
// can't remember how write this and too tried to look up.
return Request.CreateResponse(HttpStatusCode.InternalServerError;, "something here");
}
}
public Venue FindVenue(int venueId)
{
try
{
// how to pass back business logic error now without my wrapper?
if(venueId == 0)
{
// what here?
}
var venue = context.Venues.Where(x => x.Id == venueId).FirstOrDefault();
if(venue == null)
{
var foundVenue = thirdPartyService.GetVenue(venueId);
if(foundVenue == null)
{
// what here?
}
else
{
var city = cityService.FindCity(foundVenue.CityName);
if(city == null)
{
city = cityService.CreateCity(foundVenue.CityName);
if(city == null)
{
// what here?
}
CreateVenue(VenueId, city.Response, foundVenue.Name);
}
}
return venue;
}
}
catch (SqlException ex)
{
// should there be a try catch here now?
// I am guessing I am going to need to have this here if I need to do a rollback and can't do it in the controller
// throw exception here. Maybe this won't exist if no rollback is needed.
}
return null;
}
public City CreateCity(string CityName)
{
// if it crashes something I guess will catch it. Don't think I need to rollback here as only one statement being sent to database.
City newCity = new City { Name = "N" };
context.Cities.Add(newCity);
context.SaveChanges();
return newCity;
}
As you see with option 2, I might still need to wrap it in try catches for rollbacks and I am not sure how to handle advanced business validation.
Also with catching everything in the controller and sending back vanilla objects(without my wrapper) I am unsure how to do fine grain HttpStatus codes(say like notFound,Create and such)

Sorry for the brief response, but here is my general rule - if an exception occurs which you expect might happen, deal with it - either by retrying or telling the user something went wrong and giving them options to fix it.
If an unexpected exception occurs, if it's something you can deal with (e.g a timeout which you can retry) try to deal with it, otherwise get out - just think what any MS app does - e.g. office - you get an apology that something went wrong and the app ends. It's better to end gracefully than to potentially corrupt data and leave things in a real mess.

This is an article with Java-specific concepts and examples, but the broad principles here are the way to go.
Distinguish between fault exceptions, which are catastrophic and unrecoverable, and contingency exceptions, which are very much recoverable. Let the faults "bubble" to the fault barrier, where you handle appropriately. For example, you might log the error, E-mail someone or send a message to a message queue, and present the user with a nice, informative error page.
Whatever you do, be sure to preserve all the exception information from the source.
Hope that helps.

Throw an exception wherever your code determines that something has gone wrong.
You always need to handle exceptions in methods which are called directly by the end-user. This is to cater for unexpected errors which your code doesn't have specific handling for. Your generic handling code would typically log the error and may or may not include letting the user know that an unexpected error has occurred.
But if there are errors which you can expect ahead of time, you'll often want to handle these lower down in the code, nearer to the point at which they occur, so that your application can "recover" from the error and continue.

I think exceptions are useful any time you need to return details of a failure from a method, whilst being able to use the ideal return type for the method you're calling.
You said in your question:
Now for me I try to return error messages back to the the controller
and try not to really catch anything in the controller.
If the service method is supposed to ideally return a Venue object, how do you return this potential error message back to the controller? an out parameter? change the return type to something which has an error message property on it?
If you're doing either of those options, I think you're reinventing the wheel... i.e. creating a way to return exception information when one already exists.
Finally, Exceptions are strongly typed representations of what went wrong. If you return an error message, then that is fine to send back to the user, but if you need to programatically do different things based on the details of the error, then you don't want to be switching on magic string.
For example, wouldn't it be handy to differentiate between authorization errors and not found errors so you can return the most appropriate http status code to the user?
Don't forget that the Exception class has a Message property you can simply return to the user if you want to use it that way

To make sure I understand the question, your are creating a web service and want to know when to handle and when to throw exceptions.
In this situation I would strongly recommend that you catch all exceptions. "Unhandled" exceptions are very bad form. On web sites they result in displays that range from meaningless to dangerous by exposing internal information that you do no want the public to see.
If this is a good sized program I suggest that you create your own MyException class which derives from System.Exception. The purpose of this is provide a place for you to add additional information specific to your application. Here are some typical things I like to add to my MyException classes:
An ID number that will help me find the location in the code where the problem occurred.
A "LogMessage" method that logs the exception, sometimes to the Windows Event Log. Whether or not you log and to which log you write depends on what you want recorded, and the severity of the situation.
An indicator that shows the exception has been logged so the above method will not log twice even if it gets called more than once.
Anything else that might be useful given the circumstance.
I also like to put the text of the messages in an external resource file, like an XML document, and key them to the error number that you assign. This allows you to change the error text to improve clarity without having to redeploy the application.
Catch all exceptions and create a new instance of your MyException type and put the original exception into inner exception property. Below the first level of my application, I always throw one of my MyException instances rather than the original exception.
At the top level (application level), NEVER let an exception go unhandled and never throw your own exception. A better way is to return an error code and message in your data contract. That way the client application will only get what you want them to see. The only exceptions they'll need to worry about are the ones outside your scope, i.e. configuration errors or communication failures. In other words if they are able to invoke your service and the network stays connected you should give them a response they can interpret.
Hope this helps.
PS I didn't include a sample exception as I am sure a little searching will find many. Post if you want me to put up a simple sample.

Use try catch at all levels and bubble it up. Optionally, log the error in a file or database. I use text file - tab delimited. Capture at each level
1. Module Name (Use C# supplied methods to get this)
2. Method Name
3. Code Being Executed (User created - "Connecting to database")
4. Error Number
5. Error Description
6. Code Being Executed (User created - "Accessing database")
7. Error Number for the end user
8. Error Description for the end user
Additionally, I also pass a unique identifier like - Session Id in case of Web, Logged in User Id, User Name (if available)
I always have the Exception catch block. In here I set the error number as -0 and the message from the exception object as the error description. If it is SQL Server related - I capture SQL Exception. This generates an error number - I use that.
I want to extend this some more though.

How to in case of timeout to execute method again and again until it completes successfully?

I have asp.net application. All business logic in business layer.
Here is the example of the method
public void DoSomething()
{
PersonClass pc = new PersonClass();
pc.CreatePerson();
pc.AssignBasicTask();
pc.ChangePersonsStatus();
pc.CreateDefaultSettings();
}
what happens once in a while, one of the sub method can timeout, so as a result the process can be incompleted.
what I think in this case to make sure all steps completed properly is
public void DoSomething()
{
PersonClass pc = new PersonClass();
var error = null;
error = pc.CreatePerson();
if(error != timeout exception)
error = pc.AssignBasicTask();
else
return to step above
if(error != timeout exception)
error = pc.ChangePersonsStatus();
else
return to step above
if(error != timeout exception)
error = pc.CreateDefaultSettings();
else
return to step above
}
but it's just an idea, more then sure it's a proper way how to handle this.

Of course, this can be done more or less elegantly, with different options for timing out or giving up - but an easy way to achieve what you want, would be to define a retry method which keeps retrying an action until it succeeds:
public static class RetryUtility
{
public T RetryUntilSuccess<T>(Func<T> action)
{
while(true)
{
try
{
return action();
}
catch
{
// Swallowing exceptions is BAD, BAD, BAD. You should AT LEAST log it.
}
}
}
public void RetryUntilSuccess(Action action)
{
// Trick to allow a void method being passed in without duplicating the implementation.
RetryUntilSuccess(() => { action(); return true; });
}
}
Then do
RetryUtility.RetryUntilSuccess(() => pc.CreatePerson());
RetryUtility.RetryUntilSuccess(() => pc.AssignBasicTask());
RetryUtility.RetryUntilSuccess(() => pc.ChangePersonsStatus());
RetryUtility.RetryUntilSuccess(() => pc.CreateDefaultSettings());
I must urge you to think about what to do if the method keeps failing, you could be creating an infinite loop - perhaps it should give up after N retries or back off with exponentially raising retry time - you will need to define that, since we cannot know enough about your problem domain to decide that.

You have it pretty close to correct in your psuedo-code, and there a lot of ways to do this, but here is how I would do it:
PersonClass pc = new PersonClass();
while(true)
if(pc.CreatePerson())
break;
while(true)
if(pc.AssignBasicTask())
break;
This assumes that your methods return true to indicate success, false to indicate a timeoiut failure (and probably an exception for any other kind of failure). And while I didn't do it here, I would strongly recommend some sort of try counting to make sure it doesn't just loop forever and ever.

Use a TransactionScope for to make sure everything is executed as a unit. More info here: Implementing an Implicit Transaction using Transaction Scope
You should never retry a timed out operation infinitely, you may end up hanging the server or with an infinite loop or both. There should always be a threshold of how many retries is acceptable to attempt before quitting.
Sample:
using(TransactionScope scope = new TransactionScope())
{
try
{
// Your code here
// If no errors were thrown commit your transaction
scope.Complete();
}
catch
{
// Some error handling
}
}

Annoying SQL exception, probably due to some code done wrong

I started working on this "already started" project, and I'm having a really annoying error when trying to execute some interactions with SQL Server 2008:
The server failed to resume the
transaction. Desc.:
One of these errors I get in this specific method call:
The aspx.cs Call:
busProcesso openProcess = new busProcesso(pProcessoId);
try
{
if (openProcess.GetDocument() == null)
{
//Irrelevant code.
}
}
catch{ //... }
The Business class (relevant part):
public class busProcesso : IbusProcesso
{
public Processo vProcesso { get; set; }
RENDBDataContext db;
public busProcesso()
{
vProcesso = new Processo();
}
public busProcesso(decimal pProcessoId)
{
db = new RENDBDataContext();
try
{
vProcesso = db.Processos.SingleOrDefault(x => x.Id == pProcessoId);
}
catch (Exception ex)
{
throw new Exception(ex.Message, ex);
}
}
public string GetDocument()
{
try
{
string document = null;
foreach (Processo_has_Servico ps in ListaServicosProcesso())
{
if (ps.Servico.Document != null) //Get the error right at this line.
{
document = ps.Servico.Document;
}
}
return document ;
}
catch (Exception ex)
{
throw new Exception(ex.Message, ex);
}
}
public IQueryable<Processo_has_Servico> ListaServicosProcesso()
{
db = new RENDBDataContext();
try
{
return from ps in db.Processo_has_Servicos
join s in db.Servicos on ps.Servico_Id equals s.Id
where ps.Processo_Id == vProcesso.Id
select ps;
}
catch (Exception ex)
{
throw new Exception(ex.Message, ex);
}
}
}
As I said, the error occurs right at the line:
if (ps.Servico.Document != null) from the GetDocument() method.
Opening SQL Server Activity Monitor, I see there is a process for my database (.Net SqlClient Data Provider)
After some time/use (when I start to get the "server failed to resume the transaction" error), I go to the SQL Server Activity Monitor and there's around 5 or 6 more identical processes that weren't killed and (probably) should've been. When I manually kill them, the error stops for a while, until it starts again.
I'm not really good at working in OO and all, so I'm probably missing something, maybe some way to close one of these connections. Also, any help/tip about this structure will be welcome.
PS. The error doesn't happen everytime. Sometimes it runs just perfectly. Then it starts to give the error. Then it stops. Sometimes it happens just once.. pretty weird.

The code in ListaServicosProcesso is creating the context db. Then it is returning an IQueryable.
At this point no request has been sent to the database.
Then there is a for each in the code. At this point EF says "I need to get the data from the database". So it tries to get the data.
But the context db is now out of scope, so it crashes, on the first line that tries to use the data.
There are 2 ways to get around this:
return a list from ListaServicosProcesso, this will force the database call to execute
move the for each into ListaServicosProcesso
Edit
Pharabus is correct db is not out of scope. The problem is here:
db = new RENDBDataContext();
A new instance of the context is being created without the old one being disposed. Try Dispose of db at the end of ListaServicosProcesso. Even better place db in a using statement. But then the foreach must be moved inside the using statement.

Here's a couple of ideas to try.
1/ You can attach SQL server profiler to see the query that is being executed, which will allow you to copy and paste that query to see the data that is in the database. This might be help.
2/ You never check whether ps.Servico is null - you jump straight to ps.Servico.Document. If ps.Servico is null then you will get a null reference exception if you try to access any properties on that object.

I'm not sure of the exact cause of the error you're seeing (if you Google it, the references are all over the place...), but there are a few things you could improve in your code and I've found that just cleaning things up a bit often makes problems go away. Not always, but often.
I agree with the other answerers that it would help to keep better track of your DataContext(s). For example in you're creating it once in the constructor, then again in ListaServicosProcesso(). At that point vProcesso is on one DataContext and other entities will be on another, which gets messy.
I think you could simplify the whole thing a bit, for example you could combine GetDocument() and ListaServicosProcesso() like this:
public string GetDocument()
{
try
{
// Are you sure vProcesso is not null?
if (vProcesso == null)
return null;
// Only create the context if it wasn't already created,
if (db == null)
db = new RENDBDataContext();
return db.Processo_has_Servicos
.Where(ps => ps.Processo_Id == vProcesso.Id && ps.Servico.Document != null)
.Select(ps => ps.Servico.Document) // use an implicit join
.SingleOrDefault();
}
catch (Exception ex)
{
throw new Exception(ex.Message, ex);
}
}

How can I improve this exception retry scenario?

I have a web service method I am calling which is 3rd party and outside of my domain. For some reason every now and again the web service fails with a gateway timeout. Its intermittent and a call to it directly after a failed attempt can succeed.
Now I am left with a coding dilemma, I have code that should do the trick, but the code looks like amateur hour, as you'll see below.
Is this really bad code, or acceptable given the usage? If its not acceptable, how can I improve it?
Please try hard to keep a straight face while looking at it.
try
{
MDO = OperationsWebService.MessageDownload(MI);
}
catch
{
try
{
MDO = OperationsWebService.MessageDownload(MI);
}
catch
{
try
{
MDO = OperationsWebService.MessageDownload(MI);
}
catch
{
try
{
MDO = OperationsWebService.MessageDownload(MI);
}
catch
{
try
{
MDO = OperationsWebService.MessageDownload(MI);
}
catch (Exception ex)
{
// 5 retries, ok now log and deal with the error.
}
}
}
}
}

You can do it in a loop.
Exception firstEx = null;
for(int i=0; i<5; i++)
{
try
{
MDO = OperationsWebService.MessageDownload(MI);
firstEx = null;
break;
}
catch(Exception ex)
{
if (firstEx == null)
{
firstEx = ex;
}
Thread.Sleep(100 * (i + 1));
}
}
if (firstEx != null)
{
throw new Exception("WebService call failed after 5 retries.", firstEx);
}

Here's another way you might try:
// Easier to change if you decide that 5 retries isn't right for you
Exception exceptionKeeper = null;
for (int i = 0; i < MAX_RETRIES; ++i)
{
try
{
MDO = OperationsWebService.MessageDownload(MI);
break; // correct point from Joe - thanks.
}
catch (Exception ex)
{
exceptionKeeper = ex;
// 5 retries, ok now log and deal with the error.
}
}
I think it documents the intent better. It's less code as well; easier to maintain.

All of the answers so far assume that the reaction to any exception should be to retry the operation. This is a good assumption right up until it's a false assumption. You could easily be retrying an operation that is damaging your system, all because you didn't check the exception type.
You should almost never use a bare "catch", nor "catch (Exception ex). Catch a more-specific exception - one you know you can safely recover from.

Try a loop, with some kind of limit:
int retryCount = 5;
var done = false;
Exception error = null;
while (!done && retryCount > 0)
{
try
{
MDO = OperationsWebService.MessageDownload(MI);
done = true;
}
catch (Exception ex)
{
error = ex;
}
if (done)
break;
retryCount--;
}

You should use recursion (or a loop), and should only retry if you got the error you expected.
For example:
static void TryExecute<TException>(Action method, Func<TException, bool> retryFilter, int maxRetries) where TException : Exception {
try {
method();
} catch(TException ex) {
if (maxRetries > 0 && retryFilter(ex))
TryExecute(method, retryFilter, maxRetries - 1);
else
throw;
}
}
EDIT: With a loop:
static void TryExecute<TException>(Action method, Func<TException, bool> retryFilter, int maxRetries) where TException : Exception {
while (true) {
try {
method();
return;
} catch(TException ex) {
if (maxRetries > 0 && retryFilter(ex))
maxRetries--;
else
throw;
}
}
}
You can try to prevent future errors in retryFilter, perhaps by Thread.Sleep.
If the last retry fails, this will throw the last exception.

Here is some retry logic we are using. We don't do this a lot and I was going to pull it out and document it as our Retry Pattern/Standard. I had to wing it when I first wrote it so I came here to see if I was doing it correctly. Looks like I was. The version below is fully commented. See below that for an uncommented version.
#region Retry logic for SomeWebService.MyMethod
// The following code wraps SomeWebService.MyMethod in retry logic
// in an attempt to account for network failures, timeouts, etc.
// Declare the return object for SomeWebService.MyMethod outside of
// the following for{} and try{} code so that we have it afterwards.
MyMethodResult result = null;
// This logic will attempt to retry the call to SomeWebService.MyMethod
for (int retryAttempt = 1; retryAttempt <= Config.MaxRetryAttempts; retryAttempt++)
{
try
{
result = SomeWebService.MyMethod(myId);
// If we didn't get an exception, then that (most likely) means that the
// call was successful so we can break out of the retry logic.
break;
}
catch (Exception ex)
{
// Ideally we want to only catch and act on specific
// exceptions related to the failure. However, in our
// testing, we found that the exception could be any type
// (service unavailable, timeout, database failure, etc.)
// and attempting to trap every exception that was retryable
// was burdensome. It was easier to just retry everything
// regardless of the cause of the exception. YMMV. Do what is
// appropriate for your scenario.
// Need to check to see if there will be another retry attempt allowed.
if (retryAttempt < Config.MaxRetryAttempts)
{
// Log that we are re-trying
Logger.LogEvent(string.Format("Retry attempt #{0} for SomeWebService.MyMethod({1})", retryAttempt, myId);
// Put the thread to sleep. Rather than using a straight time value for each
// iteration, we are going to multiply the sleep time by how many times we
// have currently tried to call the method. This will allow for an easy way to
// cover a broader range of time without having to use higher retry counts or timeouts.
// For example, if MaxRetryAttempts = 10 and RetrySleepSeconds = 60, the coverage will
// be as follows:
// - Retry #1 - Sleep for 1 minute
// - Retry #2 - Sleep for 2 minutes (covering three minutes total)
// - Retry #10 - Sleep for 10 minutes (and will have covered almost an hour of downtime)
Thread.Sleep(retryAttempt * Config.RetrySleepSeconds * 1000);
}
else
{
// If we made it here, we have tried to call the method several
// times without any luck. Time to give up and move on.
// Moving on could either mean:
// A) Logging the exception and moving on to the next item.
Logger.LogError(string.Format("Max Retry Attempts Exceeded for SomeWebService.MyMethod({0})", MyId), ex);
// B) Throwing the exception for the program to deal with.
throw new Exception(string.Format("Max Retry Attempts Exceeded for SomeWebService.MyMethod({0})", myId), ex);
// Or both. Your code, your call.
}
}
}
#endregion
I like Samuel Neff's example of using an exception variable to see if it completely failed or not. That would have made some of the evaluations in my logic a little simpler. I could go either way. Not sure that either way has a significant advantage over the other. However, at this point in time, I'm not going to change how we do it. The important thing is to document what you are doing and why so that some idiot doesn't come through behind you and muck with everything.
Just for kicks though, to get a better idea if the code is any shorter or cleaner one way or the other, I pulled out all the comments. They came out exactly the same number of lines. I went ahead and compiled the two versions and ran them through Reflector Code Metrics and got the following:
Metric: Inside-Catch / Outside-For
CodeSize: 197 / 185
CyclomaticComplexity: 3 / 3
Instructions: 79 / 80
Locals: 6 / 7
Final exception logic inside the catch (22 lines):
MyMethodResult result = null;
for (int retryAttempt = 1; retryAttempt <= Config.MaxRetryAttempts; retryAttempt++)
{
try
{
result = SomeWebService.MyMethod(myId);
break;
}
catch (Exception ex)
{
if (retryAttempt < Config.MaxRetryAttempts)
{
Logger.LogEvent(string.Format("Retry attempt #{0} for SomeWebService.MyMethod({1})", retryAttempt, myId);
Thread.Sleep(retryAttempt * Config.RetrySleepSeconds * 1000);
}
else
{
Logger.LogError(string.Format("Max Retry Attempts Exceeded for SomeWebService.MyMethod({0})", MyId), ex);
throw new Exception(string.Format("Max Retry Attempts Exceeded for SomeWebService.MyMethod({0})", myId), ex);
}
}
}
Final exception logic after the for-loop (22 lines):
MyMethodResult result = null;
Exception retryException = null;
for (int retryAttempt = 1; retryAttempt <= Config.MaxRetryAttempts; retryAttempt++)
{
try
{
result = SomeWebService.MyMethod(myId);
retryException = null;
break;
}
catch (Exception ex)
{
retryException = ex;
Logger.LogEvent(string.Format("Retry attempt #{0} for SomeWebService.MyMethod({1})", retryAttempt, myId);
Thread.Sleep(retryAttempt * Config.RetrySleepSeconds * 1000);
}
}
if (retryException != null)
{
Logger.LogError(string.Format("Max Retry Attempts Exceeded for SomeWebService.MyMethod({0})", MyId), ex);
throw new Exception(string.Format("Max Retry Attempts Exceeded for SomeWebService.MyMethod({0})", myId), ex);
}

I'm using the following generic method for a retry scenario. I especially want to draw attention to the PreserveStackTrace method which helps to preserve the full call stack trace, because (as I learned the hard way) neither throw or throw ex yields the complete call stack trace information.
public static void RetryBeforeThrow<T>(Action action, int retries, int timeout) where T : Exception
{
int tries = 1;
do
{
try
{
action();
return;
}
catch (T ex)
{
if (retries <= 0)
{
PreserveStackTrace(ex);
throw;
}
Thread.Sleep(timeout);
}
}
while (tries++ < retries);
}
/// <summary>
/// Sets a flag on an <see cref="T:System.Exception"/> so that all the stack trace information is preserved
/// when the exception is re-thrown.
/// </summary>
/// <remarks>This is useful because "throw" removes information, such as the original stack frame.</remarks>
/// <see href="http://weblogs.asp.net/fmarguerie/archive/2008/01/02/rethrowing-exceptions-and-preserving-the-full-call-stack-trace.aspx"/>
public static void PreserveStackTrace(Exception ex)
{
MethodInfo preserveStackTrace = typeof(Exception).GetMethod("InternalPreserveStackTrace", BindingFlags.Instance | BindingFlags.NonPublic);
preserveStackTrace.Invoke(ex, null);
}

As everyone else has pointed out the correct approach is to wrap your try/catch inside some loop with a MAX_RETRY of some sort.
You might also consider adding a timeout between each loop iteration. Otherwise you're likely to burn through your retry counter before the transient issue has had a chance to resolve itself.

It seems you have the answers you need, but I thought I'd post this link, What is an Action Policy?, that I found to provide a much more elegant solution. Lokad has some rather labyrinthine implementations, but the guy's logic is pretty solid, and the end code you'd end up writing is pretty and simple.

int cnt=0;
bool cont = true;
while (cont)
{
try
{
MDO = OperationsWebService.MessageDownload(MI);
cont = false;
}
catch (Exception ex)
{
++cnt;
if (cnt == 5)
{
// 5 retries, ok now log and deal with the error.
cont = false;
}
}
}
UPDATED : Fixed code based on comments.