Have service that receives requests, generates data and saves this in files to AWS S3.
If service receives many requests can try to save up to 20 files (2 servers x 10 workers each) in parallel.
Data generated by requests and saved to S3 can be from few KB to around ~400MB
Problem is that sometimes (seems to be when service is busy/big files to save) S3 fails with the exception below:
We discussed 2 solutions:
1) Implement a retry of S3.UploadAsync() if save fails.
Not sure if will make any difference. Assume S3 already retries internally so maybe is no point to retry. If problem is that files are too big/takes to long to save this won't solve the issue maybe make it worst.
2) Increase the TransferUtilityConfig.DefaultTimeout to, lets say 10min (default is 5min).
If problem is that saving takes more than 5 minutes, this will fix the issue but exception thrown by S3 doesn't indicate is a timeout exception so maybe this will solve anything.
3) Is this an intermittent issue in AWS infrastructure? Could retry help?
Does anyone have experience/solution when this exception happens? Any other ideas?
UPDATE:
TransferUtilityConfig() does not contain DefaultTimeout if using NET 4.5. The functionality has been moved to AmazonS3Config. This offers more parameters to control the upload: Timeout, ReadWriteTimeout, MaxErrorRetry
AmazonS3Config Class
Settings are explained here
AWS Retries and Timeouts
This is the code used by the service to save:
using (var amazonS3Client = new AmazonS3Client(RegionEndpoint.GetBySystemName(_iAwsS3Settings.RegionEndpoint)))
using (var fileTransferUtility = new TransferUtility(amazonS3Client))
using (var memoryStream = new MemoryStream(data))
{
var fileTransferUtilityRequest = new TransferUtilityUploadRequest
{
BucketName = _iAwsS3Settings.BucketName,
InputStream = memoryStream,
StorageClass = S3StorageClass.ReducedRedundancy,
PartSize = 6291456, // 6 MB.
Key = fileLocation,
CannedACL = S3CannedACL.BucketOwnerFullControl
};
await fileTransferUtility.UploadAsync(fileTransferUtilityRequest, ct);
}
This is the exception given when S3 save fails:
System.AggregateException: One or more errors occurred. --->
Amazon.Runtime.AmazonServiceException: A WebException with status
SecureChannelFailure was thrown. ---> System.Net.WebException: The
request was aborted: Could not create SSL/TLS secure channel. at
System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult
asyncResult, TransportContext& context) at
System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult
asyncResult) at
System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult
iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean
requiresSynchronization) --- End of stack trace from previous
location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Amazon.Runtime.Internal.HttpHandler1.<InvokeAsync>d__91.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
Amazon.Runtime.Internal.HttpHandler1.<InvokeAsync>d__91.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Amazon.Runtime.Internal.RedirectHandler.d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Amazon.Runtime.Internal.Unmarshaller.<InvokeAsync>d__31.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Amazon.S3.Internal.AmazonS3ResponseHandler.d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__51.MoveNext()
--- End of inner exception stack trace --- at Amazon.Runtime.Internal.WebExceptionHandler.HandleException(IExecutionContext
executionContext, WebException exception) at
Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext
executionContext, Exception exception) at
Amazon.Runtime.Internal.ErrorHandler.d__51.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__91.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Amazon.Runtime.Internal.CredentialsRetriever.d__71.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__101.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
Amazon.Runtime.Internal.RetryHandler.d__10`1.MoveNext()
This is kind of an old question but I just had a very similar issue and because I was unable to find an appropriate answer on Google, I wanted to contribute my solution.
I had started implementation of an older API which had the following line:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
I was able to correct the issue by changing SecurityProtocolType.Ssl3 to SecurityProtocolType.Tls12
Further explanation:
SSL3 is not generally supported due to the Poodle vulnerability:
https://security.stackexchange.com/questions/70719/ssl3-poodle-vulnerability/70724#70724
I had not added this line of code myself and as a result, I was having great difficulty finding the source of the issue. However, while researching I had come across this post regarding a similar issue: https://github.com/aws/aws-sdk-net/issues/86
Near the bottom of this discussion, someone suggested adding the following line:
ServicePointManager.CheckCertificateRevocationList = true;
This fix was not able to correct my issue but it did lead me down the correct path. After searching for other references to the ServicePointManager, I was able to find the previously mentioned line regarding the SecurityProtocolType and correct it.
S3 SDK already implements retry logic
By default, an upload is retried 4 times
Created a console application to try to reproduce the error. Console application tried to upload 10-30 files asynchronously. Changing the values in AmazonS3Config for Timeout, ReadWriteTimeout, MaxErrorRetry produced exceptions (System.Net.WebException: The operation has timed out) but not the same we had (Could not create SSL/TLS secure channel).
We hypothesized that the problem could that the service is so busy that can not create the connection, that is why get "Could not create SSL/TLS secure channel"
My exception was:
A WebException with status ConnectFailure was thrown
For me the ServiceUrl was http instead of https and also didn't need a port number. The port number was blocked by the firewall.
Before:
http://s3ws.MyDomain.com:80
After:
https://s3ws.MyDomain.com
var s3Config = new AmazonS3Config
{
RegionEndpoint = Amazon.RegionEndpoint.EUNorth1,
ServiceURL = "https://s3ws.MyDomain.com",
ForcePathStyle = true
};
return new AmazonS3Client(accessKey, secretKey, s3Config);
Related
I Have a view which lists reservations associated with currently logged in user. Once user clicks on one of those reservations, he navigates to details view which connects to api through httpclient and gets reservation details. The code is listed below.
client.BaseAddress = new Uri(this.URL);
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token.TokenStr);
var result = await client.GetAsync($"/api/reservation/getforid?Id={id}");
client.Dispose();
if (result.StatusCode == System.Net.HttpStatusCode.OK)
{
var str = result.Content.ReadAsStringAsync().Result.Replace("\\", " ");
var model = JsonConvert.DeserializeObject<ReservationModel>(str);
return model;
}
The details page also allows for modyfing certain data of selected reservation and upon successful editing, the view is closed and the reservation data is to be reloaded again.
What happens though is Java.Net.ProtocolException is thrown on client.getAsync(). And while requests to api is made, it seems that the httpclient doesnt await for the data, but gets some false response which is unable to be translated to response object.
To sum up:
The HttpClient connects to api without any problem when the first request to show reservation detail page is made.
After modyfing reservation, and trying to reload reservation data, the exception is thrown, even though the code/data making the request is same
EDIT:
I refactored my project so that it uses shared HttpClient instance and found the line of code which is causing the problem.
private IMvxCommand submit;
public IMvxCommand Submit
{
get
{
submit = submit ?? new MvxAsyncCommand(async () =>
{
var validationResult = validateFields();
if (validationResult)
{
await _personApi.UpdatePerson(Model);
_nav.Close(this,true);
}
});
return submit;
}
}
the _personApi.UpdatePerson() method seems to be somehow messing stuff up.
After commenting it the view closes, and ReservationDetailsView which lists persons successfully fetches data. However when i uncomment that method, upon navigating to ReservationDetailViewModel the same exception gets thrown...
you can see the UpdatePerson method below:
public async Task UpdatePerson(PersonModel model)
{
var token = GetCurrentToken();
//client = new HttpClient();
// client.BaseAddress = new Uri(this.URL);
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token.TokenStr);
// client.DefaultRequestHeaders.Add("content-type","application/json");
var dynamic = new { model.Id, model.Name, model.NumberPhone, model.Pesel, model.Street, model.Email, model.WhoToInfrom, model.PostalCode, model.CityPerson };
var content = new StringContent(JsonConvert.SerializeObject(dynamic), Encoding.UTF8, "application/json");
var result = await client.PostAsync($"/api/person/update", content);
}
EDIT 2:
Still stuck on the problem, doesn't matter how many times i'll use the HttpClient it works fine until i make HttpClient call from PersonApi in EditPersonViewModel which updates person.
Any subsequent call will actually connect to my API (i have breakpoints set in, no exceptions are thrown) but the client receives response as below. Where is it coming from?
This the error which IIS logs:
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 30.07.2021 11:14:01
Event time (UTC): 30.07.2021 09:14:01
Event ID: 17452691323540639122452734bf1713
Event sequence: 19
Event occurrence: 15
Event detail code: 0
Application information:
Application domain: /LM/W3SVC/62/ROOT-4-132720999030559530
Trust level: Full
Application Virtual Path: /
Application Path: D:\www\abc\AbcHoliday\AbcHoliday\
Machine name: SERVER3
Process information:
Process ID: 11392
Process name: w3wp.exe
Account name: IIS APPPOOL\abc.syntio.pl
Exception information:
Exception type: TaskCanceledException
Exception message: A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.HttpServer.<SendAsync>d__24.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Owin.HttpMessageHandlerAdapter.<InvokeCore>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1.<Invoke>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage.<RunApp>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContext.<DoFinalWork>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.StageAsyncResult.End(IAsyncResult ar)
at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContext.EndFinalWork(IAsyncResult ar)
at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
Request information:
Request URL: http://abc.syntio.pl/api/reservation/getforid?Id=6384
Request path: /api/reservation/getforid
User host address: 10.10.0.1
User:
Is authenticated: True
Authentication Type: JWT
Thread account name: IIS APPPOOL\abc.syntio.pl
Thread information:
Thread ID: 25
Thread account name: IIS APPPOOL\abc.syntio.pl
Is impersonating: False
Stack trace: at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.HttpServer.<SendAsync>d__24.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Owin.HttpMessageHandlerAdapter.<InvokeCore>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1.<Invoke>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage.<RunApp>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContext.<DoFinalWork>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.StageAsyncResult.End(IAsyncResult ar)
at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContext.EndFinalWork(IAsyncResult ar)
at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
You're going to want a try/catch around Web API calls in the VM. You can't expect this call to succeed every single time! You can fall back on a cached response on error.
Based on the error description, it sounds like you are not able to parse the JSON correctly. You may need to provide custom JsonSerializerSettings compatible with the settings your API server is using. Newtonsoft should be throwing a JsonSerializationException with more information if the problem is due to JSON serialization/deserialization errors.
As for the Java Exceptions, Xamarin.Android by default uses AndroidClientHandler these days. There appears to be some bugs because you can catch Java exceptions (exceptions thrown from the Java.Lang.* namespace) in your .NET Standard/PCL project. That's a bad situation because the .NET Standard/PCL project doesn't know about Android.
I recommend using the following approach in a modern Xamarin.Android app using MvvmCross:
Use a single instance of HttpClient across the application. Normally the best way to do that would be to use IHttpClientFactory, but Xamarin doesn't have great support for that (yet). A good place to instantiate the HttpClient is inside your MvxAndroidSetup.
public class Setup : MvxAndroidSetup<App>
{
protected override void InitializeFirstChance()
{
base.InitializeFirstChance();
var httpClient = new HttpClient(new BasicAuthenticationDelegatingHandler(new SafeAndroidClientHandler()));
// TODO: Setup httpClient.DefaultRequestHeaders, etc.
Mvx.IoCProvider.RegisterSingleton(httpClient);
}
}
Use a SafeAndroidClientHandler to catch the Java.Lang.* exceptions being thrown by HttpClient and rethrow then wrapped in a .NET-friendly Exception. Here is my implementation. You would do something similar for the GetAsync method override.
public class SafeAndroidClientHandler : AndroidClientHandler
{
public SafeAndroidClientHandler()
{
AutomaticDecompression = System.Net.DecompressionMethods.GZip | System.Net.DecompressionMethods.Deflate;
}
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
try
{
return await base.SendAsync(request, cancellationToken).ConfigureAwait(false);
}
catch (Java.Lang.Exception ex)
{
throw new HttpRequestException(ex.Message, ex);
}
}
}
Now you can simply inject HttpClient into ViewModels, Services, etc. thanks to MvvmCross IoC. If a Java.Lang.* exception is thrown it would now be caught as a HttpRequestException in the ViewModel.
HTTPClient Class says:
// HttpClient is intended to be instantiated once per application, rather than per-use. See Remarks.
static readonly HttpClient client = new HttpClient();
Given that:
Don't call client.Dispose;
Don't create an HTTPClient on each call - create it once.
Using Microsoft.Azure.ServiceBus;
I have a session enabled queue, and I want to subscribe to both the active queue and DLQ so that when my session messages are dead lettered, I can process those messages from DLQ.
I call queueClient.RegisterSessionHandler to subscribe to the main (active queue), and queueClient.RegisterMessageHandler to subscribe to DLQ. Then every time I receive a message from DLQ, I want to complete the message (to remove from the DLQ). However, when I try to complete, I get this exception:
Encountered exception in message processing context. Action: UserCallback. Exception: System.InvalidOperationException: It is not possible for an entity that requires sessions to create a non-sessionful message receiver.
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.<DisposeMessagesAsync>d__99.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.ServiceBus.RetryPolicy.<RunOperation>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.Azure.ServiceBus.RetryPolicy.<RunOperation>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.<CompleteAsync>d__68.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Media.Stream.SubstrateReplication.Common.Azure.Messaging.Providers.BrokeredMessageBase`1.<CompleteAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Media.Stream.SubstrateReplication.Common.Azure.Messaging.Providers.ServiceBusProviderBase.<ProcessReceivedDeadLetterMessageAsync>d__17`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Media.Stream.SubstrateReplication.Common.Azure.Messaging.Providers.ServiceBusQueueProvider.<>c__DisplayClass5_0`1.<<OnMessageReceivedAsync>b__3>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.ServiceBus.MessageReceivePump.<MessageDispatchTask>d__13.MoveNext() File Name:C:\s\src\Common\Azure\Messaging\Providers\ExceptionHandler.cs Method Name:HandleExceptionAsync Line Number:35 Exception:
I found this article saying that CompleteAsync is only allowed for non-session subscription (I'm assuming the same thing applies for queue as well?). But.. why? I was able to complete messages just fine when I initially just registered my sessionHandler. Ever since I added registerMessageHandler to DLQ of the main queue, I get this error whenever I try to complete... :-(
So my question is... why am I getting this exception? What am I doing wrong?
Also another item, I think I am processing the same dead letter messages over and over, so my metrics (dead letter count) is greatly different from the one that azure portal provides. I have no idea what's happening, I don't know if same message is being added multiple times to DLQ or for some reason lock isn't working making multiple listeners process messages at the same time... has anyone experienced this?
Thank you!
I have an odd issue with my implementation of Rebus which has been working for the past couple of years without any issue, and I am trying to figure out the scope of the problem and where to focus my troubleshooting effort. A little context:
We have been running version 0.99.66
Moved to version 3.1.5 last week, then saw the issues appear
Rolled back to 0.99.66, issues continue
Using MSMQ for the transport
Running Windows Server 2016
Identical code running on other server instances without issue
So, we are experiencing seemingly random instances where messages are failing, ending up in the error queue with a Rebus error saying that the message could not be dispatched to any handlers. This could happen once, but then when an identical message type comes through the next time, it gets handled correctly.
Here is a snippet of the code in question:
public class ProcessManagerService
{
public ProcessManagerService()
{
...
BusAdapter = new BuiltinHandlerActivator();
BusAdapter.Handle<FileEventMessage>(async msg => await StartProcess(msg));
BusAdapter.Handle<ProcessRequest>(async msg => await StartProcess(msg));
Bus = Configure.With(BusAdapter)
.Logging(l => l.ColoredConsole(LogLevel.Error))
.Transport(t => t.UseMsmq(ConfigurationManager.AppSettings["Queue"]))
.Start();
}
...
public async Task StartProcess(FileEventMessage msg)
{
var svc = new StepManager() { FileEvent = msg.FileEvent };
await svc.Run();
}
public async Task StartProcess(ProcessRequest msg)
{
var svc = new StepManager();
await svc.Run(msg);
}
}
And here is an example of the exception that is thrown:
5 unhandled exceptions: 12/18/2018 7:53:00 AM -06:00:
Rebus.Exceptions.RebusApplicationException: Message with ID
c72a8b6d-e31c-4a88-937e-612bf1db8b11 and type
ClearStone.Messages.Monitoring.File.FileEventMessage,
ClearStone.Messages could not be dispatched to any handlers at
Rebus.Pipeline.Receive.DispatchIncomingMessageStep.d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at Rebus.Sagas.LoadSagaDataStep.d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Rebus.Pipeline.Receive.ActivateHandlersStep.d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Rebus.Pipeline.Receive.DeserializeIncomingMessageStep.d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Rebus.Pipeline.Receive.HandleDeferredMessagesStep.d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Rebus.Retry.Simple.SimpleRetryStrategyStep.d__8.MoveNext()
UPDATE: Here is a more detailed stack trace after wiring in Rebus source:
5 unhandled exceptions: 12/20/2018 9:39:05 AM -06:00: Rebus.Exceptions.RebusApplicationException: Message with ID 84c3605a-41de-4300-9596-97e7288d2bcb and type ClearStone.Messages.Monitoring.File.FileEventMessage, ClearStone.Messages could not be dispatched to any handlers
at Rebus.Pipeline.Receive.DispatchIncomingMessageStep.d__1.MoveNext() in C:\Temp\rebus_0_99_66_archive\Rebus\Pipeline\Receive\DispatchIncomingMessageStep.cs:line 61
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at Rebus.Sagas.LoadSagaDataStep.d__7.MoveNext() in C:\Temp\rebus_0_99_66_archive\Rebus\Sagas\LoadSagaDataStep.cs:line 77
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at Rebus.Pipeline.Receive.ActivateHandlersStep.d__3.MoveNext() in C:\Temp\rebus_0_99_66_archive\Rebus\Pipeline\Receive\ActivateHandlersStep.cs:line 48
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at Rebus.Pipeline.Receive.DeserializeIncomingMessageStep.d__2.MoveNext() in C:\Temp\rebus_0_99_66_archive\Rebus\Pipeline\Receive\DeserializeIncomingMessageStep.cs:line 36
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at Rebus.Pipeline.Receive.HandleDeferredMessagesStep.d__12.MoveNext() in C:\Temp\rebus_0_99_66_archive\Rebus\Pipeline\Receive\HandleDeferredMessagesStep.cs:line 114
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at Rebus.Retry.Simple.SimpleRetryStrategyStep.d__8.MoveNext() in C:\Temp\rebus_0_99_66_archive\Rebus\Retry\Simple\SimpleRetryStrategyStep.cs:line 105
Assuming the obvious and that it is something in this particular server instance/enviornment, I am trying to figure out why Rebus is behaving this way, and what in my environment may be causing this. Any direction as to where to start looking would be greatly appreciated!
Sounds pretty weird :) when people have this problem, it's almost always because they have somehow set up multiple Rebus instances to consume messages off of the same queue.
In some rare cases, it's because .Start() is called on the bus before handlers are added to the container/built-in handler activator, but that doesn't seem to be the problem in your case.
Can you tell me more about your setup? If it's about as simple as what you're showing above, maybe you can reproduce it in a separate application?
I've a .NET Core 2 web api that call a connected WCF service. In my development environment it works fine, but when I deploy it in production environment, with IIS and Kestrel, I get this exception:
System.AggregateException: One or more errors occurred. (There was no endpoint listening at http://192.168.100.33:3433/Test.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.) ---> System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at http://192.168.100.33:3433/Test.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: The server name or address could not be resolved
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Threading.Tasks.RendezvousAwaitable`1.GetResult()
at System.Net.Http.WinHttpHandler.<StartRequest>d__105.MoveNext()
--- End of inner exception stack trace ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.ServiceModel.Channels.ServiceModelHttpMessageHandler.<SendAsync>d__41.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at System.Net.Http.HttpClient.<FinishSendAsyncUnbuffered>d__59.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.ServiceModel.Channels.HttpChannelFactory`1.HttpClientRequestChannel.HttpClientChannelAsyncRequest.<SendRequestAsync>d__13.MoveNext()
--- End of inner exception stack trace ---
at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannelProxy.TaskCreator.<>c__DisplayClass1_0.<CreateGenericTask>b__0(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at WebApplication1.Controllers.ValuesController.Get() in c:\temp\TestWS\WebApplication1\Controllers\ValuesController.cs:line 29
---> (Inner Exception #0) System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at http://192.168.100.33:3433/Test.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: The server name or address could not be resolved
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Threading.Tasks.RendezvousAwaitable`1.GetResult()
at System.Net.Http.WinHttpHandler.<StartRequest>d__105.MoveNext()
--- End of inner exception stack trace ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.ServiceModel.Channels.ServiceModelHttpMessageHandler.<SendAsync>d__41.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at System.Net.Http.HttpClient.<FinishSendAsyncUnbuffered>d__59.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.ServiceModel.Channels.HttpChannelFactory`1.HttpClientRequestChannel.HttpClientChannelAsyncRequest.<SendRequestAsync>d__13.MoveNext()
--- End of inner exception stack trace ---
at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannelProxy.TaskCreator.<>c__DisplayClass1_0.<CreateGenericTask>b__0(IAsyncResult asyncResult)<---
The same WCF service called by classic web api on the same server works fine.
If I call it by web browser from the server it work fine too.
This is my test code:
BasicHttpBinding bindingStr = new BasicHttpBinding();
EndpointAddress endpointSessionSTR = new EndpointAddress(new Uri("http://192.168.100.33:3433/Test.svc"));
TestStrutture.StruttureClient cliStr = new TestStrutture.StruttureClient(bindingStr, endpointSessionSTR);
var retVal = cliStr.GetClassiStrutturaAsync();
retVal.Wait();
var classi = retVal.Result;
What did I forget to do?
On CMD:
netsh winhttp reset proxy
You can try to list the proxys first using.
netsh winhttp show proxy
I have not discovered why this happens, but that did the trick for me. On netcore 2.0 and 2.1.
I've been scratching my head about this one for a bit, and I was hoping someone better at parsing error messages might be able to point me in the right direction.
I have a .NET console application which utilizes the Dropbox API. The affected code snippet is printed below. When I run the executable from my machine (in a folder, co-located with the Dropbox DLL) it connects fine and performs some lookup as intended. When another user on the same network runs it on their machine (similar settings), the HttpRequestException printed below is logged after 20 seconds (consistently). Additionally, if I wrap this as a service for testing and run it on a third machine, it runs as expected under my account user/pass, but logs the same error when run under another user/pass.
Can anyone with better eyes at parsing error messages help steer me toward the solution? Or does anyone have ideas as to what I might be missing? I appreciate any help, even shots in the dark.
var _AccessToken = [value]
//DropboxCertHelper.InitializeCertPinning();
var httpClient = new HttpClient(new WebRequestHandler { ReadWriteTimeout = 10000 })
{
Timeout = TimeSpan.FromMinutes(20)
};
try
{
var config = new DropboxClientConfig("MyApplication")
{
HttpClient = httpClient
};
var client = new DropboxClient(_AccessToken, config);
ListFolderResult listing = new ListFolderResult();
listing = await client.Files.ListFolderAsync(_AppPath, true);
foreach (var item in listing.Entries.Where(i => i.IsFolder))
{
Log(item.PathDisplay);
}
}
catch (Exception e)
{
Log(e);
}
And this is the logged error:
System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.Net.WebException: Unable to connect to the remote server
---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 162.125.7.7:443
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)
at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at Dropbox.Api.DropboxRequestHandler.<RequestJsonString>d__2f.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Dropbox.Api.DropboxRequestHandler.<RequestJsonStringWithRetry>d__1a.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Dropbox.Api.DropboxRequestHandler.<Dropbox.Api.Stone.ITransport.SendRpcRequestAsync>d__5`3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at MyApplication.Program.<MyMethod>d__6.MoveNext()