I am looking into an issue with some legacy code which is relaying a Http Request to another URL. It is an API Controller that reads the request and forwards it to another address asynchronously.
Very rarely, it throws a "Cannot access a disposed object" exception - full stack trace shown later in this question. It seems to be on the line where the HttpClient is calling the SendAsync method. I think it might be the ResponseHeadersRead option - I suspect it is happening when a large packet is being sent and it gets closed because it has just read the header and quit. Just thought I'd sanity check this with you all, for your thoughts. I'll change the option to be ResponseContentsRead option and see how that goes (but it can take a long time for the error to surface).
Here is the code:
using (var client = new HttpClient())
{
var request = BuildRelayHttpRequest(Request);
await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
}
private static HttpRequestMessage BuildRelayHttpRequest(HttpRequestMessage incomingRequest)
{
var forwardToUrl = new Uri(ConfigurationManager.AppSettings["ForwardFeedURL"]);
var relayRequest = new HttpRequestMessage(incomingRequest.Method, forwardToUrl);
if (incomingRequest.Method != HttpMethod.Get && incomingRequest.Content != null)
{
relayRequest.Content = incomingRequest.Content;
}
//Copies contents
relayRequest.Content = incomingRequest.Content;
return relayRequest;
}
And here is the exception:
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Web.Http.WebHost.HttpControllerHandler+LazyStreamContent'.
at System.Net.Http.HttpContent.CheckDisposed()
at System.Net.Http.HttpContent.CopyToAsync(Stream stream, TransportContext context)
at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at CHO.Web.Services.ETrains.Controllers.ETrainsApiController.<CopyandForwardFeedAsyn>d__18.MoveNext() in \Controllers\MyAPIController.cs:line 289
Note, line 289 is the "await client.SendAsync" line of code
Odds are there is an error code being set by the server.
Follow your code with a response.EnsureSuccessStatusCode(); that is wrapped in a Try Catch block:
try
{
var request = BuildRelayHttpRequest(Request);
await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
response.EnsureSuccessStatusCode();
// Handle success
}
catch (HttpRequestException)
{
// Handle failure
}
Related
I've got a dll that relies on a SOAP call to a web service, and I was provided with the WSDL file from the vendor, and after some testing, I've got code that will successfully call the service and get the response with no issue. However, when the vendor moved the service to a new server, my code starting failing.
Specifically, I get a NullRefernceException that, even though it is square in a try/catch block, ignores all of that and crashes anyway.
my call looks like this:
new ChannelFactory<GRChannel>(binding, address).Using(async(factory) =>
{
factory.Credentials.UserName.UserName = "TestUser";
factory.Credentials.UserName.Password = "1234";
var proxy = factory.CreateChannel();
proxy.Open();
var context = new OperationContext((IClientChannel)proxy);
var prevOpContext = OperationContext.Current;
OperationContext.Current = context;
try
{
var results = await proxy.ZBAPI_GOODSMVT_CREATEAsync(payload); //<-- Null happens here
System.Xml.Serialization.XmlSerializer serializer = new System.Xml.Serialization.XmlSerializer(typeof(ZBAPI_GOODSMVT_CREATEResponse));
using (StringWriter text = new StringWriter())
{
serializer.Serialize(text, results.ZBAPI_GOODSMVT_CREATEResponse);
Message = text.ToString();
}
}
catch
{
//Nothing here matters, we never make it to this point
}
});
the Stack Trace is this at the time of failure:
at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannelProxy.TaskCreator.<>c__DisplayClass1_0.<CreateGenericTask>b__0(IAsyncResult asyncResult)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at RFCDLL.RFC.<>c__DisplayClass5_0.<<RunRequest>b__0>d.MoveNext() in C:\Users\User01\source\repos\RFCDLL\RFCDLL\RFC.cs:line 46
obviously, having a null failure is fine, and expected when the service isn't running, I just want to make sure the error handling works, and I can not have the whole app crash when that happens.
I made a simple app where chunks of a file are streamed from client to server. Server-side I have handled exceptions such that a response of my own is returned to the client. When I throw an exception before reading from the stream has been completed, however, even though it gets caught and a custom response is returned, client-side I still get an unhandled RpcException with status Cancelled.
public override async Task<UploadFileResponse> UploadFile(
IAsyncStreamReader<UploadFileRequest> requestStream,
ServerCallContext context)
{
try
{
bool moveNext = await requestStream.MoveNext();
using (var stream = System.IO.File.Create($"foo.txt"))
{
while (moveNext)
{
// If something goes wrong here, before the stream has been fully read, an RpcException
// of status Cancelled is caught in the client instead of receiving an UploadFileResponse of
// type 'Failed'. Despite the fact that we catch it in the server and return a Failed response.
await stream.WriteAsync(requestStream.Current.Data.ToByteArray());
moveNext = await requestStream.MoveNext();
throw new Exception();
}
// If something goes wrong here, when the stream has been fully read, we catch it and successfully return
// a response of our own instead of an RpcException.
// throw new Exception();
}
return new UploadFileResponse()
{
StatusCode = UploadStatusCode.Ok
};
}
catch (Exception ex)
{
return new UploadFileResponse()
{
Message = ex.Message,
StatusCode = UploadStatusCode.Failed
};
}
}
Perhaps the way I approach implementing this operation is wrong. I can see why the server would return a Cancelled RPC exception because we indeed cancel the call before the stream has been fully read but I don't understand why it overrides the custom response. It might be that handling both would have to be done client-side - a failed response and a potential RPC exception.
I found some materials on the topic - Server and Client.
Apparently it is common to throw RpcExceptions whenever there should be an invalid response as also shown in the official gRPC Github repository here.
I am experimenting with a new NServiceBus project utilizing Azure Storage Queues for message transport and JSON serialization using custom message unwrapping logic seen here:
var jsonSerializer = new Newtonsoft.Json.JsonSerializer();
transportExtensions.UnwrapMessagesWith(cloudQueueMessage =>
{
using (var stream = new MemoryStream(cloudQueueMessage.AsBytes))
using (var streamReader = new StreamReader(stream))
using (var textReader = new JsonTextReader(streamReader))
{
try
{
var jObject = JObject.Load(textReader);
using (var jsonReader = jObject.CreateReader())
{
// Try deserialize to a NServiceBus envelope first
var wrapper = jsonSerializer.Deserialize<MessageWrapper>(jsonReader);
if (wrapper.MessageIntent != default)
{
// This was a envelope message
return wrapper;
}
}
// Otherwise this was an EventGrid event
using (var jsonReader = jObject.CreateReader())
{
var #event = jsonSerializer.Deserialize<EventGridEvent>(jsonReader);
var wrapper = new MessageWrapper
{
Id = #event.Id,
Headers = new Dictionary<string, string>
{
{ "NServiceBus.EnclosedMessageTypes", #event.EventType },
{ "NServiceBus.MessageIntent", "Publish" },
{ "EventGrid.topic", #event.Topic },
{ "EventGrid.subject", #event.Subject },
{ "EventGrid.eventTime", #event.EventTime.ToString("u") },
{ "EventGrid.dataVersion", #event.DataVersion },
{ "EventGrid.metadataVersion", #event.MetadataVersion },
},
Body = Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(#event.Data)),
MessageIntent = MessageIntentEnum.Publish
};
return wrapper;
}
}
catch
{
logger.Error("Message deserialization failed, sending message to error queue");
throw;
}
}
});
The custom message unwrapping logic works correctly for properly formatted JSON messages and when an improperly formatted JSON message is put into the input queue the custom message unwrapping logic will error out on the first line inside the usings where I create the jObject which is the expected behavior. However, when the custom message unwrapping logic fails the error will get caught by the logic in the MessageRetrieved class which is part of the NServiceBus.Azure.Transports.WindowsAzureStorageQueues NuGet package (v8.2.0) seen below:
public async Task<MessageWrapper> Unwrap()
{
try
{
Logger.DebugFormat("Unwrapping message with native ID: '{0}'", rawMessage.Id);
return unwrapper.Unwrap(rawMessage);
}
catch (Exception ex)
{
await errorQueue.AddMessageAsync(rawMessage).ConfigureAwait(false);
await inputQueue.DeleteMessageAsync(rawMessage).ConfigureAwait(false);
throw new SerializationException($"Failed to deserialize message envelope for message with id {rawMessage.Id}. Make sure the configured serializer is used across all endpoints or configure the message wrapper serializer for this endpoint using the `SerializeMessageWrapperWith` extension on the transport configuration. Please refer to the Azure Storage Queue Transport configuration documentation for more details.", ex);
}
}
The first line of the try catch runs correctly adding the message to the configured error queue, however, when it does that, it appears to be changing the message ID and popreceipt of the raw message as seen here:
Initial Message Values
Updated Message Values
Then when the next line runs attempting to remove the original message from the input queue it is unable to find it as according to this article https://learn.microsoft.com/en-us/rest/api/storageservices/delete-message2#remarks it requires the original message ID and pop reciept which have now changed leading to the following error being thrown:
2020-04-20 14:17:58,603 WARN : Azure Storage Queue transport failed pushing a message through pipeline
Type: Microsoft.WindowsAzure.Storage.StorageException
Message: The remote server returned an error: (404) Not Found.
Source: Microsoft.WindowsAzure.Storage
StackTrace:
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 50
at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass7.<CreateCallbackVoid>b__5(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 121
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.Transport.AzureStorageQueues.MessageRetrieved.<Unwrap>d__3.MoveNext() in C:\BuildAgent\work\3c19e2a032c05076\src\Transport\MessageRetrieved.cs:line 40
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.Transport.AzureStorageQueues.MessagePump.<InnerReceive>d__7.MoveNext() in C:\BuildAgent\work\3c19e2a032c05076\src\Transport\MessagePump.cs:line 153
TargetSite: T EndExecuteAsync[T](System.IAsyncResult)
Is this an issue with the NServiceBus package logic, or is something in my custom message unwrapping logic causing these values to change?
This is a bug. When unwrapping is failing, the message is not yet going through the processing pipeline. As a result of that, the normal recoverability is not applicable. The CloudQueueMessage needs to be "cloned" and the clone to be sent to the error queue while the original message used to remove it from the input queue. I've raised a bug issue in GitHub and you can track the process there.
I'm using Azure Service Bus Topics with the AMQP protocol in the West Europe datacenter.
This is a schematic way of how the solution implented works:
private Microsoft.Azure.ServiceBus.SubscriptionClient CreateClient() {
string serviceBusConnectionString;
strin serviceBusTopicName;
string subscriptionName;
var subscriptionClient = new Microsoft.Azure.ServiceBus.SubscriptionClient(serviceBusConnectionString, serviceBusTopicName, subscriptionName) {
PrefetchCount = 0
};
return subscriptionClient;
}
public async Task<ISubscriptionClient> CreateSubscriptionClientAsync() {
//Some logic on the subscriptionClient, caching, creating a new one if it doesn't exists, etc.
return CreateClient()
}
private async Task CallbackAsync(Message msg, CancellationToken cancellationToken) {
//Do stuff with the Message
//when you're done Completethe message
}
public async Task<string> OpenAsync(CancellationToken cancellationToken) {
subscriptionClient = await CreateSubscriptionClientAsync().ConfigureAwait(false);
var messageHandlerOptions = new MessageHandlerOptions(ExceptionReceivedHandlerAsync) {
AutoComplete = false,
MaxAutoRenewDuration = TimeSpan.FromHours(8)
};
subscriptionClient.RegisterMessageHandler(CallbackAsync, messageHandlerOptions);
return string.Empty;
}
But last night I had couple of thousand exception like this one:
Exception: "System.InvalidOperationException: The AMQP object g2b-sessionXXXXXXXX is closing. Operation 'attach' cannot be performed.
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.<OnReceiveAsync>d__86.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.<>c__DisplayClass64_0.<<ReceiveAsync>b__0>d.MoveNext()
I'm using the Microsoft.Azure.ServiceBus v3.1.0.0,
On line I didn't find anything useful about this "Operation 'attach' cannot be performed".
On the message bus exception list page I didn't find any information about this specific problem. And in the status history site there is no reference about any outage involving Service Bus.
Did anyone experience this problem before?
What is causing the exception?
Do I need to implement any retry logic? How?
Any help is apreciated, thanks.
In the following code:
[System.Diagnostics.CodeAnalysis.SuppressMessage("Await.Warning", "CS4014:Await.Warning")]
private async Task<bool> Refresh()
{
log.Info("Refreshing Token.");
Debug.WriteLine("Refreshing Token.");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(TOKEN_URL);
request.Method = "POST";
request.ContentType = "application/json; charset=UTF-8";
request.Headers.Add("Authorization", "basic " + authCode);
request.Accept = "application/json, text/javascript, */*; q=0.01";
request.ContentLength = payload.Length;
log.Debug(request.Headers["Authorization"]);
Debug.WriteLine(request.Headers["Authorization"]);
using (Stream writeStream = request.GetRequestStream())
{
await writeStream.WriteAsync(payload, 0, payload.Length);
}
lock (tokenLock)
{
Debug.WriteLine($"Write Lock enabled? {tokenLock.IsWriteLockHeld}");
tokenLock.EnterWriteLock();
Debug.WriteLine($"Write Lock enabled? {tokenLock.IsWriteLockHeld}");
}
try
{
string body;
using (HttpWebResponse response = (HttpWebResponse)await request.GetResponseAsync())
{
int numericStatusCode = (int)response.StatusCode;
Debug.WriteLine($"Response Code: {numericStatusCode}");
if (response.StatusCode != HttpStatusCode.OK)
{
log.Error($"!!!!! Request failed. Received HTTP {response.StatusCode}");
body = string.Empty;
}
else
{
string responseValue = string.Empty;
using (Stream responseStream = response.GetResponseStream())
{
if (responseStream != null)
{
using (StreamReader reader = new StreamReader(responseStream))
{
responseValue = await reader.ReadToEndAsync();
}
}
}
body = responseValue;
Debug.WriteLine($"Response Body = {body}");
log.Trace($"Response Body = {body}");
}
}
Debug.WriteLine($"Write Lock enabled? {tokenLock.IsWriteLockHeld}");
if (!string.IsNullOrEmpty(body))
{
_token = JsonConvert.DeserializeObject<AuthTokenInfo>(body, serializerSettings);
refreshUri = _token.RefreshTokenServerUri;
payload = Encoding.GetEncoding("utf-8").GetBytes(
JsonConvert.SerializeObject(new { grant_type = "refresh_token", _token.RefreshToken })
);
Debug.WriteLine($"Write Lock enabled? {tokenLock.IsWriteLockHeld}");
Debug.WriteLine($"Token Refreshed, Expires In = {_token.ExpiresIn}");
Debug.WriteLine($"Access Token = {_token.AccessToken}");
Debug.WriteLine($"New Token Refresh URI: {refreshUri}");
Debug.WriteLine($"New Refresh Token: {_token.RefreshToken}");
if (_token != null)
{
int refreshTime = 60 * 1000; // (Token.ExpiresIn.Value - (15 * 60)) * 1000;
log.Info($"Refreshing token in {refreshTime} milliseconds.");
Debug.WriteLine($"Refreshing token in {refreshTime} milliseconds.");
Task.Delay(refreshTime).ContinueWith(async (action) =>
{
log.Info("Refreshing token NOW.");
Debug.WriteLine("Refreshing token NOW.");
await Refresh();
});
Debug.WriteLine("Refresh scheduled.");
}
}
}
finally
{
lock(tokenLock)
{
Debug.WriteLine($"Write Lock enabled? {tokenLock.IsWriteLockHeld}");
tokenLock.ExitWriteLock();
Debug.WriteLine($"Write Lock enabled? {tokenLock.IsWriteLockHeld}");
}
}
return true;
}
When I execute this code, my debug output shows:
Refreshing Token.
Write Lock enabled? False
Write Lock enabled? True
Response Code: 200
Write Lock enabled? False
Write Lock enabled? False
Token Refreshed, Expires In = 3600
Refreshing token in 60000 milliseconds.
Refresh scheduled.
Write Lock enabled? False
Exception thrown: 'System.Threading.SynchronizationLockException' in System.Core.dll
Exception thrown: 'System.AggregateException' in mscorlib.dll
Exception thrown: 'System.TypeInitializationException' in InContactApi.dll
Exception thrown: 'System.TypeInitializationException' in mscorlib.dll
Exception thrown: 'System.AggregateException' in mscorlib.dll
An unhandled exception of type 'System.AggregateException' occurred in mscorlib.dll
One or more errors occurred.
Unhandled Exception: System.AggregateException: One or more errors occurred. ---> System.TypeInitializationException: The type initializer for 'InContact.Auth' threw an exception. ---> System.AggregateException: One or more errors occurred. ---> System.Threading.SynchronizationLockException: The write lock is being released without being held.
at System.Threading.ReaderWriterLockSlim.ExitWriteLock()
at InContact.AuthToken.<Refresh>d__12.MoveNext() in C:\Users\chill\source\repos\interactive_intelligence\InContactApi\InContactApi\Auth.cs:line 206
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait()
at InContact.AuthToken..ctor() in C:\Users\chill\source\repos\interactive_intelligence\InContactApi\InContactApi\Auth.cs:line 106
at InContact.Auth..cctor() in C:\Users\chill\source\repos\interactive_intelligence\InContactApi\InContactApi\Auth.cs:line 236
--- End of inner exception stack trace ---
at InContact.Auth.get_BaseURL()
at InContact.InContactApi.MakeRequestURL(String subURL, Dictionary`2 query, String callerName) in C:\Users\chill\source\repos\interactive_intelligence\InContactApi\InContactApi\InContactApi.cs:line 127
at InContact.InContactApi.<GetFolderListing>d__26.MoveNext() in C:\Users\chill\source\repos\interactive_intelligence\InContactApi\InContactApi\InContactApi.cs:line 607
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at CallLogger.ICRecordings.<DirTraverse>d__8.MoveNext() in C:\Users\chill\source\repos\interactive_intelligence\CallLogger\CallLogger\ICRecordings.cs:line 73
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at System.Threading.Tasks.Task`1.get_Result()
at CallLogger.Program.Main() in C:\Users\chill\source\repos\interactive_intelligence\CallLogger\CallLogger\Program.cs:line 32
The program '[34036] PhoneLogger.exe' has exited with code 0 (0x0).
I am not understanding how my write lock is unlocking partway through the code, with the only write unlock line I have being at the end in a finally block.
Can anyone shed some light on this for me, and/or suggest a better approach?
I am dealing with an OAuth system that has an access token that must be refreshed every hour (when I am finally done with it). I have implemented this Refresh method to accomplish the goal, and use a Task.Delay().ContinueWith() call to schedule the refresh to run automatically. I am using a ReadWriterLockSlim so that I can lock the reads from continuing while the refresh is happening. Otherwise I want them to work normally. I need them locked because once I request the new token from the server on the refresh, I can no longer use the old auth token.
Aleks Andreev, thank you.
So the solution is to not use ReadWriterLockSlim, but rather to install the Nito.AsyncEx NuGet module and then use the AsyncReadWriterLock from it. Because ReadWriterLockSlim does not work with async/await.