We have .net 6 service where we are using polly package (Polly, Version=7.0.0.0) for retrying transient api calls. We are seeing System.NullReferenceException: Object reference not set to an instance of an object error a lot. Though, the retry seems to be working, I think we are missing something from the code.
we are able to see logs for retries:
NotExtended delaying for 1000 ms, then making retry 1 {...data..}
NotExtended delaying for 1000 ms, then making retry 2 {...data..}
NotExtended delaying for 1000 ms, then making retry 3 {...data..}
Since I am new to how Polly works, not sure what exactly we are missing from the below code. any suggestions would be much helpful. Thanks in advance.
P.S.: I am pasting snippet of code where we use polly package but not the lines of code NOT related to polly implementation to avoid confusion.
Not sure if this error is occurring for few api calls or for all of them.
using Polly;
using Polly.Extensions.Http;
using Polly.Wrap;
services.AddHttpClient<IApiService, ApiService>()
.AddPolicyHandler(GetPollyRetryPolicy(builder.services));
private static AsyncPolicyWrap<HttpResponseMessage> GetPollyRetryPolicy(IServiceCollection services)
{
var asyncPolicy = HttpPolicyExtensions
.HandleTransientHttpError()
.WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(retryAttempt),
onRetryAsync: async (outcome, timespan, retryAttempt, context) =>
{
var serviceProvider = services.BuildServiceProvider();
var data = new
{
url = outcome.Result?.RequestMessage?.RequestUri?.ToString(),
method = outcome.Result?.RequestMessage?.Method?.Method,
resultContent = await outcome.Result?.Content?.ReadAsStringAsync()
};
serviceProvider.GetService<ILogger>()
?.Warn($"{outcome.Result.StatusCode} delaying for {timespan.TotalMilliseconds} ms, " +
$"then making retry {retryAttempt}", data: data, ex: outcome.Exception);
});
return Policy.TimeoutAsync(60).WrapAsync(asyncPolicy);
}
Error we are receiving is below:
System.NullReferenceException: Object reference not set to an instance of an object.
--- End of stack trace from previous location ---
at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
at Polly.AsyncPolicy`1.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
at Polly.Wrap.AsyncPolicyWrapEngine.<>c__DisplayClass2_0`1.<<ImplementationAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Polly.Timeout.AsyncTimeoutEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Func`2 timeoutProvider, TimeoutStrategy timeoutStrategy, Func`5 onTimeoutAsync, Boolean continueOnCapturedContext)
at Polly.Timeout.AsyncTimeoutEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Func`2 timeoutProvider, TimeoutStrategy timeoutStrategy, Func`5 onTimeoutAsync, Boolean continueOnCapturedContext)
at Polly.AsyncPolicy.ExecuteAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
at Polly.Wrap.AsyncPolicyWrapEngine.ImplementationAsync[TResult](Func`3 func, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext, IAsyncPolicy outerPolicy, IAsyncPolicy`1 innerPolicy)
at Polly.AsyncPolicy`1.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
As Stephen Cleary has mentioned it in his comment: Building a whole new service provider within each retry is very odd. It is not just odd, but not necessary at all.
The AddPolicyHandler has an overload which provides access to the IServiceProvider:
services.AddHttpClient<IApiService, ApiService>()
.AddPolicyHandler((provider, _) => GetPollyRetryPolicy(provider))
With this technique in our hand you can simply ask for an ILogger like this inside the onRetryAsync delegate:
var logger = provider.GetRequiredService<ILogger>();
I would also suggest to prefer the static Wrap method over the instance method:
Policy.WrapAsync(Policy.TimeoutAsync(60), asyncPolicy);
Here I have detailed the differences between the alternatives.
Please also note that the ordering matters:
Timeout outer, Retry inner: Timeout act a global, over-aching constraint
Retry outer, Timeout inner: Timeout act a local constraint, so it resets itself for each retry attempt
Related
I have an API Controller setup which return 277,000+ items:
[HttpPost]
public async IAsyncEnumerable<LocationDto> GetLocations([FromBody] LocationReportQueryDto locationReportQuery, CancellationToken token = default)
{
var result = await locationReportDataAccess.GetFilteredLocationsAsync(locationReportQuery, token);
foreach (var location in result)
{
yield return location;
}
}
and instead of streaming each location back, Asp.Net is actually buffering the response.
Client side code:
public async Task<List<ItemLocDto>> GetFilteredLocationsAsync(LocationReportQueryDto locationReportQuery, CancellationToken token = default)
{
var httpClient = httpClientFactory.CreateClient("DataAccess");
var response = await httpClient.PostAsJsonAsync("/reports/LocationReport/GetFilteredLocations", locationReportQuery, token);
response.EnsureSuccessStatusCode();
var list = await response.Content.ReadFromJsonAsync<List<ItemLocDto>>(cancellationToken: token);
if (list == null) throw new HttpRequestException("LocationReportDataAccess GetFilteredLocationsAsync HTTP Call - Response is null");
return list;
}
I'm getting this exception upon returning from this method:
blazor.server.js:1 [2022-12-18T04:51:19.544Z] Error: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Http.HttpContent.LimitMemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.MemoryStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
--- End of stack trace from previous location ---
at System.Net.Http.HttpConnection.ChunkedEncodingReadStream.CopyToAsyncCore(Stream destination, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Portal.Infrastructure.DataAccess.Reports.LocationReportDataAccess.GetFilteredLocationsAsync(LocationReportQueryDto locationReportQuery, CancellationToken token) in C:\Users\LOFT\RiderProjects\Portal\src\Libraries\Portal.Infrastructure\DataAccess\Reports\LocationReportDataAccess.cs:line 30
at Portal.Services.Reporting.LocationResultsService.GetResultsAsync(LocationInputConfig config, CancellationToken token) in C:\Users\LOFT\RiderProjects\Portal\src\Libraries\Portal.Services\Reporting\LocationResultsService.cs:line 137
at Portal.Web.Pages.Reports.LocationList.OnGenerateClick() in C:\Users\LOFT\RiderProjects\Portal\src\Presentation\Portal.Web\Pages\Reports\LocationList.razor:line 679
at Microsoft.AspNetCore.Components.ComponentBase.CallStateHasChangedOnAsyncCompletion(Task task)
at Microsoft.AspNetCore.Components.RenderTree.Renderer.GetErrorHandledTask(Task taskToHandle, ComponentState owningComponentState)
I cannot use pagination because I'm using a legacy database, so I'm trying to stream all of the data from the API endpoint.
The "result" is of type List<LocationDto,> could this be the problem, or do I need to reduce a buffer size somewhere?
Your exception stack trace shows that you're getting the OOM from within GetFilteredLocationsAsync. This is because it returns a List<ItemLocDto>, presumably reading the entire response into memory as JSON and then parsing them into ItemLocDto objects while building/extending a List<T> to contain those objects. It's the GetFilteredLocationsAsync that is buffering into memory.
So, you'll need to change something about how /reports/LocationReport/GetFilteredLocations is called. If it can stream results, then you may be able to stream from the source to GetLocations without buffering in-memory as a List<T>. Otherwise, you'll probably need some sort of pagination on both APIs.
I get an error while uploading files as StreamContent in parallel from .NET 6 / Blazor Server-Side to API via HttpClient. Here is the stripped down code and the full text of the error:
P.S. The RemoteJSRuntime that is passed to Microsoft.AspNetCore.Components.Server.Circuits.RemoteJSDataStream.ReceiveData by the framework is NULL, so an error is thrown. However, I don't understand why NULL is passed. I cannot catch it, extinguish it, somehow influence it.
P.S.S. It is not related to ApiClient, it works fine. The error only occurs on startup after the first page load, oddly enough, after reloading the page and starting again, everything works.
Source code:
https://github.com/abberdeen/BlazorServer.ParallelFileUpload
Related issue:
https://github.com/dotnet/aspnetcore/issues/38854
public void AddFilesToUploadQueue(List<FileUploadDto> files)
{
foreach (var item in files)
{
fileUploadQueue.Enqueue(item);
}
for (int i = 0; i < files.Count; i++)
{
Task t = factory.StartNew(() => UploadOne());
tasks.Add(t);
}
}
private async Task UploadOne()
{
semaphore.WaitOne();
FileUploadDto item;
if (!fileUploadQueue.TryDequeue(out item))
{
return;
}
// Make HttpRequest
await UploadFileAsync(item);
semaphore.Release();
}
blazor.server.js:1 [2021-12-06T19:07:53.123Z] Error: System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.AspNetCore.Components.Server.Circuits.RemoteJSDataStream.ReceiveData(RemoteJSRuntime runtime, Int64 streamId, Int64 chunkId, Byte[] chunk, String error)
at Microsoft.AspNetCore.Components.Rendering.RendererSynchronizationContext.<>c__11`1.<<InvokeAsync>b__11_0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Components.Server.Circuits.CircuitHost.ReceiveJSDataChunk(Int64 streamId, Int64 chunkId, Byte[] chunk, String error)
Errors in the output window:
System.TimeoutException: Did not receive any data in the allotted time.
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at System.IO.Pipelines.PipeReaderStream.ReadAsyncInternal(Memory`1 buffer, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Components.Server.Circuits.RemoteJSDataStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Components.Forms.BrowserFileStream.CopyFileDataIntoBuffer(Int64 sourceOffset, Memory`1 destination, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Components.Forms.BrowserFileStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
at System.IO.Stream.<CopyToAsync>g__Core|29_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
at System.Net.Http.StreamToStreamCopy.<CopyAsync>g__DisposeSourceAsync|1_0(Task copyTask, Stream source)
at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask)
at System.Net.Http.MultipartContent.SerializeToStreamAsyncCore(Stream stream, TransportContext context, CancellationToken cancellationToken)
at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask)
at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask)
at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at ~~~~~BlazorApp.Services.AuthHttpClientHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in ~~~~~BlazorApp\Services\AuthHttpClientHandler.cs:line 27
at System.Net.Http.Handlers.ProgressMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at ~~~~~BlazorApp.Services.FileUploadApiService.PostAndReportProgress(String url, MultipartFormDataContent formData, String guid, EventHandler`1 progressHandler, CancellationToken cancellationToken) in ~~~~~BlazorApp\Services\FileUploadApiService.cs:line 106
--- End of inner exception stack trace ---
at ~~~~~BlazorApp.Services.FileUploadApiService.PostAndReportProgress(String url, MultipartFormDataContent formData, String guid, EventHandler`1 progressHandler, CancellationToken cancellationToken) in ~~~~~BlazorApp\Services\FileUploadApiService.cs:line 122
at ~~~~~BlazorApp.Services.FileUploadApiService.ChatUploadFileAsync(StreamContent file, Int32 chatId, String guid, EventHandler`1 progressHandler, CancellationToken cancellationToken) in ~~~~~BlazorApp\Services\FileUploadApiService.cs:line 59
at ~~~~~BlazorApp.Services.FileUploadManager.UploadFileAsync(FileUploadDto item) in ~~~~~BlazorApp\Services\FileUploadManager.cs:line 171
at ~~~~~BlazorApp.Services.FileUploadManager.<StartUploadTask>b__18_0() in ~~~~~BlazorApp\Services\FileUploadManager.cs:line 108
Thanks for the sample. I was able to reproduce the System.TimeoutException.
I am unable to reproduce the error with this code (FileUploadManager changes only). Can you try this?
public async void AddFilesToUploadQueue(IReadOnlyList<IBrowserFile> files)
{
lock (fileUploadQueue)
{
foreach (var item in files)
{
fileUploadQueue.Enqueue(item);
}
}
for (int i = 0; i < files.Count; i++)
{
IBrowserFile? file = null;
lock(fileUploadQueue)
{
fileUploadQueue.TryDequeue(out file);
}
tasks.Add(UploadOneFile(file));
}
//await Task.WhenAll(tasks);
}
protected async Task UploadOneFile(IBrowserFile file)
{
await semaphoreSlim.WaitAsync();
try
{
using (var readStream = file.OpenReadStream(int.MaxValue))
{
var streamContent = fileUploadService.CreateStreamContent(readStream, file.Name, file.ContentType);
var result = await fileUploadService.UploadFileAsync(streamContent);
}
}
finally
{
semaphoreSlim.Release();
}
}
I am running a BackgroundTask in my ASP.NET Core 5 app which creates some entities inside my database. My idea was to run it in multiple tasks to speed up the generator.
public class GeneratorService : BackgroundService
{
private const int TaskCount = 8;
private uint _index;
private readonly List<Task> _tasks;
private readonly IServiceScopeFactory _serviceScopeFactory;
public GeneratorService(IServiceScopeFactory serviceScopeFactory)
{
_serviceScopeFactory = serviceScopeFactory;
_tasks = new List<Task>();
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (stoppingToken.IsCancellationRequested == false)
{
for (uint i = 0; i < TaskCount; i++)
{
var i1 = i;
var task = new Task(async () => await Generate(_index + i1, stoppingToken));
task.Start();
_tasks.Add(task);
}
Task.WaitAll(_tasks.ToArray(), stoppingToken);
_tasks.Clear();
_index += TaskCount;
await Task.Delay(10, stoppingToken);
}
}
private async Task Generate(uint index, CancellationToken stoppingToken)
{
using var scope = _serviceScopeFactory.CreateScope();
var context = scope.ServiceProvider.GetService<DataContext>();
var repository = scope.ServiceProvider.GetService<Repository>();
var generator = scope.ServiceProvider.GetService<Generator>();
// A simple return await FirstOrDefaultAsync(e => e.Number == index)
var entity = await repository.GetByNumberAsync(index);
if (entity== null) ...
generator.GenerateChildren(entity);
await context.SaveChangesAsync(stoppingToken);
Console.WriteLine($"Entity {index} generated");
}
}
The generator loop runs for about 100 - 200 entities, but then it starts throwing me exceptions:
fail: Microsoft.EntityFrameworkCore.Query[10100]
An exception occurred while iterating over the results of a query for context type 'Namespace.DataContext'.
Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
---> System.ComponentModel.Win32Exception (258): Der Wartevorgang wurde abgebrochen.
at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__169_0(Task`1 result)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
ClientConnectionId:f8508f09-1298-487c-8513-93bd4289014e
Error Number:-2,State:0,Class:11
Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
---> System.ComponentModel.Win32Exception (258): Der Wartevorgang wurde abgebrochen.
at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__169_0(Task`1 result)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
ClientConnectionId:f8508f09-1298-487c-8513-93bd4289014e
Error Number:-2,State:0,Class:11
I am registering the DataContext in an extension method:
public static IServiceCollection ConfigureDatabase(this IServiceCollection services,
IConfiguration configuration)
{
var connectionString = // I get my connection string via a json file
services.AddDbContext<DataContext>(options =>
{
options.UseSqlServer(connectionString);
options.EnableSensitiveDataLogging();
});
return services;
}
This is a SQL Server Database running on my localhost.
If I am right, the DataContext is registered as a Scoped Service. So if I am creating a scope inside the BackgroundTask, it will automatically dispose the DataContext after the Task completed.
Or is this a SQL Server related issue where it could not handle that many requests simultaneously?
I'm using rabbitmq to create the bus.
After upgrading from 6.2.2 to 6.2.3, I'm getting the AlreadyClosedException in my sagas. It seems to relate to message re-delivery as when I turn re-delivery off then these exceptions go away.
Re-delivery config
rec.UseScheduledRedelivery(r =>
{
r.Handle<Exception>(IsTransientException);
r.Ignore<Exception>(IsHarmlessException);
r.Intervals(TimeSpan.FromSeconds(FirstRedeliveryIntervalInSeconds),
TimeSpan.FromSeconds(SecondRedeliveryIntervalInSeconds),
TimeSpan.FromSeconds(ThirdRedeliveryIntervalInSeconds));
});
Error tack trace:
RabbitMQ.Client.Exceptions.AlreadyClosedException: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=406, text='PRECONDITION_FAILED - invalid expiration '-34318': {value_negative,-34318}', classId=60, methodId=40
at RabbitMQ.Client.Impl.SessionBase.Transmit(Command cmd)
at RabbitMQ.Client.Impl.ModelBase.TransmitAndEnqueue(Command cmd, IRpcContinuation k)
at RabbitMQ.Client.Impl.ModelBase.ModelRpc(MethodBase method, ContentHeaderBase header, Byte[] body)
at RabbitMQ.Client.Framing.Impl.Model._Private_ExchangeDeclare(String exchange, String type, Boolean passive, Boolean durable, Boolean autoDelete, Boolean internal, Boolean nowait, IDictionary2 arguments)
at RabbitMQ.Client.Impl.ModelBase.ExchangeDeclare(String exchange, String type, Boolean durable, Boolean autoDelete, IDictionary2 arguments)
at MassTransit.RabbitMqTransport.Contexts.RabbitMqModelContext.<>c__DisplayClass16_0.b__0()
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location where exception was thrown ---
at MassTransit.RabbitMqTransport.Pipeline.ConfigureTopologyFilter1.ConfigureTopology(ModelContext context)
at MassTransit.RabbitMqTransport.Pipeline.ConfigureTopologyFilter1.<>c__DisplayClass3_0.<-Send>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at GreenPipes.PipeExtensions.OneTimeSetup[T](PipeContext context, Func2 setupMethod, PayloadFactory1 payloadFactory)
at MassTransit.RabbitMqTransport.Pipeline.ConfigureTopologyFilter1.GreenPipes.IFilter<MassTransit.RabbitMqTransport.ModelContext>.Send(ModelContext context, IPipe1 next)
at MassTransit.RabbitMqTransport.Transport.RabbitMqMoveTransport.Move(ReceiveContext context, Action2 preSend)
at MassTransit.Pipeline.Filters.ErrorTransportFilter.GreenPipes.IFilter<MassTransit.ExceptionReceiveContext>.Send(ExceptionReceiveContext context, IPipe1 next)
at MassTransit.Pipeline.Filters.GenerateFaultFilter.GreenPipes.IFilter.Send(ExceptionReceiveContext context, IPipe1 next)
at GreenPipes.Filters.RescueFilter2.GreenPipes.IFilter.Send(TContext context, IPipe1 next)
at MassTransit.Pipeline.Filters.DeadLetterFilter.GreenPipes.IFilter<MassTransit.ReceiveContext>.Send(ReceiveContext context, IPipe1 next)
at MassTransit.Transports.ReceivePipeDispatcher.Dispatch(ReceiveContext context, ReceiveLockContext receiveLock)
at MassTransit.Transports.ReceivePipeDispatcher.Dispatch(ReceiveContext context, ReceiveLockContext receiveLock)
at MassTransit.Transports.ReceivePipeDispatcher.Dispatch(ReceiveContext context, ReceiveLockContext receiveLock)
at MassTransit.RabbitMqTransport.Pipeline.RabbitMqBasicConsumer.<>c__DisplayClass12_0.<b__0>d.MoveNext()
Also:
---> MassTransit.RabbitMqTransport.MessageNotConfirmedException: rabbitmq://mq/scheduler => The message was not confirmed: PRECONDITION_FAILED - invalid expiration '-34318': {value_negative,-34318}
at MassTransit.RabbitMqTransport.Contexts.RabbitMqModelContext.MassTransit.RabbitMqTransport.ModelContext.BasicPublishAsync(String exchange, String routingKey, Boolean mandatory, IBasicProperties basicProperties, Byte[] body, Boolean awaitAck)
at MassTransit.RabbitMqTransport.Transport.RabbitMqSendTransport.SendPipe1.Send(ModelContext modelContext)
at MassTransit.RabbitMqTransport.Transport.RabbitMqSendTransport.SendPipe1.Send(ModelContext modelContext)
at GreenPipes.Agents.PipeContextSupervisor1.GreenPipes.IPipeContextSource<TContext>.Send(IPipe1 pipe, CancellationToken cancellationToken)
at GreenPipes.Agents.PipeContextSupervisor1.GreenPipes.IPipeContextSource<TContext>.Send(IPipe1 pipe, CancellationToken cancellationToken)
at GreenPipes.Agents.PipeContextSupervisor1.GreenPipes.IPipeContextSource<TContext>.Send(IPipe1 pipe, CancellationToken cancellationToken)
at MassTransit.Scheduling.EndpointScheduleMessageProvider.ScheduleSend[T](ScheduleMessage1 message, IPipe1 pipe, CancellationToken cancellationToken)
at MassTransit.Scheduling.BaseScheduleMessageProvider.MassTransit.Scheduling.IScheduleMessageProvider.ScheduleSend[T](Uri destinationAddress, DateTime scheduledTime, Task1 message, IPipe1 pipe, CancellationToken cancellationToken)
at MassTransit.Pipeline.Filters.RedeliveryRetryFilter2.Send(TContext context, IPipe1 next)
--- End of inner exception stack trace ---
I didn't change any of my code and the method throws an LUIS.Runtime.Models.APIErrorException
_services.LuisServices[LuisConfiguration].RecognizeAsync(turnContext, cancellationToken);
I tried to use a different LUIS model and an old version of my bot that definately worked. It seems like something was changed in the BotBuilder and I can't figure out what. This is the stacktrace:
Exception caught : Microsoft.Azure.CognitiveServices.Language.LUIS.Runtime.Models.APIErrorException: Operation returned an invalid status code 'Forbidden'
at Microsoft.Azure.CognitiveServices.Language.LUIS.Runtime.Prediction.ResolveWithHttpMessagesAsync(String appId, String query,
Nullable`1 timezoneOffset, Nullable`1 verbose,
Nullable`1 staging, Nullable`1 spellCheck, String bingSpellCheckSubscriptionKey,
Nullable`1 log, Dictionary`2 customHeaders, CancellationToken cancellationToken)
at Microsoft.Azure.CognitiveServices.Language.LUIS.Runtime.PredictionExtensions.ResolveAsync(IPrediction operations, String appId, String query,
Nullable`1 timezoneOffset, Nullable`1 verbose,
Nullable`1 staging,
Nullable`1 spellCheck,
String bingSpellCheckSubscriptionKey, Nullable`1 log, CancellationToken cancellationToken)
at Microsoft.Bot.Builder.AI.Luis.LuisRecognizer
.RecognizeInternalAsync(ITurnContext turnContext, CancellationToken cancellationToken) in D:\a\1\s\libraries\Microsoft.Bot.Builder.AI.LUIS
\LuisRecognizer.cs:line 421
at Microsoft.Bot.Builder.AI.Luis.LuisRecognizer
.RecognizeAsync(ITurnContext turnContext, CancellationToken cancellationToken) in D:\a\1\s\libraries\Microsoft.Bot.Builder.AI.LUIS
\LuisRecognizer.cs:line 112
at NLPMortBot.BasicBot.OnTurnAsync(ITurnContext turnContext, CancellationToken cancellationToken) in nlpmortbot\NLPMortBot\BasicBot.cs:line 121
at Microsoft.Bot.Builder.MiddlewareSet.ReceiveActivityWithStatusAsync(ITurnContext turnContext, BotCallbackHandler callback,
CancellationToken cancellationToken) in D:\a\1\s\libraries\Microsoft.Bot.Builder\MiddlewareSet.cs:line 55
at Microsoft.Bot.Builder.BotAdapter.RunPipelineAsync(ITurnContext turnContext, BotCallbackHandler callback,
CancellationToken cancellationToken) in D:\a\1\s\libraries\Microsoft.Bot.Builder\BotAdapter.cs:line 167
Thanks in advance guys !
This can happen if you are using the Starter Key instead of an actual Subscription Key from an 'endpoint runtime resource in Azure'. More information can be found here: https://learn.microsoft.com/en-us/azure/cognitive-services/luis/luis-how-to-azure-subscription
After creating the LUIS Azure Resource Key, it can be assigned within the LUIS Portal:
More information can be found here: https://learn.microsoft.com/en-us/azure/cognitive-services/luis/get-started-portal-deploy-app#assign-the-resource-key-to-the-luis-app-in-the-luis-portal