I have configurable count of Server Bus queue consumers in a single process. The code uses ReceiveAsync method of QueueClient class and it invokes QueueClient.Close on cancellation.
It works pretty well but it turned out that there is some issue with closing QueueClient - only one client ends immediately, all others hang until serverWaitTime timeout expires.
Look at the code and its output:
using System;
using System.Diagnostics;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.ServiceBus.Messaging;
public class Program
{
private static void Main()
{
CancellationTokenSource source = new CancellationTokenSource();
var cancellationToken = source.Token;
var logger = new Logger();
Task.Run(() =>
{
Task.Delay(TimeSpan.FromSeconds(10)).Wait();
source.Cancel();
logger.Log("Cancellation requested.");
});
string connectionString = "...";
string queueName = "...";
var workers = Enumerable.Range(1, 3).Select(i => new Worker(connectionString, queueName, logger));
var tasks = workers.Select(worker => Task.Run(() => worker.RunAsync(cancellationToken), cancellationToken)).ToArray();
Task.WaitAll(tasks);
logger.Log("The end.");
}
}
class Worker
{
private readonly Logger _logger;
private readonly QueueClient _queueClient;
public Worker(string connectionString, string queueName, Logger logger)
{
_logger = logger;
_queueClient = QueueClient.CreateFromConnectionString(connectionString, queueName);
}
public async Task RunAsync(CancellationToken cancellationToken)
{
_logger.Log($"Worker {GetHashCode()} started.");
using (cancellationToken.Register(() => _queueClient.Close()))
while (!cancellationToken.IsCancellationRequested)
{
try
{
var message = await _queueClient.ReceiveAsync(TimeSpan.FromSeconds(20));
_logger.Log($"Worker {GetHashCode()}: Process message {message.MessageId}...");
}
catch (OperationCanceledException ex)
{
_logger.Log($"Worker {GetHashCode()}: {ex.Message}");
}
}
_logger.Log($"Worker {GetHashCode()} finished.");
}
}
class Logger
{
private readonly Stopwatch _stopwatch;
public Logger()
{
_stopwatch = new Stopwatch();
_stopwatch.Start();
}
public void Log(string message) => Console.WriteLine($"{_stopwatch.Elapsed}: {message}");
}
Output:
00:00:00.8125644: Worker 12547953 started.
00:00:00.8127684: Worker 45653674 started.
00:00:00.8127314: Worker 59817589 started.
00:00:10.4534961: Cancellation requested.
00:00:11.4912900: Worker 45653674: The operation cannot be performed because the entity has been closed or aborted.
00:00:11.4914054: Worker 45653674 finished.
00:00:22.3242631: Worker 12547953: The operation cannot be performed because the entity has been closed or aborted.
00:00:22.3244501: Worker 12547953 finished.
00:00:22.3243945: Worker 59817589: The operation cannot be performed because the entity has been closed or aborted.
00:00:22.3252456: Worker 59817589 finished.
00:00:22.3253535: The end.
So as you can see the worker 45653674 stopped immediately but two others stopped only 10 seconds later.
I found some helpful information in this article: https://developers.de/blogs/damir_dobric/archive/2013/12/03/service-bus-undocumented-scaling-tips-amp-tricks.aspx. The issue goes away if each queue client works via its own physical connection.
So to fix the issue it's necessary to replace the following code:
_queueClient = QueueClient.CreateFromConnectionString(connectionString, queueName);
with
var factory = MessagingFactory.CreateFromConnectionString(connectionString);
_queueClient = factory.CreateQueueClient(queueName);
Related
I have a need in my asp.net webapi (framework .Net 4.7.2) to call Redis (using StackExchange.Redis) in order to delete a key in a fire and forget way and I am making some stress test.
As I am comparing the various way to have the max speed :
I have already test executing the command with the FireAndForget flag,
I have also measured a simple command to Redis by await it.
And I am now searching a way to collect a list of commands received in a window of 15ms and execute them all in one go by pipeling them.
I have first try to use a Task.Run Action to call Redis but the problem that I am observing is that under stress, the memory of my webapi keep climbing.
The memory is full of System.Threading.IThreadPoolWorkItem[] objects with the folowing code :
[HttpPost]
[Route("api/values/testpostfireforget")]
public ApiResult<int> DeleteFromBasketId([FromBody] int basketId)
{
var response = new DeleteFromBasketResponse<int>();
var cpt = Interlocked.Increment(ref counter);
Task.Run(async () => {
await db.StringSetAsync($"BASKET_TO_DELETE_{cpt}",cpt.ToString())
.ConfigureAwait(false);
});
return response;
}
So I think that under stress my api keep enqueing background task in memory and execute them one after the other as fast as it can but less than the request coming in...
So I am searching for a way to have only one long lived background thread running with the asp.net webapi, that could capture the commands to send to Redis and execute them by pipeling them.
I was thinking in runnning a background task by implementing IHostedService interface, but it seems that in this case the background task would not share any state with my current http request. So implementing a IhostedService would be handy for a scheduled background task but not in my case, or I do not know how...
Based on StackExchange.Redis documentation you can use CommandFlags.FireAndForget flag:
[HttpPost]
[Route("api/values/testpostfireforget")]
public ApiResult<int> DeleteFromBasketId([FromBody] int basketId)
{
var response = new DeleteFromBasketResponse<int>();
var cpt = Interlocked.Increment(ref counter);
db.StringSet($"BASKET_TO_DELETE_{cpt}", cpt.ToString(), flags: CommandFlags.FireAndForget);
return response;
}
Edit 1: another solution based on comment
You can use pub/sub approach. Something like this should work:
public class MessageBatcher
{
private readonly IDatabase target;
private readonly BlockingCollection<Action<IDatabaseAsync>> tasks = new();
private Task worker;
public MessageBatcher(IDatabase target) => this.target = target;
public void AddMessage(Action<IDatabaseAsync> task) => tasks.Add(task);
public IDisposable Start(int batchSize)
{
var cancellationTokenSource = new CancellationTokenSource();
worker = Task.Factory.StartNew(state =>
{
var count = 0;
var tokenSource = (CancellationTokenSource) state;
var box = new StrongBox<IBatch>(target.CreateBatch());
tokenSource.Token.Register(b => ((StrongBox<IBatch>)b).Value.Execute(), box);
foreach (var task in tasks.GetConsumingEnumerable(tokenSource.Token))
{
var batch = box.Value;
task(batch);
if (++count == batchSize)
{
batch.Execute();
box.Value = target.CreateBatch();
count = 0;
}
}
}, cancellationTokenSource, cancellationTokenSource.Token, TaskCreationOptions.LongRunning, TaskScheduler.Current);
return new Disposer(worker, cancellationTokenSource);
}
private class Disposer : IDisposable
{
private readonly Task worker;
private readonly CancellationTokenSource tokenSource;
public Disposer(Task worker, CancellationTokenSource tokenSource) => (this.worker, this.tokenSource) = (worker, tokenSource);
public void Dispose()
{
tokenSource.Cancel();
worker.Wait();
tokenSource.Dispose();
}
}
}
Usage:
private readonly MessageBatcher batcher;
ctor(MessageBatcher batcher) // ensure that passed `handler` is singleton and already already started
{
this.batcher= batcher;
}
[HttpPost]
[Route("api/values/testpostfireforget")]
public ApiResult<int> DeleteFromBasketId([FromBody] int basketId)
{
var response = new DeleteFromBasketResponse<int>();
var cpt = Interlocked.Increment(ref counter);
batcher.AddMessage(db => db.StringSetAsync($"BASKET_TO_DELETE_{cpt}", cpt.ToString(), flags: CommandFlags.FireAndForget));
return response;
}
I have a console app that has multiple BackgroundServices, each reading from the same Kafka topic using the Confluent.Kafka nuget package (v1.6.2). The topic has 3 partitions.
When the app starts, all the background services have their constructors called, however only one of the ExecuteAsync methods is ever called. If I add a Task.Delay() - the number of milliseconds doesn't seem to matter - at the start of each ExecuteAsync, everything works fine and all the background services run.
No exceptions are raised, as far as I can tell.
Does anyone have an idea of what may be happening, or where to look further?
Here's the code:
using Confluent.Kafka;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
namespace KafkaConsumer
{
class Program
{
static void Main(string[] args)
{
CreateHostBuilder(args).Build().Run();
}
private static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureServices((hostContext, services) =>
{
services.AddHostedService<ConsumerA>();
services.AddHostedService<ConsumerB>();
services.AddHostedService<ConsumerC>();
});
}
public class ConsumerA : BackgroundService
{
private readonly ILogger<ConsumerA> _logger;
private readonly IConsumer<Ignore, string> _consumer;
public ConsumerA(ILogger<ConsumerA> logger)
{
_logger = logger;
var config = new ConsumerConfig()
{
BootstrapServers = #"server:port",
GroupId = "Group1",
AutoOffsetReset = AutoOffsetReset.Earliest
};
_consumer = new ConsumerBuilder<Ignore, string>(config).Build();
_logger.LogInformation("ConsumerA constructor");
}
protected override async Task ExecuteAsync(CancellationToken cancellationToken)
{
// await Task.Delay(10);
_logger.LogInformation("ConsumerA starting");
_consumer.Subscribe(new List<string> { "topic" });
while (!cancellationToken.IsCancellationRequested)
{
_ = _consumer.Consume(cancellationToken);
}
}
}
public class ConsumerB : BackgroundService
{
private readonly ILogger<ConsumerB> _logger;
private readonly IConsumer<Ignore, string> _consumer;
public ConsumerB(ILogger<ConsumerB> logger)
{
_logger = logger;
var config = new ConsumerConfig()
{
BootstrapServers = #"server:port",
GroupId = "Group1",
AutoOffsetReset = AutoOffsetReset.Earliest
};
_consumer = new ConsumerBuilder<Ignore, string>(config).Build();
_logger.LogInformation("ConsumerB constructor");
}
protected override async Task ExecuteAsync(CancellationToken cancellationToken)
{
// await Task.Delay(10);
_logger.LogInformation("ConsumerB starting");
_consumer.Subscribe(new List<string> { "topic" });
while (!cancellationToken.IsCancellationRequested)
{
_ = _consumer.Consume(cancellationToken);
}
}
}
public class ConsumerC : BackgroundService
{
private readonly ILogger<ConsumerC> _logger;
private readonly IConsumer<Ignore, string> _consumer;
public ConsumerC(ILogger<ConsumerC> logger)
{
_logger = logger;
var config = new ConsumerConfig()
{
BootstrapServers = #"server:port",
GroupId = "Group1",
AutoOffsetReset = AutoOffsetReset.Earliest
};
_consumer = new ConsumerBuilder<Ignore, string>(config).Build();
_logger.LogInformation("ConsumerC constructor");
}
protected override async Task ExecuteAsync(CancellationToken cancellationToken)
{
// await Task.Delay(10);
_logger.LogInformation("ConsumerC starting");
_consumer.Subscribe(new List<string> { "topic" });
while (!cancellationToken.IsCancellationRequested)
{
_ = _consumer.Consume(cancellationToken);
}
}
}
}
And the output:
(with no delays):
info: KafkaConsumer.ConsumerA[0]
ConsumerA constructor
info: KafkaConsumer.ConsumerB[0]
ConsumerB constructor
info: KafkaConsumer.ConsumerC[0]
ConsumerC constructor
info: KafkaConsumer.ConsumerA[0]
ConsumerA starting
(with delays added):
info: KafkaConsumer.ConsumerA[0]
ConsumerA constructor
info: KafkaConsumer.ConsumerB[0]
ConsumerB constructor
info: KafkaConsumer.ConsumerC[0]
ConsumerC constructor
info: Microsoft.Hosting.Lifetime[0]
Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
Content root path: c:\users\..\kafkaconsumer\bin\Debug\net5.0
info: KafkaConsumer.ConsumerC[0]
ConsumerC starting
info: KafkaConsumer.ConsumerA[0]
ConsumerA starting
info: KafkaConsumer.ConsumerB[0]
ConsumerB starting
When starting up the BackgroundServices, the framework is evidently doing something like this:
var starting1 = service1.ExecuteAsync(...); //all called in sequence without awaits inbetween
var starting2 = service2.ExecuteAsync(...);
var starting3 = service3.ExecuteAsync(...);
...
//will await the startings all at once later on
Of course, when it does this in one of your services, it immediately gets trapped in a synchronous loop in which it blockingly polls the Kafka consumer. The thread of execution is never yielded back to the framework to continue calling other services.
You can get around this by doing your synchronous looping on separate threads, leaving the framework to happily go about its business:
protected Task ExecuteAsync(...)
{
return Task.Run(() => { //runs the below on a separate thread from the threadpool
_logger.LogInformation("ConsumerC starting");
_consumer.Subscribe(new List<string> { "topic" });
while (!cancellationToken.IsCancellationRequested)
{
_ = _consumer.Consume(cancellationToken);
}
});
}
When dealing with async APIs, there's a general expectation that you don't sit on your given thread, as doing so can cause problems for things above you that are expecting the thread back. When you await things, the point of execution 'stays' in your code but really the thread is given back to the caller while a continuation is queued to carry on doing your stuff at the right time (transparently, mostly).
Unfortunately as far as I know the Kafka libraries don't have APIs for playing along with this, and so they require full threads of their own.
This is because your execute method is async but you don't use await inside that to inform SynchronizationContext.
Write your executeAsync method like this:
protected override async Task ExecuteAsync(CancellationToken cancellationToken)
{
// await Task.Delay(10);
_logger.LogInformation("ConsumerC starting");
await Task.Run(() => _consumer.Subscribe(new List<string> { "topic" }));
while (!cancellationToken.IsCancellationRequested)
{
_await Task.Run(() => consumer.Consume(cancellationToken));
}
}
The below code reads the messages Asynchronously BUT I want to read one message at one time and the next message in the queue next day or after 1 week kind of use case. How can I read only one message Synchronously and keep the rest of the messages in the azure service bus queue for next time read.
enter code here
using System;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Azure.ServiceBus;
namespace ConsoleApp2
{
class Program
{
const string ServiceBusConnectionString = "xxxxxx";
const string QueueName = "xxx";
static IQueueClient queueClient;
public static async Task Main(string[] args)
{
queueClient = new QueueClient(ServiceBusConnectionString, QueueName);
RegisterOnMessageHandlerAndReceiveMessages();
Console.ReadKey();
await queueClient.CloseAsync();
}
static void RegisterOnMessageHandlerAndReceiveMessages()
{
var messageHandlerOptions = new MessageHandlerOptions(ExceptionReceivedHandler)
{
MaxConcurrentCalls = 10,
AutoComplete = false
};
queueClient.RegisterMessageHandler(ProcessMessagesAsync, messageHandlerOptions);
}
static async Task ProcessMessagesAsync(Message message, CancellationToken token)
{
Console.WriteLine($"Received message: SequenceNumber:{message.SystemProperties.SequenceNumber} Body:{Encoding.UTF8.GetString(message.Body)}");
await queueClient.CompleteAsync(message.SystemProperties.LockToken);
}
static Task ExceptionReceivedHandler(ExceptionReceivedEventArgs exceptionReceivedEventArgs)
{
Console.WriteLine($"Message handler encountered an exception {exceptionReceivedEventArgs.Exception}.");
var context = exceptionReceivedEventArgs.ExceptionReceivedContext;
Console.WriteLine("Exception context for troubleshooting:");
Console.WriteLine($"- Endpoint: {context.Endpoint}");
Console.WriteLine($"- Entity Path: {context.EntityPath}");
Console.WriteLine($"- Executing Action: {context.Action}");
return Task.CompletedTask;
}
}
}
You should be able to achieve this by using MessageReceiver class.
Take a look at this example for the implementation details:
https://github.com/Azure/azure-service-bus/blob/master/samples/DotNet/Microsoft.ServiceBus.Messaging/ReceiveLoop/Program.cs
I have a background task that runs every few seconds in my ASP.NET Core application. It returns the following error:
An exception occurred while iterating over the results of a query for context type 'ThaiLiveApi.Data.DataContext'.
System.InvalidOperationException: A second operation started on this context before a previous operation completed.
This is usually caused by different threads using the same instance of DbContext.
I would think because im using the dbcontext in scoped that this would not be possible and only use one dbcontext for that scope. Am I missing something?
Here is my worker code:
public class FacebookWorker : IHostedService
{
private readonly IServiceScopeFactory _serviceScopeFactory;
private readonly ILogger<FacebookWorker> _logger;
public FacebookWorker(IServiceScopeFactory serviceScopeFactory, ILogger<FacebookWorker> logger)
{
_serviceScopeFactory = serviceScopeFactory;
_logger = logger;
}
private readonly int JobIntervalInSecs = 5;
private Timer _timer;
public Task StartAsync(CancellationToken cancellationToken)
{
if (cancellationToken.IsCancellationRequested)
{
cancellationToken.ThrowIfCancellationRequested();
}
// Invoke the DoWork method every 5 seconds.
_timer = new Timer(callback: async o => await DoWork(o),
state: null, dueTime: TimeSpan.FromSeconds(0),
period: TimeSpan.FromSeconds(JobIntervalInSecs));
return Task.CompletedTask;
}
private async Task DoWork(object state) {
// allow only a certain number of concurrent work. In this case,
// only allow one job to run at a time.
if (State.numberOfActiveJobs < State.maxNumberOfActiveJobs) {
// Update number of running jobs in one atomic operation.
try {
Interlocked.Increment(ref State.numberOfActiveJobs);
_logger.LogInformation("Fetching comments " + DateTime.Now);
using (var scope = _serviceScopeFactory.CreateScope()) {
var facebookService = scope.ServiceProvider.GetRequiredService<IFacebookService>();
await facebookService.FetchAndHandleComments();
}
}
finally {
Interlocked.Decrement(ref State.numberOfActiveJobs);
}
}
}
}
}
I suppose I'm getting this error because of my Worker? Is it possible this is caused in another part of my code?
It happens when a process started but last process is not complete. try to use using block when call DbContext. One more thing Check your DbContext instance is it static or not.
After spending a very frustrating and unproductive day on this, I'm posting here in search of help.
I am using a third-party library that initiates a network connection in an unknown manner (I do know however it's a managed wrapper for an unmanaged lib). It lets you know about the status of the connection by invoking an event StatusChanged(status).
Since obviously invoking the network is costly and I may not need it for my Service, I inject an AsyncLazy<Connection> which is then invoked if necessary. The Service is accessed by ParallelForEachAsync which is an extension I made to process Tasks concurrently, based on this post.
If accessed sequentially, all is well. Any concurrency, even 2 parallel tasks will result in a deadlock 90% of the time. I know it's definitely related to how the third-party lib interacts with my code because a) I am not able to reproduce the effect using the same structure but without invoking it and b) the event StatusChanged(Connecting) is received fine, at which point I assume the network operation is started and I never get a callback for StatusChanged(Connected).
Here's a as-faithful-as-possible repro of the code structure which doesn't reproduce the deadlock unfortunately.
Any ideas on how to go about resolving this?
class Program
{
static void Main(string[] args)
{
AsyncContext.Run(() => MainAsync(args));
}
static async Task MainAsync(string[] args)
{
var lazy = new AsyncLazy<Connection>(() => ConnectionFactory.Create());
var service = new Service(lazy);
await Enumerable.Range(0, 100)
.ParallelForEachAsync(10, async i =>
{
await service.DoWork();
Console.WriteLine("did some work");
}, CancellationToken.None);
}
}
class ConnectionFactory
{
public static Task<Connection> Create()
{
var tcs = new TaskCompletionSource<Connection>();
var session = new Session();
session.Connected += (sender, args) =>
{
Console.WriteLine("connected");
tcs.SetResult(new Connection());
};
session.Connect();
return tcs.Task;
}
}
class Connection
{
public async Task DoSomethinElse()
{
await Task.Delay(1000);
}
}
class Session
{
public event EventHandler Connected;
public void Connect()
{
Console.WriteLine("Simulate network operation with unknown scheduling");
Task.Delay(100).Wait();
Connected(this, EventArgs.Empty);
}
}
class Service
{
private static Random r = new Random();
private readonly AsyncLazy<Connection> lazy;
public Service(AsyncLazy<Connection> lazy)
{
this.lazy = lazy;
}
public async Task DoWork()
{
Console.WriteLine("Trying to do some work, will connect");
await Task.Delay(r.Next(0, 100));
var connection = await lazy;
await connection.DoSomethinElse();
}
}
public static class AsyncExtensions
{
public static async Task<AsyncParallelLoopResult> ParallelForEachAsync<T>(
this IEnumerable<T> source,
int degreeOfParallelism,
Func<T, Task> body,
CancellationToken cancellationToken)
{
var partitions = Partitioner.Create(source).GetPartitions(degreeOfParallelism);
bool wasBroken = false;
var tasks =
from partition in partitions
select Task.Run(async () =>
{
using (partition)
{
while (partition.MoveNext())
{
if (cancellationToken.IsCancellationRequested)
{
Volatile.Write(ref wasBroken, true);
break;
}
await body(partition.Current);
}
}
});
await Task.WhenAll(tasks)
.ConfigureAwait(false);
return new AsyncParallelLoopResult(Volatile.Read(ref wasBroken));
}
}
public class AsyncParallelLoopResult
{
public bool IsCompleted { get; private set; }
internal AsyncParallelLoopResult(bool isCompleted)
{
IsCompleted = isCompleted;
}
}
EDIT
I think I understand why it's happening but not sure how to solve it. While the context is waiting for DoWork, DoWork is waiting for the lazy connection.
This ugly hack seems to solve it:
Connection WaitForConnection()
{
connectionLazy.Start();
var awaiter = connectionLazy.GetAwaiter();
while (!awaiter.IsCompleted)
Thread.Sleep(50);
return awaiter.GetResult();
}
Any more elegant solutions?
I suspect that the 3rd-party library is requiring some kind of STA pumping. This is fairly common with old-style asynchronous code.
I have a type AsyncContextThread that you can try, passing true to the constructor to enable manual STA pumping. AsyncContextThread is just like AsyncContext except it runs the context within a new thread (an STA thread in this case).
static void Main(string[] args)
{
using (var thread = new AsyncContextThread(true))
{
thread.Factory.Run(() => MainAsync(args)).Wait();
}
}
or
static void Main(string[] args)
{
AsyncContext.Run(() => async
{
using (var thread = new AsyncContextThread(true))
{
await thread.Factory.Run(() => MainAsync(args));
}
}
}
Note that AsyncContextThread will not work in all STA scenarios. I have run into issues when doing (some rather twisted) COM interop that required a true UI thread (WPF or WinForms thread); for some reason the STA pumping wasn't sufficient for those COM objects.