I feel like I didn't fully understand how Rebus works...
I have a system where an API should queue messages in a SQL table, and a Worker Service that will do the processing of the messages. I noticed that the publisher tries to route the messages to a handler and if it does not find a proper handler it raises an exception and moves the message to the error queue... I just want my API to queue the message and let the Worker Service to get the message when ready...
My API configuration is:
services.AddRebus((configure) => configure
.Logging(l => l.NLog())
.Transport(t => t.UseSqlServer(transportOptions, "QueueMsgs"))
.Options(o => o.SetNumberOfWorkers(1))
.Options(o => o.SimpleRetryStrategy(maxDeliveryAttempts: 10))
);
And I try to send messages like:
await _bus.Send(user);
If I debug the Worker Service at the same time, everything works perfectly, but my goal is to not need the Worker Service to be active to keep the messages in the queue...
What can I do?
I tried to use publish to queue the message
await _bus.Publish(user);
And adding the Subscription:
services.AddRebus((configure) => configure
.Logging(l => l.NLog())
.Transport(t => t.UseSqlServer(transportOptions, "QueueMsgs"))
.Options(o => o.SetNumberOfWorkers(1))
.Options(o => o.SimpleRetryStrategy(maxDeliveryAttempts: 10))
.Subscriptions(s => s.StoreInSqlServer(sqlConnectionString, "QueueMsgsSubs"))
//.Options(o => )
);
I'm quite lost, to be honest.
Please any help will be highly appreciated. Thank you!!
As I suspected, the fault was on me, I didn't have the correct configuration:
This is for the service configuration:
public static IServiceCollection AddRebusInjection(this IServiceCollection services, string sqlConnectionString)
{
string queueTable = "QueueMsgs";
services.AddRebus((configure) => configure
.Logging(l => l.NLog())
.Transport(t => t.UseSqlServerAsOneWayClient(sqlConnectionString))
.Options(o => o.SimpleRetryStrategy(maxDeliveryAttempts: 10, errorQueueAddress: "queueErrors"))
.Routing(r => r.TypeBased()
.Map<UserDTO>(queueTable)
.Map<StudyRoleDTO>(queueTable))
);
return services;
}
And we need to send messages with .Send:
[HttpPost, Route("RegisterAdminUser")]
public async Task<IActionResult> SetAdminUser(SetAdminUserRequest request)
{
_logger.LogTrace($"Received a SetAdminUser request: {JsonConvert.SerializeObject(request, Formatting.None)}");
var user = new UserDTO()
{
Name = request.LastName +", "+request.FirstName,
FirstName = request.FirstName,
LastName = request.LastName,
Email = request.Email,
UserGUID = request.UserGuid,
AdminType = await _userService.GetAdminTypeAsync(request.Roles)
};
await _userService.CheckUserAsync(user);
await _bus.Send(user);
return Accepted();
}
And that's it. Now my messages are nice and comfy waiting for the consumer to get them :)
Related
I am making multiple async calls to a couple of different URLS, both urls should return the same result but i would like to compare the results from both or check for certain values in the responses. i am not sure how to compare or look for specific values in the responses outside of status codes, is there an easy way to do this? also would like to take note of the response and if it was a failure i want to be able to keep track of that later in my code to not use that url again and im not sure how i would go about this
Code:
private async Task<ClientModel> getClientInfoAsync(string clientID)
{
ClientModel c = null;
try
{
var client = new HttpClient();
//Start requests for all of them
var requests = urls.Select
(
url => client.GetAsync(getURL(url, "Client", clientID))
).ToList();
//Wait for all the requests to finish
await Task.WhenAll(requests);
//Get the responses
var responses = requests.Select
(
task => task.Result
);
foreach (var r in responses)
{
// Extract the message body
var s = await r.Content.ReadAsStringAsync();
if (r.IsSuccessStatusCode)
{
c = r.Content.ReadAsAsync<ClientModel>().Result;
SetLastSuccessfulCommunicationDetails(); //after this call HERE I THINK IS WHERE I WOULD COMPARE RESPONSES AND GO FROM THERE
}
}
}
catch (Exception ex)
{
string errMsg = "Error getting the client info";
//...catch error code here...
}
return c;
}
Basically im unsure of how to deal with the responses and only return one client model (c) based on my comparison and status of the response. let me know if i need to include any further information
If I can assume that you have a method that looks like this:
private Task<ClientModel> DetermineClientModelFromResponses(IEnumerable<string> responses)
...then you can use Microsoft's Reactive Framework (aka Rx) - NuGet System.Reactive and add using System.Reactive.Linq;.
It let's you do this:
private async Task<ClientModel> GetClientInfoAsync(string clientID) =>
await DetermineClientModelFromResponses(
await Observable.Using(
() => new HttpClient(),
client =>
urls
.ToObservable()
.SelectMany(url => Observable.FromAsync(() => client.GetAsync(getURL(url, "Client", clientID))))
.Where(response => response.IsSuccessStatusCode)
.SelectMany(response => Observable.FromAsync(() => response.Content.ReadAsStringAsync()))
.ToArray()));
...or, alternatively, this:
private async Task<ClientModel> GetClientInfoAsync(string clientID) =>
await DetermineClientModelFromResponses(
await Observable.Using(
() => new HttpClient(),
client =>
(
from url in urls.ToObservable()
from response in Observable.FromAsync(() => client.GetAsync(getURL(url, "Client", clientID)))
where response.IsSuccessStatusCode
from text in Observable.FromAsync(() => response.Content.ReadAsStringAsync())
select text
).ToArray()));
If you're OK with first successful response wins, then this should work for you, but you need to ensure you have at least one succeed:
private async Task<ClientModel> GetClientInfoAsync(string clientID) =>
await Observable.Using(
() => new HttpClient(),
client =>
(
from url in urls.ToObservable()
from response in Observable.FromAsync(() => client.GetAsync(getURL(url, "Client", clientID)))
where response.IsSuccessStatusCode
from text in Observable.FromAsync(() => response.Content.ReadAsAsync<ClientModel>())
select text
).Take(1));
To make this more robust to errors you have a few strategies.
I remodelled you code to make a simple example:
async Task Main()
{
var result = (string)"no result";
try
{
result = await GetClientInfoAsync("123");
}
catch (NotImplementedException ex)
{
Console.WriteLine(ex.Message);
}
Console.WriteLine(result);
}
private List<string> urls = new List<string>() { "Hello" };
private async Task<string> GetClientInfoAsync(string clientID) =>
await Observable.Using(
() => new HttpClient(),
client =>
(
from url in urls.ToObservable()
from response in Observable.FromAsync(() => Test1(url))
from text in Observable.FromAsync(() => Test2(response))
select $"{clientID}:{text}"
)
.Concat(Observable.Return<string>(null))
.Take(1));
private Random _random = new Random();
Task<string> Test1(string url)
{
if (_random.NextDouble() > 0.3)
{
throw new NotImplementedException("Test1!");
}
return Task.Run(() => $"{url}!");
}
Task<string> Test2(string response)
{
if (_random.NextDouble() > 0.3)
{
throw new NotImplementedException("Test2!");
}
return Task.Run(() => $"{response}#");
}
This code will end GetClientInfoAsync as soon as there's an exception and it lets it bubble up to the Main method. That might not be sufficient for you.
One alternative is to add normal try/catch code to each of Test1 and Test2 to ensure they never fail.
Alternatively, you can add "try again" functionality quite easily.
private async Task<string> GetClientInfoAsync(string clientID) =>
await Observable.Using(
() => new HttpClient(),
client =>
(
from url in urls.ToObservable()
from response in Observable.Defer(() => Observable.FromAsync(() => Test1(url))).Retry(5)
from text in Observable.Defer(() => Observable.FromAsync(() => Test2(response))).Retry(5)
select $"{clientID}:{text}"
)
.Concat(Observable.Return<string>(null))
.Take(1));
Note that now Test1 and Test2 retry 5 times each.
There's still a chance that the error gets through, but that's normal coding, right?
Note that I also added .Concat(Observable.Return<string>(null)) to ensure that the query produces one value if no values comes from the query itself. The Concat waits for the main query to end before it concatenates the null result, so if the main query produces no values then null will come out.
First of all, don't use .Result if you can help it. There's a cleaner way to grab all the responses.
var responses = await Task.WhenAll(requests);
One down-side to that approach is that it'll throw an exception if any of the requests throws an exception. So if you want to intelligently handle that case you'll need to try/catch and have some special logic to deal with that.
You can continue that pattern for each asynchronous step, to deal with your results as collections:
var validResponses = responses.Where(r =>r.IsSuccessStatusCode);
var clientModels = await Task.WhenAll(
validResponses
.Select(async r => await r.Content.ReadAsAsync<ClientModel>()));
One down-side to this approach is that you end up waiting for all the requests to complete before you start reading any of the response content. So it might make more sense to put the whole process of issuing a request, getting its response, and collecting any data along the way into a single async method, and just say var responseData = await Task.WhenAll(urls.Select(ThatMethod)). That way, as soon as one request comes back you can start consuming its response.
When it comes to comparing the results, it's impossible to help without knowing what kind of comparison you're looking for. But let's just say you want the one with the highest value for some property on the model, for instance:
var lastResponseModel = clientModels
.OrderByDescending(c => c.LastSaved)
.First();
return lastResponseModel;
I have a RMQ Cluster setup on Amazon EC2. Now when I try to create a RabbitMQ host by setting the cluster members, I get the Broker Unreachable exception every time. However, if I comment that UseCluster method, things start working fine. But I am not able to enjoy full benefits of cluster in that case.
Following is the piece of code I am working with:
foreach(var server in servers)
{
try
{
var bus = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(new Uri(server), hst =>
{
hst.Username(settings.UserName);
hst.Password(settings.Password);
hst.UseCluster(c =>
{
c.ClusterMembers = servers;
});
});
//Check if the connection is reachable
host.Settings.GetConnectionFactory().CreateConnection();
registrationAction?.Invoke(cfg, host);
});
AddBusToContainer(container, bus);
return bus;
}
catch (BrokerUnreachableException ex)
{
EventLog.WriteEntry($"Error trying to connect to {server}", $"{ex}", EventLogEntryType.Error);
continue;
}
}
In the above code, if I comment the UseCluster implementation, I am able to successfully test the connection. But I receive the BrokerUnreachableException with the above code.
Does anyone have any idea about how to configure RMQ Cluster when creating a MassTransit bus? It would be very helpful.
Thanks in advance.
To configure a cluster, you should only configure a single bus instance (you're configuring multiple in your example). I don't know what version of MassTransit you're using (it looks older, given the syntax), but this represents the current approach if you're hand-coding the bus.
var bus = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
cfg.Host("cluster", hst =>
{
hst.Username(settings.UserName);
hst.Password(settings.Password);
hst.UseCluster(c =>
{
foreach(var server in servers)
c.Node(server);
});
});
registrationAction?.Invoke(cfg);
});
AddBusToContainer(container, bus);
return bus;
Though I would suggest using the newer syntax with your container, to ensure components are properly registered.
container.AddMassTransit(x =>
{
x.UsingRabbitMq(cfg =>
{
cfg.Host("cluster", hst =>
{
hst.Username(settings.UserName);
hst.Password(settings.Password);
hst.UseCluster(c =>
{
foreach(var server in servers)
c.Node(server);
});
});
registrationAction?.Invoke(cfg);
});
})
Consuming a message published with NServiceBus, using IQueueClient/Functions-ServiceBusTrigger (Microsoft.Azure.ServiceBus)
I'm working in a WebJob using .NET Core and Microsoft.Azure.ServiceBus to consume a message that has been published by a separate service using NServiceBus. My initial approach with this WebJob was to use a class Functions.cs with a method ProcessMessage that uses the attribute ServiceBusTrigger
Below is an example of how my Function.cs looks like:
public class Functions
{
public Task ProcessAuditLogMessage([ServiceBusTrigger("MyQueue")]
Message message)
{
var messageBody = Encoding.UTF8.GetString(message.Body);
var auditLogMessage = JsonConvert
.DeserializeObject<MyMessage>(messageBody);
_logger.Information("Hello World");
return Task.CompletedTask;
}
}
In Program.cs, I have:
class Program
{
static async Task Main()
{
var serviceCollection = new ServiceCollection();
serviceCollection.AddWebJobs(o => o.AllowPartialHostStartup = true);
var builder = new HostBuilder()
.UseServiceProviderFactory(
new AutofacServiceProviderFactory(serviceCollection))
.ConfigureContainer<ContainerBuilder>(b =>
{
BuildAutofacContainer();
})
.ConfigureWebJobs(b =>
{
b.AddServiceBus(o =>
{
o.ConnectionString = configProvider.AuditLogServiceBus;
});
});
var host = builder.Build();
using (host)
{
await host.RunAsync();
}
}
private static IContainer BuildAutofacContainer(IServiceColletion
serviceCollection)
{
...
}
}
I was expecting that this logic would consume the messages published in the Queue but so far the messages get moved to the Dead-letter count with a DeadLetterReason: MaxDeliveryCountExceeded and the error description Message could not be consumed after 10 delivery attempts which gives me the indication that at least there is an attempt to get these messages delivered to my consumer.
Unfortunately that's all I have in terms of error messages/logs (I'm in the process to set up some logs from my Azure Dashboard and see if I can get more detailed logs)
Has anyone come across the scenario o having to consume messages, that have been published with NServiceBus, using Microsoft.Azure.ServiceBus instead of NServiceBus (on the consumer side). Maybe I'm missing something...
I created consumer and producer using MassTransit:
var busControl = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(messageBrokerUri, h =>
{
h.Username(communicationConfiguration.MessageBrokerUsername);
h.Password(communicationConfiguration.MessageBrokerPassword);
});
cfg.ReceiveEndpoint(host, communicationConfiguration.FileResultAggregationServiceAddress, ec =>
{
ec.Consumer(serviceCollection.GetRequiredService<IFileResultAggregationConsumer>);
});
});
busControl.Start();
My question is: if the consumer consume message1 and in the middle i will shut down the consumer service then rabbitMQ will not get an ack. after 2 minutes the consumer will be alive.
What will happen to message1?
If the message will gone, how can i use Masstransit to make the message not disappear
I'm trying to listen to the error queue to process failed messages but I can't seem to get MassTransit not to set bindings on message that i want it to listen to within the configuration. The configuration is below and is using v3 of MassTransit:
var hostAddress = new Uri("rabbitmq://localhost/");
var username = "guest";
var password = "guest";
_busControl = MassTransit.Bus.Factory.CreateUsingRabbitMq(configurator =>
{
var host = configurator.Host(hostAddress, h =>
{
h.Username(username);
h.Password(password);
});
configurator.ReceiveEndpoint(host, "myqueue_error",
endpointConfigurator =>
{
endpointConfigurator.Handler<SomeMessage>(context =>
{
return Console.Out.WriteLineAsync("Woop");
});
});
});
In the above example it will set bindings up for anything that publishes SomeMessage and direct them in to the myqueue_error which I only want messages going in to this queue which has been forward from the service that are failing. Is there anyway to consume messages from a queue but tell MassTransit not to get bindings up for them?
Update - Potential Solution
It seems that I don't need to setup a ReceiveEndpoint but I can just rename the controlbus to accept the message that I care about, This will then be able to process these messages without creating exchange bindings to the messages.
Below is the altered code, not sure if this is an ideal way but it works
var hostAddress = new Uri("rabbitmq://localhost/");
var username = "guest";
var password = "guest";
_busControl = MassTransit.Bus.Factory.CreateUsingRabbitMq(configurator =>
{
configurator.Host(hostAddress, h =>
{
h.Username(username);
h.Password(password);
});
// We need to make the queue look like the error queue
configurator.BusQueueName = $"{_queue}_error";
configurator.Durable = true;
configurator.AutoDelete = false;
configurator.SetQueueArgument("x-expires", null);
});
var connectHandle = _busControl.ConnectHandler<SomeMessage>(context => Console.Out.WriteLineAsync("Woop"));
_busHandle = _busControl.Start();
_busHandle.Ready.Wait();
// Wait
// Clean up
connectHandle.Disconnect();
_busHandle.Stop
From a lot of digging around I found a better solution which I totally missed from the documentation.
It seems that we can listen to messages by subscribing consumers to listen to Fault This works perfect for what I've been trying to achieve and we can also keep the error queues in tack.
http://docs.masstransit-project.com/en/mt3/usage/exceptions.html#handling-exceptions
So the final bit of configuration that i settle with is the following:
var hostAddress = new Uri("rabbitmq://localhost/");
var username = "guest";
var password = "guest";
_busControl = MassTransit.Bus.Factory.CreateUsingRabbitMq(configurator =>
{
var host = configurator.Host(hostAddress, h =>
{
h.Username(username);
h.Password(password);
});
configurator.ReceiveEndpoint(host, "error_listener",
endpointConfigurator =>
{
endpointConfigurator.Handler<Fault<SomeMessage>>(context =>
{
return Console.Out.WriteLineAsync("Woop");
});
});
});