Multiple ServiceStack applications with one RabbitMQ server - c#

I have created 2 ServiceStack applications that run as Windows services via TopShelf and make use of one RabbitMQ server. Unfortunately when I start the second application the following exception occurs:
Exception in Rabbit MQ Server: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=406, text="PRECONDITION_FAILED - cannot redeclare exchange 'mx.servicestack.topic' in vhost '/' with different type, durable, internal or autodelete value"
The startup code contains the following code:
App 1
...
var rabbitMqServer = new RabbitMqServer();
rabbitMqServer.RegisterHandler<BusMessages.CrawlRequest>(
n =>
{
var request = n.GetBody();
this.Crawl(request);
return null;
});
rabbitMqServer.Start();
...
App 2
...
var rabbitMqServer = new RabbitMqServer();
rabbitMqServer.RegisterHandler<SendMailRequest>(
message =>
{
SendMail(message.GetBody());
return null;
});
rabbitMqServer.Start();
...
The problem seems to be with the exchange named mx.servicestack.topic, which is defaulted by ServiceStack. Does anyone know a solution to circumvent this or change the Exchange name so I can use multiple (rather default) ServiceStack applications in combination with the same RabbitMQ server?
Update
As I was looking into it more deeply it seemed to be a bug in ServiceStack.RabbitMq v4.0.31 (used in App 1). In that version the default exchange mx.servicestack.topic is added as a fanout exchange type instead of a topic exchange type. App 2 was using ServiceStack.RabbitMq v4.0.40 which tries to add/use the exchange mx.servicestack.topic as a topic exchange type, as it should be. Upgrading the ServiceStack packages to version 4.0.40 for App 1 fixed this issue.
I prefer the way of segregation for different applications like Alain explains in his answer https://stackoverflow.com/a/31209330/1278669.
However, for different applications working in the same (small) customers' domain it's very doable to use the default exchanges like ServiceStack creates.
Last but not least, I found a dirty workaround to get App 2 running next to App 1 without upgrading the ServiceStack packages of App 1. That's done by doing the following:
...
QueueNames.ExchangeTopic = "mx.App2.topic";
var rabbitMqServer = new RabbitMqServer();
...

You need multiple vhosts in the RabbitMQ server to segregate your ServiceStack applications.
Instead of amqp://localhost:5672 you can use amqp://localhost:5672/vhostname when configuring your RabbitMqServer as described here:
https://github.com/ServiceStack/ServiceStack/wiki/Rabbit-MQ
In a practical deployment the RabbitMQ server wouldn't be on localhost. I'm using that above as a short step from where you currently are using the built-in default which is amqp://localhost:5672 when invoking new RabbitMqServer().
Virtual hosts need to be added on the RabbitMQ server ahead of time and users need to be created for them separately. They are effectively separate AMQP servers with shared infrastructure.
You can add vhosts with rabbitmqctl as follows
rabbitmqctl add-vhost vhostname

Related

IBMMQDotnetClient vs IBMXMSDotnetClient

I am working on a project, which needs to connect to IBM MQ using c#, and considering which NuGet package is the best one.
However, there are 2 NuGet packages IBMMQDotnetClient and IBMXMSDotnetClient and both of them are provided by the official (IBM).
After reading a few references,
Difference between nuget packages IBMMQDotnetClient and IBMXMSDotnetClient (IBM MQ support for .Net Core)
https://www.ibm.com/support/pages/xms-net-%E2%80%93-overview
It looks like that me that there are 2 main differences between these 2 API.
IBMXMSDotnetClient provides more functionalities
IBMXMSDotnetClient provides JMS-like API. If someone is using Java before, it is more easier to port the Java code to .Net
Another difference I noticed is that the way they are connecting to the MQ server (QueueManager vs Connection and Session). But it is not really a problem, just different styles as far as I can see.
IBMMQDotnetClient
MQEnvironment.Hostname = hostName;
MQEnvironment.Channel = channel;
// Queue Manager object
MQQueueManager queueManager = new MQQueueManager(queueManagerName);
int openOptions = MQC.MQOO_INPUT_AS_Q_DEF | MQC.MQOO_OUTPUT;
MQQueue system_default_local_queue = queueManager .AccessQueue(queueName, openOptions);
IBMXMSDotnetClient
XMSFactoryFactory factoryFactory;
IConnectionFactory cf;
IDestination destination;
factoryFactory = XMSFactoryFactory.GetInstance(XMSC.CT_WMQ);
cf = factoryFactory.CreateConnectionFactory();
cf.SetStringProperty(XMSC.WMQ_HOST_NAME, hostName);
cf.SetIntProperty(XMSC.WMQ_PORT, port);
cf.SetStringProperty(XMSC.WMQ_CHANNEL, channel);
cf.SetIntProperty(XMSC.WMQ_CONNECTION_MODE, XMSC.WMQ_CM_CLIENT);
// Queue Manager just as an simple string property here
cf.SetStringProperty(XMSC.WMQ_QUEUE_MANAGER, queueManagerName);
If it is the case, I cannot see any reason why I do not choose IBMXMSDotnetClient.
But the weird thing is that there are 327K downloads for IBMMQDotnetClient while there are only 160k downloads for IBMXMSDotnetClient.
Or there are some key advantages of IBMMQDotnetClient I missed?
IBMMQDotNet provides MQ native APIs in .NET language while IBMXMSDotNet provides JMS style of APIs in .NET. However there is one major difference between the two APIs: IBMXMSDotNet provides asynchronous message consumer while IBMMQDotNet does not. Asynchronous message consumption is a type of consuming messages where IBMXMSDotNet automatically calls application registered callback method whenever a message arrives in MQ queue. With IBMMQDotNet, you will need to do a synchronous Get call (with or without timeout) to receive messages.
Hope this helps
Update: 16th May 2022
One example: In case of MQDotNet, you can change selection criteria to read messages matching another criteria without reopening a queue. For example you can say "select message matching groupId x" first from the queue to read messages matching the given groupId. Then you can say "select message matching groupId y" from the same queue without reopening queue. In case of XMSDotNet, the selection createria is specified while creating the consumer (which internally opens a MQ queue). Consumer must be recreated if you want to change selection criteria to consume messages with another criteria

Filtering connections on my ASP.NET Core sever between Razor/MVC and gRPC, depending on Kestrel endpoint

In my .NET 6 ASP application I plan on exposing 3 types of network endpoints:
Razor pages-based web UI, listening either on a TCP port or Unix socket
Public gRPC API, listening on TCP or Unix socket as well
Server management gRPC API for internal use only, listening only on a Unix socket for security reasons
The public parts (Razor and public gRPC) could be used behind a reverse proxy, which is why I plan on allowing both TCP and Unix sockets (in case the proxy is on the same machine, but I can make them TCP-only). I also need to have them all in the same Host, so they can access the same services (same instances, not just services configured in similar ways), and so I can also register regular Hosted Services. Lastly, since both Microsoft and gRPC documentation say to not use Grpc.Core, I do not wish to use it, even though it would completely solve my problem.
Since it is not possible to have several WebHosts inside a single Generic Host, my best bet is now to rely on MapWhen to direct the requests depending on which endpoint received the connection. However, the only data I have for the MapWhen predicate idicates the connection origin using the IPAddress class (via HttpContext.Connection), which cannot represent a Unix socket, so I can't use it to reliably identify the connection source.
If I go the MapWhen route, the code would have the following structure:
public void KestrelSetupCallback(KestrelServerOptions options) {
// The two *Endpoint below are either IPEndpoint or UnixSocketEndpoint
// obtained from app configuration
options.Listen(razorEndpoint, o => o.Protocols = HttpProtocols.Http1AndHttp2);
options.Listen(apiEndpoint, o => o.Protocols = HttpProtocols.Http2);
options.ListenUnixSocket(managementSocketPath, o => o.Protocols = HttpProtocols.Http2);
}
public void WebappSetupCallback(IApplicationBuilder app) {
// Additional middlewares can be added here if needed by the predicates
app.MapWhen(RazorPredicate, RazorSetupCallback);
app.MapWhen(ApiPredicate, ApiSetupCallback);
app.MapWhen(ManagementPredicate, ManagementSetupCallback);
}
I have found about the ListenOptions.Use method , which I could call on each endpoint's ListenOptions to add a connection middleware that would set a Feature identifying the endpoint. I am not sure if that is possible though (if the IFeatureCollection is writable at this point), and I would like to know if there are other options.
Would this approach work, given my use case? Would I need to alter the code structure?
Aside from that, what approach could I take to implement the *Predicate methods?
Is there a better alternate method to achieve my use case? This feels very XYZ Problem to me and I'm afraid of missing a feature designed precisely for that.
There are ~3 ways to do this (there's more but lets start here):
Split the pipeline like you have above. Use MapWhen to determine what conditions should run which branch
You can use RequireHost on endpoints/controllers etc to determine which ones will get matched depending on the incoming host.
You can boot up multiple hosts in the same process and treat them like separate islands. They'll have separate config, DI, logging etc.

Self hosted GRPC server

I am attempting to transition away from WCF to move our codebase across to .NET Core. Our services are all hosted as Windows services at present so am trying to self-host the gRPC service as well (rather than building AspNetCore applications). I have successfully built a server using Grpc.Core.Server, and the client side as well with Grpc.Net.Client.GrpcChannel, see the code snippets below for reference.
Server:
var builder = ServerServiceDefinition.CreateBuilder();
// Binder is a small class ripped from the CodeFirst example
var binder = new Binder();
binder.Bind(builder, serviceType, service: serv);
var serverServiceDefinition = builder.Build();
var server = new Grpc.Core.Server
{
Services = { serverServiceDefinition },
Ports = { new ServerPort(host, port, ServerCredentials.Insecure) }
};
server.Start();
Client:
var channel = GrpcChannel.ForAddress(Uri, new GrpcChannelOptions()
{
//HttpHandler = new GrpcWebHandler(new System.Net.Http.HttpClientHandler())
});
var service = channel.CreateGrpcService<TService>();
However because our applications are still running in .Net Framework 4.8 I get the runtime exception when testing out this code:
System.PlatformNotSupportedException : gRPC requires extra configuration on .NET implementations that don't support gRPC over HTTP/2. An HTTP provider must be specified using GrpcChannelOptions.HttpHandler.The configured HTTP provider must either support HTTP/2 or be configured to use gRPC-Web. See https://aka.ms/aspnet/grpc/netstandard for details.
That leads me to add in the Grpc.Net.Client.Web.GrpcWebHandler on the client side to switch over to Grpc-web as per the link in the error.
However, I am now struggling to do the equivalent for the server to support Grpc-web. The guide here suggests to either (1) use Grpc.AspNetCore.Web or (2) use "Envoy proxy" to get the server supporting it. The problem with (1) is that I'm not using AspNetCore so I don't think this solution is appropriate, and I can't find any lightweight/easy way to do (2) in a simple C# solution.
Without the server-side support added, I get this exception:
Grpc.Core.RpcException : Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: An error occurred while sending the request. WebException: The server committed a protocol violation. Section=ResponseStatusLine", DebugException="System.Net.Http.HttpRequestException: An error occurred while sending the request.
Which I assume is obviously related to the fact the server isn't supporting the Grpc-web requests. So I am at a bit of a dead end with regards to this now. I feel I need to work out how to self-host AspNetCore servers and move to that instead of Grpc.Core.Server, which will open up option (1), but I am finding little to no evidence that is actually possible.
So I guess my main question is: Is there any way to support Grpc-web clients in a server hosted via Grpc.Core.Server?
And if the answer is no --> How can I self-host a GRPC server that will support Grpc-web clients?
As per this getting started guide I have discovered protobuf-net.Grpc.Native which appears to solve the problem I have at the moment. I also discovered I was missing a default constructor for my [DataContract], which I think was unrelated to the errors I was receiving but may have been contributing.

Dynamically set Endpoint definition in BizTalk Send Port using MQSeries adapter

Introduction
We exchange income data with an external party. Each year income tax regulations change and a new message schema has to be implemented. Altogether we now have 8 different schema versions each of which are deployed in a separate 'year income tax' application and this amount increases by 1 each year.
Because we pay our hosting company per installed application, we want to decrease the amount of applications installed.
All these applications are functionally equal, which means we validate incoming messages, and forward valid messages into a specific MQSeries queue. Each invalid message is routed to a response queue. Each application has it's own 'valid' and 'invalid' message queues.
The plan
One generic application that processes all 8(+) messages. New schemas must be deployable without application changes or downtime for previous, running 'income year tax' flows.
So far...
I can receive multiple messages on the same BizTalk receive port (MessageType XmlDocument) and am able to validate these messages dynamically in an orchestration by calling a custom receive pipeline (XML Disassembler + XML Validator). Exceptions as well as valid messages are processed as prescribed. There are no references between the Schemas and the generic application, so schemas can be deployed without need to stop running processes. So far, so good.
The orchestration has 1 receive shape, and 2 send shapes (valid, invalid).
SSO contains the values for routing the 'valid' and 'invalid' messages to their correct queue. Based on the incoming messagetype SSO is questioned for the correct 'valid' or 'invalid' queuedefinition.
The problem
I have previously dealt with dynamic FTP, FILE, WCF and SMTP ports, which all worked flawlessly after supplying the adapter with the correct Context Properties. Even MSMQ seems to have a fairly straightforward approach on dynamically setting transport properties.
However, I cannot seem to find MQSeries MQMT ContextProperties to set the queuedefinition dynamically.
Microsoft does not provide much information on this, and extensive searches on the internet hasn't provided me with anything useful (examples) either.
I tried matching IBM's docs with Microsoft's, but altogether I am now stuck.
I would suggest to use MQSC adapter for IBM MQ integration. It is part of Host Integration Server MSI. It only requires MQ client to be installed on the server Vs MQ Server for Windows installation required by MQSeries adapter.
Set the OutboundTransportLocation property in following format mqsc://{channelName}/tcp/{server{({port})/{queuemanager}/{queuename}
TransportType = MQSC
Context Properties - Schema can be found within assembly MQSeriesEx.MQSPropertySchemaEx with namespace (http://schemas.microsoft.com/BizTalk/2003/mqs-properties).
There are only few context properties you would need to set if at all required.
Channel_HeartBeat
Channel_MaxMessageLength
Channel_UserId
Channel_Password
ConnectionTimeout
If additional properties are required than use MQSeries.MQSPropertySchema context properties.
Thanks Vikas for your suggestion.
I followed your directions and found it works!
However, I found it a little more complicated than needed as it required me configuring channel names for each flow.
The solution that best suited me was the one I had in mind all along, and it was right before me. My attempts failed because I made a fatal mistake by setting the outgoing message's properties where I should have set the dynamic send port's properties.
SendPort(Microsoft.XLANGs.BaseTypes.Address)="MQS://SERVER/QMANAGER/QUEUENAME";

Configuring mq websphere 7 with .net

I am trying to connect to a remote queue using c#.
I tried many ways to connect to the remote queue but it always fails with common errors like: MQRC_CHANNEL_CONFIG_ERROR or MQRC_HOST_NOT_AVAILABLE.
What I am doing is this:
string channel = "QM_TEST.SVRCONN";
string hostname = "<serverIp>";
string queueName = "QM_TEST";
string port = 1414;
props.Add(MQC.HOST_NAME_PROPERTY, hostname);
props.Add(MQC.CHANNEL_PROPERTY, channel);
props.Add(MQC.PORT_PROPERTY, port );
props.Add(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES_MANAGED);
MQQueueManager mqQueue = new MQQueueManager(queueName, props);
I have tried changing this but all failed.
I think that my problem is the server configurations..
can you point me to a full guide to how to configure a server and connect to it with .net?
My problem is connecting to a REMOTE server using .net and not to a local server.
Thank you!
The problem was that the CCSID between the client and the server were different.
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/index.jsp?topic=%2Fcom.ibm.mq.csqzaf.doc%2Fcs12480_.htm
On the client side I had to put
Environment.SetEnvironmentVariable("MQCCSID", "437");
Thats why I got:
MQRC_CHANNEL_CONFIG_ERROR
I'm guessing the problem (or at least a problem) is here:
MQQueue mqQueue = new MQQueueManager(queueName, props);
This should be
queueManager = new MQQueueManager(queueManagerName, properties);
If you have installed the WebSphere MQ client to the default location, there are many sample programs under the following directory:
C:\Program Files (x86)\IBM\WebSphere MQ\tools\dotnet\samples\cs\base\
There are a number of sample programs there for various tasks. If you have the latest V7.1 client installed then you will see the following programs:
SimpleAsyncPut
SimpleClientAutoReconnectGet
SimpleClientAutoReconnectPut
SimpleGet
SimpleMessageProperties
SimplePublish
SimplePut
SimpleReadAhead
SimpleSharingConversation
SimpleSubscribe
SimpleXAGet
SimpleXAPut
There are also WCF and XMS samples.
If you need the client code, please see my response to another SO question here for links.
Update:
Here's the normal diagnostic process.
If the WMQ components were installed by relocating libraries or classes from somewhere else, perform an install using the full vendor-supplied client media. This includes troubleshooting utilities such as trace, dspmqver, etc. It also resolves any library or class mismatch issues.
Use the pre-compiled client programs to test the connection. The amqsputc, amqsgetc and amqsbcgc programs require the MQSERVER environment variable as described here. The Q program from SupportPac MA01 is a separate download but has the advantage of NOT requiring any environment variables, CCDT files or other dependencies.
If the sample programs fail, check the QMgr's error logs at [WMQ install]/qmgrs/[QMgr name]/errors/AMQERR01.LOG for messages. Also check for FDC files and errors in [WMQ install]/errors.
If no errors on the QMgr side, attempt the connection again while using a client-side trace as described here and here.
Most client problems are resolved through installation of the full WMQ client as supplied by IBM. (Conversely that implies most people are installing by grabbing DLL or JAR files.) If the problem persists, error log inspection on the QMgr and client side usually reveals the underlying cause. If these do not work then tracing usually diagnoses the remaining issues.
UPDATE 2:
Per the error messages posted at MQSeries.net, the channel has a security exit set. A security exit is external code that the channel calls out to when starting a channel. There is no way to know what the exit expects or does without having access to the code or docs of the exit. If the exit is written in-house, you'll need to talk to the programmer to figure out what it requires. If the exit is a commercial product then you will need to get the documentation for it.
Alternatively, alter the channel so that SCYEXIT is blank to disable the exit.
The data posted at MQSeries.net was as follows:
MQ9575: DCE Security: failed to get the user's login name.
EXPLANATION:
System call 192.168.50.55 to get the login name of the user running WebSphere
MQ client application process 5 failed with error value -1. This occurred in
security exit function create_cred. The exit will now attempt to open channel
using the DCE default login context.
ACTION:
If you wish to run using the DCE default login context take no action. If you
wish to run using the user's login name as the DCE security exit principal
examine the documentation for the operating system on which you are running MQ
clients and reconfigure the operating system as necessary to allow the
192.168.50.55 call to succeed.
Note that it states the call is failing in the security exit.

Categories

Resources