I've tried my best to figure this out, but I'm guessing my inexperience with Dotnet/C# and message queues is showing through.
The problem
In a nutshell the situation I'm dealing with is, I have two projects.
The first project is a legacy Dotnet project using the Apache NMS library to consume messages from an ActiveMQ queue.
The queue is normally connected in production via SSL, but developing locally I have connected via TCP which all works fine and actively connects & dequeues as expected.
Once the messages are dequeued they are deserialised using the JsonConvert DeserializeObject method.
The second project is a NodeJS project which I've created to put messages onto the queue.
The messages are just a simple object, stringified to JSON.
I'm using the AMQP Rhea npm package to connect and send the messages, in AMQP format.
This again all works fine and the messages are enqueued as expected.
The problem is when I come to deserialising the message pulled from the queue.
There's an exception because it encounters a prefix to my JSON message - I have no idea where the prefix comes from.
I have a creeping suspicion it's because there's a clash between the protocols I'm using and that's causing the issue, but before I find myself in that particular rabbit hole - or just resorting to using a shaky method of removing the prefix - I'm hoping one of you with more experience can help point me in the right direction.
Code
Here's a couple of snippets of the offending C#:
This is how I'm dequeueing the messages:
string dequeuedMessage = string.Empty;
List<string> messages = new List<string>();
IMessage message = _consumer.Receive(waitForMessage);
if (message is ITextMessage || message is IBytesMessage)
{
dequeuedMessage = (message is ITextMessage) ? (message as ITextMessage).Text : Encoding.UTF8.GetString((message as IBytesMessage).Content);
messages.Add(dequeuedMessage);
}
else
break;
What's interesting is the message always comes through as type IBytesMessage, not what the previous author was expecting (ITextMessage), which is why I'm having that creeping suspicion.
And this is how I'm deserialising the message:
private IMessage BuildMessageObject(string message)
{
IMessage convertedMessage = JsonConvert.DeserializeObject<QueueMessage>(message);
return convertedMessage;
}
Here's a look at the prefix:
\0SpE\0SsE\0Sw\xb1\0\0\x01\xac{"Message":"Message Content Snipped for Brevity"}
It appears to always be the same, so maybe it means something to someone.
What I've Tried & Expectations
To quickly debug, I tried connecting to the queue in the Dotnet project using AMQP, but it appears that the Apache NMS package doesn't support this.
I'm not in a position to completely rewrite the connection with another package either.
I've also written something to dequeue the message in my Node app and I get the JSON back without the prefix, but this connects with the AMQP protocol which may be why?
Thanks for any help.
Edit: I should add that I'm using ActiveMQ Classic and not Artemis.
Answering my own question, it appears it is an issue with the amqp/rhea package and using the AMQP protocol as I suspected.
The problem, without running into details I can't fully explain, is that the message from the queue has been encoded and the message body includes that encoding.
The NMS library uses ActiveMQ's native OpenWire protocol.
I'm probably going to have to resolve this by moving from using the amqp/rhea package and trying to find an openwire equivalent - which is preferable than trying to find an AMQP equivalent for the Dotnet project.
Edit:
As it turns out, the rhea package is sufficient to solve the problem.
In sending the message body, I've instead created it as a Buffer instead of type Message:
const messageAsBuffer = Buffer.from(message);
sender.send(messageAsBuffer, undefined, 1);
where undefined can be a message tag and 1 can be any integer that is not 0, specifying the message is not in AMQP format.
Related
I am working on a project, which needs to connect to IBM MQ using c#, and considering which NuGet package is the best one.
However, there are 2 NuGet packages IBMMQDotnetClient and IBMXMSDotnetClient and both of them are provided by the official (IBM).
After reading a few references,
Difference between nuget packages IBMMQDotnetClient and IBMXMSDotnetClient (IBM MQ support for .Net Core)
https://www.ibm.com/support/pages/xms-net-%E2%80%93-overview
It looks like that me that there are 2 main differences between these 2 API.
IBMXMSDotnetClient provides more functionalities
IBMXMSDotnetClient provides JMS-like API. If someone is using Java before, it is more easier to port the Java code to .Net
Another difference I noticed is that the way they are connecting to the MQ server (QueueManager vs Connection and Session). But it is not really a problem, just different styles as far as I can see.
IBMMQDotnetClient
MQEnvironment.Hostname = hostName;
MQEnvironment.Channel = channel;
// Queue Manager object
MQQueueManager queueManager = new MQQueueManager(queueManagerName);
int openOptions = MQC.MQOO_INPUT_AS_Q_DEF | MQC.MQOO_OUTPUT;
MQQueue system_default_local_queue = queueManager .AccessQueue(queueName, openOptions);
IBMXMSDotnetClient
XMSFactoryFactory factoryFactory;
IConnectionFactory cf;
IDestination destination;
factoryFactory = XMSFactoryFactory.GetInstance(XMSC.CT_WMQ);
cf = factoryFactory.CreateConnectionFactory();
cf.SetStringProperty(XMSC.WMQ_HOST_NAME, hostName);
cf.SetIntProperty(XMSC.WMQ_PORT, port);
cf.SetStringProperty(XMSC.WMQ_CHANNEL, channel);
cf.SetIntProperty(XMSC.WMQ_CONNECTION_MODE, XMSC.WMQ_CM_CLIENT);
// Queue Manager just as an simple string property here
cf.SetStringProperty(XMSC.WMQ_QUEUE_MANAGER, queueManagerName);
If it is the case, I cannot see any reason why I do not choose IBMXMSDotnetClient.
But the weird thing is that there are 327K downloads for IBMMQDotnetClient while there are only 160k downloads for IBMXMSDotnetClient.
Or there are some key advantages of IBMMQDotnetClient I missed?
IBMMQDotNet provides MQ native APIs in .NET language while IBMXMSDotNet provides JMS style of APIs in .NET. However there is one major difference between the two APIs: IBMXMSDotNet provides asynchronous message consumer while IBMMQDotNet does not. Asynchronous message consumption is a type of consuming messages where IBMXMSDotNet automatically calls application registered callback method whenever a message arrives in MQ queue. With IBMMQDotNet, you will need to do a synchronous Get call (with or without timeout) to receive messages.
Hope this helps
Update: 16th May 2022
One example: In case of MQDotNet, you can change selection criteria to read messages matching another criteria without reopening a queue. For example you can say "select message matching groupId x" first from the queue to read messages matching the given groupId. Then you can say "select message matching groupId y" from the same queue without reopening queue. In case of XMSDotNet, the selection createria is specified while creating the consumer (which internally opens a MQ queue). Consumer must be recreated if you want to change selection criteria to consume messages with another criteria
Introduction
We exchange income data with an external party. Each year income tax regulations change and a new message schema has to be implemented. Altogether we now have 8 different schema versions each of which are deployed in a separate 'year income tax' application and this amount increases by 1 each year.
Because we pay our hosting company per installed application, we want to decrease the amount of applications installed.
All these applications are functionally equal, which means we validate incoming messages, and forward valid messages into a specific MQSeries queue. Each invalid message is routed to a response queue. Each application has it's own 'valid' and 'invalid' message queues.
The plan
One generic application that processes all 8(+) messages. New schemas must be deployable without application changes or downtime for previous, running 'income year tax' flows.
So far...
I can receive multiple messages on the same BizTalk receive port (MessageType XmlDocument) and am able to validate these messages dynamically in an orchestration by calling a custom receive pipeline (XML Disassembler + XML Validator). Exceptions as well as valid messages are processed as prescribed. There are no references between the Schemas and the generic application, so schemas can be deployed without need to stop running processes. So far, so good.
The orchestration has 1 receive shape, and 2 send shapes (valid, invalid).
SSO contains the values for routing the 'valid' and 'invalid' messages to their correct queue. Based on the incoming messagetype SSO is questioned for the correct 'valid' or 'invalid' queuedefinition.
The problem
I have previously dealt with dynamic FTP, FILE, WCF and SMTP ports, which all worked flawlessly after supplying the adapter with the correct Context Properties. Even MSMQ seems to have a fairly straightforward approach on dynamically setting transport properties.
However, I cannot seem to find MQSeries MQMT ContextProperties to set the queuedefinition dynamically.
Microsoft does not provide much information on this, and extensive searches on the internet hasn't provided me with anything useful (examples) either.
I tried matching IBM's docs with Microsoft's, but altogether I am now stuck.
I would suggest to use MQSC adapter for IBM MQ integration. It is part of Host Integration Server MSI. It only requires MQ client to be installed on the server Vs MQ Server for Windows installation required by MQSeries adapter.
Set the OutboundTransportLocation property in following format mqsc://{channelName}/tcp/{server{({port})/{queuemanager}/{queuename}
TransportType = MQSC
Context Properties - Schema can be found within assembly MQSeriesEx.MQSPropertySchemaEx with namespace (http://schemas.microsoft.com/BizTalk/2003/mqs-properties).
There are only few context properties you would need to set if at all required.
Channel_HeartBeat
Channel_MaxMessageLength
Channel_UserId
Channel_Password
ConnectionTimeout
If additional properties are required than use MQSeries.MQSPropertySchema context properties.
Thanks Vikas for your suggestion.
I followed your directions and found it works!
However, I found it a little more complicated than needed as it required me configuring channel names for each flow.
The solution that best suited me was the one I had in mind all along, and it was right before me. My attempts failed because I made a fatal mistake by setting the outgoing message's properties where I should have set the dynamic send port's properties.
SendPort(Microsoft.XLANGs.BaseTypes.Address)="MQS://SERVER/QMANAGER/QUEUENAME";
On several occasions I have received the following error from a .Net (C#, 4.0) application out of the blue on sending a message thru a producer:
CWSMQ0082E: Failed to send to CompCode: 2, Reason: 2009. A problem was encountered whilst sending a message. See the linked exception for more information.
Of course, the LinkedException (why not use the InnerException IBM???) is null i.e. no more information available.
Code I'm using (pretty straightforward):
var m = _session.CreateBytesMessage();
m.WriteBytes(mybytearray);
m.JMSReplyTo = myreplytoqueue;
m.SetIntProperty(XMSC.JMS_IBM_MSGTYPE, MQC.MQMT_DATAGRAM);
m.SetIntProperty(XMSC.JMS_IBM_REPORT_COA, MQC.MQRO_COD);
m.SetIntProperty(XMSC.JMS_IBM_REPORT_COD, MQC.MQRO_COA);
myproducer.Send(m, DeliveryMode.Persistent, mypriority, myttl);
(Offtopic: I hate the SetIntProperty way of setting properties. Which <expletive deleted> came up with that idea? It takes ages to look up all sorts of constants all over the place and its allowed values.)
The exception is thrown on the .Send method. I'm using XMS.Net (IA9H / 2.0.0.7). The only Google result that turns up turns out to have a different reason code (and even if it were the same, it should be fixed in my version if I understand correctly). This occurs randomly (though it seems to happen more often when it's been a while since a message has been sent/received) and I have no way to reproduce this.
I have ab-so-lute-ly no idea how to troubleshoot this or even where to start looking. Is this something caused by the server-side? Is it caused by XMS.net or some underlying IBM WebSphere MQ infrastructure?
Some results that I found that seem similar are suggesting to set SHARECNV to any value higher than 0 or to "true" / "yes" but the documentation explicitly tells me the default is 10. Also; I have no idea if this is the cause so changing it to another value feels like a shotgun approach.
Anybody any idea on how to go about solving this? I could of course just catch the exception, tear everything (channels, sessions, whatever) down and restart but that's just plain ugly IMHO.
The 2009 return code means "Connection Broken." Basically, the underlying TCP socket is gone and the client finds out about it at the time of the API call. It is possible to tune the channels using heartbeat and keepalive so that WMQ tries harde to keep the socket alive. However if the socket is timed out by the underlying infrastructure, nothing WMQ can do will help. Examples we've seen are that firewalls and load balancers are often set to detect idle connections and sever them.
Modern versions of WMQ client will attempt to reconnect transparently. The application just blocks a bit longer when this occurs.
Short of using the automatic reconnect, the only solution is in fact to rebuild the connection. Since it will get a new connection handle, all the object handles must be rebuilt as well.
Many of the tuning functions described here are available through the client configuration file, available in v7.0 and greater clients. In particular, the TCP stanza of that file enables keepalive. (The TCP spec says that if keepalive is provided, it must be disabled by default.) The QMgr has a similar ini file with configuration stanzas, including one for keepalive. The latest WMQ client is available as SupportPac MQC71 if you need that.
In cases where the main exception is sufficient enough to indicate the error, the inner exception will be null. In your case it's MQ reason code 2009 which means a connection to queue manager has been broken. The socket through which your application and queue manager were communicating was closed for some reason. The reason for socket close could be a network blip.
Along with suggestions T.Rob noted above, You could also run a XMS and Queue manager trace to understand the problem further. Please see the Troubleshooting chapter in XMS InfoCenter.
HTH
I am trying inialise a tls tunnel with the .net SslStream but after opening the stream I always get the following error:
"Unable to read data from the transport connection: An established connection was aborted by the software in your host machine."
After I establish a tls connection and after sending a second message.
I've been searching for an answer for the last four days but there isn't any helpful information online!
edit:
I am trying to connect to talk.google.com
and I'm using the code sample from MSDN. Only difference is that I'm sending data before and when it is time to use tls i do the following:
public void SecureStream()
{
netStream.Flush();
sslStream = new SslStream(netStream, false, new RemoteCertificateValidationCallback(ValidateServerCertificate), null);
sslStream.AuthenticateAsClient("talk.google.com");}
edit: I managed to eliminate the first error (small bug on how i was handling the send) now I always get
"Unable to read data from the transport connection: An established connection was aborted by the software in your host machine."
edit2: Im not sending any whitespaces I rewrote the message passing part and I still have the same problem.
I start with
String streamInit = "<stream:stream xmlns='jabber:client' xmlns:stream='http://etherx.jabber.org/streams' to='google.com' version='1.0'>";
client.Send(streamInit);
Then on receive I have the following
static void client_MessageReceived(SyncronousClient source, string Result)
{
if (Regex.IsMatch(Result, "<starttls xmlns=\"urn:ietf:params:xml:ns:xmpp-tls\"><required/></starttls>"))
{
String startTlS = "<starttls xmlns='urn:ietf:params:xml:ns:xmpp-tls'/>";
source.Send(startTlS);
}
else if (Regex.IsMatch(Result, "<proceed xmlns=\"urn:ietf:params:xml:ns:xmpp-tls\"/>"))
{
//Do TLS Magic
source.SecureStream();
String streamReInit = "<stream:stream xmlns='jabber:client'xmlns:stream='http://etherx.jabber.org/streams'to='google.com'version='1.0'>";
source.Send(streamReInit);
}
else if (Regex.IsMatch(Result, "<mechanisms xmlns=\"urn:ietf:params:xml:ns:xmpp-sasl\">"))
{
//String AuthType = "<auth xmlns='urn:ietf:params:xml:ns:xmpp-sasl' mechanism='X-GOOGLE-TOKEN'/>";
String AuthType = "<auth xmlns=\"urn:ietf:params:xml:ns:xmpp-sasl\" mechanism=\"PLAIN\"/>";
source.Send(AuthType);
}}
It's unlikely to be your problem (unless .Net has started doing SNI under the covers), but when you call AuthenticateAsClient, pass in the same domain name that you used in your stream's to attribute (in this case, google.com). As well, you might need gmail.com instead of google.com:
sslStream.AuthenticateAsClient("gmail.com", null, SslProtocols.Tls, false);
As csharptest.net alluded to, make sure you don't have a keepalive timer that sends extra whitespace, or wait to start the timer until after TLS works. The only other way I can imagine your getting that error is if you don't have a ciphersuite that the server implements, but I know the .Net SslStream works against GTalk.
Lastly, use one of the existing .Net libraries for XMPP (there are 5 listed here), and you can start writing much more fun code right away. You're about to run into the inadequacies of the .Net XML system, and your regex-based approach won't work when you start getting partial stanzas or multiple stanzas in a single read.
That really doesn't make sense to me. The server, if using SSL, requires that the client perform the SSL handshake upon connection. Thus, I'm not sure what you mean by "I'm sending data before...". It sounds like your not immediately calling the AuthenticateAsClient. If this is the case, I suspect that is your problem. AFAIK, you cannot use the same socket/connection connection for both SSL and non-SSL communications. Either the server requires SSL, or it does not support it, it should never do both.
My previous answer above was in ignorance. Indeed it appears that the standard does in fact require that the connect send and receive data prior to the initialization of the SSL handshake. Really odd that they would do that... but whatever. After briefly reading through parts of the RFC it appears that you are expected to begin the SSL client auth immediately after the closing '>'. No trailing whitespace allowed which may be your problem?
At the moment I have a C# service that is reading messages off the queue (Websphere MQ) and writing them in a database.
Everytime I do a GET the message dissappears from queue. I would like an additional functionality though. I prefer to read a message off the queue and remove it in from the queue only after the write in the database was succesful. Please note I do all these in a multithreaded application. I know there is a way to browse the queue but this doesn't really provide the functionality I need.
I'm writing my firts WMQ application, and I know I'll run into this issue very soon, so I found your question.
I've found this http://www.mqseries.net/phpBB2/viewtopic.php?t=43043&sid=11ad2d587dbd19056836ccc3f8943e5f (specifing MQOO_BROWSE option while opening the queue) in other forum, I haven't tried it yet, but it think it worth a try...
[]'s
I have implemented similar functionality in C++. Hopefully this helps you or someone.
You can browse messages without removing them from the queue using options MQGMO_BROWSE_FIRST and MQGMO_BROWSE_NEXT.
How do I browse a Websphere MQ message without removing it?
Store message identifiers in a list or in any other suitable data structure.
Write messages to the database.
Then get messages from the queue normally without BROWSE option. ImqQueue::Get takes two parameters: options and ImqMessage. Set message identifier to ImqMessage-class before calling get. ImqMessage acts as a filter. You can select
only those messages that have been written successfully to the database.
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/index.jsp?topic=%2Fcom.ibm.mq.amqzan.doc%2Fuc10330_.htm