Introduction
We exchange income data with an external party. Each year income tax regulations change and a new message schema has to be implemented. Altogether we now have 8 different schema versions each of which are deployed in a separate 'year income tax' application and this amount increases by 1 each year.
Because we pay our hosting company per installed application, we want to decrease the amount of applications installed.
All these applications are functionally equal, which means we validate incoming messages, and forward valid messages into a specific MQSeries queue. Each invalid message is routed to a response queue. Each application has it's own 'valid' and 'invalid' message queues.
The plan
One generic application that processes all 8(+) messages. New schemas must be deployable without application changes or downtime for previous, running 'income year tax' flows.
So far...
I can receive multiple messages on the same BizTalk receive port (MessageType XmlDocument) and am able to validate these messages dynamically in an orchestration by calling a custom receive pipeline (XML Disassembler + XML Validator). Exceptions as well as valid messages are processed as prescribed. There are no references between the Schemas and the generic application, so schemas can be deployed without need to stop running processes. So far, so good.
The orchestration has 1 receive shape, and 2 send shapes (valid, invalid).
SSO contains the values for routing the 'valid' and 'invalid' messages to their correct queue. Based on the incoming messagetype SSO is questioned for the correct 'valid' or 'invalid' queuedefinition.
The problem
I have previously dealt with dynamic FTP, FILE, WCF and SMTP ports, which all worked flawlessly after supplying the adapter with the correct Context Properties. Even MSMQ seems to have a fairly straightforward approach on dynamically setting transport properties.
However, I cannot seem to find MQSeries MQMT ContextProperties to set the queuedefinition dynamically.
Microsoft does not provide much information on this, and extensive searches on the internet hasn't provided me with anything useful (examples) either.
I tried matching IBM's docs with Microsoft's, but altogether I am now stuck.
I would suggest to use MQSC adapter for IBM MQ integration. It is part of Host Integration Server MSI. It only requires MQ client to be installed on the server Vs MQ Server for Windows installation required by MQSeries adapter.
Set the OutboundTransportLocation property in following format mqsc://{channelName}/tcp/{server{({port})/{queuemanager}/{queuename}
TransportType = MQSC
Context Properties - Schema can be found within assembly MQSeriesEx.MQSPropertySchemaEx with namespace (http://schemas.microsoft.com/BizTalk/2003/mqs-properties).
There are only few context properties you would need to set if at all required.
Channel_HeartBeat
Channel_MaxMessageLength
Channel_UserId
Channel_Password
ConnectionTimeout
If additional properties are required than use MQSeries.MQSPropertySchema context properties.
Thanks Vikas for your suggestion.
I followed your directions and found it works!
However, I found it a little more complicated than needed as it required me configuring channel names for each flow.
The solution that best suited me was the one I had in mind all along, and it was right before me. My attempts failed because I made a fatal mistake by setting the outgoing message's properties where I should have set the dynamic send port's properties.
SendPort(Microsoft.XLANGs.BaseTypes.Address)="MQS://SERVER/QMANAGER/QUEUENAME";
Related
We are using MassTransit with RabbitMQ and part of our implementation includes an outbox pattern.
Now i'm trying to create a docker container whose only purpose is to dispatch messages from outboxes in several databases.
The container gets a list of connection strings to the various databases and then starts to dispatch messages from their outboxes.
Currently we store the following information in our outbox (with examples):
MessageType: SomeNamespace.SomeType, SomeContract
MessageBody: {"SomeProperty":"MyValue"}
TransmitMethod: Send/Publish
QueueName: SomeQueueName
My question is if it's possible to dispatch these messages without having access to the contract types?
I can add more information to the table if needed to make this happen.
You can look at how the MassTransit message scheduler support for Quartz.NET captures and ultimately sends the message on the transport. In this case, it's saving the serialized message from the transport and reloading the JSON into the message body at serialization time.
You might also find useful details in the relational outbox draft PR.
We're using ActiveMQ locally to transfer data between 5 processes that turn simultaneously.
I have some data I need to send to a process, both at runtime (which works perfectly fine), but also a default value on start. Thing is it is published when the process starts, it just doesn't read because it wasn't subscribed to the topic at the time the data was sent.
I have multiple solutions : I could delay the first publishing for a moment so that the process has time to launch (which doesn't seem very appealing) ; or is there a way to send all stored previously non-treated messages to some process that just subscribed ?
I'm coding in C#.
I don't have any experience with ActiveMQ, but other message system usually have an option which marks the subscription as persistent, which means that; after the first subscription; the message queue itself checks if a certain message is delivered to that system and retries with a timeout. In this scenario you need to start the receiver at least 1 time.
If this is not an option and you want to plug in receiver afterwards, you might want to consider a setup of your messages which allows you to retrieve the full state, i.e. if you send total-messages instead of differential- messages.
After a little google, I came upon this definition durable subscribers, I hope this helps:
See:
http://activemq.apache.org/how-do-durable-queues-and-topics-work.html
and
http://activemq.apache.org/manage-durable-subscribers.html
since you are using C# client i don't konw if this is supported
topic = new ActiveMQTopic("TEST.Topic?consumer.retroactive=true");
http://activemq.apache.org/retroactive-consumer.html
So, another solution is to configure this behavior on the broker side by adding that to the activemq.xml and restart :
The subscription recovery policy allows you to go back in time when
you subscribe to a topic.
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" >
<subscriptionRecoveryPolicy>
<timedSubscriptionRecoveryPolicy recoverDuration="10000" />
<fixedCountSubscriptionRecoveryPolicy maximumSize="10000" />
</subscriptionRecoveryPolicy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
http://activemq.apache.org/subscription-recovery-policy.html
I went around the issue by sending a message from each process when they're launched back to the main one, and then only sending the info I needed to send.
I have a redis instance that publishes messages via different topics. Instead of implementing a complex heartbeat mechanism (complex because the instance would stop publishing messages after some time if they are not consumed), is there a way to check whether pubs are consumed by anyone?
For example, instance RedisServer publishes messages to topic1 and topic2. RedisClient1 subscribes to topic1 and RedisClient2 subscribes to topic2. When RedisClient2 for whatever reason stops consuming messages of topic2 then I want RedisServer to know about it and decide when to stop publishing messages to topic2. The discontinuation of topic2 consumption is unpredictable hence I am not able to inform RedisServer of the discontinuation/unsubscription.
I thought if there was a way for a redis instance to know whether messages of a certain topic are consumed or not then that would be very helpful information.
Any idea whether that is possible?
Given you are using a recent-enough version of redis (> 2.8.0) these two commands may help you:
PUBSUB CHANNELS [pattern]
Which lists the currently active channels ( = channel having at least one subscriber) matching the pattern.
PUBSUB NUMSUB [chan1 ... chanN]
Which returns the number of subscribers for the specified channels (doesn't work for patterns however).
Note: Both solutions won't enable you to determine if a message was truely processed! If you need to know about completion of tasks (if your messages are triggering something), then I would recommend searching for a full blown job queue (for example Resque, if you want to stick with Redis)
Edit: Here's the Redis doc. for all of the above: http://redis.io/commands/pubsub
You can also use the result of PUBLISH. It will give you the number of subscribers that received the message: http://redis.io/commands/publish
This way you don't need to poll the PUBSUB command, just do your "stop publishing" messages logic after you publish a message.
At most you publish one message with no one subscribing.
Edit 2:
Client Library: After reviewing it is not easily suggested that this is for the .NET client library.
DLL: Google.Apis.Admin.email_migration_v2.dll
What steps will reproduce the problem?
Generate a process which contains a
Google.Apis.Admin.email_migration_v2.AdminService instance for each
unique Google Apps Gmail mailbox that will have messages sent to it.
All of the AdminService objects generated use the same OAuth2.0
credentials and application name. Each AdminService object generated
will only send messages to one Google Apps user’s mailbox. For
example, if we were sending messages to five different Google Apps
Gmail mailboxes we would generate five AdminService objects to send
messages; one for each user’s mailbox.
Biggest thing to note is that each AdminService object created is created on a separate process.
AdminService objects were given a FileDataStore object to change the location of where the refresh token is stored; C:\ProgramData\SomeFile\SomeFile.
Supplied appropriate scopes to the credentials.
Begin sending mail messages on each process. Using one thread to send messages in each process, so only one message is sent at a time to each user’s mailbox.
Each message sent gets its own instance of MailItem and MailResource.InsetMedia
The MailResource.InsertMedia object is generated for each item by calling AdminService.Mail.Insert(MailItem, string, Stream, string) method.
When our code makes the call to MailResource.InsertMediaUpload.UploadAsync(CancellationTokenSource).Result is where we can receive the error.
The error is caught and handled (logged) from the return type of the aforementioned call; the type is Google.Apis.Upload.IUploadProgress. The exception is handled using the IUploadProgress.Exception property.
What is the expected output? What do you see instead?
The expected output would be a successful message response or the exception property of the IUploadProgress to be null after the return of the task. Instead we are receiving the following error message:
The service admin has thrown an exception:
Google.GoogleApiException:Google.Apis.Requests.RequestError
Limit reached. [412]
Errors [Message[Limit reached.] Location[If-Match - header] Reason[conditionNotMet] Domain[global]]
at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at Google.Apis.Upload.ResumableUpload`1.d__e.MoveNext()
What version of the product are you using?
Google.Apis.Admin.Email_Migration_v2 (1.8.1.20)
What is your operating system?
Windows Server 2008 R2 Enterprise (SP1)
What is your IDE?
Visual Studio 2013 Premium
What is the .NET framework version?
4.0.30319
Please provide any additional information below.
Non-consecutive messages can fail (with the 412 http status code
provided above) during the process of sending the messages. Once we
receive this error other messages sent after the failed message(s)
can succeed. (Items can fail at any point during the process
beginning, middle or end.)
Each message sent has nearly identical content. The size of the
messages range from 1KB to 100KB including the size of all associated
attachments, not all messages have attachments.
Reprocessing the failed items at a later time results in successful
message responses and the appropriate items are sent to the user’s
Google Apps Gmail Inbox.
The maximum number of Google Apps user’s mailboxes sent to at one
time was ten.
After checking the quotas of our Google Developers Console project:
We were nowhere near the specified limit of 20 requests a second for
the Email Migration API; maxed out at sending 7 requests a second.
Only 2% of the maximum daily requests had been reached.
All messages sent had the same label; the label was well under the
225 character limit. Actually all of the labels/sub-labels applied
together only surmounted to 40 characters.
This error message can still be received when sending to only one
Google Apps user’s mailbox; only using one process and one thread.
Each process is normally sending anywhere from 1000-5000 messages.
I have not found a lot of specific documentation to explain this particular error in enough detail to remedy the problem at hand.
Questions:
So what exactly does this 412 http status code mean? What limit is being encountered that this message is referring to?
Shouldn’t we be receiving some form of 5XX error from the server if we are hitting a limit? In which case wouldn’t the built in exponential back off policy kick in?
a. Unless the server is checking the POST request for a pre-condition about a server side limit then telling the client to back off which is what a 412 error seems to typically indicate. In that case please give as much detail as possible for question 1.
Sorry for the extensive post! Thanks for your time! I will also be creating a defect/issue in Google's .NET issue tracker and providing a link.
Edit 1:
For anyone interested in following this issue here is a link to the submitted item in Google's issue tracker for .NET.
Submitted Issue
For reference it is issue 492.
I am not quite sure where you see the "the specified limit of 20 requests a second for the Email Migration API". Reminder: the QPS limit you see in the Google Developers Console project is not the actual default limit. You can change that limit to anything you want, and thus, that's not the actual limit for the API. It is really just for managing the consumption of the API quota (some APis will have a much higher QPS where you can adjust it to lower for different projects across your console).
According to the email migration APi documentation, the QPS is 1 request per second (the link is here: https://developers.google.com/admin-sdk/email-migration/v2/limits).
I have experienced 412 errors when the QPS limit is being hit, and I have also seen the 412 error returned when I am uploading too much data to a single domain. How much data are you loading all at once? I would suggest doing an exponential backoff to see if the issue would disappear.
I believe I have found an answer to this problem, though I will advise a disclaimer, I do not work for Google and cannot be 100% sure of the accuracy; you've been warned. This should at least hold true for the .NET version of Google's Email Migration v2 API. I cannot guarantee how other APIs work because I do not use them..
Through working with this API in spurts for well over eight months now, it appears that if an application or multiple applications are to send messages to a single Google Apps user/mailbox consistently, at a faster rate than which Google servers can process, then at some rate you should start to get a bunch of GoogleApiExceptions stating "412 - Limit Reached" when sending new messages. What we have gathered through using our application is that each Google Apps user/mailbox has its own pending items queue. When you send a message to Google Apps it is first put into this queue before being processed by a Google Server and put into the user's mailbox. If this queue becomes full and you attempt to send another message you will receive a 412 error.
Options are to wait before sending another message, you'll have to wait however long the Google server takes to process the next message in the user's queue before sending another; which is unpredictable. The better option in my opinion is to start sending messages to another Google Apps user; because each user appears to have its own message queue. Be sure to stop sending to the user who is consistently getting 412 errors. This will give the Google server some time to process that user's packed message queue. Note each pending messages queue appeared to hold about 100-150 items before throwing 412 errors.
503 errors appear to occur when sending messages into a user's mailbox queue at a higher rate than 1 request per second. As Emily has stated "the QPS limit you see in the Google Developers Console project is not the actual default limit" it is truly 1 QPS per Google Apps user.
As for the exponential back-off it is supposed to be implemented automatically see this. Note Peleyal appears to be the gentleman in charge of the API; can be noted from the download page for the API.
This took us a little while to figure out so cheers if you're having this issue! Please if you find any contradicting information correct any mistakes found in this answer or make your own!!
At the moment I have a C# service that is reading messages off the queue (Websphere MQ) and writing them in a database.
Everytime I do a GET the message dissappears from queue. I would like an additional functionality though. I prefer to read a message off the queue and remove it in from the queue only after the write in the database was succesful. Please note I do all these in a multithreaded application. I know there is a way to browse the queue but this doesn't really provide the functionality I need.
I'm writing my firts WMQ application, and I know I'll run into this issue very soon, so I found your question.
I've found this http://www.mqseries.net/phpBB2/viewtopic.php?t=43043&sid=11ad2d587dbd19056836ccc3f8943e5f (specifing MQOO_BROWSE option while opening the queue) in other forum, I haven't tried it yet, but it think it worth a try...
[]'s
I have implemented similar functionality in C++. Hopefully this helps you or someone.
You can browse messages without removing them from the queue using options MQGMO_BROWSE_FIRST and MQGMO_BROWSE_NEXT.
How do I browse a Websphere MQ message without removing it?
Store message identifiers in a list or in any other suitable data structure.
Write messages to the database.
Then get messages from the queue normally without BROWSE option. ImqQueue::Get takes two parameters: options and ImqMessage. Set message identifier to ImqMessage-class before calling get. ImqMessage acts as a filter. You can select
only those messages that have been written successfully to the database.
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/index.jsp?topic=%2Fcom.ibm.mq.amqzan.doc%2Fuc10330_.htm