I need to implement pipeline if Service Fabric's Reliable Services, and I need some guidelines about what of these approaches is preferable from the viewpoint of reliability simplicity and simple good design:
I have been investigating this topic a lot as well (to be applied to my work for NServiceBus and MessageHandler) and would like to provide my thoughts on the matter. However I haven't determined what the best model is yet.
If you disregard the practical implementation with ServiceFabric I would categorize the proposed approach in the following order when it comes to reliability:
C) The store and forward model is probably the best of the 3 models when it comes to interservice communication, all services can work independently from each other and are in no way subject to networking outages (at the downside of added latency)
A) Input queue per service: Each service free from impact by network outages for it's own work. However when it wishes to send messages to another service it may be impacted by network outages and needs retry built in to accomodate for this.
B) Output queue per service: Is probably the least of the 3 models as each service is directly dependent on a resource of the others, this results in to much dependency on network availability between the nodes.
If you look at it from a simplicity point of view, I would categorize them the following way
A) Input queue per service: As the message source needs to actively route messages to a given destination queue, it is fairly simpel to implement business processes or workflows (what I assume your pipeline is going to represent) using a routing pattern (either static routing or dynamic f.e. using a routing slip pattern
C) Store and forward: Again routing is an explicit part of your implementation, so both static and dynamic routing patterns are possible, however the practical implemenation is harder as you need to build and manage a messagepump that transfers messages from the transfer queue (output) to the destination queue and the associated need to flow context from the message source into the message pump. (Shameless plug: NServiceBus is a framework that can take away the complexity for you and make this scenario as simple as A)
B) Output queue per service: Each service needs to be setup to explicitly read from another's queue, this approach would only allow static routing as the routing rules are embedded in where you read from only (this severely limit you from a functional perspective)
If we take ServiceFabric's implementation details into account, then I assume you want to make use of the IReliableQueue implementation? This implementation has some shortcomings though, that make me wonder if these patterns can actually be implemented properly on ServiceFabric's native storage infrastructure.
The storage infrastructure is only available on Statefull services, so Stateless services (like Rest API's or other protocol termination gateway's) cannot be part of the pipeline (usually you want one of these as an entry point)
Only 1 thread can access a reliable queue at the same time, so it is impossible to write and read from the same queue at the same time. This severely limits throughput of the queue.
Accessing a reliable queue requires a local transaction, but these transactions are limited to a single partition. So it's also impossible to scale out your statefull services to create a competing consumer pattern.
Given these shortcomings, I'm still inclined to use another type of queueing infrastructure for SF Services instead of SF's persistence model, for example Azure Service Bus or Azure Storage Queues (Which NserviceBus allows as well).
In short, I'll support both A and C, with a slight preference for C, but I'm not convinced about using reliable queues as an implementation until these shortcomings have been resolved.
Related
Under Service Oriented Architecture (SOA), I am interested in the question of whether a service should own its own data or not.
One of the constraints is that if anything fails at any point, we need to be able to roll the state of the entire system back to a prior state so we can retry or resume an operation.
If each service owns its own data, then does this imply that the system deals with change better from the programmers point of view?
However, if each service owns its own data, are there any mechanisms to roll the entire system back to a prior state so a failed operation can be resumed or retried?
It sounds like the granularity of what you call services might be wrong. A single service can have multiple endpoints (using same or different protocols) and if a message received on one endpoint requires rolling back state that was received on another it is still an internal transaction within the boundary of the service.
If we consider the simplistic example of order and customer services. The order services may have contracts with messages relating to the whole order or to an order line and cancelling the order will undo state that was affected by both. Usually the address change in the customer service would not be rolled back with that.
Sometimes service actions are tied together in a longer business process, to continue on the example above let's also add an invoicing service. so when we cancel an order we also want to cancel the invoice. However it is important to note that business rules within the realm of the invoicing service can behave differently, for instance and not "roll back" e.g. canceling an order late may require cancelation fees. This sort of long running interaction is what I call a saga (you can see a draft of that pattern here)
Also note that distributed transactions between services is usually not a good idea for several reasons (like holding locks for an external party you don't necessarily trust) you can read more about that here
The problem you raised here is (partially) solved by the two-phase commit protocol (see wikipedia article)
To avoid implementing this complex algorithm, you can dedicate one of the service of the architecture to data management. If you need data synchronization between different databases, try to do it on the lowest layer (ie system or DBMS).
SOA system defines more services within one system. This can provide more autonomous services in order to every service can be hosted on different machine.
But it does not mean that you can not provide unified persistent layer for all (domain) models which can point into one storage => simple business transaction when the whole system is spread into more computers or transaction for one system.
Autonomous domain model is useful besides other things during refactoring to avoid situation where a change in one model causes a change in another service => global changes in the whole application.
In short: No. Services don't "own" data.
Data are truths about the world, and implicitly durable and shared. Logical services (API) don't always map to real-world data in a 1-1 way. Physical services (code) are implementations that are very refactorable, which opposes the durable nature of data.
When you partition the data, you lose descriptive power and analytic insight. But where it really kills you is integrity. Data cannot be kept coherent across silos as you scale. For complex data, you need those foreign keys.
Put another way: a platform only has one "logical" DB (per environment), because there is only one universe. There are many valid reasons to break up a DB, such as HW limits, performance, coordination, replication, and compliance. But treat them as needed evils, used only when needed.
But I think you may be asking a different question: "should a long-running, data-based transaction be managed by a single authoritative service?" And typically, that answer is: Yes. That transaction service can implement the multiple steps to sequence the flow as it sees fit, such as 2-phase commit. All your other services should use that transaction service to execute the transaction.
BUT! That transaction service must interact with the DB as a shared resource using only atomic semantics. That includes all the transaction states (intent, then action, then result) so that recovery and rollbacks are possible. The database must be empowered to maintain integrity in the event of faults. I cannot stress this enough: everything, always must decompose into atomic DB operations if you want fault tolerance.
Imagine I want to have a small network of worker drones possibly on separate threads and possibly on separate processes or even on different PCs. The work items are created by a central program.
I'm looking for an existing product or service that will do this all for me. I know that there is MSMQ and also MQSeries. MQSeries is too expensive. MSMQ is notoriously unreliable. A database backed system would be fine, but I don't want to own/manage/write it. I want to use someone else's work queue system.
Related Articles:
Here is a similar question, but it's advocating building a custom queue mechanism.
The queue that I like a lot is this one from Google App Engine.
http://www.codeproject.com/KB/library/DotNetMQ.aspx
If you follow some guidelines you can use a database as a queue store with good success, see Using tables as Queues.
SQL Server comes with its own built-in message queuing, namely Service Broker. It allows you to avoid many of the MSMQ pitfalls when it comes to scalability, reliability and high availability and disaster recovery scenarios.
Servcie Broker is fully integrated in the database (no external store, one consistent backup/restore, one unit of failover, no need for expensive two-phase-commit DTC between message store and database, one single T-SQL API to access and program both the messages and your data) and also has some nice unique features such as transactional messaging with guaranteed Exactly-Once-In-Order delivery, correlated message locking, internal activation etc.
I have used Rabbit MQ in the past for a pet project, you could add that to your list for Queue systems.
As far as a framework to wrap the Queue's, you could take a look at http://www.nservicebus.com/ we have done a couple of basic projects here at work with that. And here's a quick example to get started: http://meisinger2.wordpress.com/2009/11/09/nservicebus-fifteen-minutes/
I have successfully used MassTransit in the past. It supports using MSMQ as well as RabbitMQ.
We're in a tight spot at work and I need some clarity:
Basically: We can use WCF to read/write to an MSMQ and get type
safety on the objects we pass.
Alternative: We could use something like NService Bus to do the
exact same thing.
Now at my work we're all well versed in WCF, but none of us are well versed in using a Service Bus.
So could someone please help me with some pros/cons gains/losses for using WCF or NService Bus as right now it looks to me that it would be easier to use WCF (as long as we don't need advanced transactions etc)?
How easy would you estimate it would be to later change from WCF to NService Bus?
Kind regards
If all you want to do is push messages into MSMQ, you don't need either.
NServiceBus doesn't just enqueue messages. Per NServiceBus service that you write, you can specify to which messages it subscribes and which it publishes. NServiceBus then takes care of the dispatching and delivering of the messages, using MSMQ as the transport mechanism.
NServiceBus makes it easy this way for applications to just publish and subscribe to messages without requiring knowledge of where they come from, or when a service has moved to another server.
However, NServiceBus is not as easy to use as you might think, there is not a lot of good documentation available. It thus takes some time to plough through it.
And since v2.5, you need a commercial license if you want to use it on more than one thread.
As others have mentioned, I wouldn't use NServiceBus as a way to simply get messages onto a queue. As Roy says, NServiceBus is not trivial to implement. It's an architectural choice, not simply an implementation choice.
However, when my team moved from a WCF solution to NServiceBus, we were very happy (after the learning curve). One of the advantages is that the design of the system makes it harder to do the wrong thing. That is to say, if something is hard to do in NServiceBus, then you are probably not thinking about the architecture the way you should (if you want an asynchronous, scalable, maintainable system).
I am creating a mass mailer application, where a web application sets up a email template and then queues a bunch of email address for sending. The other side will be a Windows service (or exe) that will poll this queue, picking up the messages for sending.
My question is, what would the advantage be of using SQL Service Broker (or MSMQ) over just creating my own custom queue table?
Everything I'm reading is suggesting I use Service Broker, but I really don't see what the huge advantage over a flat table (that would be a lot simpler to work with for me). For reference the application will be used to send 50,000-100,000 emails almost daily.
Do you know how to implement a queue over a flat table? This is not a silly question, implementing a queue over a table correctly is much harder than it sounds. Queue-like-tables are notoriously deadlock prone and you need to carefully consider the table design and the enqueue and dequeue operations. Also, do you know how to scale your pooling of the table? And how are you goind to handle retries and timeouts (ie. what timers are used for)?
I'm not saying you should use SSB. The lerning curve is very steep and is primarily a distributed applicaiton platform, not a local queueing product so some features, like dialogs, will actually be obstacles for you rather than advantages. I'm just saying that you must consider also the difficulties of flat-table-queues. If you never implemented a flat-table-queue then be warned, there are many dragons under that bridge.
50k-100k messages per day is nothing, is only one message per second. If you want 100k per minute, then we have something to talk about.
If you every need to port to another vendor's database, you will have less problem if you used normal tables.
As you seem to only have one reader and one write from your queue, I would tend to use a standard table until you hit problem. However if you start to feel the need to use “locking hints” etc, that the time to switch to the Service Broker Queues.
I would not use MSMQ, if both the sender and the reader need a database connection to work. MSMQ would be good if the sender did not talk to the database at all, as it lets the sender keep working when the database is down. However having to setup and maintain both the MSMQ and the database is likely to be more work then it is worth for most systems.
For advantages of Service Broker see this link:
http://msdn.microsoft.com/en-us/library/ms166063.aspx
In general we try to use a tool or standard functionality rather than building things ourselves. This lowers the cost and can make upgrading easier.
I know this is old question, but is sufficiently abstract to be relevant for long enough time.
After using both paradigms I would suggest flat table. It is surprisingly scalable and nifty. Correct hints need to be used.
Once the application goes distributed, or starts using mutiple allways on groups with different RW and RO servers, the Service Broker (or any other method of distributed communication) becomes a neccessity.
Flat table
needs only few hints (higly dependent on isolation level) to work scalably and reliably in the consumer (READPAST, UPDLOCK, ROWLOCK)
the order of message processing is not set in stone
the consumer must make sure that the message stays in the queue if the processing fails
needs some polling mechanism (job, CDC (here lies madness :)), external application...)
turn of maintenance jobs and automatic statistics for the table
Service broker
needs extremely overblown "infrastructure" (message types, contracts, services, queues, activation procedures, must be enabled after each server restart, conversations need to be correctly created and dropped...)
is extremely opaque - we have spent ages trying to make it run after it mysteriously stopped working
there is a predefined order of message processing
the tables it uses can cause deadlocks themselfs if SB is overused
is the only way (except for linked servers...) to send messages directly from database on RW server of one HA group to a database that is RO in this HA group (without any external app)
is the only way to send messages between different servers (linked servers are a big NONO (unless they become an YESYES - you know the drill - it depends)) (without any external app)
Say I need to design an in-memory service because of a very high load read/write system. I want to dump the results of the objects every 2 minutes. How would I access the in-memory objects/data from within a web application?
(I was thinking a Windows service would be running in the background handling the in-memory service etc.)
I want the fastest possible solution, and I would guess most people would say use a web service? What other options would I have? I just don't understand how I could hook into the Windows service's objects etc.
(Please don't ask why I would want to do this, maybe you're right and it's a bad idea but I am also curious if this type of architecture is possible.)
Update
I was looking at this site swoopo.com that I would think has a lot of hits near the end of auctions, but since the auction keeps resetting the hits to the database would be just crazy so I was thinking if they did it in memory then dumped to db every x minutes...
What you're describing is called a cache, with a facade front-end.
You write a facade to which you commit your changes and acquire your datasets. The facade queues up reads and writes and commits when the queue is full or after a certain amount of time has passed. Your web application has a single point of access to the data (the facade), and the facade is structured in such a way to avoid writing and reading from storage too often.
Most relational database management systems do this for you. They do this kind of optimization and queuing internally so writing another layer on top of it would only slow things down. So don't write a cache if you're using an RDBMS.
Regarding the specifics of accessing such a facade, you can treat it as just an object, and implement it however you want (its own thread, a thread pool, a Web service, a Windows service, whatever).
Any remoting technology would work such as sockets, pipes and the like.
Check out: www.remobjects.com
You could use a Windows Message Queues or a Service Bus, or even .NET remoting.
See http://www.nservicebus.com/, or http://code.google.com/p/masstransit/.
You could hook into the Windows Services objects by using Remoting or WCF, both offer very fast interprocess communication. Sockets are fast too but are more cumbersome to program compared to WCF. There is a ton of WCF documentation and support online.
Databases provide a level of caching for you. The advantage of an in memory golden copy such as the one you propose is that it never has to read from disk when a request comes in and if you host it on the same machine as your IIS (provided you have enough RAM for both) there is no extra network hop, making it much faster that querying a db. However, the downside to this approach is that it does not scale as well if you need to add machines to load balance.
Third party messaging providers such as TIBCO are also worth looking at.