I have a public ASP.NET Core service with 5 instances, one on each service fabric cluster node. I also have a worker service with just one instance, the primary because this is a stateful service. I want to make a call from the worker service to a specific instance of the frontend.
I am using the following standard code to create a connection...
IFrontend c = ServiceProxy.Create<IFrontend>(new Uri("fabric:/MyApp/FrontendService"));
This works but will connect to one of the 5 ASP.NET Core services and it could be any of them. I want to connect to a specific one. Is there some specific format where you can provide the service instance identifier?
There is no straight solution for this. One thing I can think of is to have the ASP.NET Core service send a message to the backend to register the WebSocket is present, and have that message include the node name the frontend service is running on (via ServiceContext.NodeContext.NodeName).
Then have a pub/sub mechanism to send send a message (including the node name of the designated ASP.NET Core service instance you want to address) from the backend to all ASP.NET Core service instance and let only the ASP.NET Core service instance handle the message if the node names match.
You could use this project for that
References:
Call a specific instance of a service in Azure Service Fabric
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/b7cc7df3-9872-4000-8cc6-c48cb47b0b3f/calling-all-stateless-service-instances-via-serviceproxy?forum=AzureServiceFabric
As PeterBons mentioned, there is no straight solution for this kind of problem, there are many catches to be aware before you decide on any approach related to your initial plan, I can point a few, maybe will help you make a better decision:
There is no guarantee that the user is connected to FE1 when the response comes back, even though you keep a connection open, the connection might fail, and the user might connect to FE2 when it happens, your response will be redirected to the wrong server.
The user might be still connected to same node on reconnection, but the original service might be moved around, is common on SF that services move around the nodes in case of failures or Load balancing, in this scenario, the connection might drop and reconnect on same node, if you keep track of the partition, the partition might be already on another node, receiving an useless message.
The Worker response might fail when sending the message to FE1 and you will need to handle retries in the Worker, the same can happen if FE to the User fail, you also have to add a retry logic in there, increasing the complexity.
Some approaches that might work:
Make the communication asynchronous using a message bus in the middle, so that both services does not care about each other state, every response sent from Worker to FE will be asynchronous, any failure might be handled on their time independently. You might want to use:
One Message Queue per partition or
A single Pub/Sub Topic to all partitions and each one handle what is forwarded to then.
Or, can use a PaaS service that manages that for you, like Azure SignalR Service, in this case you need only to have an unique identification for the client and the worker keep it to send an answer back.
Related
I am working on a Service Fabric Application, in which I am running my Application that contains a bunch of ASP.NET Core Web APIs. Now when I run my application on my local service fabric cluster that is configured with 5 nodes, the application runs successfully and I am able to send post requests the exposed Web APIs. Actually I want to hit the code running on a same cluster node with different post requests to the exposed APIs on that particular node.
For further explanation, for example there is an API exposed on Node '0' that accept a post request and execute a Job, and also there is an API that abort the running job. Now when I request to execute a Job, it starts to execute on Node '0' but when I try to abort the Job, the service fabric cluster forward the request to a different node for example say node '1'. In resulting I could not able to abort the running Job because there is no running Job available on Node '1'. I don't know how to handle this situation.
For states, I am using a Stateless service of type ASP.Net Core Web API and running the app on 5 nodes of my local service fabric cluster.
Please suggest what should be the best approach.
Your problem is because you are running your APIs to do a Worker task.
You should use your API to schedule the work in the Background(Process\Worker) and return to the user a token or operation id. The user will use this token to request the status or cancel the task.
The first step: When you call your API the first time, you could generate a GUID(Or insert in DB) and put this message in a queue(i.e: Service Bus), and then return the GUID to the caller.
The second step: A worker process will be running in your cluster listening for messages from this queue and process these messages whenever a message arrives. You can make this a single thread service that process message by message in a loop, or a multi-threaded service that process multiple messages using one Thread for each message. It will depend how complex you want to be:
In a single threaded listener, to scale you application, you have to span multiple instances so that multiple tasks will run in parallel, you can do that in SF with a simple scale command and SF will distribute the service instances across your available nodes.
In a multi-threaded version you will have to manage the concurrency for better performance, you might have to consider memory, cpu, disk and so on, otherwise you risk having too much load in a single node.
The third step, the cancellation: The cancellation process is easy and there are many approaches:
Using a similar approach and enqueue a cancellation message
Your service will listen for the cancellation in a separate thread and cancel the running task(if running).
Using a different queue to send the cancellation messages is better
If running multiple listener instances you might consider a topic instead of a queue.
Using a cache key to store the job status and check on every iteration if the cancellation has been requested.
Table with job status, where you check on every iteration as you would do with the cache key.
Creating a Remote endpoint to make a direct call to the service and trigger a cancellation token.
There are many approaches, these are simple, and you might make use of multiple in combination to have a better control of your tasks.
You'll need some storage to do that.
Create a table (e.g JobQueue). Before starting to process the job, you store in a database, store the status (e.g Running, it could be an enum), and then return the ID to the caller. Once you need to abort/cancel the job, you call the abort method from the API sending the ID you want to abort. In the abort method, you just update the status of the job to Aborting. Inside the first method (which runs the job), you'll need to check this table onde in a while, if it's aborting, then you stop the job (and update the status to Aborted). Or you could just delete from the database once the job has been aborted or finished.
Alternatively, if you want the data to be temporary, you could use a sixth server as a cache server and store data there. This cache server could be a clustered server as well, but then you would need to use something like Redis.
In a scale-out scenario where one server consists of master+worker endpoints and another server consists of workers, is it safe to call bus.Publish from an endpoint when it finishes handling a given event? (Keeping in mind bus.Publish could be invoked from an endpoint sitting on the worker server).
My initial reaction is that it's not safe since it sounds like the example where you should never call publish from a web server...
We could certainly use the WCF wrapper and call out to a service that exists only on the master+worker endpoint server, but does anyone have any practical experience with this?
Thanks!
Each logical subscriber has a receiving endpoint. If you're using the distributor, this is the distributor endpoint, or distributor queue, if you will. So the subscriber will subscribe to specific events and specify it's receiving endpoint. The publisher will have no idea if it's a single endpoint instance, or if it's a distributor receiving the message.
The distributor will then send the message to a worker that is ready to process the message.
This is explained in more detail and with some clarifying images on this page: http://docs.particular.net/nservicebus/scalability-and-ha/distributor/publish-subscribe
In the end, we made our web apps "send-only endpoints" which essentially means they simply send commands directly to an endpoint via a chosen transport (in our case MSMQ). Once we need to scale, we will eventually implement "Sender Side Distribution" rather than utilizing the distributor.
From the NSB support team: "If you add more endpoints, Sender Side Distribution is the way to go. It acts as a round-robin mechanism running on the sender side which would send messages to a different 'worker' endpoint when you scale out."
https://docs.particular.net/transports/msmq/sender-side-distribution
If you only need to fire-and-forget messages from a website or some other app/service, I'd recommend this approach - it's quite simple.
I have three wcf services A,B and C respectively ,since i wanted it to be SOA(Service Oriented Architecture) the way my setup works is when i send a request from client to server.
All the services are self hosted windows services.
Client sends request to service A (client has no clue about the other services B and C);
Service A eventually sends that request to Service B and Service C.
Service B and C sends response back to Service A which would be sent back to the client by service A.
Issue i m facing :If i make any changes in the code of Service B and rebuild and restart the service ,i am having issue getting the response back but when i restart all the remaining services then it works fine.
In other words my client doesn't get the response back unless i restart all the services(A,B and C) even though i just changed the code in only one service and rebuilt it.I know the thing works if i restart all the three services but i want to know is this the problem in my way of designing or it is something i have to deal with self hosted windows services.And all the services(A,B,C) are independent as none depends on each other.
Did some one ever see such things happened in SOA.I would be glad if some one can guide me to appropriate solution ?
Replace WCF between services with any sort of queue (one service publishes something, other can read when they are ready). Can be anything. Can be a simple table where you read from if there is something new. Can be RabbitMQ, NServiceBus, etc, whatever works for you.
Define messages you put into the queue: commands and events. Both are simple classes with properties, no logic there. Commands represent what the system is asked to do (RegisterUser, PlaceOrder, ect), events represent what the system has done (UserRegistered, OrderApproved, PaymentReceived, etc). Be explicit about actions, Don't do something like "I have changed all the properties of a user on the client, now I call SaveUser(user)". Your service supposes to know how to change objects, clients should only command what to do.
Never break your contract. It is easy, easier than it sounds: you can add things to your message contracts, but cannot remove. In other word you just keep your contract backwards compatible.
Now you have a much better design: services communicate only through messages in queues, messages are backward compatible. This means that you can stop any of the services at any time without impacting others: they will continue sending messages into queues, and when the stopped service comes back again it will catch up processing all the stuff from the queue.
Then, if you want, you can use the same approach with client interactions: if instead of calling WCF clients would only put their commands in some sort of a queue then service upgrades or other downtime would not impact user experience.
Example: if I use WCF to place an order or to put an item into a shopping card then if there is a problem or a service is down for maintenance I will not be able to do it. I would click a button and have a nasty error. More importantly my order will not make into the system.
In contrast, if there is a queue in the middle, I only put my command into the queue. Now even if my service is down at the moment, or experience a high load (and therefore slow) then my user experience is still the same and does not degrade. It is just my command will be processed a bit later, but as a client I don't really care. And my order will not be lost in this scenario. The system became fault-tolerate and self-balanced.
There are all sorts of fantastic tricks you can do if you simply put a queue in the middle instead of experiencing problems with spatial and temporal coupling that comes with WCF :)
And what I described is just the beginning... :)
You may want to consider using a service bus such as NServiceBus to help you accomplish your functionality.
The first issue it will help you address is the decoupling of your services via publish/subscribe messaging pattern. Rather than invoking web services in one or the other service, publish events that notify the respective services when something has occurred. In your case this would look something like this:
Client invokes web service in Service A.
Service A publishes a message "Client Command Received" which Service B and C subscribe to.
Service B and C handle this event and then publish events of their own.
Service A subscribes to both events and replies to the client.
The first and immediate benefit of using something NServiceBus is reliability. On top of that you are able to easily version your message without affecting your client or your respective services. NServiceBus has full WCF integration so your client can continue to send messages to your service as before.
One of the things that makes your scenario interesting is that you can't guarantee when Service B and C send their responses back to you. Do you keep the connection to the client open until Service has received their responses? Do you need both responses before you can send a the client its response? What happens if either or one of the service crash? What if there is a time limit to how long you can wait before a response is received by Service A? All of these questions and more can be answered with a feature in NServiceBus called Sagas. Check it out.
If using NServiceBus is not possible then things become more difficult. WCF doesn't support publish/subscribe out of the box so you will have to bake your own. At a minimum I would recommend using this to decouple your services. How you manage state and temporal coupling in your services is another matter. Save yourself the trouble.
There are other frameworks out there but if you want a developer centric, cost effective way to create a .NET based solution then recommend using NServiceBus.
I am building a WCF service that will expose several operations, it will run in IIS because it needs HTTPS endpoints. Most of the operations will perform within seconds or less; however, one or two of these operations will take between 5-90 minutes.
The primary consumer of this service will be an ASP.NET MVC application; what is the correct way to do handle this?
Should I jackup the timeout and do some ajax calls? Should I add a table to my database, and have the long running operations update this database, and have the web interface poll this table every minute? I'm not sure what (if there is) the generally accepted best practice for this.
I wrote something similar for my senior project, basically a job scheduling framework.
I chose to go down the path of storing the "status" of the "job" in the database.
I wrote a manager windows service that implemented a WCF client (proxy)
I wrote a WCF Service that implemented my "worker host".
The manager service would read the queue from the database, and hand out work to all of my "worker hosts". The reason I had windows service perform this task as opposed to just having the UI talk directly to the worker host, was because it gave an extra level of control over the whole process.
I didn't like the idea of having "the network cable unplugged" from my worker host, and never getting a status update again from this specific job. So, the windows service gives me the ability to constantly monitor the progress of the WCF worker host, and if a connection error ever occurs (or something else unexpected), I can update the status to failed. Thus, no orphaned jobs.
Take a look at this
WCF Long Running Operations
There could be other options but they are nearly the same. You can also come up with some push notifications (I assume no data is returned) as one int the following link
WCF Push
I can't deny the performance benefit of a duplex async call, but some things about makes me feel wary.
My concern is that given a client object instantiated, will WCF be able to tell which particular client service instance will receive the callback argument?
Can anyone tell me if this is a good idea? If not why not?
new DuplexChannelFactory<IServerWithCallback>(
new ClientService(),
new NetTcpBinding(),
new EndpointAddress("net.tcp://localhost:1234/"+Guid.NewGuid()))
If the virtual path above is reserved how can it be discarded. I want the client service lifetime to be fairly short. IE make a request and receive a response and when done receiving, kill it. How bad is the performance penalty in making the client service lifetime short as opposed to pooling it and keeping it alive longer.
The idea is to avoid timeout issue. When done receiving, sending, dispose ASAP. By convention - can't pass the client services around. If you need info, create a new one, simple - just like EF/L2S etc.
From inside the WCF service itself, how do I kill the session with the client. ie. I don't want the client ending the session - I know I can decorate my operation accordingly, but I want the service to terminate itself programmatically when certain conditions are met.
I can affix the port and forward accordingly to resolve any firewall issue, but what I'm worried about is if the client were to sit behind a load-balancer. How would the service know which particular server to call?
I think in the end Duplex services is simply another failed architecture from Microsoft. This is one of those things that looked really good on paper but just falls apart upon closer examination.
There are too many weaknesses:
1) Reliance on session to establish client listener by the server. This is session information is stored in memory. Hence the server itself cannot be load balanced. Or if it were load balanced you need to turn ip affinity on, but now if one of the servers is bombarded you can't simply add another one and expect all these sessions to automagically migrate over to the new server.
2) For each client sitting behind a router/firewall/loadbalancer, a new end point with specific port needs to be created. Otherwise the router will not be able to properly route the callback messages to the appropriate client. An alternative is to have a router that allows custom programming to redirect specific path to a particular server. Again a tall order. Or another way is for the client with the callback to host its own database and share data via a database <-- Might work in some situation where licensing fees is not an issue... but it introduces a lot of complexity and so onerous on the client plus it mixes the application and services layer together (which might be acceptable in some exceptional situation, but not on top of the huge setup cost)
3) All this basically says that duplex is practically useless. If you need call back then you will do well to setup a wcf host on the client end. It will be simpler and much more scalable. Plus there is less coupling between client and server.
The best duplex solution for scalable architecture is in the end not using one.
It will depend on how short you need the clients new'd up and how long they will last. Pooling would not be an option if you specifically need a new client each time, but if the clients keep doing the same thing why not have a pool of them waiting to be used, if they fault out recreate that same client again.
In reality in a callback scenario if the service is calling back to the client (really calling a function on the client) to pass information the service is now the client and vice versa. You can have the service that's making the callback .Close() the connection but it will be open until the GC can dispose of it, from my experience that can take longer than expected. So in short the client should be responsible (the client being the one making the call to something) for shutting itself down, or disconnecting, the service should only give back answers or take data from a client.
In duplex callbacks the service now calling back to the client will get the address of the client abstracted behind the duplexchannelfactory. If the service can't call back to the client I don't think there's much that can be done, you'd have to ensure the port that your clients are calling to the service is open to receive callbacks I would guess.