We have a TIBCO EMS solution that uses built-in server failover in a 2-4 server environment. If the TIBCO admins fail-over services from one EMS server to another, connections are supposed to be transfered to the new server automatically at the EMS service level. For our C# applications using the EMS service, this is not happening - our user connections are not being transfered to the new server after failover and we're not sure why.
Our application connection to EMS at startup only so if the TIBCO admins failover after users have started our application, they users need to restart the app in order to reconnect to the new server (our EMS connection uses a server string including all 4 production EMS servers - if the first attempt fails, it moves to the next server in the string and tries again).
I'm looking for an automated approach that will attempt to reconnect to EMS periodically if it detects that the connection is dead but I'm not sure how best to do that.
Any ideas? We are using TIBCO.EMS.dll version 4.4.2 and .Net 2.x (SmartClient app)
Any help would be appreciated.
First off, yes, I am answering my own question. Its important to note, however, that without ajmastrean, I would be nowhere. thank you so much!
ONE:
ConnectionFactory.SetReconnAttemptCount, SetReconnAttemptDelay, SetReconnAttemptTimeout should be set appropriately. I think the default values re-try too quickly (on the order of 1/2 second between retries). Our EMS servers can take a long time to failover because of network storage, etc - so 5 retries at 1/2s intervals is nowhere near long enough.
TWO:
I believe its important to enable the client-server and server-client heartbeats. Wasn't able to verify but without those in place, the client might not get the notification that the server is offline or switching in failover mode. This, of course, is a server side setting for EMS.
THREE:
you can watch for failover event by setting Tibems.SetExceptionOnFTSwitch(true); and then wiring up a exception event handler. When in a single-server environment, you will see a "Connection has been terminated" message. However, if you are in a fault-tolerant multi-server environment, you will see this: "Connection has performed fault-tolerant switch to ". You don't strictly need this notification, but it can be useful (especially in testing).
FOUR:
Apparently not clear in the EMS documentation, connection reconnect will NOT work in a single-server environment. You need to be in a multi-server, fault tolerant environment. There is a trick, however. You can put the same server in the connection list twice - strange I know, but it works and it enables the built-in reconnect logic to work.
some code:
private void initEMS()
{
Tibems.SetExceptionOnFTSwitch(true);
_ConnectionFactory = new TIBCO.EMS.TopicConnectionFactory(<server>);
_ConnectionFactory.SetReconnAttemptCount(30); // 30retries
_ConnectionFactory.SetReconnAttemptDelay(120000); // 2minutes
_ConnectionFactory.SetReconnAttemptTimeout(2000); // 2seconds
_Connection = _ConnectionFactory.CreateTopicConnectionM(<username>, <password>);
_Connection.ExceptionHandler += new EMSExceptionHandler(_Connection_ExceptionHandler);
}
private void _Connection_ExceptionHandler(object sender, EMSExceptionEventArgs args)
{
EMSException e = args.Exception;
// args.Exception = "Connection has been terminated" -- single server failure
// args.Exception = "Connection has performed fault-tolerant switch to <server url>" -- fault-tolerant multi-server
MessageBox.Show(e.ToString());
}
This post should sum up my current comments and explain my approach in more detail...
The TIBCO 'ConnectionFactory' and 'Connection' types are heavyweight, thread-safe types. TIBCO suggests that you maintain the use of one ConnectionFactory (per server configured factory) and one Connection per factory.
The server also appears to be responsible for in-place 'Connection' failover and re-connection, so let's confirm it's doing its job and then lean on that feature.
Creating a client side solution is going to be slightly more involved than fixing a server or client setup problem. All sessions you have created from a failed connection need to be re-created (not to mention producers, consumers, and destinations). There are no "reconnect" or "refresh" methods on either type. The sessions do not maintain a reference to their parent connection either.
You will have to manage a lookup of connection/session objects and go nuts re-initializing everyone! or implement some sort of session failure event handler that can get the new connection and reconnect them.
So, for now, let's dig in and see if the client is setup to receive failover notification (tib ems users guide pg 292). And make sure the raised exception is caught, contains the failover URL, and is being handled properly.
Client applications may receive notification of a failover by setting the tibco.tibjms.ft.switch.exception system property
Perhaps the library needs that to work?
Related
We have a big system with bunch of services communicating with each other via TIBCO EMS Messaging.
Now, one of the services keeps throwing this exception when it tries to connect to EMS server:
TIBCO.EMS.IllegalStateException: connect failed: server in standby mode
However, other services on the same EMS server are working just fine without any exception.
I tried to reproduce this by creating a console application which would use same credentials to connect to EMS server on the same destination (Queue) but couldn't reproduce this error.
EMS connection is fault-tolerant and it might have been failed over to secondary server, however, now I see primary server is back up.
I also tried to connect to primary server only (which I can see is up & running) but it fails with the same exception.
Below is the stack trace for reference:
TIBCO.EMS.IllegalStateException: connect failed: server in standby mode
at TIBCO.EMS.CFImpl._CreateConnection(String userName, String password, Boolean xa)
at TIBCO.EMS.ConnectionFactory.CreateConnection(String userName, String password)
at Ems.makeConn()
We're using TIBCO.EMS.dll v. 8.1.0.10 on .NET 4.0.
Any idea what can cause this exception ?
Could your connection string only mention one of the hosts inside an FT/HA setup ? This looks like the error message returned when you are trying to connect to only one of the HA/FT host, but not the one active at the moment.
If it is the case, the connection string will work most of the time, but a proper EMS HA connection string include two (or more!) EMS host and port. Only one is active at the same time.
It typically looks like this:
serverUrl=tcp://server0:7222, tcp://server1:7344
See source doc here in TIBCO docs.
It turned out tools we were using to check if server is active (we don't have admin rights on the servers) was misbehaving (showing incorrect status), so none of those servers were really active (not primary nor secondary).
As soon as server was brought up this error message went away.
I am building a c#/wpf project.
It's architecture is this:
A console application which will be on a virtual machine (or my home computer) that will be the server side.
A wpf application that will be the client app.
Now my problem is this - I want the server to be able to send changes to the clients. If for example I have a change for client ABC, I want the server to know how to call a service on the clients computer.
The problem is, that I don't know how the server will call the clients.
A small example in case I didn't explain it well:
The server is on computer 1, and there are two clients, on computers 2 and 3.
Client 2 has a Toyota car and client 3 has a BMW car.
The server on computer 1 wants to tell client 2 that it has a new car, an Avenger.
How do I keep track and call services on the clients?
I thought of saving their ip address (from calling ipconfig from the cmd) in the DB - but isn't that based on the WI-FI/network they are connected to?
Thanks for any help!
You could try implementing SignalR. It is a great library that uses web sockets to push data to clients.
Edit:
SignalR can help you solve your problem by allowing you to set up Hubs on your console app (server) that WPF application (clients) can connect to. When the clients start up you will register them with a specified Hub. When something changes on the server, you can push from the server Hub to the client. The client will receive the information from the server and allow you to handle it as you see fit.
Rough mockup of some code:
namepsace Server{}
public class YourHub : Hub {
public void SomeHubMethod(string userName) {
//clientMethodToCall is a method in the WPF application that
//will be called. Client needs to be registered to hub first.
Clients.User(userName).clientMethodToCall("This is a test.");
//One issue you may face is mapping client connections.
//There are a couple different ways/methodologies to do this.
//Just figure what will work best for you.
}
}
}
namespace Client{
public class HubService{
public IHubProxy CreateHubProxy(){
var hubConnection = new HubConnection("http://serverAddress:serverPort/");
IHubProxy yourHubProxy = hubConnection.CreateHubProxy("YourHub");
return yourHubProxy;
}
}
}
Then in your WPF window:
var hubService = new HubService();
var yourHubProxy = hubService.CreateHubProxy();
yourHubProxy.Start().Wait();
yourHubProxy.On("clientMethodToCall", () => DoSometingWithServerData());
You need to create some kind of subscription model for the clients to the server to handle a Publish-Subscribe channel (see http://www.enterpriseintegrationpatterns.com/patterns/messaging/PublishSubscribeChannel.html). The basic architecture is this:
Client sends a request to the messaging channel to register itself as a subscriber to a certain kind of message/event/etc.
Server sends messages to the channel to be delivered to subscribers to that message.
There are many ways to handle this. You could use some of the Azure services (like Event hub, or Topic) if you don't want to reinvent the wheel here. You could also have your server application track all of these things (updates to IP addresses, updates to subscription interest, making sure that messages don't get sent more than once; taking care of message durability [making sure messages get delivered even if the client is offline when the message gets created]).
In general, whatever solution you choose is plagued with a common problem - clients hide behind firewalls and have dynamic IP addresses. This makes it difficult (I've heard of technologies claiming to overcome this but haven't seen any in action) for a server to push to a client.
In reality, the client talks and the server listens and response. However, you can use this approach to simulate a push by;
1. polling (the client periodically asks for information)
2. long polling (the client asks for information and the server holds onto the request until information arrives or a timeout occurs)
3. sockets (the client requests server connection that is used for bi-directional communication for a period of time).
Knowing those terms, your next choice is to write your own or use a third-party service (azure, amazon, other) to deliver messages for you. I personally like long polling because it is easy to implement. In my application, I have the following setup.
A web API server on Azure with and endpoint that listens for message requests
A simple loop inside the server code that checks the database for new messages every 100ms.
A client that calls the API, handling the response.
As mentioned, there are many ways to do this. In your particular case, one way would be as follows.
Client A calls server API to listen for message
Server holds onto call, waiting for new message entry in database
Client B calls server API to post new message
Server saves message to database
Server instance from step 2 sees new message
Server returns message to Client A.
Also, the message doesn't have to be stored in a database - it just depends on your needs.
Sounds like you want to track users à la https://www.simple-talk.com/dotnet/asp.net/tracking-online-users-with-signalr/ , but in a desktop app in the sense of http://www.codeproject.com/Articles/804770/Implementing-SignalR-in-Desktop-Applications or damienbod.wordpress.com/2013/11/20/signalr-a-complete-wpf-client-using-mvvm/ .
Is there a way to stop a remote connection?
In more detail if a WCF connection is created using the ChannelFactory
var aChannelFactory = new ChannelFactory<ISomeService>(aBinding, aEndpoint);
and the connection is used via GSM, it makes sense to enlarge timeouts so that
aChannelFactory.CreateChannel();
has the possibility to connect to the remote site.
Now the problem is that a user may have the wish to stop the creation of a connection (Abort a getting connencted window, only use offline part of a programm).
But how is this possible?
The CreateChannel-call is synchronous.
Even if it is wrapped in a thread, nowadays Thread.Abort should not be used.
Is there any way to stop the creation of a channel automatically?
Or must it be "faked" to a user, so that the connection still is running until its timeout (or success) even is user decided to work temporarely offline?
As Grant Thomas stated it is not necessary to abort the connection.
I'm using C# on Windows Server 2008, and I want to receive a message from a public transactional queue on another machine in the same domain. The error looks like this:
System.Messaging.MessageQueueException: Cannot import the transaction.
at System.Messaging.MessageQueue.ReceiveCurrent(TimeSpan timeout, Int32 action, CursorHandle cursor, MessagePropertyFilter filter, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
at System.Messaging.MessageQueue.Receive(TimeSpan timeout, MessageQueueTransactionType transactionType)
at JobManagerLib.JobProcessor.Process(Action waitForNewMessageCallback) in C:\Dev\OtherProjects\POC\WindowsService\JobManagerSample\JobManagerLib\JobProcessor.cs:line 132
I've tried DTCPing, which succeeds in one direction but fails in the other. Here is the pertinent part of the log:
++++++++++++hosts ++++++++++++
127.0.0.1 localhost
::1 localhost
08-20, 15:47:22.739-->Error(0x424) at clutil.cpp #256
08-20, 15:47:22.739-->-->OpenCluster
08-20, 15:47:22.739-->-->1060(The specified service does not exist as an installed service.)
++++++++++++++++++++++++++++++++++++++++++++++
DTCping 1.9 Report for DEV-MSMQ2
++++++++++++++++++++++++++++++++++++++++++++++
RPC server is ready
++++++++++++Validating Remote Computer Name++++++++++++
08-20, 15:47:26.207-->Start DTC connection test
Name Resolution:
dev-msmq1-->192.168.22.11-->Dev-msmq1
08-20, 15:47:26.222-->Start RPC test (DEV-MSMQ2-->dev-msmq1)
RPC test failed
Does anybody have any idea why this might be failing? The Windows Firewall has been opened for MSDTC. It's hard to find much info about Windows 2008 and MSMQ.
EDIT: The queue names are FormatName:DIRECT=OS:dev-msmq1\getmap, and FormatName:DIRECT=OS:dev-msmq1\logevent. They are public, transactional queues, and Everyone has peek/receive permission on them. The pertinent part of my code is as follows:
using (TransactionScope tx = new TransactionScope(TransactionScopeOption.RequiresNew))
{
using (var queue = new MessageQueue(QueueName))
{
queue.Formatter = new XmlMessageFormatter(new string[] { _targetParameterType });
var message = queue.Receive(TimeOut, MessageQueueTransactionType.Automatic);
string messageId = message.Label;
...
}
}
Thanks
So I did find a solution: abandon the whole thing altogether and switch to using WCF and the net.Msmq binding. Now the queue communication is working fine.
Make sure, both machine's clocks are in sync. I seen this before where authentication will fail because the server and the client are off time by a minute. This will happen even if the queues are public and have permissions on everyone.
Just for completeness, allowing only MSDTC and MSMQ through the firewall is not enough when running IPv4:
You need to let ICMP traffic through the firewall as well (IPv6 can resolve host names regardless of the firewall, but your DTCPIng log indicates you are running IPv4).
I have struggled with the same error you see in DTCPing, and in my case it turned out to be triggered by the firewall blocking the ICMP traffic.
My problem is that I want to have a server application (on a remote computer) to publish certain events to several client computers. The server and client communicate using .Net-Remoting so currently I am using remoted .Net-Events to get the functionality. But there is one drawback: when the server (the event publisher) comes offline and is restarted, the clients lose the connection since the remote object references become invalid.
I am looking into Loosely Coupled Events and Transient COM Subscriptions to solve this issue. I put together a small demo application with one publisher and two subscribers. It works beautifully on one computer.
I am using the COMAdmin-Libraries to create a transient subscription for the event subscribers. The code looks like this:
MyEventHandler handler = new MyEventHandler();
ICOMAdminCatalog catalog;
ICatalogCollection transientCollection;
ICatalogObject subscription;
catalog = (ICOMAdminCatalog)new COMAdminCatalog();
transientCollection = (ICatalogCollection)catalog.GetCollection("TransientSubscriptions");
subscription = (ICatalogObject)transientCollection.Add();
subscription.set_Value("Name", "SubTrans");
subscription.set_Value("SubscriberInterface", handler);
string eventClassString = "{B57E128F-DB28-451b-99D3-0F81DA487EDE}";
subscription.set_Value("EventCLSID", eventClassString);
string sinkString = "{9A616A06-4F8D-4fbc-B47F-482C24A04F35}";
subscription.set_Value("InterfaceID", sinkString);
subscription.set_Value("FilterCriteria", "");
subscription.set_Value("PublisherID", "");
transientCollection.SaveChanges();
handler.Event1 += OnEvent1;
handler.Event2 += OnEvent2;
My question now is: what do I have to change in the subscription to make this work over a network? Is it even possible?
What about MSMQ? It seems perfect for what you are trying to achieve? You can use a traditional publish/subscribe model or multicast the messages.
This might be a step too far, but have you considered using WCF and the callback element of WCF?
Callback effectively turns the what was client into a server. To be honest, I don't know a great deal about callback and have only experimented. Perhaps worth a 10 minute google though.
If your server comes offline every once and a while I cannot see how you can avoid to poll it to check that it is alive.
As you are talking about COM and remote computers, I suspect you'll have to do some DCOM security configuration.