I've read in the Active MQ documentation that Temporary Queues are deleted by the broker when the connection that was used to create them is closed.
I'm using Apache NMS v1.5.0 and Active MQ 5.1.3, and temporary queues are always persisting even after the connection has gone out of scope.
I have a client / server scenario, whereby the client creates a temp queue and creates a message, specifying the temp queue in the ReplyTo property of the message.
The server component then reads the message and starts sending messages to the reply to queue.
Unfortunately, when the client closes it's connection, the temporary queue it created does not get deleted.
The following code snippet should illustrate what i mean.
I create one connection, and create a temporary queue using that connection.
I close the connection and create a second one.
I should not be able to produce and consume messages on the temporary queue using a session
created by the second connection, and yet i can.
Can someone tell me if i'm doing something wrong here. How can i get Active MQ to delete the Temporary Queue.
Any help much appreciated.
[Test]
public void TempQueueTest()
{
var cf = new ConnectionFactory("tcp://activemq-broker:61616");
using (var connection = cf.CreateConnection())
{
connection.Start();
using (var session = connection.CreateSession())
{
var normalQueue = session.GetQueue("normalQueue");
ITemporaryQueue tempQueue = session.CreateTemporaryQueue();
using (var producer = session.CreateProducer(normalQueue))
{
// create a messasge and put on a normal queue
//specify the temp queue as the reply to queue
var mesage = new ActiveMQTextMessage("hello");
mesage.ReplyTo = tempQueue as ActiveMQDestination;
producer.Send(mesage);
}
}
connection.Stop();
}
// ok, connection has been disposed, so the temp queue should be destroyed
// create a new connection
using (var connection = cf.CreateConnection())
{
connection.Start();
using (var session = connection.CreateSession())
{
var normalQueue = session.GetQueue("normalQueue");
using (var consumer = session.CreateConsumer(normalQueue))
{
var message = consumer.Receive() as ActiveMQTextMessage;
// replyToDest is the temp queue created with the previous connection
var replyToDest = message.ReplyTo;
using (var producer = session.CreateProducer(replyToDest))
{
// i shouldn't be able to send a message to this temp queue
producer.Send(new ActiveMQTextMessage("this shouldn't work"));
}
using (var tempConsumer = session.CreateConsumer(replyToDest))
{
// is shouldn't be able to receive messages on the temp queue as it should be destroyed
var message1 = tempConsumer.Receive() as ActiveMQTextMessage;
}
}
}
connection.Stop();
}
}
Given the ancient versions you are using I don't know there's any way fix what's going on here. The code looks correct but there were a large number of fixes between the v1.5.0 release of the NMS libs and the current 1.6.0 many of which fixed issues with Temp Destinations. I'd suggest you try and move on to later versions to see if your problem goes away.
Right now you'd probably have to use JMX to access the broker and remove old temp destinations.
Related
I have a basic producer app and a consumer app. if I run both and have both start consuming on their respective topics, I have a great working system. My thought was that if I started the producer and sent a message that I would be able to then start the consumer and have it pick up that message. I was wrong.
Unless both are up and running, I lose messages (or they do not get consumed).
my consumer app looks like this for comsuming...
Uri uri = new Uri("http://localhost:9092");
KafkaOptions options = new KafkaOptions(uri);
BrokerRouter brokerRouter = new BrokerRouter(options);
Consumer consumer = new Consumer(new ConsumerOptions(receiveTopic, brokerRouter));
List<OffsetResponse> offset = consumer.GetTopicOffsetAsync(receiveTopic, 100000).Result;
IEnumerable<OffsetPosition> t = from x in offset select new OffsetPosition(x.PartitionId, x.Offsets.Max());
consumer.SetOffsetPosition(t.ToArray());
IEnumerable<KafkaNet.Protocol.Message> msgs = consumer.Consume();
foreach (KafkaNet.Protocol.Message msg in msgs)
{
do some stuff here based on the message received
}
unless I have the code between the lines, it starts at the beginning every time I start the application.
What is the proper way to manage topic offsets so messages are consumed after a disconnect happens?
If I run
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic chat-message-reply-XXX consumer-property fetch-size=40000000 --from-beginning
I can see the messages, but when I connect my application to that topic, the consumer.Consume() does not pick up the messages it has not already seen. I have tried this with and without runing the above bat file to see if that makes any difference. When I look at the consumer.SetOffsetPosition(t.ToArray()) call (t specifically) it shows that the offset is the count of all messages for the topic.
Please help,
Set auto.offset.reset configuration in your ConsumerOptions to earliest. When the consumer group starts the consume messages, it will consume from the latest offset because the default value for auto.offset.reset is latest.
But I looked at kafka-net API now, it does not have a AutoOffsetReset property, and it seems pretty insufficient with its configuration in consumers. It also lacks documentation with method summaries.
I would suggest you use Confluent .NET Kafka Nuget package because it is owned by Confluent itself.
Also, why are calling GetTopicOffsets and setting that offset back again in consumer. I think when you configure your consumer, you should just start reading messages with Consume().
Try this:
static void Main(string[] args)
{
var uri = new Uri("http://localhost:9092");
var kafkaOptions = new KafkaOptions(uri);
var brokerRouter = new BrokerRouter(kafkaOptions);
var consumerOptions = new ConsumerOptions(receivedTopic, brokerRouter);
var consumer = new Consumer(consumerOptions);
foreach (var msg in consumer.Consume())
{
var value = Encoding.UTF8.GetString(msg.Value);
// Process value here
}
}
In addition, enable logs in your KafkaOptions and ConsumerOptions, they will help you a lot:
var kafkaOptions = new KafkaOptions(uri)
{
Log = new ConsoleLog()
};
var consumerOptions = new ConsumerOptions(topic, brokerRouter)
{
Log = new ConsoleLog()
});
I switched over to use Confluent's C# .NET package and it now works.
I'm having an issue with ActiveMQ, I'm trying to connect using MaxReconnectAttemps but its seems to ignore the property. I'm putting an invalid destination so it tries to connect twice but it seems to be trying to connect indefinitely.
Any ideas as to set it up?
Thanks,
IConnectionFactory factory = new ConnectionFactory(("failover://(tcp://localhost:61616)?initialReconnectDelay=2000&maxReconnectAttempts=2"));
using (Connection connection = factory.CreateConnection(username,password) as Connection)
{
connection.ClientId = "ClientId";
using (ISession session = connection.CreateSession())
{
IQueue queue = session.GetQueue(queueName);
var producer = session.CreateProducer(queue);
producer.DeliveryMode = MsgDeliveryMode.Persistent;
ITextMessage request = session.CreateTextMessage("Hello World!");
producer.Send(request);
}
}
Since you are using the .NET client you need to use a prefix on the URI options for the failover transport, so to configure maxReconnectAttempts you need to pass the option like this:
failover:(tcp://localhost:61616)?transport.maxReconnectAttempts=3
It's a good idea to look at the documentation for the client you are using which is here.
I have four application server for my application.Application is working on all server using load balancing.If one of my server goes down I have to check it manually using my system hosts file.To avoid this manual process I have created one program using C#.I write server IP address one by one in host file and remove previous one.
private void RunWithUAC()
{
List<string> lstIPAddress = new List<string>();
lstIPAddress.Add("1.1.1.1 example.com");
lstIPAddress.Add("1.1.1.1 example.com");
lstIPAddress.Add("1.1.1.1 example.com");
lstIPAddress.Add("1.1.1.1 example.com");
var systemPath = Environment.GetFolderPath(Environment.SpecialFolder.System);
Console.WriteLine(systemPath);
var path = #"C:\Windows\System32\drivers\etc\hosts";
foreach (var item in lstIPAddress)
{
System.IO.File.WriteAllText(path, string.Empty);
try
{
File.WriteAllText(path, item);
WebRequest request = WebRequest.Create("https://example.com");
request.Timeout = 10000;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
}
catch (Exception)
{
MessageBox.Show(item);
}
Thread.Sleep(1000);
}
}
But When second server goes down.It will give me timeout error for third server.
Please check the code and let me know what is wrong with this code.
Probably some kind of connection pooling, HTTP pipelining or keep-alive. This is the wrong approach in the first place.
Connect directly to the right IP (WebRequest.Create("https://1.1.1.1")). If you need to send a Host header add that manually to the request.
I'm writing an application that uses WebSphere MQ for messaging. For my unittests (flowtests), I want to verify that I have put the right messages on the response queue.
I'm trying to figure out how to do this. My main obstacle is that I think it might be scary to clear a queue before I run my unittest, because the same queue might be used by another application.
I thought a decent workaround would be to create a new queue manager and queue for my unittest and delete it after using it.
So my question is: Is it possible to create a queue manager and queue using C#?
For future reference and future people who want to create queues. I figured out how to create and delete IBM MQ queues (not queuemanagers) with PCF messaging. It is not very straightforward, but it can be done.
We have implemented it in a library and are using it to create and delete queues before and after integration tests respectivally. The most important part of code in this library is shown in the code sample below. Just add a reference to amqmdnet.dll and below code will create a queue and delete it.
string queueManagerName = "QM_LOCAL";
string queueName = "DeleteMeQueue";
Hashtable options = new Hashtable();
// This is a connection to a local server. For a remote server use 'TRANSPORT_MQSERIES_CLIENT', 'TRANSPORT_MQSERIES_XACLIENT' or 'TRANSPORT_MQSERIES_MANAGED'
options.Add(IBM.WMQ.MQC.TRANSPORT_PROPERTY, "TRANSPORT_MQSERIES_BINDINGS");
// For 'TRANSPORT_MQSERIES_CLIENT', 'TRANSPORT_MQSERIES_XACLIENT' or 'TRANSPORT_MQSERIES_MANAGED' uncomment the below
// string hostName = "RemoteServerName";
// string channelName = "SYSTEM.ADMIN.SVRCONN";
// int portNumber = 1414;
// options.Add(IBM.WMQ.MQC.HOST_NAME_PROPERTY, hostName);
// options.Add(IBM.WMQ.MQC.CHANNEL_PROPERTY, channelName);
// options.Add(IBM.WMQ.MQC.PORT_PROPERTY, portNumber);
// options.Add(IBM.WMQ.MQC.CONNECT_OPTIONS_PROPERTY, IBM.WMQ.MQC.MQC.MQCNO_STANDARD_BINDING);
IBM.WMQ.MQQueueManager queueManager = null;
IBM.WMQ.PCF.PCFMessageAgent agent = null;
try
{
// Initialize a connection to the (remote) queuemanager and a PCF message agent.
queueManager = new IBM.WMQ.MQQueueManager(queueManagerName, options);
agent = new IBM.WMQ.PCF.PCFMessageAgent(queueManager);
// Create queue
IBM.WMQ.PCF.PCFMessage createRequest = new IBM.WMQ.PCF.PCFMessage(IBM.WMQ.PCF.CMQCFC.MQCMD_CREATE_Q);
createRequest.AddParameter(IBM.WMQ.MQC.MQCA_Q_NAME, queueName);
createRequest.AddParameter(IBM.WMQ.MQC.MQIA_Q_TYPE, IBM.WMQ.MQC.MQQT_LOCAL);
createRequest.AddParameter(IBM.WMQ.MQC.MQIA_DEF_PERSISTENCE, IBM.WMQ.MQC.MQPER_PERSISTENT);
createRequest.AddParameter(IBM.WMQ.MQC.MQCA_Q_DESC, "Created by " + Environment.UserName + " on " + DateTime.UtcNow.ToString("o"));
IBM.WMQ.PCF.PCFMessage[] createResponses = agent.Send(createRequest);
// Delete queue
IBM.WMQ.PCF.PCFMessage deleteRequest = new IBM.WMQ.PCF.PCFMessage(IBM.WMQ.PCF.CMQCFC.MQCMD_DELETE_Q);
deleteRequest.AddParameter(IBM.WMQ.MQC.MQCA_Q_NAME, queueName);
IBM.WMQ.PCF.PCFMessage[] deleteResponses = agent.Send(deleteRequest);
}
finally
{
// Disconnect the agent and queuemanager.
if (agent != null) agent.Disconnect();
if (queueManager != null && queueManager.IsConnected) queueManager.Disconnect();
}
Creation of queue manager and queues are administrative jobs. Creation of queue manager can not be done using an user defined application. You have to use the command crtmqm <qmname> provided by MQ to create queue manager.
I would suggest you ask your queue manager administrator to create dedicated queue for you. Only your unit test use this queue and no other user is allowed to put/get messages to this queue.
I have win service that work with MQ.
But i want that it works using ssl channel and database with public/private keys(for that)
May you explain me how to do it.
P.S. I'm not very good at MQ
now i connect to MQ using this code
MQEnvironment.Hostname = ConfigurationManager.AppSettings["HostnameIN"];
MQEnvironment.Channel = ConfigurationManager.AppSettings["ChannelIN"];
MQEnvironment.Port = int.Parse(ConfigurationManager.AppSettings["PortIN"]);
Environment.SetEnvironmentVariable("MQCCSID", ConfigurationManager.AppSettings["MQCCSID"]);
var mqQueueManagerName = ConfigurationManager.AppSettings["QueueManagerNameIN"];
var mqQueueName = ConfigurationManager.AppSettings["QueueNameIN"];
const int openOptions = MQC.MQOO_BROWSE | MQC.MQOO_INPUT_AS_Q_DEF;
var qMgr = new MQQueueManager(mqQueueManagerName);
var getOptions = new MQGetMessageOptions();
and get all messages using this
using (var mqQueue = qMgr.AccessQueue(mqQueueName, openOptions))
{
try
{
//while (mqQueue.CurrentDepth>0)
while (true)
{
var message = new MQMessage();
//message.Version = 2;
getOptions.Options = MQC.MQGMO_WAIT | MQC.MQGMO_BROWSE_NEXT;
mqQueue.Get(message, getOptions);
mqMessages.Add(message);
}
}
In order to set up MQ to use SSL on the channel you're using, you don't need to make any application changes at all - you simply need to configure the channel you're using on the queue manager to require SSL. The libraries within the client, JVM, and the queue manager will handle establishing that secure connection for you. So in theory all you need to do is make the MQSC/MQ Explorer changes which will configure SSL on the channel.
Recommend you read the following page in the IBM knowledge center. It provides a number of scenarios for various methods of connecting a client securely to the queue manager:
http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.sce.doc/q014220_.htm