Develop Reference

need help setting QueueMessageEncoding.Base64 for new Azure queueClient version

Recently updated to current library 12.8 for azure queue processing.
Inserted message no longer work on existing routines as they are encoded as UTF-8 vs Base 64.
found the thread talking about this and see that MS has implemented a new method to set encoding.
https://github.com/Azure/azure-sdk-for-net/issues/10242
I am unable to set the encoding however and just need a push in the right direction.
This is a .NEt 4.8 Console Application
code I am currently using:
private static void insertQueueMessage(string messageToInsert, string queueName)
{
// Get the connection string from app settings
string connectionString = ConfigurationManager.AppSettings["StorageConnectionString"];
// Instantiate a QueueClient which will be used to create and manipulate the queue
QueueClient queueClient = new QueueClient(connectionString, queueName);
// Send a message to the queue
queueClient.SendMessage(messageToInsert);
}
What I have tried:
queueClient.SendMessage(messageToInsert,QueueMessageEncoding.Base64);
and
QueueClient queueClient = new QueueClient(connectionString, queueName,QueueMessageEncoding.Base64);
How do I code this to work?

Stupid easy answer, I feel like a dolt missing this.
QueueClient queueClient = new QueueClient(connectionString, queueName, new QueueClientOptions
{
MessageEncoding = QueueMessageEncoding.Base64
});

How does the offset for a topic work in Kafka (Kafka_net)

I have a basic producer app and a consumer app. if I run both and have both start consuming on their respective topics, I have a great working system. My thought was that if I started the producer and sent a message that I would be able to then start the consumer and have it pick up that message. I was wrong.
Unless both are up and running, I lose messages (or they do not get consumed).
my consumer app looks like this for comsuming...
Uri uri = new Uri("http://localhost:9092");
KafkaOptions options = new KafkaOptions(uri);
BrokerRouter brokerRouter = new BrokerRouter(options);
Consumer consumer = new Consumer(new ConsumerOptions(receiveTopic, brokerRouter));
List<OffsetResponse> offset = consumer.GetTopicOffsetAsync(receiveTopic, 100000).Result;
IEnumerable<OffsetPosition> t = from x in offset select new OffsetPosition(x.PartitionId, x.Offsets.Max());
consumer.SetOffsetPosition(t.ToArray());
IEnumerable<KafkaNet.Protocol.Message> msgs = consumer.Consume();
foreach (KafkaNet.Protocol.Message msg in msgs)
{
do some stuff here based on the message received
}
unless I have the code between the lines, it starts at the beginning every time I start the application.
What is the proper way to manage topic offsets so messages are consumed after a disconnect happens?
If I run
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic chat-message-reply-XXX consumer-property fetch-size=40000000 --from-beginning
I can see the messages, but when I connect my application to that topic, the consumer.Consume() does not pick up the messages it has not already seen. I have tried this with and without runing the above bat file to see if that makes any difference. When I look at the consumer.SetOffsetPosition(t.ToArray()) call (t specifically) it shows that the offset is the count of all messages for the topic.
Please help,

Set auto.offset.reset configuration in your ConsumerOptions to earliest. When the consumer group starts the consume messages, it will consume from the latest offset because the default value for auto.offset.reset is latest.
But I looked at kafka-net API now, it does not have a AutoOffsetReset property, and it seems pretty insufficient with its configuration in consumers. It also lacks documentation with method summaries.
I would suggest you use Confluent .NET Kafka Nuget package because it is owned by Confluent itself.
Also, why are calling GetTopicOffsets and setting that offset back again in consumer. I think when you configure your consumer, you should just start reading messages with Consume().
Try this:
static void Main(string[] args)
{
var uri = new Uri("http://localhost:9092");
var kafkaOptions = new KafkaOptions(uri);
var brokerRouter = new BrokerRouter(kafkaOptions);
var consumerOptions = new ConsumerOptions(receivedTopic, brokerRouter);
var consumer = new Consumer(consumerOptions);
foreach (var msg in consumer.Consume())
{
var value = Encoding.UTF8.GetString(msg.Value);
// Process value here
}
}
In addition, enable logs in your KafkaOptions and ConsumerOptions, they will help you a lot:
var kafkaOptions = new KafkaOptions(uri)
{
Log = new ConsoleLog()
};
var consumerOptions = new ConsumerOptions(topic, brokerRouter)
{
Log = new ConsoleLog()
});

I switched over to use Confluent's C# .NET package and it now works.

Is there an MQ/XMS equivalent for the MQ/JMS setTargetClient

I have a XMS publish app, that is working, but it is including JMS headers as part of the message. My subscribe app is actually a python app, and I was wondering if it is possible to remove the JMS headers from the XMS app. I know it is possible in JMS, but is it possible in C# / XMS.
My C# code is fairly simple (with some of the details left out -
// Get an instance of factory.
factoryFactory = XMSFactoryFactory.GetInstance(XMSC.CT_WMQ);
// Create WMQ Connection Factory.
cf = factoryFactory.CreateConnectionFactory();
// Set the properties
cf.SetStringProperty(XMSC.WMQ_HOST_NAME, conn.host);
...
// Create connection.
connectionWMQ = cf.CreateConnection();
// Create session
sessionWMQ = connectionWMQ.CreateSession(false, AcknowledgeMode.AutoAcknowledge);
// Create destination
destination = sessionWMQ.CreateTopic(conn.topic_name);
// Create producer
producer = sessionWMQ.CreateProducer(destination);
// Start the connection to receive messages.
connectionWMQ.Start();
// Create a text message and send it.
textMessage = sessionWMQ.CreateTextMessage();
textMessage.Text = xmsJson.toJsonString();
producer.Send(textMessage);
In MQ /JMS I can use setTargetClient to remove the JMS headers -
private void setToNoJMSHeaders(Destination destination) {
try {
MQDestination mqDestination = (MQDestination) destination;
mqDestination.setTargetClient(WMQConstants.WMQ_CLIENT_NONJMS_MQ);
} catch (JMSException jmsex) {
logger.warning("Unable to set target destination to non JMS");
}
}
I was wondering if I can do the same to the topic destination in XMS
// Create destination
destination = sessionWMQ.CreateTopic(conn.topic_name);
// Configure the destination to Non-JMS
... ???

Yes, you should be able to do this
Try
// Create destination
destination = sessionWMQ.CreateTopic(conn.topic_name);
destination.SetIntProperty(XMSC.WMQ_TARGET_CLIENT, XMSC.WMQ_TARGET_DEST_MQ);
See: https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.1.0/com.ibm.mq.ref.dev.doc/prx_wmq_target_client.htm

Create security connection

I have win service that work with MQ.
But i want that it works using ssl channel and database with public/private keys(for that)
May you explain me how to do it.
P.S. I'm not very good at MQ
now i connect to MQ using this code
MQEnvironment.Hostname = ConfigurationManager.AppSettings["HostnameIN"];
MQEnvironment.Channel = ConfigurationManager.AppSettings["ChannelIN"];
MQEnvironment.Port = int.Parse(ConfigurationManager.AppSettings["PortIN"]);
Environment.SetEnvironmentVariable("MQCCSID", ConfigurationManager.AppSettings["MQCCSID"]);
var mqQueueManagerName = ConfigurationManager.AppSettings["QueueManagerNameIN"];
var mqQueueName = ConfigurationManager.AppSettings["QueueNameIN"];
const int openOptions = MQC.MQOO_BROWSE | MQC.MQOO_INPUT_AS_Q_DEF;
var qMgr = new MQQueueManager(mqQueueManagerName);
var getOptions = new MQGetMessageOptions();
and get all messages using this
using (var mqQueue = qMgr.AccessQueue(mqQueueName, openOptions))
{
try
{
//while (mqQueue.CurrentDepth>0)
while (true)
{
var message = new MQMessage();
//message.Version = 2;
getOptions.Options = MQC.MQGMO_WAIT | MQC.MQGMO_BROWSE_NEXT;
mqQueue.Get(message, getOptions);
mqMessages.Add(message);
}
}

In order to set up MQ to use SSL on the channel you're using, you don't need to make any application changes at all - you simply need to configure the channel you're using on the queue manager to require SSL. The libraries within the client, JVM, and the queue manager will handle establishing that secure connection for you. So in theory all you need to do is make the MQSC/MQ Explorer changes which will configure SSL on the channel.
Recommend you read the following page in the IBM knowledge center. It provides a number of scenarios for various methods of connecting a client securely to the queue manager:
http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.sce.doc/q014220_.htm

ActiveMQ NMS Temporary Queue not destroyed when Connection is closed

I've read in the Active MQ documentation that Temporary Queues are deleted by the broker when the connection that was used to create them is closed.
I'm using Apache NMS v1.5.0 and Active MQ 5.1.3, and temporary queues are always persisting even after the connection has gone out of scope.
I have a client / server scenario, whereby the client creates a temp queue and creates a message, specifying the temp queue in the ReplyTo property of the message.
The server component then reads the message and starts sending messages to the reply to queue.
Unfortunately, when the client closes it's connection, the temporary queue it created does not get deleted.
The following code snippet should illustrate what i mean.
I create one connection, and create a temporary queue using that connection.
I close the connection and create a second one.
I should not be able to produce and consume messages on the temporary queue using a session
created by the second connection, and yet i can.
Can someone tell me if i'm doing something wrong here. How can i get Active MQ to delete the Temporary Queue.
Any help much appreciated.
[Test]
public void TempQueueTest()
{
var cf = new ConnectionFactory("tcp://activemq-broker:61616");
using (var connection = cf.CreateConnection())
{
connection.Start();
using (var session = connection.CreateSession())
{
var normalQueue = session.GetQueue("normalQueue");
ITemporaryQueue tempQueue = session.CreateTemporaryQueue();
using (var producer = session.CreateProducer(normalQueue))
{
// create a messasge and put on a normal queue
//specify the temp queue as the reply to queue
var mesage = new ActiveMQTextMessage("hello");
mesage.ReplyTo = tempQueue as ActiveMQDestination;
producer.Send(mesage);
}
}
connection.Stop();
}
// ok, connection has been disposed, so the temp queue should be destroyed
// create a new connection
using (var connection = cf.CreateConnection())
{
connection.Start();
using (var session = connection.CreateSession())
{
var normalQueue = session.GetQueue("normalQueue");
using (var consumer = session.CreateConsumer(normalQueue))
{
var message = consumer.Receive() as ActiveMQTextMessage;
// replyToDest is the temp queue created with the previous connection
var replyToDest = message.ReplyTo;
using (var producer = session.CreateProducer(replyToDest))
{
// i shouldn't be able to send a message to this temp queue
producer.Send(new ActiveMQTextMessage("this shouldn't work"));
}
using (var tempConsumer = session.CreateConsumer(replyToDest))
{
// is shouldn't be able to receive messages on the temp queue as it should be destroyed
var message1 = tempConsumer.Receive() as ActiveMQTextMessage;
}
}
}
connection.Stop();
}
}

Given the ancient versions you are using I don't know there's any way fix what's going on here. The code looks correct but there were a large number of fixes between the v1.5.0 release of the NMS libs and the current 1.6.0 many of which fixed issues with Temp Destinations. I'd suggest you try and move on to later versions to see if your problem goes away.
Right now you'd probably have to use JMX to access the broker and remove old temp destinations.