Recently updated to current library 12.8 for azure queue processing.
Inserted message no longer work on existing routines as they are encoded as UTF-8 vs Base 64.
found the thread talking about this and see that MS has implemented a new method to set encoding.
https://github.com/Azure/azure-sdk-for-net/issues/10242
I am unable to set the encoding however and just need a push in the right direction.
This is a .NEt 4.8 Console Application
code I am currently using:
private static void insertQueueMessage(string messageToInsert, string queueName)
{
// Get the connection string from app settings
string connectionString = ConfigurationManager.AppSettings["StorageConnectionString"];
// Instantiate a QueueClient which will be used to create and manipulate the queue
QueueClient queueClient = new QueueClient(connectionString, queueName);
// Send a message to the queue
queueClient.SendMessage(messageToInsert);
}
What I have tried:
queueClient.SendMessage(messageToInsert,QueueMessageEncoding.Base64);
and
QueueClient queueClient = new QueueClient(connectionString, queueName,QueueMessageEncoding.Base64);
How do I code this to work?
Stupid easy answer, I feel like a dolt missing this.
QueueClient queueClient = new QueueClient(connectionString, queueName, new QueueClientOptions
{
MessageEncoding = QueueMessageEncoding.Base64
});
Related
I have a basic producer app and a consumer app. if I run both and have both start consuming on their respective topics, I have a great working system. My thought was that if I started the producer and sent a message that I would be able to then start the consumer and have it pick up that message. I was wrong.
Unless both are up and running, I lose messages (or they do not get consumed).
my consumer app looks like this for comsuming...
Uri uri = new Uri("http://localhost:9092");
KafkaOptions options = new KafkaOptions(uri);
BrokerRouter brokerRouter = new BrokerRouter(options);
Consumer consumer = new Consumer(new ConsumerOptions(receiveTopic, brokerRouter));
List<OffsetResponse> offset = consumer.GetTopicOffsetAsync(receiveTopic, 100000).Result;
IEnumerable<OffsetPosition> t = from x in offset select new OffsetPosition(x.PartitionId, x.Offsets.Max());
consumer.SetOffsetPosition(t.ToArray());
IEnumerable<KafkaNet.Protocol.Message> msgs = consumer.Consume();
foreach (KafkaNet.Protocol.Message msg in msgs)
{
do some stuff here based on the message received
}
unless I have the code between the lines, it starts at the beginning every time I start the application.
What is the proper way to manage topic offsets so messages are consumed after a disconnect happens?
If I run
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic chat-message-reply-XXX consumer-property fetch-size=40000000 --from-beginning
I can see the messages, but when I connect my application to that topic, the consumer.Consume() does not pick up the messages it has not already seen. I have tried this with and without runing the above bat file to see if that makes any difference. When I look at the consumer.SetOffsetPosition(t.ToArray()) call (t specifically) it shows that the offset is the count of all messages for the topic.
Please help,
Set auto.offset.reset configuration in your ConsumerOptions to earliest. When the consumer group starts the consume messages, it will consume from the latest offset because the default value for auto.offset.reset is latest.
But I looked at kafka-net API now, it does not have a AutoOffsetReset property, and it seems pretty insufficient with its configuration in consumers. It also lacks documentation with method summaries.
I would suggest you use Confluent .NET Kafka Nuget package because it is owned by Confluent itself.
Also, why are calling GetTopicOffsets and setting that offset back again in consumer. I think when you configure your consumer, you should just start reading messages with Consume().
Try this:
static void Main(string[] args)
{
var uri = new Uri("http://localhost:9092");
var kafkaOptions = new KafkaOptions(uri);
var brokerRouter = new BrokerRouter(kafkaOptions);
var consumerOptions = new ConsumerOptions(receivedTopic, brokerRouter);
var consumer = new Consumer(consumerOptions);
foreach (var msg in consumer.Consume())
{
var value = Encoding.UTF8.GetString(msg.Value);
// Process value here
}
}
In addition, enable logs in your KafkaOptions and ConsumerOptions, they will help you a lot:
var kafkaOptions = new KafkaOptions(uri)
{
Log = new ConsoleLog()
};
var consumerOptions = new ConsumerOptions(topic, brokerRouter)
{
Log = new ConsoleLog()
});
I switched over to use Confluent's C# .NET package and it now works.
I'm writing an application that uses WebSphere MQ for messaging. For my unittests (flowtests), I want to verify that I have put the right messages on the response queue.
I'm trying to figure out how to do this. My main obstacle is that I think it might be scary to clear a queue before I run my unittest, because the same queue might be used by another application.
I thought a decent workaround would be to create a new queue manager and queue for my unittest and delete it after using it.
So my question is: Is it possible to create a queue manager and queue using C#?
For future reference and future people who want to create queues. I figured out how to create and delete IBM MQ queues (not queuemanagers) with PCF messaging. It is not very straightforward, but it can be done.
We have implemented it in a library and are using it to create and delete queues before and after integration tests respectivally. The most important part of code in this library is shown in the code sample below. Just add a reference to amqmdnet.dll and below code will create a queue and delete it.
string queueManagerName = "QM_LOCAL";
string queueName = "DeleteMeQueue";
Hashtable options = new Hashtable();
// This is a connection to a local server. For a remote server use 'TRANSPORT_MQSERIES_CLIENT', 'TRANSPORT_MQSERIES_XACLIENT' or 'TRANSPORT_MQSERIES_MANAGED'
options.Add(IBM.WMQ.MQC.TRANSPORT_PROPERTY, "TRANSPORT_MQSERIES_BINDINGS");
// For 'TRANSPORT_MQSERIES_CLIENT', 'TRANSPORT_MQSERIES_XACLIENT' or 'TRANSPORT_MQSERIES_MANAGED' uncomment the below
// string hostName = "RemoteServerName";
// string channelName = "SYSTEM.ADMIN.SVRCONN";
// int portNumber = 1414;
// options.Add(IBM.WMQ.MQC.HOST_NAME_PROPERTY, hostName);
// options.Add(IBM.WMQ.MQC.CHANNEL_PROPERTY, channelName);
// options.Add(IBM.WMQ.MQC.PORT_PROPERTY, portNumber);
// options.Add(IBM.WMQ.MQC.CONNECT_OPTIONS_PROPERTY, IBM.WMQ.MQC.MQC.MQCNO_STANDARD_BINDING);
IBM.WMQ.MQQueueManager queueManager = null;
IBM.WMQ.PCF.PCFMessageAgent agent = null;
try
{
// Initialize a connection to the (remote) queuemanager and a PCF message agent.
queueManager = new IBM.WMQ.MQQueueManager(queueManagerName, options);
agent = new IBM.WMQ.PCF.PCFMessageAgent(queueManager);
// Create queue
IBM.WMQ.PCF.PCFMessage createRequest = new IBM.WMQ.PCF.PCFMessage(IBM.WMQ.PCF.CMQCFC.MQCMD_CREATE_Q);
createRequest.AddParameter(IBM.WMQ.MQC.MQCA_Q_NAME, queueName);
createRequest.AddParameter(IBM.WMQ.MQC.MQIA_Q_TYPE, IBM.WMQ.MQC.MQQT_LOCAL);
createRequest.AddParameter(IBM.WMQ.MQC.MQIA_DEF_PERSISTENCE, IBM.WMQ.MQC.MQPER_PERSISTENT);
createRequest.AddParameter(IBM.WMQ.MQC.MQCA_Q_DESC, "Created by " + Environment.UserName + " on " + DateTime.UtcNow.ToString("o"));
IBM.WMQ.PCF.PCFMessage[] createResponses = agent.Send(createRequest);
// Delete queue
IBM.WMQ.PCF.PCFMessage deleteRequest = new IBM.WMQ.PCF.PCFMessage(IBM.WMQ.PCF.CMQCFC.MQCMD_DELETE_Q);
deleteRequest.AddParameter(IBM.WMQ.MQC.MQCA_Q_NAME, queueName);
IBM.WMQ.PCF.PCFMessage[] deleteResponses = agent.Send(deleteRequest);
}
finally
{
// Disconnect the agent and queuemanager.
if (agent != null) agent.Disconnect();
if (queueManager != null && queueManager.IsConnected) queueManager.Disconnect();
}
Creation of queue manager and queues are administrative jobs. Creation of queue manager can not be done using an user defined application. You have to use the command crtmqm <qmname> provided by MQ to create queue manager.
I would suggest you ask your queue manager administrator to create dedicated queue for you. Only your unit test use this queue and no other user is allowed to put/get messages to this queue.
I've read in the Active MQ documentation that Temporary Queues are deleted by the broker when the connection that was used to create them is closed.
I'm using Apache NMS v1.5.0 and Active MQ 5.1.3, and temporary queues are always persisting even after the connection has gone out of scope.
I have a client / server scenario, whereby the client creates a temp queue and creates a message, specifying the temp queue in the ReplyTo property of the message.
The server component then reads the message and starts sending messages to the reply to queue.
Unfortunately, when the client closes it's connection, the temporary queue it created does not get deleted.
The following code snippet should illustrate what i mean.
I create one connection, and create a temporary queue using that connection.
I close the connection and create a second one.
I should not be able to produce and consume messages on the temporary queue using a session
created by the second connection, and yet i can.
Can someone tell me if i'm doing something wrong here. How can i get Active MQ to delete the Temporary Queue.
Any help much appreciated.
[Test]
public void TempQueueTest()
{
var cf = new ConnectionFactory("tcp://activemq-broker:61616");
using (var connection = cf.CreateConnection())
{
connection.Start();
using (var session = connection.CreateSession())
{
var normalQueue = session.GetQueue("normalQueue");
ITemporaryQueue tempQueue = session.CreateTemporaryQueue();
using (var producer = session.CreateProducer(normalQueue))
{
// create a messasge and put on a normal queue
//specify the temp queue as the reply to queue
var mesage = new ActiveMQTextMessage("hello");
mesage.ReplyTo = tempQueue as ActiveMQDestination;
producer.Send(mesage);
}
}
connection.Stop();
}
// ok, connection has been disposed, so the temp queue should be destroyed
// create a new connection
using (var connection = cf.CreateConnection())
{
connection.Start();
using (var session = connection.CreateSession())
{
var normalQueue = session.GetQueue("normalQueue");
using (var consumer = session.CreateConsumer(normalQueue))
{
var message = consumer.Receive() as ActiveMQTextMessage;
// replyToDest is the temp queue created with the previous connection
var replyToDest = message.ReplyTo;
using (var producer = session.CreateProducer(replyToDest))
{
// i shouldn't be able to send a message to this temp queue
producer.Send(new ActiveMQTextMessage("this shouldn't work"));
}
using (var tempConsumer = session.CreateConsumer(replyToDest))
{
// is shouldn't be able to receive messages on the temp queue as it should be destroyed
var message1 = tempConsumer.Receive() as ActiveMQTextMessage;
}
}
}
connection.Stop();
}
}
Given the ancient versions you are using I don't know there's any way fix what's going on here. The code looks correct but there were a large number of fixes between the v1.5.0 release of the NMS libs and the current 1.6.0 many of which fixed issues with Temp Destinations. I'd suggest you try and move on to later versions to see if your problem goes away.
Right now you'd probably have to use JMX to access the broker and remove old temp destinations.
I have been attempting to start an instance of EC2 in C# without luck.
When passing in an instance id to start the instance I get an error that the instance cannot be found despite that I am passing in an instance ID that I have obtained from the object property.
Amazon made huge efforts to integrate its AWS Cloud .Net SDK To VS2008 & VS 2010
1 - Download and Install the AWS SDK msi
2 - Create an AWS Console project, enter your credentials (available from your AWS Console under your login name menu on the top right corner)
3 - Add the following code (see below images).
4 - Your're done. It's very straightforward. You can check the programmatic start/stop success by refreshing your AWS Console Screen.
AmazonEC2 ec2 = AWSClientFactory.CreateAmazonEC2Client();
//Start Your Instance
ec2.StartInstances(new StartInstancesRequest().WithInstanceId("i-00000000"));
//Stop it
ec2.StopInstances(new StopInstancesRequest().WithInstanceId("i-00000000"));
You just need to replace "i-00000000" by your instance Id (available in your AWS Management Console)
Hope this helps those googling this and stumbling upon this question (as I did myself) start off quickly. Following these simple steps via these wizards will spare you considerable headaches.
Try something like this with the AWSSDK to start new instances of an "image id":
RunInstancesResponse response = Client.RunInstances(new RunInstancesRequest()
.WithImageId(ami_id)
.WithInstanceType(instance_type)
.WithKeyName(YOUR_KEYPAIR_NAME)
.WithMinCount(1)
.WithMaxCount(max_number_of_instances)
.WithUserData(Convert.ToBase64String(Encoding.UTF8.GetBytes(bootScript.Replace("\r", ""))))
);
(Note: The .WithUserData() is optional and is used above to pass a short shell script.)
If the call is successful the response should contain a list of instances. You can use something like this to create a list of "instance ids":
if (response.IsSetRunInstancesResult() && response.RunInstancesResult.IsSetReservation() && response.RunInstancesResult.Reservation.IsSetRunningInstance())
{
List<string> instance_ids = new List<string>();
foreach (RunningInstance ri in response.RunInstancesResult.Reservation.RunningInstance)
{
instance_ids.Add(ri.InstanceId);
}
// do something with instance_ids
...
}
Be mindful that Amazon AWS instances exist only in one region. If your instance id i-12345 is in the EU-West-1 region, and you just make a new EC2Client and tell the client to start i-12345 it may well complain that it cannot find that instance, because the client started up in the us-east-1 region, which does not have i-12345 instance.
Your call that creates the client should specify the region, if it is not the default region (I've no idea which AWS region is default, so I specify every time):
AmazonEC2 ec2 = AWSClientFactory.CreateAmazonEC2Client(
new Amazon.EC2.AmazonEC2Config().WithServiceURL("https://eu-west-1.ec2.amazonaws.com")
);
Ok, this is the FULL, end-to-end instructions.
1. Install AWSSDK.Core and AWSSDK.EC2 using Nuget Package Manager.
2. Then copy this whole class to your project. AccessKey and Secret are obtained in AWS IAM. You will need to ensure the user you create has "AmazonEC2FullAccess" (You can probably use a lower-level permission policy, I am just lazy here :D). region is your AW S EC2 instance region. and Instance ID can be found in the EC2 dashboard list. Simple, works perfectly... You can also write extra code to manage the response object.
3. Be mindful if you are behind a proxy, you will have to configure it (I havent included code here).
public class AWSClass : IDisposable
{
Amazon.EC2.AmazonEC2Client _client;
public AWSClass(string region, string AccessKey, string Secret)
{
RegionEndpoint EndPoint = RegionEndpoint.GetBySystemName(region);
Amazon.Runtime.BasicAWSCredentials Credentials = new Amazon.Runtime.BasicAWSCredentials(AccessKey, Secret);
_client = new AmazonEC2Client(Credentials, EndPoint);
}
public void Dispose()
{
_client = null;
}
public void StopInstance(string InstanceID)
{
StopInstancesResponse response = _client.StopInstances(new StopInstancesRequest
{
InstanceIds = new List<string> {InstanceID }
});
//Can also do something with the response object too
}
public void StartInstance(string InstanceID)
{
StartInstancesResponse response = _client.StartInstances(new StartInstancesRequest
{
InstanceIds = new List<string> { InstanceID }
});
}
}
try this.
var startRequest = new StartInstancesRequest
{
InstanceIds = new List<string>() { instanceId }
};
bool isError = true;
StartInstancesResponse startInstancesResponse = null;
while (isError)
{
try
{
startInstancesResponse=amazonEc2client.StartInstances(startRequest);
isError = false;
}
catch (Exception ex)
{
Console.WriteLine(ex.Message + "\n" + ex.StackTrace);
isError = true;
}
}
I've installed the M4 release of the Apache Qpid Java broker on a Windows box, and started it using the out-of-the-box configuration (via the qpid-server.bat script).
I'm now trying to publish a message to a queue using the RabbitMQ C# client library (version 1.5.3, compiled for .NET 3.0); my code is:
public void PublishMessage(string message)
{
ConnectionFactory factory = new ConnectionFactory();
factory.Parameters.VirtualHost = "...";
IProtocol protocol = Protocols.FromEnvironment();
using (IConnection conn = factory.CreateConnection(protocol, "localhost", 5672))
{
using (IModel ch = conn.CreateModel())
{
string exchange = "...";
string routingKey = "...";
ch.BasicPublish(exchange, routingKey, null, Encoding.UTF8.GetBytes(message));
}
}
}
Basically, I'm unsure what values to use for factory.Parameters.VirtualHost and the strings exchange and routingKey. I've tried various combinations, but nothing seems to work - the closest I've got is seeing the following in the Qpid server log:
2009-03-19 17:11:04,248 WARN [pool-1-thread-1] queue.IncomingMessage (IncomingMessage.java:198) - MESSAGE DISCARDED: No routes for message - Message[(HC:896033 ID:1 Ref:1)]: 1; ref count: 1
which looks as though the Qpid server is receiving the message, but doesn't know what to do with it.
Any advice on what configuration values I need in my client code (bearing in mind I'm using the default Qpid config in virtualhosts.xml) would be much appreciated. More general information on virtual hosts, exchanges, queues and routing keys, and how Qpid links them all together, would also be very useful.
Thank you in advance,
Alan
Just for reference, I managed to get this working in the end. The code below sends a message to the queue test-queue in the test.direct exchange on the localhost virtual host (all part of the default Qpid broker configuration):
public void PublishMessage(string message)
{
ConnectionFactory factory = new ConnectionFactory();
factory.Parameters.VirtualHost = "/localhost";
IProtocol protocol = Protocols.AMQP_0_8_QPID;
using (IConnection conn = factory.CreateConnection(protocol, "localhost", 5672))
{
using (IModel ch = conn.CreateModel())
{
ch.ExchangeDeclare("test.direct", "direct");
ch.QueueDeclare("test-queue");
ch.QueueBind("test-queue", "test.direct", "TEST", false, null);
ch.BasicPublish("test.direct", "TEST", null, Encoding.UTF8.GetBytes(message));
}
}
}