Develop Reference

Kafka consumer is not consuming message

I am new in Kafka. kafka consumer is not reading message from the given topic.
I am checking with kafka console as well. it is not working. i donot understand the problem. it was working fine earlier.
public string MessageConsumer(string brokerList, List<string> topics, CancellationToken cancellationToken)
{
//ConfigurationManager.AutoLoadAppSettings("", "", true);
string logKey = string.Format("ARIConsumer.StartPRoducer ==>Topics {0} Key{1} =>", "", string.Join(",", topics));
string message = string.Empty;
var conf = new ConsumerConfig
{
BootstrapServers = "localhost:9092",
GroupId = "23",
EnableAutoCommit = false,
AutoOffsetReset = AutoOffsetResetType.Latest,
};
using (var c = new Consumer<Ignore, string>(conf))
{
try
{
c.Subscribe(topics);
bool consuming = true;
// The client will automatically recover from non-fatal errors. You typically
// don't need to take any action unless an error is marked as fatal.
c.OnError += (_, e) => consuming = !e.IsFatal;
while (consuming)
{
try
{
TimeSpan timeSpan = new TimeSpan(0, 0, 5);
var cr = c.Consume(timeSpan);
// Thread.Sleep(5000);
if (cr != null)
{
message = cr.Value;
Console.WriteLine("Thread" + Thread.CurrentThread.ManagedThreadId + "Message : " + message);
CLogger.WriteLog(ELogLevel.INFO, $"Consumed message Partition '{cr.Partition}' at: '{cr.TopicPartitionOffset} thread: { Thread.CurrentThread.ManagedThreadId}'. Message: {message}");
//Console.WriteLine($"Consumed message Partition '{cr.Partition}' at: '{cr.TopicPartitionOffset}'. Topic: { cr.Topic} value :{cr.Value} Timestamp :{DateTime.UtcNow.ToString("yyyy-MM-dd HH:mm:ss.fff", CultureInfo.InvariantCulture)} GrpId: { conf.GroupId}");
c.Commit();
}
Console.WriteLine($"Calling the next Poll ");
}
catch (ConsumeException e)
{
CLogger.WriteLog(ELogLevel.ERROR, $"Error occured: {e.Error.Reason}");
Console.WriteLine($"Error occured: {e.Error.Reason}");
}
//consuming = false;
}
// Ensure the consumer leaves the group cleanly and final offsets are committed.
c.Close();
}
catch (Exception ex)
{
}
}
return message;
}
What is the issue with this code or there is installation issue with kafka

Is there a Producer actively sending data?
Your consumer is starting from the latest offsets based on the AutoOffsetReset, so it wouldn't read existing data in the topic
The console consumer also defaults to the latest offset
And if you haven't changed the GroupId, then your consumer might have worked once, then you consumed data, then commited the offsets for that group. When the consumer starts again in the same group, it will only resume from the end of the topic, or the offset of the last commit
You also have an empty catch (Exception ex), which might be hiding some other error

Try removing the timespan from consume method.

Specified argument was out of the range of valid values. creationOptions at apnsBroker.Start()

I created an console app with .NET 4.0 like this:
static void Test2(string deviceToken, string message)
{
try
{
//Get Certificate
var appleCert = System.IO.File.ReadAllBytes("Certificates_moi.p12");
// Configuration (NOTE: .pfx can also be used here)
var config = new ApnsConfiguration(ApnsConfiguration.ApnsServerEnvironment.Sandbox, appleCert, "vnpt1234");
// Create a new broker
var apnsBroker = new ApnsServiceBroker(config);
// Wire up events
apnsBroker.OnNotificationFailed += (notification, aggregateEx) =>
{
aggregateEx.Handle(ex =>
{
// See what kind of exception it was to further diagnose
if (ex is ApnsNotificationException)
{
var notificationException = (ApnsNotificationException)ex;
// Deal with the failed notification
var apnsNotification = notificationException.Notification;
var statusCode = notificationException.ErrorStatusCode;
string desc = "Apple Notification Failed: ID={apnsNotification.Identifier}, Code={statusCode}";
Console.WriteLine(desc);
//lblStatus.Text = desc;
}
else
{
string desc = "Apple Notification Failed for some unknown reason : {ex.InnerException}";
// Inner exception might hold more useful information like an ApnsConnectionException
Console.WriteLine(desc);
//lblStatus.Text = desc;
}
// Mark it as handled
return true;
});
};
apnsBroker.OnNotificationSucceeded += (notification) =>
{
//lblStatus.Text = "Apple Notification Sent successfully!";
Console.WriteLine("Apple Notification Sent successfully!");
};
var fbs = new FeedbackService(config);
fbs.FeedbackReceived += (string devicToken, DateTime timestamp) =>
{
// Remove the deviceToken from your database
// timestamp is the time the token was reported as expired
};
// Start Proccess
apnsBroker.Start();
if (deviceToken != "")
{
apnsBroker.QueueNotification(new ApnsNotification
{
DeviceToken = deviceToken,
Payload = JObject.Parse(("{\"aps\":{\"badge\":1,\"sound\":\"oven.caf\",\"alert\":\"" + (message + "\"}}")))
});
}
apnsBroker.Stop();
}
catch (Exception ex)
{
throw;
}
}
There is an error at apnsBroker.Start();
- $exception {"Specified argument was out of the range of valid values.\r\nParameter name: creationOptions"} System.Exception {System.ArgumentOutOfRangeException}
Can anybody help me, please?

I fixed this issue. Move to another machine running .NET 4.5 and it worked.

Reuse MQMessage object

I have this test code that listens for messages forever. If it gets one it prints it out. What I want to do is avoid having to construct a MQMessage object prior to each get(). How do i reuse a MQMessage for multiple calls to get()?
using System;
using IBM.WMQ;
namespace WMQ {
class Program {
static void Main(string[] args) {
string QueueManagerName = "A1PATA00";
string channelName = "ECACHE";
string connectionName = "A1PATA00.WORLDSPAN.COM(1414)";
var queueManager = new MQQueueManager(QueueManagerName, channelName, connectionName);
MQQueue get = queueManager.AccessQueue("SPLASH.ECAC.2", MQC.MQOO_INPUT_AS_Q_DEF | MQC.MQGMO_FAIL_IF_QUIESCING);
var gmo = new MQGetMessageOptions();
gmo.Options = MQC.MQGMO_FAIL_IF_QUIESCING | MQC.MQGMO_WAIT;
gmo.WaitInterval = 10000000;// wait time
// var queueMessage = new MQMessage(); <--- i want to do this new once!
while (true) {
var queueMessage = new MQMessage(); // <-- only works if I do this every time i do a get
get.Get(queueMessage, gmo);
var strReturn = queueMessage.ReadString(queueMessage.MessageLength);
Console.WriteLine(strReturn);
}
}
}
}

When I see questions like this, I shake my head. You do not
understand object oriented concepts in the .NET VM (framework) and
garbage collection.
Also, why don't you write pure C# code?
Finally, in a world where security is very important, your code does
not support either SSL/TLS and/or UserID & Password authentication.
(1) Please notice where I am defining the MQMessage object (very important).
There is basically no difference in memory usage or speed between the following 2 code snippets:
(A)
MQMessage msg = null;
while (true)
{
msg = new MQMessage();
get.Get(msg, gmo);
Console.WriteLine(msg.ReadString(msg.MessageLength));
}
(B)
MQMessage msg = new MQMessage();
while (true)
{
get.Get(msg, gmo);
Console.WriteLine(msg.ReadString(msg.MessageLength));
msg.ClearMessage();
msg.MessageId = MQC.MQMI_NONE;
msg.CorrelationId = MQC.MQCI_NONE;
}
I prefer (A) because it is cleaner and easier to read.
(2) When you use 'var', you are forcing the .NET framework to guess at what you are doing. Do pure object oriented programming.
i.e.
MQMessage msg = new MQMessage();
(3) Explicitly setting the channel name and connection name in the MQQueueManager does not allow setting of MQ security information. Also, do NOT use the MQEnvironment class as it is NOT thread safe. It is far better to put all the information in a Hashtable and pass the Hashtable to the MQQueueManager class. Here is a MQ .NET managed-mode example using a Hashtable for MQ connection information:
using System;
using System.Collections;
using System.Collections.Generic;
using System.Text;
using IBM.WMQ;
namespace MQTest02
{
class MQTest02
{
private Hashtable inParms = null;
private Hashtable qMgrProp = null;
private System.String qManager;
private System.String outputQName;
/*
* The constructor
*/
public MQTest02()
: base()
{
}
/// <summary> Make sure the required parameters are present.</summary>
/// <returns> true/false
/// </returns>
private bool allParamsPresent()
{
bool b = inParms.ContainsKey("-h") && inParms.ContainsKey("-p") &&
inParms.ContainsKey("-c") && inParms.ContainsKey("-m") &&
inParms.ContainsKey("-q");
if (b)
{
try
{
System.Int32.Parse((System.String)inParms["-p"]);
}
catch (System.FormatException e)
{
b = false;
}
}
return b;
}
/// <summary> Extract the command-line parameters and initialize the MQ variables.</summary>
/// <param name="args">
/// </param>
/// <throws> IllegalArgumentException </throws>
private void init(System.String[] args)
{
inParms = Hashtable.Synchronized(new Hashtable());
if (args.Length > 0 && (args.Length % 2) == 0)
{
for (int i = 0; i < args.Length; i += 2)
{
inParms[args[i]] = args[i + 1];
}
}
else
{
throw new System.ArgumentException();
}
if (allParamsPresent())
{
qManager = ((System.String)inParms["-m"]);
outputQName = ((System.String)inParms["-q"]);
qMgrProp = new Hashtable();
qMgrProp.Add(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES_MANAGED);
qMgrProp.Add(MQC.HOST_NAME_PROPERTY, ((System.String)inParms["-h"]));
qMgrProp.Add(MQC.CHANNEL_PROPERTY, ((System.String)inParms["-c"]));
try
{
qMgrProp.Add(MQC.PORT_PROPERTY, System.Int32.Parse((System.String)inParms["-p"]));
}
catch (System.FormatException e)
{
qMgrProp.Add(MQC.PORT_PROPERTY, 1414);
}
if (inParms.ContainsKey("-u"))
qMgrProp.Add(MQC.USER_ID_PROPERTY, ((System.String)inParms["-u"]));
if (inParms.ContainsKey("-x"))
qMgrProp.Add(MQC.PASSWORD_PROPERTY, ((System.String)inParms["-x"]));
if (inParms.ContainsKey("-s"))
qMgrProp.Add(MQC.SECURITY_EXIT_PROPERTY, ((System.String)inParms["-s"]));
System.Console.Out.WriteLine("MQTest02:");
Console.WriteLine(" QMgrName ='{0}'", qManager);
Console.WriteLine(" Output QName ='{0}'", outputQName);
System.Console.Out.WriteLine("QMgr Property values:");
foreach (DictionaryEntry de in qMgrProp)
{
Console.WriteLine(" {0} = '{1}'", de.Key, de.Value);
}
}
else
{
throw new System.ArgumentException();
}
}
/// <summary> Connect, open queue, read a message, close queue and disconnect.
///
/// </summary>
/// <throws> MQException </throws>
private void testReceive()
{
MQQueueManager qMgr = null;
MQQueue queue = null;
int openOptions = MQC.MQOO_INPUT_AS_Q_DEF + MQC.MQOO_FAIL_IF_QUIESCING;
MQGetMessageOptions gmo = new MQGetMessageOptions();
MQMessage receiveMsg = null;
try
{
qMgr = new MQQueueManager(qManager, qMgrProp);
System.Console.Out.WriteLine("MQTest02 successfully connected to " + qManager);
queue = qMgr.AccessQueue(outputQName, openOptions, null, null, null); // no alternate user id
System.Console.Out.WriteLine("MQTest02 successfully opened " + outputQName);
receiveMsg = new MQMessage();
queue.Get(receiveMsg, gmo);
System.Console.Out.WriteLine("Message Data>>>" + receiveMsg.ReadString(receiveMsg.MessageLength));
}
catch (MQException mqex)
{
System.Console.Out.WriteLine("MQTest02 cc=" + mqex.CompletionCode + " : rc=" + mqex.ReasonCode);
}
catch (System.IO.IOException ioex)
{
System.Console.Out.WriteLine("MQTest02 ioex=" + ioex);
}
finally
{
try
{
queue.Close();
System.Console.Out.WriteLine("MQTest02 closed: " + outputQName);
}
catch (MQException mqex)
{
System.Console.Out.WriteLine("MQTest02 cc=" + mqex.CompletionCode + " : rc=" + mqex.ReasonCode);
}
try
{
qMgr.Disconnect();
System.Console.Out.WriteLine("MQTest02 disconnected from " + qManager);
}
catch (MQException mqex)
{
System.Console.Out.WriteLine("MQTest02 cc=" + mqex.CompletionCode + " : rc=" + mqex.ReasonCode);
}
}
}
/// <summary> main line</summary>
/// <param name="args">
/// </param>
// [STAThread]
public static void Main(System.String[] args)
{
MQTest02 mqt = new MQTest02();
try
{
mqt.init(args);
mqt.testReceive();
}
catch (System.ArgumentException e)
{
System.Console.Out.WriteLine("Usage: MQTest02 -h host -p port -c channel -m QueueManagerName -q QueueName [-u userID] [-x passwd] [-s securityExit]");
System.Environment.Exit(1);
}
catch (MQException e)
{
System.Console.Out.WriteLine(e);
System.Environment.Exit(1);
}
System.Environment.Exit(0);
}
}
}
To run MQTest02 using your information, it would be:
MQTest02.exe -h A1PATA00.WORLDSPAN.COM -p 1414 -m A1PATA00 -c ECACHE -q SPLASH.ECAC.2

In the IBM MQ Knowledge center page "ClearMessage method" it documents the following:
This method clears the data buffer portion of the MQMessage object.
Any Message Data in the data buffer is lost, because MessageLength,
DataLength, and DataOffset are all set to zero.
The Message Descriptor (MQMD) portion is unaffected; an application
might need to modify some of the MQMD fields before reusing the
MQMessage object. To set the MQMD fields back use New to replace the
object with a new instance.
In the IBM MQ Knowledge center page "MQMessage .NET class" it documents the following:
public byte[] MessageId {get; set;}
For an MQQueue.Get call, this field specifies the message identifier
of the message to be retrieved. Normally, the queue manager returns
the first message with a message identifier and correlation identifier
that match the message descriptor fields. Allow any message identifier
to match using the special value MQC.MQMI_NONE.
public byte[] CorrelationId {get; set;}
For an MQQueue.Get call, the correlation identifier of the message to
be retrieved. The queue manager returns the first message with a
message identifier and a correlation identifier that match the message
descriptor fields. The default value, MQC.MQCI_NONE, helps any
correlation identifier to match.
Try this:
var queueMessage = new MQMessage(); <--- i want to do this new once!
while (true) {
//var queueMessage = new MQMessage(); // <-- only works if I do this every time i do a get
queueMessage.ClearMessage();
queueMessage.MessageId = MQC.MQMI_NONE;
queueMessage.CorrelationId = MQC.MQCI_NONE;
get.Get(queueMessage, gmo);

Connecting to the MQ Server using CCDT

I'm trying to connect to the MQ using the information present in the CCDT file. I can currently connect to the MQ using all the details, and get and put messages from and to the queue.
After extensive googling, I've been unable to find any sample code which allows me to connect using the CCDT file.
One of my colleagues forwarded me his JMS connection code, but I've been unable to port it to C#.
The JAVA code is as follows -
public class MQTest {
public static void main(String[] args) {
MQQueueManager queueManager = null;
URL ccdtFileUrl = null;
MQMessage mqMessage = null;
//MQPutMessageOptions myPMO = null
try {
String QM = "IB9QMGR";
String QUEUE1 = "TEST";
System.out.println("Starting MQClient Put Program: ");
ccdtFileUrl = new URL("file:///D:/AMQCLCHL.TAB") ;
ccdtFileUrl.openConnection();
queueManager = new MQQueueManager("SDCQMGR.T1", ccdtFileUrl);
System.out.println("Connected to QMGR ");
int openOptions = MQC.MQOO_OUTPUT;
MQQueue InQueue = queueManager.accessQueue(QUEUE1,openOptions,null,null,null);
MQMessage inMessage = new MQMessage();
inMessage.writeString("###Testing####");
InQueue.put(inMessage);
System.out.println("Message Id is :" + inMessage.messageId);
System.out.println(inMessage.toString());
InQueue.close();
queueManager.disconnect() ;
}
catch(MQException ex){
System.out.println("MQ Error - Reason code :" + ex.reasonCode);
}
catch (Exception e){
System.out.println("Error : " + e);
}
}
}
Instead of URL, I used the URI (in C#) to set file location. (This may be wrongly used. Not sure what else to use though.)
Uri ccdtFileUrl = new Uri("file:///D:/AMQCLCHL.TAB") ;
but I can't use openConnection() on a URI. Also,
queueManager = new MQQueueManager("SDCQMGR.T1",ccdtFileUrl);
gives an argument overload exception. As URI is not supported in C#.
I've tried looking up samples but I've found some JMS samples and thats it. Looking for some sample code to connect in C#.

You will need to set MQCHLLIB and MQCHLTAB environment variables to use CCDT. You can set these two variables either from command prompt,app.config or code in the application itself.
Following example demonstrates usage of CCDT:
MQQueueManager qm = null;
System.Environment.SetEnvironmentVariable("MQCHLLIB", "C:\\ProgramData\\IBM\\MQ\\qmgrs\\QM1\\#ipcc");
System.Environment.SetEnvironmentVariable("MQCHLTAB", "AMQCLCHL.TAB");
try
{
**Hashtable props = new Hashtable();
props.Add(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES_CLIENT);
qm = new MQQueueManager("QM1",props);**
MQQueue queue1 = qm.AccessQueue("SYSTEM.DEFAULT.LOCAL.QUEUE", MQC.MQOO_OUTPUT | MQC.MQOO_FAIL_IF_QUIESCING);
MQMessage msg = new MQMessage();
msg.WriteUTF("Hello this message is from .net client");
queue1.Put(msg);
queue1.Close();
qm.Disconnect();
}
catch (Exception ex)
{
Console.Write(ex);
}

MSMQ transactional query exception handling

I'm trying out MSMQ with c#... I started by writing 2 simple console applications: one is the sender of a message (a class in a shared class library project) and one listener. worked great.
Next, what I wanted to learn is what happens in case of a failure on the receiving side (listener). So it got me started with transitional messaging. On a failure I see that the message stays in queue, but the server keeps asking for it. What I actually want to achieve is that the message is marked for X times retries at interval of times and then throw it to an error queue.
Now for my code...
Sender
Console.WriteLine("I am the sender");
Message newMessage;
MessageQueue queue = null;
string queueName = #".\Private$\MyQueue";
using (MessageQueueTransaction msgTx = new MessageQueueTransaction())
{
queue = new MessageQueue(queueName);
queue.DefaultPropertiesToSend.Recoverable = true;
while (Console.ReadLine() != null)
{
msgTx.Begin();
Person person = new Person
{
FirstName = "Shlomi",
LastName = "Or",
Birthday = new DateTime(1982, 5, 6)
};
queue.Send(person, "person_label", msgTx);
msgTx.Commit();
}
queue.Close();
}
Listener
Console.WriteLine("I am the reciever");
int counter = 0;
while (!Console.KeyAvailable)
{
Message newMessage;
MessageQueue queue = null;
string queueName = #".\Private$\MyQueue";
using (TransactionScope scope = new TransactionScope(TransactionScopeOption.Required))
{
try
{
if (!MessageQueue.Exists(queueName))
queue = MessageQueue.Create(queueName, true);
else
queue = new MessageQueue(queueName);
if (queue.CanRead)
{
try
{
newMessage = queue.Receive(MessageQueueTransactionType.Automatic);
//newMessage.Formatter = new XmlMessageFormatter(new String[] { "System.String,mscorlib" });
//Console.WriteLine(newMessage.Body.ToString());
newMessage.Formatter = new XmlMessageFormatter(new Type[] { typeof(Person), typeof(Object) });
Person person = (Person)newMessage.Body;
Console.WriteLine(person.FirstName);
throw new Exception("Some error");
scope.Complete();
}
catch(Exception ex)
{
Console.WriteLine("exception occured " + (++counter));
}
}
}
finally
{
queue.Dispose();
}
}
}
when I run this code, what I see happening when I send only one message is that the error is thrown and it keeps trying to get the same message over and over. What I want is as explained above - some kind of retry and then move to error queue. how can I do that?