Develop Reference

Kafka consumer (using kafka-net) fetching all messages from the beginning by default

I'm building a Kafka consumer for my .net proyect. I'm using kafka-net (A native c# client fro Apache Kafka mady by James Roland).
The problem I have is that this code (based on the documentation) fetches all messages from the beginning by default:
private void StartKafkaConsumer(string ipKafka, string portKafka, string topicKafka)
{
string topic = topicKafka;
Uri uri = new Uri($"http://{ipKafka}:{portKafka}");
var options = new KafkaOptions(uri);
using (var router = new BrokerRouter(options))
{
using (var consumer = new Consumer(new ConsumerOptions(topic, router)))
{
foreach (var message in consumer.Consume())
{
Console.WriteLine(Encoding.UTF8.GetString(message.Value));
}
}
}
}
...
StartKafkaConsumer("localhost", "9092", "test"); //this fetches messages sent weeks ago, since the creation of the 'test' topic
Basically, this code does the same than this command:
.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning
All I want to do is to fetch messages from the time the client gets connected to the Apache server, not from the beginning. I know it's possible since I tried the last command without the "--from-beginning" part and it worked.
Any suggestion would be appreciated.

Look at the ConsumerOptions methods. There should be an option to set a property/config value:
auto.offset.reset
Set the above property/config to latest. At that point, anytime you connect with an unknown/new consumer group id, the consumer will default to start at the latest offset.
If, however, the consumer group id was already known (i.e., had already consumed at least once from that topic/partition), it will attempt to get the last committed offset + 1. If that offset is unavailable, because maybe it has gone past the retention threshold, then it will default to latest.
Here's more detailed documentation:
https://kafka.apache.org/documentation/#newconsumerconfigs

Can we filter messages from Amazon SQS queue by message attributes?

For now I have tried to filter the messages based on Message Attribute Name="Class". As you can see in the below code
//Specify attribute list
List<string> AttributesList = new List<string>();
AttributesList.Add("Class");
receiveMessageRequest.MessageAttributeNames = AttributesList;
receiveMessageRequest.QueueUrl = urlSQS;
receiveMessageRequest.MaxNumberOfMessages = 10;
ReceiveMessageResponse receiveMessageResponse = objClient.ReceiveMessage(receiveMessageRequest);
But the messages are not been filtered based on the provided MessageAttributeName = "class".

receiveMessageRequest.MessageAttributeNames = AttributesList;
This tells SQS which message attributes you want it to return with the message if the are present on the message. It is not a message filter. If the attributes aren't present, nothing happens.
But your confusion seems understandable -- it's not actually obvious why the API even has this functionality, though it may be a holdover from when SQS supported only smaller messages than it does today, or it may be so that you can avoid spending any time parsing information from the response that you will end up discarding. I almost always just ask for All.

Please note this regarding messaging services on AWS
SQS : No filtering support ( while picking up messages)
SNS : Supports attributes-based filtering: subscriber can set a subscription attribute (a subscription filter policy) which is applied on incoming messages and only relevant messages can be sent to the subscriber.
EventBridge: Amazon EventBridge supports declarative filtering using event patterns. With event pattern content filtering you can write complex rules that only trigger under very specific conditions. For instance, you might want a rule that will trigger only when a field of the event is within a specific numeric range, if the event comes from a specific IP address, or only if a specific field does not exist in the event JSON.
Please refer my article for a detailed difference between main messaging frameworks on AWS.
https://www.linkedin.com/pulse/mastering-art-decoupling-application-architecture-aws-amit-meena/

It depends on how the message in question gets onto the queue. If you are pushing the message via SNS then yes you can filtered messages; https://docs.aws.amazon.com/sns/latest/dg/message-filtering.html
Any other filtering mechanism doesn't exist right now.
Hope that helps!

As per the AWS SDK method, we can use following code to do the filter.
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest("QUEUE URL" );
receiveMessageRequest.setMaxNumberOfMessages(Integer.valueOf(1));
private static AmazonSQS sqs;
List<Message> messages = sqs.receiveMessage(receiveMessageRequest.withMessageAttributeNames("Attribute Name")).getMessages();
If you want all the message then use given code
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest("QUEUE URL" );
receiveMessageRequest.setMaxNumberOfMessages(Integer.valueOf(1));
private static AmazonSQS sqs;
List<Message> messages = sqs.receiveMessage(receiveMessageRequest.withMessageAttributeNames("All")).getMessages();

Retrieve emails in descending/reverse order using MAILKIT c#

I am using mailkit currently to retrieve emails. I am using this code :
var message = inbox.GetMessage(i);
Emails Email = new Emails;
Email.From = message.From.ToString;
Email.EmailDate = message.Date.ToString;
InboxMails.Add(Email);
The problem is, as I am just retrieving the Dates, I can see that the emails are retrieved in the reverse order. I mean, Mailkit fetches the emails from the last email to the first where it should fetch from first to last. E.g. the last email in my mailbox is from 2/3/2014 and the first one is from 1/1/2018.
Now Mailkit loads the last one first and eventually gets to the first one, any way to retrieve in the correct descending order - from the first to the last email?
Also, is there a way to get a Unique Id for each message by which I can find / filter the messages?
I tried this :
for (int i = inbox.Count -1; i >= 0; i--)
But it doesnt even return a single message

Mailkit: fetch messages and copying them to MySQL, attachments to drive

I've been using this code of MailSystem.Net so far to get emails from an Imap INBOX and added options to retrieve mail using $"SENTSINCE {Date}".
string mailBox = "INBOX";
public IEnumerable<Message> GetMailsSince(string mailBox) {
return GetMails(mailBox, $"SENTSINCE {DateTime.Now.AddDays(-3).ToString("dd-MMM-yyyy")}").Cast<Message>();
}
private MessageCollection GetMails(string mailBox, string searchPhrase) {
Mailbox mails = Client.SelectMailbox(mailBox);
MessageCollection messages = mails.SearchParse(searchPhrase);
return messages;
}
But even after studying mailkit for hours I can't seem to distill out how to do the same thing. My goal is to get a list of message object whose properties I can then map to another class I created which writes it into a mysql database. I also want to save the attachments to disk. All this works fine so far but performance is an issue. I'm hoping mailkit will greatly improve that.
My main source has been the sample here but because I'm not familiar with async programming yet and treeviews it's hard to see through it.
How can I hard code that I want "INBOX" as ´IMailFolder´?
Where or how can I use the "SENTSINCE {Date}" filter in Mailkit?
How do I get an ´IEnumerable´ of Mailkits equivalent to the Message object in mailsystem (´IMessageSummary´ maybe)?
If you can point me to some code or even convert the linked MailSystem.Net example to Mailkit that would be fantastic.

MimeMessage is the equivalent of MailSystem.NET's Message object, but that's not what you want. What you want is MailKit's IMessageSummary which will allow you to download individual MIME parts (aka "attachments").
It also allows you to get summary information about the message (flags, received date (aka "InternalDate") pre-parsed/decoded common header values (such as subject, sender, recipients, etc) really quickly because the IMAP server has those pieces of information cached in its database for quick retrieval.
using (var client = new ImapClient ()) {
client.Connect ("imap.mail-server.com", 993, SecureSocketOptions.SslOnConnect);
client.Authenticate ("username", "password");
// if you don't care about modifying message flags or deleting
// messages, you can open the INBOX in read-only mode...
client.Inbox.Open (FolderAccess.ReadOnly);
// search for messages sent since a particular date
var uids = client.Inbox.Search (SearchQuery.SentAfter (date));
// using the uids of the matching messages, fetch the BODYSTRUCTUREs
// of each message so that we can figure out which MIME parts to
// download individually.
foreach (var item in client.Inbox.Fetch (uids, MessageSummaryItems.BodyStructure MessageSummaryItems.UniqueId)) {
foreach (var attachment in item.Attachments.OfType<BodyPartBasic> ()) {
var part = (MimePart) client.Inbox.GetBodyPart (item.UniqueId, attachment);
using (var stream = File.Create (part.FileName))
part.ContentObject.DecodeTo (stream);
}
}
}
Note: Each property on IMessageSummary has a corresponding MessageSummaryItems enum value that you will need to use in order to have that property populated.
For example, if you want to use IMessageSummary.Envelope, you will need to include MessageSummaryItems.Envelope in your Fetch() request.
Since MessageSummaryItems is marked with the [Flags] attribute, you can bitwise-or enum values together like this:
MessageSummaryItems.BodyStructure | MessageSummaryItems.Envelope and both pieces of information will be fetched.
Update:
Here's the inefficient way that is closer to how MailSystem.NET does it.
using (var client = new ImapClient ()) {
client.Connect ("imap.mail-server.com", 993, SecureSocketOptions.SslOnConnect);
client.Authenticate ("username", "password");
// if you don't care about modifying message flags or deleting
// messages, you can open the INBOX in read-only mode...
client.Inbox.Open (FolderAccess.ReadOnly);
// search for messages sent since a particular date
var uids = client.Inbox.Search (SearchQuery.SentAfter (date));
// using the uids of the matching messages, fetch the BODYSTRUCTUREs
// of each message so that we can figure out which MIME parts to
// download individually.
foreach (var uid in uids) {
var message = client.Inbox.GetMessage (uid);
foreach (var attachment in message.Attachments.OfType<MimePart> ()) {
using (var stream = File.Create (attachment.FileName))
attachment.ContentObject.DecodeTo (stream);
}
}
}
Note: if you care about saving message/rfc822 attachments, then take a look at this StackOverflow answer: MailKit save Attachments

The "Inbox" folder is always available on an IMAP mail acccount. With MailKit it is available as ImapClient.Inbox. For the date filtering you could use the DateSearchQuery class. The getting started page of MailKit pretty much covers all your questions.

Mimekit/mailkit download message body?

I have been making my own mail client recently and added a receive option, I used mimekit and mailkit as plugins and was able to download most of my mails with a reader that is supposed to show content(right now it shows subject, to, from, date)
The way I downloaded the subject, to, ... is msg.envelope.subject, msg.envelope.to
But I cannot download the body like this :( when doing either msg.body, msg.textbody, msg.bodyparts, ... they all result in NOTHING, the place where it should be is just empty, I can't get it downloaded :(
Can anyone help me?

There are 2 ways to get the message body:
1. Download the Whole Message
This method is probably the easiest way.
To do this, all you need to do is call:
var message = folder.GetMessage (uid);
or
var message = folder.GetMessage (index);
I would recommend always using the UniqueId of the message. Since you are already using the Fetch method, all you need to do to make sure that you have the UniqueId of the message is to include the MessageSummaryItems.UniqueId in your fetch request:
var messages = folder.Fetch (0, -1, MessageSummaryItems.UniqueId |
MessageSummaryItems.Envelope | ...);
Once you have the message, you can do whatever you want with it.
For rendering the message, I would recommend taking a look at the MessageReader sample included in the MimeKit GitHub repository.
It will show you how to properly render a MimeMessage.
2. Download Only What You Need
This method is a bit harder but can be more efficient as far as network bandwidth usage goes.
The first thing you need to do is to make sure to include the MessageSummaryItems.BodyStructure bit flag in the Fetch method. For example:
var messages = folder.Fetch (0, -1, MessageSummaryItems.Envelope |
MessageSummaryItems.BodyStructure);
(You'll probably want other fields, but that's just an example to show you how to bitwise-or flags together to request multiple message summary items).
By requesting the BodyStructure of the messages, you'll be able to make use of the msg.Body property.
Each msg.Body will be a BodyPart object which is an abstract class. The 2 main subclasses are BodyPartMultipart and BodyPartBasic. You can use the as cast or the is keyword to figure out which one it is:
var multipart = msg.Body as BodyPartMultipart;
if (multipart != null) {
// the top-level body part is a multi-part
} else {
// the body is a basic singleton part
}
This is how you would iterate over the subparts of a BodyPartMultipart:
foreach (var part in multipart.BodyParts) {
// each part will either be a BodyPartMultipart
// or a BodyPartBasic, just like before...
}
There are also 2 subclasses of BodyPartBasic which are: BodyPartText and BodyPartMessage. A BodyPartText is a textual-based MIME part (meaning it has a MIME-type of text/*) whereas a BodyPartMessage is an embedded message (and will have a MIME-type of message/rfc822).
Since MIME is recursive, you'll need to implement a recursive function to walk the MIME tree structure to find whatever MIME part you are looking for.
For your convenience, the TextBody and HtmlBody properties on the IMessageSummary interface will locate and return the text/plain and text/html body parts, respectively.
It should be noted, however, that these properties only work in cases where the structure of the message follows the standard convention (notice I said convention, there is no formal standard dictating the location of the message text within a MIME hierarchy).
It should also be noted that if your mail client will be rendering the HTML body, the HTML body part may be part of a group of related MIME parts (i.e. a child of a multipart/related), but the HtmlBody property will not be able to return that and so implementing your own recursive logic will be a better option.
For an example of how to do this, check out the ImapClientDemo sample in the MailKit GitHub repository. The logic currently resides in the MainWindow.cs code.