Develop Reference

How to ingest large amount of logs in ASP.Net Web API

I am new to API development and I want to create a Web API end point which will be receiving a large amount of log data. And I want to send that data to Amazon s3 bucket via Amazon Kinesis delivery stream. Below is a sample application which works FINE, but I have NO CLUE how to INGEST large inbound of data and in What format my API should be receiving data? How my API Endpoint should look like.
[HttpPost]
public async void Post() // HOW to allow it to receive large chunk of data?
{
await WriteToStream();
}
private async Task WriteToStream()
{
const string myStreamName = "test";
Console.Error.WriteLine("Putting records in stream : " + myStreamName);
// Write 10 UTF-8 encoded records to the stream.
for (int j = 0; j < 10000; ++j)
{
// I AM HARDCODING DATA HERE FROM THE LOOP COUNTER!!!
byte[] dataAsBytes = Encoding.UTF8.GetBytes("testdata-" + j);
using (MemoryStream memoryStream = new MemoryStream(dataAsBytes))
{
PutRecordRequest putRecord = new PutRecordRequest();
putRecord.DeliveryStreamName = myStreamName;
Record record = new Record();
record.Data = memoryStream;
putRecord.Record = record;
await kinesisClient.PutRecordAsync(putRecord);
}
}
}
P.S: IN real world app I will not have that for loop. I want my API to ingest large data, what should be the definition of my API? Do I need to use something called multiform/data, file? Please guide me.

Here is my thought process. As you are exposing a API for the logging, your input should contain below attributes
Log Level (info, debug, warn, fatal)
Log message (string)
Application ID
Application Instance ID
application IP
Host (machine in which the error was logged)
User ID (for whom the error occurred)
Time stamp in Utc (time at which the error occurred)
Additional Data (customisable as xml / json)
I will suggest exposing the API as AWS lambda via Gateway API as it will help in scaling out as load increases.
To take sample for how to build API and use model binding, you may refer https://learn.microsoft.com/en-us/aspnet/web-api/overview/formats-and-model-binding/model-validation-in-aspnet-web-api

I don't have much context so basically will try to provide answer from how I see it.
First instead of sending data to webapi I would send data directly to S3. In azure there is Share Access Token so you send request to you api to give you url where to upload file(there is many options but you can limit by time, limit by IP who can upload). So to upload file 1. Do call to get upload Url, 2. PUT to that url. Looks like in Amazon it called Signed Policy.
After that write lambda function which will be triggered on S3 upload, this function will be sending event (Again I dont know how its in AWS but in azure I will send Blob Queue message) this event will contain url to file and start position.
Write second Lambda which listens to events and do actually processing, so in my apps sometimes i know that to process N items it take 10 seconds so I usually choose N to be something not longer that 10-20 seconds, due to nature of deployments. After you processed N rows and not yet finished send same event but now Start position = Start position on the begging + N. More info how to read range
Designing this way you can process large files, even more you can be smarter because you can send multiple events where you can say Start Line, End Line so you will be able to process your file in multiple instances.
PS. Why I would not recommend you upload files to WebApi its because those files will be in memory, so lets say you have 1GB files sending from multiple sources in this case you will kill your servers in minutes.
PS2. Format of file depends, could be json since its the easiest way to read those files, but keep in mind that if you have large files it will be expensive to read whole file to memory. Here is example how to read them properly. So other option could be just flat file then will be easy to read it, since then you can read range and process it
PS3. In azure I would use Azure Batch Jobs

How to get the message using message id in Mailkit?

I am using mail kit to send receive mail, and manage all records in my database. I am storing all user action for particular mail and then execute it using my code.
I am storing message id in the table for the unique message, now I want to get the message using messageid. Is there any way to do it?

Firstly, don't expect the Message-Id header to be globally unique. Any hacker could easily create their own message and re-use a known Message-Id to try and confuse software that depends on Message-Ids being unique.
That said, you'll need to use the IMailFolder.Search() API combined with SearchQuery.HeaderContains() to search for messages with a particular Message-Id header.
var uids = folder.Search (SearchQuery.HeaderContains ("Message-Id", "blah#blah.com"));

Getting display name from EWS when passing in just an email address

I've using a custom GetMailTips SOAP call (since the EWS for Core 2.0 doesn't support it) to get Out of Office info for a batch of email addresses.
How can I get the display names of the users that I am passing in the email address for?
I can call ResolveName of the managed API and that works but it has to be done one at a time and that is slow. I would like to ideally get this info out when I make my GetMailTips request and failing that make a call with all the email addresses to get the Display Names all at once. I read there is meant to be a ResolveNames method but that's not in the API either.
Any help appreciated

Autodiscover can return that for multiple users eg
AutodiscoverService adService = new AutodiscoverService(ExchangeVersion.Exchange2013_SP1);
adService.Credentials = new NetworkCredential("user#d.com", "pass");
adService.RedirectionUrlValidationCallback = adAutoDiscoCallBack;
List<String> Users = new List<string>();
Users.Add("user1#domain.com");
Users.Add("user2#domain.com");
GetUserSettingsResponseCollection usrSettings = adService.GetUsersSettings(Users, UserSettingName.UserDisplayName);
foreach(GetUserSettingsResponse usr in usrSettings)
{
Console.WriteLine(usr.Settings[UserSettingName.UserDisplayName]);
}
Another way would be to create a Message and add the email address as recipients then save it to the drafts folders and the address should get resolved against the GAL.

how can I send message to specific user in specific group using asp.net signalR

I can send message to all group in chat
return context.Clients.Group(sGrp).recieveNotification(message, user, chatid, uuid);
I can send message to all group in chat
return context.Clients.User(currentuser).seenyou(myname);
the problem when i send message to specific user in specific group
is there is way to do this .

The only way to send a message to a specific user is by addressing the user by its connection Id. In this case it doesn't matter if he is in a specific group or not. Other than that there's no such thing as a built-in list, that you can iterate on, you need to manage those references by yourself.
Of course, if the desired user is the sender, you can use Clients.Caller.

Using Commands in my IRC Client

im currently trying to make a simple IRC Gui Client. Im using the SmartIrc4net as a base, as it seems to be the most supportive out of all of them, found here: http://www.codeproject.com/Articles/8323/SmartIrc4net-the-C-IRC-library
Now What I am having problem with is the action commands. For example to make yourself an oper, you would type
/oper admin password or to changehost, would be /sethost mynewhost
My problem is that when I pass that value through a TextBox, instead of making me admin, or changing my host. My input just gets displayed as text in the chat.
Here is my code:
string[] serverlist;
serverlist = new string[] { "mydomain.com" };
int port = 6667;
string channel = "#MyChannel#";
try
{
irc.Connect(serverlist, port);
irc.Login("SmartIRC", "SmartIrc4net Test Bot");
irc.RfcJoin(channel);
irc.SendMessage(SendType.Message, channel, "/oper admin mypass");
irc.SendMessage(SendType.Action, channel, "/sethost mynewhost");
irc.Listen();
But when I pass those values, all it does is just display what I typed in the chat, without actually making me oper, or changing my sethost.
Is there anyway that I could actually make it pass commands through to the IRC server, instead of just displaying the raw text on the chat?
Any help would be appreciated, Thanks

This is because you are explicitly sending a message. IRC itself has no notion of /commands, this is done all in the client. What you are doing is to just send a message with a specific text that happens to start with /. I.e. what the server receives is
PRIVMSG #channel :/oper admin mypass
instead of
OPER admin mypass
You just need to figure out a way of sending raw IRC commands to the server. The page you linked to doesn't offer much documentation on that part, though. But judging from the layers this should be in either Layer 2 or Layer 1.

There is a lot of more options then SendMessages.
You have example irc.RfcOper(username, pasword) for Oper.
If you want to send raw data command for things it does not support on the fly example sethost you can just send a WriteLine directly.
irc.WriteLine("sethost mynewhost")
Open the IrcCommands.cs to see a list of commands and Rfc2812.cs to see how they are transfered.
I however recommend you to read or at least peek at Rfc2812 standard that you can find here https://www.rfc-editor.org/rfc/rfc2812