Develop Reference

How to send email reports at large scale for every 12 AM using azure webjob?

I have an Azure web job that is running continuously and will send report emails at 12 AM when a user's timezone time reaches 12 AM. It is expected that these reports will send at 12 AM. Consider the system has 100 users and 50 of them are in the same timezone. it is expected that these 50 users should get their mail at 12 AM. But what happening is that all are getting at the different time. The starting time is at 12 AM.
Here is my logic
foreach(var item in reports){
if(Convert.ToDateTime(item.LastSendDate).Date < UserTimezoneDate.Date){
// send mail logic using report id
}
}
Someone suggestdd that use multithreading, it is a possible solution, but I want to implement it a large scale basis. So if I use multithreading is it a good approach? Is it good to use service fabric? If yes how can implement it(logic only)?

MailKit SMTP slow when sending messages

I've been using MailKit to retrieve some emails using IMAP and forwarding them using SMTP (more info here), but it takes really long for the SMTP to send the email.
I'm using mailkit via NuGet
This is the code I'm using
<!-- language: c# -->
var before = DateTime.Now;
Console.Write("\tForwarding email... ");
smtpClient.Send(forwardMessage, fromMailboxAddress, new[] { toMailboxAddress });
Console.WriteLine(" done! ({0})", DateTime.Now - before);
And the time it takes is usually more than 30s. What is making me suspect there's somethign wrong is that the email is actually forwarded almost instantly: few seconds (or even less) than the code reaches the smtpClient.Send method, I can see the message appearing in the destination email account (I have Thunderbird opened at the same time), but something makes the code to be still doing something in the Send code line.
Is there a way to know what the code is doing and why does it take so long?

You can see what the SmtpClient.Send() code is doing here: SmtpClient.cs:1543
I can't think of any reason it would take more than 30 seconds to send if you are seeing the message show up at the destination within seconds after client.Send() is being called.
My only guess is that the server is taking a long time to send a response to the DATA (or BDAT) command (which is the command that actually sends the raw message data).
In other words, my guess is that it would have to be this line: SmtpClient.cs:1517 or this line: SmtpClient.cs:1488
This is the ReadResponse() method: SmtpClient.cs:320. Most likely the waiting will be in the Poll() call or, if the stream does not support polling (SslStream), then it will be stuck in the stream.Read() call waiting for a response from the server.

Google.Apis.Admin.Email_Migration_v2 [HTTP Status Code 412 – Limit Reached]

Edit 2:
Client Library: After reviewing it is not easily suggested that this is for the .NET client library.
DLL: Google.Apis.Admin.email_migration_v2.dll
What steps will reproduce the problem?
Generate a process which contains a
Google.Apis.Admin.email_migration_v2.AdminService instance for each
unique Google Apps Gmail mailbox that will have messages sent to it.
All of the AdminService objects generated use the same OAuth2.0
credentials and application name. Each AdminService object generated
will only send messages to one Google Apps user’s mailbox. For
example, if we were sending messages to five different Google Apps
Gmail mailboxes we would generate five AdminService objects to send
messages; one for each user’s mailbox.
Biggest thing to note is that each AdminService object created is created on a separate process.
AdminService objects were given a FileDataStore object to change the location of where the refresh token is stored; C:\ProgramData\SomeFile\SomeFile.
Supplied appropriate scopes to the credentials.
Begin sending mail messages on each process. Using one thread to send messages in each process, so only one message is sent at a time to each user’s mailbox.
Each message sent gets its own instance of MailItem and MailResource.InsetMedia
The MailResource.InsertMedia object is generated for each item by calling AdminService.Mail.Insert(MailItem, string, Stream, string) method.
When our code makes the call to MailResource.InsertMediaUpload.UploadAsync(CancellationTokenSource).Result is where we can receive the error.
The error is caught and handled (logged) from the return type of the aforementioned call; the type is Google.Apis.Upload.IUploadProgress. The exception is handled using the IUploadProgress.Exception property.
What is the expected output? What do you see instead?
The expected output would be a successful message response or the exception property of the IUploadProgress to be null after the return of the task. Instead we are receiving the following error message:
The service admin has thrown an exception:
Google.GoogleApiException:Google.Apis.Requests.RequestError
Limit reached. [412]
Errors [Message[Limit reached.] Location[If-Match - header] Reason[conditionNotMet] Domain[global]]
at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at Google.Apis.Upload.ResumableUpload`1.d__e.MoveNext()
What version of the product are you using?
Google.Apis.Admin.Email_Migration_v2 (1.8.1.20)
What is your operating system?
Windows Server 2008 R2 Enterprise (SP1)
What is your IDE?
Visual Studio 2013 Premium
What is the .NET framework version?
4.0.30319
Please provide any additional information below.
Non-consecutive messages can fail (with the 412 http status code
provided above) during the process of sending the messages. Once we
receive this error other messages sent after the failed message(s)
can succeed. (Items can fail at any point during the process
beginning, middle or end.)
Each message sent has nearly identical content. The size of the
messages range from 1KB to 100KB including the size of all associated
attachments, not all messages have attachments.
Reprocessing the failed items at a later time results in successful
message responses and the appropriate items are sent to the user’s
Google Apps Gmail Inbox.
The maximum number of Google Apps user’s mailboxes sent to at one
time was ten.
After checking the quotas of our Google Developers Console project:
We were nowhere near the specified limit of 20 requests a second for
the Email Migration API; maxed out at sending 7 requests a second.
Only 2% of the maximum daily requests had been reached.
All messages sent had the same label; the label was well under the
225 character limit. Actually all of the labels/sub-labels applied
together only surmounted to 40 characters.
This error message can still be received when sending to only one
Google Apps user’s mailbox; only using one process and one thread.
Each process is normally sending anywhere from 1000-5000 messages.
I have not found a lot of specific documentation to explain this particular error in enough detail to remedy the problem at hand.
Questions:
So what exactly does this 412 http status code mean? What limit is being encountered that this message is referring to?
Shouldn’t we be receiving some form of 5XX error from the server if we are hitting a limit? In which case wouldn’t the built in exponential back off policy kick in?
a. Unless the server is checking the POST request for a pre-condition about a server side limit then telling the client to back off which is what a 412 error seems to typically indicate. In that case please give as much detail as possible for question 1.
Sorry for the extensive post! Thanks for your time! I will also be creating a defect/issue in Google's .NET issue tracker and providing a link.
Edit 1:
For anyone interested in following this issue here is a link to the submitted item in Google's issue tracker for .NET.
Submitted Issue
For reference it is issue 492.

I am not quite sure where you see the "the specified limit of 20 requests a second for the Email Migration API". Reminder: the QPS limit you see in the Google Developers Console project is not the actual default limit. You can change that limit to anything you want, and thus, that's not the actual limit for the API. It is really just for managing the consumption of the API quota (some APis will have a much higher QPS where you can adjust it to lower for different projects across your console).
According to the email migration APi documentation, the QPS is 1 request per second (the link is here: https://developers.google.com/admin-sdk/email-migration/v2/limits).
I have experienced 412 errors when the QPS limit is being hit, and I have also seen the 412 error returned when I am uploading too much data to a single domain. How much data are you loading all at once? I would suggest doing an exponential backoff to see if the issue would disappear.

I believe I have found an answer to this problem, though I will advise a disclaimer, I do not work for Google and cannot be 100% sure of the accuracy; you've been warned. This should at least hold true for the .NET version of Google's Email Migration v2 API. I cannot guarantee how other APIs work because I do not use them..
Through working with this API in spurts for well over eight months now, it appears that if an application or multiple applications are to send messages to a single Google Apps user/mailbox consistently, at a faster rate than which Google servers can process, then at some rate you should start to get a bunch of GoogleApiExceptions stating "412 - Limit Reached" when sending new messages. What we have gathered through using our application is that each Google Apps user/mailbox has its own pending items queue. When you send a message to Google Apps it is first put into this queue before being processed by a Google Server and put into the user's mailbox. If this queue becomes full and you attempt to send another message you will receive a 412 error.
Options are to wait before sending another message, you'll have to wait however long the Google server takes to process the next message in the user's queue before sending another; which is unpredictable. The better option in my opinion is to start sending messages to another Google Apps user; because each user appears to have its own message queue. Be sure to stop sending to the user who is consistently getting 412 errors. This will give the Google server some time to process that user's packed message queue. Note each pending messages queue appeared to hold about 100-150 items before throwing 412 errors.
503 errors appear to occur when sending messages into a user's mailbox queue at a higher rate than 1 request per second. As Emily has stated "the QPS limit you see in the Google Developers Console project is not the actual default limit" it is truly 1 QPS per Google Apps user.
As for the exponential back-off it is supposed to be implemented automatically see this. Note Peleyal appears to be the gentleman in charge of the API; can be noted from the download page for the API.
This took us a little while to figure out so cheers if you're having this issue! Please if you find any contradicting information correct any mistakes found in this answer or make your own!!

MSMQ Poison message and TimeToReachQueue

I tried creating a poison message scenario in the following manner.
1- Created a message queue on a server (transactional queue).
2- Created a receiver app that handles incoming messages on that server.
3- Created a client app located on a client machine which sends messages to that server with the specific name for the queue.
4- I used the sender client app with the following code (C# 4.0 framework):
System.Messaging.Message mm = new System.Messaging.Message("Some msg");
mm.TimeToBeReceived = new TimeSpan(0, 0, 50);
mm.TimeToReachQueue = new TimeSpan(0, 0, 30);
mm.UseDeadLetterQueue = true;
mq.Send(mm);
So this is setting the timeout to reach queue to 30 seconds.
First test worked fine. Message went through and was received by the server app.
My second test, I disconnected my ethernet cable, then did another send from the client machine.
I can see in the message queue on the client machine that the message is waiting to be sent ("Waiting for connection"). My problem is that when it goes beyond the 30 sec (or 50sec too), the message never goes in the Dead-letter queue on the client machine.
Why is it so ? ... I was expecting it to go there some it timed-out.
Tested on Windows 7 (client) / Windows server 2008 r2 (server)

Your question is a few days old already. Did you find out anything?
My interpretation of your scenario would be that the unplugged cable is the key.
In the scenario John describes, there is an existing connection and the receiver could not process the message correctly within the set time limit.
In you scenario, however, the receiving endpoint never gets the chance to process the message, so the timeout can never occur. As you said, the state of the message is Waiting for connection. A message that was never sent cannot logically have a timeout to reach its destination.
Just ask yourself, how many resources Windows/ MSMQ would unneccessaryly sacrifice - and how often - to check MessageQueues for how-many conditions if the queues is essentially inactive? There might be a lot of queues with a lot of messages on a system.
The behavior I would expect is that if you plug the network cable back in and the connection is re-established that then, only when it is needed, your poison message wil be checked for the timeout and eventually moved to the DeadLetter queue.
You might want to check this scenario out - or did you already check it out the meantime?

Could this work (send mass newsletter email w/ ASP.NET (C#))

I am working on a site and it has to have a newsletter sending functionality. A couple of weeks ago I posted a question on stackoverflow, and the good people suggested that I do a console app which I call from the site and it sends the mails. I did this, and it works locally, but I wasn't able to get it working on the server (security issue). Been trying to make this work for the past 3 days. So then, I decidet to try and change how the email is sent.
To make a long story short, now I split all the emails in "groups" of 50, and then make a new MailMessage object, and mailMessage.To.add(sample#sample.com) emails to it. Then I send it 50 by 50.
It works for now, but my question is, how "BAD" is this solution? I tryed adding the emails to bcc, but for some reason (for me) this works only locally ( !?! ).
And one more thing which I couldn't find on line, what is the maximum number of mails I can add to the MailMessage object?
Thank you for your time!
Andrej

its very bad since each user which receives your newsletter will have access to the 49 other contacts (where do i sign up?)
If you are having problems with multiple BCC's, contact you SMTP host.
Did you try sending each separately? if it is taking to long to send, you can build a multi-threaded solution which will help.

The correct way is to send each one his mail and you create a queue in memory on in database for store mail before send.
Remember send all emails in 1 time if you use public smtp server is bad thing, you can be banned for that server,this means you must schedule such as 1000 mail at time or less.
i have wrote this years ago if you need i can post same code