The scenario is a Windows Service connects to an outlook.com or a gmail.com POP3 account using OpenPOP. It finds existing emails if there are any (i.e. client.getMessageCount() returns a value larger than zero).
The service keeps the connection open and rechecks for new messages.
The problem is that if i send email to that account after the service has connected then the getMessageCount() always returns zero until the service automatically re-cycles the pop connection.
I'm hoping someone has had the same issue.
I cannot resolve the issue by more frequent pop diconnect/reconnects as some providers (like outlook.com) have limits to time between connects.
The POP3 specification mentions that a mailbox is locked while a client has it open. Therefore no messages can appear.
Think of this scenario: Most servers assigns new messages a low message number. You have just fetched all the message numbers:
1 - Foo message
2 - Bar message
You now want to delete the Foo message, but the server updates the maildrop to look like:
1 - Baz message
2 - Foo message
3 - Bar message
You send the command DELE 1, and have now marked the Baz and not the Foo message to be deleted. The POP3 protocol was developed when internet was a sparse resource and you were not always connected. If you want to have live updates to a maildrop, you should use the much newer IMAP protocol instead.
If you keep using POP3, you will have to disconnect and connect again to have updates. Notice that most servers does not allow clients to continue to connect/disconnect in a fast manner, they will simply reject your authentication if you are connecting too often.
Also notice that Gmail have a strange POP3 implementation. See What non-standard behaviour features does Gmail exhibit, when it is programmatically used as a POP3 server?
Related
We have got a single POP e-mail account. This e-mail account is filled with:
mails with subject XYZ
mails with subject ABC
We have two Windows services
Service 1 - Which picks up emails with Subject XYZ only (Reads an email, stores it in the database and then deletes it from the mailbox)
Service 2 - Which picks up emails with Subject ABC only (Reads an email, stores it in the database and then deletes it from the mailbox)
Can these two separate C# services (Service 1 & 2) (.Net framework 4.0) run simultaneously on this single POP enabled mailbox and work without any access conflicts over the mailbox?
Yes - there is generally no restrictions on how many clients can connect with the same account sequentially.
Notes
you should not be constantly be connecting - polling and disconnecting is the expected approach.
you may have issues deleting items due to way POP3 addresses items - may need to synchronize deletion in some way if server does allow 2 clients for the same user to be in TRANSACTION state at the same time (unlikely).
server may have some protection against frequent access from different IP in addition to exclusive single user access - you need to check with particular service provider if there are any restrictions (also likely for 2-3 well known clients you'd be fine).
See POP3 RFC 1939 for details.
Yes, multiple processes should be able to read from the same mailbox simultaneously.
The caveats are
That the server supports simultaneous access (it should)
That neither process is deleting messages.
Caveat #2 is important in that both processes have to read through the list of messages to find messages that are appropriate to them, but neither process will know which messages the other process has already read. If either process deletes a message, it opens up the possibility that one process will already know the message exists and attempt to read said message after the second process has deleted it. (A race condition.)
A better approach might be to have a single "Mail Reader" process read all messages, enter them into different queues (perhaps stored in a database) which are then handled by the other two "Do Work" processes. After the Mail Reader process has successfully placed the message into a queue, then it can safely delete the message from the mailbox.
BACKGROUND
First time using Pop3.
Using OpenPop library.
Have configured Gmail to work with Pop3.
PROBLEM
Have been debugging code to see what happens (and try to get body text). Messages downloaded first 2 or 3 times.. got the UIDs with this:
var uids = client.GetMessageUids();
This time, it is no longer fetching any UIDs and I can't figure out why. Note, I did prematurely terminate the app a couple of times. My Gmail settings are configured to keep the mails on the server even after downloading them with a pop client. I even tried setting some mails to unread status in the hope that would work, but it doesn't.
So, any ideas what would cause this method to stop working after 2 or 3 goes? Does Gmail have some kind of security issue or maybe know that this client has acquired the messages already and doesn't send the messages anymore or what?
EDIT
I sent an email to that address and now it does show 1 message. So what I really need to know is HOW does the client (or Gmail) know which messages have been downloaded or not? This is very important, because if an error occurs and I cannot store the email for my app, the next time a refresh is done, the message will not be downloaded again and so messages will be missing from the application. Is there a way to reset it? Where is this being recorded?
OpenPop does not store anything about messages by itself. It simply fetches whatever you tell it to. Gmail is a strange POP3 provider, as seen in What non-standard behaviour features does Gmail exhibit, when it is programmatically used as a POP3 server?
Gmail does not present the same message if it has been downloaded by any POP3 client. You could use the recent:username login method to see the last 30 days worth of email always.
How do you 'verify' that a message sent using MSMQ to a private local queue was actually delivered? I'm especially thinking of a scenario where the listener (a C# service in my case) is not running and therefore delivery can't be successful.
You can only guarantee that it can get to the queue without taking extra steps. To deal with the "not running receiver" scenario, you would need to code the receiver to send a message back to the server when it processes the message. The original sender would be responsible for tracking the sent messages and verifying that the client has recieved them.
That's one of the decisions you should be taking when deciding whether or not to use MSMQ as opposed to a remoting or a web service scenario. For example, we had a project used for notifying all of our retail locations when an emergency occurred (such as a product recall/food safety issue.) We needed to know immediately if the store's listener was running so we chose remoting, and when the sender received an error indicating one of the listeners was not listenting, we would need to pick up the phone and call.
Just something to keep in mind.
Edit - clarification
I was really giving out two options above.
Code the client to send a message back to the sender when it receives a message.
Use another option, such as remoting, where you can detect if the client is running and receives the message.
It's always sent to the queue.
If your service isn't running to receive it, it just sits there, waiting patiently, until someone receives it.
You know it's been sent to the queue because .Send() returns without crashing.
You can probably pull this info out using administrative queues
When you send a message you can specify the AcknowledgeType which will allow you find out (through positive or negative acknowledgement) whether the message reached the queue and/or was received from the queue. Acknowledgements are sent as messages, by MSMQ, to the AdministrativeQueue so make sure you assign that property on the Message object.
You can check the administrative queue for acknowledgements by correlation ID which is ID of the original message.
I have the need from an asp.net web site to send out many SMS messages at once, and also poll a POP3 account for an incoming mail, and then SMS that out to many recipients, one at a time.
The way I am thinking of doing this is a windows service that would connect to my sql back-end to see if there are SMS messages to be sent out, like every 10-20 seconds or so. If so, get all the messages in a list, delete them from the table, and then proceed to send them out.
Same way with the pop account.
Any ideas on how to best provide this service without causing blocking in the asp.net web page when it is kicked off (e.g. messages added to sql server)?
Platform is windows server 2003 R2, sql 2008 standard, asp.net 3.5 SP1.
Thanks for your advice.
We have implemented similar scenarios using SQL Server service broker's Queueing mechanism. The idea is that every inserted SMS record is caught by a trigger which inserts a message containing the SmsID into the service broker Queue.
You then need a stored procedure which receives messages from the Queue. If there are no messages, your procedure will run until the next entry is inserted. That's OK, since it does not take up resources to listen to the Queue.
Next you'll need a Windows service who continuously (recursively) calls the STP, assembles the SMS and sends it.
The Advantage of the Service Broker Queue over a flag in a table is thread safety. This way you could have as many instances of your Service as you want w/o having to worry too much about concurrency issues.
You can find a nice Service Broker tutoial here: http://www.developer.com/db/article.php/3640771
Instead of using an Sql Server for the queuing you could use MSMQ (Microsoft Message Queuing) for this.
MSMQ is quite easy to set up and once it is up and running it is more scalable than Sql Server.
So what you could do was to setup a new queue in MSMQ that would receive the messages you wanted to send. The message would normally be some sort of Message object that describe the message, the sender and the recipient.
Then you would either setup a service that would poll the queue at a regular interval or you could setup MSMQ to start a class of your choice each time a new Message was sent to the queue.
If you need a log of the messages you could have the service / sender object write to a log in sql server when the message was sent.
Our business sends a newsletter to a vast number of subscribers every week. When the business was very young, before I joined, they used a "free" version of some mass mailer that took six hours to send 5K mails and fell foul of every reverse DNS check on the internet.
I upgraded this to a bespoke .Net widget that ran on the correct server and could send up to about 20k mails in half an hour with full DNS compliance. Unfortunately (or fortunately depending on your standpoint) our mail list has now outgrown this simple tool. In particular its lack of adequate throttling, it can make more mails than the server can comfortably send at once. I need to actually monitor how full the IIS SMTP server's available outgoing mail storage allocation is and throttle the load accordingly.
Unfortunately I can find no information on where a mail object goes when (or even if) it is turned into a mail. I can implement a filesystemwatcher if I have a place to watch, currently I don't. If no actual mail file is ever created I guess I will have to create one to implement the functionality but I need to know where to put it. It would also be more reassuring to allow the system to confirm sending somehow but I have no idea how to go about retrieving data from the system that says a mail has been sent.
Extensive Googling has proven vague on these points; so I was wondering if anyone here knew where I could get a guide to these problems, or could otherwise point me in the right direction.
Many thanks.
EDIT: In the end I gave up trying to measure throughput on the IIS SMTP server as a bad job. It just didn't seem to want to play. I'm now carrying out my logging in a separate location and just shunting it through to the SMTP server thereafter. I still don't know of anyone who really bothers trying to keep tabs on the doings of the IIS SMTP server and so this question as of this writing goes unanswered.
Oh well...
Okay so I've been working on this project for ages now and I thought I might share my findings with the world.
The IIS SMTP Server
All mails created using the IIS SMTP server are sent, in the first instance, to the Pickup Directory. If you are sending one mail then you will have to operate in Matrix time to actually ever see it there because it will probably just go, straight away.
On an individual mail's way out of the door it passes through the queue folder in IIS.
If you wanted to watch the Performance Counter to monitor this process you ould look at the "Remote Queue Length". (The reason for this is that the "Local Queue Length" monitors mails sent "Locally" within the network. "Remote" in this instance refers to "Outside into the world". The specific definition of "Local" escapes me as we send no local mail but I imagine it means queued to go to mailboxes contained within the specific installation of IIS on the server or any local grouping thereof.)
From an Exchange point of view it seems to be the equivalent of mails sent within the Exchange Domain and those sent out of that domain into the wider world.
Anyhow. The Remote Queue Length doesn't tell the whole story. You also have to look at the Remote Retry Queue, the number of Current Outbound Connections and, for belt and braces sake the actual number of files in the queue directory.
Here's why:
Remote Queue: All messages that have not
yet been sent, however many times
this has been tried. The number of
mails currently assigned to any open
connections are not counted as they
are in a state of "being tried".
Remote Retry Queue: All messages that
have not yet been sent that have, at
some point in the past, been assigned
to an open connection for delivery.
Obviously the delivery must have
failed or the message would have been
delivered. Any messages currently
assigned to an open connection for a
retry are not counted.
Current Outbound Connections: Shows when the
server is attempting to send queued
mails, more than one message can be
assigned to an outbound connection.
Messages thus assigned are not
counted in either the Remote Queue or
the Remote Retry queue. Physical
Files in the queue directory: This
shows the number of mails still in
the Queue directory. This will
decrease as mails are successfully
delivered.
Example: If you have 0 outbound connections and 50 mails in the Queue directory then the Remote Queue, Retry Queue and Physical files will all read at 50. When a retry flag is raised (this is a setting in IIS) the number of connections increases and the number of mails in the queues decreases. Until a mail is delivered the number of physical files remains the same. However as more than one mail can be sent on a current connection 1 connection may result in Remote Queue and Retry Queue lengths of 47 or lower. If, during the retry event, any mails are successfully delivered the number of physical files in the Queue directory will then decrease. When the connection closes the queue counters should all stabilise again.
Logging
It is possible with .Net's Mail library to Specify a Pickup directory separate from the IIS default. Here you can queue mails and get a bespoke service to occasionally move the mails into the IIS directory where the IIS service will take over and send out queued mails.
To do this you will be looking for the SmtpClient object's "DeliveryMethod" property which should be set to SmtpDeliveryMethod.SpecifiedPickupDirectory.
To actually set the SpecifiedPickupDirectory you should set the SmtpClient's PickupDirectoryLocation property.
When mails are delivered to this location they are stored as .eml files. The filename is a GUID. This means that multiple emails will be despatched in an essentially random order. You could, in theory, write code to address this situation if desired. The .eml file follows a standard format which can be read by opening the .eml in notepad. Parsing this will allow you to extract information for a log.
I hope this high level overview of the way the SMTP server in IIS works is of some assistance to someone in a similar position to the one I was in in March.
I would use the PerformanceCounter component to read the SMTP Service's Local Queue Length counter. That should keep you in control :-)
If your .net widget is bespoke, why not just throttle it's output to some (definable) throughput?
As an alternative you might be able to fiddle with some registry settings for the SMTP server.
http://blog.rednael.com/CommentView,guid,dc20366c-3629-490a-a8ee-7e8f496ef58b.aspx
Apparently there are also some WMI counters (SMTP Server\Remote Queue Length and SMTP Server\Remote Retry Queue Length) that will give you useful information.
http://www.tech-archive.net/Archive/Internet-Server/microsoft.public.inetserver.iis.smtp_nntp/2008-02/msg00011.html