Develop Reference

Microsoft Graph API - Exchange Online Messages call returns ServiceUnavailable

I am fetching messages from Exchange in Office365 using Microsoft Graph API.
However, for some folders I seem to get intermittent exceptions.
What we are using:
Microsoft.Graph Version 3.9.0 - Microsoft Graph Client Library for .Net
Microsoft.Graph.Core Version 1.21.0 - Microsoft Graph Core Client Library for .Net
This is the call being used:
'GET /v1.0/users/{id}/mailFolders/{id}/messages'
And this is the error (ServiceUnavailable with UnknownError as inner exception):
Status Code: ServiceUnavailable Microsoft.Graph.ServiceException:
Code: UnknownError Message: Error while processing response. Inner
error:
AdditionalData:
date: 2020-08-04T13:55:33
request-id: ** ClientRequestId: **
Code: UnknownError Message: Error while processing response. Inner error:
AdditionalData:
date: 2020-08-04T13:55:33
request-id: ** ClientRequestId: **
What I've tried:
Throttling:
These are usually the errors we would see with throttling. However, in this case, there seems to be no indication of throttling being applied. There isn't any 'back-off' time returned in the result. Other requests to different folders returns just fine too. By applying our own 'back-off' time (ranging between 5mins-20mins does not seem so make a difference either).
Beta endpoint:
The call posted above shows /v1.0 used. We've also switched to the /beta endpoint, with no difference.
Amount of mails retrieved:
Graph allows us to retrieve up to 999 mails at a time. We've reduced that all the way down to a mail or 2 at a time, but it still returns with the same error.
Delta token:
We've also tried switching over to using the delta token in order to retrieve the mails. This also returns with the same error.
Graph downgrade:
Hoping that there is some difference in the last few versions, we downgraded Graph. There was no difference.
Check local sync issues:
I've noticed in the past (quite a while back), that when doing this call for a folder that has potential local sync issues, this is the same type of error response returned. In this case, there is no reason to believe that these are local sync issues.
Additional:
When setting up the httpProvider, I've removed the default retry handlers as well. I've seen that using the default retry handler, it would automatically catch the 'ServiceException' and do internal retries (not adhering to back offs (not that there is any)), and would result in a tooManyRetries or a timeout (hiding the actual issue). By removing the default retry handler, we can see the actual 'ServiceException' error returned by the server.
When:
Based on our telemetry, this seems to have started happening a lot more frequently since around the 11-13th of June. Before that we did not experience any issues.
There are days that the requests work, but they are few and far in between.
This is quite a big issue for us, so any suggestions would be greatly appreciated. Any specific Microsoft Support channel that I can log this with would also help.
Thanks in advance.

Paypal Sandbox (express-checkout) return internal error

since yesterday evening i've a problem with my "express-checkout API"
Every time if I called the uri
https://www.sandbox.paypal.com/cgi-bin/webscr?cmd=_express-checkout&token=EC-0W349......
I get the error
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, webmaster#paypal.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
Yesterday in the morning everything was ok and since I didn't change anything also nothing on the API class.
My Server support the sha256 and the protocol Tls12.
I've nothing in the Logfile from server.
The Live environment works fine but the sandbox environment doesn't work since yesterday.
Is it an error on paypal site or on my side?
I've send also a mail to paypal but I get an autoresponse Message.
Please help me and sry for my bad english.
Thanks and best regards.

This may be related to the recent sandbox downtime over the past 24 hours or so. The problem was diagnosed by the team and they were able to repair the problem. Can you check again to see if it's working now please?

C# EWS Exchange API => sometimes work

I've seen loads post on this forum about exchange API, but I've never found something like I have.
I' m trying to connect to Exchange using Exchange API in order to get all room's meetings.
My problem : Sometimes, it's working, I can connect and get all my informations. And 5m later, impossible to connect. I got Unauthorized.
It's not an error on code because it's working, I think it's a problem with exchange server, but I really don't know what it is...
On the next picture, there are traces.
On the left, when it's working, And the same 30m later (same identification, same room) which is not working.
http://hpics.li/2d9b216
If you have any ideas ? I really don't know what to do.
thanks (and sorry for my english !)

I notice that the working request comes from server A (look at the value of X-FEServer header), and the failing one comes from server B. Perhaps there is a configuration problem on server B.

thx for this.
I've done some tests : I've modified my host file in my computer. when the url is pointing to serveur A, it's working. When it's on the other server, not working. It's the same configuration on both, cannot find what's wrong. So I will conserve my actual host file (pointing always on serveur A).

Google.Apis.Admin.Email_Migration_v2 [HTTP Status Code 412 – Limit Reached]

Edit 2:
Client Library: After reviewing it is not easily suggested that this is for the .NET client library.
DLL: Google.Apis.Admin.email_migration_v2.dll
What steps will reproduce the problem?
Generate a process which contains a
Google.Apis.Admin.email_migration_v2.AdminService instance for each
unique Google Apps Gmail mailbox that will have messages sent to it.
All of the AdminService objects generated use the same OAuth2.0
credentials and application name. Each AdminService object generated
will only send messages to one Google Apps user’s mailbox. For
example, if we were sending messages to five different Google Apps
Gmail mailboxes we would generate five AdminService objects to send
messages; one for each user’s mailbox.
Biggest thing to note is that each AdminService object created is created on a separate process.
AdminService objects were given a FileDataStore object to change the location of where the refresh token is stored; C:\ProgramData\SomeFile\SomeFile.
Supplied appropriate scopes to the credentials.
Begin sending mail messages on each process. Using one thread to send messages in each process, so only one message is sent at a time to each user’s mailbox.
Each message sent gets its own instance of MailItem and MailResource.InsetMedia
The MailResource.InsertMedia object is generated for each item by calling AdminService.Mail.Insert(MailItem, string, Stream, string) method.
When our code makes the call to MailResource.InsertMediaUpload.UploadAsync(CancellationTokenSource).Result is where we can receive the error.
The error is caught and handled (logged) from the return type of the aforementioned call; the type is Google.Apis.Upload.IUploadProgress. The exception is handled using the IUploadProgress.Exception property.
What is the expected output? What do you see instead?
The expected output would be a successful message response or the exception property of the IUploadProgress to be null after the return of the task. Instead we are receiving the following error message:
The service admin has thrown an exception:
Google.GoogleApiException:Google.Apis.Requests.RequestError
Limit reached. [412]
Errors [Message[Limit reached.] Location[If-Match - header] Reason[conditionNotMet] Domain[global]]
at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at Google.Apis.Upload.ResumableUpload`1.d__e.MoveNext()
What version of the product are you using?
Google.Apis.Admin.Email_Migration_v2 (1.8.1.20)
What is your operating system?
Windows Server 2008 R2 Enterprise (SP1)
What is your IDE?
Visual Studio 2013 Premium
What is the .NET framework version?
4.0.30319
Please provide any additional information below.
Non-consecutive messages can fail (with the 412 http status code
provided above) during the process of sending the messages. Once we
receive this error other messages sent after the failed message(s)
can succeed. (Items can fail at any point during the process
beginning, middle or end.)
Each message sent has nearly identical content. The size of the
messages range from 1KB to 100KB including the size of all associated
attachments, not all messages have attachments.
Reprocessing the failed items at a later time results in successful
message responses and the appropriate items are sent to the user’s
Google Apps Gmail Inbox.
The maximum number of Google Apps user’s mailboxes sent to at one
time was ten.
After checking the quotas of our Google Developers Console project:
We were nowhere near the specified limit of 20 requests a second for
the Email Migration API; maxed out at sending 7 requests a second.
Only 2% of the maximum daily requests had been reached.
All messages sent had the same label; the label was well under the
225 character limit. Actually all of the labels/sub-labels applied
together only surmounted to 40 characters.
This error message can still be received when sending to only one
Google Apps user’s mailbox; only using one process and one thread.
Each process is normally sending anywhere from 1000-5000 messages.
I have not found a lot of specific documentation to explain this particular error in enough detail to remedy the problem at hand.
Questions:
So what exactly does this 412 http status code mean? What limit is being encountered that this message is referring to?
Shouldn’t we be receiving some form of 5XX error from the server if we are hitting a limit? In which case wouldn’t the built in exponential back off policy kick in?
a. Unless the server is checking the POST request for a pre-condition about a server side limit then telling the client to back off which is what a 412 error seems to typically indicate. In that case please give as much detail as possible for question 1.
Sorry for the extensive post! Thanks for your time! I will also be creating a defect/issue in Google's .NET issue tracker and providing a link.
Edit 1:
For anyone interested in following this issue here is a link to the submitted item in Google's issue tracker for .NET.
Submitted Issue
For reference it is issue 492.

I am not quite sure where you see the "the specified limit of 20 requests a second for the Email Migration API". Reminder: the QPS limit you see in the Google Developers Console project is not the actual default limit. You can change that limit to anything you want, and thus, that's not the actual limit for the API. It is really just for managing the consumption of the API quota (some APis will have a much higher QPS where you can adjust it to lower for different projects across your console).
According to the email migration APi documentation, the QPS is 1 request per second (the link is here: https://developers.google.com/admin-sdk/email-migration/v2/limits).
I have experienced 412 errors when the QPS limit is being hit, and I have also seen the 412 error returned when I am uploading too much data to a single domain. How much data are you loading all at once? I would suggest doing an exponential backoff to see if the issue would disappear.

I believe I have found an answer to this problem, though I will advise a disclaimer, I do not work for Google and cannot be 100% sure of the accuracy; you've been warned. This should at least hold true for the .NET version of Google's Email Migration v2 API. I cannot guarantee how other APIs work because I do not use them..
Through working with this API in spurts for well over eight months now, it appears that if an application or multiple applications are to send messages to a single Google Apps user/mailbox consistently, at a faster rate than which Google servers can process, then at some rate you should start to get a bunch of GoogleApiExceptions stating "412 - Limit Reached" when sending new messages. What we have gathered through using our application is that each Google Apps user/mailbox has its own pending items queue. When you send a message to Google Apps it is first put into this queue before being processed by a Google Server and put into the user's mailbox. If this queue becomes full and you attempt to send another message you will receive a 412 error.
Options are to wait before sending another message, you'll have to wait however long the Google server takes to process the next message in the user's queue before sending another; which is unpredictable. The better option in my opinion is to start sending messages to another Google Apps user; because each user appears to have its own message queue. Be sure to stop sending to the user who is consistently getting 412 errors. This will give the Google server some time to process that user's packed message queue. Note each pending messages queue appeared to hold about 100-150 items before throwing 412 errors.
503 errors appear to occur when sending messages into a user's mailbox queue at a higher rate than 1 request per second. As Emily has stated "the QPS limit you see in the Google Developers Console project is not the actual default limit" it is truly 1 QPS per Google Apps user.
As for the exponential back-off it is supposed to be implemented automatically see this. Note Peleyal appears to be the gentleman in charge of the API; can be noted from the download page for the API.
This took us a little while to figure out so cheers if you're having this issue! Please if you find any contradicting information correct any mistakes found in this answer or make your own!!

IBM WebSphere XMS.Net CWSMQ0082E error

On several occasions I have received the following error from a .Net (C#, 4.0) application out of the blue on sending a message thru a producer:
CWSMQ0082E: Failed to send to CompCode: 2, Reason: 2009. A problem was encountered whilst sending a message. See the linked exception for more information.
Of course, the LinkedException (why not use the InnerException IBM???) is null i.e. no more information available.
Code I'm using (pretty straightforward):
var m = _session.CreateBytesMessage();
m.WriteBytes(mybytearray);
m.JMSReplyTo = myreplytoqueue;
m.SetIntProperty(XMSC.JMS_IBM_MSGTYPE, MQC.MQMT_DATAGRAM);
m.SetIntProperty(XMSC.JMS_IBM_REPORT_COA, MQC.MQRO_COD);
m.SetIntProperty(XMSC.JMS_IBM_REPORT_COD, MQC.MQRO_COA);
myproducer.Send(m, DeliveryMode.Persistent, mypriority, myttl);
(Offtopic: I hate the SetIntProperty way of setting properties. Which <expletive deleted> came up with that idea? It takes ages to look up all sorts of constants all over the place and its allowed values.)
The exception is thrown on the .Send method. I'm using XMS.Net (IA9H / 2.0.0.7). The only Google result that turns up turns out to have a different reason code (and even if it were the same, it should be fixed in my version if I understand correctly). This occurs randomly (though it seems to happen more often when it's been a while since a message has been sent/received) and I have no way to reproduce this.
I have ab-so-lute-ly no idea how to troubleshoot this or even where to start looking. Is this something caused by the server-side? Is it caused by XMS.net or some underlying IBM WebSphere MQ infrastructure?
Some results that I found that seem similar are suggesting to set SHARECNV to any value higher than 0 or to "true" / "yes" but the documentation explicitly tells me the default is 10. Also; I have no idea if this is the cause so changing it to another value feels like a shotgun approach.
Anybody any idea on how to go about solving this? I could of course just catch the exception, tear everything (channels, sessions, whatever) down and restart but that's just plain ugly IMHO.

The 2009 return code means "Connection Broken." Basically, the underlying TCP socket is gone and the client finds out about it at the time of the API call. It is possible to tune the channels using heartbeat and keepalive so that WMQ tries harde to keep the socket alive. However if the socket is timed out by the underlying infrastructure, nothing WMQ can do will help. Examples we've seen are that firewalls and load balancers are often set to detect idle connections and sever them.
Modern versions of WMQ client will attempt to reconnect transparently. The application just blocks a bit longer when this occurs.
Short of using the automatic reconnect, the only solution is in fact to rebuild the connection. Since it will get a new connection handle, all the object handles must be rebuilt as well.
Many of the tuning functions described here are available through the client configuration file, available in v7.0 and greater clients. In particular, the TCP stanza of that file enables keepalive. (The TCP spec says that if keepalive is provided, it must be disabled by default.) The QMgr has a similar ini file with configuration stanzas, including one for keepalive. The latest WMQ client is available as SupportPac MQC71 if you need that.

In cases where the main exception is sufficient enough to indicate the error, the inner exception will be null. In your case it's MQ reason code 2009 which means a connection to queue manager has been broken. The socket through which your application and queue manager were communicating was closed for some reason. The reason for socket close could be a network blip.
Along with suggestions T.Rob noted above, You could also run a XMS and Queue manager trace to understand the problem further. Please see the Troubleshooting chapter in XMS InfoCenter.
HTH