Develop Reference

"Resource temporarily unavailable" error from System.Net.Sockets

I am working on a .NET API that runs inside of a docker container. At some point it makes a call to a Python Flask API that is also running in a container.
var response = await httpClient.GetAsync("http://service-name:8000/actual/url")
which then produces the following error:
System.Net.Http.HttpRequestException: Resource temporarily unavailable
---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken
cancellationToken)
Has anyone had experience with this before and potentially knows a solution? I cant find much on the web about it at all. I have some seen some mentions of the issue potentially being related to the Flask API not using async methods but that doesnt make sense to me.
The Flask API produces the appropriate responses when accessed through a web browser or Postman using localhost:8000/actual/url and the container logs these responses. I have tried using the localhost URL in the .NET API but that does not work either.
If anymore information is needed please leave a comment and I will do my best to update the post quickly.
-- Christie

TLDR
A reason for the "Resource temporarily unavailable" error is when during name resolution the DNS Server responds with RCODE 2 (Server failure).
Long answer
I noticed the same behavior in a dotnet application running in a dotnet runtime alpine docker container. Here are the results of my investigation:
The error message "Resource temporarily unavailable" corresponds to the EAGAIN error code which gets returned by various functions from the C standard library. At first I suspected the connect() function because the C# stack trace indicates the error happening during the ConnectAsync() call of the c# socket. And indeed the EAGAIN error code appears in the man page of connect() with this description: "No more free local ports or insufficient entries in the routing cache".
I simulated a system with depleted local ports and noticed that a different exception gets thrown in that case, which rules out local port availability as a root cause for the original exception. Regarding the other mentioned cause in the man page it turns out that the routing cache was removed from Linux in 2012. commit
I started to look around for EAGAIN in the source of the musl C lib which is used in the dotnet runtime alpine docker container. After a while I finally noticed the gethostbyname2_r function which is used for resolving a domain name to an ip address via DNS. During System.Net.Sockets.Socket.ConnectAsync() the hostname is still a string and the name resolving happens in native code using the gethostbyname2_r function (or one of its variations).
The final question is: When does gethostbyname2_r return the EAGAIN error code? It's when the RCODE field in the header of the DNS Response has the value 2, which stands for "Server failure". source line 166
To verify this result I ran a simple mock DNS server which always returns the RCODE 2 in the DNS response. The resulting c# exception along with the stack trace matched the original exception exactly.

System.Net.WebException: The remote name could not be resolved

This kind of question has been asked several times, and I understand why it happens, and probably nothing we can do about it except retry.
I do have one question on name resolution though.
I am using AWS .Net SDK for 3.5 .Net. I am uploading a big file (>500MB up to 1.5GB, medical images). I call TransferUtility.Upload() method.
For most part the program works great.
Occasionally we get this error in the middle of the upload. Usually happens when the internet is slow.
I can catch the exception and retry, which means rery from the beginning since exception happens inside the AWS code.
My question is, if the program has resolved the s3 bucket name and has been uploading for a while why would it give me name resolution error instead of just using the cached resolved name?
Does each thread resolve the name independently and one of thread is failing since the network is saturated? Is this a computer setting? This error we were able to reproduce pretty consistently on a Windows 10 machine with Charter as ISP uploading a 800MB file.
The error occurred after about 250MB upload was done.
This is the actual exception
Exception during upload :Amazon.Runtime.AmazonServiceException:
A WebException with status NameResolutionFailure was thrown. --->
System.Net.WebException: The remote name could not be resolved: 'my-bucket.s3.amazonaws.com'

This web exception is telling you the there was an issue with the "Name Resolution". What it doesn't tell you is that the "name" it's referring to is the "EndpointRegion", for example: USEast1, USEast2 etc.
When using the Amazon.S3.Transfer.Transferutility it's crucial that the EndpointRegion you use in the Upload call MATCHES that of the bucket you're uploading into.
In my case using RegionEndpoint.GetBySystemName("USEast1") vs RegionEndpoint.GetBySystemName("US-East-1") was the difference maker.

Another cause for this issue could be DNS resolution. If your system is not able to perform DNS resolves it will give you this same error.

Google.Apis.Admin.Email_Migration_v2 [HTTP Status Code 412 – Limit Reached]

Edit 2:
Client Library: After reviewing it is not easily suggested that this is for the .NET client library.
DLL: Google.Apis.Admin.email_migration_v2.dll
What steps will reproduce the problem?
Generate a process which contains a
Google.Apis.Admin.email_migration_v2.AdminService instance for each
unique Google Apps Gmail mailbox that will have messages sent to it.
All of the AdminService objects generated use the same OAuth2.0
credentials and application name. Each AdminService object generated
will only send messages to one Google Apps user’s mailbox. For
example, if we were sending messages to five different Google Apps
Gmail mailboxes we would generate five AdminService objects to send
messages; one for each user’s mailbox.
Biggest thing to note is that each AdminService object created is created on a separate process.
AdminService objects were given a FileDataStore object to change the location of where the refresh token is stored; C:\ProgramData\SomeFile\SomeFile.
Supplied appropriate scopes to the credentials.
Begin sending mail messages on each process. Using one thread to send messages in each process, so only one message is sent at a time to each user’s mailbox.
Each message sent gets its own instance of MailItem and MailResource.InsetMedia
The MailResource.InsertMedia object is generated for each item by calling AdminService.Mail.Insert(MailItem, string, Stream, string) method.
When our code makes the call to MailResource.InsertMediaUpload.UploadAsync(CancellationTokenSource).Result is where we can receive the error.
The error is caught and handled (logged) from the return type of the aforementioned call; the type is Google.Apis.Upload.IUploadProgress. The exception is handled using the IUploadProgress.Exception property.
What is the expected output? What do you see instead?
The expected output would be a successful message response or the exception property of the IUploadProgress to be null after the return of the task. Instead we are receiving the following error message:
The service admin has thrown an exception:
Google.GoogleApiException:Google.Apis.Requests.RequestError
Limit reached. [412]
Errors [Message[Limit reached.] Location[If-Match - header] Reason[conditionNotMet] Domain[global]]
at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at Google.Apis.Upload.ResumableUpload`1.d__e.MoveNext()
What version of the product are you using?
Google.Apis.Admin.Email_Migration_v2 (1.8.1.20)
What is your operating system?
Windows Server 2008 R2 Enterprise (SP1)
What is your IDE?
Visual Studio 2013 Premium
What is the .NET framework version?
4.0.30319
Please provide any additional information below.
Non-consecutive messages can fail (with the 412 http status code
provided above) during the process of sending the messages. Once we
receive this error other messages sent after the failed message(s)
can succeed. (Items can fail at any point during the process
beginning, middle or end.)
Each message sent has nearly identical content. The size of the
messages range from 1KB to 100KB including the size of all associated
attachments, not all messages have attachments.
Reprocessing the failed items at a later time results in successful
message responses and the appropriate items are sent to the user’s
Google Apps Gmail Inbox.
The maximum number of Google Apps user’s mailboxes sent to at one
time was ten.
After checking the quotas of our Google Developers Console project:
We were nowhere near the specified limit of 20 requests a second for
the Email Migration API; maxed out at sending 7 requests a second.
Only 2% of the maximum daily requests had been reached.
All messages sent had the same label; the label was well under the
225 character limit. Actually all of the labels/sub-labels applied
together only surmounted to 40 characters.
This error message can still be received when sending to only one
Google Apps user’s mailbox; only using one process and one thread.
Each process is normally sending anywhere from 1000-5000 messages.
I have not found a lot of specific documentation to explain this particular error in enough detail to remedy the problem at hand.
Questions:
So what exactly does this 412 http status code mean? What limit is being encountered that this message is referring to?
Shouldn’t we be receiving some form of 5XX error from the server if we are hitting a limit? In which case wouldn’t the built in exponential back off policy kick in?
a. Unless the server is checking the POST request for a pre-condition about a server side limit then telling the client to back off which is what a 412 error seems to typically indicate. In that case please give as much detail as possible for question 1.
Sorry for the extensive post! Thanks for your time! I will also be creating a defect/issue in Google's .NET issue tracker and providing a link.
Edit 1:
For anyone interested in following this issue here is a link to the submitted item in Google's issue tracker for .NET.
Submitted Issue
For reference it is issue 492.

I am not quite sure where you see the "the specified limit of 20 requests a second for the Email Migration API". Reminder: the QPS limit you see in the Google Developers Console project is not the actual default limit. You can change that limit to anything you want, and thus, that's not the actual limit for the API. It is really just for managing the consumption of the API quota (some APis will have a much higher QPS where you can adjust it to lower for different projects across your console).
According to the email migration APi documentation, the QPS is 1 request per second (the link is here: https://developers.google.com/admin-sdk/email-migration/v2/limits).
I have experienced 412 errors when the QPS limit is being hit, and I have also seen the 412 error returned when I am uploading too much data to a single domain. How much data are you loading all at once? I would suggest doing an exponential backoff to see if the issue would disappear.

I believe I have found an answer to this problem, though I will advise a disclaimer, I do not work for Google and cannot be 100% sure of the accuracy; you've been warned. This should at least hold true for the .NET version of Google's Email Migration v2 API. I cannot guarantee how other APIs work because I do not use them..
Through working with this API in spurts for well over eight months now, it appears that if an application or multiple applications are to send messages to a single Google Apps user/mailbox consistently, at a faster rate than which Google servers can process, then at some rate you should start to get a bunch of GoogleApiExceptions stating "412 - Limit Reached" when sending new messages. What we have gathered through using our application is that each Google Apps user/mailbox has its own pending items queue. When you send a message to Google Apps it is first put into this queue before being processed by a Google Server and put into the user's mailbox. If this queue becomes full and you attempt to send another message you will receive a 412 error.
Options are to wait before sending another message, you'll have to wait however long the Google server takes to process the next message in the user's queue before sending another; which is unpredictable. The better option in my opinion is to start sending messages to another Google Apps user; because each user appears to have its own message queue. Be sure to stop sending to the user who is consistently getting 412 errors. This will give the Google server some time to process that user's packed message queue. Note each pending messages queue appeared to hold about 100-150 items before throwing 412 errors.
503 errors appear to occur when sending messages into a user's mailbox queue at a higher rate than 1 request per second. As Emily has stated "the QPS limit you see in the Google Developers Console project is not the actual default limit" it is truly 1 QPS per Google Apps user.
As for the exponential back-off it is supposed to be implemented automatically see this. Note Peleyal appears to be the gentleman in charge of the API; can be noted from the download page for the API.
This took us a little while to figure out so cheers if you're having this issue! Please if you find any contradicting information correct any mistakes found in this answer or make your own!!

HttpModule ATL Server Service InputStream Failure

I'm trying to add some SOAP message logging capabilities to an old, old ATL Server web service that is running in integrated mode in IIS 7.5 on a Windows Server 2008 box, but running into a strange problem. For further background, I've added the assembly that contains the HttpModule to the modules element of the web.config for the ATL Server web service.
I've been following the answer provided here and the logging itself works great.
However, whenever I use this logging capability, the service responds with "SOAP Invalid Request", while the log file has the SOAP message as expected. I've done lots of fiddling around with it and figured out that this only happens if/when I access the request object's InputStream property in my handler for the BeginRequest event. It will fail if I even simply set a variable to the length of the InputStream like this:
private void OnBegin(object sender, EventArgs e)
{
var request = _application.Request;
//this will blow up
var foo = request.InputStream.Position;
}
If I don't touch the InputStream in my handler (which doesn't do much good when I'm only doing this to log the contents of the request, obviously), the request goes through perfectly.
I can access header values in the Request object and various other properties of the HttpApplication involved, but accessing the InputStream causes the service to choke.
Is there something intrinsic with ATL Server that will prevent me from doing this logging? Do I need to add some sort of locking or other safeguard in my BeginRequest handler to make sure this behaves? Is my handler hosing up the InputStream somehow causing it to be unusable for the service?
Another way of approaching this is to ask if there is a way to see the request as it gets to my service (i.e. after this HttpModule executes)?
It may also be worth noting that I am using SoapUI to test the service.
EDIT:
I've now done some failed request tracing in IIS and I get this error message:
ModuleName IsapiModule
Notification 128
HttpStatus 500
HttpReason Internal Server Error
HttpSubStatus 0
ErrorCode 0
ConfigExceptionInfo
Notification EXECUTE_REQUEST_HANDLER
ErrorCode The operation completed successfully. (0x0)
This comes in the handler for the ATL Server web service (i.e. the DLL for the service). Directly before that is the "GENERAL_READ_ENTITY_START" and "GENERAL_READ_ENTITY_END" messages, and the "END" has this message:
BytesReceived 0
ErrorCode 2147942438
ErrorCode Reached the end of the file. (0x80070026)
Does that mean what I think it means? That the handler isn't getting any data? Is this more evidence pointing towards my HttpModule messing with the Request's InputStream?

Are you sure your request object is valid? You're doing things slightly differently here from the sample you reference. They are extracting the stream from the sender argument whereas you obviously rely on a member variable.

So I finally determined that this wasn't a workable approach: I couldn't get the HttpModule to fire at all in IIS 6 (which I would need to have it to do for it to be an acceptable solution). I tried setting the Filter property on the Request object and all sorts of other crazy ideas, but none that led me to be able to both record the request body in the HttpModule and have the service still work.
So I did more digging and came upon this article on codeproject that talks about the inner workings of ATL Server, specifically the HandleRequest method in atlsoap.h. I mucked around in there for a while and figured out a way to get at the request body in there, and it was pretty simple to write it to a file manually from there.
For those curious, this is the final code I added to HandleRequest():
//****************************************REQUEST LOGGING**********************************************************
BYTE* bytes = pRequestInfo->pServerContext->GetAvailableData();
FILE* pFile;
pFile = fopen("C:\\IISLog\\ATL.txt", "a");
fwrite(bytes, 1, pRequestInfo->pServerContext->GetAvailableBytes(), pFile);
fclose(pFile);
//****************************************REQUEST LOGGING**********************************************************
I am going to still play around with it a bit more, but I have what appears to be a workable solution.

IBM WebSphere XMS.Net CWSMQ0082E error

On several occasions I have received the following error from a .Net (C#, 4.0) application out of the blue on sending a message thru a producer:
CWSMQ0082E: Failed to send to CompCode: 2, Reason: 2009. A problem was encountered whilst sending a message. See the linked exception for more information.
Of course, the LinkedException (why not use the InnerException IBM???) is null i.e. no more information available.
Code I'm using (pretty straightforward):
var m = _session.CreateBytesMessage();
m.WriteBytes(mybytearray);
m.JMSReplyTo = myreplytoqueue;
m.SetIntProperty(XMSC.JMS_IBM_MSGTYPE, MQC.MQMT_DATAGRAM);
m.SetIntProperty(XMSC.JMS_IBM_REPORT_COA, MQC.MQRO_COD);
m.SetIntProperty(XMSC.JMS_IBM_REPORT_COD, MQC.MQRO_COA);
myproducer.Send(m, DeliveryMode.Persistent, mypriority, myttl);
(Offtopic: I hate the SetIntProperty way of setting properties. Which <expletive deleted> came up with that idea? It takes ages to look up all sorts of constants all over the place and its allowed values.)
The exception is thrown on the .Send method. I'm using XMS.Net (IA9H / 2.0.0.7). The only Google result that turns up turns out to have a different reason code (and even if it were the same, it should be fixed in my version if I understand correctly). This occurs randomly (though it seems to happen more often when it's been a while since a message has been sent/received) and I have no way to reproduce this.
I have ab-so-lute-ly no idea how to troubleshoot this or even where to start looking. Is this something caused by the server-side? Is it caused by XMS.net or some underlying IBM WebSphere MQ infrastructure?
Some results that I found that seem similar are suggesting to set SHARECNV to any value higher than 0 or to "true" / "yes" but the documentation explicitly tells me the default is 10. Also; I have no idea if this is the cause so changing it to another value feels like a shotgun approach.
Anybody any idea on how to go about solving this? I could of course just catch the exception, tear everything (channels, sessions, whatever) down and restart but that's just plain ugly IMHO.

The 2009 return code means "Connection Broken." Basically, the underlying TCP socket is gone and the client finds out about it at the time of the API call. It is possible to tune the channels using heartbeat and keepalive so that WMQ tries harde to keep the socket alive. However if the socket is timed out by the underlying infrastructure, nothing WMQ can do will help. Examples we've seen are that firewalls and load balancers are often set to detect idle connections and sever them.
Modern versions of WMQ client will attempt to reconnect transparently. The application just blocks a bit longer when this occurs.
Short of using the automatic reconnect, the only solution is in fact to rebuild the connection. Since it will get a new connection handle, all the object handles must be rebuilt as well.
Many of the tuning functions described here are available through the client configuration file, available in v7.0 and greater clients. In particular, the TCP stanza of that file enables keepalive. (The TCP spec says that if keepalive is provided, it must be disabled by default.) The QMgr has a similar ini file with configuration stanzas, including one for keepalive. The latest WMQ client is available as SupportPac MQC71 if you need that.

In cases where the main exception is sufficient enough to indicate the error, the inner exception will be null. In your case it's MQ reason code 2009 which means a connection to queue manager has been broken. The socket through which your application and queue manager were communicating was closed for some reason. The reason for socket close could be a network blip.
Along with suggestions T.Rob noted above, You could also run a XMS and Queue manager trace to understand the problem further. Please see the Troubleshooting chapter in XMS InfoCenter.
HTH