Develop Reference

"Resource temporarily unavailable" error from System.Net.Sockets

I am working on a .NET API that runs inside of a docker container. At some point it makes a call to a Python Flask API that is also running in a container.
var response = await httpClient.GetAsync("http://service-name:8000/actual/url")
which then produces the following error:
System.Net.Http.HttpRequestException: Resource temporarily unavailable
---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken
cancellationToken)
Has anyone had experience with this before and potentially knows a solution? I cant find much on the web about it at all. I have some seen some mentions of the issue potentially being related to the Flask API not using async methods but that doesnt make sense to me.
The Flask API produces the appropriate responses when accessed through a web browser or Postman using localhost:8000/actual/url and the container logs these responses. I have tried using the localhost URL in the .NET API but that does not work either.
If anymore information is needed please leave a comment and I will do my best to update the post quickly.
-- Christie

TLDR
A reason for the "Resource temporarily unavailable" error is when during name resolution the DNS Server responds with RCODE 2 (Server failure).
Long answer
I noticed the same behavior in a dotnet application running in a dotnet runtime alpine docker container. Here are the results of my investigation:
The error message "Resource temporarily unavailable" corresponds to the EAGAIN error code which gets returned by various functions from the C standard library. At first I suspected the connect() function because the C# stack trace indicates the error happening during the ConnectAsync() call of the c# socket. And indeed the EAGAIN error code appears in the man page of connect() with this description: "No more free local ports or insufficient entries in the routing cache".
I simulated a system with depleted local ports and noticed that a different exception gets thrown in that case, which rules out local port availability as a root cause for the original exception. Regarding the other mentioned cause in the man page it turns out that the routing cache was removed from Linux in 2012. commit
I started to look around for EAGAIN in the source of the musl C lib which is used in the dotnet runtime alpine docker container. After a while I finally noticed the gethostbyname2_r function which is used for resolving a domain name to an ip address via DNS. During System.Net.Sockets.Socket.ConnectAsync() the hostname is still a string and the name resolving happens in native code using the gethostbyname2_r function (or one of its variations).
The final question is: When does gethostbyname2_r return the EAGAIN error code? It's when the RCODE field in the header of the DNS Response has the value 2, which stands for "Server failure". source line 166
To verify this result I ran a simple mock DNS server which always returns the RCODE 2 in the DNS response. The resulting c# exception along with the stack trace matched the original exception exactly.

Azure service bus relay connecting to unknown ip address: 40.112.124.x:9352

We deliver on-premise software that is exposed to the cloud using Azure Service bus relay, the basic code we use to expose is as follows (I have removed everything identifiable):
ServiceHost sh = new ServiceHost(typeof(BasicHttpEntityService));
BasicHttpRelayBinding basicHttpRelayBinding = new BasicHttpRelayBinding();
Uri uriEndPointAddress = ServiceBusEnvironment.CreateServiceUri("https", "ourdomain", "test-url-appendage");
m_shRelayServiceHost.AddServiceEndpoint(
typeof(IMyService),
basicHttpRelayBinding,
uriEndPointAddress
).Behaviors.Add(
new TransportClientEndpointBehavior
{
TokenProvider = TokenProvider.CreateSharedSecretTokenProvider(
"MyUser",
"MyPassword")
});
sh.Open();
This works fine at most of our customers, however, one of our customers has a strict firewall policy.
According to the SB guidelines we have found, we asked them to open ports 9351-9354 to ourdomain.servicebus.windows.net. Now we find out that when there is an incoming request, the service connects to both 'ourdomain' (we see this succeeds in Wireshark, and also in the WCF log) AND an unknown (to us) service on 40.112.124.x:9352 (the last octet changes with every request).
I have been able to reproduce the problem in my development environment by disallowing connections to any 40.x.x.x address on any port. This is what happens in the WCF log:
System.Net.Sockets.SocketException (0x80004005): An attempt was made to access a socket in a way forbidden by its access permissions 40.112.124.25:9352
Server stack trace:
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at Microsoft.ServiceBus.RelayedConnectionSession.ConnectAsyncResult.<GetAsyncSteps>b__4(ConnectAsyncResult thisRef, IAsyncResult r)
at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
Exception rethrown at [0]:
at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.ServiceBus.RelayedConnectionSession.EndConnect(IAsyncResult result)
There is no DNS-request going out during this time, so there is no host name that provides any clues to the function of this outgoing connection.
From my investigation, this appears to be a Microsoft controlled subnet, so I'm fine with the relay service connecting to it, but I would like to know:
Is this additional connection optional?
If not, should we allow the entire subnet?
Could this IP-range change in the future? Is it hardcoded somewhere?

In the end, we requested support from Microsoft.
In short their answers were as follows:
Is this additional connection optional?
No it is not optional. For the relay listener, there is a control channel on port 5671, this connection is always there. Then there is a data channel on portal 9352, this connection established when there is a relay client tries to communicate with the listener.
Could this IP-range change in the future?
Currently, for relay this IP can change, so you need to allow the IP range for the entire datacenter in your region (https://www.microsoft.com/en-us/download/confirmation.aspx?id=41653). The SB product team will be working on to significantly reduce this IP range in the future, to make it much more predictable. There is no exact ETA on this future.
So the good news is they are working on it. The bad news is, that right now, we will need to add a LOT of IP addresses to the white-list to ensure smooth operation.

Windows Server can not use WCF Server

I can use this program on my own computer, but I can not use on the server.
Server use supreme authority Administrator to open the program.
Server WCF HTTP Activation Feature with .NET4.5 is opening.
Server endpoint address use "http://localhost" like following
endpoint address="http://localhost" binding="basicHttpBinding" bindingConfiguration="NewBinding0" name="ProductService" contract="ProductService.IWCFProductService"
Wrong Message:
The communication object, System.ServiceModel.ServiceHost, cannot be used for communication because it is in the Faulted state.
Stack trace at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout)
at System.ServiceModel.ServiceHostBase.System.IDisposable.Dispose()
at FileUtilityHelperService.Program.Main(String[] args)

This simply means that there has been an unexpected exception somewhere in your code. The error message "...because it is in the Faulted state" means that communication between the server and the client is unusable.
What you need is better error handeling.
Use try/catch around your code that can potentially generate an exception
Always order exeptions from the most specific to the least specific
You would probably like to log your errors to a file or database, log4net is great for this (and comes as a nuget package)

Can Linq AsParallel() dispose of SoapHttpClientProtocol objects prematurely?

In an ASP.Net MVC 4 web application that I'm working on. I have one page that basically generates a report by getting data from a SOAP service.
My code basically looks like this
List<CustomThings> serverInfos = ServerInfos;
serverInfos.AsParallel().ForAll(srvInfo =>
{
SoapHttpClientProtocol soapProxy = CreateProxy(srvInfo);
//call make soap calls through the soap client
//store results in the proper places
}
The reason I'm doing AsParallel here is because doing several requests over HTTP in a serial fashion takes forever. I should throw in that this code does work, although sporadically.
Is it possible that things are getting disposed of in an unpredictable fashion, and PLINQ is not a good solution for what I'm trying to do here?
Is it possible that another threading issue could cause an error which makes the soap client "give up"?
Additional Info
This particular soap proxy is talking to an ArcGIS Server. Normally, you can check the server logs and see when particular requests are inititiated and if the requests failed. There is nothing showing in these logs.
Here's an example of an inner exception stack trace I get from the AsParallel code.
Exception: System.AggregateException: One or more errors occurred.
---> System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the
server. ---> System.IO.IOException: Unable to read data from the
transport connection: An existing connection was forcibly closed by
the remote host. ---> System.Net.Sockets.SocketException: An existing
connection was forcibly closed by the remote host at
System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32
size, SocketFlags socketFlags) at
System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset,
Int32 size) --- End of inner exception stack trace --- at
System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset,
Int32 size) at System.Net.PooledStream.Read(Byte[] buffer, Int32
offset, Int32 size) at
System.Net.Connection.SyncRead(HttpWebRequest request, Boolean
userRetrievedStream, Boolean probeRead) --- End of inner exception
stack trace --- at
System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest
request) at
System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest
request) at
System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String
methodName, Object[] parameters) at
ESRI.ArcGIS.SOAP.FeatureServerProxy.Query(Int32 LayerOrTableID, String
DefinitionExpression, QueryFilter QueryFilter, ServiceDataOptions
ServiceDataOptions, String GdbVersion, Double MaximumAllowableOffset)
at
System.Linq.Parallel.SelectQueryOperator2.SelectQueryOperatorResults.GetElement(Int32
index) at System.Linq.Parallel.QueryResults1.get_Item(Int32 index)
at
System.Linq.Parallel.PartitionedDataSource1.ListContiguousIndexRangeEnumerator.MoveNext(T&
currentElement, Int32& currentKey) at
System.Linq.Parallel.PipelineSpoolingTask2.SpoolingWork() at
System.Linq.Parallel.SpoolingTaskBase.Work() at
System.Linq.Parallel.QueryTask.BaseWork(Object unused) at
System.Linq.Parallel.QueryTask.<.cctor>b__0(Object o) at
System.Threading.Tasks.Task.InnerInvoke() at
System.Threading.Tasks.Task.Execute()

PLINQ does not even know your connection object exists. It cannot close it.
Read the message carefully:
An existing connection was forcibly closed by the remote host.
The server closed the connection in an unexpected way. Your client is not at fault.
Interpreting the exception precisely is an essential debugging skill. This information was right there in the exception message.
Maybe you are generating too much load. Set a sustainable degree of parallelism. The default heuristics are for CPU work, not for IO.
.WithDegreeOfParallelism(10)
A connection that was expected to be kept alive was closed by the server.
This could mean that the server does not support HTTP keep alive.

I don't think you are doing anything terribly wrong with AsParallel for your Soap HTTP requests, and i don't think it is a threading issue.
However, the parallel requests obviously push your client/server to the number of connection limits, and that is why you are seeing the connections getting closed.
I would bet your client, server or both are not configured to handle the number of concurrent connections you are issuing. That is why it works when you run the requests in serial fashion.
I guess you don't have access to server config, so one thing you could do is to control the number of parallel requests you issue to the server at the same time by setting the ParallelEnumerable.WithDegreeOfParallelism setting like in the following snippet:
.AsParallel()
.WithDegreeOfParallelism(15)
That way you control the parallelism, and don't risk overloading the server with a large number of requests on a small number of connections.
Regarding the client you should make sure that you have set the max. number of concurrent client connections to an appropriate number, just to make sure that your requests can use separate connections to the server, and prevent reusing connections which could cause your Keep-Alive issues.
The server could close the connection if the number of requests using a connection has exceeded the keep alive max number of connections or if it exceeds the timeout settings.
You can set the client connection limit programmatically using the ServicePointManager.DefaultConnectionLimit setting. E.g. you could set it to 50:
System.Net.ServicePointManager.DefaultConnectionLimit = 50;
Here is an example setting max. connections to 50 using the config file:
<configuration>
<system.net>
<connectionManagement>
<add address="*" maxconnection="50" />
</connectionManagement>
</system.net>
I used "50" just as an example, you should determine/calculate/measure what is the best setting for your setup.
Also make sure you are disposing your HTTP Connections properly after each request to prevent connection timeouts.

System.ServiceModel.CommunicationObjectFaultedException

I am facing the following problem on one server only on another server or locally it is working properly.
I am using IIS6 Windows 2003. Is there any setting related problem?
System.ServiceModel.CommunicationObjectFaultedException: The
communication object, System.ServiceModel.Channels.ServiceChannel,
cannot be used for communication because it is in the Faulted state.
Server stack trace: at
System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan
timeout) Exception rethrown at [0]: at
System.Runtime.Remoting.Proxies.RealProxy.

You cannot use a channel that has been faulted. You will need to recreate a new channel.
My toolkit has inbuilt support for this so that you can seamlessly make another call to the channel. It handles this scenario internally.
See http://neovolve.codeplex.com/releases/view/53499.

First, we need to find out what the real problem or issue is. For this set includeExceptionDetailInFaults to true as below in you service App.Config file:
serviceDebug includeExceptionDetailInFaults="true"
Then you will get the real exception detail when you catch an exception. I was getting AccessDeniedException. So I ran this command from Power Shell:
netsh http add urlacl url=http://+:8080/ user=Domainxxx\UserNamexxx. Problem solved.