I have an dotnet 5 service that sends emails using the SmtpClient. This service is deployed as a linux docker image running in AWS EKS (Kubernetes).
Running locally on my machine, everything works fine, but when it's deployed to our K8s environment, every SmtpClient.SentAsync call sits for around 10 minutes, before it finally crashes with the following exception:
System.Net.Mail.SmtpException: Failure sending mail.
---> System.Net.Sockets.SocketException (99): Cannot assign requested address
I've researched this error, and there is a lot of talk about using/not using localhost as a host name. In my case, I'm not using local host at all. I'm just trying to send outbound emails using SmtpClient, where the host is a 3rd party SMTP server.
I've tried running this as a linux docker container on my own dev machine, and it also works fine, so it seems to not like something about my EKS environment, yet I can't figure out what that is. Any ideas?
Update:
FYI, the pod can access the internet just fine, using the HttpClient for example. I can also make http calls to it's API on port 80. So it's binding just fine in that regard. It just seems to be an issue when using the SmtpClient.
Stack Track:
System.Net.Mail.SmtpException: Failure sending mail.
---> System.Net.Sockets.SocketException (99): Cannot assign requested address
at System.Net.Sockets.Socket.BeginConnectEx(EndPoint remoteEP, Boolean flowContext, AsyncCallback callback, Object state)
at System.Net.Sockets.Socket.UnsafeBeginConnect(EndPoint remoteEP, AsyncCallback callback, Object state, Boolean flowContext)
at System.Net.Sockets.Socket.PostOneBeginConnect(MultipleAddressConnectAsyncResult context)
--- End of stack trace from previous location ---
at System.Net.Sockets.Socket.DoMultipleAddressConnectCallback(Object result, MultipleAddressConnectAsyncResult context)
at System.Net.Sockets.Socket.MultipleAddressConnectCallback(IAsyncResult result)
--- End of stack trace from previous location ---
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.Sockets.TcpClient.EndConnect(IAsyncResult asyncResult)
at System.Net.Mail.SmtpConnection.ConnectAndHandshakeAsyncResult.InitializeConnectionCallback(IAsyncResult result)
--- End of stack trace from previous location ---
at System.Net.Mail.SmtpConnection.ConnectAndHandshakeAsyncResult.End(IAsyncResult result)
at System.Net.Mail.SmtpClient.ConnectCallback(IAsyncResult result)
...
Related
I'm encountering a situation on production that is difficult to debug.
Sometimes the connection to an external service can't be established with the following error:
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 93.39.196.220:443
at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)
at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
Sometimes it works without any problem. This is what I've tried:
Replacing the way I perform the connection (from Restsharp to HttpClient)
HttpClient is long lived as suggested by this post
Adjusting the timeout
Using async and sync code
Analyzing SNAT Port exhaustion
More info:
The IPs of my server are authorized by the external service (and if they weren't I would be receiving a different error)
While on production I'm having the problem I'm able to connect to the service from development or staging (so it shouldn't be a problem with the external service)
While I'm having trouble contacting this service I don't have trouble contacting a different (but similar) service
I'm on a Web App Service on Azure. I'm using the .NET framework 4.7.2
The same code sometimes works and sometimes it doesn't
This is (more or less) the current iteration
private async Task<ResultSet> SendRequestAsync(HttpClient client, RateRequestBody document){
ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls12;
var response = await client.PostAsXmlAsync(baseUrl + "/XMLServices", document);
if (response.IsSuccessStatusCode){
...
}else{
...
}
}
Any ideas on what I might have missed?
Thanks
The answer may lay here: Default SecurityProtocol in .NET 4.5
Assuming both servers are operating correctly, one works fine with your client code and the other gives you problems based on your 'More Info' section: "While I'm having trouble contacting this service I don't have trouble contacting a different (but similar) service"
I recommend removing or commenting out the following line and giving it a try:
//ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls12;
...or replace it with:
ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
First, for some context: I am using .NetCore to develop an API with Identity. Everything is on a Cloud server, inside a Docker. When a user is created, an email is sent to the new User using a mailkit and the webmail server through Plesk (Hosted on the same machine). The docker is accessed via a redirection trough Apache using a ProxyPass from the subdomain to the port on localhost
Everything works great while debugging trough JetBrain's Rider, but it is not able to process the email in the docker on the server.
Here is the stack:
System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000001, 11): Resource temporarily unavailable
at System.Net.Dns.InternalGetHostByName(String hostName)
at System.Net.Dns.ResolveCallback(Object context)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult)
at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult)
at System.Net.Dns.<>c.<GetHostAddressesAsync>b__25_1(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
I have yet to try and run the docker on another linux machine to test.
One of my current guess would be a problem with the SSL certificate, but I don't think it would cause a problem with the DNS or any internal socket.
Another guess is thats its a problem for the Docker to get the DNS Hostname, but since it works ok in a local.
Edit: I tried multiple time to run the docker on the mac and the error is still triggered once in a while but not always. It is although always triggered on the server and never send the email
After two weeks of research, I finally stumbled upon a solution for this:
The problem is related to the network, that was obvious, but it's precisely about how containers are isolated from one another. Problem is, the container has no outbound connection. A solution that work inside a standalone container is to use the --network host parameter, which would expose the host network to the container. Note that using this would remove the port mapping from the container since the container's port 5000 is now linked to the host's port 5000
Hope this solution can help others
as expected of a beginner IoT Edge developer, I started following the Microsoft tutorial on how to deploy custom code to an IoT Edge server Tutorial: Develop a C# IoT Edge module for Linux devices.
I was able to install Ubuntu 18.04 server on an industrial intel PC, download and configure the edgeHub and edgeAgent modules, create a free Azure and Docker Hub account, etc. Even the example code from the tutorial was succesfully built and deployed to my target device and all seemed well, all modules were talking to each other, etc.
Once I got the demo code to run, I started modifying the code to better suit my end goal, which is to capture network traffic generated by our industrial equipment, which uses UDP multicast/broadcast. Apparently I needed to configure my docker image to run in 'host' networking mode. And indeed, as soon as I told the azure edgeAgent to start the module container in host mode, the UDP packets started to come in.
HOWEVER, now my sample module is no longer able to connect to the iot hub and I'm at a complete loss. I've tried to run the edgeHub container in host mode as well, this doesn't seem to make any difference. The exact error I'm getting as a result of the ModuleClient.OpenAsync method is:
Unhandled exception. System.AggregateException: One or more errors occurred. (Transient network error occurred, please retry.)
---> Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: Transient network error occurred, please retry.
---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.InternalGetHostByName(String hostName)
at System.Net.Dns.ResolveCallback(Object context)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult)
at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult)
at System.Net.Dns.<>c.<GetHostAddressesAsync>b__25_1(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.OpenAsyncInternal(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.OpenAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass23_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
--- End of inner exception stack trace ---
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass33_0.<<OpenInternalAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()
at SampleModule.Program.Init() in /app/SampleModule/Program.cs:line 54
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait()
at SampleModule.Program.Main(String[] args) in /app/SampleModule/Program.cs:line 26
The code used for connecting:
ITransportSettings[] settings = { mqttSetting };
ModuleClient ioTHubModuleClient = await ModuleClient.CreateFromEnvironmentAsync(settings);
await ioTHubModuleClient.OpenAsync();
Console.WriteLine("IoT Hub module client initialized.");
So apparently the ModuleClient can't find the edgeHub anymore (Name or service not known?). I understood that the 'CreateFromEnvironmentAsync' uses environment variables set by the edgeAgent, but which ones? Anyone have any idea why this is? Even pointers in the right direction to start debugging this issue are greatly appreciated!
If I change my docker containers back to 'bridge' networking, the OpenAsync method works perfectly, but my UDP broadcast messages are no longer received, of course.
After some more digging around, I found the solution myself.
Module fails to restart because of transient network error
This article, although I had read it already multiple times, suggested that something was wrong with the /etc/hosts file. Sure enough, the entry for my device (127.0.1.1) was faulty, somewhere along the way I must have changed the hostname of my device and this change wasn't reflected in the hosts file.
Since that change, I got both AMQP and MQTT to work, however both with the edgeHub container also running in host mode. In bridge mode I ran across a new issue where the docker-proxy didn't bind the exported ports (which resulted in a new 'Connection refused' exception). It remains a mystery why...
Recently we were facing an issue in client place is that we were unable to communicate to web service even we checked that there is no system proxy set in that system, to overcome this we had given a dummy proxy (localhost:port) and its started working well but after some days with dummy proxy it was unable to communicate to service if we remove dummy proxy then it started working well. i am not able to understand what is blocking me to communicate to service. what may be the issue and how can we overcome this problem. This issue is happening only on some system.
Actually it has to go and hit this URL http://********-tech.in:8080/******dp/ClientVersion but suddenly it started hitting this one 125.21.244.38:8080 why i am not able to conclude.
2017-05-05 00:19:44,721 ERROR [Heartbeat Service ] - Exception while sending heartbeat to manager.
System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 125.21.244.38:8080
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
at System.Net.HttpWebRequest.GetRequestStream()
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
at HeartbeatWSImplService.heartbeat(String probeUid, heartbeatRequest heartbeatRequest)
at ProHance.ProbeHeartBeat.HeartBeat.timer_Elapsed(Object sender, ElapsedEventArgs e)
Please read this post here they are telling to set the default proxy for the webservice request, they are trying to the proxy details while connecting to google. If there is no proxy configured you can try the method mentioned there. Set the default proxy.
We already had a server with signalr hub where clients connect using signalr. We recently moved it to two nodes with a load balancer for scalability. But after moving it to load balancer now clients are unable to connect to the server using signalr. Clients first try using web sockets and it gives the following error on signalr trace.
fae26beb-a806-4957-b52a-39b80856e492 - Auto: Failed to connect to using transport webSockets. System.Net.WebSockets.WebSocketException (0x80004005): Unable to connect to the remote server ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
at System.Net.TlsStream.EndRead(IAsyncResult asyncResult)
at System.Net.PooledStream.EndRead(IAsyncResult asyncResult)
at System.Net.Connection.ReadCallback(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.WebSockets.ClientWebSocket.<ConnectAsyncCore>d__21.MoveNext()
at System.Net.WebSockets.ClientWebSocket.<ConnectAsyncCore>d__21.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNet.SignalR.Client.Transports.WebSocketTransport.<PerformConnect>d__1.MoveNext()
But after couple of times trying to connect it manages to connect using SSE (Server Sent Events).
fae26beb-a806-4957-b52a-39b80856e492 - SSE: OnMessage(Data: {"C":"s-0,BA9F","S":1,"M":[]})
fae26beb-a806-4957-b52a-39b80856e492 - ChangeState(Connecting, Connected)
fae26beb-a806-4957-b52a-39b80856e492 - SSE: OnMessage(Data: {"C":"s-0,BAA0","M":[{"H":"PrintConnectorHub","M":"SignalRConnectedSuccessfully","A":[true]}]})
fae26beb-a806-4957-b52a-39b80856e492 - OnMessage({"R":false,"I":"3"})
But when the server tries to send a message back to the client, client does not receive the message. We have implemented SignalR scale out with SQL Server and following are the log traces in the server.
TRACE SignalR.SqlMessageBus Stream 0 : SqlReceiver last payload ID=47781, new payload ID=47782 (Microsoft.AspNet.SignalR.SqlServer.SqlReceiver.ProcessRecord)
TRACE SignalR.SqlMessageBus Stream 0 : Updated receive reader initial payload ID parameter=47782 (Microsoft.AspNet.SignalR.SqlServer.SqlReceiver.ProcessRecord)
TRACE SignalR.SqlMessageBus Stream 0 : Payload 47782 containing 1 message(s) received (Microsoft.AspNet.SignalR.SqlServer.SqlReceiver.ProcessRecord)
TRACE SignalR.SqlMessageBus Stream 0 : 1 records received (Microsoft.AspNet.SignalR.SqlServer.ObservableDbOperation.ExecuteReaderWithUpdates)
TRACE SignalR.SqlMessageBus Created DbCommand: CommandType=Text, CommandText=SELECT [PayloadId], [Payload], [InsertedOn] FROM [SignalR].[Messages_0] WHERE [PayloadId] > #PayloadId, Parameters= [Name=PayloadId, Value=47782] (Microsoft.AspNet.SignalR.SqlServer.DbOperation.TraceCommand)
Signalr initialization in clients
signalrHostUrl = "SignalR_Host";
try
{
hubConnection = new HubConnection(signalrHostUrl);
hubConnection.Error += ex =>
{
signalRlogger.Error(ex, "Exception from hubConnection");
};
hubConnection.ConnectionSlow += () => ConnectionSlow();
hubConnection.Closed += () => SignalrConnectionConnectionClosed();
hubConnection.StateChanged += signalRConnectionStateController.StateChanged;
hubConnection.Start().Wait();
}
catch (Exception ex)
{
signalRlogger.Error(ex, "initialzing hub connection");
}
Before moving to the load balancer the clients were perfectly able to connect via web sockets and communicate. But after moving to load balancer it completely fails to connect via web sockets. We even tried to open a web socket connection by this tool (https://chrome.google.com/webstore/detail/simple-websocket-client/pfdhoblngboilpfeibdedpjgfnlcodoo?hl=en) which also fails to open a connection. Although it manages to connect via SSE it still unable to communicate to clients. What are the complications when using web sockets in load balancing and in this situation why signalr fails to make a connection to the clients?
When the other machine tries to decrypt the connection token you got when negotiating the connection it can't and the request fails. You need to ensure that all machines you are using have the same machine key. This way all the machines will be able to decrypt connectionTokens created by other machines.
Are you able to enable stickyness on the load balancers? For example session based stickyness. This will make sure the clients connect to the same machine. The downside is that if a machine fails the client won't be able to reconnect.