Develop Reference

DotNetCore SocketException "Cannot assign requested address" SmtpClient

I have an dotnet 5 service that sends emails using the SmtpClient. This service is deployed as a linux docker image running in AWS EKS (Kubernetes).
Running locally on my machine, everything works fine, but when it's deployed to our K8s environment, every SmtpClient.SentAsync call sits for around 10 minutes, before it finally crashes with the following exception:
System.Net.Mail.SmtpException: Failure sending mail.
---> System.Net.Sockets.SocketException (99): Cannot assign requested address
I've researched this error, and there is a lot of talk about using/not using localhost as a host name. In my case, I'm not using local host at all. I'm just trying to send outbound emails using SmtpClient, where the host is a 3rd party SMTP server.
I've tried running this as a linux docker container on my own dev machine, and it also works fine, so it seems to not like something about my EKS environment, yet I can't figure out what that is. Any ideas?
Update:
FYI, the pod can access the internet just fine, using the HttpClient for example. I can also make http calls to it's API on port 80. So it's binding just fine in that regard. It just seems to be an issue when using the SmtpClient.
Stack Track:
System.Net.Mail.SmtpException: Failure sending mail.
---> System.Net.Sockets.SocketException (99): Cannot assign requested address
at System.Net.Sockets.Socket.BeginConnectEx(EndPoint remoteEP, Boolean flowContext, AsyncCallback callback, Object state)
at System.Net.Sockets.Socket.UnsafeBeginConnect(EndPoint remoteEP, AsyncCallback callback, Object state, Boolean flowContext)
at System.Net.Sockets.Socket.PostOneBeginConnect(MultipleAddressConnectAsyncResult context)
--- End of stack trace from previous location ---
at System.Net.Sockets.Socket.DoMultipleAddressConnectCallback(Object result, MultipleAddressConnectAsyncResult context)
at System.Net.Sockets.Socket.MultipleAddressConnectCallback(IAsyncResult result)
--- End of stack trace from previous location ---
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.Sockets.TcpClient.EndConnect(IAsyncResult asyncResult)
at System.Net.Mail.SmtpConnection.ConnectAndHandshakeAsyncResult.InitializeConnectionCallback(IAsyncResult result)
--- End of stack trace from previous location ---
at System.Net.Mail.SmtpConnection.ConnectAndHandshakeAsyncResult.End(IAsyncResult result)
at System.Net.Mail.SmtpClient.ConnectCallback(IAsyncResult result)
...

Azure IoT Edge ClientModule can't connect in host mode

as expected of a beginner IoT Edge developer, I started following the Microsoft tutorial on how to deploy custom code to an IoT Edge server Tutorial: Develop a C# IoT Edge module for Linux devices.
I was able to install Ubuntu 18.04 server on an industrial intel PC, download and configure the edgeHub and edgeAgent modules, create a free Azure and Docker Hub account, etc. Even the example code from the tutorial was succesfully built and deployed to my target device and all seemed well, all modules were talking to each other, etc.
Once I got the demo code to run, I started modifying the code to better suit my end goal, which is to capture network traffic generated by our industrial equipment, which uses UDP multicast/broadcast. Apparently I needed to configure my docker image to run in 'host' networking mode. And indeed, as soon as I told the azure edgeAgent to start the module container in host mode, the UDP packets started to come in.
HOWEVER, now my sample module is no longer able to connect to the iot hub and I'm at a complete loss. I've tried to run the edgeHub container in host mode as well, this doesn't seem to make any difference. The exact error I'm getting as a result of the ModuleClient.OpenAsync method is:
Unhandled exception. System.AggregateException: One or more errors occurred. (Transient network error occurred, please retry.)
---> Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: Transient network error occurred, please retry.
---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.InternalGetHostByName(String hostName)
at System.Net.Dns.ResolveCallback(Object context)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult)
at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult)
at System.Net.Dns.<>c.<GetHostAddressesAsync>b__25_1(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.OpenAsyncInternal(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.OpenAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass23_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
--- End of inner exception stack trace ---
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass33_0.<<OpenInternalAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()
at SampleModule.Program.Init() in /app/SampleModule/Program.cs:line 54
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait()
at SampleModule.Program.Main(String[] args) in /app/SampleModule/Program.cs:line 26
The code used for connecting:
ITransportSettings[] settings = { mqttSetting };
ModuleClient ioTHubModuleClient = await ModuleClient.CreateFromEnvironmentAsync(settings);
await ioTHubModuleClient.OpenAsync();
Console.WriteLine("IoT Hub module client initialized.");
So apparently the ModuleClient can't find the edgeHub anymore (Name or service not known?). I understood that the 'CreateFromEnvironmentAsync' uses environment variables set by the edgeAgent, but which ones? Anyone have any idea why this is? Even pointers in the right direction to start debugging this issue are greatly appreciated!
If I change my docker containers back to 'bridge' networking, the OpenAsync method works perfectly, but my UDP broadcast messages are no longer received, of course.

After some more digging around, I found the solution myself.
Module fails to restart because of transient network error
This article, although I had read it already multiple times, suggested that something was wrong with the /etc/hosts file. Sure enough, the entry for my device (127.0.1.1) was faulty, somewhere along the way I must have changed the hostname of my device and this change wasn't reflected in the hosts file.
Since that change, I got both AMQP and MQTT to work, however both with the edgeHub container also running in host mode. In bridge mode I ran across a new issue where the docker-proxy didn't bind the exported ports (which resulted in a new 'Connection refused' exception). It remains a mystery why...

ASP.NET Core w/ EF Core API returning "Failed to authenticate HTTPS connection." From Postman Get request

As an exercise in learning EF Core with C#/ASP.NET Core, I went through this tutorial to the end: ASP.NET Core Web API with EF Core Code-First Approach.
Once I finished I was able to go and run 'dotnet watch run' from the source folder, and loading up the end point http://localhost:5000/api/employee I was able to get back a response in Chrome, from a redirect to the https://localhost:5001/api/employee endpoint.
I'm stuck because I want to get that same response in Postman, but I'm just seeing the following:
This is what I see from Powershell where I'm running the server:
PS C:\Users\foo\source\repos\EFCoreCodeFirstSample\EFCoreCodeFirstSample> dotnet watch run
watch : Started
info: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[0]
User profile is available. Using 'C:\Users\C M\AppData\Local\ASP.NET\DataProtection-Keys' as key repository and Windows DPAPI to encrypt keys at rest.
Hosting environment: Development
Content root path: C:\Users\foo\source\repos\EFCoreCodeFirstSample\EFCoreCodeFirstSample
Now listening on: http://localhost:5000
Now listening on: https://localhost:5001
Application started. Press Ctrl+C to shut down.
dbug: HttpsConnectionAdapter[1]
Failed to authenticate HTTPS connection.
System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Security.SslState.ThrowIfExceptional()
at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
at System.Net.Security.SslStream.EndAuthenticateAsServer(IAsyncResult asyncResult)
at System.Net.Security.SslStream.<>c.<AuthenticateAsServerAsync>b__51_1(IAsyncResult iar)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionAdapter.InnerOnConnectionAsync(ConnectionAdapterContext context)
Everything from
dbug: HttpsConnectionAdapter[1]
Failed to authenticate HTTPS connection
and below shows up when I run the Postman Get request. I'm guessing it has something to do with how my startup.cs is configured... but I'm not sure where to start. Searching on Stack Overflow here got me a lot of questions regarding SSL certs, and pfx files, but nothing pertaining to this issue in particular. I'm new to this and in school, so maybe I simply missed the answer because it's beyond my scope of knowledge at this point? I don't understand why it would work in chrome, but not Postman.

SignlaR not working with load balancing

We already had a server with signalr hub where clients connect using signalr. We recently moved it to two nodes with a load balancer for scalability. But after moving it to load balancer now clients are unable to connect to the server using signalr. Clients first try using web sockets and it gives the following error on signalr trace.
fae26beb-a806-4957-b52a-39b80856e492 - Auto: Failed to connect to using transport webSockets. System.Net.WebSockets.WebSocketException (0x80004005): Unable to connect to the remote server ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
at System.Net.TlsStream.EndRead(IAsyncResult asyncResult)
at System.Net.PooledStream.EndRead(IAsyncResult asyncResult)
at System.Net.Connection.ReadCallback(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.WebSockets.ClientWebSocket.<ConnectAsyncCore>d__21.MoveNext()
at System.Net.WebSockets.ClientWebSocket.<ConnectAsyncCore>d__21.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNet.SignalR.Client.Transports.WebSocketTransport.<PerformConnect>d__1.MoveNext()
But after couple of times trying to connect it manages to connect using SSE (Server Sent Events).
fae26beb-a806-4957-b52a-39b80856e492 - SSE: OnMessage(Data: {"C":"s-0,BA9F","S":1,"M":[]})
fae26beb-a806-4957-b52a-39b80856e492 - ChangeState(Connecting, Connected)
fae26beb-a806-4957-b52a-39b80856e492 - SSE: OnMessage(Data: {"C":"s-0,BAA0","M":[{"H":"PrintConnectorHub","M":"SignalRConnectedSuccessfully","A":[true]}]})
fae26beb-a806-4957-b52a-39b80856e492 - OnMessage({"R":false,"I":"3"})
But when the server tries to send a message back to the client, client does not receive the message. We have implemented SignalR scale out with SQL Server and following are the log traces in the server.
TRACE SignalR.SqlMessageBus Stream 0 : SqlReceiver last payload ID=47781, new payload ID=47782 (Microsoft.AspNet.SignalR.SqlServer.SqlReceiver.ProcessRecord)
TRACE SignalR.SqlMessageBus Stream 0 : Updated receive reader initial payload ID parameter=47782 (Microsoft.AspNet.SignalR.SqlServer.SqlReceiver.ProcessRecord)
TRACE SignalR.SqlMessageBus Stream 0 : Payload 47782 containing 1 message(s) received (Microsoft.AspNet.SignalR.SqlServer.SqlReceiver.ProcessRecord)
TRACE SignalR.SqlMessageBus Stream 0 : 1 records received (Microsoft.AspNet.SignalR.SqlServer.ObservableDbOperation.ExecuteReaderWithUpdates)
TRACE SignalR.SqlMessageBus Created DbCommand: CommandType=Text, CommandText=SELECT [PayloadId], [Payload], [InsertedOn] FROM [SignalR].[Messages_0] WHERE [PayloadId] > #PayloadId, Parameters= [Name=PayloadId, Value=47782] (Microsoft.AspNet.SignalR.SqlServer.DbOperation.TraceCommand)
Signalr initialization in clients
signalrHostUrl = "SignalR_Host";
try
{
hubConnection = new HubConnection(signalrHostUrl);
hubConnection.Error += ex =>
{
signalRlogger.Error(ex, "Exception from hubConnection");
};
hubConnection.ConnectionSlow += () => ConnectionSlow();
hubConnection.Closed += () => SignalrConnectionConnectionClosed();
hubConnection.StateChanged += signalRConnectionStateController.StateChanged;
hubConnection.Start().Wait();
}
catch (Exception ex)
{
signalRlogger.Error(ex, "initialzing hub connection");
}
Before moving to the load balancer the clients were perfectly able to connect via web sockets and communicate. But after moving to load balancer it completely fails to connect via web sockets. We even tried to open a web socket connection by this tool (https://chrome.google.com/webstore/detail/simple-websocket-client/pfdhoblngboilpfeibdedpjgfnlcodoo?hl=en) which also fails to open a connection. Although it manages to connect via SSE it still unable to communicate to clients. What are the complications when using web sockets in load balancing and in this situation why signalr fails to make a connection to the clients?

When the other machine tries to decrypt the connection token you got when negotiating the connection it can't and the request fails. You need to ensure that all machines you are using have the same machine key. This way all the machines will be able to decrypt connectionTokens created by other machines.
Are you able to enable stickyness on the load balancers? For example session based stickyness. This will make sure the clients connect to the same machine. The downside is that if a machine fails the client won't be able to reconnect.

WebListenerException:The specified network name is no longer available

There is an odd exception in logs of my MVC 6 (beta 7) project:
01:29:55.8657 Error Flush
System.IO.IOException ---> Microsoft.Net.Http.Server.WebListenerException: The specified network name is no longer available
--- End of inner exception stack trace ---
01:29:55.8901 Error ProcessRequestAsync
System.IO.IOException ---> Microsoft.Net.Http.Server.WebListenerException: The specified network name is no longer available
--- End of inner exception stack trace ---
at Microsoft.Net.Http.Server.ResponseStream.FlushInternal(Boolean endOfRequest)
at Microsoft.Net.Http.Server.ResponseStream.Dispose(Boolean disposing)
at System.IO.Stream.Close()
at Microsoft.Net.Http.Server.Response.Dispose()
at Microsoft.Net.Http.Server.RequestContext.Dispose()
at Microsoft.AspNet.Server.WebListener.MessagePump.<ProcessRequestAsync>d__23.MoveNext()
I have found such errors were outside the MVC 6 (the fist link, the second link)
But in my case a source of the exception is the Microsoft.Net.Http.Server.ResponseStream.FlushInternal method.
The error does not interrupt the application. It occurs after some queries (I did not notice a pattern), and is rarely.
Why this error occurs periodically in my log?
Can I do take any actions to avoid the error?
Thank you!

That sounds like the client disconnected before the response was sent. That can be difficult to detect without writing to the socket. We have a (private) work item to de-emphasize these error messages in the logs.