First, for some context: I am using .NetCore to develop an API with Identity. Everything is on a Cloud server, inside a Docker. When a user is created, an email is sent to the new User using a mailkit and the webmail server through Plesk (Hosted on the same machine). The docker is accessed via a redirection trough Apache using a ProxyPass from the subdomain to the port on localhost
Everything works great while debugging trough JetBrain's Rider, but it is not able to process the email in the docker on the server.
Here is the stack:
System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000001, 11): Resource temporarily unavailable
at System.Net.Dns.InternalGetHostByName(String hostName)
at System.Net.Dns.ResolveCallback(Object context)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult)
at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult)
at System.Net.Dns.<>c.<GetHostAddressesAsync>b__25_1(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
I have yet to try and run the docker on another linux machine to test.
One of my current guess would be a problem with the SSL certificate, but I don't think it would cause a problem with the DNS or any internal socket.
Another guess is thats its a problem for the Docker to get the DNS Hostname, but since it works ok in a local.
Edit: I tried multiple time to run the docker on the mac and the error is still triggered once in a while but not always. It is although always triggered on the server and never send the email
After two weeks of research, I finally stumbled upon a solution for this:
The problem is related to the network, that was obvious, but it's precisely about how containers are isolated from one another. Problem is, the container has no outbound connection. A solution that work inside a standalone container is to use the --network host parameter, which would expose the host network to the container. Note that using this would remove the port mapping from the container since the container's port 5000 is now linked to the host's port 5000
Hope this solution can help others
Related
I have an dotnet 5 service that sends emails using the SmtpClient. This service is deployed as a linux docker image running in AWS EKS (Kubernetes).
Running locally on my machine, everything works fine, but when it's deployed to our K8s environment, every SmtpClient.SentAsync call sits for around 10 minutes, before it finally crashes with the following exception:
System.Net.Mail.SmtpException: Failure sending mail.
---> System.Net.Sockets.SocketException (99): Cannot assign requested address
I've researched this error, and there is a lot of talk about using/not using localhost as a host name. In my case, I'm not using local host at all. I'm just trying to send outbound emails using SmtpClient, where the host is a 3rd party SMTP server.
I've tried running this as a linux docker container on my own dev machine, and it also works fine, so it seems to not like something about my EKS environment, yet I can't figure out what that is. Any ideas?
Update:
FYI, the pod can access the internet just fine, using the HttpClient for example. I can also make http calls to it's API on port 80. So it's binding just fine in that regard. It just seems to be an issue when using the SmtpClient.
Stack Track:
System.Net.Mail.SmtpException: Failure sending mail.
---> System.Net.Sockets.SocketException (99): Cannot assign requested address
at System.Net.Sockets.Socket.BeginConnectEx(EndPoint remoteEP, Boolean flowContext, AsyncCallback callback, Object state)
at System.Net.Sockets.Socket.UnsafeBeginConnect(EndPoint remoteEP, AsyncCallback callback, Object state, Boolean flowContext)
at System.Net.Sockets.Socket.PostOneBeginConnect(MultipleAddressConnectAsyncResult context)
--- End of stack trace from previous location ---
at System.Net.Sockets.Socket.DoMultipleAddressConnectCallback(Object result, MultipleAddressConnectAsyncResult context)
at System.Net.Sockets.Socket.MultipleAddressConnectCallback(IAsyncResult result)
--- End of stack trace from previous location ---
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.Sockets.TcpClient.EndConnect(IAsyncResult asyncResult)
at System.Net.Mail.SmtpConnection.ConnectAndHandshakeAsyncResult.InitializeConnectionCallback(IAsyncResult result)
--- End of stack trace from previous location ---
at System.Net.Mail.SmtpConnection.ConnectAndHandshakeAsyncResult.End(IAsyncResult result)
at System.Net.Mail.SmtpClient.ConnectCallback(IAsyncResult result)
...
as expected of a beginner IoT Edge developer, I started following the Microsoft tutorial on how to deploy custom code to an IoT Edge server Tutorial: Develop a C# IoT Edge module for Linux devices.
I was able to install Ubuntu 18.04 server on an industrial intel PC, download and configure the edgeHub and edgeAgent modules, create a free Azure and Docker Hub account, etc. Even the example code from the tutorial was succesfully built and deployed to my target device and all seemed well, all modules were talking to each other, etc.
Once I got the demo code to run, I started modifying the code to better suit my end goal, which is to capture network traffic generated by our industrial equipment, which uses UDP multicast/broadcast. Apparently I needed to configure my docker image to run in 'host' networking mode. And indeed, as soon as I told the azure edgeAgent to start the module container in host mode, the UDP packets started to come in.
HOWEVER, now my sample module is no longer able to connect to the iot hub and I'm at a complete loss. I've tried to run the edgeHub container in host mode as well, this doesn't seem to make any difference. The exact error I'm getting as a result of the ModuleClient.OpenAsync method is:
Unhandled exception. System.AggregateException: One or more errors occurred. (Transient network error occurred, please retry.)
---> Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: Transient network error occurred, please retry.
---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.InternalGetHostByName(String hostName)
at System.Net.Dns.ResolveCallback(Object context)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult)
at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult)
at System.Net.Dns.<>c.<GetHostAddressesAsync>b__25_1(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.OpenAsyncInternal(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.OpenAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass23_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
--- End of inner exception stack trace ---
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass33_0.<<OpenInternalAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()
at SampleModule.Program.Init() in /app/SampleModule/Program.cs:line 54
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait()
at SampleModule.Program.Main(String[] args) in /app/SampleModule/Program.cs:line 26
The code used for connecting:
ITransportSettings[] settings = { mqttSetting };
ModuleClient ioTHubModuleClient = await ModuleClient.CreateFromEnvironmentAsync(settings);
await ioTHubModuleClient.OpenAsync();
Console.WriteLine("IoT Hub module client initialized.");
So apparently the ModuleClient can't find the edgeHub anymore (Name or service not known?). I understood that the 'CreateFromEnvironmentAsync' uses environment variables set by the edgeAgent, but which ones? Anyone have any idea why this is? Even pointers in the right direction to start debugging this issue are greatly appreciated!
If I change my docker containers back to 'bridge' networking, the OpenAsync method works perfectly, but my UDP broadcast messages are no longer received, of course.
After some more digging around, I found the solution myself.
Module fails to restart because of transient network error
This article, although I had read it already multiple times, suggested that something was wrong with the /etc/hosts file. Sure enough, the entry for my device (127.0.1.1) was faulty, somewhere along the way I must have changed the hostname of my device and this change wasn't reflected in the hosts file.
Since that change, I got both AMQP and MQTT to work, however both with the edgeHub container also running in host mode. In bridge mode I ran across a new issue where the docker-proxy didn't bind the exported ports (which resulted in a new 'Connection refused' exception). It remains a mystery why...
I've set up an instance of Identity Server 3 and it's running locally using a self-signed SSL certificate which I've added to my trusted certificates store.
Separately I have an example MVC project which is using the OWIN Katana middleware to authenticate against this local IS3 instance. This all works well.
I've now moved the two applications to a testing environment outside my local machine, replicating the setup as close as possible. Both are running under SSL using a self-signed certificate which I've also installed to my local trusted certificate store. However, clicking sign-in from my test app throws this exception:
The remote certificate is invalid according to the validation
procedure.
Here is the stack trace:
[AuthenticationException: The remote certificate is invalid according
to the validation procedure.]
System.Net.TlsStream.EndWrite(IAsyncResult asyncResult) +298
System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar) +150
[WebException: The underlying connection was closed: Could not
establish trust relationship for the SSL/TLS secure channel.]
System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
+764 System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
+78
[HttpRequestException: An error occurred while sending the request.]
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) +14139120
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) +62
Microsoft.IdentityModel.Protocols.d__0.MoveNext()
+366
[IOException: Unable to get document from:
https://[OBFUSCATED]/core/.well-known/openid-configuration]
Microsoft.IdentityModel.Protocols.d__0.MoveNext()
+736 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) +14139120
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) +62
Microsoft.IdentityModel.Protocols.d__0.MoveNext() +290
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) +14139120
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) +62
Microsoft.IdentityModel.Protocols.d__3.MoveNext()
+917
It's pretty obvious my SSL setup is not sufficient to meet the validation, but I'm not clear what I need to change. Is it simply the case that I cannot test on a remote deployment environment without a full SSL certificate? Or am I missing something. If you need to know more about my setup let me know what would help. As I say, this same applications run fine locally so I assume this is more about server config. Thanks.
Update 1: more detail about testing environment
My testing environment is a pair of load balanced web servers. My Identity Server application and my example client application are both hosted on these servers under different URLs. Both use the same self-signed SSL certificate. This SSL certificate has been added to the Trusted Root Certs and the Trusted People stores on each web server. I've therefore followed all the advice on other examples (e.g. here). Is there something about this setup (e.g. both sites using the same certificate) that will cause the validation procedure to fail?
Solved
I'm not 100% sure what the problem was here but I think it must have been a problem with how I generated my self-signed certs. I decided to wipe out the existing SSL certs, then used the following Powershell script on each web server to create a new self-signed on each then copied the new certificates. Using these certificates for my bindings worked fine. Make sure the $AddionalHostNames variable matches your IS3 application's hostname.
$AddionalHostNames="my.is3.host.name"
New-SelfSignedCertificate -DnsName $env:computername, $AddionalHostNames -CertStoreLocation cert:\LocalMachine\My
We deliver on-premise software that is exposed to the cloud using Azure Service bus relay, the basic code we use to expose is as follows (I have removed everything identifiable):
ServiceHost sh = new ServiceHost(typeof(BasicHttpEntityService));
BasicHttpRelayBinding basicHttpRelayBinding = new BasicHttpRelayBinding();
Uri uriEndPointAddress = ServiceBusEnvironment.CreateServiceUri("https", "ourdomain", "test-url-appendage");
m_shRelayServiceHost.AddServiceEndpoint(
typeof(IMyService),
basicHttpRelayBinding,
uriEndPointAddress
).Behaviors.Add(
new TransportClientEndpointBehavior
{
TokenProvider = TokenProvider.CreateSharedSecretTokenProvider(
"MyUser",
"MyPassword")
});
sh.Open();
This works fine at most of our customers, however, one of our customers has a strict firewall policy.
According to the SB guidelines we have found, we asked them to open ports 9351-9354 to ourdomain.servicebus.windows.net. Now we find out that when there is an incoming request, the service connects to both 'ourdomain' (we see this succeeds in Wireshark, and also in the WCF log) AND an unknown (to us) service on 40.112.124.x:9352 (the last octet changes with every request).
I have been able to reproduce the problem in my development environment by disallowing connections to any 40.x.x.x address on any port. This is what happens in the WCF log:
System.Net.Sockets.SocketException (0x80004005): An attempt was made to access a socket in a way forbidden by its access permissions 40.112.124.25:9352
Server stack trace:
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at Microsoft.ServiceBus.RelayedConnectionSession.ConnectAsyncResult.<GetAsyncSteps>b__4(ConnectAsyncResult thisRef, IAsyncResult r)
at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
Exception rethrown at [0]:
at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.ServiceBus.RelayedConnectionSession.EndConnect(IAsyncResult result)
There is no DNS-request going out during this time, so there is no host name that provides any clues to the function of this outgoing connection.
From my investigation, this appears to be a Microsoft controlled subnet, so I'm fine with the relay service connecting to it, but I would like to know:
Is this additional connection optional?
If not, should we allow the entire subnet?
Could this IP-range change in the future? Is it hardcoded somewhere?
In the end, we requested support from Microsoft.
In short their answers were as follows:
Is this additional connection optional?
No it is not optional. For the relay listener, there is a control channel on port 5671, this connection is always there. Then there is a data channel on portal 9352, this connection established when there is a relay client tries to communicate with the listener.
Could this IP-range change in the future?
Currently, for relay this IP can change, so you need to allow the IP range for the entire datacenter in your region (https://www.microsoft.com/en-us/download/confirmation.aspx?id=41653). The SB product team will be working on to significantly reduce this IP range in the future, to make it much more predictable. There is no exact ETA on this future.
So the good news is they are working on it. The bad news is, that right now, we will need to add a LOT of IP addresses to the white-list to ensure smooth operation.
I have developed a Quartz.Net windows service to run scheduled jobs and I have set up an SQLite jobstore for it to work. Here are the settings of my AdoJobStore:
# SQLite settings
quartz.jobStore.type = Quartz.Impl.AdoJobStore.JobStoreTX, Quartz
quartz.jobStore.misfireThreshold = 60000
quartz.jobStore.lockHandler.type = Quartz.Impl.AdoJobStore.UpdateLockRowSemaphore, Quartz
quartz.jobStore.useProperties = true
quartz.jobStore.dataSource = default
quartz.jobStore.tablePrefix = qrtz_
quartz.jobStore.driverDelegateType = Quartz.Impl.AdoJobStore.SQLiteDelegate, Quartz
quartz.dataSource.default.provider = SQLite-10
quartz.dataSource.default.connectionString = Data Source=.\jobs.db;Version=3
Before giving you a headache, I should say that this is a working solution! That means the job store works just fine! Each job consists of running an .exe file that will get its parameters from an SQL Server database using a connection string that is stored in a text file.
So if the service is started automatically, everything works fine.
After installing the Windows service, we give it a username that has access to the destination database in service's Log On tab. We set the service to start automatically (and have tried the Automatic Delayed mode as well). In the Recovery tab of the service we also command it to Restart in case it crashes.
However, the service does not seems to be starting by itself on some machines and that has kept me wondering for quite a while now. On machines with failure, this is what my service logs when it fails to start:
2015-02-13 15:09:15,674 [1] ERROR Quartz.Server.QuartzServer [(null)] - Server initialization failed:Unable to bind scheduler to remoting.
Quartz.SchedulerException: Unable to bind scheduler to remoting. ---> System.Net.Sockets.SocketException: No such host is known
at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6)
at System.Net.Dns.GetHostEntry(String hostNameOrAddress)
at System.Runtime.Remoting.Channels.CoreChannel.GetMachineIp()
at System.Runtime.Remoting.Channels.Tcp.TcpServerChannel.SetupMachineName()
at System.Runtime.Remoting.Channels.Tcp.TcpServerChannel..ctor(IDictionary properties, IServerChannelSinkProvider sinkProvider, IAuthorizeRemotingConnection authorizeCallback)
at System.Runtime.Remoting.Channels.Tcp.TcpChannel..ctor(IDictionary properties, IClientChannelSinkProvider clientSinkProvider, IServerChannelSinkProvider serverSinkProvider)
at Quartz.Simpl.RemotingSchedulerExporter.RegisterRemotingChannelIfNeeded()
at Quartz.Simpl.RemotingSchedulerExporter.Bind(IRemotableQuartzScheduler scheduler)
at Quartz.Core.QuartzScheduler.Bind()
at Quartz.Core.QuartzScheduler.Initialize()
--- End of inner exception stack trace ---
at Quartz.Core.QuartzScheduler.Initialize()
at Quartz.Impl.StdSchedulerFactory.Instantiate()
at Quartz.Impl.StdSchedulerFactory.GetScheduler()
at Quartz.Server.QuartzServer.Initialize() [See nested exception: System.Net.Sockets.SocketException (0x80004005): No such host is known
at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6)
at System.Net.Dns.GetHostEntry(String hostNameOrAddress)
at System.Runtime.Remoting.Channels.CoreChannel.GetMachineIp()
at System.Runtime.Remoting.Channels.Tcp.TcpServerChannel.SetupMachineName()
at System.Runtime.Remoting.Channels.Tcp.TcpServerChannel..ctor(IDictionary properties, IServerChannelSinkProvider sinkProvider, IAuthorizeRemotingConnection authorizeCallback)
at System.Runtime.Remoting.Channels.Tcp.TcpChannel..ctor(IDictionary properties, IClientChannelSinkProvider clientSinkProvider, IServerChannelSinkProvider serverSinkProvider)
at Quartz.Simpl.RemotingSchedulerExporter.RegisterRemotingChannelIfNeeded()
at Quartz.Simpl.RemotingSchedulerExporter.Bind(IRemotableQuartzScheduler scheduler)
at Quartz.Core.QuartzScheduler.Bind()
at Quartz.Core.QuartzScheduler.Initialize()]
The service is installed on host 127.0.0.1 port 555 and does not seem to need a dependency because it works under some other machines.
I have a feeling that the service is trying to start itself before there is an access to the database or before the TCP host is established. But if that's the case, why does it work on other devices?
I have asked the same question before but have failed to find a working answer. Any help from Quartz.Net dev team or Windows Services experts is greatly appreciated.
Disclaimer. I'm not part of the Quartz.net dev team, nor am I an expert on Windows services
The service is installed on host 127.0.0.1 port 555 and does not seem to need a dependency because it works under some other machines.
Your reasoning is false. It could be that the required services just happen to have always started on the other machines, but without setting a dependency this is not guaranteed.
So, since you are getting a DNS error, I would suggest you start by setting your service do that it depends on Dnscache and Tcpip .