as expected of a beginner IoT Edge developer, I started following the Microsoft tutorial on how to deploy custom code to an IoT Edge server Tutorial: Develop a C# IoT Edge module for Linux devices.
I was able to install Ubuntu 18.04 server on an industrial intel PC, download and configure the edgeHub and edgeAgent modules, create a free Azure and Docker Hub account, etc. Even the example code from the tutorial was succesfully built and deployed to my target device and all seemed well, all modules were talking to each other, etc.
Once I got the demo code to run, I started modifying the code to better suit my end goal, which is to capture network traffic generated by our industrial equipment, which uses UDP multicast/broadcast. Apparently I needed to configure my docker image to run in 'host' networking mode. And indeed, as soon as I told the azure edgeAgent to start the module container in host mode, the UDP packets started to come in.
HOWEVER, now my sample module is no longer able to connect to the iot hub and I'm at a complete loss. I've tried to run the edgeHub container in host mode as well, this doesn't seem to make any difference. The exact error I'm getting as a result of the ModuleClient.OpenAsync method is:
Unhandled exception. System.AggregateException: One or more errors occurred. (Transient network error occurred, please retry.)
---> Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: Transient network error occurred, please retry.
---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.InternalGetHostByName(String hostName)
at System.Net.Dns.ResolveCallback(Object context)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult)
at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult)
at System.Net.Dns.<>c.<GetHostAddressesAsync>b__25_1(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.OpenAsyncInternal(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.OpenAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass23_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
--- End of inner exception stack trace ---
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass33_0.<<OpenInternalAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()
at SampleModule.Program.Init() in /app/SampleModule/Program.cs:line 54
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait()
at SampleModule.Program.Main(String[] args) in /app/SampleModule/Program.cs:line 26
The code used for connecting:
ITransportSettings[] settings = { mqttSetting };
ModuleClient ioTHubModuleClient = await ModuleClient.CreateFromEnvironmentAsync(settings);
await ioTHubModuleClient.OpenAsync();
Console.WriteLine("IoT Hub module client initialized.");
So apparently the ModuleClient can't find the edgeHub anymore (Name or service not known?). I understood that the 'CreateFromEnvironmentAsync' uses environment variables set by the edgeAgent, but which ones? Anyone have any idea why this is? Even pointers in the right direction to start debugging this issue are greatly appreciated!
If I change my docker containers back to 'bridge' networking, the OpenAsync method works perfectly, but my UDP broadcast messages are no longer received, of course.
After some more digging around, I found the solution myself.
Module fails to restart because of transient network error
This article, although I had read it already multiple times, suggested that something was wrong with the /etc/hosts file. Sure enough, the entry for my device (127.0.1.1) was faulty, somewhere along the way I must have changed the hostname of my device and this change wasn't reflected in the hosts file.
Since that change, I got both AMQP and MQTT to work, however both with the edgeHub container also running in host mode. In bridge mode I ran across a new issue where the docker-proxy didn't bind the exported ports (which resulted in a new 'Connection refused' exception). It remains a mystery why...
Related
I have an dotnet 5 service that sends emails using the SmtpClient. This service is deployed as a linux docker image running in AWS EKS (Kubernetes).
Running locally on my machine, everything works fine, but when it's deployed to our K8s environment, every SmtpClient.SentAsync call sits for around 10 minutes, before it finally crashes with the following exception:
System.Net.Mail.SmtpException: Failure sending mail.
---> System.Net.Sockets.SocketException (99): Cannot assign requested address
I've researched this error, and there is a lot of talk about using/not using localhost as a host name. In my case, I'm not using local host at all. I'm just trying to send outbound emails using SmtpClient, where the host is a 3rd party SMTP server.
I've tried running this as a linux docker container on my own dev machine, and it also works fine, so it seems to not like something about my EKS environment, yet I can't figure out what that is. Any ideas?
Update:
FYI, the pod can access the internet just fine, using the HttpClient for example. I can also make http calls to it's API on port 80. So it's binding just fine in that regard. It just seems to be an issue when using the SmtpClient.
Stack Track:
System.Net.Mail.SmtpException: Failure sending mail.
---> System.Net.Sockets.SocketException (99): Cannot assign requested address
at System.Net.Sockets.Socket.BeginConnectEx(EndPoint remoteEP, Boolean flowContext, AsyncCallback callback, Object state)
at System.Net.Sockets.Socket.UnsafeBeginConnect(EndPoint remoteEP, AsyncCallback callback, Object state, Boolean flowContext)
at System.Net.Sockets.Socket.PostOneBeginConnect(MultipleAddressConnectAsyncResult context)
--- End of stack trace from previous location ---
at System.Net.Sockets.Socket.DoMultipleAddressConnectCallback(Object result, MultipleAddressConnectAsyncResult context)
at System.Net.Sockets.Socket.MultipleAddressConnectCallback(IAsyncResult result)
--- End of stack trace from previous location ---
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.Sockets.TcpClient.EndConnect(IAsyncResult asyncResult)
at System.Net.Mail.SmtpConnection.ConnectAndHandshakeAsyncResult.InitializeConnectionCallback(IAsyncResult result)
--- End of stack trace from previous location ---
at System.Net.Mail.SmtpConnection.ConnectAndHandshakeAsyncResult.End(IAsyncResult result)
at System.Net.Mail.SmtpClient.ConnectCallback(IAsyncResult result)
...
First, for some context: I am using .NetCore to develop an API with Identity. Everything is on a Cloud server, inside a Docker. When a user is created, an email is sent to the new User using a mailkit and the webmail server through Plesk (Hosted on the same machine). The docker is accessed via a redirection trough Apache using a ProxyPass from the subdomain to the port on localhost
Everything works great while debugging trough JetBrain's Rider, but it is not able to process the email in the docker on the server.
Here is the stack:
System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000001, 11): Resource temporarily unavailable
at System.Net.Dns.InternalGetHostByName(String hostName)
at System.Net.Dns.ResolveCallback(Object context)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult)
at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult)
at System.Net.Dns.<>c.<GetHostAddressesAsync>b__25_1(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
I have yet to try and run the docker on another linux machine to test.
One of my current guess would be a problem with the SSL certificate, but I don't think it would cause a problem with the DNS or any internal socket.
Another guess is thats its a problem for the Docker to get the DNS Hostname, but since it works ok in a local.
Edit: I tried multiple time to run the docker on the mac and the error is still triggered once in a while but not always. It is although always triggered on the server and never send the email
After two weeks of research, I finally stumbled upon a solution for this:
The problem is related to the network, that was obvious, but it's precisely about how containers are isolated from one another. Problem is, the container has no outbound connection. A solution that work inside a standalone container is to use the --network host parameter, which would expose the host network to the container. Note that using this would remove the port mapping from the container since the container's port 5000 is now linked to the host's port 5000
Hope this solution can help others
I have a Stateless Service Fabric project (.NET Core) that I need to kick off a Docker job from. I'm using Docker.DotNet and the following code works well in a small Console App, however will not work in Service Fabric:
var dockerClient = new DockerClientConfiguration(new Uri("npipe://./pipe/docker_engine")).CreateClient();
// error occurs on next line (in Service Fabric)...
dockerClient.Images
.CreateImageAsync(new ImagesCreateParameters
{
FromImage = "jbarlow83/ocrmypdf",
Tag = "latest"
},
new AuthConfig(),
new Progress<JSONMessage>());
I see this error in Service Fabric Explorer if I try to run it from the Stateless SF project:
System.UnauthorizedAccessException: Access to the path is denied.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.Pipes.NamedPipeClientStream.ConnectInternal(Int32 timeout, CancellationToken cancellationToken, Int32 startTime)
at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Docker.DotNet.DockerClient.<>c__DisplayClass6_0.<<-ctor>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Net.Http.Client.ManagedHandler.d__33.MoveNext()
I'm not sure if this is actually a permission issue, or if it's related to how Service Fabric network isolation works.
I'm trying this on my local development instance, and this will eventually (hopefully) go to an on-premise setup.
Is there a way to access named pipes on the node that is hosting a SF application? Or perhaps another suggested way to run Docker through a .NET Core SF application?
Just came across this when I ran into this issue today, and ended up stumbling onto a different solution if you don't want to switch away from the Named Pipe connection. If you set up your Service Fabric service to run as a LocalSystem Account instead of the default account, it should work as well.
You can follow the instructions here to change the Account your Stateless Service runs under in the Manifest
This works because according to this article on Named Pipes, only certain types of accounts get access to the Named Pipe, one of which is LocalSystem.
I was able to get this working by changing my DockerClientConfiguration line to no longer use named pipes, and instead use http://localhost:2375:
var dockerClient = new DockerClientConfiguration(new Uri("http://localhost:2375")).CreateClient();
And then enabled Expose daemon on tcp://localhost:2375 without TLS in the Docker CE General Settings:
HttpClient throw a random exception ( maybe 2-3 times out of 10 attempts ) with Azure Container Instance
Exception Info: System.Net.WebException at
System.Net.HttpWebRequest.EndGetResponse(System.IAsyncResult) at
System.Net.Http.HttpClientHandler.GetResponseCallback(System.IAsyncResult)
Exception Info: System.Net.Http.HttpRequestException at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at
System.Net.Http.HttpClient+\u003cFinishSendAsync\u003ed__58.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSucem.Net.Http.HttpClient+\u003cFinishSendAsync\u003ed__58.MoveNext()
at
SystetHelpers.MSIResourcesAccessInfoHelper+\u003cGetResourcesAccessInfo\u003ed__0.MoveNext()\n.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\n
at System.Runtime.m[[System.__Canon, mscorlib, Version=4.0.0.0,
Culture=neutral,
PublicKeyToken=b77a5c56193ompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.ThreC.Main(System.String[])
On the code side, I am simply firing
await httpClient.GetAsync($"http://xyz.eastus.cloudapp.azure.com/api/controller/{id}");
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsAsync<ResourceAccessInfo>();
The behavior is very random it happens only 2-3 time out of 10 attempts
I have also applied re-try when the first attempt then try the second time but still it fails
Same works fine with VM containers always.
Updated:
Additional Exception Info:
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---\u003e System.Net.WebException: The remote name could not be resolved: \u0027cloudbridge1.eastus.cloudapp.azure.com\u0027\r\n at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)\r\n at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)\r\n --- End of inner exception stack trace ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Net.Http.HttpClient.\u003cFinishSendAsync\u003ed__58.MoveNext()\r\n--- End ofstack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at UcClearly.MSIAPIHelpers.MSIResourcesAccessInfoHelper.\u003cGetResourcesAccessInfo\u003ed__0.MoveNext()
Which suggest remote server DNS not resolved, I tried with public IP of the server also but same random behavior, I am not sure why Azure Container instance behave this random way
This behaviour is expected.
Windows containers slow network readiness
On initial creation, Windows containers may have no inbound or outbound connectivity for up to 30 seconds (or longer, in rare cases). If your container application needs an Internet connection, add delay and retry logic to allow 30 seconds to establish Internet connectivity. After initial setup, container networking should resume appropriately.
https://learn.microsoft.com/en-us/azure/container-instances/container-instances-troubleshooting#windows-containers-slow-network-readiness
I currently try to migrate from VMs to Docker Windows Containers. Containers are hosten on a Windows Server 2016 Datacenter. The OS is installed on a physical root computer hosted by Hetzner. My Docker Containers contain a console app, written in c#, performaning updates on a MariaDB database, hosted in google-cloud.
Before every console app ran in a seperated VM with its own ipv4. Now they run in a container and getting the following exception
Unhandled Exception: System.TypeInitializationException: The type initializer for 'A.C' threw an exception. ---> MySql.Data.MySqlClient.MySqlException: Authentication to host '##.##.##.##' for user '####' using
method 'mysql_native_password' failed with message: Reading from the stream has failed. ---> MySql.Data.MySqlClient.MySqlException: Reading from the stream has failed. ---> System.IO.EndOfStreamException:
Attempted to read past the end of the stream.
at MySql.Data.MySqlClient.MySqlStream.ReadFully(Stream stream, Byte[] buffer, Int32 offset, Int32 count)
at MySql.Data.MySqlClient.MySqlStream.LoadPacket()
--- End of inner exception stack trace ---
at MySql.Data.MySqlClient.MySqlStream.LoadPacket()
at MySql.Data.MySqlClient.MySqlStream.ReadPacket()
at MySql.Data.MySqlClient.Authentication.MySqlAuthenticationPlugin.ReadPacket()
--- End of inner exception stack trace ---
at MySql.Data.MySqlClient.Authentication.MySqlAuthenticationPlugin.AuthenticationFailed(Exception ex)
at MySql.Data.MySqlClient.Authentication.MySqlAuthenticationPlugin.ReadPacket()
at MySql.Data.MySqlClient.Authentication.MySqlAuthenticationPlugin.Authenticate(Boolean reset)
at MySql.Data.MySqlClient.NativeDriver.Authenticate(String authMethod, Boolean reset)
at MySql.Data.MySqlClient.NativeDriver.Open()
at MySql.Data.MySqlClient.Driver.Open()
at MySql.Data.MySqlClient.Driver.Create(MySqlConnectionStringBuilder settings)
at MySql.Data.MySqlClient.MySqlPool.CreateNewPooledConnection()
at MySql.Data.MySqlClient.MySqlPool.GetPooledConnection()
at MySql.Data.MySqlClient.MySqlPool.TryToGetDriver()
at MySql.Data.MySqlClient.MySqlPool.GetConnection()
at MySql.Data.MySqlClient.MySqlConnection.Open()
at ServiceStack.OrmLite.OrmLiteConnection.Open()
at ServiceStack.OrmLite.OrmLiteConnectionFactory.OpenDbConnection()
at A.C.b()
at A.C..cctor()
--- End of inner exception stack trace ---
at A.C.A()
at A.b..ctor()
at A.a.A()
at A.d.A(String[] )
at A.d.a[A](String[] )
at A.A.A(String[] )
Docker uses a NAT network. Hence I think there is a problem with the connection regarding to multiple connections from the same IP.
Have anyone of you had similiar experience or knows how to fix this? Im a little bit stucked and glad for any help. Thanks in advance.
If you have any more questions regarding my setup or anything else please let me know.
Regards
Michael
This appears to be a common problem in MySql.Data; see the comment from Rui Fan at the end of bug 76597 for a potential workaround.