I have an issue with my app communicating with a website, but only when the bindings are set to use a Specific IP (i.e., 172.16.9.86). If I leave the binding as '*' (All Unassigned) the app works correctly.
The app is making the call to the website using HttpClient (.NET Core 3.1) like so:
try
{
System.Net.ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
responseBody = await client.GetStringAsync("https://myServer.myDomain.com/SecureKeyTest/rdTemplate/rdGetSecureKey.aspx?Username=admin");
Debug.WriteLine(responseBody);
}
catch (HttpRequestException e)
{
Debug.WriteLine("Message :{0} ", e.Message);
}
I can take that same URL and paste it into Chrome and it works to retrieve the SecureKey, even when the binding is still set to a specific IP address, so I don't understand how the HttpClient call is different from the call from Chrome.
I have scoured hundreds of posts I have found on this issue and almost all claim that adding the SecurityProtocol settings is supposed to fix this issue, but it hasn't. The certificate used for this website is a fully registered company certificate, not a self-generated one.
EDIT: forgot to add the exception message I get:
The SSL connection could not be established, see inner exception.System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
--- End of inner exception stack trace
--- at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request) at System.Net.Security.SslStream.ThrowIfExceptional() at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult) at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result) at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult) at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_1(IAsyncResult iar) at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown --- at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
Simply because you don't have a corresponding IP based certificate mapping for that IP address in Windows HTTP API,
https://docs.jexusmanager.com/tutorials/https-binding.html#ip-based-bindings
Related
I followed this tutorial link and installed Rabbit MQ on Oracle Cloud server. I am able to use Management Plugin in browser. But when I try to push a message via dotnet I am getting this error. Firewall is allowed for ports 5671,5672 and 15672 also in oracle ingress rules."
"RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the
specified endpoints were reachable\r\n ---> System.AggregateException:
One or more errors occurred. (Connection failed)\r\n --->
RabbitMQ.Client.Exceptions.ConnectFailureException: Connection
failed\r\n ---> System.Net.Sockets.SocketException (11001): No such
host is known.\r\n at
System.Net.NameResolutionPal.ProcessResult(SocketError errorCode,
GetAddrInfoExContext* context)\r\n at
System.Net.NameResolutionPal.GetAddressInfoExCallback(Int32 error,
Int32 bytes, NativeOverlapped* overlapped)\r\n--- End of stack trace
from previous location ---\r\n at
RabbitMQ.Client.Impl.TcpClientAdapter.ConnectAsync(String host, Int32
port)\r\n at RabbitMQ.Client.Impl.TaskExtensions.TimeoutAfter(Task
task, TimeSpan timeout)\r\n at
RabbitMQ.Client.Impl.SocketFrameHandler.ConnectOrFail(ITcpClient
socket, AmqpTcpEndpoint endpoint, TimeSpan timeout)\r\n --- End of
inner exception stack trace ---\r\n at
RabbitMQ.Client.Impl.SocketFrameHandler.ConnectOrFail(ITcpClient
socket, AmqpTcpEndpoint endpoint, TimeSpan timeout)\r\n at
RabbitMQ.Client.Impl.SocketFrameHandler.ConnectUsingAddressFamily(AmqpTcpEndpoint
endpoint, Func2 socketFactory, TimeSpan timeout, AddressFamily family)\r\n at RabbitMQ.Client.Impl.SocketFrameHandler..ctor(AmqpTcpEndpoint endpoint, Func2 socketFactory, TimeSpan connectionTimeout, TimeSpan
readTimeout, TimeSpan writeTimeout)\r\n at
RabbitMQ.Client.ConnectionFactory.CreateFrameHandler(AmqpTcpEndpoint
endpoint)\r\n at
RabbitMQ.Client.EndpointResolverExtensions.SelectOne[T](IEndpointResolver
resolver, Func2 selector)\r\n --- End of inner exception stack trace ---\r\n at RabbitMQ.Client.EndpointResolverExtensions.SelectOne[T](IEndpointResolver resolver, Func2 selector)\r\n at
RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.Init(IEndpointResolver
endpoints)\r\n at
RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver
endpointResolver, String clientProvidedName)\r\n --- End of inner
exception stack trace ---\r\n at
RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver
endpointResolver, String clientProvidedName)\r\n at "
when trying to run rabbitmq-diagnostics listeners I am getting this error in logs
** Connection attempt from node '' rejected. Invalid challenge reply. **
Can anyone assist me on this issue?
After adding the below registry key to enable TLS1.3 on Windows Server 2022
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.3\Client]
"DisabledByDefault"=dword:00000000
"Enabled"=dword:00000001
The .NET Client application throws the below exception
System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
---> System.ComponentModel.Win32Exception (0x80090308): The token supplied to the function is invalid
--- End of inner exception stack trace ---
at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
at System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions)
at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)
Is there anything i am missing? In the .NET Client application i had set the protocol version to TLS13.
Thanks in advance.
Try the below before you call the server if server is also on windows 2022+
Otherwise it will not work if server is not 2022+
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls13
I realize that this a rather service specific subject, but I think that the reason for my problem is a general one, since I'm a novice in the gRPC arena.
I'm attempting to call a simple method PeerVersion in the Concordium blockchain gRPC API (https://github.com/Concordium/concordium-grpc-api) from a .NET Core 3.1 app, but I get what seems to be a rather general error in regard to SSL.
I can add that I have testet this call and others with BloomRPC (https://github.com/uw-labs/bloomrpc) and it works just fine.
You wont be able to call the API successfully without access to a Concordium node, but I recon that someone with better insight into gRPC than me, could maybe see what I'm doing wrong.
C# code
static async Task Main(string[] args)
{
using var channel = GrpcChannel.ForAddress("https://192.168.1.18:10001"); // my local node running on the Concordium testnet
var client = new Concordium.P2P.P2PClient(channel);
var metadata = new Grpc.Core.Metadata() // required token for the API
{
{ "authentication", "rpcadmin" }
};
var request = new Concordium.Empty();
var reply = client.PeerVersion(request, metadata);
Console.WriteLine("peer version: " + reply.Value);
Console.WriteLine("press any key to exit...");
Console.ReadKey();
}
When I run this I get the following exception. (Danish text reads "An existing connection was forcibly disconnected by an external host" or something like that).
- $exception {"Status(StatusCode=\"Unavailable\", Detail=\"Error starting gRPC call. HttpRequestException: The SSL connection could not be established, see inner exception. IOException: Unable to read data from the transport connection: En eksisterende forbindelse blev tvangsafbrudt af en ekstern vært.. SocketException: En eksisterende forbindelse blev tvangsafbrudt af en ekstern vært.\", DebugException=\"System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.IO.IOException: Unable to read data from the transport connection: En eksisterende forbindelse blev tvangsafbrudt af en ekstern vært..
---> System.Net.Sockets.SocketException (10054): En eksisterende forbindelse blev tvangsafbrudt af en ekstern vært.
--- End of inner exception stack trace ---
at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
at System.Net.Security.SslStream.ThrowIfExceptional()
at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_1(IAsyncResult iar)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)\")"} Grpc.Core.RpcException
Is anyone able to give me some pointers as to what could be the problem here?
For anyone that knows BloobRPC, here is a screenshot of request and response.
Ah, I got the answer to my problem elsewhere. It turns out that my assumption that the Concordium node requires a secure connection was wrong, in fact it doesn't support secure connections, so the URL was the wrong part.
So to fix the problem, I needed to create the client like this:
// to allow non secure connections
AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true);
using var channel = GrpcChannel.ForAddress("http://192.168.1.18:10001");
Environment: Windows Server 2012 R2 64-bit.
C# .NET Framework version 4.5.1.
I am trying to use this program to download a bunch of files from a SharePoint 2013 site: https://github.com/nddipiazza/Sharepoint-Exporter
In this project there is a [FileDownloader.cs][1] file that pulls requests to download files from a BlockingCollection and downloads them to file.
When I run this I pretty quickly get hit with socket errors:
System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted 10.5.50.2:443
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at SpPrefetchIndexBuilder.FileDownloader.attemptToDownload(FileToDownload toDownload, Int32 numRetry)
---> (Inner Exception #0) System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted 10.5.50.2:443
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace ---<---
Basically it seems like I am hitting this code too hard:
public void AttemptToDownload(FileToDownload toDownload, int numRetry)
{
try
{
var responseResult = client.GetAsync(SpPrefetchIndexBuilder.topParentSite + toDownload.serverRelativeUrl);
if (responseResult.Result != null && responseResult.Result.StatusCode == System.Net.HttpStatusCode.OK)
{
using (var memStream = responseResult.Result.Content.ReadAsStreamAsync().GetAwaiter().GetResult())
{
using (var fileStream = File.Create(toDownload.saveToPath))
{
memStream.CopyTo(fileStream);
}
}
Console.WriteLine("Thread {0} - Successfully downloaded {1} to {2}", Thread.CurrentThread.ManagedThreadId, toDownload.serverRelativeUrl, toDownload.saveToPath);
}
else
{
Console.WriteLine("Got non-OK status {0} when trying to download url {1}", responseResult.Result.StatusCode, SpPrefetchIndexBuilder.topParentSite + toDownload.serverRelativeUrl);
}
}
catch (Exception e)
{
if (numRetry >= NUM_RETRIES)
{
Console.WriteLine("Gave up trying to download url {0} to file {1} after {2} retries due to error: {3}", SpPrefetchIndexBuilder.topParentSite + toDownload.serverRelativeUrl, toDownload.saveToPath, NUM_RETRIES, e);
}
else
{
AttemptToDownload(toDownload, numRetry + 1);
}
}
}
Is there something wrong with how I'm using HttpClient? I read all the forums and it says to re-use a static reference, and I am doing that. All of my threads share the same static HttpClient reference.
Here is a netstat -a -o -n from when the downloads have been running for a few minutes. There are a lot of TIMED_WAIT sitting there. https://pastebin.com/GTYmqwue In this test i was using SharePoint on port 80. Why is that?
When I run it again a couple minutes later, the number of TIMED_WAIT have increased hundreds more. There must be a leak of some sort going on?
Why is HttpClient leaving 1000's of TIMED_WAIT connections sitting there? How can I get them to close?
I tried to set ServiePointManager.DefaultConnectionLimit = numThreads but it still grows to 1000's of connections.
I think I figured it out finally
I accidentally had a really aggressive client.Timeout = TimeSpan.FromMinutes(5);
I changed this to client.Timeout = TimeSpan.FromSeconds(15);
Now I think the connections aren't leaking anymore.
UPDATE 4 Even More confused
I thought I was onto something, but the same error keeps popping up. This solution seemed promising, but isn't 100%, but thought it might help someone narrow down what's wrong.
Steps:
Restart API and Identity Server IIS Application Pools
Get Access Token Via SoapUI using QA URL
Hit API in QA with SoapUI
Failure - Doesn't work (See below for this stacktrace)
Hit API locally with SoapUI (Using same QA Identity Server URL)
Local API URL returns expected data
Hit API using QA URL using SoapUI (I'm just changing the endpoint in SoapUI)
QA API URL returns expected data
I was able to repeat that 4 times before I would still get the Error when hitting QA. After restarting this process again, I can now get my QA environment to work without having to use the API locally. Any ideas what's going on? Seems to be a setup issue/proxy/certificate issue, but no idea how to debug that.
This is the error that I'm seeing now:
Server Error in '/' Application.
An existing connection was forcibly closed by the remote host
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Stack Trace:
[SocketException (0x2746): An existing connection was forcibly closed by the remote host]
System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult) +8156963
System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult) +48
[IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.]
System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult) +8111720
System.Net.TlsStream.EndRead(IAsyncResult asyncResult) +275
System.Net.Connection.ReadCallback(IAsyncResult asyncResult) +45
[WebException: The underlying connection was closed: An unexpected error occurred on a receive.]
System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) +764
System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar) +78
[HttpRequestException: An error occurred while sending the request.]
[AggregateException: One or more errors occurred.]
System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) +4451240
Microsoft.IdentityModel.Protocols.<GetDocumentAsync>d__0.MoveNext() in c:\workspace\WilsonForDotNet45Release\src\Microsoft.IdentityModel.Protocol.Extensions\Configuration\HttpDocumentRetriever.cs:53
[IOException: Unable to get document from: https://securityeliqa.twcable.com/core/.well-known/openid-configuration]
Microsoft.IdentityModel.Protocols.<GetDocumentAsync>d__0.MoveNext() in c:\workspace\WilsonForDotNet45Release\src\Microsoft.IdentityModel.Protocol.Extensions\Configuration\HttpDocumentRetriever.cs:59
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13847892
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
Microsoft.IdentityModel.Protocols.<GetAsync>d__0.MoveNext() in c:\workspace\WilsonForDotNet45Release\src\Microsoft.IdentityModel.Protocol.Extensions\Configuration\OpenIdConnectConfigurationRetriever.cs:81
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13847892
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
Microsoft.IdentityModel.Protocols.<GetConfigurationAsync>d__3.MoveNext() in c:\workspace\WilsonForDotNet45Release\src\Microsoft.IdentityModel.Protocol.Extensions\Configuration\ConfigurationManager.cs:0
[InvalidOperationException: IDX10803: Unable to create to obtain configuration from: 'https://securityeliqa.twcable.com/core/.well-known/openid-configuration'.]
Microsoft.IdentityModel.Protocols.<GetConfigurationAsync>d__3.MoveNext() in c:\workspace\WilsonForDotNet45Release\src\Microsoft.IdentityModel.Protocol.Extensions\Configuration\ConfigurationManager.cs:212
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13847892
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
Microsoft.IdentityModel.Protocols.<GetConfigurationAsync>d__0.MoveNext() in c:\workspace\WilsonForDotNet45Release\src\Microsoft.IdentityModel.Protocol.Extensions\Configuration\ConfigurationManager.cs:0
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13847892
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() +31
IdentityServer.WebApi.AccessTokenValidation.<<RetrieveMetadata>b__0>d__4.MoveNext() in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\Plumbing\DiscoveryDocumentIssuerSecurityTokenProvider.cs:123
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13847892
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() +31
IdentityServer.WebApi.AccessTokenValidation.AsyncHelper.RunSync(Func`1 func) in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\Plumbing\AsyncHelper.cs:18
IdentityServer.WebApi.AccessTokenValidation.DiscoveryDocumentIssuerSecurityTokenProvider.RetrieveMetadata() in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\Plumbing\DiscoveryDocumentIssuerSecurityTokenProvider.cs:141
IdentityServer.WebApi.AccessTokenValidation.DiscoveryDocumentIssuerSecurityTokenProvider..ctor(String discoveryEndpoint, IdentityServerBearerTokenAuthenticationOptions options, ILoggerFactory loggerFactory) in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\Plumbing\DiscoveryDocumentIssuerSecurityTokenProvider.cs:43
Owin.IdentityServerBearerTokenValidationAppBuilderExtensions.ConfigureLocalValidation(IdentityServerBearerTokenAuthenticationOptions options, ILoggerFactory loggerFactory) in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\IdentityServerBearerTokenValidationAppBuilderExtensions.cs:129
Owin.IdentityServerBearerTokenValidationAppBuilderExtensions.UseIdentityServerBearerTokenAuthentication(IAppBuilder app, IdentityServerBearerTokenAuthenticationOptions options) in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\IdentityServerBearerTokenValidationAppBuilderExtensions.cs:39
Company.WebApi.waAddressQualification.Startup.Configuration(IAppBuilder app) in e:\Source Code\GitHub\waDemo\Startup.cs:23
[TargetInvocationException: Exception has been thrown by the target of an invocation.]
System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor) +0
System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments) +128
System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) +146
Owin.Loader.<>c__DisplayClass12.<MakeDelegate>b__b(IAppBuilder builder) +93
Owin.Loader.<>c__DisplayClass1.<LoadImplementation>b__0(IAppBuilder builder) +209
Microsoft.Owin.Host.SystemWeb.OwinAppContext.Initialize(Action`1 startup) +843
Microsoft.Owin.Host.SystemWeb.OwinBuilder.Build(Action`1 startup) +51
Microsoft.Owin.Host.SystemWeb.OwinHttpModule.InitializeBlueprint() +101
System.Threading.LazyInitializer.EnsureInitializedCore(T& target, Boolean& initialized, Object& syncLock, Func`1 valueFactory) +137
Microsoft.Owin.Host.SystemWeb.OwinHttpModule.Init(HttpApplication context) +172
System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers) +618
System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context) +172
System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context) +402
System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext) +343
[HttpException (0x80004005): Exception has been thrown by the target of an invocation.]
System.Web.HttpRuntime.FirstRequestInit(HttpContext context) +579
System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context) +112
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +712
Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.6.1055.0
UPDATE 3 Even More confused
At this point I believe it's the server that is cancelling my requests, but even with that I can't be 100% sure. Here are some additional data points:
Service is setup on both a Dev and QA box, Dev box gets the error about 1 out of 100 requests, QA, about 50% of the time.
The QA box has Identity Server setup as https behind a load balancer, Dev is http
The service setup mimics Identity Server where in QA it is https behind a load balancer, Dev is http
A set of three address in QA will usually have at least one address fail, I see the request in the server log for the two that work, shouldn't I see the third? If the third request is not in the server logs does that mean it never got there?
What really gets me is I can't seem to see an error anywhere but on the client, where the HttpClient.GetAsync call is made. If the server is cancelling the request, shouldn't there be a log for each one it cancels?
I've found in the Event log on on the windows server hosting the service errors about cancelling a task, however not all failed requests end up here?
The operation was canceled. at System.Threading.CancellationToken.ThrowOperationCanceledException() at System.Web.Http.Owin.HttpMessageHandlerAdapter.d__13.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Owin.HttpMessageHandlerAdapter.d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult() at IdentityServer.WebApi.AccessTokenValidation.ScopeRequirementMiddleware.d__0.MoveNext() in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\Plumbing\ScopeRequirementMiddleware.cs:line 48 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1.d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult() at IdentityServer.WebApi.AccessTokenValidation.IdentityServerBearerTokenValidationMiddleware.d__6.MoveNext() in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\IdentityServerBearerTokenValidationMiddleware.cs:line 81 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage.d__5.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContext.d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.StageAsyncResult.End(IAsyncResult ar) at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
Original Questions
I'd like to know if I'm call my Demo API correctly, or if the API is causing me an issue. If I send three requests through at the same time I get the issue on at least one of the addresses I'm trying to process.
Update Number one
In the Inner Exception I'm getting the following message, I was able to capture a more complete error as follows: "The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host"
I'm using .NET 4.6.1 for both client and server side
I'm not using a proxy.
I'm able to successfully call the API about 1/3 of the time
I've experimented with the couple of different options in calling the service, the first was just wrapping the DemoApiClient in a lock, the next was to use Task Run with the async and await keywords and finally retrieving the token for authentication using Task Run or just calling it and having ConfigureAwait(false) listed on the await lines. I'm not sure if any of this matters, or if the server is just failing when called. The server isn't that complex, but I'd rather not post that code here. It does validate the token and then process an address, makes a couple of database calls and returns data found from the database.
UPDATE 2
I test with three addresses at a time, when I send them through SoapUI to the server all three are process fine, but with the .Net C# client using the code below only one of the three requests seem to make it to the server. I was expecting to see two stacktraces and one successful request, but the IIS log only shows one request coming in.
The code I'm using to call the API is here:
using Demo.IdentityServer.IdentityModel.Client;
using Newtonsoft.Json;
using PartnerPortal.Model;
using System;
using System.Configuration;
using System.Diagnostics;
using System.Net.Http;
using System.Linq;
using System.Collections.Generic;
using System.Threading.Tasks;
namespace DemoPortal.Client
{
public static class DemoApiClient
{
public readonly static string waDemoUrl = ConfigurationManager.AppSettings["waDemoURL"];
public static async Task<DemoData> GetDemoData(RequestAddress requestAddress)
{
try
{
using (var httpClient = new HttpClient() { MaxResponseContentBufferSize = 10000000, Timeout = TimeSpan.FromMilliseconds(5000) })
{
string reasonPhrase = "";
var demoUri = new Uri(string.Format(waDemoUrl + "api/DemoApp/GetSomeData?trackingId={0}&clientKey={1}&address={2}", requestAddress.TrackingId, requestAddress.ClientKey, requestAddress.Address));
var httpResult = new HttpResponseMessage();
var result = RequestAccessToken.RequestToken().Result;
var accessToken = await Task.Run(() => RequestAccessToken.RequestToken().Result);
//var accessToken = RequestAccessToken.RequestToken().Result;
//which of the above two options should I use? Does it matter?
httpClient.SetBearerToken(accessToken);
HttpResponseMessage response = await httpClient.GetAsync(demoUri);
HttpContent httpContent = response.Content;
if (httpResult.IsSuccessStatusCode)
{
var content = await httpContent.ReadAsStringAsync();
DemoData demoData = null;
demoData = JsonConvert.DeserializeObject<DemoData>(content);
return demoData;
}
else
{
reasonPhrase = httpResult.ReasonPhrase;
if (reasonPhrase.ToUpper() == "UNAUTHORIZED")
{
throw new KeyNotFoundException("Not authorized");
}
}
}
}
catch (Exception ex)
{
Debug.WriteLine("Message is:" + ex);
throw (ex);
}
return null;
}
}
}
And this is how the token is retrieved:
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.Configuration;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using System.Web;
using Thinktecture.IdentityModel.Tokens.Http;
namespace Demo.IdentityServer.IdentityModel.Client
{
public static class RequestAccessToken
{
public readonly static string securityUrl = ConfigurationManager.AppSettings["securityUrl"];
public readonly static string clientSecret = ConfigurationManager.AppSettings["clientSecret"];
public static async Task<string> RequestToken()
{
var url = new Uri(securityUrl);
var fields = new Dictionary<string, string>
{
{ OAuth2Constants.GrantType, OAuth2Constants.GrantTypes.ClientCredentials },
{ OAuth2Constants.Scope, "Read"}
};
using (var httpClient = new HttpClient())
{
var cancellationToken = new CancellationToken();
httpClient.DefaultRequestHeaders.Authorization = new BasicAuthenticationHeaderValue("Demo", clientSecret);
var ss = JsonConvert.SerializeObject(fields);
var data = new FormUrlEncodedContent(fields);
var s = data.ReadAsStringAsync();
//var response = await httpClient.PostAsync(url, data, cancellationToken).ConfigureAwait(false);
var response = await httpClient.PostAsync(url, data, cancellationToken);
// Should I use ConfigureAwait(false) or call method with a Task(Run() ... ?
if (response.StatusCode == HttpStatusCode.OK || response.StatusCode == HttpStatusCode.BadRequest)
{
//var content = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
var content = await response.Content.ReadAsStringAsync();
var token = new TokenResponse(content);
var accessToken = token.AccessToken;
return accessToken;
}
else
{
return new TokenResponse(response.StatusCode, response.ReasonPhrase).AccessToken;
}
}
}
}
}
SOLUTION Found!!! The entire issue stemmed from a misconfigured Load Balancer. Something with the SNAT pool. I'm not the network guy, and whatever he did fixed it. Something like the load balancer was referencing the box name directly instead of the SNAT Pool name, or the SNAT Pool had the wrong name. Either way once updated the intermittent successes became always successes.
I can tell you that you are flirting with SNAT port exhaustion at scale. It makes sense to instantiate HttpClient within a using block since it implements IDisposable. However, doing so will lead to HttpClient monopolizing one SNAT port for 240 seconds after it is disposed. See here for a great explanation of this problem.