Develop Reference

Query health check endpoints in ASP.NET

I implemented a health check endpoint following this doc. My ASP.NET application is dockerized and runs using docker-compose, with the port mapped/exposed.
Question: I am not sure how to query the health check endpoint from clients such as postman.
When I send a GET request to the /healthz endpoint as the following, postman throws the following error.
http://host.docker.internal:1200/healthz
Error: Client network socket disconnected before secure TLS connection was established
while I can see the following in the logs of the docker container.
[05:01:03 DBG] Connection id "0HMLSHSV1HRD2" accepted.
[05:01:03 DBG] Connection id "0HMLSHSV1HRD2" started.
[05:01:03 INF] Request starting HTTP/1.1 GET http://host.docker.internal:1200/healthz - -
[05:01:03 DBG] Wildcard detected, all requests with hosts will be allowed.
[05:01:03 VRB] All hosts are allowed.
[05:01:03 DBG] 1 candidate(s) found for the request path '/healthz'
[05:01:03 DBG] Request matched endpoint 'Health checks'
[05:01:03 DBG] Static files was skipped as the request already matched an endpoint.
[05:01:03 DBG] Https port '1200' loaded from configuration.
[05:01:03 DBG] Redirecting to 'https://host.docker.internal:1200/healthz'.
[05:01:03 DBG] Connection id "0HMLSHSV1HRD2" completed keep alive response.
[05:01:03 INF] Request finished HTTP/1.1 GET http://host.docker.internal:1200/healthz - - - 307 0 - 89.5220ms

GRPC on Google Cloud Run : upstream connect error or disconnect/reset before headers. reset reason: remote reset

EDIT
It seems that my first error I describe is very easy to reproduce. Actually, Google Run fails to run any GRPC query on a .NET5 GRPC server it seems (at least, it did work before but as of today, February 21st, it seems that something changed). To reproduce:
Create a .NET5 GRPC server (also fails with .NET6):
dotnet new grpc -o TestGrpc
Change Program.cs so that it listens on $PORT, typically:
public static IHostBuilder CreateHostBuilder(string[] args)
{
var port = Environment.GetEnvironmentVariable("PORT") ?? "8080";
var url = string.Concat("http://0.0.0.0:", port);
return Host.CreateDefaultBuilder(args)
.ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.UseStartup<Startup>().UseUrls(url);
});
}
A very simple Dockerfile to have an image for the server (fails also with a more standard one, like here):
FROM mcr.microsoft.com/dotnet/sdk:5.0
COPY . ./
RUN dotnet restore ./TestGrpc.csproj
RUN dotnet build ./TestGrpc.csproj -c Release
CMD dotnet run --project ./TestGrpc.csproj
Build and push to Google Artifcats Registry.
Create a Cloud Run instance with HTTP/2 enabled (Ketrel requires HTTP/2 so we need to set
HTTP/2 end-to-end, yet I tested without as well but it's not better).
Use Grpcurl for instance and try:
grpcurl {CLOUD_RUN_URL}:443 list
And you will obtain the same error as I got with my (more complex) project:
Failed to list services: rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: remote reset
On the Google Cloud Run instance I only have the log:
2022-02-21T16:44:32.528530Z POST 200 1.02 KB 41 ms grpcurl/v1.8.6 grpc-go/1.44.1-dev https://***/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo
(I don't really understand why it's a 200 though... and never seems to reach the actual server implementation, just as if there was some kind of middleware blocking the query to reach the implementation... )
I'm pretty sure this used to worked as I started my project this way (and then changed the protos, the service, etc. ). If anyone has a clue I'd be more than grateful :-)
INITIAL POST (less precise than explanations above but I leave it here if it may give clues)
I have a server running within a Docker (.NET5 GRPC application). This server, when deployed locally works perfectly fine. But recently I have an error when I deploy it on Google Cloud Run: upstream connect error or disconnect/reset before headers. reset reason: remote reset when it was working fine before. I keep on having this error from any client I use, for instance with Curl:
curl -v https://{ENDPOINT}/{Proto-base}/{Method} --http2
* Trying ***...
* TCP_NODELAY set
* Connected to *** (***) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*.a.run.app
* start date: Feb 7 02:07:06 2022 GMT
* expire date: May 2 02:07:05 2022 GMT
* subjectAltName: host "***" matched cert's "*.a.run.app"
* issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1C3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5564aad30860)
> GET /{Proto}/{Method} HTTP/2
> Host: ***
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 503
< content-length: 85
< content-type: text/plain
< date: Mon, 21 Feb 2022 13:51:31 GMT
< server: Google Frontend
< traceparent: 00-5a74487dafb5687961deeb17e0158ca9-5ab63cd23680e7d7-01
< x-cloud-trace-context: 5a74487dafb5687961deeb17e0158ca9/6536478782730069975;o=1
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
<
* Connection #0 to host *** left intact
upstream connect error or disconnect/reset before headers. reset reason: remote reset
Same happens with Grpcurl:
grpcurl ***:443 list {Proto-base}
Failed to list methods for service "***.Company": rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: remote reset
I cannot find much resource on this error as most of the threads I read deal with another type of reset reason (like protocol, or connection, etc.). But I strictly have no idea what remote reset means and what I did wrong.
Looking at the logs in Google Cloud Run, I can see that the server is definitly hit, though I added trace logging in the route which is not triggered, hence it never reaches my code:
2022-02-21T14:44:22.840580Z POST 200 1.01 KB 1 msgrpc-python/1.44.0 grpc-c/22.0.0 (linux; chttp2) https://***/{Protos-base}/{Method}
(if I reached my code it should print some "Hellos" everywhere which it doesn't)
Has anyone ever found this?
P.S.: there are many things around about Envoy, but I don't even use this. I simply have a Cloud Run instance (with HTTP/2 - and I tried without but it fails due to protocol issue).

It is an actual bug from Envoy and Google Cloud Run. There is a quick fix if you're using .NET6, otherwise it's a bit more hacky. I will just copy here the answer provided by Amanda Tarafa Mas from Google Cloud Platform on the github issue I opened:
Here are the potential fixes:
When using .NET 6 you can set KestrelServerOptions.AllowAlternateSchemes to true.
If on a lower .NET version, consider something like GRPC :scheme pseudo-header passed from proxy/loadbalancer causes ConnectionAbortedException dotnet/aspnetcore#30532 (comment). Or consider upgrading to .NET 6.
What's happening:
Cloud Run has dependency on Envoy, which has a behavior change since 04/15/2021, see "preserve_downstream_scheme" in release notes:
https://www.envoyproxy.io/docs/envoy/latest/version_history/v1.18.0
Envoy recently removed the old behaviour: https://www.envoyproxy.io/docs/envoy/latest/version_history/current#removed-config-or-runtime
In turn, this exposes this .NET issue: GRPC :scheme pseudo-header passed from proxy/loadbalancer causes ConnectionAbortedException dotnet/aspnetcore#30532, for which the Kestrel configuration flag was added, but only for .NET 6.
I'm looking into having this documented somewhere. #meteatamel can you update the tutorial so that it uses the Kestrel option?
For me setting KestrelServerOptions.AllowAlternate was enough to make my GRPC server work again.
As #Craig said, you can track the issue here and see if it gets resolved.

Enabling Web Requests go via reverse proxy in F#

Code is in F# but also tagging C# in case for any suggestions. It's an SSL based request for which it couldn't resolve the proxy name, I've tried both http://localhost and https://localhost besides 127.0.0.1
Code:
let request = WebRequest.Create("https://foo.example.com") :?> HttpWebRequest
let myproxy = WebProxy("http://127.0.0.1", 60103);
myproxy.BypassProxyOnLocal <- false;
request.Proxy <- myproxy;
Error:
System.Net.WebException: An error occurred while sending the request. Couldn't resolve proxy name ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.CurlException: Couldn't resolve proxy name
My configurations
Reverse Proxy Settings
Local port: 60103
Remote Hot: foo.example.com
Remote Port: 443
(Check box is enabled for both "Enable Reverse Proxies" and the above record itself)
SSL Proxy Settings
Host: foo.example.com
Port: 443
Both checkboxes are checked for "Enable SSL Proxying" and the record itself.
Under Help -> Proxying -> Install Charles Root Certificate, the certificate was installed and marked as trusted in keychain

It has been painful and I realize the mistake and here's the answer with the solution.
The cause of problem was a solution that worked for me in nodejs and it caused a confusion in here while not realizing that C#/F# provides a WebProxy class for that. nodejs doesn't have a WebProxy class (as per my knowledge) it so the concept is to use reverse proxy in nodejs, that is to send the request to localhost and a specific port which maps to the remote url.
So keep your URL same as where it should be pointed to and use WebProxy class here to point to localhost and port (8888 in my case) where Charles Proxy is listening and intercepting requests.
Thanks Fyodor Soikin for point out not to use the url (http:// etc.) and just use hostname.
// let url = "http://localhost:60103"; // don't do this, reverse proxy settings
let url = "https://example.com"; // keep the url intact
let request = WebRequest.Create(url) :?> HttpWebRequest
let myproxy = WebProxy("localhost", 8888); // port where Charles proxy is running
myproxy.BypassProxyOnLocal <- false;
request.Proxy <- myproxy;

kestrel + nginx intermittent 502

I have an ASP.NET core 1.0 app running on ubuntu 16.04 behind nginx/1.10.0. However, I notice that intermittently nginx throws the following error (in nginx error-log)
2017/06/08 05:19:19 [error] 11572#11572: *119049 upstream prematurely closed connection while reading response header from upstream, client: <ipaddress>, server: <servername>, request: "POST /<uri> HTTP/1.1", upstream: "http://127.0.0.1:5000/<uri>", host: "<servername>"
which results in a 502 Bad Gateway to the client although the request completes successfully in the application
2017-06-08 05:19:14.399 [DBG] Executed action method "<Controller.Action>", returned result "Microsoft.AspNetCore.Mvc.CreatedAtRouteResult
Could this be an nginx configuration issue or ASP.NET/Kestrel issue? Can someone give me hints to debug this?
Further notes:
I also see the following in the application log just after request completion
2017-06-08 05:19:14.399 [DBG] Executed action method "<Controller.Action>", returned result "Microsoft.AspNetCore.Mvc.CreatedAtRouteResult
2017-06-08 05:19:18.632 [DBG] Some connections failed to close gracefully during server shutdown.
2017-06-08 05:19:22.402 [DBG] Hosting starting
2017-06-08 05:19:22.689 [DBG] Hosting started
and the nginx error is just around the same time 2017/06/08 05:19:19
Also, normally in the logs I see messages around Executing ObjectResult and Executed time etc. which I don't see with the above request
2017-06-08 00:10:14.391 [DBG] Selected output formatter '"Microsoft.AspNetCore.Mvc.Formatters.JsonOutputFormatter"' and content type '"application/json"' to write the response.
2017-06-08 00:10:14.391 [INF] Executing ObjectResult, writing value "Microsoft.AspNetCore.Mvc.ControllerContext".
2017-06-08 00:10:14.504 [INF] Executed action "<Controller.Action>" in 1875.3845ms

SignalR with Redis Backplane Behind F5 - StatusCode: 400, ReasonPhrase: 'Bad Request'

I'm using SignalR version 2.1.2 with SignalR.Redis 2.1.2 on Server 2012 R2, IIS 8.5 with WebSockets enabled.
All is running perfectly in my development environment. I can even stand up copies on different servers (e.g. http machine1/myapp/signalr, http machine2/myapp/signalr) of the site configured to use the same backplane, and both UI's get messages pubb'd to them perfectly.
I then moved "myapp" to our next environment, which is a cluster of 2 machines sitting behind an F5 load balancer, with a dns alias setup to route to the F5, and then round robin "myapp". The website itself can connect to signalr just fine, and can receive published messages it subscribes to, BUT when I try to publish to the site via the alias (e.g. http myappalias/signalr), I get a 400, Bad Request error response. Here is an example of the error.
InnerException: Microsoft.AspNet.SignalR.Client.Infrastructure.StartException
_HResult=-2146233088
_message=Error during start request. Stopping the connection.
HResult=-2146233088
IsTransient=false
Message=Error during start request. Stopping the connection.
InnerException: System.AggregateException
_HResult=-2146233088
_message=One or more errors occurred.
HResult=-2146233088
IsTransient=false
Message=One or more errors occurred.
InnerException: Microsoft.AspNet.SignalR.Client.HttpClientException
_HResult=-2146233088
_message=StatusCode: 400, ReasonPhrase: 'Bad Request', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Pragma: no-cache
Transfer-Encoding: chunked
X-Content-Type-Options: nosniff
Persistent-Auth: true
Cache-Control: no-cache
Date: Thu, 13 Nov 2014 22:30:22 GMT
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Content-Type: text/html
Expires: -1
}
Here is some test code I'm using to publish test messages to each environment, where it fails on "connection.Start().Wait()"
class Program
{
static void Main(string[] args)
{
var connection = new HubConnection("http://myappalias/signalr");
connection.Credentials = System.Net.CredentialCache.DefaultNetworkCredentials;
var proxy = connection.CreateHubProxy("MyAppHub");
connection.Start().Wait();
ConsoleKeyInfo key = Console.ReadKey();
do
{
proxy.Invoke("NewMessage", new Message() { Payload = "Hello" });
Console.WriteLine("Message fired.");
key = Console.ReadKey();
} while (key.Key != ConsoleKey.Escape);
}
}
Now, if I don't use the "myappalias", and instead hit the server head on, it works perfectly. It appears either the F5 is the problem, the client needs to be configured differently for this scenario or I have to do something different when setting up signlar's startup class. Here is an example of the startup class I'm using.
[assembly: OwinStartup(typeof(MyApp.Startup))]
namespace MyApp
{
public class Startup
{
private static readonly ILog log = LogManager.GetLogger
(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);
public void Configuration(IAppBuilder app)
{
try
{
log.Debug(LoggingConstants.Begin);
string redisServer = ConfigurationManager.AppSettings["redis:server"];
int redisPort = Convert.ToInt32(ConfigurationManager.AppSettings["redis:port"]);
HubConfiguration configuration = new HubConfiguration();
configuration.EnableDetailedErrors = true;
configuration.EnableJavaScriptProxies = false;
configuration.Resolver = GlobalHost.DependencyResolver.UseRedis(redisServer, redisPort, string.Empty, "MyApp");
app.MapSignalR("/signalr", configuration);
log.Info("SIGNALR - Startup Complete");
}
finally
{
log.Debug(LoggingConstants.End);
}
}
}
}
I download the client source code, and wired that in directly instead of the nuget package, so I could step through everything. I seems it successfully negotiates, and then attempt to "connect" with SSE's and then LongPolling transports, but fails at both.
Question 1.1
Anyone know of an alternative to Signalr for .NET that supports scaling with load balancing in a less "I want to pull my hair out" kind of way?

It should not be necessary to configure source address affinity to use SignalR behind a load balancer. It's certainly not wrong to set up session affinity, but that doesn't fix your underlying problem.
If you look closely at the content of the 400 response, you probably see a message similar to "The ConnectionId is in the incorrect format."
SignalR uses the server's machine key to create an anti-CSRF token, but this requires that all the servers in your farm share a machine key for the token to be properly decrypted in when SignalR requests hop servers. The /negotiate request that you see succeed is the request that retrieves the anti-CSRF token. When the SignalR client then uses the anti-CSRF token to make a /connect request, it failed because the /connect request was processed by a different server that didn't create the token and is unable to decrypt it.
This explains why setting up session affinity fixed your problem, but sharing a machine key will help you avoid this problem even if something goes wrong with session affinity.
Here is an issue that filed on GitHub by someone who experienced a similar issue: https://github.com/SignalR/SignalR/issues/2292.

The problem was fixed by switching the profile for "MyApp" in the F5, to using the "source_addr" profile built into the F5 as a parent profile with a timeout of 1 hour. Here is a description of what that profile does:
Source address affinity persistence Also known as simple persistence,
source address affinity persistence supports TCP and UDP protocols,
and directs session requests to the same server based solely on the
source IP address of a packet.
EDIT
This ended up "Working" for a while, but if I deploy a publisher (something that simply publishes through the signalr client) without republishing the Hub, the publisher times out trying to connect over and over and over again. uhg.