Develop Reference

GRPC on Google Cloud Run : upstream connect error or disconnect/reset before headers. reset reason: remote reset

EDIT
It seems that my first error I describe is very easy to reproduce. Actually, Google Run fails to run any GRPC query on a .NET5 GRPC server it seems (at least, it did work before but as of today, February 21st, it seems that something changed). To reproduce:
Create a .NET5 GRPC server (also fails with .NET6):
dotnet new grpc -o TestGrpc
Change Program.cs so that it listens on $PORT, typically:
public static IHostBuilder CreateHostBuilder(string[] args)
{
var port = Environment.GetEnvironmentVariable("PORT") ?? "8080";
var url = string.Concat("http://0.0.0.0:", port);
return Host.CreateDefaultBuilder(args)
.ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.UseStartup<Startup>().UseUrls(url);
});
}
A very simple Dockerfile to have an image for the server (fails also with a more standard one, like here):
FROM mcr.microsoft.com/dotnet/sdk:5.0
COPY . ./
RUN dotnet restore ./TestGrpc.csproj
RUN dotnet build ./TestGrpc.csproj -c Release
CMD dotnet run --project ./TestGrpc.csproj
Build and push to Google Artifcats Registry.
Create a Cloud Run instance with HTTP/2 enabled (Ketrel requires HTTP/2 so we need to set
HTTP/2 end-to-end, yet I tested without as well but it's not better).
Use Grpcurl for instance and try:
grpcurl {CLOUD_RUN_URL}:443 list
And you will obtain the same error as I got with my (more complex) project:
Failed to list services: rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: remote reset
On the Google Cloud Run instance I only have the log:
2022-02-21T16:44:32.528530Z POST 200 1.02 KB 41 ms grpcurl/v1.8.6 grpc-go/1.44.1-dev https://***/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo
(I don't really understand why it's a 200 though... and never seems to reach the actual server implementation, just as if there was some kind of middleware blocking the query to reach the implementation... )
I'm pretty sure this used to worked as I started my project this way (and then changed the protos, the service, etc. ). If anyone has a clue I'd be more than grateful :-)
INITIAL POST (less precise than explanations above but I leave it here if it may give clues)
I have a server running within a Docker (.NET5 GRPC application). This server, when deployed locally works perfectly fine. But recently I have an error when I deploy it on Google Cloud Run: upstream connect error or disconnect/reset before headers. reset reason: remote reset when it was working fine before. I keep on having this error from any client I use, for instance with Curl:
curl -v https://{ENDPOINT}/{Proto-base}/{Method} --http2
* Trying ***...
* TCP_NODELAY set
* Connected to *** (***) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*.a.run.app
* start date: Feb 7 02:07:06 2022 GMT
* expire date: May 2 02:07:05 2022 GMT
* subjectAltName: host "***" matched cert's "*.a.run.app"
* issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1C3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5564aad30860)
> GET /{Proto}/{Method} HTTP/2
> Host: ***
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 503
< content-length: 85
< content-type: text/plain
< date: Mon, 21 Feb 2022 13:51:31 GMT
< server: Google Frontend
< traceparent: 00-5a74487dafb5687961deeb17e0158ca9-5ab63cd23680e7d7-01
< x-cloud-trace-context: 5a74487dafb5687961deeb17e0158ca9/6536478782730069975;o=1
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
<
* Connection #0 to host *** left intact
upstream connect error or disconnect/reset before headers. reset reason: remote reset
Same happens with Grpcurl:
grpcurl ***:443 list {Proto-base}
Failed to list methods for service "***.Company": rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: remote reset
I cannot find much resource on this error as most of the threads I read deal with another type of reset reason (like protocol, or connection, etc.). But I strictly have no idea what remote reset means and what I did wrong.
Looking at the logs in Google Cloud Run, I can see that the server is definitly hit, though I added trace logging in the route which is not triggered, hence it never reaches my code:
2022-02-21T14:44:22.840580Z POST 200 1.01 KB 1 msgrpc-python/1.44.0 grpc-c/22.0.0 (linux; chttp2) https://***/{Protos-base}/{Method}
(if I reached my code it should print some "Hellos" everywhere which it doesn't)
Has anyone ever found this?
P.S.: there are many things around about Envoy, but I don't even use this. I simply have a Cloud Run instance (with HTTP/2 - and I tried without but it fails due to protocol issue).

It is an actual bug from Envoy and Google Cloud Run. There is a quick fix if you're using .NET6, otherwise it's a bit more hacky. I will just copy here the answer provided by Amanda Tarafa Mas from Google Cloud Platform on the github issue I opened:
Here are the potential fixes:
When using .NET 6 you can set KestrelServerOptions.AllowAlternateSchemes to true.
If on a lower .NET version, consider something like GRPC :scheme pseudo-header passed from proxy/loadbalancer causes ConnectionAbortedException dotnet/aspnetcore#30532 (comment). Or consider upgrading to .NET 6.
What's happening:
Cloud Run has dependency on Envoy, which has a behavior change since 04/15/2021, see "preserve_downstream_scheme" in release notes:
https://www.envoyproxy.io/docs/envoy/latest/version_history/v1.18.0
Envoy recently removed the old behaviour: https://www.envoyproxy.io/docs/envoy/latest/version_history/current#removed-config-or-runtime
In turn, this exposes this .NET issue: GRPC :scheme pseudo-header passed from proxy/loadbalancer causes ConnectionAbortedException dotnet/aspnetcore#30532, for which the Kestrel configuration flag was added, but only for .NET 6.
I'm looking into having this documented somewhere. #meteatamel can you update the tutorial so that it uses the Kestrel option?
For me setting KestrelServerOptions.AllowAlternate was enough to make my GRPC server work again.
As #Craig said, you can track the issue here and see if it gets resolved.

Technology for cross-platform embedable .net Rest API

We have a .net server application, originally developed for windows. Now, we move it to linux platform (using mono + postgre instead of ms sql) and we also want to implement RESTful API for it, which can use AppDomain resources such as caches, some internal objects and so on. And so, we need a framework to built such an API. It'll be very good if can also built some web-interface using Razor or something, but it's not main in this case.
So far I've tried NancyFx which looks really promising, but I've encountered some errors running it on mono, for example, when I run my test app on windows with nancy self-hosting, it works really good. But, when executed with mono (on windows also), it lacks performance and gives me a little strange responses.
For example, for a simple method like below:
Get["/IsActive"] = param =>
{
var json = Response.AsJson(true);
return json;
};
I get simply true in .net execution and
P/1.1 200 OK
Content-Type: application/json; charset=utf-8
Server: Mono-HTTPAPI/1.0
Date: Mon, 13 Jul 2015 18:05:24 GMT
Transfer-Encoding: chunked
Keep-Alive: timeout=15,max=100
4
true
0
and on mono I get this response really slow. I guess it's some mono issues with HttpHandler or something...
So, what platform now is best for building embedable REST service for my server .net application running on linux with Mono?

Best way to tell if an SMTP message has an IMF header?

I am writing an SMTP parser that needs to handle standard smtp as well as SMTP messages sent from a system that I do not control, which do not include an IMF header. For example,
Standard SMTP with IMF header:
From: "Blah blah blah" blah#bluhblah.blah
To: Derp#dederp.com
Subject: This is a standard SMTP message
Date: Mon, 5 Oct 2009 11:36:07 +0530
MIME-Version: 1.0
Content-Type: text / plain
Content-Language: en-us
Testing testing 1 2 3
Bastardized SMTP:
Testing testing 1 2 3
Preferably in C#, but any language is fine.
No speculation please, I need someone who understands the SMTP / IMF specs to respond. I take that back, speculate all you want. lol

Unless you really, really want to write your own SMTP parser; you'll want a 3rd party library to help. We chose OpenPop. It works well, handles all the things we've thrown at it, and is still supported.
No, I'm not affiliated with them.

couple of things.
You don't want a SMTP parser, you want a Mime message parser. Parsing mime messages is not a trivial exercise. There are 1000s of broken message formats out there, or just plain weird formats, yet still RFC2822 (Mime Message) compliant.
I recommend just using a Mime message parser (either one from codeproject or like source -- if you want a free one), or else a commercial prouduct.
Then, just loop through the headers to see if they match your rules.
--Dave
PS: I'm partial to aspNetMime, since I wrote it.

SMTP server Configuration [closed]

Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 9 years ago.
Improve this question
I have Windows XP and configured SMTP server in my IIS. Specified AllUnAssigned in IP address of general tab and specified 127.0.0.1 in connection under Access tab.
Now I tried sending mails using my local SMTP server using the following code,
MailMessage amessage = new MailMessage();
amessage.To.Add(new MailAddress("xxx#gmail.com"));
amessage.From = new MailAddress("yyy#gmail.com");
amessage.Subject = "TestMail";
amessage.Body = "This is a testmail";
SmtpClient clienta = new SmtpClient("localhost");
clienta.Timeout = 500;
clienta.Credentials = CredentialCache.DefaultNetworkCredentials;
try
{
clienta.Send(amessage);
}
catch (Exception ex)
{
Console.WriteLine(ex);
Console.ReadLine();
}
Console.ReadLine();
I don't get any error in executing the above code but no mails were sent/received. When I checked in the Inetpub I find the below file:
From: postmaster#munged
To: yyy#gmail.com
Date: Tue, 26 Apr 2011 11:07:22 +0530
MIME-Version: 1.0
Content-Type: multipart/report;
report-type=delivery-status;
boundary="9B095B5ADSN=_01CC03D260FE2C6C00000003munged?xxx"
Message-ID:
Subject: Delivery Status Notification
(Failure)
This is a MIME-formatted message.
Portions of this message may be
unreadable without a MIME-capable mail
program.
--9B095B5ADSN=_01CC03D260FE2C6C00000003munged?munged
Content-Type: text/plain;
charset=unicode-1-1-utf-7
This is an automatically generated
Delivery Status Notification.
Delivery to the following recipients
failed.
xxx#gmail.com
--9B095B5ADSN=_01CC03D260FE2C6C00000003munged?munged
Content-Type: message/delivery-status
Reporting-MTA: dns;munged
Received-From-MTA: dns;munged
Arrival-Date: Tue, 26 Apr 2011
11:07:16 +0530
Final-Recipient: rfc822;xxx#gmail.com
Action: failed Status: 5.0.0
Diagnostic-Code: smtp;550-5.7.1
[122.178.191.78] The IP you're using
to send mail is not authorized to
550-5.7.1 send email directly to our
servers. Please use the SMTP relay at
your 550-5.7.1 service provider
instead. Learn more at
550 5.7.1
http://mail.google.com/support/bin/answer.py?answer=10336
m9si19863225wfl.114
--9B095B5ADSN=_01CC03D260FE2C6C00000003munged?munged
Content-Type: message/rfc822
Received: from munged
([127.0.0.1]) by munged with
Microsoft SMTPSVC(6.0.2600.2180);
Tue, 26 Apr 2011 11:07:16 +0530
MIME-Version: 1.0 From: yyy#gmail.com
To: xxx#gmail.com Date: 26 Apr 2011
11:07:16 +0530 Subject: TestMail
Content-Type: text/plain;
charset=us-ascii
Content-Transfer-Encoding:
quoted-printable Return-Path:
munged#gmail.com Message-ID:
X-OriginalArrivalTime: 26 Apr 2011
05:37:16.0515 (UTC)
FILETIME=[009D2330:01CC03D4]
This is a testmail
--9B095B5ADSN=_01CC03D260FE2C6C00000003munged?munged--
Can anyone please help me on this?

Here's the important part of the error message:
Diagnostic-Code: smtp;550-5.7.1 [122.178.191.78] The IP you're using to send mail is not authorized to 550-5.7.1 send email directly to our servers. Please use the SMTP relay at your 550-5.7.1 service provider instead.
To solve this, you'll need to setup an SMTP relay to send to your ISP's email servers. They'll send to Google for you.
For clarity:
This is NOT an application/code/C# problem, but an IIS problem.
Google will NOT accept SMTP connections from random nodes on the internet. That means: YOUR Windows XP SMTP server will not have direct sending ability.
You MUST send to a known good whitelisted SMTP host.
Your best option is to configure IIS to act as a relay to a known good SMTP server.
Your best bet for a known good SMTP server is your ISP's. This is likely smtp.yourISP.com.
This tutorial's Steps 2 & 3 will help you achieve this. Your 'smart host' will be your ISP's SMTP server DNS name or IP address.

I think that if you want to send emails through gmail from your PC, you should encrypt your data with SSL. You have to options here:
use specific asp functions
use a software like stunnel (it is not so difficult to configure)

You can solve this using one of the following two ways
Option 1. Change '127.0.01' to 'localhost' in connection under Access tab.
Option 2. Change the code used to connect using '127.0.01' instead of localhost.
SmtpClient clienta = new SmtpClient("127.0.01");

gzip HttpWebRequest

I am submitting HTTP POST requests via HttpWebRequest which contain a large amount of content. I would like to gzip the message content. Is this possible?
Does IIS 7 have to be configured to handle the compressed content? It has already been configured to serve compressed responses.
I've tried adding a Content-Encoding = gzip header and writing to the request stream wrapped in a GZipStream but the server returns a 504 (GatewayTimeout) which seems odd.

I don't believe IIS7 supports GZIP requests, out of the box. Here's why. On my IIS7 machine, gzip.dll does not export decompression methods.
c:\Windows\System32\inetsrv>c:\vc9\bin\dumpbin.exe -exports gzip.dll
Microsoft (R) COFF/PE Dumper Version 9.00.30729.01
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file gzip.dll
File Type: DLL
Section contains the following exports for gzip.dll
00000000 characteristics
47919400 time date stamp Sat Jan 19 01:09:04 2008
0.00 version
1 ordinal base
6 number of functions
6 number of names
ordinal hint RVA name
1 0 0000242D Compress
2 1 00002E13 CreateCompression
3 2 000065AE DeInitCompression
4 3 000012EE DestroyCompression
5 4 0000658D InitCompression
6 5 000065B6 ResetCompression
Summary
1000 .data
1000 .reloc
1000 .rsrc
6000 .text
I think this represents a change in gzip.dll. I believe in prior versions of gzip.dll, there were 12 exported methods, including 6 that did Decompression.

The vast majority of web servers do not support compressed request bodies. mod_deflate can be configured to support it on Apache but seldom actually is (as a zip-bomb is an easy potential DoS attack). I'm not aware of an IIS solution.
If you are talking back to your own server there is of course nothing stopping you doing the compression at the application level. If you have to pass a standard form type for the backend to read, you should pick multipart/form-data, as URL-encoding would bloat the binary data of the compressed content parameter.

I got the same error.
Solved by adding executionTimeout to web.config:
<httpRuntime maxRequestLength="1048576" executionTimeout="300" />
ExecutionTimeout- is on seconds...