I'm using SignalR version 2.1.2 with SignalR.Redis 2.1.2 on Server 2012 R2, IIS 8.5 with WebSockets enabled.
All is running perfectly in my development environment. I can even stand up copies on different servers (e.g. http machine1/myapp/signalr, http machine2/myapp/signalr) of the site configured to use the same backplane, and both UI's get messages pubb'd to them perfectly.
I then moved "myapp" to our next environment, which is a cluster of 2 machines sitting behind an F5 load balancer, with a dns alias setup to route to the F5, and then round robin "myapp". The website itself can connect to signalr just fine, and can receive published messages it subscribes to, BUT when I try to publish to the site via the alias (e.g. http myappalias/signalr), I get a 400, Bad Request error response. Here is an example of the error.
InnerException: Microsoft.AspNet.SignalR.Client.Infrastructure.StartException
_HResult=-2146233088
_message=Error during start request. Stopping the connection.
HResult=-2146233088
IsTransient=false
Message=Error during start request. Stopping the connection.
InnerException: System.AggregateException
_HResult=-2146233088
_message=One or more errors occurred.
HResult=-2146233088
IsTransient=false
Message=One or more errors occurred.
InnerException: Microsoft.AspNet.SignalR.Client.HttpClientException
_HResult=-2146233088
_message=StatusCode: 400, ReasonPhrase: 'Bad Request', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Pragma: no-cache
Transfer-Encoding: chunked
X-Content-Type-Options: nosniff
Persistent-Auth: true
Cache-Control: no-cache
Date: Thu, 13 Nov 2014 22:30:22 GMT
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Content-Type: text/html
Expires: -1
}
Here is some test code I'm using to publish test messages to each environment, where it fails on "connection.Start().Wait()"
class Program
{
static void Main(string[] args)
{
var connection = new HubConnection("http://myappalias/signalr");
connection.Credentials = System.Net.CredentialCache.DefaultNetworkCredentials;
var proxy = connection.CreateHubProxy("MyAppHub");
connection.Start().Wait();
ConsoleKeyInfo key = Console.ReadKey();
do
{
proxy.Invoke("NewMessage", new Message() { Payload = "Hello" });
Console.WriteLine("Message fired.");
key = Console.ReadKey();
} while (key.Key != ConsoleKey.Escape);
}
}
Now, if I don't use the "myappalias", and instead hit the server head on, it works perfectly. It appears either the F5 is the problem, the client needs to be configured differently for this scenario or I have to do something different when setting up signlar's startup class. Here is an example of the startup class I'm using.
[assembly: OwinStartup(typeof(MyApp.Startup))]
namespace MyApp
{
public class Startup
{
private static readonly ILog log = LogManager.GetLogger
(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);
public void Configuration(IAppBuilder app)
{
try
{
log.Debug(LoggingConstants.Begin);
string redisServer = ConfigurationManager.AppSettings["redis:server"];
int redisPort = Convert.ToInt32(ConfigurationManager.AppSettings["redis:port"]);
HubConfiguration configuration = new HubConfiguration();
configuration.EnableDetailedErrors = true;
configuration.EnableJavaScriptProxies = false;
configuration.Resolver = GlobalHost.DependencyResolver.UseRedis(redisServer, redisPort, string.Empty, "MyApp");
app.MapSignalR("/signalr", configuration);
log.Info("SIGNALR - Startup Complete");
}
finally
{
log.Debug(LoggingConstants.End);
}
}
}
}
I download the client source code, and wired that in directly instead of the nuget package, so I could step through everything. I seems it successfully negotiates, and then attempt to "connect" with SSE's and then LongPolling transports, but fails at both.
Question 1.1
Anyone know of an alternative to Signalr for .NET that supports scaling with load balancing in a less "I want to pull my hair out" kind of way?
It should not be necessary to configure source address affinity to use SignalR behind a load balancer. It's certainly not wrong to set up session affinity, but that doesn't fix your underlying problem.
If you look closely at the content of the 400 response, you probably see a message similar to "The ConnectionId is in the incorrect format."
SignalR uses the server's machine key to create an anti-CSRF token, but this requires that all the servers in your farm share a machine key for the token to be properly decrypted in when SignalR requests hop servers. The /negotiate request that you see succeed is the request that retrieves the anti-CSRF token. When the SignalR client then uses the anti-CSRF token to make a /connect request, it failed because the /connect request was processed by a different server that didn't create the token and is unable to decrypt it.
This explains why setting up session affinity fixed your problem, but sharing a machine key will help you avoid this problem even if something goes wrong with session affinity.
Here is an issue that filed on GitHub by someone who experienced a similar issue: https://github.com/SignalR/SignalR/issues/2292.
The problem was fixed by switching the profile for "MyApp" in the F5, to using the "source_addr" profile built into the F5 as a parent profile with a timeout of 1 hour. Here is a description of what that profile does:
Source address affinity persistence Also known as simple persistence,
source address affinity persistence supports TCP and UDP protocols,
and directs session requests to the same server based solely on the
source IP address of a packet.
EDIT
This ended up "Working" for a while, but if I deploy a publisher (something that simply publishes through the signalr client) without republishing the Hub, the publisher times out trying to connect over and over and over again. uhg.
Related
I have a .NET Core 3.1 C# application which is calling an API via HTTPS (and presenting its public key as part of getting the token as that certificate is later used to decrypt information sent back separately). On just about all our machines, it is working, but on one Windows 8.1 machine, we get the following series of exceptions when we try to initially connect for an authentication token:
The SSL connection could not be established, see inner exception.
---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
---> System.ComponentModel.Win32Exception (0x80090326): The message received was unexpected or badly formatted.
The exception is thrown from System.Net.Http.HttpClient.FinishSendAsyncBuffered so I suspect it is happening at the HTTPS level and our certificate stuff is not really relevant here anyway.
Our code to get the token looks like this:
The constructor for the auth service:
public XXXXAuthService(IXXDbService dbService, XXXXApiConfig config)
{
_dbService = dbService;
_config = config;
// try forcing TLS1.2 for SSL connection exceptions thrown in some operating environments
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
_httpClient = new HttpClient {BaseAddress = new Uri(config.BaseUrl)};
_httpClient.DefaultRequestHeaders.Accept.Clear();
_httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
}
Code to get the auth token:
private async Task<string> GetXXXXBearerToken(string userId, DateTime creationTime)
{
var token = await GenerateProviderJwtForXXXX(userId, creationTime);
var kvp = new List<KeyValuePair<string, string>>
{
new KeyValuePair<string, string>("grant_type", "urn:ietf:params:oauth:grant-type:token-exchange"),
new KeyValuePair<string, string>("subject_token", token),
new KeyValuePair<string, string>("subject_token_type", "urn:ietf:params:oauth:token-type:jwt")
};
var data = new FormUrlEncodedContent(kvp);
var publicKey = await GetXXXXPublicKey();
_httpClient.DefaultRequestHeaders.Remove("X-XXXX-Public-Cert");
_httpClient.DefaultRequestHeaders.Add("X-XXXX-Public-Cert", publicKey);
var response = await _httpClient.PostAsync("Identity/token", data);
if (!response.IsSuccessStatusCode)
throw new Exception("XXXX Token Server Error: " + response.ReasonPhrase);
var result = await response.Content.ReadAsStringAsync();
var authResponse = JsonConvert.DeserializeObject<OAuthResponse>(result);
if (!string.IsNullOrEmpty(authResponse.access_token))
return authResponse.access_token;
System.Diagnostics.Trace.WriteLine("Token Exchange Result: " + result);
if (!string.IsNullOrEmpty(authResponse.error))
{
var outcome = new XXX.XXXX.Model.OperationOutcome();
outcome.Issue.Add(new XXX.XXXX.Model.OperationOutcome.IssueComponent()
{
//some code to throw an error is here
}
throw new XXX.XXXX.Rest.XXXXOperationException("Bearer Token Exchange failed", response.StatusCode);
}
Unfortunately none of the existing questions/advice anywhere on Stack Overflow, or the rest of the web, for this particular error seems to have helped. They are primarily about version discrepancies between client and server which seems not to be the case here as I am forcing TLS 1.2 (which is active and enabled on the failing machine).
Interestingly, I can visit the server URL in a browser via HTTPS just fine, which suggests there is something about my code that is the problem rather than the machine, but it works everywhere else.
I have confirmed that:
The certificate I am using to authenticate the connection on the machine is valid and has a chain of trust (though as above I don't think we are getting that far as the TLS connection itself is failing)
The server we are calling supports TLS 1.2 (by forcing it)
I can get to the website for the URL independently via the browser
Is there something I need to do either in the code or on the machine to get this call to work everywhere?
Things I have tried to resolve the issue
Installing all Windows 8.1 updates to present day
Forcing TLS 1.2 in the code (see above code sample)
Limiting VM to TLS 1.2 only
I might be able to at least point you in the right direction…
Same Symptoms
I had a .NET Core 3.1 web app running on IIS (Windows Server 2012 R2) that got the exact same error and stacktrace when it tried to connect to another server using TLS 1.2. I also had the symptom where I could connect with the browser (Chrome), but not with the app. (Would have been interesting to see if Internet Explorer browser worked though.)
Root Cause
The TLS handshake was failing because the two servers were unable to agree on a common cipher suite. (Using Wireshark, I discovered that when my app tried to connect it provided a more limited set of cipher suites than when the Chrome browser made the call.)
Solution
In my case, I used IIS Crypto (a small free tool: https://www.nartac.com/Products/IISCrypto/) to enable additional cipher suites on my web app's server. I downloaded and ran IIS Crypto, checkmarked additional cipher suites on its Cipher Suites tab, and then restarted the machine.
One of the new cipher suites worked with my app and the destination server, so the TLS handshake was successful and the error was resolved.
One quick caveat: Some cipher suites are more secure than others, so you'll want to read up on best practices.
Addendum
If you want to further diagnose the failure, I'd recommend installing Wireshark (another free tool: https://www.wireshark.org/#download) on the machine with your .NET Core app. If a TLS Handshake Failure is the issue, you will see a message like: Alert (Level: Fatal, Description: Handshake Failure)
This primer on wireshark output helped me:
https://blog.catchpoint.com/2017/05/12/dissecting-tls-using-wireshark/
I faced a simular issue, and in order to help others here's what I concluded:
Sucessfully executing this code doesn't mean that your application supports the specified protocol version, and the "SSL Error" can still occur later on when trying to establish a connection:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
In my case I was trying to force Tls13 and found out that my app configuration didn't actually support it:
net core 3.0 running on a Windows Server Datacenter 2019, version 1809
So I had to change my configuration to the following which provides support for the protocol version I needed:
Net framework 5.0 on a Windows Server Datacenter 2022, OS build 20348.288
I was trying to connect to an endpoint that suddenly dropped Tls 1.2 support (not sure why) and from then on only accepted Tls 1.3.
I'm following the following link https://docs.aws.amazon.com/AmazonS3/latest/dev/HLuploadFileDotNet.html
to upload files from local machine to an S3 bucket on VPC. The application is also testing and running on the on-premise machine.
var s3Client = new AmazonS3Client(RegionEndpoint.USEast2);
var fileTransferUtility = new TransferUtility(s3Client);
await fileTransferUtility.UploadAsync("c:\tmp\test.txt", "bucketName");
However, the code gets the following error.
A socket operation was attempted to an unreachable network
Should an Url be given?
Here is the network traffic captured by Fiddler. However, it gets a different exception for the code.
GET http://1xx.1xx.1xx.2xx/latest/meta-data/iam/security-credentials HTTP/1.1
Host: 1xx.1xx.1xx.2xx
HTTP/1.1 503 Service Unavailable
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/html; charset=utf-8
Proxy-Connection: Keep-Alive
Connection: Keep-Alive
Content-Length: 787
Network Error
Network Error (tcp_error)
A communication error occurred: "Operation timed out"
The Web Server may be down, too busy, or experiencing other problems preventing it from responding to requests. You may wish to try again at a later time.
For assistance, contact your network support team.
.aws\config
[default]
region = USWest2
I had the same error today, even though I had a valid $USERPROFILE\.aws\credentials file - it was actually because $USERPROFILE\AppData\Local\AWSToolkit\RegisteredAccounts.json couldn't be decrypted (not sure why), which causes AWS to think you don't have have local credentials, and hence tries to make a connection to the EC2 metadata URL which is http://169.254.169.254/latest/meta-data/?. On a local development machine that won't be accessible. For me, deleting the $USERPROFILE\AppData\Local\AWSToolkit\RegisteredAccounts.json file did the trick. FWIW, I only managed to figure this out by reading through the source of the AWS SDK...
I have deployed an ASP .net MVC web app to Azure App service.
I do a GET request from my site to some controller method which gets data from DB(DbContext). Sometimes the process of getting data from DB may take more than 4 minutes. That means that my request has no action more than 4 minutes. After that Azure kills the connection - I get message:
500 - The request timed out. The web server failed
to respond within the specified time.
This is a method example:
[HttpGet]
public async Task<JsonResult> LongGet(string testString)
{
var task = Task.Delay(360000);
await task;
return Json("Woke", JsonRequestBehavior.AllowGet);
}
I have seen a lot of questions like this, but I got no answer:
Not working 1
Cant give other link - reputation is too low.
I have read this article - its about Azure Load Balancer which is not available for webapps, but its written that common way of handling my problem in Azure webapp is using TCP Keep-alive. So I changed my method:
[HttpGet]
public async Task<JsonResult> LongPost(string testString)
{
ServicePointManager.SetTcpKeepAlive(true, 1000, 5000);
ServicePointManager.MaxServicePointIdleTime = 400000;
ServicePointManager.FindServicePoint(Request.Url).MaxIdleTime = 4000000;
var task = Task.Delay(360000);
await task;
return Json("Woke", JsonRequestBehavior.AllowGet);
}
But still get same error.
I am using simple GET request like
GET /Home/LongPost?testString="abc" HTTP/1.1
Host: longgetrequest.azurewebsites.net
Cache-Control: no-cache
Postman-Token: bde0d996-8cf3-2b3f-20cd-d704016b29c6
So I am looking for the answer what am I doing wrong and how to increase request timeout time in Azure Web app. Any help is appreciated.
Azure setting on portal:
Web sockets - On
Always On - On
App settings:
SCM_COMMAND_IDLE_TIMEOUT = 3600
WEBSITE_NODE_DEFAULT_VERSION = 4.2.3
230 seconds. That's it. That's the in-flight request timeout in Azure App Service. It's hardcoded in the platform so TCP keep-alives or not you're still bound by it.
Source -- see David Ebbo's answer here:
https://social.msdn.microsoft.com/Forums/en-US/17305ddc-07b2-436c-881b-286d1744c98f/503-errors-with-large-pdf-file?forum=windowsazurewebsitespreview
There is a 230 second (i.e. a little less than 4 mins) timeout for requests that are not sending any data back. After that, the client gets the 500 you saw, even though in reality the request is allowed to continue server side.
Without knowing more about your application it's difficult to suggest a different approach. However what's clear is that you do need a different approach --
Maybe return a 202 Accepted instead with a Location header to poll for the result later?
I just changed my Azure Web Site from Shared Enviroment to Standard, and it works.
I need to develop a C# client able to post XML docs into a SAP Bussiness Connector 4.6.
The Client:
.NET until 4.6, VS 2013 available, SO Win 8.1 Pro 64 bits.
The Server:
Windows 2000, with SAP Business Connector 4.6 (it's really equal to WebMethods 4.6), configured with https and Client certificate as authentication method. The server has a certificate that doesn't match the URL used in local development against it.
I'm trying several .NET methods and clients. I'm actually trying httpClient to connect with BC, without success.
The actual code is very similar to this:
try
{
System.Net.Http.HttpClient client;
//HttpClient uses the HttpMessageHandler pipeline for sending and receiving requests
//WebRequestHandler derives from HttpClientHandler but adds properties that generally only are available on full .NET
System.Net.Http.WebRequestHandler wrHandler = new System.Net.Http.WebRequestHandler();
System.Security.Cryptography.X509Certificates.X509Certificate x509cert = System.Security.Cryptography.X509Certificates.X509Certificate2.CreateFromCertFile("MyClientCertificate.crt");
wrHandler.ClientCertificates.Add(x509cert);
client = new System.Net.Http.HttpClient(wrHandler);
//THIS SKIPS SERVER CERTIFICATE
System.Net.ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
System.Net.Http.HttpRequestMessage request = new System.Net.Http.HttpRequestMessage();
request.Method = System.Net.Http.HttpMethod.Post;
request.RequestUri = new System.Uri("https://192.168.12.12:3333/invoke/wm.PartnerMgr.flows.UCLR.0000000001:ORDERS");
request.Content = new System.Net.Http.StringContent(CtrXMLToSendContent.Text, Encoding.UTF8, CtrClient1MediaType.Text);
System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Ssl3 | System.Net.SecurityProtocolType.Tls;
Task<System.Net.Http.HttpResponseMessage> taskResponse = client.SendAsync(request);
if (taskResponse.Wait(-1)) //-1 = infinite
{
//Response received
System.Net.Http.HttpResponseMessage response = taskResponse.Result;
TextResponseData.Text = response.ToString();
Task<String> taskResponseStr = response.Content.ReadAsStringAsync();
taskResponseStr.Wait();
String responseContent = taskResponseStr.Result;
TextResponseContent.Text = responseContent;
}
else
{
TextClient1Infolog.AppendText("Wait ");
}
}
catch (Exception _e)
{
TextClient1Infolog.AppendText("Error :" + Environment.NewLine);
TextClient1Infolog.AppendText(_e.ToString());
}
When I launch it, it throws an exception, giving this error:
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.
There's no server response at all I can explore or debug. I think it's a problem of handshaking SSL protocol. I must insist: There's no response text from server, no 402 or 404 message errors at all: Nothing is received.
Firefox over that direction says no connection. IE, with TSL 1.0, 1.2 and 1.3 activated, says it cannot connect; activating SSL 3.0 and reopening page says there's a problem with server certificate (yes, because I'm accesing from local network, where certificate states for something like "this.server.com").
Fiddler4 doesn't say me much (and autocertificates appears to be confusing something), SoapUI appears to be not very useful here (haven't see way to set certificates and SSL handshakes).
Found a tool (TestSSLServer, link: http://www.bolet.org/TestSSLServer/) that gave me this info:
Supported versions: SSLv3 TLSv1.0 Deflate compression: no Supported
cipher suites (ORDER IS NOT SIGNIFICANT):
SSLv3
RSA_EXPORT_WITH_RC4_40_MD5
RSA_WITH_RC4_128_MD5
RSA_WITH_RC4_128_SHA
RSA_EXPORT_WITH_RC2_CBC_40_MD5
RSA_EXPORT_WITH_DES40_CBC_SHA
RSA_WITH_DES_CBC_SHA
RSA_WITH_3DES_EDE_CBC_SHA
(TLSv1.0: idem)
---------------------- Server certificate(s):
188659f61762af0de690bf5cb76a8554e7ff7f23: CN=my.server.com,
OU=Domain Control Validated
---------------------- Minimal encryption strength: weak encryption (40-bit) Achievable encryption strength: strong encryption
(96-bit or more) BEAST status: vulnerable CRIME status: protected
So I suppose I have to force SSL3 handshake (I think this is OK in my code), validate server certificate (too)... perhaps I must do something with client certificate.
Must I generate one for my client computer and load it? How can I generate one with one of the encriptions of the TestSSLServer's list? Wich file should be installed in what computer?
And I'm open to hear about other http clients (ServerXMLHTTP, etc) or tools able to test xml postings over xml. Any ideas about how to proceed for now?
I've created a Sample Application based on SuperWebSocket (running v0.3). I manage to get connect to the WebSocket Server through Telnet, but for some reason I'm having trouble doing it through JavaScript (running Chrome 17.0.963.46 m).
Through Telnet I can connect through either localhost:911 or 192.168.1.147:911.
My Application is running on http://localhost/Raphael-Test/, and I've tried running through both localhost and the local networks IP, both gets stuck at "Connecting" ie. status 0.
Is there anything obvious I'm missing, any configuration that should be done in the Web Application itself? I should add that I've successfully tried out the LiveChat demo, got it working through JavaScript.
This is my current Client Implementation running when the Page has been fully loaded:
ws = new WebSocket("ws://192.168.1.147:911");
ws.onopen = function () {
alert("connected");
};
ws.onmessage = function (evt) {
var msg = evt.data;
alert(msg);
};
Handshake (with NO response):
GET / HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: 192.168.1.147:911
Origin: http://192.168.1.147
Sec-WebSocket-Key: 8bl46pmPrixTYRJ/5i9Sug==
Sec-WebSocket-Version: 13
It turns out I used the SuperSocket Server base classes instead of SuperWebSocket on the server side. This made the TCP connection itself work as expected, but did of course not handle the WebSocket Handshake and therefore the connection failed.