This is not one of the "why does my code not download over HTTPS" questions. I have seen and read all of them years ago, and since then I have implemented this same approach a number of times, successfully.
My very simple C# code that targets 4.7.1 framework and runs on 4.7.2 framework attempts to download resources over HTTPS. It follows all commonly known recommendations as regards to configuring ServicePointManager, WebClient, and WebRequest. Initially, I tested it with an Apache 2.4.20 instance on our LAN, and it worked fine. Next, I tested it with a couple of other servers on the public Internet, and there I ran into the error that occurs about 5 seconds after making the request, even though the page opens in a browser instantaneously:
{"The request was aborted: Could not create SSL/TLS secure channel."}
I checked the server certificates. They are trusted by Mozilla and Windows and open in any browser known to me, in a fully trusted mode. The sites respond over TLS 1.2. There is nothing visibly wrong with the servers or their certificates. For a sanity check, I took Google's homepage address https://www.google.ca/?gws_rd=ssl, and it opened just fine. Some sites open, other sites do not. I tried enabling one TLS version at a time, and each worked equally well with each of the working sites, whereas none of them worked with the non-working sites.
Here is my code. It is written to be used with basic auth but tested without it as well. This is not an authentication issue.
ServicePointManager.Expect100Continue = true;
ServicePointManager.DefaultConnectionLimit = 9999;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11; // | SecurityProtocolType.Tls | SecurityProtocolType.Ssl3; // tested by enabling only 1 protocol at a time locally. Each worked.
if (ServicePointManager.ServerCertificateValidationCallback == null) // for testing only
{
ServicePointManager.ServerCertificateValidationCallback += (se, cert, chain, sslerror) => { return true; };
}
using (var client = new WebClientEx()) // Ex adds Timeout and CookieContainer
{
client.CachePolicy = new System.Net.Cache.RequestCachePolicy(System.Net.Cache.RequestCacheLevel.BypassCache);
client.Headers.Add("Cache-Control", "no-cache");
client.Encoding = Encoding.UTF8;
var cc = new CredentialCache();
// Using one or the other, depending on the situation, or none
//cc.Add(url, "Basic", new NetworkCredential(user, pass));
//cc.Add(new Uri(url.GetLeftPart(UriPartial.Authority)), "Digest", new NetworkCredential(user, pass));
client.Credentials = cc;
return client.DownloadData(url);
}
Before falling back onto WebClient, I have tried with the plain HttpRequest for the sake of its reliable timing out, which is important for my implementation:
var req = (HttpWebRequest)WebRequest.Create(url);
req.Timeout = (int)timeout.TotalMilliseconds;
req.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate; // tried with or w/o
var encoded = Convert.ToBase64String(Encoding.GetEncoding("ISO-8859-1").GetBytes($"{user}:{pass}"));
req.Headers.Add("Authorization", "Basic " + encoded); // tried with or w/o
var credentialCache = new CredentialCache();
credentialCache.Add(url, "Basic", new NetworkCredential(user, pass)); // tried with or w/o
req.Credentials = credentialCache;
req.PreAuthenticate = true; // tried with or w/o
using (var response = (HttpWebResponse)req.GetResponse())
{
using (var ms = new MemoryStream())
{
response.GetResponseStream().CopyTo(ms);
return ms.ToArray();
}
}
Both variants of my code equally worked with my local Apache and Google but did not work with some other sites. I cannot find any faults with my code. One of the sites that did not work is https://jigsaw.w3.org/HTTP/Basic/ that opens fine in any browser. This is the site that most of our software uses for unit testing, so I am expected to produce something that works with it during the test phase and with any arbitrary web server out there in production.
I turned on System.Net tracing and compared the successful and failed download attempts, and the difference is that one proceeds with the handshake while the other fails right away. There is no useful info in the trace.
Any ideas what is amiss?
All services and technologies mentioned in this question are beyond my control and have to remain whatever they currently are. I am powerless to change any versions or to install any updates. It has to work the way it currently is. If you are not happy with it, please skip over to other questions.
The answer is "nothing is wrong with the code". Something is actually wrong with those select servers that I had to test with. They are old or niche, and they are non-compliant to a degree that .NET framework refuses to interoperate with them. As soon as I tested with modern, compliant servers, everything worked as expected, both over HTTP or HTTPS and both with basic or digest authentication.
Related
Hopefully someone can help with this problem. Recently our machines were updated with KB4344167 which includes security updates for .NET 4.7.1. Unfortunately this update has broken our code for a Webrequest. When we run the code below we get this error:
The request was aborted: Could not create SSL/TLS secure channel.
// Create a request for the URL.
WebRequest request = WebRequest.Create(url);
//specify to use TLS 1.2 as default connection
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
request.Timeout = int.Parse(configmanager.GetSetting("Webtimeout"));
// Set proxy
request.Proxy = WebRequest.DefaultWebProxy;
request.Proxy.Credentials = CredentialCache.DefaultCredentials;
// Define a cache policy for this request only.
HttpRequestCachePolicy noCachePolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.NoCacheNoStore);
request.CachePolicy = noCachePolicy;
ServicePointManager.ServerCertificateValidationCallback = (s, cert, chain, ssl) => true;
// Get the response.
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
When the security update is uninstalled from the machine the code executes fine. Are we missing something in the code above? Thats about the only thing I can think of.
Any help is greatly appreciated!
#Damien_The_Unbeliever had the correct answer. Ultimately the problem was the order of the ServicePointManager and the Webrequest.Create. Reversing those lines, so the ServicePointManager is defined before the Webrequest.Create fixed the issue. I still don't know why adding the ServicePointManager after the Create fixed our original issue when our server moved to TLS 1.2, but we're not going to worry about that now.
I ran into something similar. It appears MS may have broken something in their attempt to only enable TLS 1.2. https://support.microsoft.com/en-us/help/4458166/applications-that-rely-on-tls-1-2-strong-encryption-experience-connect
So far, I've tried adding the suggested config to the app.config and it worked like a charm. No more SSL/TLS errors.
<runtime>
<AppContextSwitchOverrides value="Switch.System.Net.DontEnableSchUseStrongCrypto=false" />
</runtime>
NOTE: we found this on servers that are selectively patched, i.e. they don't yet have the MS fix. Our development machines never saw the problem.
I'm aware that there are tons of questions and answers on StackOverflow and all over the internet with exactly the same or very similar error message and I'm pretty sure I've read and tried 90% of them. I still can't get this fixed.
Below is the code with WebClient, as suggested by Visual Studio. But I also tried with HttpWebRequest/WebRequest, despite the warning that it's obsolete. No change, I got the same error: "The request was aborted: Could not create SSL/TLS secure channel.".
Now here's the code. makeProxy is really just 3 lines, a new WebProxy plus the credentials for it. I believe it works because (a) if I provide a wrong credential there then I get an authentication error, (b) if I try to go to a http and not a https page, then I get it back, so I'm out on the internet.
protected string getToken() {
string url = ConfigurationManager.AppSettings["domo_api_url_auth"];
WebClient c = new WebClient();
c.Proxy = makeProxy();
c.Credentials = new NetworkCredential(ConfigurationManager.AppSettings["domo_api_cid"], ConfigurationManager.AppSettings["domo_api_pwd"]);
c.Encoding = System.Text.Encoding.UTF8;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
// ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
// ServicePointManager.ServerCertificateValidationCallback += ValidateRemoteCertificate;
Stream responseStream = c.OpenRead(url);
return new StreamReader(responseStream).ReadToEnd();
}
My problem is that the message is not very helpful. I also tried to debug it, didn't help. I thought of the certificate on the site of the web service I'm trying to reach and went through the process of allowing that cert for W3SVC. Didn't help. And I get this same error for every https site, like google.com or stackoverflow.com or my own company's web site. But when I go to a random news site with http only, everything works fine.
The extremely suspicious thing is that it doesn't work even when I uncomment that first ServerCertificateValidationCallback line, and when I uncomment the second, supposedly redirecting the code to the validation, it literally never gets there. Like the validation wouldn't even start.
How to troubleshoot this? I don't even understand on which point of the web request-response process it fails.
You likely need to activate TLS in addition to SSLv3:
ServicePointManager.SecurityProtocol =
SecurityProtocolType.Ssl3 |
SecurityProtocolType.Tls |
SecurityProtocolType.Tls11 |
SecurityProtocolType.Tls12;
I am trying to write C# code which makes a web request against a REST service endpoint used for calculating sales tax within a web application. This is a third party service, and it is secured using SSL. There are two environments, UAT and production. The code that runs the webrequest looks like this:
...
var req = WebRequest.Create(url) as HttpWebRequest;
req.Method = "POST";
req.ContentType = "application/json";
...
using (var webresponse = req.GetResponse())
{
using (var responseStream = new StreamReader(webresponse.GetResponseStream()))
{
var respJson = responseStream.ReadToEnd();
calcResult = BuildResponse(calcRequest, respJson, consoleWriteRawReqResponse);
}
}
return calcResult;
This works fine against the UAT environment. But when I run the same code against the production environment, I get the error:
"Could not create SSL/TLS secure channel"
I am able to execute both requests from Postman without issue, without any special modifications.
This led me down the path of investigating this error, and I found many helpful SO posts discussing the topic, including:
The request was aborted: Could not create SSL/TLS secure channel
Could not create SSL/TLS secure channel, despite setting ServerCertificateValidationCallback
These helped by pointing me in the right direction, which was to look at setting the ServicePointManager.SecurityProtocol setting to a different value, and using the ServicePointManager.ServerCertificateValidationCallback to investigate errors.
What I found after playing with these is the following:
The UAT environment call will work with the default setting of Ssl3 | Tls (default for .NET 4.5.2), while the production environment will not.
The production call will work ONLY when I set this setting to explicitly to Ssl3.
That code looks like this:
...
var req = WebRequest.Create(url) as HttpWebRequest;
req.Method = "POST";
req.ContentType = "application/json";
...
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
ServicePointManager.ServerCertificateValidationCallback = new RemoteCertificateValidationCallback(CertValidationCallback);
using (var webresponse = req.GetResponse())
{
using (var responseStream = new StreamReader(webresponse.GetResponseStream()))
{
var respJson = responseStream.ReadToEnd();
calcResult = BuildResponse(calcRequest, respJson, consoleWriteRawReqResponse);
}
}
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls;
return calcResult;
This is particularly confusing because in looking at the endpoints in a web browser, I can see that they are both secured by the same wildcard certificate and are both using TLS 1.0.
So I would expect that setting ServicePointManager.SecurityProtocol to TLS would work, but it does not.
I really want to avoid setting ServicePointManager.SecurityProtocol explicitly to SSL3 because our application is a web application and has multiple other integration points that communicate over SSL. These are all working fine and I do not want to adversely affect their functionality. Even if I set this setting right before the call, and then change it back right after, I run the risk of hitting concurrency issues since ServicePointManager.SecurityProtocol is static.
I investigated that topic as well, and did not like what I read. There are mentions of using different app domains:
.NET https requests with different security protocols across threads
How to use SSL3 instead of TLS in a particular HttpWebRequest?
But that seems overly complex / hacky to me. Is dealing with creating an app domain really the only solution? Or is this something I simply should not be trying to solve and instead take it up with the owner of the service in question? It is very curious to me that it would work with TLS on one environment / server, but not the other.
EDIT
I did some more playing with this. I changed my client to use the approach outlined quite well in this blog post (using a different app domain to isolate the code that changes the ServicePointManager.SecurityProtocol):
https://bitlush.com/blog/executing-code-in-a-separate-application-domain-using-c-sharp
This actually worked quite well, and could be fall back solution. But I also learned that the provider of the service in question has a different endpoint (same URL, different port) that is secured using TLS 1.2. Thankfully, by expanding my SecurityProtocol setting like so in the global.asax.cs application start event:
ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
I am able to communicate with the service fine in all environments. It also does not affect my existing integrations with other services (CyberSource, for example).
However - now there is a new but related question. Why is it that this call ONLY works if I expand SecurityProtocolType as above? My other integrations, like CyberSource, did not require this. Yet this one does. And they all appear to be secured using TLS 1.2 from what I saw in the browser.
If you run 4.0 you can use it like this:
ServicePointManager.SecurityProtocol = (SecurityProtocolType)3072; // SecurityProtocolType.Tls12
This one worked for me:
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls
| SecurityProtocolType.Tls11
| SecurityProtocolType.Tls12
| SecurityProtocolType.Ssl3;
Some of the advanced TLS configurations on a web server are hidden. Your production server has likely been modified to protect against DROWN, logjam, FREAK, POODLE and BEAST attacks.
see
https://tecadmin.net/enable-tls-on-windows-server-and-iis/
https://support.microsoft.com/en-us/help/187498/how-to-disable-pct-1.0,-ssl-2.0,-ssl-3.0,-or-tls-1.0-in-internet-information-services
To make changes to these advanced TLS settings, it's not as simple as clicking some buttons in IIS. Well it could be that simple if you use a third-party tool like this: https://www.nartac.com/Products/IISCrypto
These configurations work fine for major recent web browsers, but .Net seems to struggle with such modern secure server configurations (unless you manually override defaults as you discovered).
Conclusion: it's not obvious, your UAT and Production environments seem the same, but they're not.
I need to develop a C# client able to post XML docs into a SAP Bussiness Connector 4.6.
The Client:
.NET until 4.6, VS 2013 available, SO Win 8.1 Pro 64 bits.
The Server:
Windows 2000, with SAP Business Connector 4.6 (it's really equal to WebMethods 4.6), configured with https and Client certificate as authentication method. The server has a certificate that doesn't match the URL used in local development against it.
I'm trying several .NET methods and clients. I'm actually trying httpClient to connect with BC, without success.
The actual code is very similar to this:
try
{
System.Net.Http.HttpClient client;
//HttpClient uses the HttpMessageHandler pipeline for sending and receiving requests
//WebRequestHandler derives from HttpClientHandler but adds properties that generally only are available on full .NET
System.Net.Http.WebRequestHandler wrHandler = new System.Net.Http.WebRequestHandler();
System.Security.Cryptography.X509Certificates.X509Certificate x509cert = System.Security.Cryptography.X509Certificates.X509Certificate2.CreateFromCertFile("MyClientCertificate.crt");
wrHandler.ClientCertificates.Add(x509cert);
client = new System.Net.Http.HttpClient(wrHandler);
//THIS SKIPS SERVER CERTIFICATE
System.Net.ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
System.Net.Http.HttpRequestMessage request = new System.Net.Http.HttpRequestMessage();
request.Method = System.Net.Http.HttpMethod.Post;
request.RequestUri = new System.Uri("https://192.168.12.12:3333/invoke/wm.PartnerMgr.flows.UCLR.0000000001:ORDERS");
request.Content = new System.Net.Http.StringContent(CtrXMLToSendContent.Text, Encoding.UTF8, CtrClient1MediaType.Text);
System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Ssl3 | System.Net.SecurityProtocolType.Tls;
Task<System.Net.Http.HttpResponseMessage> taskResponse = client.SendAsync(request);
if (taskResponse.Wait(-1)) //-1 = infinite
{
//Response received
System.Net.Http.HttpResponseMessage response = taskResponse.Result;
TextResponseData.Text = response.ToString();
Task<String> taskResponseStr = response.Content.ReadAsStringAsync();
taskResponseStr.Wait();
String responseContent = taskResponseStr.Result;
TextResponseContent.Text = responseContent;
}
else
{
TextClient1Infolog.AppendText("Wait ");
}
}
catch (Exception _e)
{
TextClient1Infolog.AppendText("Error :" + Environment.NewLine);
TextClient1Infolog.AppendText(_e.ToString());
}
When I launch it, it throws an exception, giving this error:
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.
There's no server response at all I can explore or debug. I think it's a problem of handshaking SSL protocol. I must insist: There's no response text from server, no 402 or 404 message errors at all: Nothing is received.
Firefox over that direction says no connection. IE, with TSL 1.0, 1.2 and 1.3 activated, says it cannot connect; activating SSL 3.0 and reopening page says there's a problem with server certificate (yes, because I'm accesing from local network, where certificate states for something like "this.server.com").
Fiddler4 doesn't say me much (and autocertificates appears to be confusing something), SoapUI appears to be not very useful here (haven't see way to set certificates and SSL handshakes).
Found a tool (TestSSLServer, link: http://www.bolet.org/TestSSLServer/) that gave me this info:
Supported versions: SSLv3 TLSv1.0 Deflate compression: no Supported
cipher suites (ORDER IS NOT SIGNIFICANT):
SSLv3
RSA_EXPORT_WITH_RC4_40_MD5
RSA_WITH_RC4_128_MD5
RSA_WITH_RC4_128_SHA
RSA_EXPORT_WITH_RC2_CBC_40_MD5
RSA_EXPORT_WITH_DES40_CBC_SHA
RSA_WITH_DES_CBC_SHA
RSA_WITH_3DES_EDE_CBC_SHA
(TLSv1.0: idem)
---------------------- Server certificate(s):
188659f61762af0de690bf5cb76a8554e7ff7f23: CN=my.server.com,
OU=Domain Control Validated
---------------------- Minimal encryption strength: weak encryption (40-bit) Achievable encryption strength: strong encryption
(96-bit or more) BEAST status: vulnerable CRIME status: protected
So I suppose I have to force SSL3 handshake (I think this is OK in my code), validate server certificate (too)... perhaps I must do something with client certificate.
Must I generate one for my client computer and load it? How can I generate one with one of the encriptions of the TestSSLServer's list? Wich file should be installed in what computer?
And I'm open to hear about other http clients (ServerXMLHTTP, etc) or tools able to test xml postings over xml. Any ideas about how to proceed for now?
I have written a WinForms app that uploads addresses from a spreadsheet, and geocodes them using an external geocoding service. This all works fine on my local machine, but the time has come for it to be installed on other peoples computers for testing. The app no longer works now though, generating the below error:
System.Net.WebException: The remote server returned an error: (407) Proxy Authentication Required.
Having read a lot and chatted breifly to our network guys, it seems i need to establish the Security Context for the users account and work with this to correct the error.
Has anyone got any pointers about how I should be going about this?
Thanks in advance!
C
It depends on how your uploading the data. If your using a http request (as it looks like you are) it will look something like;
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create("https://test.example.com/");
req.Method = "POST";
req.ContentType = "text/xml";
req.Credentials = new NetworkCredential("TESTACCOUNT", "P#ssword");
StreamWriter writer = new StreamWriter(req.GetRequestStream());
writer.Write(input);
writer.Close();
var rsp = req.GetResponse().GetResponseStream();