Develop Reference

Inconsistent SSL error when accessing different sites

This is not one of the "why does my code not download over HTTPS" questions. I have seen and read all of them years ago, and since then I have implemented this same approach a number of times, successfully.
My very simple C# code that targets 4.7.1 framework and runs on 4.7.2 framework attempts to download resources over HTTPS. It follows all commonly known recommendations as regards to configuring ServicePointManager, WebClient, and WebRequest. Initially, I tested it with an Apache 2.4.20 instance on our LAN, and it worked fine. Next, I tested it with a couple of other servers on the public Internet, and there I ran into the error that occurs about 5 seconds after making the request, even though the page opens in a browser instantaneously:
{"The request was aborted: Could not create SSL/TLS secure channel."}
I checked the server certificates. They are trusted by Mozilla and Windows and open in any browser known to me, in a fully trusted mode. The sites respond over TLS 1.2. There is nothing visibly wrong with the servers or their certificates. For a sanity check, I took Google's homepage address https://www.google.ca/?gws_rd=ssl, and it opened just fine. Some sites open, other sites do not. I tried enabling one TLS version at a time, and each worked equally well with each of the working sites, whereas none of them worked with the non-working sites.
Here is my code. It is written to be used with basic auth but tested without it as well. This is not an authentication issue.
ServicePointManager.Expect100Continue = true;
ServicePointManager.DefaultConnectionLimit = 9999;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11; // | SecurityProtocolType.Tls | SecurityProtocolType.Ssl3; // tested by enabling only 1 protocol at a time locally. Each worked.
if (ServicePointManager.ServerCertificateValidationCallback == null) // for testing only
{
ServicePointManager.ServerCertificateValidationCallback += (se, cert, chain, sslerror) => { return true; };
}
using (var client = new WebClientEx()) // Ex adds Timeout and CookieContainer
{
client.CachePolicy = new System.Net.Cache.RequestCachePolicy(System.Net.Cache.RequestCacheLevel.BypassCache);
client.Headers.Add("Cache-Control", "no-cache");
client.Encoding = Encoding.UTF8;
var cc = new CredentialCache();
// Using one or the other, depending on the situation, or none
//cc.Add(url, "Basic", new NetworkCredential(user, pass));
//cc.Add(new Uri(url.GetLeftPart(UriPartial.Authority)), "Digest", new NetworkCredential(user, pass));
client.Credentials = cc;
return client.DownloadData(url);
}
Before falling back onto WebClient, I have tried with the plain HttpRequest for the sake of its reliable timing out, which is important for my implementation:
var req = (HttpWebRequest)WebRequest.Create(url);
req.Timeout = (int)timeout.TotalMilliseconds;
req.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate; // tried with or w/o
var encoded = Convert.ToBase64String(Encoding.GetEncoding("ISO-8859-1").GetBytes($"{user}:{pass}"));
req.Headers.Add("Authorization", "Basic " + encoded); // tried with or w/o
var credentialCache = new CredentialCache();
credentialCache.Add(url, "Basic", new NetworkCredential(user, pass)); // tried with or w/o
req.Credentials = credentialCache;
req.PreAuthenticate = true; // tried with or w/o
using (var response = (HttpWebResponse)req.GetResponse())
{
using (var ms = new MemoryStream())
{
response.GetResponseStream().CopyTo(ms);
return ms.ToArray();
}
}
Both variants of my code equally worked with my local Apache and Google but did not work with some other sites. I cannot find any faults with my code. One of the sites that did not work is https://jigsaw.w3.org/HTTP/Basic/ that opens fine in any browser. This is the site that most of our software uses for unit testing, so I am expected to produce something that works with it during the test phase and with any arbitrary web server out there in production.
I turned on System.Net tracing and compared the successful and failed download attempts, and the difference is that one proceeds with the handshake while the other fails right away. There is no useful info in the trace.
Any ideas what is amiss?
All services and technologies mentioned in this question are beyond my control and have to remain whatever they currently are. I am powerless to change any versions or to install any updates. It has to work the way it currently is. If you are not happy with it, please skip over to other questions.

The answer is "nothing is wrong with the code". Something is actually wrong with those select servers that I had to test with. They are old or niche, and they are non-compliant to a degree that .NET framework refuses to interoperate with them. As soon as I tested with modern, compliant servers, everything worked as expected, both over HTTP or HTTPS and both with basic or digest authentication.

SChannel error: The request was aborted: Could not create SSL/TLS secure channel

I am trying to access and consume the XML at the following site: https://www.dhs.gov/ntas/1.1/alerts.xml. However, I keep getting the message: 'The request was aborted: Could not create SSL/TLS secure channel.' when I run my application on our dev server. I do not see anything in our server's event logs.
Strangely, when I access the feed from my localhost, it works fine.
My application is using .Net framework 4.6.2. As I understand, it should support TLS12 by default (unless I am wrong). When surf directly to the link, the browsers I tried -- Chrome, Firefox, Edge -- can access and display the XML successfully. IE11 is the exception and cannot display it.
I have tried adding TLS12 support manually in the following ways:
ServicePointManager.SecurityProtocol = (SecurityProtocolType)3072
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls
| SecurityProtocolType.Tls11
| SecurityProtocolType.Tls12
ServicePointManager.SecurityProtocol = (SecurityProtocolType)3072 | (SecurityProtocolType)768 | SecurityProtocolType.Tls;
Here is the code I am using. As noted above this works on localhost.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(NTASUri);
request.UserAgent = Request.ServerVariables["HTTP_USER_AGENT"];
WebResponse response = request.GetResponse();
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string responseFromServer = reader.ReadToEnd();
XmlDocument xmlDoc = new XmlDocument();
xmlDoc.LoadXml(responseFromServer);
return xmlDoc.InnerXml;
Does anyone have any ideas on what can be done to get this working?

My systems person ended up figuring this out, thankfully. We are now able to consume the feed on our server.
In the registry, the cipher suites listed under the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Cryptography\Configuration\Local\SSL\00010002\ key were limited and only a subset of those supported by Windows.
Deleting the Functions string value which specifies a set of cipher suites returns the Windows client to the default behavior/order.
Further information:
https://learn.microsoft.com/en-us/windows-server/identity/ad-fs/operations/manage-ssl-protocols-in-ad-fs
https://www.windows-security.org/2c488aac52906551ff218fd5c2bdaddc/ssl-cipher-suite-order
Hope this helps someone else in this situation.

KB4344167 security update breaks TLS Code

Hopefully someone can help with this problem. Recently our machines were updated with KB4344167 which includes security updates for .NET 4.7.1. Unfortunately this update has broken our code for a Webrequest. When we run the code below we get this error:
The request was aborted: Could not create SSL/TLS secure channel.
// Create a request for the URL.
WebRequest request = WebRequest.Create(url);
//specify to use TLS 1.2 as default connection
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
request.Timeout = int.Parse(configmanager.GetSetting("Webtimeout"));
// Set proxy
request.Proxy = WebRequest.DefaultWebProxy;
request.Proxy.Credentials = CredentialCache.DefaultCredentials;
// Define a cache policy for this request only.
HttpRequestCachePolicy noCachePolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.NoCacheNoStore);
request.CachePolicy = noCachePolicy;
ServicePointManager.ServerCertificateValidationCallback = (s, cert, chain, ssl) => true;
// Get the response.
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
When the security update is uninstalled from the machine the code executes fine. Are we missing something in the code above? Thats about the only thing I can think of.
Any help is greatly appreciated!

#Damien_The_Unbeliever had the correct answer. Ultimately the problem was the order of the ServicePointManager and the Webrequest.Create. Reversing those lines, so the ServicePointManager is defined before the Webrequest.Create fixed the issue. I still don't know why adding the ServicePointManager after the Create fixed our original issue when our server moved to TLS 1.2, but we're not going to worry about that now.

I ran into something similar. It appears MS may have broken something in their attempt to only enable TLS 1.2. https://support.microsoft.com/en-us/help/4458166/applications-that-rely-on-tls-1-2-strong-encryption-experience-connect
So far, I've tried adding the suggested config to the app.config and it worked like a charm. No more SSL/TLS errors.
<runtime>
<AppContextSwitchOverrides value="Switch.System.Net.DontEnableSchUseStrongCrypto=false" />
</runtime>
NOTE: we found this on servers that are selectively patched, i.e. they don't yet have the MS fix. Our development machines never saw the problem.

HTTPS over proxy doesn't work

I'm aware that there are tons of questions and answers on StackOverflow and all over the internet with exactly the same or very similar error message and I'm pretty sure I've read and tried 90% of them. I still can't get this fixed.
Below is the code with WebClient, as suggested by Visual Studio. But I also tried with HttpWebRequest/WebRequest, despite the warning that it's obsolete. No change, I got the same error: "The request was aborted: Could not create SSL/TLS secure channel.".
Now here's the code. makeProxy is really just 3 lines, a new WebProxy plus the credentials for it. I believe it works because (a) if I provide a wrong credential there then I get an authentication error, (b) if I try to go to a http and not a https page, then I get it back, so I'm out on the internet.
protected string getToken() {
string url = ConfigurationManager.AppSettings["domo_api_url_auth"];
WebClient c = new WebClient();
c.Proxy = makeProxy();
c.Credentials = new NetworkCredential(ConfigurationManager.AppSettings["domo_api_cid"], ConfigurationManager.AppSettings["domo_api_pwd"]);
c.Encoding = System.Text.Encoding.UTF8;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
// ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
// ServicePointManager.ServerCertificateValidationCallback += ValidateRemoteCertificate;
Stream responseStream = c.OpenRead(url);
return new StreamReader(responseStream).ReadToEnd();
}
My problem is that the message is not very helpful. I also tried to debug it, didn't help. I thought of the certificate on the site of the web service I'm trying to reach and went through the process of allowing that cert for W3SVC. Didn't help. And I get this same error for every https site, like google.com or stackoverflow.com or my own company's web site. But when I go to a random news site with http only, everything works fine.
The extremely suspicious thing is that it doesn't work even when I uncomment that first ServerCertificateValidationCallback line, and when I uncomment the second, supposedly redirecting the code to the validation, it literally never gets there. Like the validation wouldn't even start.
How to troubleshoot this? I don't even understand on which point of the web request-response process it fails.

You likely need to activate TLS in addition to SSLv3:
ServicePointManager.SecurityProtocol =
SecurityProtocolType.Ssl3 |
SecurityProtocolType.Tls |
SecurityProtocolType.Tls11 |
SecurityProtocolType.Tls12;

HTTP 407 proxy authentication error when calling a web service

I'm working on a .NET app that calls 3rd party web services over the internet. The services do not use SOAP, so we manually construct an XML request document, send it to the service via HTTP, and retrieve an XML response.
Our code is a Windows service that is run in the context of a normal Windows domain account, and sits behind a proxy server (Microsoft ISA Server) configured to require NTLM authentication. The account running our service has permission to access the internet through the proxy server.
The code looks like this:
// Create the request object.
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
request.Method = "POST";
// Configure for authenticating proxy server requiring Windows domain credentials.
request.Proxy = New WebProxy(proxyAddress) { UseDefaultCredentials = true };
// Set other required headers.
request.Accept = acceptableMimeType;
request.Headers.Add(HttpRequestHeader.AcceptCharset, acceptableCharset);
request.Headers.Add(HttpRequestHeader.AcceptEncoding, "none");
request.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-gb");
request.Headers.Add(HttpRequestHeader.CacheControl, "no-store");
request.Headers.Add(HttpRequestHeader.ContentEncoding, "none");
request.Headers.Add(HttpRequestHeader.ContentLanguage, "en-gb");
request.ContentType = requestMimeType;
request.ContentLength = requestBytes.Length;
// Make the method call.
using(Stream stream = request.GetRequestStream()) {
stream.Write(requestBytes, 0, requestBytes.Length);
}
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
// Extract the data from the response without relying on the HTTP Content-Length header
// (we cannot trust all providers to set it correctly).
const int bufferSize = 1024 * 64;
List<byte> responseBytes = new List<byte>();
using(Stream stream = new BufferedStream(response.GetResponseStream(), bufferSize)) {
int value;
while((value = stream.ReadByte()) != -1) {
responseBytes.Add((byte) value);
}
}
This works fine if the proxy server is turned off, or the URL has been whitelisted as not requiring authentication, but as soon as authentication is active, it always fails with an HTTP 407 error.
I put the above code in a test harness, and tried every method I could think of for configuring the request.Proxy property, without success.
I then noticed that all the 3rd party web services that we have to call are HTTPS. When I tried accessing them as HTTP instead, the proxy authentication started working. Is there some extra hoop I have to jump through to get proxy authentication and HTTPS to play nicely?
PS: The same problems occur with the open source SmoothWall proxy server, so I can't just write it off as a bug in ISA Server.
PPS: I'm aware that you can configure proxy settings in app.config, but (a) doing it in code shouldn't make any difference, and (b) the application design requires that we read the proxy settings from a database at runtime.

Have you tried setting the proxy in the app.config ?
To disable the proxy, in the App.config file add the following configuration
<system.net>
<defaultProxy enabled="false" useDefaultCredentials="false">
<proxy/>
<bypasslist/>
<module/>
</defaultProxy>
</system.net>
To enable the proxy and to use the default proxy settings(specified in IE) add this configuration in your App.config
<system.net>
<defaultProxy enabled="true" useDefaultCredentials="true">
<proxy/>
<bypasslist/>
<module/>
</defaultProxy>
</system.net>

I did have a similar situation
Did you noticed it worked when you accessed the internet before you ran the code? and if you had not accessed the internet for ages (20mins for me) you got the error.
have you tried to set the proxy credentials directly?
//setup the proxy
request.Proxy = new WebProxy("proxyIp", 8080);
request.Proxy.Credentials = CredentialCache.DefaultCredentials;
I hope this fixes your issue too

I think I will have to write off this question. My original posted code does appear to work sometimes. Our proxy server is extremely unreliable; one minute it will block an internet connection from any software, and the next it will allow it. The IT guys seem powerless to do anything about it, and we (everyone outside the IT department) have no authority to make changes to the network infrastructure.
If anyone has any ideas on how to "harden" my code to compensate for an unreliable proxy server, then I'd be interested to hear them. :-)

Is there something wrong with your proxy server's certificate? If your service can't establish HTTPS then it will throw an error.