I've seen a lot of discussion about "TF400324: Team Foundation services are not available from server", but everything I've read relates to DNS or proxies, and reflect being unable to connect to TFS at all, through any channel. My case is different: I cannot reach my TFS server when using the .NET library, but it works fine using Visual Studio's workflow tools, and I can reach the same URL just fine in a browser.
Zee code, it is here:
private TfsConfigurationServer _server;
...
Uri url = new Uri(serverName + rootFolder);
var creds1 = new NetworkCredential(username, password, Environment.UserDomainName);
var creds2 = new BasicAuthCredential(creds1);
var creds3 = new TfsClientCredentials(creds2) { AllowInteractive = false };
_server = new TfsConfigurationServer(url, creds3);
// Throws Microsoft.TeamFoundation.TeamFoundationServiceUnavailableException
_server.Authenticate();
The creds are needlessly elaborate because I've tried various suggested solutions, but I don't think that's relevant; I get the same behavior with any other creds I've tried, or no creds at all.
If I copy the exact URL to a browser, I can authenticate and proceed. Within Visual Studio I can connect to TFS using Source Control Explorer and other tools, without explicitly authenticating. What might the library want handled differently?
Additional detail from the error:
Team Foundation services are not available from server https://tfs.imahufflepuff.com:8080/tfs/Root/Project.
Technical information (for administrator):
The underlying connection was closed: An unexpected error occurred on a send.
---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send.
---> System.IO.IOException: The handshake failed due to an unexpected packet format.
We don't have an in-house TFS admin, otherwise I'd kick this issue over to him. I've tried to use Fiddler to get additional detail, but VS consistently refuses to show up there. I can reach an externally-hosted API while debugging, so I don't think there's a network or proxy problem locking down VS.
Have you tried using TfsTeamProjectCollection class instead of TfsConfigurationServer ?
E.g.
Uri url = new Uri(serverName + rootFolder);
var creds = new NetworkCredential(username, password, Environment.UserDomainName);
var server = new TfsTeamProjectCollection(url, creds);
server.Authenticate();
You can also try debugging this issue using Fiddler. You'll have to change VS proxy settings before starting Fiddler:
Either set the registry key reg add hkcu\SOFTWARE\Microsoft\VisualStudio\12.0\TeamFoundation\RequestSettings
/v BypassProxyOnLocal /t REG_SZ /d False
or
Set environment variable TFS_BYPASS_PROXY_ON_LOCAL=0
Seems like an SSL handshake issue at its root, which has nothing to do with the TFS and HTTP protocol and authentication, they sit above SSL. So make sure you have a valid certificate, matching hostname, good validity, accessible CRL where the cert is not revoked, etc. Also check in other browsers or openssl.exe ("openssl.exe s_client -connect servername:8080") for more diagnostic info. You didn't mention but a proxy or an SSL-level certificate authentication could also cause problems, should be easily debuggable by the methods I mentioned above.
Related
I have an RDS instance running Postgres which has IAM Authentication enabled. I can successfully connect using an IAM token via powershell, but I'm having trouble with certificates when trying to connect using the Npgsql library with C# (targeting .Net Core 3.1) from Visual Studio 2019 Version 16.8.5 (on .Net Framework 4.8).
Note, I've been referencing this AWS page, which gives download links for the certificates.
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html
I've also been referencing this page to guide me through getting this set up.
https://aws.amazon.com/premiumsupport/knowledge-center/rds-postgresql-connect-using-iam/
I can connect with PowerShell, using:
$ENV:RDSHOST="my-postgres-instance.eu-west-1.rds.amazonaws.com"
$ENV:PGPASSWORD="$(aws rds generate-db-auth-token --hostname $ENV:RDSHOST --port 5432 --region eu-west-1 --username my_db_user)"
psql -h $ENV:RDSHOST -p 5432 "sslmode=verify-full sslrootcert=rds-ca-2019-root.pem dbname=my_db user=my_db_user password=$ENV:PGPASSWORD"
However when I try to open a connection with C# using the Npgsql library with the code below, I get an exception:
var host = "my-postgres-instance.eu-west-1.rds.amazonaws.com";
var port = 5432;
var user = "my_db_user";
var token = RDSAuthTokenGenerator.GenerateAuthToken(RegionEndpoint.EUWest1, host, port, user);
var builder = new NpgsqlConnectionStringBuilder
{
Host = host,
Port = port,
Database = "my_db",
Username = user,
Password = token,
SslMode = SslMode.Require,
RootCertificate = #"C:\Path\To\Cert\rds-combined-ca-bundle.pem",
};
using (var conn = new NpgsqlConnection(builder.ConnectionString))
{
conn.Open();
}
Npgsql.NpgsqlException: 'Exception while performing SSL handshake
AuthenticationException: The remote certificate is invalid according to the validation procedure.
It's worth noting that I can connect setting TrustServerCertificate = true, however as I understand it, this is masking the problem and reducing security, and is not advisable.
The AWS page linked above mentions:
If your application is on Microsoft Windows and requires a PKCS7 file, you can download the PKCS7 certificate bundle. This bundle contains both the intermediate and root certificates at https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.p7b.
I've tried specifying this p7b file as the NpgsqlConnectionStringBuilder.ClientCertificate, but this causes a different exception when opening the connection:
Internal.Cryptography.CryptoThrowHelper.WindowsCryptographicException: 'Cannot find the original signer.'
Anyone able to help me understand how to connect? It's worth noting that I'm not very familiar with detting up SSL connections or dealing with certificates, so bear that in mind.
Similar questions have been asked multiple times before - but before flagging this one as duplicate please read on. Most of these questions are very old. I have worked through a lot of questions and answers and did not find a suitable solution.
We have an Azure Cloud Service project in .net 4.5. It connects to dozens of our customers' APIs (not necessarily cloud hosted) without any problems, but a single API fails with this error message:
The request was aborted: Could not create SSL/TLS secure channel
What am I missing here?
This is the code (slightly condensed) that I am using to connect to the API (this runs per API, so the base URL does not change):
ServicePointManager.ServerCertificateValidationCallback += ValidateRemoteCertificate;
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol =
SecurityProtocolType.Tls12 |
SecurityProtocolType.Tls11 |
SecurityProtocolType.Tls |
SecurityProtocolType.Ssl3;
ApiClient = HttpClientFactory.Create();
ApiClient.DefaultRequestHeaders.Authorization = null;
ApiClient.DefaultRequestHeaders.TryAddWithoutValidation("Authorization", "Basic {passwordToken}");
ApiClient.DefaultRequestHeaders.Accept.Clear();
ApiClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
SemaphoreSlim throttler = new SemaphoreSlim(initialCount: 50);
var sp = ServicePointManager.FindServicePoint(new Uri(baseUrl));
sp.SetTcpKeepAlive(true, 30000, 30000);
foreach (var request in urls)
{
Result = new HttpResponseMessage();
Result = await ApiClient.GetAsync(url);
...
}
This is what makes it hard to debug:
This problem only occurs in production, i.e. when running as an Azure Cloud Service. Not when debugging locally.
It only occurs with requests sent through HttpClient. Not with WebClient.
Further research (comparing the APIs) revealed that this API is the only one that has enabled SNI and ONLY supports TLS1.2.
Suggestions considered from other questions/answers regarding SNI in .net Framework:
To prevent misunderstandings: This is about the cloud service connecting to an API, not about a connection that is being made to the cloud service.
The HttpClient instance is being reused for all requests to a single API. (This is important as this answer suggests that the SNI tag will be created with the domain HttpClient has been initialized with). I have also tried configuring TLS after the Factory instantiated the HttpClient. No change.
The certificates are valid of course. No self-signed certificates but regular trusted ones off the shelf. Opening the API in any browser also works like a charm.
TLS1.2 is not enabled by default in .net framework 4.5, but the line ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12should actually enable it. Is there anything wrong with the way I am doing it?
Calling the API in curl (WSL and Azure remote bash) with curl --user 'user:pwd' https://myurl also works perfectly and returns the expected data.
Testing tls1.2 with openssl like openssl s_client -connect hostname:443 -tls1_2 does not reveal any issues. The chain is displayed correctly, and a TLSv1.2 session is confirmed. Testing the server's SNI feature with openssl with openssl s_client -connect host:443 -tls1_2 -servername host -tlsextdebug -msg reveals SNI support by returning TLS server extension "server name" (id=0), len=0 I get the same certificate if I provide a completely different fantasy hostname though.
I captured the TLS/SNI handshake when debugging locally (see screenshot below). No issues. My ability to debug ends with the cloud service. I would love to see the handshake between the cloud service and the API in WireShark, but I don't know of any option to analyze network traffic at that layer on an Azure cloud service. But if anyone knows how to capture the handshake process, I'd appreciate some hints.
The server selects ECDHE-RSA-AES256-GCM-SHA384 as a cipher suite during the handshake with openssl which is pretty much default for TLS1.2. I don't have access to the cloud services list of cipher suites that it would provide at Client Hello - any idea how to find out?
I don't have any proof that SNI is actually causing the problem, but this is the only difference between this API and dozens of others I can spot.
Stack trace:
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel. at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar) --- End of inner exception stack trace --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at WCFServiceWebRole1.Controllers... in [the line calling GetAsync()]
I ended up re-creating a simple API on one of my servers and configured the software in such a way to send its requests there. That way I could capture the TLS handshake and analyze it in Wireshark. These are the supported cipher suites (client side, which is Azure cloud service):
And these are the cipher suites supported by the API which is not working:
I would assume that there should be a match so that server and client can agree on one. However, I cannot find a match... guess that is what is causing the problem. In fact, the list of supported cipher suites is much longer in a local debugging session - and there is at least one match which explains why it works locally.
I have a .NET Core 3.1 C# application which is calling an API via HTTPS (and presenting its public key as part of getting the token as that certificate is later used to decrypt information sent back separately). On just about all our machines, it is working, but on one Windows 8.1 machine, we get the following series of exceptions when we try to initially connect for an authentication token:
The SSL connection could not be established, see inner exception.
---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
---> System.ComponentModel.Win32Exception (0x80090326): The message received was unexpected or badly formatted.
The exception is thrown from System.Net.Http.HttpClient.FinishSendAsyncBuffered so I suspect it is happening at the HTTPS level and our certificate stuff is not really relevant here anyway.
Our code to get the token looks like this:
The constructor for the auth service:
public XXXXAuthService(IXXDbService dbService, XXXXApiConfig config)
{
_dbService = dbService;
_config = config;
// try forcing TLS1.2 for SSL connection exceptions thrown in some operating environments
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
_httpClient = new HttpClient {BaseAddress = new Uri(config.BaseUrl)};
_httpClient.DefaultRequestHeaders.Accept.Clear();
_httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
}
Code to get the auth token:
private async Task<string> GetXXXXBearerToken(string userId, DateTime creationTime)
{
var token = await GenerateProviderJwtForXXXX(userId, creationTime);
var kvp = new List<KeyValuePair<string, string>>
{
new KeyValuePair<string, string>("grant_type", "urn:ietf:params:oauth:grant-type:token-exchange"),
new KeyValuePair<string, string>("subject_token", token),
new KeyValuePair<string, string>("subject_token_type", "urn:ietf:params:oauth:token-type:jwt")
};
var data = new FormUrlEncodedContent(kvp);
var publicKey = await GetXXXXPublicKey();
_httpClient.DefaultRequestHeaders.Remove("X-XXXX-Public-Cert");
_httpClient.DefaultRequestHeaders.Add("X-XXXX-Public-Cert", publicKey);
var response = await _httpClient.PostAsync("Identity/token", data);
if (!response.IsSuccessStatusCode)
throw new Exception("XXXX Token Server Error: " + response.ReasonPhrase);
var result = await response.Content.ReadAsStringAsync();
var authResponse = JsonConvert.DeserializeObject<OAuthResponse>(result);
if (!string.IsNullOrEmpty(authResponse.access_token))
return authResponse.access_token;
System.Diagnostics.Trace.WriteLine("Token Exchange Result: " + result);
if (!string.IsNullOrEmpty(authResponse.error))
{
var outcome = new XXX.XXXX.Model.OperationOutcome();
outcome.Issue.Add(new XXX.XXXX.Model.OperationOutcome.IssueComponent()
{
//some code to throw an error is here
}
throw new XXX.XXXX.Rest.XXXXOperationException("Bearer Token Exchange failed", response.StatusCode);
}
Unfortunately none of the existing questions/advice anywhere on Stack Overflow, or the rest of the web, for this particular error seems to have helped. They are primarily about version discrepancies between client and server which seems not to be the case here as I am forcing TLS 1.2 (which is active and enabled on the failing machine).
Interestingly, I can visit the server URL in a browser via HTTPS just fine, which suggests there is something about my code that is the problem rather than the machine, but it works everywhere else.
I have confirmed that:
The certificate I am using to authenticate the connection on the machine is valid and has a chain of trust (though as above I don't think we are getting that far as the TLS connection itself is failing)
The server we are calling supports TLS 1.2 (by forcing it)
I can get to the website for the URL independently via the browser
Is there something I need to do either in the code or on the machine to get this call to work everywhere?
Things I have tried to resolve the issue
Installing all Windows 8.1 updates to present day
Forcing TLS 1.2 in the code (see above code sample)
Limiting VM to TLS 1.2 only
I might be able to at least point you in the right direction…
Same Symptoms
I had a .NET Core 3.1 web app running on IIS (Windows Server 2012 R2) that got the exact same error and stacktrace when it tried to connect to another server using TLS 1.2. I also had the symptom where I could connect with the browser (Chrome), but not with the app. (Would have been interesting to see if Internet Explorer browser worked though.)
Root Cause
The TLS handshake was failing because the two servers were unable to agree on a common cipher suite. (Using Wireshark, I discovered that when my app tried to connect it provided a more limited set of cipher suites than when the Chrome browser made the call.)
Solution
In my case, I used IIS Crypto (a small free tool: https://www.nartac.com/Products/IISCrypto/) to enable additional cipher suites on my web app's server. I downloaded and ran IIS Crypto, checkmarked additional cipher suites on its Cipher Suites tab, and then restarted the machine.
One of the new cipher suites worked with my app and the destination server, so the TLS handshake was successful and the error was resolved.
One quick caveat: Some cipher suites are more secure than others, so you'll want to read up on best practices.
Addendum
If you want to further diagnose the failure, I'd recommend installing Wireshark (another free tool: https://www.wireshark.org/#download) on the machine with your .NET Core app. If a TLS Handshake Failure is the issue, you will see a message like: Alert (Level: Fatal, Description: Handshake Failure)
This primer on wireshark output helped me:
https://blog.catchpoint.com/2017/05/12/dissecting-tls-using-wireshark/
I faced a simular issue, and in order to help others here's what I concluded:
Sucessfully executing this code doesn't mean that your application supports the specified protocol version, and the "SSL Error" can still occur later on when trying to establish a connection:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
In my case I was trying to force Tls13 and found out that my app configuration didn't actually support it:
net core 3.0 running on a Windows Server Datacenter 2019, version 1809
So I had to change my configuration to the following which provides support for the protocol version I needed:
Net framework 5.0 on a Windows Server Datacenter 2022, OS build 20348.288
I was trying to connect to an endpoint that suddenly dropped Tls 1.2 support (not sure why) and from then on only accepted Tls 1.3.
I am setting up an FTP Server (FileZilla) and using the netftp codeplex project System.Net.FtpClient to connect to it over SSL/TLS using an X509Certificate2.
I've downloaded FileZilla Server from sourceforge.
I've downloaded System.Net.FtpClient v14.06.17 from codeplex.
I think I have isolated the problem, but maybe someone has some additional thoughts as to the underlying cause. A workaround is provided at the bottom of this post.
An exception occurs calling client.GetListing() and client.GetNameListing(), however, uploading (OpenWrite) and downloading (OpenRead) both work fine. I've tested many versions of FileZilla to isolate what version it started breaking.
0.9.42 works (2013-12-16)
0.9.43 bad (2014-01-02)
0.9.46 bad (2014-08-03)
0.9.50 bad (2015-03-19) - current
The exception that occurs is:
outer message: A call to SSPI failed, see inner exception.
inner message: The message received was unexpected or badly formatted
The C# code sample that causes the error:
FtpClient client = new FtpClient();
client.Credentials = new NetworkCredential("blah", "blah123");
client.Host = "127.0.0.1";
client.Port = 21;
client.DataConnectionEncryption = true;
client.EncryptionMode = FtpEncryptionMode.Explicit;
//client.ClientCertificates.Add(cert); // tried both with and without
client.ValidateCertificate += (cli, e) => { e.Accept = true; };
client.Connect();
var list = client.GetNameListing(); // exception on this line
On the Codeplex Documentation page, jptrosclair wrote:
You do not use pem certificates, use p12 instead. See this Stack
Overflow thread for more information. If you get SPPI exceptions
with an inner exception about an unexpected or badly formatted
message, you are probably using the wrong type of certificate.
The link refers to asp.net, not FileZilla. My guess is that IIS doesn't support PEM and was most likely sending a bunch of gibberish through the stream. However, if you use a certificate format that your server supports, then there should be no problem. As far as I have read, FileZilla only supports PEM format (or CRT/CER which are still PEM format).
Also, this doesn't explain why OpenRead and OpenWrite succeed, but GetListing and GetNameListing fail.
Regardless, I still tested multiple different certificate generation tools, including FileZilla's built-in generator, C:\Program Files (x86)\Microsoft SDKs\Windows\v7.1A\Bin\makecert.exe, and OpenSSL (windows). The problem still persisted.
Here is a link similar discussion from skovachev https://netftp.codeplex.com/discussions/535815
His workaround was to clone the connection before getting the listing.
Workaround:
I've tested the following with all FileZilla versions listed above. The fix is to download the System.Net.FtpClient source code from codeplex and in the FtpSocketStream.cs class edit the following method:
public void ActivateEncryption(string targethost, X509CertificateCollection clientCerts)
change:
m_sslStream.AuthenticateAsClient(targethost, clientCerts, SslProtocols.Tls | SslProtocols.Ssl3 | SslProtocols.Ssl2, true);
to:
m_sslStream.AuthenticateAsClient(targethost, clientCerts, SslProtocols.Tls | SslProtocols.Ssl3, true);
(Basically just remove the Ssl2 flag).
My guess is something changed in FileZilla v0.9.43 when the Ssl2 flag is set when connecting to the server. It is strange that the first time AuthenticateAsClient is called it will succeed, but the second time it will fail.
As for uploading and downloading succeeding, the default EnableThreadSafeDataConnections is true, which causes the client to be cloned and reconnect to the server. However, GetListing and GetNameListing do not clone the connection. Putting a breakpoint in the ActivateEncryption method, the AuthenticateAsClient will be called multiple times. Maybe there is some connection pooling happening under the hood in the SslStream class and multiple AuthenticateAsClient are causing problems. It's worth noting that the FtpClient.ValidateCertificate event is only called once even though AuthenticateAsClient is called twice. This gives some evidence of some caching is happening.
Is there a way to get a System.Net.WebRequest or System.Net.WebClient to respect the hosts or lmhosts file?
For example: in my hosts file I have:
10.0.0.1 www.bing.com
When I try to load Bing in a browser (both IE and FF) it fails to load as expected.
Dns.GetHostAddresses("www.bing.com")[0]; // 10.0.0.1
WebRequest.Create("http://10.0.0.1").GetResponse(); // throws exception (expected)
WebRequest.Create("http://www.bing.com/").GetResponse(); // unexpectedly succeeds
Similarly:
WebClient wc = new WebClient();
wc.DownloadString("http://www.bing.com"); //succeeds
Why would System.Net.Dns respect the hosts file but System.Net.WebRequest ignore it? What do I need to change to make the WebRequest respect the hosts file?
Additional Info:
If I disable IPv6 and set my IPv4 DNS Server to 127.0.0.1, the above code works (fails) as expected. However if I add my normal DNS servers back as alternates, the unexpected behavior resumes.
I've reproduced this on 3 Win7 and 2 Vista boxes. The only constant is my company's network.
I'm using .NET 3.5 SP1 and VS2008
Edit
Per #Richard Beier's suggestion, I tried out System.Net tracing. With tracing ON the WebRequest fails as it should. However as soon as I turn tracing OFF the behavior reverts to the unexpected success. I have reproduced this on the same machines as before in both debug and release mode.
Edit 2
This turned out to be the company proxy giving us issues. Our solution was a custom proxy config script for our test machines that had "bing.com" point to DIRECT instead of the default proxy.
I think that #Hans Passant has spotted the issue here. It looks like you have a proxy setup in IE.
Dns.GetHostAddresses("www.bing.com")[0]; // 10.0.0.1
This works because you are asking the OS to get the IP addresses for www.bing.com
WebRequest.Create("http://www.bing.com/").GetResponse(); // unexpectedly succeeds
This works because you are asking the framework to fetch a path from a server name. The framework uses the same engine and settings that IE frontend uses and hence if your company has specified by a GPO that you use a company proxy server, it is that proxy server that resolves the IP address for www.bing.com rather than you.
WebRequest.Create("http://10.0.0.1").GetResponse(); // throws exception (expected)
This works/fails because you have asked the framework to fetch you a webpage from a specific server (by IP). Even if you do have a proxy set, this proxy will still not be able to connect to this IP address.
I hope that this helps.
Jonathan
I'm using VS 2010 on Windows 7, and I can't reproduce this. I made the same hosts-file change and ran the following code:
Console.WriteLine(Dns.GetHostAddresses("www.bing.com")[0]); // 10.0.0.1
var response = WebRequest.Create("http://www.bing.com/").GetResponse(); // * * *
Console.WriteLine(new StreamReader(response.GetResponseStream()).ReadToEnd());
I got an exception on the line marked "* * *". Here's the exception detail:
System.Net.WebException was unhandled
Message=Unable to connect to the remote server
Source=System
StackTrace:
at System.Net.HttpWebRequest.GetResponse()
at ConsoleApplication2.Program.Main(String[] args) in c:\Data\Projects\ConsoleApplication2\ConsoleApplication2\Program.cs:line 17
InnerException: System.Net.Sockets.SocketException
Message=A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.0.0.1:80
Source=System
ErrorCode=10060
Maybe it's an issue with an earlier .NET version, that's now fixed in .NET 4 / VS 2010? Which version of .NET are you using?
I also found this thread from 2007, where someone else ran into the same problem. There are some good suggestions there, including the following:
Turn on system.net tracing
Work around the problem by using Dns.GetHostAddresses() to resolve it to an IP. Then put the IP in the URL - e.g. "http://10.0.0.1/". That may not be an option for you though.
In the above thread, mariyaatanasova_msft also says: "HttpWebRequest uses Dns.GetHostEntry to resolve the host, so you may get a different result from Dns.GetHostAddresses".
You should overwrite the default proxy.
HttpWebRequest & WebRequest will set a default proxy if present in Internet Explorer and your file hosts will be bypassed.
request.Proxy = new WebProxy();
The following is just an example of code:
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("www.bing.com");
request.Proxy = new WebProxy();
request.Method = "POST";
request.AllowAutoRedirect = false;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
//some code here
}
}
catch (exception e)
{
//Some other code here
}