Develop Reference

Unity TlsException: Handshake failed UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED

I'm trying to update my application's TcpClient to use TLS with SslStream instead of the normal Stream, the code i'm using for this seems to work outside of Unity, but fails when integrated in my Unity 2019.1.8 (tested on 2018 and 2017 as well) project.
To establish a connection and open a new SslStream I use the following code:
public static void InitClient(string hostName, int port, string certificateName)
{
client = new TcpClient(hostName, port);
if (client.Client.Connected)
{
Debug.LogFormat("Client connected succesfully");
}
else
{
Debug.LogErrorFormat("Client couldn't connect");
return;
}
stream = new SslStream(client.GetStream(), false, new RemoteCertificateValidationCallback(ValidateServerCertificate), null);
try
{
stream.AuthenticateAsClient(certificateName);
}
catch (AuthenticationException e)
{
Debug.LogErrorFormat("Error authenticating: {0}", e);
if (e.InnerException != null)
{
Debug.LogErrorFormat("Inner exception: {0}", e);
}
Debug.LogErrorFormat("Authentication failed - closing connection");
stream.Close();
client.Close();
}
}
And for validating the certificate
public static bool ValidateServerCertificate(object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors)
{
if (sslPolicyErrors == SslPolicyErrors.None)
return true;
Debug.LogErrorFormat("Certificate error: {0}", sslPolicyErrors);
return false;
}
In Unity 2019.1.8 the client connects and will attempt to validate the remote certificate, which fails with the error TlsException: Handshake failed - error code: UNITYTLS_INTERNAL_ERROR, verify result: UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED.
Making ValidateServerCertificate always return true lets my client connect without issue.
I tried replicating the issue in a standalone C# Console Application targeting .net framework 4.7.1 using the exact same code. Launching the client in this application will return true from ValidateServerCertificate from the sslPolicyErrors == SslPolicyErrors.None check.
I know that the certificate is a valid cert, issued by a trusted CA (as verified by the fact that the cert is accepted from a console app, and it having a padlock in browsers).
Why does the validation fail in Unity, but nowhere else?

Why Unity can't validate option 1:
Although the certificate is valid and correct (e.g. it works when using it in a web browser), it did not include the intermediate certificates chain to the root CA. This has as a result that no chain of trust can be formed (Unity doesn't cache/retrieve intermediates), resulting in the UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED flag being set.
To fix this I needed to append the certificate chain to my leaf certificate so that Unity can verify the entire chain up to the root CA. To find the certificate chain for your certificate you can use a "TLS certificate chain composer".
Depending on what software you use you may either need to include the chain in your certificate, or keep it in a seperate file. From whatsmychaincert (i'm in no way affiliated to this site, merely used it):
Note: some software requires you to put your site's certificate (e.g. example.com.crt) and your chain certificates (e.g. example.com.chain.crt) in separate files, while other software requires you to put your chain certificates after your site's certificate in the same file.
Why Unity can't validate option 2:
When your server's SSL certificate has expired Unity will throw the exact same UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED error without additional information that the certificate has been expired, so make sure the "Valid to" date is in the future (this would also cause the certificate to be denied when used in a web browser).
Why a browser/Console app can validate:
Software can have different implementation on how it deals with incomplete chains. It can either throw an error, stating that the chain is broken and can thus not be trusted (as is the case with Unity), or cache and save the intermediates for later use/retrieve it from previous sessions (as browsers and Microsoft's .net(core) do).
As explained in this answer (emphasis mine)
In general, SSL/TLS clients will try to validate the server certificate chain as received from the server. If that chain does not please the client, then the client's behaviour depends on the implementation: some clients simply give up; others (especially Windows/Internet Explorer) will try to build another chain using locally known intermediate CA and also downloading certificates from URL found in other certificates (the "authority information access" extension).
Not trusting the certificate when no chain is provided is a deliberate decision made by Unity due to cross-platform compatability, as stated in an [answer to issue 1115214][3]:
We may be able to solve this issue by doing a verification via the system specific TLS api instead of using OpenSSL/MbedTLS to validate against root certificates as we do today, however this solution would then not work cross-platform. So we don't want to implement it today, as it would hide the misconfigured server from the user on some but not all platforms.
I figured out this was a solution to my particular situation while asking this question, so decided to self-answer it for future references. However UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED can have all kind of causes, this just being one of them.

For anyone stumbling across this who's using UnityWebRequest and Let's Encrypt:
There is a known issue on Unity's Issue Tracker where you will find it's been fixed in only certain versions:
Fixed in 2020.1.0a11
Fixed in 2019.3.3f1
Fixed in 2018.4.18f1
Changing Unity versions for an existing project is not a trivial task so we opted to buy a legit SSL cert which, although not cheap, was the simplest solution.

My solution was:
> Public class ForceAcceptAll : CertificateHandler {
> protected override bool ValidateCertificate(byte[] certificateData)
> {
> return true;
> } }
//-------
var cert = new ForceAcceptAll();
// www is a UnityWebRequest
www.certificateHandler = cert;
Nota: if used incorrectly (and found by the Google, your build might get rejected).
So use it with caution.
more information here.

Root Certificate Pinning in C#/.NET

I want to implement certificate/public key pinning in my C# application. I already saw a lot of solutions that pin the certificate of the server directly as e.g. in this question. However, to be more flexible I want to pin the root certificate only. The certificate the server gets in the setup is signed by an intermediate CA which itself is signed by the root.
What I implemented so far is a server that loads its own certificate, the private key, intermediate certificate, and the root certificate from an PKCS#12 (.pfx) file. I created the file using the following command:
openssl pkcs12 -export -inkey privkey.pem -in server_cert.pem -certfile chain.pem -out outfile.pfx
The chain.pem file contains the root and intermediate certificate.
The server loads this certificate and wants to authenticate itself against the client:
// certPath is the path to the .pfx file created before
var cert = new X509Certificate2(certPath, certPass)
var clientSocket = Socket.Accept();
var sslStream = new SslStream(
new NetworkStream(clientSocket),
false
);
try {
sslStream.AuthenticateAsServer(cert, false, SslProtocols.Tls12, false);
} catch(Exception) {
// Error during authentication
}
Now, the client wants to authenticate the server:
public void Connect() {
var con = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
con.Connect(new IPEndPoint(this.address, this.port));
var sslStream = new SslStream(
new NetworkStream(con),
false,
new RemoteCertificateValidationCallback(ValidateServerCertificate),
null
);
sslStream.AuthenticateAsClient("serverCN");
}
public static bool ValidateServerCertificate(
object sender,
X509Certificate certificate,
X509Chain chain,
SslPolicyErrors sslPolicyErrors
)
{
// ??
}
The problem now is that the server only sends its own certificate to the client. Also the chain parameter does not contain further information.
This is somehow plausible as the X509Certificate2 cert (in the server code) only contains the server certificate and no information about the intermediate or root certificate. However, the client is not able to validate the whole chain as (at least) the intermediate certificate is missing.
So far, I did not find any possiblity to make .NET send the whole certificate chain, but I do not want to pin the server certificate iself or the intermediate one as this destroys the flexibility of root certificate pinning.
Therefore, does anyone know a possibility to make SslStream sending the whole chain for authentication or implement the functionality using another approach? Or do I have to pack the certificates differently?
Thanks!
Edit:
I made some other tests to detect the problem. As suggested in the comments, I created a X509Store that contains all the certificates. After that, I built a X509Chain using my server's certificate and the store. On the server itself, the new chain contains all the certificates correctly, but not in the ValidateServerCertificate function..

SslStream will never send the whole chain (except for self-issued certificates). The convention is to send everything except for the root, because the other side either already has and trusts the root or doesn't have (thus/or doesn't trust the root), and either way it was a waste of bandwidth.
But SslStream can only send the intermediates when it understands the intermediates.
var cert = new X509Certificate2(certPath, certPass);
This only extracts the end-entity certificate (the one with the private key), it discards any other certificates in the PFX. If you want to load all of the certificates you need to use X509Certificate2Collection.Import. But... that doesn't really help you either. SslStream only accepts the end-entity certificate, it expects the system to be able to build a functioning chain for it.
In order to build a functioning chain, your intermediate and root certificates need to be in any of:
Provided as manual input via X509Chain.ChainPolicy.ExtraStore
Since the chain in question is built by SslStream you can't really do this here.
CurrentUser\My X509Store
*LocalMachine\My X509Store
CurrentUser\CA X509Store
**LocalMachine\CA X509Store
CurrentUser\Root X509Store
**LocalMachine\Root X509Store
*LocalMachine\ThirdPartyRoot X509Store
An http (not-s) location identified in an Authority Access Identifier extension in the certificate.
The stores marked with * don't exist on .NET Core on Linux. The stores marked with ** do exist on Linux, but cannot be modified by a .NET application.
That's also not quite sufficient, because (at least for SslStream on Linux and probably macOS on .NET Core) it still only sends the intermediates if it built a chain it trusted. So the server needs to actually trust the root certificate for it to send the intermediates. (Or the client needs to trust the root for the client cert)
On the other side, the same rules apply. The difference is that in the callback you can choose to rebuild the chain to add the extra certificates.
private static bool IsExpectedRootPin(X509Chain chain)
{
X509Certificate2 lastCert = chain.ChainElements[chain.ChainElements.Count - 1].Certificate;
return lastCert.RawBytes.SequenceEquals(s_pinnedRootBytes);
}
private static bool ValidateServerCertificate(
object sender,
X509Certificate certificate,
X509Chain chain,
SslPolicyErrors sslPolicyErrors
)
{
if ((sslPolicyErrors & ~SslPolicyErrors.RemoteCertificateChainErrors) != 0)
{
// No cert, or name mismatch (or any future errors)
return false;
}
if (IsExpectedRootPin(chain))
{
return true;
}
chain.ChainPolicy.ExtraStore.Add(s_intermediateCert);
chain.ChainPolicy.ExtraStore.Add(s_pinnedRoot);
chain.ChainPolicy.VerificationFlags |= X509VerificationFlags.AllowUnknownCertificateAuthority;
if (chain.Build(chain.ChainElements[0].Certificate))
{
return IsExpectedRootPin(chain);
}
return false;
}
Of course, the problem with this approach is that you need to also understand and provide the intermediate on the remote side. The real solution to this is that the intermediates should be available on an HTTP distribution endpoint and the issued certificates should carry the Authority Information Access extension to be able to locate them dynamically.

RabbitMq Broker not accepting PFX

After having a working solution of RabbitMq over SSL using the certs generated by OpenSSL, I was given a requirement from our client that we cannot use OpenSSL to generate the certs. The server admin went and created the certificate chain via IIS.
We did the following to the certs after creation:
exported the server certs to a PFX and extracted cert.pem and cert.key
exported the client cert to a PFX and extracted cert.pem and cert.key
exported the Root CA to rootca.pem
updated the config file for RabbitMq
To test the connections and certificates were created properly, we went through the Troubleshooting Steps here.
Everything passed fine. We are able to connect using openssl s_client on port 5671 as expected using the client's cert.pem and cert.key generated from client.pfx. We can see the connection made in management console, as well as in the logs, and they communicate back and forth.
Now, using the client.pfx in the .NET client, we are getting an error about the certificate validation:
MassTransit.RabbitMqTransport.RabbitMqConnectionException: Connect failed: admin#mitvs-atm01:5671/ ---> RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
How can I be getting an error using the client.pfx but not get an error when using the cert.pem and cert.key that I extracted from it?

This error message means that client can't validate server certificate. Your problem is likely to do with rootca.pem not being trusted on your client's machine. Follow the appropriate OS guide to make the root CA certificate trusted.

When testing the connections through OpenSSL using s_client, it performs no or very basic verification and .NET apps can provide a function that applies any logic. The most common verification type is matching certificate's CN against server's hostname but it really can be anything.
So, even on a trusted Root CA, the CN still needs to match the hostname, which isn't picked up using OpenSSL commands.
Regenerating the cert and amending that fixed the problem.
Also, you could effectively add the code below and it will disregard this error:
h.UseSsl(s =>
{
s.ServerName = SslHostName;
s.CertificatePath = SslFileLocation;
s.CertificatePassphrase = SslPassphrase;
// The below allows the Root CA's CN to be different than the others, but adds vulnerability
s.AllowPolicyErrors(System.Net.Security.SslPolicyErrors.RemoteCertificateNameMismatch);
});
Please note - Ignoring errors carries the risk of exposing your system to MITM attacks (unless you implement your own verification logic that's not based on CN/hostname matching, of course).

How to validate server SSL certificate for LDAP+SSL connection

Our application works with Active Directory users and groups. We are using LDAP on port 389 for Active Directory operations. Now, one of our clients want us add an option for using LDAP + SSL for Active Directory communication.
They told us that they have a local CA installed on their domain and using self signed certificate for LDAPS. They also told us that they will provide the certificate, no mutual-trust needed and we should use Windows certificate store.
I have developed a test application for LDAP+SSL operations and saw that server sends its certificate when a client initiates an LDAP+SSL connection. I can establish the connection only by returning true from the server certificate verification method.
The questions are;
- Which certificate (root, the ceritificate used for LDAP+SSL...) should the customer give us?
What should be the format of the certificate for working on .Net environment?
How should I verify the server's certificate when connecting the server?
What they mean by "we should use Windows certificate store"? Do they want us add the server's certificate automatically to trusted certificate store of the local machine?
Sample code I used for LDAP+SSL connection,
LdapConnection _connection = new LdapConnection(new LdapDirectoryIdentifier(m_DomainName, m_PortNo));
_connection.Timeout = TimeSpan.FromMinutes(10);
_connection.AuthType = AuthType.Basic;
_connection.Credential = new NetworkCredential(m_UserName, m_Password);
_connection.SessionOptions.ProtocolVersion = 3;
_connection.SessionOptions.SecureSocketLayer = true;
_connection.SessionOptions.VerifyServerCertificate = (ldapCon, serverCertificate) =>
{
//TODO: Verify server certificate
return true;
};
_connection.SessionOptions.QueryClientCertificate = (con, trustedCAs) => null;
_connection.Bind();

Which certificate (root, the ceritificate used for LDAP+SSL...) should the customer give us?
The root certificate that signed the LDAP server cert. They can also give you the whole chain in advance, but that will be sent during TLS handshake anyway. You only need to have the root cert in advance.
What should be the format of the certificate for working on .Net environment?
Anything that you can import into certmgr.msc. Pfx is the usual choice on Windows.
How should I verify the server's certificate when connecting the server?
You should not write validation yourself. Certificate validation is tricky business, and it's already done for you. Use the built-in stuff (also see below).
What they mean by "we should use Windows certificate store"? Do they want us add the server's certificate automatically to trusted certificate store of the local machine?
Yes. They send you the root cert they used for signing the ldap server cert, which you can then import as a trusted root. Once this is done, you don't need to do any manual validation, it will just work™ :) with valid certificates and will not work with invalid ones.
Note that once you add their root cert as trusted, they can forge any server certificate for the client their root is installed on, and anything they sign will be considered valid on that client.
Bonus: adding semi-custom validation and debugging certificate errors
One problem that you may face is that error messages are not very helpful. If the certificate cannot be validated, you will get a very generic error message that has no hint about the actual problem. You may want to hook into the validation process for other reasons too.
For this purpose, you can define your own validation:
private bool VerifyServerCertificate(LdapConnection ldapConnection, X509Certificate certificate)
{
X509Certificate2 certificate2 = new X509Certificate2( certificate );
return certificate2.Verify();
}
And then add it to the ldap connection:
_connection.SessionOptions.VerifyServerCertificate =
new VerifyServerCertificateCallback( VerifyServerCertificate );
This way you can catch exceptions on Verify() etc. But again, if the certificate is valid (can be verified by the client), this is not strictly needed, it's done automatically anyway. You only need this if you want something not implemented, like for example you could just return true in VerifyServerCertificate to accept any cert including the invalid ones (this would be a Bad Idea and makes a secure connection useless, but may be good for debugging, etc).
Another thing you could implement in this method is certificate pinning for additional security, but that's beyond the scope of this answer.

BTW: Since introduction of the StartTLS extended operation for LDAP v3 in May 2000 (RFC 2830) LDAPS (made for LDAP v2) is deprecated.

SSL Socket between .Net and Java with client authentication

I am trying to create an SSL Socket Server/Client between .NET and Java. In this case, my SSL Socket Server will run in .net and the client runs in Java under Linux. My problem is that the connection fails during the handshaking, specifically when the server request a certificate from the client, the client is unable to send something back and the connection fails.
In .net I am using sslStream to establish the connection and on Java I am using the standard SSLSocket. Some code snippets are below, but this is what I have so far:
On the server side (Windows), I have a private certificate in the Personal/Certificates folders under MMC. I have a public certificate from the client in the Trusted People/Certificates. Both certificates were issued by the same CA. The certificate chain for both certificates have multiple levels, but it is the same for both. The root level certificate in the chain is also installed in the trusted Certification Authorities/Certificates folder.
On the client side (Linux), I have a keystore that contains the private certificate that matches the public certificate installed at the server. I have a trust store that contains the public certificate from the server, matching the server's private one.
On the server side (.net) I am using a Socket that does an asynchronous read and then it gets wrapped into an SSLStream, the code snippet is like this:
NetworkStream ns = new NetworkStream(socket, false);
SslStream ssl = new SslStream(ns, true);
ssl.AuthenticateAsServer(serverCertificate, true, SslProtocols.Default, true);
The client code is pretty much standard code:
SSLSocketFactory factory = (SSLSocketFactory) SSLSocketFactory.getDefault();
InetAddress addr = InetAddress.getByName(servername);
SSLSocket socket = (SSLSocket) factory.createSocket(addr,port);
socket.setUseClientMode(true);
socket.setNeedClientAuth(true);
socket.setWantClientAuth(true);
socket.startHandshake();
os = new DataOutputStream(socket.getOutputStream());
is = new DataInputStream(socket.getInputStream());
byte[] outBuf = new byte[50];
os.write("SEND SOMETHING".getBytes("UTF-8"));
is.read(outBuf);
In java I have set the proper varialbes to point to the trust and key store with their password.
Now, following the standard SSL Handshake, this is what happens:
ClientHello
ServerHello
Server sends public certificate
Client matches the public certificate with the one on the trust store
Server sends the Certificate request
With the certificate request the server sends a list of valid CAs, on this list only the my root CA is sent (among a long list of other well known CAs.).
Client certificate is null.
Server receives a null certificate from the client, thus closes the connection.
And that is it, the client won't send a valid certificate back to the server. I have some questions on this:
Has anybody experienced something like this?
Regarding that list of CAs sent by the server (Windows), How does .net determine what to send to the client? Is there a way to modify that list?
Do I need to send the all the authorities in the chain used to sign my certificate in that list of CAs? or is the Root one enough?
Am I missing something on either side of my code?
Any help will be greatly appreciated it.
In

The following two statements are useless on the client side (although they shouldn't hurt):
socket.setNeedClientAuth(true);
socket.setWantClientAuth(true);
The fact that you see the Certificate Request message and the Client Certificate message shows that the server is configured properly.
The most likely cause that comes to mind for the absence of certificate in the client certificate message is that the keystore (on the client side) might not be configured properly. You may be interested in this answer to make sure that your client key store is configured properly. More specifically, you need to make sure that the private key for your client certificate was imported in the same alias as the certificate chain (and that it's the chain going back to a CA advertised in the Certificate Request message).
(Regarding the rest of your question, I'm not sure how to modify the CA list sent by the server when using SslStream in C#. This earlier question would seem to suggest there is no solution, although newer versions of .Net may have addresses the issue since this question was asked. I haven't been able to find anything that would do it by looking at the SslStream API documentation and related classes, but this doesn't mean it doesn't exist.)