Develop Reference

Two way SSL - X509Certificate2.CreateFromPemFile - WINHTTP_CALLBACK_STATUS_REQUEST_ERROR

I am trying to call an API which requires two way SSL (TLS 1.2) and I have the following code:
var myCert = X509Certificate2.CreateFromPemFile(_publicFilename, _privateFilename);
using (var handler = new WinHttpHandler())
{
handler.ClientCertificateOption = ClientCertificateOption.Manual;
handler.ServerCertificateValidationCallback = (x, y, z, w) => true;
handler.ClientCertificates.Add(myCert);
handler.SslProtocols = SslProtocols.Tls12;
using (var client = new HttpClient(handler))
{
var postData = new StringContent("", UnicodeEncoding.UTF8, "application/json");
var response = await client.PostAsync("<API Endpoint>", postData);
string responseString = await response.Content.ReadAsStringAsync();
Console.WriteLine(responseString);
}
}
However, when I call that I get the following WinHttp exception:
Error 12185 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'No
credentials were available in the client certificate.'.
Also in the windows event viewer I get the following error message:
The TLS client credential's certificate does not have a private key
information property attached to it. This most often occurs when a
certificate is backed up incorrectly and then later restored. This
message can also indicate a certificate enrollment failure.
Unable to figure out what the issue is. I am using the exact same certificate and key in postman and it works fine.

TLS on Windows (WinHttpHandler, SslStream, or the default HTTP handler (which uses SslStream)) requires that the certificate have a named private key.
You can do that on a temporary basis by importing a PFX without PersistKeySet... but how do you get a PFX? Well, pretty easy.
var myCert = X509Certificate2.CreateFromPemFile(_publicFilename, _privateFilename);
using (var tmpCert = new X509Certificate2(myCert.Export(X509ContentType.Pfx)))
using (var handler = new WinHttpHandler())
{
...
handler.ClientCertificates.Add(tmpCert);
...
}
The named key will be deleted when tmpCert gets Disposed. If you have complex or long life, you can create the cert not in a using statement. If the cert gets garbage collected and the process stays active long enough to run the finalizers the key will get cleaned up then.

Why does Request Sync on HomeGraph API return 403 Forbidden?

Problem
When I call "Request Sync" on the Google HomeGraph API I receive a "403 Forbidden" response.
Background
I'm writing a Smart Home Action, and have successfully implemented SYNC, QUERY and EXECUTE. Testing on my mobile I can see and interact with devices okay. I'm now trying to implement Request Sync, but can't appear to interact with the API. I am making what seems to be successful requests for an Access Token. The token always begins with "ya29.c." which in my naïve understanding suggests an empty header and payload (trying it on https://jwt.io). However, when testing it at https://accounts.google.com/o/oauth2/tokeninfo?access_token= it appears valid, showing both my service account unique ID and the scope I intended. When I make a call to the API, either manually posting the data, or via Google's own code, it gives me a blunt 403 error. I do not know where I can get any more information on this error other than the exception objects. I'm new to GCP and couldn't find any sort of log. Given I've tried different methods and all return a 403 I'm inclined to suspect the issue is more a account or credential-related than the code, but can't be certain.
API Key
(I'm no longer able to reproduce any errors relating to API keys being missing or invalid).
Although the documentation doesn't show it, I've seen some people use an API key. When I don't include the API key with a p12 certificate, or include an incorrect one it errors (either with API key missing, or API key invalid accordingly). I have created an unrestricted API key in IAM, and am using that. I can't appear to explicitly relate this to HomeGraph API, but it says that it can call any API.
Code
This example fetches an access token, then tries to call the API via POST with and without the API key. It then tries to authenticate and call the API via the Google library code. Each fails with a 403.
using Google;
using Google.Apis.Auth.OAuth2;
using Google.Apis.HomeGraphService.v1;
using Google.Apis.HomeGraphService.v1.Data;
using Google.Apis.Services;
using Lambda.Core.Constants;
using System.IO;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Security.Cryptography.X509Certificates;
using System.Text;
using static Google.Apis.HomeGraphService.v1.DevicesResource;
public class Example
{
public void RequestSync()
{
const string UrlWithoutKey = #"https://homegraph.googleapis.com/v1/devices:requestSync";
const string UrlWithKey = #"https://homegraph.googleapis.com/v1/devices:requestSync?key=" + OAuthConstants.GoogleApiKey;
string accessToken = this.GetAccessToken();
// Manual Attempt 1
try
{
string response = this.CallRequestSyncApiManually(accessToken, UrlWithoutKey);
}
catch (WebException ex)
{
// Receive 403, Forbidden
string msg = ex.Message;
}
// Manual Attempt 2
try
{
string response = this.CallRequestSyncApiManually(accessToken, UrlWithKey);
}
catch (WebException ex)
{
// Receive 403, Forbidden
string msg = ex.Message;
}
// SDK Attempt
try
{
this.CallRequestSyncApiWithSdk();
}
catch (GoogleApiException ex)
{
// Google.Apis.Requests.RequestError
// The caller does not have permission[403]
// Errors[Message[The caller does not have permission] Location[- ] Reason[forbidden] Domain[global]]
// at Google.Apis.Requests.ClientServiceRequest`1.ParseResponse(HttpResponseMessage response) in Src\Support\Google.Apis\Requests\ClientServiceRequest.cs:line 243
// at Google.Apis.Requests.ClientServiceRequest`1.Execute() in Src\Support\Google.Apis\Requests\ClientServiceRequest.cs:line 167
string msg = ex.Message;
}
}
private string GetAccessToken()
{
string defaultScope = "https://www.googleapis.com/auth/homegraph";
string serviceAccount = OAuthConstants.GoogleServiceAccountEmail; // "??????#??????.iam.gserviceaccount.com"
string certificateFile = OAuthConstants.CertificateFileName; // "??????.p12"
var oAuth2 = new GoogleOAuth2(defaultScope, serviceAccount, certificateFile); // As per https://stackoverflow.com/questions/26478694/how-to-produce-jwt-with-google-oauth2-compatible-algorithm-rsa-sha-256-using-sys
bool status = oAuth2.RequestAccessTokenAsync().Result;
// This access token at a glance appears invalid due to an empty header and payload,
// But verifies ok when tested here: https://accounts.google.com/o/oauth2/tokeninfo?access_token=
return oAuth2.AccessToken;
}
private string CallRequestSyncApiManually(string accessToken, string url)
{
string apiRequestBody = #"{""agentUserId"": """ + OAuthConstants.TestAgentUserId + #"""}";
var client = new HttpClient();
var request = (HttpWebRequest)WebRequest.Create(url);
var data = Encoding.ASCII.GetBytes(apiRequestBody);
request.Method = "POST";
request.Accept = "application/json";
request.ContentType = "application/json";
request.ContentLength = data.Length;
request.Headers.Add("Authorization", $"Bearer {accessToken}");
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);
using (var stream = request.GetRequestStream())
{
stream.Write(data, 0, data.Length);
}
var response = (HttpWebResponse)request.GetResponse();
var responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();
return responseString;
}
private void CallRequestSyncApiWithSdk()
{
var certificate = new X509Certificate2(OAuthConstants.CertificateFileName, OAuthConstants.CertSecret, X509KeyStorageFlags.Exportable);
var credential = new ServiceAccountCredential(
new ServiceAccountCredential.Initializer(OAuthConstants.GoogleServiceAccountEmail)
{
Scopes = new[] { "https://www.googleapis.com/auth/homegraph" },
}.FromCertificate(certificate));
var service = new HomeGraphServiceService(
new BaseClientService.Initializer()
{
// Complains if API key is not provided, even though we're using a certificate from a Service Account
ApiKey = OAuthConstants.GoogleApiKey,
HttpClientInitializer = credential,
ApplicationName = OAuthConstants.ApplicationName,
});
var request = new RequestSyncRequest(
service,
new RequestSyncDevicesRequest
{
AgentUserId = OAuthConstants.TestAgentUserId
});
request.Execute();
}
}
Account Configuration
Account screenshots. (I'm not allowed to post images yet, so they're links)
HomeGraph is enabled
My API Key is unrestricted
My Service Account has Owner & Service Account Token Creator enabled
Updates
I have tried skipping manually obtaining the access token, as per Devunwired's suggestion. Whilst this does eliminate the error I was getting from not providing the API key, I still end up with the 403. My reasoning for doing the access token part manually was part of debugging a 403 I was getting with the API call. That way I could at least see part of the process working. I'm happy to use the library version for the solution as the access token doesn't appear to be the issue.
public void GoogleLibraryJsonCredentialExample()
{
try
{
GoogleCredential credential;
using (var stream = new FileStream(OAuthConstants.JsonCredentialsFileName, FileMode.Open, FileAccess.Read))
{
credential = GoogleCredential.FromStream(stream).CreateScoped(new[] { OAuthConstants.GoogleScope });
}
var service = new HomeGraphServiceService(
new BaseClientService.Initializer()
{
HttpClientInitializer = credential,
ApplicationName = OAuthConstants.ApplicationName,
});
var request = new RequestSyncRequest(
service,
new RequestSyncDevicesRequest
{
AgentUserId = OAuthConstants.TestAgentUserId
});
request.Execute();
}
catch (Exception ex)
{
// Receive 403, Forbidden
string msg = ex.Message;
}
}
Concerns
Is it possible that I need to be making the API call from a verified or white-listed domain? At the moment I'm running it from a console app running on my development machine. My understanding of domain verification is that it does not apply to incoming calls, and therefore shouldn't be the problem.

I am making what seems to be successful requests for an Access Token.
You should not need to manually request OAuth access tokens when using the Google client libraries. They generally handle this process internally using the credentials you provide from the GCP console.
Although the documentation doesn't show it, I've seen some people use an API key. Indeed, it is mandatory to include it for the SDK approach.
We do not recommend using the API key method to access the Home Graph API. You should be using service account credentials. API keys will technically work for the Request Sync method, but you will not be able to authenticate Report State using an API key.
The fact that you are receiving an error trying to build the HomeGraphServiceService without an API key may be indicative that the credential you are using isn't set up correctly (no private key or possibly missing scopes). The recommended method for supplying service account credentials is to download them in the JSON format rather than certificate, and the code to generate a credential from JSON should look something like this:
GoogleCredential credential;
using (var stream = new FileStream(serviceAccountCredentialFilePath, FileMode.Open, FileAccess.Read))
{
credential = GoogleCredential.FromStream(stream).CreateScoped(scopes);
}
You can find additional C# examples for authenticating APIs in the authentication guide.

The problem was nothing to do with my permission to talk to the HomeGraph API or that user. Instead it was where HomeGraph wanted to call my Smart Home Action, but the access token had expired. When attempting to refresh the token, an erroneous implementation on my part led to a blunt 403, which Google was then relaying back to me.
For those interested, the issue was that rather than omitting the expiry date for a token that should never expire, I was setting it to DateTime.MaxValue (subsequently sent through some further processing). Unfortunately when this is finally cast to an int, it is a value that exceeds int.Max. The subsequent time on the expiry was set to epoch (i.e. in the past), and therefore the token validation failed due to expiry.
For anyone else still having the same issue, double check your agentUserId matches exactly the value shown in your SYNC output payload. In my case I'd checked this.
Many thanks to anyone who's looked at this.

How to use Firebase REST Streaming with Hammock in Windows Phone 8?

I'm working with the Nest API, which supports REST Streaming via Firebase. I have REST working, however I cannot get it to stream correctly. This is very important for my app, and REST just isn't effective for what I want to do.
I'm using Hammock for the requests, and here's the code:
public class NestAPI
{
private RestClient client { get; set; }
public NestAPI()
{
this.client = new RestClient();
this.client.Authority = "https://developer-api.nest.com/";
this.client.HasElevatedPermissions = true;
}
public void BeginStreaming()
{
RestRequest request = new RestRequest();
request.AddParameter("auth", App.accessToken);
request.RetryPolicy = new RetryPolicy() { RetryCount = 3 };
//Enables streaming
//request.AddHeader("Accept", "text/event-stream");
//request.StreamOptions = new StreamOptions() { Duration = new TimeSpan(96, 0, 0), ResultsPerCallback = 1 };
this.client.BeginRequest<object>(request, new RestCallback<object>(this.StreamCompletedEvent));
}
private void StreamCompletedEvent(RestRequest request, RestResponse<object> response, object userState)
{
//TO DO: check for errors first
string json = response.Content;
}
public void EndStreaming()
{
this.client.CancelStreaming();
}
}
This code works and does return JSON, however I can't seem to enable streaming. When I uncomment the lines below "Enables streaming", the callback event never fires. It's important to note that authentication is done using the uri parameter, "auth".
Unfortunately, there doesn't seem to be Firebase libraries available, and REST is my only option. I want to know when JSON properties change and want to set different values while streaming.

I'm not familiar with Hammock, but can you make sure that it's set to follow redirects? The streaming endpoint typically issues HTTP 307 to get inform the client of the correct server to connect to.

I've never used Hammock, but looking through source code (briefly) it appears you need to set it up as a streaming request with StreamOptions. Twitter has some open source that uses this here https://github.com/camertron/twitter-windows/blob/master/Source/Twitter/Classes/API/Streaming/UserStream.cs.
The way you have Hammock configured here it's waiting for an entire request to complete before calling your callback. This will (almost) never happen with a streaming request as the server keeps the connection open to push new results.

setExpressCheckout and SSL/TLS error

I'm trying to develop a simple application that will enable users to purchase services off a website through the Paypal API. This application is running on ASP.NET with C#.
I have had very little luck trying to get the Paypal API to co-operate. The method I'm calling is SetExpressCheckout with all the appropriate variables.
I did my research and discovered that since I'm testing in Localhost, it may affect Paypal's ability to communicate with the application. So the next thing I tried was accessing my application through an open port and a publicly accessible IP address, but the same error occurs on the call to SetExpressCheckout.
Here is the error:
Exception Details: System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.
Source Error:
Line 1790: [return: System.Xml.Serialization.XmlElementAttribute("SetExpressCheckoutResponse", Namespace="urn:ebay:api:PayPalAPI")]
Line 1791: public SetExpressCheckoutResponseType SetExpressCheckout([System.Xml.Serialization.XmlElementAttribute(Namespace="urn:ebay:api:PayPalAPI")] SetExpressCheckoutReq SetExpressCheckoutReq) {
Line 1792: object[] results = this.Invoke("SetExpressCheckout", new object[] {
Line 1793: SetExpressCheckoutReq});
Line 1794: return ((SetExpressCheckoutResponseType)(results[0]));
Source File: c:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\anan_p2\730602d6\31a8d74e\App_WebReferences.c8vgyrf8.2.cs Line: 1792
I've also tried generating certificates using OpenSSL and uploading them to the Paypal account's encrypted seller option but still no effect.
Thank you very much for reading through my question!
Update: As requested here is the code being used.
String hostingOn = ConfigurationManager.AppSettings["default_site_url"];
reqDetails.ReturnURL = hostingOn + "marketplace_confirm.aspx";
reqDetails.CancelURL = hostingOn + "marketplace.aspx";
reqDetails.NoShipping = "1";
reqDetails.ReqConfirmShipping = "0";
reqDetails.OrderTotal = new BasicAmountType()
{
currencyID = CurrencyCodeType.CAD,
Value = payment_amt.Value,
};
SetExpressCheckoutReq req = new SetExpressCheckoutReq()
{
SetExpressCheckoutRequest = new SetExpressCheckoutRequestType()
{
Version = UtilPayPalAPI.Version,
SetExpressCheckoutRequestDetails = reqDetails
}
};
PayPalAPIAASoapBinding paypal = new PayPalAPIAASoapBinding();
paypal.SetExpressCheckout(req);
I am also using the https://api-aa-3t.paypal.com/2.0/ url for accessing the API

Since early 2016, Paypal started requiring TLS 1.2 protocol for communications in the Sandbox, and will enforce it for the live environment starting June 17. See here for reference.
In most .NET applications TLS 1.2 will come disabled by default, and therefore you'll need to enable it.
You need to add the following line, for example, at the beginning of you Application_Start method:
public class Site : HttpApplication
{
protected void Application_Start()
{
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
// other configuration
}
}

You're probably connecting to api.paypal.com or api.sandbox.paypal.com, and not sending along your API certificate. The API certificate is a client SSL certificate used to complete the SSL chain.
If you don't have or are not using an API certificate, you should connect to api-3t.paypal.com or api-3t.sandbox.paypal.com for Live or Sandbox respectively.

I've been working with a PayPal (NVP/Signature) Express Checkout integration and have been hit with this SSL/TLS error.
Nothing I did seemed to get around it but then I found the following code to add above my request. For reference, I'm using MVC3/.NET 4 so Tls1.2 isn't available to me by default (like in .NET 4.5 +). This first three lines of this code gets around that. I hope it helps people!
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = (SecurityProtocolType)3072;
ServicePointManager.DefaultConnectionLimit = 9999;
var url = "https://[paypal-api-url]/nvp";
var uri = new Uri(url);
var request = WebRequest.Create(uri);
var encoding = new UTF8Encoding();
var requestData = encoding.GetBytes(data);
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
request.Timeout = (300 * 1000);
request.ContentLength = requestData.Length;
using (var stream = request.GetRequestStream())
{
stream.Write(requestData, 0, requestData.Length);
}
var response = request.GetResponse();
...

Thanks a lot that really helps me.
For reference here is my code for establishing the interface in VB.NET
'Create a service Binding in code
Dim ppEndpointAddress As New System.ServiceModel.EndpointAddress("https://api-3t.sandbox.paypal.com/2.0/")
Dim ppBinding As New System.ServiceModel.BasicHttpBinding(System.ServiceModel.BasicHttpSecurityMode.Transport)
Dim ppIface As New PayPalAPI.PayPalAPIAAInterfaceClient(ppBinding, ppEndpointAddress)
Dim ppPaymentReq As New PayPalAPI.DoDirectPaymentReq()
ppPaymentReq.DoDirectPaymentRequest = ppRequest

Is it possible to determine if a HTTPS proxy is being used just by looking at the certificate?

Since HTTPS proxies will replace the SSL certificate with their own, what are my options to determine if a given HTTPS connection has a proxy in the middle?
I will use this information to determine my application policy, since there are cases where I want a 100% end-to-end encrypted tunnel with no decryption by any 3rd party.
Even better if you can tell me how to determine this via C# in a .NET application or Silverlight.
For starters, here is a sample method to validate a certificate using .NET, but I'm still not sure how to use this to determine what part of the cert to validate. In addition, I think the ServicePointManger is more of a "global" connection class. Using this may be too broad when I'm testing a single HTTP connection, and I'm not sure if ServicePointManager is available within Silverlight.
http://msdn.microsoft.com/en-us/library/bb408523.aspx

You have a couple of options. The first option is to use the ServicePointManager class. You are correct in that it manages all service points, but you can use the "sender" parameter in the callback method to differentiate between the different service points:
void SomeMethod()
{
ServicePointManager.ServerCertificateValidationCallback +=
ValidateServerCertificate;
var url = "https://mail.google.com/mail/?shva=1#inbox";
var request = (HttpWebRequest)WebRequest.Create(url);
request.GetResponse();
}
private static bool ValidateServerCertificate(object sender,
X509Certificate certificate, X509Chain chain,
SslPolicyErrors sslpolicyerrors)
{
if(sender is HttpWebRequest)
{
var request = (HttpWebRequest) sender;
if(request.RequestUri.ToString() == "https://mail.google.com/mail/?shva=1#inbox")
{
return (certificate.GetPublicKeyString() == "The public key string you expect");
}
}
return true;
}
This option will work for manually-created HttpWebRequest and WCF-created requests, as the "sender" will be HttpWebRequest for both. I'm not sure if the "sender" will be anything other than an HttpWebRequest.
The second option is to get the certificate from the service point directly:
void SomeMethod()
{
ServicePointManager.ServerCertificateValidationCallback +=
ValidateServerCertificate;
var url = "https://mail.google.com/mail/?shva=1#inbox";
var request = (HttpWebRequest)WebRequest.Create(url);
request.GetResponse();
var serverCert = request.ServicePoint.Certificate;
// Validate the certificate.
}
I couldn't figure out if it's possible to get the ServicePoint used by a WCF proxy. If it's not possible, this option won't work for WCF. Other than that, the biggest difference is that the first option prevents the connection if the certificate validation fails, while the second method won't validate until after the connection has been made.
If you just need to determine if a request is going to pass through a proxy:
var httpRequest = (HttpWebRequest)WebRequest.Create("someurl");
var isUsingProxy = DoesRequstUseProxy(request);
bool DoesRequestUseProxy(HttpWebRequest request)
{
if(request.Proxy == null)
{
return false;
}
return request.Proxy.GetProxy(request.RequestUri) != request.RequestUri;
}