I have an MVC 5 web application running on .NET 4.7.2 and hosted in an Azure AppService, that uses Azure Key Vault to hold secrets. The project uses the Microsoft.Azure.KeyVault 3.0.3 NuGet package and the secrets are accessed using the KeyVaultClient and .GetSecretAsync(). All resources are located in the same Azure region.
For the most part this works very well, and for about 90% of the time it returns the secret in milliseconds.
But every now and then the call to access the Key Vault fails. This doesn't manifest itself as an exception thrown by the SDK, but the web app hangs. Eventually - and normally in around 1 minute but sometimes longer - the secret is returned and all is fine again. This is because the SDK uses a retry pattern, which will keep trying to get the secret.
Looking at Application Insights for the AppService I can see that the GET request generated by the SDK gets an HTTP 500 response from the Key Vault and a SocketException is thrown, with a result code of ConnectFailure.
The exception is:
Looking at the telemetry and stepping through the code there is no element of commonality or obvious cause. It seems to be entirely random.
The bottom line is the Azure hosted AppService sometimes cannot connect to an Azure hosted Key Vault in the same datacentre, using the latest framework and SDK version.
Has anyone else seen this or have any idea? I've searched around and found a few instances of people experiencing the same issue, but nobody has a cause or solution.
EDIT (1): I have now tried spinning up a new Key Vault in a different region entirely, and the problem remains exactly the same.
We experienced the same behavior on our project, where KeyVault would be fast and reliable most of the time, and then intermittently stop responding or take a very long time to return once in a while with no obvious reason to explain why. This occurred in all tiers of our application, from the API, to Azure Functions, to command line tools.
Eventually, we had to work around this by caching secrets in memory to avoid hitting the KeyVault too often, where our AppSettings class would cache these internally. In addition to this, we also configured our DI container to treat this class as a singleton.
Here is a very simplified example:
public class MyAppSettings : IAppSettings
{
private readonly ObjectCache _cache = MemoryCache.Default;
private readonly object _lock = new Object();
private KeyValueClient _kvClient;
public string MySecretValue => GetSecret("MySecretValue");
private KeyValueClient GetKeyVaultClient()
{
// Initialize _kvClient if required
return _kvClient;
}
private string GetSecret(string name)
{
lock (_lock)
{
if (_cache.Contains(key))
return (string) _cache.Get(key);
// Sanitize name if required, remove reserved chars
// Construct path
var path = "...";
// Get value from KV
var kvClient = GetKeyVaultClient();
Task<SecretBundle> task = Task.Run(async() => await kvClient.GetSecretAsync(path));
var value = task.Result;
// Cache it
_cache.Set(name, value, DateTime.UtcNow.AddHours(1));
return value;
}
}
}
This isn't production ready - you'll need to modify this and implement the GetKeyVaultClient method to actually return your KeyVaultClient object, and also the GetSecret method should sanitize the key name being retrieved.
In our DI registry, we had this setup to use a singleton like this:
For<IAppSettings>().Use<MyAppSettings>().Singleton();
These two changes seemed to work well for us, and we haven't had any issues with this for a while now.
Another option is to deploy the secrets from keyvault to your app service application as app settings in your deployment pipeline.
Pros:
Keep the secrets out of source control
Remove the runtime dependency on keyvault
Faster reliable local access to the secrets
Cons:
Updating the secrets requires a redeploy
Related
I want the azure functions to use the connection string that is stored in azure key vault, not from application settings in azure portal.
I have stored the secret in azure portal and fetching it using the below code in Startup.cs file, but it is not working.
Below is code piece I have tried:
Startup.cs
public override void Configure(IFunctionsHostBuilder builder)
{
// Reading connection string from app settings, (but I don't want this method)
// var configuration = Environment.GetEnvironmentVariable("SQL:ConnectionString");
var configuration = AzureKeyVault.GetSecretKey();
builder.Services.AddHttpClient();
Server.Module.Load(builder.Services, configuration);
}
AzureKeyVault Class
public class AzureKeyVault
{
public static String GetSecretKey()
{
var azureServiceTokenProvider = new AzureServiceTokenProvider();
var keyVault = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
var secretKey = Task.Run(async () => await keyVault.GetSecretAsync("https://somesecreturl.azure.net/", "mysecretkeyname")).Result;
var connectionString = secretKey.Value;
return connectionString;
}
}
The above code works fine in local, but after deploying it is not picking the connection string from azure key vault (for which I have written code), but I am getting an error Function host is not running
When I add connection string in application settings, function app is working fine with Environment.GetEnvironmentVariable("SQL:ConnectionString")
I am a newbie in azure. Azure functions are designed in a way that it can able to read the connection string only from application settings alone?
Any help would be highly appreciated. Thanks!
Key Vault Access
Since it is working for you locally (I assume the local instance of the function uses your AZ CLI identity with your account that has access to Key Vault, because you have added that secret beforehand), I believe your issue might be connected with granting Key Vault Access for your Function App. Please make sure you have granted the access as described here: https://learn.microsoft.com/en-us/azure/app-service/app-service-key-vault-references?tabs=azure-cli#granting-your-app-access-to-key-vault.
Key Vault References
If my assumption is correct and you are trying to hide the secret in Key Vault rather than storing it as plain-text in App Settings, you can always insert a reference like:
#Microsoft.KeyVault(VaultName=myvault;SecretName=mysecret)
as your application setting and then you will be able to read it in your application just like it would be a regular env variable:
var configuration = Environment.GetEnvironmentVariable("SQL:ConnectionString");
This will make your code much simpler and will allow you to decide in the future if you want to take configuration from another service (for example App Configuration Service) without any code changes to your app.
Note: using reference will not fix your access issue, you need to fix that first anyways.
As part of a Microservice based solution that we are building we have a number of Azure Functions sitting in one Azure Function App. The functions Orchestrate numerous requests to different APIs some of which take a long time to complete. We added Application Insights to the functions to allow for some tracking of the requests made, but dependency tracking is not working yet in Azure Functions. It is possible to manually track dependencies but that involves inserting some tracking code around each dependency call, however we want to avoid manually tracking dependencies on each and every call.
One of the solutions I have thought of would be to create a request tracker that tracks all outgoing web requests from the functions. Within the request tracker I could then track the dependency requests including their time. I want to hook the request tracker into some sort of web traffic handler, unfortunately I was unable to find much about doing this. A lot of posts mention using System.Net trace writer for this, but as far as I can see this requires a Web.config to setup and functions do not have one.
I have seen a few posts mentioning to create a request wrapper and place that on my outgoing requests, but unfortantely that is not an option as we use a number of packages that make requests internally. If you have any ideas that could get me going in the right direction please let me know. Thanks
Update:
I added the following helper method which allows me to manually track tasks as dependency requests
public static async Task<T> TrackDependency<T>(this Task<T> task, string dependecyName, string callName, string operationId)
{
var telemtryClient = new TelemetryClient();
var startTime = DateTime.UtcNow;
var timer = System.Diagnostics.Stopwatch.StartNew();
var success = true;
T result = default(T);
try
{
result = await task;
}
catch (Exception)
{
success = false;
}
finally
{
timer.Stop();
var dependencyTelemetry = new DependencyTelemetry(dependecyName, callName, startTime, timer.Elapsed, success);
dependencyTelemetry.Context.Operation.Id = operationId;
telemtryClient.Track(dependencyTelemetry);
}
return result;
}
It can then be used as follows:
client.Accounts.UpdateWithHttpMessagesAsync(accountId, account).TrackDependency("Accounts", "UpdateAccounts", requestContextProvider.CorrelationId);
I can now see individual request dependencies in Application Insights, but obviously the actual telemetry on them is very limited, it does not contain path info or much else.
So when you say dependency tracking is not working in Azure Functions, what exactly do you mean? Have you actually added and configured the Application Insights SDK to your actual function yet? The out-of-the-box monitoring experience with Azure Functions doesn't automatically add dependency tracing, but if you actually add/configure the Application Insights SDK in your function project it should start tracking everything going on in there.
I am trying to test a .NET core console program to publish a message to SNS. As I had issues trying to get it to work in Lambda, I want to try it in a non-Lambda environment. In Lambda, security is covered by the role, but in a console program, I presume that I have to specify my access key and secret somehow.
I've read this page: http://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/net-dg-config-creds.html#net-dg-config-creds-sdk-store, but still totally confused.
I'm running on my local development computer, not an EC2 instance. No intent to go to production with this, just trying to test some code.
I'm on Visual Studio 2015, .NET Core 1.0. I've used Nuget to get the following:
"AWSSDK.Extensions.NETCore.Setup": "3.3.3",
"AWSSDK.SimpleNotificationService": "3.3.0.23",
Based on the answer to How to set credentials on AWS SDK on NET Core? I created the /user/.aws/credentials file (assuming credentials was the file name and not the directory name).
But that question/answer doesn't address how to actually use this file. The code I'm running is below.
public static void Main(string[] args)
{
Console.WriteLine("Started");
//var awsCredentials = new Amazon.Runtime.AWSCredentials()
var client = new Amazon.SimpleNotificationService.AmazonSimpleNotificationServiceClient(Amazon.RegionEndpoint.EUWest2);
//var client = new Amazon.SimpleNotificationService.AmazonSimpleNotificationServiceClient(awsCredentials, Amazon.RegionEndpoint.EUWest2);
//Amazon.SimpleNotificationService.Model.PublishResponse publishResp = null;
SendMessage(client).Wait();
Console.WriteLine("Completed call to SendMessage: Press enter to end:");
Console.ReadLine();
}
The error I'm getting on the new client is:
An unhandled exception of type 'Amazon.Runtime.AmazonServiceException' occurred in AWSSDK.Core.dll
Additional information: Unable to find credentials
I see there is a way to pass an AWSCredentials object to that constructor, but I don't understand how to build it. Amazon.Runtime.AWSCredentials is an abstract class, so I can't use it in a "new" statement.
Based on Dan Pantry's answer, here is a simple short answer with code highlighted (note the region enum in the second line):
var awsCredentials = new Amazon.Runtime.BasicAWSCredentials("myaccesskey", "mysecretkey");
var client = new Amazon.SimpleNotificationService.AmazonSimpleNotificationServiceClient(
awsCredentials, Amazon.RegionEndpoint.EUWest2);
Use a role if possible, but above works when needed. Then the question is where to store the access key/secret key; could be environment variable, config file, prompt the user, or any of the usual suspects.
AWS-CLI and Python use credentials from here: c:\Users\username\.aws\credentials, so the C# could just read that file so as not to put the codes in the C# program itself. But then each user/developer that runs the program would need to set their credentials there.
There is also now a concept of running Lambda on your local machine, but I haven't tried it yet:
https://dzone.com/articles/run-aws-lambda-functions-locally-on-windows-machin#:~:text=Step%201%3A%20Download%20SAM%20local,version%20with%20the%20command%20below.&text=Step%203%3A%20Write%20your%20lambda,yaml%20on%20the%20root%20level.
So the point is that if you are going to do Lambda, but you need to test locally first, this would probably be worth trying.
You'll want to construct one of its child classes instead of the abstract one. You can take a look at the class hierarchy here.
For posterity, the options are:
AnonymousAWSCredentials - Authenticates as an anonymous user.
BasicAWSCredentials - You provide your credentials to the class constructor directly.
EnvironmentAWSCredentials - Credentials are pulled from the environment variables of the running executable.
InstanceProfileAWSCredentials - Pulls credentials from the Instance Profile of the EC2 instance running the executable. This, obviously, only works on EC2.
SessionAWSCredentials - Similar to BasicAWSCredentials, except utilises an AWS Session using a temporary session token from AWS STS.
RefreshingSessionAWSCredentials - Similar to SessionAWSCredentials, but refreshes when the STS token expires.
Note that the default strategy in the absence of a credentials object involves checking the Environment Variables and then the instance profile.
If you want to have the program pull credentials from ~/.aws/credentials, you'll need to do some legwork. There used to be a StoredProfileAWSCredentials class, but that appears to have been removed - you can find more information by looking at this github issue. This is only useful, really, in development as you won't be using ~/.aws/credentials in production but probably instance profiles - I'd suggest instead using the default strategy and using Environment AWS credentials in test or development environments.
I take this approach at work since we use a command line tool to grab us limited time tokens from AWS STS and plunk them into the current shell for use for the next hour.
EDIT: It appears you're using AWS Lambda. These have federated access to AWS resources based on the roles assigned to them, so this should work using the default credential strategy in the aws-sdk library which uses instance profiles. So this is only really necessary for development/testing, in which case I would again recommend just using environment variables.
This is a really old question, and the existing answers work, but I really don't like hard-coding my Access Key Id and Secret Key values directly into source code, even for throw-away projects I'm doing on my local machine. For one thing, I might revoke those keys in the future, so I want to leverage the credentials in my .aws\credentials file.
To do that for my .NET core apps (including console apps, etc), I first add two NuGet packages:
Microsoft.Extensions.Configuration.Json
AWSSDK.Extensions.NETCore.Setup
Then, I add an applications.json file to my project, which contains the following (note - you need to right-click the file, and set "Copy to output" as either "copy if newer" or "always"):
{
"AWS": {
"Profile": "default",
"ProfilesLocation": "C:\\Users\\my-user-profile-folder\\.aws\\credentials",
"Region": "us-west-2"
}
}
Finally, I create an instance of the AWS SDK client using the following:
var builder = new ConfigurationBuilder().AddJsonFile("appsettings.Development.json", optional: false, reloadOnChange: true);
var options = builder.Build().GetAWSOptions();
var s3client = options.CreateServiceClient<IAmazonS3>();
This way, if I update my credentials file, I'm fine. Or if my code gets zipped up and emailed to a friend or co-worker, I don't accidentally send them my credentials also.
There is another way to do this, without needing to add the NuGet packages also, which many people might prefer. You can use the new SharedCredentialsFile class and AWSCredentialsFactory, like this (using the "default" profile here, and assumes your credential file is in the default location, same as the other method):
var sharedFile = new SharedCredentialsFile();
sharedFile.TryGetProfile("default", out var profile);
AWSCredentialsFactory.TryGetAWSCredentials(profile, sharedFile, out var credentials);
var s3Client = new AmazonS3Client(credentials);
Note - I'm not checking that the two Try* methods are succeeding here, which you probably should do. Details on using these classes are here: https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/net-dg-config-creds.html#how-to-create-an-amazons3client-using-the-sharedcredentialsfile-class
While keeping your credentials in the shared "credentials" file, you can redefine the ProfilesLocation when creating the CredentialProfileStoreChain
//define your file location here:
var chain = new CredentialProfileStoreChain(#"C:\aws\credentials");
// input the name of your credentials here:
if (chain.TryGetAWSCredentials("nameofprofile", out AWSCredentials awsCredentials))
{
//executes if the credentials were found and inserted into awsCredentials
}
else
{
// executes if the credentials were not found
}
Taken from here: https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/creds-locate.html
For those struggling with profile names, here is where you can find it.
Contents of your ~/.aws/credentials:
[YOUR_PROFILE_NAME]
aws_access_key_id = ***
aws_secret_access_key = ***
aws_security_token = ***
aws_session_expiration = ***
aws_session_token = ***
So then in your application you access the credentials like this:
var chain = new CredentialProfileStoreChain();
var result = chain.TryGetAWSCredentials("YOUR_PROFILE_NAME", out var credentials);
Resources:
accessing credentials and profiles: https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/net-dg-config-creds.html#creds-locate
named profiles: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html
I'm moving the deployment of a web app from an Azure Website into a Web Role in a Cloud Service.
Part of the migration has involved reserving some local storage in the role config and changing interactions with the local file-system to use the following mantra to find a path that is good for writing to:
LocalResource tempStorageResource = RoleEnvironment
.GetLocalResource("SomeRoleStorage");
var targetFolderPath = tempStorageResource.RootPath;
However, I'd like to keep things working in the WebSite instance. I'm going to write a path provider that abstracts the actual location away. Part of implementing this will require detecting whether I'm running locally/in the debugger, but I also need to know whether the running code is running under a WebSite or a WebRole. How can I do this?
public class AzurePathProvider : ILocalStoragePathProvider
{
public string GetStoragePath(string key)
{
var isWebRole = //????;
if(isWebRole)
{
LocalResource tempStorageResource =
RoleEnvironment
.GetLocalResource(key);
return tempStorageResource.RootPath;
}
else
{
return "/some/other/storage/location";
}
}
}
Check for RoleEnvironment.IsAvailable to decide if the code is running in Cloud Service or not. It will always be true when your code is running in Cloud Service otherwise it will be false.
Furthermore to detect if the code is running in compute emulator, you can check for RoleEnvironment.IsEmulated along with RoleEnvironment.IsAvailable.
So I've got a ServiceReference added to a C# Console Application which calls a Web Service that is exposed from Oracle.
I've got everything setup and it works like peaches when it's not using SSL (http). I'm trying to set it up using SSL now, and I'm running into issues with adding it to the Service References (or even Web References). For example, the URL (https) that the service is being exposed on, isn't returning the appropriate web methods when I try to add it into Visual Studio.
The underlying connection was closed: An unexpected error occurred on a send.
Received an unexpected EOF or 0 bytes from the transport stream.
Metadata contains a reference that cannot be resolved: 'https://srs204.mywebsite.ca:7776/SomeDirectory/MyWebService?WSDL'
Another quandary I've got is in regards to certificate management and deployment. I've got about 1000 external client sites that will need to use this little utility and they'll need the certificate installed in the appropriate cert store in order to connect to the Web Service. Not sure on the best approach to handling this. Do they need to be in the root store?
I've spent quite a few hours on the web looking over various options but can't get a good clean answer anywhere.
To summarize, I've got a couple of questions here:
1) Anybody have some good links on setting up Web Services in Visual Studio that use SSL?
2) How should I register the certificate? Which store should it exist in? Can I just use something like CertMgr to register it?
There's gotta be a good book/tutorial/whatever that will show me common good practices on setting something like this up. I just can't seem to find it!
Well, I've figured this out. It took me far longer than I care to talk about, but I wanted to share my solution since it's a HUGE pet peeve of mine to see the standard. "Oh I fixed it! Thanks!" posts that leave everyone hanging on what actually happened.
So.
The root problem was that by default Visual Studio 2008 uses TLS for the SSL handshake and the Oracle/Java based Webservice that I was trying to connect to was using SSL3.
When you use the "Add Service Reference..." in Visual Studio 2008, you have no way to specify that the security protocol for the service point manager should be SSL3.
Unless.
You take a static WSDL document and use wsdl.exe to generate a proxy class.
wsdl /l:CS /protocol:SOAP /namespace:MyNamespace MyWebService.wsdl
Then you can use the C Sharp Compiler to turn that proxy class into a library (.dll) and add it to your .Net projects "References".
csc /t:library /r:System.Web.Services.dll /r:System.Xml.dll MyWebService.cs
At this point you also need to make sure that you've included System.Web.Services in your "References" as well.
Now you should be able to call your web service without an issue in the code. To make it work you're going to need one magic line of code added before you instantiate the service.
// We're using SSL here and not TLS. Without this line, nothing workie.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
Okay, so I was feeling pretty impressed with myself as testing was great on my dev box. Then I deployed to another client box and it wouldn't connect again due to a permissions/authority issue. This smelled like certificates to me (whatever they smell like). To resolve this, I used certmgr.exe to register the certificate for the site to the Trusted Root on the Local Machine.
certmgr -add -c "c:\someDir\yourCert.cer" -s -r localMachine root
This allows me to distribute the certificate to our client sites and install it automatically for the users. I'm still not sure on how "security friendly" the different versions of windows will be in regards to automated certificate registrations like this one, but it's worked great so far.
Hope this answer helps some folks. Thanks to blowdart too for all of your help on this one and providing some insight.
It sounds like the web service is using a self signed certificate. Frankly this isn't the best approach.
Assuming you're a large organisation and it's internal you can setup your own trusted certificate authority, this is especially easy with Active Directory. From that CA the server hosting the Oracle service could request a certificate and you can use AD policy to trust your internal CA's root certificate by placing it in the trusted root of the machine store. This would remove the need to manually trust or accept the certificate on the web service.
If the client machines are external then you're going to have to get the folks exposing the service to either purchase a "real" certificate from one of the well known CAs like Verisign, Thawte, GeoTrust etc. or as part of your install bundle the public certificate and install it into Trusted Root certificate authorities at the machine level on every machine. This has problems, for example no way to revoke the certificate, but will remove the prompt.
Thanks for this great tip, took a quick look around at your stuff and you have a lot of good ideas going on. Here's my little bit to add -- I'm figuring out webMethods and (surprise!) it has the same problems as the Oracle app server you connected to (SSL3 instead of TLS). Your approach worked great, here's my addendum.
Given static class "Factory," provide these two handy-dandy items:
/// <summary>
/// Used when dispatching code from the Factory (for example, SSL3 calls)
/// </summary>
/// <param name="flag">Make this guy have values for debugging support</param>
public delegate void CodeDispatcher(ref string flag);
/// <summary>
/// Run code in SSL3 -- this is not thread safe. All connections executed while this
/// context is active are set with this flag. Need to research how to avoid this...
/// </summary>
/// <param name="flag">Debugging context on exception</param>
/// <param name="dispatcher">Dispatching code</param>
public static void DispatchInSsl3(ref string flag, CodeDispatcher dispatcher)
{
var resetServicePoint = false;
var origSecurityProtocol = System.Net.ServicePointManager.SecurityProtocol;
try
{
System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Ssl3;
resetServicePoint = true;
dispatcher(ref flag);
}
finally
{
if (resetServicePoint)
{
try { System.Net.ServicePointManager.SecurityProtocol = origSecurityProtocol; }
catch { }
}
}
}
And then to consume this stuff (as you have no doubt already guessed, but put a drum roll in here anyway):
var readings = new ArchG2.Portal.wmArchG201_Svc_fireWmdReading.wmdReading[] {
new ArchG2.Portal.wmArchG201_Svc_fireWmdReading.wmdReading() {
attrID = 1, created = DateTime.Now.AddDays(-1), reading = 17.34, userID = 2
},
new ArchG2.Portal.wmArchG201_Svc_fireWmdReading.wmdReading() {
attrID = 2, created = DateTime.Now.AddDays(-2), reading = 99.76, userID = 3
},
new ArchG2.Portal.wmArchG201_Svc_fireWmdReading.wmdReading() {
attrID = 3, created = DateTime.Now.AddDays(-5), reading = 82.17, userID = 4
}
};
ArchG2.Portal.Utils.wmArchG201.Factory.DispatchInSsl3(ref flag, (ref string flag_inner) =>
{
// creates the binding, endpoint, etc. programatically to avoid mucking with
// SharePoint web.config.
var wsFireWmdReading = ArchG2.Portal.Utils.wmArchG201.Factory.Get_fireWmdReading(ref flag_inner, LH, Context);
wsFireWmdReading.fireWmdReading(readings);
});
That does the trick -- when I get some more time I'll solve the threading issue (or not).
Since I have no reputation to comment, I'd like to mention that Mat Nadrofsky's answer and code sample for forcing SSL3 is also the solution for an error similar to
An error occurred while making the
HTTP request to https://xxxx/whatever.
This could be due to the fact that the
server certificate is not configured
properly with HTTP.SYS in the HTTPS
case. This could also be caused by a
mismatch of the security binding
between the client and the server.
Just use
// We're using SSL here and not TLS. Without this line, nothing workie.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
as mentioned by Mat. Tested with an SAP NetWeaver PI server in HTTPS. Thanks!
Mat,
I had such issues too and I have a way to avoid using certmgr.exe to add certificates to trusted root on a remote machine.
X509Store store;
store = new X509Store("ROOT", StoreLocation.LocalMachine);
store.Open(OpenFlags.ReadWrite);
store.Add(certificate);
The 'certificate object' can be created like this:
X509Certificate2 certificate = new X509Certificate2("Give certificate location path here");