I have tried a lot of things and I admit defeat (I have read a lot of responses on here but none have helped me so far). I am trying to setup signed URLs for files held on Cloudfont. I am able to create signed URLs for S3 but I cannot get anything to work for Cloudfront. For cloudfront I am using the following from the AWS SDK:
var url = AmazonCloudFrontUrlSigner.GetCannedSignedURL( AmazonCloudFrontUrlSigner.Protocol.http, "cdn.coffeebreakgrooves.com", privateKey,
file, cloudFrontKeyPairID, DateTime.Now.AddDays(2));
I get a signed URL generated but I get access denied when following the link, which when I read about it suggested I setup Origin Access Identity. So I then went to my distribution settings and setup Origin Access Identity and chose:
Restrict Bucket Access: Yes
Origin Access Identity: Use an Existing Identity
Grant Read Permissions: Yes, Update Bucket Policy
Then all files become publicly available on Cloudfront, regardless of any settings I have for ACL in S3 (so even if file.txt has no permissions for anyone in S3 it can then be accessed via Cloudfront) and I can't tell if the signed URLs work or not because the download works with or without the querystring and the files have become publicly available. Essentially, how can I make my files private but downloadable with a signed URL (and is my signing method correct?). If I delete the generated bucket policy access is restricted again. I think I need to know how to set the bucket policy so that the origin access identity can only access the bucket with a signed URL... maybe.
Many thanks in advance for any help!
After a bit of a break and a rethink here is where I was going wrong. It isn't possible to have some content secured and other not secured in the same distribution. Either a whole distribution is secured or not. Here is my solution.
Setup a new bucket for your secure items in AWS
Add a new distribution in Cloudfront pointing to the new bucket created in 1 and choose Yes for 'Restrict Viewer Access' and 'Yes' for 'Forward Query Strings' (this is only to add the ability to add content disposition to specific downloads) and choose 'Self' for 'Trusted Signers'
At the top of AWS click on your name and choose 'Security Credentials' and choose 'Continue' as we chose 'Self' above.
Click on 'CloudFront Key Pairs' and choose 'Create New Key Pair'. Download the key files when offered (they won't be offered again), you need the private key. Also copy the Access Key ID as you'll need that.
Go to your distributions, click on the i next to the secure distribution, click on the origins tab, click 'create origin' or select the origin and choose Edit then choose 'Yes' for Restrict Bucket Access, Create a New Identity and Yes Update Bucket Policy. This essentially means that Cloudfront can authenticate against your bucket.
In your project go to NuGet and search for 'AWS' and install the AWS SDK.
Copy the private key file (pk***.pem) to a folder above your website root (or somewhere relatively private)
Add some code as per the following to generate a secure URL with a Content Disposition header.
I have to say that I couldn't have solved this without the help of Torsten's post on https://forums.aws.amazon.com/thread.jspa?messageID=421768 which is in PHP but pointed me in the right direction:
string cloudFrontKeyPairID = "myaccesskeyidfrompoint4";
string pathtokey = HttpContext.Current.Request.MapPath("~/").Replace("wwwroot", "ssl") + "pk-mykeyidfilenamesavedin4.pem";
FileInfo privateKey = new FileInfo(pathtokey);
string file = "folder/mytrack.mp3?response-content-disposition=" +
HttpContext.Current.Server.UrlEncode("attachment;filename='a_filename_with_no_spaces.mp3'");
//I can't figure out how to do spaces or odd characters.
url = AmazonCloudFrontUrlSigner.GetCannedSignedURL(
AmazonCloudFrontUrlSigner.Protocol.http,
"customcname.mydomain.com",
privateKey,
file,
cloudFrontKeyPairID,
DateTime.Now.AddDays(2));
I hope that helps someone, I will be using this as a personal resource anyway! Enabling the Origin Access Identity on an existing bucket which doesn't have 'Restrict Viewer Access' set it essentially opens up permissions for all items on your bucket. This may or may not be desirable! If I have anything wrong please let me know, this is all pretty new to me.
Related
This error is driving me crazy, had the same issue with a lambda web api, but that worked when uploaded, so that will do for now
This project however will be a NuGet package, and i need to be able to simply upload a file to S3
I have the code in place using the following : "https://docs.aws.amazon.com/AmazonS3/latest/dev/HLuploadFileDotNet.html"
im trying to run this local, but getting the Unable to get IAM security credentials from EC2 Instance Metadata Service." error
within visual studio, i can see the AWS Explorer, and i can create a bucket from here... so i have my user setup... but when i run local... there is no user???
i know its going to be a one line missing value somewhere :(
Thank you #Geeshan for your answer, it gave me some direction on where to look
However, I had my credentials file setup, but the answer to my question was you also need to have a profile setup with the name "default"
Also, now know that you dont need a matching "default" IAM user.
On the AWS IAM User page, you create your users as normal, no need to create a default user. In Visual studio, when you create a new profile, if your profile is named "default", and you enter they key and secret key for one of your IAM user, running your application will use that user (I miss understood this profile name, and was assuming it needs to match your IAM user, so initially i create a "Default" IAM user. which i now know, is not needed)
If you dont want to use the default then, you can choose a user by using "TryGetProfile and TryGetAWSCredentials in your application code (see below)
This is due to missing credentials in your local machine. Setting up the AWS Explorer will not provide the credentials to running your code.
One way to setup the credentials is to use a Credentials File in C:\users\awsuser\.aws\credentials (asuming you are using windows).
The following is an example of a profile in the credentials file.
[default]
aws_access_key_id = {accessKey}
aws_secret_access_key = {secretKey}
Here you can read more about managing credentials for .Net
I am storing pdf in my blob storage upon click of button. I thought of adding SAS tokens which I am able to add. I am doing something similar to 'GetBlobSasUri()' mentioned in https://learn.microsoft.com/en-us/azure/storage/blobs/storage-dotnet-shared-access-signature-part-2 and then I am saving that url. I am able to generate a SAS token and it is getting attached with the link I am saving in the blob. And after the token expires I am not able to open the url. But if I remove the SAS token which I added to the link, I am able to open the pdf document. I checked the access policy to make sure it is private. But it didn't help as in private mode I am able to open the pdf document by simply removing the SAS token.
And I am also wondering is it possible to add SAS token each time a PDF document opens. So that if one sends the url to somebody, that person will not be able to open the link once the token has expired. Right now what I have is on click of button a pdf is generated by storing it in blob and it has a SAS token appended to it.
I am also storing the token in my database along with the url to open pdf, does that make sense?
Can someone please suggest!
What is the container access level? Is it public or blob? It needs to be private if you don't want the URL (w/o the SAS token) to work.
Also, I agree with David; I wouldn't store the security token in your database. It should be generated as needed. You'll have to write something to generate it. It's not clear how you're serving the PDF Files to the user. To do what you're asking, you would need an app that would let the user select one, and then the app would give the user a URL with a SAS token on it that expires in a few minutes. There's no way that I know of for a PDF file to get a SAS token or generate a SAS Token on its own.
Although question is old, this will help someone looking for solution.
Please check access level for the file itself along with parent container. Setting this to private will make sure URL without token would not work. You should get ErrorCode, "ResourceNotFound".
Also, since this token is meant to be short lived, it should not be stored in Database and should be generated on-the-fly.
As a best security practice, tokens should not be stored in Database. You could store them in Azure KeyValult or similar.
I am running Google Translate API in C#.
Running locally on my computer the next code works, but online on a server it throws the following error:
using Google.Cloud.Translation.V2;
TranslationClient client = TranslationClient.Create();
var response = client.TranslateText(sentence, targetLanguage, sourceLanguage: sourceLanguage);
"The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information."
Locally this runs just by installing Cloud SDK Installer which does all the settings, there is no need for authentication in code.
On the server, should I use instead OAuth 2.0 or Service account keys ?
Can someone assist me on how to solve this?
EDIT: Can someone confirm to me if it is necessary to have access to the local server to run commands in command line like here https://cloud.google.com/storage/docs/authentication ? This would be pretty ridiculous, instead of just writing code. For example Youtube API does not require local access.
Follow directions to get json file:
https://cloud.google.com/translate/docs/reference/libraries
Then run this code first:
System.Environment.SetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS", "c:\mypath\myfile.json");
To generate a private key in JSON or PKCS12 format:
Open the list of credentials in the Google Cloud Platform Console.
OPEN THE LIST OF CREDENTIALS
Click Create credentials.
Select Service account key. A Create service account key window
opens.
Click the drop-down box below Service account, then click New service account.
Enter a name for the service account in Name.
Use the default Service account ID or generate a different one.
Select the Key type: JSON or P12.
Click Create. A Service account created
window is displayed and the private key for the Key type you
selected is downloaded automatically. If you selected a P12 key, the
private key's password ("notasecret") is displayed.
Click Close.
You can find more details here
https://cloud.google.com/storage/docs/authentication
Its all in the errormessage. You have two options
Run the Google Compute Engine on the machine you have your program running on and input your credentials there.
Use a service account and set the "GOOGLE_APPLICATION_CREDENTIALS" environment variable to reference your credentials file (which is a .json file that you can download from the google developer console.)
PS: Do not store your credentials file anywhere on the server where it may be accessed by someone else!
You must download API key from
https://console.developers.google.com/iam-admin/serviceaccounts
After that download .P12 file file to use it in your code
var certificate = new X509Certificate2(#"key3.p12", "notasecret", X509KeyStorageFlags.Exportable);
notasecret is default password
The easiest answer to my question , to avoid local settings on the server, is the third option of using the Translation API described below: using API keys.
This means just a simple POST to an endpoint that has the API key in the link.
https://cloud.google.com/docs/authentication/#getting_credentials_for_server-centric_flow
https://cloud.google.com/docs/authentication/api-keys
I am just starting out with the c# Google.Apis.Gmail.V1 classes and using a service account to read a mailbox.
My code works fine when I use the following snippet
ServiceAccountCredential credential = new ServiceAccountCredential(
new ServiceAccountCredential.Initializer(ServiceAccountEmailAddress)
{
User = "abc#test.domain.com",
Scopes = new[] { "https://www.googleapis.com/auth/gmail.readonly" }
}.FromCertificate(certificate));
With that code I can call the following successfully
if (credential.RequestAccessTokenAsync(CancellationToken.None).Result)
{
}
But I need to modify the email messages so I changed the scope from
https://www.googleapis.com/auth/gmail.readonly
to
https://www.googleapis.com/auth/gmail.modify
I now get an exception when requesting the access token
{"Error:\"unauthorized_client\", Description:\"Unauthorized client or scope in request.\", Uri:\"\""}
I have checked the service account (*.iam.gserviceaccount.com) in the Google Developers Console and I have tried all options for permissions including OWNER which should give me Full access to all resources but no joy.
I think I am just missing a simple step but I am unsure of where to look next.
TL;DR
I would read through this, but here is the short version. I know this is an older post, but hopefully it finds you!
If you have not updated/white-listed the service account's privileges/scopes in the Google Admin Console you will need to do that, make sure the domain has API access enabled, make sure the service account is setup properly, when creating the "certificate" object be aware its parameters so that it is being instantiated correctly, check the permissions on the account being impersonated and finally make sure you've made an appropriate Google Apps service account key (could have easily made an inappropriate key type.)
White-listing Google APIs in the Admin Console
This gives the Google Apps service account the abilityto use whatever scopes you provide in your Google Apps domain.
Login to the Google Apps Admin Console by using the following link.
https://admin.google.com/
The Google Apps user account must have sufficient privileges to modify domain related settings. It does not have to be the account used to create the Google Apps project in the developer console. If the account does not have privilege you will be directed to a completely different screen with no options to click on varying domain controlling web apps like "Security", "Roles", "Support", "Groups" and etc. Instead you'll dumped onto a page that shows things like "Gmail", "Drive", "Docs" and etc. that is typical user apps. The current link it drops you at is https://apps.google.com/user/hub
Click “Security.”
Click “Show more” option at the bottom of the security options list.
Click “Advanced Settings” to get the more options.
Select the “Manage API client access” link.
Now certain API scopes must be white-listed for the desired service account. In the “Client Name” text box provide the service account’s client ID. The client ID is obtained in the developer console. In the “One or more API scopes” add the desired scopes; comma delimited.
Note, if there are existing scopes they will be removed so be sure to re-add any that will be needed.
Enable Domain Wide API Access
Login to the Google Apps Admin Console by using the following link.
https://admin.google.com/
Go to the “Security” page.
Under “API reference” section
Make sure that “Enable API access” is enabled.
Creating an Appropriate Google Apps Service Account Key (Probably this)
Go to the Google Developer Console. Login as the Google Apps user that created the Google Apps project/service account. https://console.developers.google.com/
Navigate to the particular project with which you created the service account.
Click the "Service Account" button on the left of the project's page to bring up a page with all of the project's service accounts.
Click the vertical ellipse widget all the way to the right of the desired service account's row. Select “Create Key.”
Select .p12 key as it looks like this is what you're trying to use. Click "Create." Be sure to protect this key.
I have found that if the key is not created this way then it leaves open the possiblity for making either an API key or an OAuth 2.0 client/user key. These are the wrong types of keys to use in this case you would need to have created a service account key. The way outlined above forces you to create a service account key.
Modifying the Existing Google Apps Service Account's Settings
I'm not going over how to setup the actual service account, one thing you may need in your case is to make sure that the service account has domain wide delegation enabled. This is toggled in the Google Developer Console. Should be pretty easy to find.
Code
You do not provide your entire code base for creating the token, so just want to add a few things you might be doing improperly.
Make sure when you create the certificate that the secret you provide is the default "notasecret" string. This secret is currently the default value provided by all keys distributed by Google and is immutable during key creation. I had a link to prove this, but have since lost it.
X509Certificate2 certificate = new X509Certificate2(certificateFilePath, "notasecret", X509KeyStorageFlags.Exportable);
Just trying to advocate proper coding. While I have found some bugs in the past with Google's constant values that required additional string manipulation (adding additional slashes.) You should really be using the string constants that they provide in place of literals. I only say to use these because it provides a layer of abstraction, who is to say Google will never change the literal; unlikely.
In your case the new scope is:
GmailService.Scope.GmailModify
While the old scope was:
GmailService.Scope.GmailReadonly
Otherwise, everything code wise looks good to me.
Another thing to try would be to make sure that the actual Google Apps user account being impersonated by the service account has sufficient privileges. I would suspect a different error if this were the case, would be getting a 403 in the response instead. Anyway, in your case this is the "abc#test.domain.com" account. Once again you would go to the Google Admin Console, check its roles make sure it has sufficient roles checked for whatever it is you're trying to do. I don't know what specifically you'll need in this case, best bet would be to give it the same permissions as the "Super Admin" role then remove permissions as you go to see what it might actually need. Otherwise, if possible just give it "Super Admin."
If I was a gambler I would put my money on an inappropriately created service account key. I just recently ran into this and it was the only thing that produced the same exact error you're receiving. Other things would get me the same "Description" value in the response token, but not the same "Error" value. I'm not really even sure how the culprit key was made, because I didn't make it. I just know the fix was to recreate a new key with the steps above and that fixed the issue.
Usually when I upload to S3 storage, I use an AmazonS3Client like this:
var client = Amazon.AWSClientFactory.CreateAmazonS3Client(accessKey, secretKey, s3Config)
This works fine for internal use but now I am looking at providing an app to external users and don't want our (sacret) access & secret keys to be out there. I've set up an S3 bucket with a bucket policy allowing uploads (PutObject) from anonymous users but how do I use the Amazon SDK now? I can't seem to find any way without providing the access and secret key.
You should not open a bucket up for public write, likely. You are open to lots of attacks and will need to keep a close eye on your log files, etc.
A better solution would be to keep the default private access on the bucket, then create an IAM user who only has upload (and perhaps download) permissions for the required area. Then when someone wants to upload a file, you can use a call to your server which has the IAM keys to calculate and return a 'pre signed post' which will allow your client app to post a new file to the server. You can then use any auth tool you want on your server to decide whether or not to allow someone to upload, including no auth - but have abuse detection. When you do this the secret key for the IAM user is never sent down to the client, which may be in a debug session etc.
Since the whole post is pre signed, you can also decide where the file is allowed to go, the uploaded file name, etc and return that in the server response.
You just need to pass null for accessKey and secretKey and you can use the SDK for any anonymously allowed operation.
Check out this related question of mine it includes an official response from an Amazon employee from their developer forum! Relevant information from the linked question:
This is from an official Amazon employee on their forum:
As of the 1.3.8.0 release of the SDK you can pass null for the access
and secret key and the SDK will skip the signing process and try the
operations like GetObject as a public operation.
Norm