Currently I am working in CRUD operations using Amazon S3 for 3.5 .net , I am using 3.1.5 version.
I found this code to check if the bucket exists :
AmazonS3Client s3Client = new AmazonS3Client ();
///setup the client configuration
S3DirectoryInfo directoryInfo = new S3DirectoryInfo(s3Client, bucketName);
bucketExists = directoryInfo.Exists;
Is there another elegant way (c# code) to check if the bucket exists?
I originally followed the answer here but I switched to a slightly different method so I thought I'd share it. This method creates the bucket if it doesn't already exist.
internal async Task CreateBucketAsync(string bucket, CancellationToken token)
{
if (string.IsNullOrEmpty(bucket)) return;
using (var amazonClient = GetAmazonClient)
{
if (AmazonS3Util.DoesS3BucketExist(amazonClient, bucket)) return;
await amazonClient.PutBucketAsync(new PutBucketRequest { BucketName = bucket, UseClientRegion = true }, token);
await SetMultiPartLifetime(amazonClient, bucket, token);
}
}
Your Code is written in c#, you are looking for other way to check if the directory exists? I think your way is better.
You can create a list of all the subfolders in the root and store it in other place (text file or list or whatever you want) and then you don't need to create every time connection to amazon.
S3DirectoryInfo s3Root = new S3DirectoryInfo(s3Client, "bucketofcode");
foreach (S3DirectoryInfo subDirectory in s3Root.GetDirectories())
{
Console.WriteLine(subDirectory.Name);
}
From here https://blogs.aws.amazon.com/net/post/Tx2N8LWZYHZHGQI/The-Three-Different-APIs-for-Amazon-S3
Related
I am using .Net Core along with Amazon's .net sdk to push and pull things from S3. I am using a folder structure in S3 that involves inserting an empty directory with several sub directories.
At a later time I insert files into those directories and move them around. Now I need to be able to remove the directory entirely.
I am able to delete all of the contents of the directory by using
await client.DeleteObjectAsync(bucketName, keyName, null).ConfigureAwait(false);
where I loop through all the files I want to delete in the given bucket. However, it always leaves me with the empty folder structure, in S3 I see that it has a content of 0 Bytes but I don't want to have to sort through thousands of empty folders to find the ones that actually have data.
Is there any way to remove an empty folder from S3 using AWS .NET SDK?
Update:
I am able to delete everything in the folder I want except for the folder itself.
using (IAmazonS3 client = new AmazonS3Client(awsCreds, Amazon.RegionEndpoint.USEast1))
{
try
{
DeleteObjectsRequest deleteRequest = new DeleteObjectsRequest();
ListObjectsRequest listRequest = new ListObjectsRequest
{
BucketName = bucketName,
Prefix = prefix,
//Marker = prefix,
};
ListObjectsResponse response = await client.ListObjectsAsync(listRequest).ConfigureAwait(false);
// Process response
foreach (S3Object entry in response.S3Objects)
{
deleteRequest.AddKey(entry.Key);
}
deleteRequest.BucketName = bucketName;
var response2 = await client.DeleteObjectsAsync(deleteRequest).ConfigureAwait(false);
return true;
}
catch (AmazonS3Exception amazonS3Exception)
{
if (amazonS3Exception.ErrorCode != null
&& (amazonS3Exception.ErrorCode.Equals("InvalidAccessKeyId", StringComparison.Ordinal)
|| amazonS3Exception.ErrorCode.Equals("InvalidSecurity", StringComparison.Ordinal)))
{
logger.LogError("AwsS3Service.DeleteFileFromBucket Error - Check the provided AWS Credentials.");
}
else
{
logger.LogError($"AwsS3Service.DeleteFileFromBucket Error - Message: {amazonS3Exception.Message}");
}
}
}
This deletes the entire contents of the directory I choose along with all sub directories. But the main directory remains, is there any way to remove that main directory.
Your code is 99% of the way there. The only thing you need to do is add the prefix variable to your keys to be deleted as well. Technically, it is a 0-byte object that needs to be 'deleted' as well.
For example, after your loop through all the objects in the response, go ahead and add the prefix variable that was added to find all those things.
foreach (S3Object entry in response.S3Objects)
{
deleteRequest.AddKey(entry.Key);
}
// Add the folder itself to be deleted as well
deleteRequest.AddKey(prefix);
I am trying to test some methods in my library using Google API. More specifically the Cloud Vision API. When I reference the library in LINQPad I get an error
FileNotFoundException: Error loading native library. Not found in any of the possible locations: C:\Users\\AppData\Local\Temp\LINQPad5_dgzgzeqb\shadow_fxuunf\grpc_csharp_ext.x86.dll,C:\Users\\AppData\Local\Temp\LINQPad5_dgzgzeqb\shadow_fxuunf\runtimes/win/native\grpc_csharp_ext.x86.dll,C:\Users\\AppData\Local\Temp\LINQPad5_dgzgzeqb\shadow_fxuunf../..\runtimes/win/native\grpc_csharp_ext.x86.dll
I have tried copying the dll into all of these locations as well as my LINQPad Plugins and the LINQPad folder. I have tried clearing Cancel and Clear query thinking I needed to reset it. I have also closed and reopened LINQPad thinking maybe it re-scans the directory on load. None of this has worked. Has LINQPad changed where to put dlls or am I missing something?
I am using Google.Cloud.Vision.V1
`
var file = new byte[128];
var _settingsCon = new SettingConnector();
var apiKey = Task.Run(() => _settingsCon.Get("Google:Key")).Result.Value;
var credential = Google.Apis.Auth.OAuth2.GoogleCredential.FromJson(apiKey);
var channel = new Grpc.Core.Channel(
ImageAnnotatorClient.DefaultEndpoint.ToString(),
credential.ToChannelCredentials());
var builder = new StringBuilder();
var image = Image.FromBytes(file);
var client = ImageAnnotatorClient.Create(channel);
var response = client.DetectDocumentText(image);
foreach (var page in response.Pages)
{
foreach (var block in page.Blocks)
{
foreach (var paragraph in block.Paragraphs)
{
builder.Append(paragraph);
}
}
}
builder.ToString().Dump();`
This is essentially the function. The file is a dummy file that would be passed in. It shouldn't matter cause it can't make the request any way. The Dump is used instead of return.
I'm trying to list objects from my S3 bucket, 3rd level of a certain folder only:
bucket
samples
XXXX
XXXX_XXXXX
XXXX_XXXXX
YYYY
YYYY_YYYYY
YYYY_YYYYY
The XXXX_XXXXX and YYYY_YYYYY folders only.
Using C#, this is my code:
using (IAmazonS3 client = new AmazonS3Client("awsAccessKeyId", "awsSecretAccessKey", RegionEndpoint.GetBySystemName("eu-central-1")))
{
ListObjectsRequest request = new ListObjectsRequest
{
BucketName = bucketName,
Prefix = "samples/",
Delimiter = "/"
};
do
{
ListObjectsResponse response = client.ListObjects(request);
if (response.S3Objects.Count() > 0)
{ // CODE }
The response.S3Objects is empty. If I remove the Delimiter from the Request ALL the objects are returned, and the loading time is too long.
I've been following the AWS S3 docs, but it simply returns nothing. Please help me understand what is wrong. Many thanks.
You need to be looking in CommonPrefixes, not S3Objects. CommonPrefixes gives you all of the prefixes up to the next delimiter, each of which you use to repeat the request, taking you another level deeper each time.
I have a UWP application which perform to capture and process images from a camera. This project leverage Microsoft Cognitive Services Face Recognition API and I'm exploring the application's existing functionality for awhile now. My goal is that when the image of a person is identified by the camera (through Face Recognition API service), I want to show the associated image of that person.
With that, the images are captured and stored in a local directory of my machine. I want to retrieve the image file and render it on the screen once the person is identified.
The code below shows the async Task method ProcessCameraCapture
private async Task ProcessCameraCapture(ImageAnalyzer e)
{
if (e == null)
{
this.UpdateUIForNoFacesDetected();
this.isProcessingPhoto = false;
return;
}
DateTime start = DateTime.Now;
await e.DetectFacesAsync();
if (e.DetectedFaces.Any())
{
string names;
await e.IdentifyFacesAsync();
this.greetingTextBlock.Text = this.GetGreettingFromFaces(e, out names);
if (e.IdentifiedPersons.Any())
{
this.greetingTextBlock.Foreground = new SolidColorBrush(Windows.UI.Colors.GreenYellow);
this.greetingSymbol.Foreground = new SolidColorBrush(Windows.UI.Colors.GreenYellow);
this.greetingSymbol.Symbol = Symbol.Comment;
GetSavedFilePhoto(names);
}
else
{
this.greetingTextBlock.Foreground = new SolidColorBrush(Windows.UI.Colors.Yellow);
this.greetingSymbol.Foreground = new SolidColorBrush(Windows.UI.Colors.Yellow);
this.greetingSymbol.Symbol = Symbol.View;
}
}
else
{
this.UpdateUIForNoFacesDetected();
}
TimeSpan latency = DateTime.Now - start;
this.faceLantencyDebugText.Text = string.Format("Face API latency: {0}ms", (int)latency.TotalMilliseconds);
this.isProcessingPhoto = false;
}
In GetSavedFilePhoto, I passed the string names argument once the person is identified.
Code below for the GetSavedFilePhoto method
private void GetSavedFilePhoto(string personName)
{
if (string.IsNullOrWhiteSpace(personName)) return;
var directoryPath = #"D:\PersonImages";
var directories = Directory.GetDirectories(directoryPath);
var filePaths = Directory.GetFiles(directoryPath, "*.jpg", SearchOption.AllDirectories);
}
However, in GetSavedFilePhoto method the variable directories returned an empty string of array when using directoryPath string variable. Directory "D:\PersonImages" is a valid and existing folder in my machine and, it contains subfolders with images inside. I also tried Directory.GetFiles to retrieve the jpg images but still returned an empty string.
I think it should work because I have used Directory class several times but not inside an asyncTask method. Does using async caused the files not returned when using I/O operation?
Sorry for this stupid question, but I really don't understand.
Any help is greatly appreciated.
Using Directory.GetFiles or Directory.GetDirectories method can get the folder/file in the local folder of the Application by the following code. But it could not open D:\.
var directories = Directory.GetDirectories(ApplicationData.Current.LocalFolder.Path);
In UWP app you can only access two locations at default (local folder and install folder), others need capabilities setting or file open picker.Details please reference file access permission.
If you need access to all files in D:\, the user must manually pick the D:\ drive using the FolderPicker, then you have permissions to access to files in this drive.
var picker = new Windows.Storage.Pickers.FileOpenPicker();
picker.ViewMode = Windows.Storage.Pickers.PickerViewMode.Thumbnail;
picker.SuggestedStartLocation =
Windows.Storage.Pickers.PickerLocationId.ComputerFolder;
picker.FileTypeFilter.Add(".jpg");
picker.FileTypeFilter.Add(".jpeg");
picker.FileTypeFilter.Add(".png");
Windows.Storage.StorageFile file = await picker.PickSingleFileAsync();
if (file != null)
{
// Application now has read/write access to the picked file
}
else
{
//do some stuff
}
After scouring the web I am completely stuck on an application I am building to push directories up to Amazon S3 using C# (Targeting .NET 4.5). I am getting NullReferenceExceptions on the line of code that pushes the directory files using the UploadDirectory(TransferUtilityUploadDirectoryRequest) method of the TransferManager class. The problem is I cannot find anything that is null! The debugger doesn't show anything null either, so I'm obviously missing something here.
I read up that if you are uploading to buckets that have periods in them, you need to change the protocol to HTTP otherwise a NullReferenceException might be thrown, however I've done this as well and am continuing to receive the error, even when I created another bucket for testing that has no periods in it.
The portion of my code up to & including the line that causes the exception is below. The class called S3Info is just a helper class that I created that just stores some configuration info such as access/secret keys and other info:
public static void uploadDirectories(S3Info info, List<DirectoryInfo> dirs, Logger logger = null)
{
AmazonS3Config alterConfig = new AmazonS3Config();
alterConfig.CommunicationProtocol = Protocol.HTTP;
AmazonS3Client s3Client = new AmazonS3Client(info.getCredentials(), alterConfig);
TransferUtility directoryTransferUtil = new TransferUtility(s3Client);
TransferUtilityUploadDirectoryRequest uploadDirRequest;
PutObjectRequest completeFileUploadRequest;
uint uploadSuccessCount = 0;
if (dirs == null || dirs.Count == 0)
{
logger.log("Nothing to upload.");
return;
}
//upload directory with PDFs
foreach (DirectoryInfo dir in dirs)
{
try
{
//configure upload request
uploadDirRequest = new TransferUtilityUploadDirectoryRequest();
uploadDirRequest.BucketName = info.selectedBucket.BucketName;
uploadDirRequest.Directory = dir.FullName;
uploadDirRequest.KeyPrefix = dir.Name + #"\";
uploadDirRequest.SearchOption = SearchOption.TopDirectoryOnly;
uploadDirRequest.SearchPattern = "*.pdf";
uploadDirRequest.Timeout = 600000; //10 minutes
//upload directory!
directoryTransferUtil.UploadDirectory(uploadDirRequest); //exception thrown here
I'm a bit stuck at this point so I'm open to any suggestions the community can provide. Thanks.
EDIT: Stack Trace-
Object reference not set to an instance of an object. :
at Amazon.S3.Transfer.Internal.UploadDirectoryCommand.Execute()
at Amazon.S3.Transfer.TransferUtility.UploadDirectory(TransferUtilityUploadDirectoryRequest request)
at S3Delivery.AmazonActions.uploadDirectories(S3Info info, List`1 dirs, Logger logger) in c:\Users\jblacker\Documents\Visual Studio 2012\Projects\S3Delivery\S3Delivery\AmazonActions.cs:line 173
Line 173 is the line referred to above.
A patched version of the SDK (version 1.5.30.1) was released earlier today that fixes this issue.
I posted this same question on the AWS Forums. Apparently the API was broken for the method: UploadDirectory() in version 1.5.30 of the .NET SDK.
Amazon has just posted a patch as version 1.5.30.1