I'm trying to duplicate a file from a bucket to another but I can't seam to see the new file on the destination bucket.
I'm getting no errors at all...
Request:
Response:
<?xml version="1.0" encoding="UTF-8"?>
<CopyObjectResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<LastModified>2012-04-08T11:26:36.000Z</LastModified
<ETag>"a5f9084078981b64737b57dbf1735fcf"</ETag>
</CopyObjectResult>
But I keep checking the Last Modified Date on S3 and I can't find any information about this new file, either I can access it directly
http://jk-v20.s3.amazonaws.com/PublicFiles/3ff28e21-4801-47c6-a6d0-e370706d303f_Content_Favicon.ico
What am I doing wrong?
Method:
public void DuplicateFileInCloud(string original, string destination)
{
try
{
CopyObjectRequest request = new CopyObjectRequest();
if (original.StartsWith("http"))
{
// could be from other bucket, URL will show all data
// example: http://jk-v30.s3.amazonaws.com/PredefinedFiles/Favicons/002.ico
string bucket = getBucketNameFromUrl(original), // jk-v30
key = getKeyFromUrl(original); // PredefinedFiles/Favicons/002.ico
request.WithSourceBucket(bucket);
request.WithSourceKey(key);
}
else
{
// same bucket: copy/paste operation
request.WithSourceBucket(this.bucketName);
request.WithSourceKey(original);
}
request.WithDestinationBucket(this.bucketName);
request.WithDestinationKey(destination);
request.CannedACL = S3CannedACL.PublicRead;
using (AmazonS3 client = Amazon.AWSClientFactory.CreateAmazonS3Client(this.accessKey, this.secretAccessKey))
{
S3Response response = client.CopyObject(request);
response.Dispose();
}
}
catch (AmazonS3Exception s3Exception)
{
throw s3Exception;
}
}
http://jk-v20.s3.amazonaws.com//PublicFiles/3ff28e21-4801-47c6-a6d0-e370706d303f_Content_Favicon.ico
Is where the file is. (Note double slash. // ..) If you hit this Url you see the ico file. So its something to do with the leading slash, which may be added automatically by your toolset.
Can you post the request (with headers), captured with something like fiddler?
The docs indicate that the source path should start with a slash (i.e., fully qualified), have you tried that?
x-amz-copy-source: /source_bucket/sourceObject
Maybe the framework does that for you, but your destination has a leading slash so maybe...
Code looks correct, I am using something similar in my working application.
It might be helpful to enable server access logging on your s3 buckets to understand what is happening behind the scenes - http://docs.amazonwebservices.com/AmazonS3/latest/dev/ServerLogs.html.
Related
I have a console application which tries to upload a document into a share point document library list.
I am successfully able to do it and also I am able to fill one of the custom Column(Column name is : "Category") value while uploading the file using C#.
I have tried to fill another custom column(Column name is : "Related Assets") value using the same procedure but i get the error stating the provided column name does not exist but when i see in actual share point portal it does exist.
So not able to solve this issue. Even i tried couple of methods as given below and i get same error message in terms of the column does not exist or it has been deleted or not able to recognize it.
Please find the screenshot of SharePoint showing the list of columns:
Please find the code i have till now which upload the document into SharePoint portal.
public static async Task<string> UploadReleaseNoteDocumentintoSpPortal(string releasenotefilepath, string releasenotefilename, string clientid, string clientsecret)
{
string status = string.Empty;
try
{
Console.WriteLine("Trying to Upload Release note file into Share Point Portal...");
string siteUrl = "<<Sp site URL>>";
Console.WriteLine("Connecting to Share Point Portal...");
var ClientContext = new OfficeDevPnP.Core.AuthenticationManager().GetAppOnlyAuthenticatedContext(siteUrl, clientid, clientsecret);
ClientContext.Load(ClientContext.Web, p => p.Title);
await ClientContext.ExecuteQueryAsync();
Console.WriteLine(ClientContext.Web.Title);
var web = ClientContext.Web;
Console.WriteLine("Connected successfully to Share Point Portal...");
List DocumentsList = web.Lists.GetByTitle("Accelerators Documents");
ClientContext.Load(DocumentsList.RootFolder);
await ClientContext.ExecuteQueryAsync();
Console.WriteLine("Reading and loading the list named : Accelerators Documents from SP");
Console.WriteLine("Converting the release note document into byte array");
byte[] bytes = System.IO.File.ReadAllBytes(releasenotefilepath);
MemoryStream stream = new MemoryStream(bytes);
Console.WriteLine("Storing the release note Data into File Create information object of SharePoint");
FileCreationInformation FileCreateInfo = new FileCreationInformation();
FileCreateInfo.Content = bytes;
FileCreateInfo.ContentStream = stream;
FileCreateInfo.Overwrite = true;
FileCreateInfo.Url = DocumentsList.RootFolder.ServerRelativeUrl + #"\" + releasenotefilename;
Console.WriteLine("Adding file to SharePoint");
var ReleaseNoteFiledata = DocumentsList.RootFolder.Files.Add(FileCreateInfo);
ReleaseNoteFiledata.Update();
ReleaseNoteFiledata.ListItemAllFields["Category"] = "Release Notes";
//ReleaseNoteFiledata.ListItemAllFields["Related Assets"] = "<<Desired Value>>";
//IN Above commented line i get the error stating Microsoft.SharePoint.Client.ServerException:
//'Column 'Related Assets' does not exist. It may have been deleted by another user.
//But in actual site if we see it exists as you can see in above screenshot
ReleaseNoteFiledata.ListItemAllFields.Update();
ClientContext.Load(ReleaseNoteFiledata);
await ClientContext.ExecuteQueryAsync();
Console.WriteLine("Adding file to SharePoint Completed Successfully...");
return status = "Successful";
}
catch (Exception ex)
{
Console.WriteLine("Exception occured while trying to upload Release note file into CoP Portal :" + ex.Message);
return status = "Error/Exception";
}
}
Please find the error message i get while trying to add value to another custom column present in SharePoint:
Microsoft.SharePoint.Client.ServerException: 'Column 'Related Assets' does not exist. It may have been deleted by another user.
Even if i use the ReleaseNoteFiledata.SetFileProperties() and pass the values as a dictionary key value pair containing the column name and its value then also i get the same error for the second custom column. If i keep only the category custom column then it works perfectly without any issue as you can see in the screenshot above.
If i select the record and see the details or properties in the SharePoint the Related assets column symbol is some like in below screenshot:
Please let me know if the supporting documents are fine or still if my issue is not understandable so that i can re frame it or provide more screenshots.
Please help me in solving the above issue or how to make this column recognizable or readable or identifiable in the code.
Thanks in Advance
Regards
ChaitanyaNG
You need to use the internal name of the column 'Related Assets' in your code. It should be Related_x0020_Assets.
You could check the internal name of the column by go to list settings-> click the column, you would see the internal name in the url.
I am using .Net Core along with Amazon's .net sdk to push and pull things from S3. I am using a folder structure in S3 that involves inserting an empty directory with several sub directories.
At a later time I insert files into those directories and move them around. Now I need to be able to remove the directory entirely.
I am able to delete all of the contents of the directory by using
await client.DeleteObjectAsync(bucketName, keyName, null).ConfigureAwait(false);
where I loop through all the files I want to delete in the given bucket. However, it always leaves me with the empty folder structure, in S3 I see that it has a content of 0 Bytes but I don't want to have to sort through thousands of empty folders to find the ones that actually have data.
Is there any way to remove an empty folder from S3 using AWS .NET SDK?
Update:
I am able to delete everything in the folder I want except for the folder itself.
using (IAmazonS3 client = new AmazonS3Client(awsCreds, Amazon.RegionEndpoint.USEast1))
{
try
{
DeleteObjectsRequest deleteRequest = new DeleteObjectsRequest();
ListObjectsRequest listRequest = new ListObjectsRequest
{
BucketName = bucketName,
Prefix = prefix,
//Marker = prefix,
};
ListObjectsResponse response = await client.ListObjectsAsync(listRequest).ConfigureAwait(false);
// Process response
foreach (S3Object entry in response.S3Objects)
{
deleteRequest.AddKey(entry.Key);
}
deleteRequest.BucketName = bucketName;
var response2 = await client.DeleteObjectsAsync(deleteRequest).ConfigureAwait(false);
return true;
}
catch (AmazonS3Exception amazonS3Exception)
{
if (amazonS3Exception.ErrorCode != null
&& (amazonS3Exception.ErrorCode.Equals("InvalidAccessKeyId", StringComparison.Ordinal)
|| amazonS3Exception.ErrorCode.Equals("InvalidSecurity", StringComparison.Ordinal)))
{
logger.LogError("AwsS3Service.DeleteFileFromBucket Error - Check the provided AWS Credentials.");
}
else
{
logger.LogError($"AwsS3Service.DeleteFileFromBucket Error - Message: {amazonS3Exception.Message}");
}
}
}
This deletes the entire contents of the directory I choose along with all sub directories. But the main directory remains, is there any way to remove that main directory.
Your code is 99% of the way there. The only thing you need to do is add the prefix variable to your keys to be deleted as well. Technically, it is a 0-byte object that needs to be 'deleted' as well.
For example, after your loop through all the objects in the response, go ahead and add the prefix variable that was added to find all those things.
foreach (S3Object entry in response.S3Objects)
{
deleteRequest.AddKey(entry.Key);
}
// Add the folder itself to be deleted as well
deleteRequest.AddKey(prefix);
I'm trying to list objects from my S3 bucket, 3rd level of a certain folder only:
bucket
samples
XXXX
XXXX_XXXXX
XXXX_XXXXX
YYYY
YYYY_YYYYY
YYYY_YYYYY
The XXXX_XXXXX and YYYY_YYYYY folders only.
Using C#, this is my code:
using (IAmazonS3 client = new AmazonS3Client("awsAccessKeyId", "awsSecretAccessKey", RegionEndpoint.GetBySystemName("eu-central-1")))
{
ListObjectsRequest request = new ListObjectsRequest
{
BucketName = bucketName,
Prefix = "samples/",
Delimiter = "/"
};
do
{
ListObjectsResponse response = client.ListObjects(request);
if (response.S3Objects.Count() > 0)
{ // CODE }
The response.S3Objects is empty. If I remove the Delimiter from the Request ALL the objects are returned, and the loading time is too long.
I've been following the AWS S3 docs, but it simply returns nothing. Please help me understand what is wrong. Many thanks.
You need to be looking in CommonPrefixes, not S3Objects. CommonPrefixes gives you all of the prefixes up to the next delimiter, each of which you use to repeat the request, taking you another level deeper each time.
I am trying to copy a blob from one location to another and it seems like this method is obsolete. Everything I've read says I should use "StartCopy". However, when I try this it doesn't copy the blob. I just get a 404 error at the destination.
I don't seem to be able to find any documentation for this. Can anyone advise me on how to do this in the latest version of the API or point me in the direction of some docs.
Uri uploadUri = new Uri(destinationLocator.Path);
string assetContainerName = uploadUri.Segments[1];
CloudBlobContainer assetContainer =
cloudBlobClient.GetContainerReference(assetContainerName);
string fileName = HttpUtility.UrlDecode(Path.GetFileName(model.BlockBlob.Uri.AbsoluteUri));
var sourceCloudBlob = mediaBlobContainer.GetBlockBlobReference(fileName);
sourceCloudBlob.FetchAttributes();
if (sourceCloudBlob.Properties.Length > 0)
{
IAssetFile assetFile = asset.AssetFiles.Create(fileName);
var destinationBlob = assetContainer.GetBlockBlobReference(fileName);
destinationBlob.DeleteIfExists();
destinationBlob.StartCopyFromBlob(sourceCloudBlob);
destinationBlob.FetchAttributes();
if (sourceCloudBlob.Properties.Length != destinationBlob.Properties.Length)
model.UploadStatusMessage += "Failed to copy as Media Asset!";
}
I'm just posting my comment as the answer to make it easier to see.
It wasn't the access level of the container. It wasn't anything to do with StartCopy either. It turned out to be these lines of code.
var mediaBlobContainer = cloudBlobClient.GetContainerReference(cloudBlobClient.BaseUri + "temporarymedia");
mediaBlobContainer.CreateIfNotExists();
Apparently I shouldn't be supplying the cloudBlobClient.BaseUri, just the name temporarymedia.
var mediaBlobContainer = cloudBlobClient.GetContainerReference("temporarymedia");
There was no relevant error message though. Hopefully it'll save another Azure newbie some time in future.
We use Amazon S3 to save picture for our project but now we need to copy some images from one place to another in S3. I have older/saved images path stored in my DB and now wants to save them using new path.
Can someone please guide me in any right direction to start with. I have been looking into CopyObjectRequest but can't figure out how to proceed.
Any help will be appreciated.
Thanks
Here is a generic example with original and destination variables being passed to the function from the caller.
public void DuplicateFileInCloud(string original, string destination)
{
try
{
CopyObjectRequest request = new CopyObjectRequest();
if (original.StartsWith("http"))
{
// example: http://jk-v30.s3.amazonaws.com/PredefinedFiles/Favicons/002.ico
string bucket = getBucketNameFromUrl(original), // i.e. jk-v30
key = getKeyFromUrl(original); // the path to your file: PredefinedFiles/Favicons/002.ico
request.WithSourceBucket(bucket);
request.WithSourceKey(key);
}
else
{
// same bucket: copy/paste operation
request.WithSourceBucket(this.bucketName);
request.WithSourceKey(original);
}
request.WithDestinationBucket(this.bucketName);
request.WithDestinationKey(destination);
request.CannedACL = S3CannedACL.PublicRead; // one way of setting public headers - see other below
using (AmazonS3 client = Amazon.AWSClientFactory.CreateAmazonS3Client(this.accessKey, this.secretAccessKey))
{
S3Response response = client.CopyObject(request);
response.Dispose();
}
}
catch (AmazonS3Exception s3Exception)
{
throw s3Exception;
}
}
Hope this is what you were looking for!
Setting public-read
response.Addheader("x-amz-acl","public-read");