I'm using this approach to encrypt files and store them in Azure block blobs. I would like to copy the encrypted blob to another blob storage account and decrypt it in the process. I know it's possible to do a "copy blob" operation which runs entirely inside Azure asynchronously and doesn't download the blob contents through my local computer in transit. I believe this is accomplished through the CloudBlockBlob.StartCopy method. But is it possible to do that with an encrypted file and decrypt it in transit to the other storage account?
Following that link above, my code looks like the following. blob.OpenRead works but blob2.StartCopy doesn't work.
BlobEncryptionPolicy policy = new BlobEncryptionPolicy(null, cloudResolver);
BlobRequestOptions options = new BlobRequestOptions() { EncryptionPolicy = policy };
CloudBlockBlob blob = container.GetBlockBlobReference("MyFile.txt");
//var blobStream = blob.OpenRead(null, options); //this works
CloudBlockBlob blob2 = container2.GetBlockBlobReference("MyFile2.txt");
blob2.StartCopy(blob, null, null, options, null); //this fails with: The remote server returned an error: (404) Not Found.
The answer is that encryption is done in the storage client library so if you do a copy blob to a new storage account it will still be encrypted.
The reason your code is failing is because the source blob is in a Private container. For cross-account copy to work, the source blob should be publicly accessible. Within same storage account, you can copy a blob from a private container. AFAIK, the error has nothing to do with encryption.
What you could do is create a SAS URL on the source blob and then use the following override of StartCopy method:
public string StartCopy(
Uri source,
AccessCondition sourceAccessCondition = null,
AccessCondition destAccessCondition = null,
BlobRequestOptions options = null,
OperationContext operationContext = null
)
Here's the sample code to do so:
private static void StartCopyAcrossAccount()
{
var sourceAccount = new CloudStorageAccount(new StorageCredentials("source-account-name", "source-account-key"), true);
var sourceContainer = sourceAccount.CreateCloudBlobClient().GetContainerReference("source-container");
var sourceBlob = sourceContainer.GetBlockBlobReference("blob-name");
var sourceBlobSas = sourceBlob.GetSharedAccessSignature(new Microsoft.WindowsAzure.Storage.Blob.SharedAccessBlobPolicy()
{
SharedAccessExpiryTime = DateTime.UtcNow.AddHours(1),
Permissions = Microsoft.WindowsAzure.Storage.Blob.SharedAccessBlobPermissions.Read
});
var sourceBlobSasUrl = sourceBlob.Uri.AbsoluteUri + sourceBlobSas;
var targetAccount = new CloudStorageAccount(new StorageCredentials("target-account-name", "target-account-key"), true);
var targetContainer = targetAccount.CreateCloudBlobClient().GetContainerReference("target-container");
var targetBlob = targetContainer.GetBlockBlobReference("blob-name");
var copyId = targetBlob.StartCopy(new Uri(sourceBlobSasUrl), null, null);
}
Related
i use Azure Blob Storage Client Libary v11 for .Net.
I wrote a program that our customers can use to upload files. I generate a URL with a SAS Token (valid for x Days) for our customer and the customer can upload files using the program. Here is an example url:
https://storage.blob.core.windows.net/123456789?sv=2019-07-07&sr=c&si=mypolicy&sig=ASDH845378ddsaSDdase324234234rASDSFR
How can I find out whether the SAS token is still valid before the upload is started?
Update:
I have no se claim in my url.
Here is my code to generate the url:
var policyName = "mypolicy";
string containerName = "123456789";
// Retrieve storage account information from connection string
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(GetSecret());
// Create a blob client for interacting with the blob service.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Create a container for organizing blobs within the storage account.
CloudBlobContainer container = blobClient.GetContainerReference(containerName);
try
{
// The call below will fail if the sample is configured to use the storage emulator in the connection string, but
// the emulator is not running.
// Change the retry policy for this call so that if it fails, it fails quickly.
BlobRequestOptions requestOptions = new BlobRequestOptions() { RetryPolicy = new NoRetry() };
await container.CreateIfNotExistsAsync(requestOptions, null);
}
catch (StorageException ex)
{
MessageBox.Show(ex.Message, Application.ProductName, MessageBoxButtons.OK, MessageBoxIcon.Error);
return string.Empty;
}
// create the stored policy we will use, with the relevant permissions and expiry time
var storedPolicy = new SharedAccessBlobPolicy()
{
SharedAccessExpiryTime = DateTime.UtcNow.AddDays(7),
Permissions = SharedAccessBlobPermissions.Read |
SharedAccessBlobPermissions.Write |
SharedAccessBlobPermissions.List
};
// get the existing permissions (alternatively create new BlobContainerPermissions())
var permissions = container.GetPermissions();
// optionally clear out any existing policies on this container
permissions.SharedAccessPolicies.Clear();
// add in the new one
permissions.SharedAccessPolicies.Add(policyName, storedPolicy);
// save back to the container
container.SetPermissions(permissions);
// Now we are ready to create a shared access signature based on the stored access policy
var containerSignature = container.GetSharedAccessSignature(null, policyName);
// create the URI a client can use to get access to just this container
return container.Uri + containerSignature;
I have found a solution myself. This blog describes two different ShardedAccessSignatures. I have adapted the code so that I now also have the se claim in my URL.
Solution:
protected void GetSharedAccessSignature(
String containerName, String blobName)
{
CloudStorageAccount cloudStorageAccount =
CloudStorageAccount.FromConfigurationSetting(“DataConnectionString”);
CloudBlobClient cloudBlobClient = cloudStorageAccount.CreateCloudBlobClient();
CloudBlobContainer cloudBlobContainer =
new CloudBlobContainer(containerName, cloudBlobClient);
CloudBlockBlob cloudBlockBlob =
cloudBlobContainer.GetBlockBlobReference(blobName);
SharedAccessPolicy sharedAccessPolicy = new SharedAccessPolicy();
sharedAccessPolicy.Permissions = SharedAccessPermissions.Read;
sharedAccessPolicy.SharedAccessStartTime = DateTime.UtcNow.AddMinutes(-10);
sharedAccessPolicy.SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(40);
String sharedAccessSignature1 =
cloudBlockBlob.GetSharedAccessSignature(sharedAccessPolicy);
String sharedAccessSignature2 =
cloudBlockBlob.GetSharedAccessSignature( new SharedAccessPolicy(), “adele”);
}
The sharedAccessSignature1 contains the se claim.
In my code of my initial questions I had used the sharedAccessSignature2.
We have a parquet formatfile (500 mb) which is located in Azure blob.How to read the file directly from blob and save in memory of c# ,say eg:Datatable.
I am able to read parquet file which is physically located in folder using the below code.
public void ReadParqueFile()
{
using (Stream fileStream = System.IO.File.OpenRead("D:/../userdata1.parquet"))
{
using (var parquetReader = new ParquetReader(fileStream))
{
DataField[] dataFields = parquetReader.Schema.GetDataFields();
for (int i = 0; i < parquetReader.RowGroupCount; i++)
{
using (ParquetRowGroupReader groupReader = parquetReader.OpenRowGroupReader(i))
{
DataColumn[] columns = dataFields.Select(groupReader.ReadColumn).ToArray();
DataColumn firstColumn = columns[0];
Array data = firstColumn.Data;
//int[] ids = (int[])data;
}
}
}
}
}
}
(I am able to read csv file directly from blob using sourcestream).Please kindly suggest a fastest method to read the parquet file directly from blob
Per my experience, the solution to directly read the parquet file from blob is first to generate the blob url with sas token and then to get the stream of HttpClient from the url with sas and finally to read the http response stream via ParquetReader.
First, please refer to the sample code below of the section Create a service SAS for a blob of the offical document Create a service SAS for a container or blob with .NET using Azure Blob Storage SDK for .NET Core.
private static string GetBlobSasUri(CloudBlobContainer container, string blobName, string policyName = null)
{
string sasBlobToken;
// Get a reference to a blob within the container.
// Note that the blob may not exist yet, but a SAS can still be created for it.
CloudBlockBlob blob = container.GetBlockBlobReference(blobName);
if (policyName == null)
{
// Create a new access policy and define its constraints.
// Note that the SharedAccessBlobPolicy class is used both to define the parameters of an ad hoc SAS, and
// to construct a shared access policy that is saved to the container's shared access policies.
SharedAccessBlobPolicy adHocSAS = new SharedAccessBlobPolicy()
{
// When the start time for the SAS is omitted, the start time is assumed to be the time when the storage service receives the request.
// Omitting the start time for a SAS that is effective immediately helps to avoid clock skew.
SharedAccessExpiryTime = DateTime.UtcNow.AddHours(24),
Permissions = SharedAccessBlobPermissions.Read | SharedAccessBlobPermissions.Write | SharedAccessBlobPermissions.Create
};
// Generate the shared access signature on the blob, setting the constraints directly on the signature.
sasBlobToken = blob.GetSharedAccessSignature(adHocSAS);
Console.WriteLine("SAS for blob (ad hoc): {0}", sasBlobToken);
Console.WriteLine();
}
else
{
// Generate the shared access signature on the blob. In this case, all of the constraints for the
// shared access signature are specified on the container's stored access policy.
sasBlobToken = blob.GetSharedAccessSignature(null, policyName);
Console.WriteLine("SAS for blob (stored access policy): {0}", sasBlobToken);
Console.WriteLine();
}
// Return the URI string for the container, including the SAS token.
return blob.Uri + sasBlobToken;
}
Then to get the http response stream of HttpClient from the url with sas token .
var blobUrlWithSAS = GetBlobSasUri(container, blobName);
var client = new HttpClient();
var stream = await client.GetStreamAsync(blobUrlWithSAS);
Finally to read it via ParquetReader, the code comes from Reading Data of GitHub repo aloneguid/parquet-dotnet.
var options = new ParquetOptions { TreatByteArrayAsString = true };
var reader = new ParquetReader(stream, options);
I am trying to rename a container name for users when a condition is met. I made some research and found out that there is no rename function for containers in azure blob storage. But there is a way to accomplish this by copying the files and deleting it after copy. Below is the code I made.
string ContainerName = "old-container-name-user1";
string NewContainerName = "new-container-name-user2"
CloudStorageAccount sourceAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));
CloudBlobClient sourceblobClient = sourceAccount.CreateCloudBlobClient();
CloudBlobContainer sourceBlobContainer = sourceblobClient.GetContainerReference(ContainerName);
CloudBlobContainer destBlobContainer = sourceblobClient.GetContainerReference(NewContainerName);
CloudBlockBlob blobOriginal = sourceBlobContainer.GetBlockBlobReference(ContainerName);
CloudBlockBlob blobNew = destBlobContainer.GetBlockBlobReference(NewContainerName);
blobNew.StartCopy(blobOriginal);
blobOriginal.Delete();
When I execute this code I got an error message. Below is the error.
Exception User-Unhandled
Microsoft.WindowsAzure.Storage.StorageException:'The remote server
returned an error: (404) Not Found.'
Inner Exception WebException: The remote server returned an error:
(404) Not Found.
When I also try "blobNew.StartCopyAsync(blobOriginal)" the code just goes through but when I check the containers in azure there is no container created. What do you think the problem is? Any tips on how to improve my code? Delete function does not work also.
UPDATE
I update my code and was able to copy the files from other to container to the new one. Below is the code.
string ContainerName = "old-container-name-user1"
string NewContainerName = "new-container-name-user2"
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference(ContainerName);
CloudBlobContainer destcontainer = blobClient.GetContainerReference(NewContainerName);
destcontainer.CreateIfNotExists(BlobContainerPublicAccessType.Blob);
IEnumerable<IListBlobItem> IE = container.ListBlobs(useFlatBlobListing: true);
foreach (IListBlobItem item in IE)
{
CloudBlockBlob blob = (CloudBlockBlob)item;
CloudBlockBlob destBlob = destcontainer.GetBlockBlobReference(blob.Name);
destBlob.StartCopyAsync(new Uri(GetSharedAccessUri(blob.Name, container)));
}
AccessURI Method
private static string GetSharedAccessUri(string blobName, CloudBlobContainer container)
{
DateTime toDateTime = DateTime.Now.AddMinutes(60);
SharedAccessBlobPolicy policy = new SharedAccessBlobPolicy
{
Permissions = SharedAccessBlobPermissions.Read,
SharedAccessStartTime = null,
SharedAccessExpiryTime = new DateTimeOffset(toDateTime)
};
CloudBlockBlob blob = container.GetBlockBlobReference(blobName);
string sas = blob.GetSharedAccessSignature(policy);
return blob.Uri.AbsoluteUri + sas;
}
Now its working but another problem shows up. It says
System.InvalidCastException: 'Unable to cast object of type
'Microsoft.WindowsAzure.Storage.Blob.CloudBlobDirectory' to type
'Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob'.'
But this will be posted in another question. Thanks to our friend Gauriv for pointing out my problem. Check his answer below.
Update 2
By adding useFlatBlobListing: true in IEnumerable<IListBlobItem> IE = container.ListBlobs(blobListingDetails: BlobListingDetails.Metadata); I was able to fix my problem. This line of code was put in my display.
Final Code
IEnumerable<IListBlobItem> IE = container.ListBlobs(blobListingDetails: BlobListingDetails.Metadata, useFlatBlobListing: true);
If you look at your code for creating source and destination blob:
CloudBlockBlob blobOriginal = sourceBlobContainer.GetBlockBlobReference(ContainerName);
CloudBlockBlob blobNew = destBlobContainer.GetBlockBlobReference(NewContainerName);
You'll notice that you're passing the names of the blob container and not the name of the blob. Because you don't have a blob in the container by the name of the container, you're getting 404 error.
To copy a blob container, what you have to do is list all blobs from the source container and then copy them individually in the destination container. Once all the blobs have been copied, you can delete the source container.
If you want, you can use Microsoft's Storage Explorer to achieve "rename container" functionality. It also works by copying blobs from old container to the renamed container and then deletes the old container.
static void RenameContainer()
{
var connectionString = "DefaultEndpointsProtocol=https;AccountName=account-name;AccountKey=account-key";
var storageAccount = CloudStorageAccount.Parse(connectionString);
var client = storageAccount.CreateCloudBlobClient();
var sourceContainer = client.GetContainerReference("source-container");
var targetContainer = client.GetContainerReference("target-container");
targetContainer.CreateIfNotExists();//Create target container
BlobContinuationToken continuationToken = null;
do
{
Console.WriteLine("Listing blobs. Please wait...");
var blobsResult = sourceContainer.ListBlobsSegmented(prefix: "", useFlatBlobListing: true, blobListingDetails: BlobListingDetails.All, maxResults: 1000, currentToken: continuationToken, options: new BlobRequestOptions(), operationContext: new OperationContext());
continuationToken = blobsResult.ContinuationToken;
var items = blobsResult.Results;
foreach (var item in items)
{
var blob = (CloudBlob)item;
var targetBlob = targetContainer.GetBlobReference(blob.Name);
Console.WriteLine(string.Format("Copying \"{0}\" from \"{1}\" blob container to \"{2}\" blob container.", blob.Name, sourceContainer.Name, targetContainer.Name));
targetBlob.StartCopy(blob.Uri);
}
} while (continuationToken != null);
Console.WriteLine("Deleting source blob container. Please wait.");
//sourceContainer.DeleteIfExists();
Console.WriteLine("Rename container operation complete. Press any key to terminate the application.");
}
You can rename containers with Microsoft's "Microsoft Azure Storage Explorer" (after version 0.8.3).
Actual Answer:
Regarding your error message, If the client application receives an HTTP 404 (Not found) message from the server, this means that the object the client was attempting to use does not exist in the storage service. There are several possible reasons for this, such as:
· The client or another process previously deleted the object (Make sure name is correct)
· A Shared Access Signature (SAS) authorization issue
· Client-side code does not have permission to access the object
· Network failure
In order to identify the issue in detail, you can add a try/catch and see the actual error
When I tried to download the image file from the online azure blob, it throws exception as "The given path's format is not supported". My code block is below:
StorageCredentials creds = new StorageCredentials(accountName, accountKey);
CloudStorageAccount account = new CloudStorageAccount(creds, useHttps: true);
CloudBlobClient client = account.CreateCloudBlobClient();
container = client.GetContainerReference(blobName);
CloudBlockBlob blockBlob = container.GetBlockBlobReference(MyPath);
await blockBlob.DownloadToFileAsync(Path, FileMode.OpenOrCreate);
using (var fileStream = System.IO.File.Create(Path))
{
await blockBlob.DownloadToStreamAsync(fileStream);
}
If I provide the local path as "c:\users\Joy\Downloads" in path,as like below:
var localPath = #"C:\Users\Joy \Downloads\user.jpg" ;
await blockBlob.DownloadToFileAsync(localPath, FileMode.OpenOrCreate);
using (var fileStream = System.IO.File.Create(localPath))
{
await blockBlob.DownloadToStreamAsync(fileStream);
}
It can be copied into corresponding location. But I couldn't download the file in my custom location.
According to your description, I enabled public read access to my blobs to check this issue. I created a console application and you could refer to the following code snippet for downloading the file and maintain the virtual directory in your local file system as follows:
CloudBlockBlob blockBlob = new CloudBlockBlob(new Uri("https://brucchstorage.blob.core.windows.net/images/2017/11/28/lake.jpeg"));
var localPath = Path.Combine(/*your custom root folder for storing file(s)*/AppDomain.CurrentDomain.BaseDirectory,$"downloads\\{blockBlob.Name}"); //blockBlob.Name =2017/11/28/lake.jpeg
var rootDir = new FileInfo(localPath).Directory;
if (!rootDir.Exists) //make sure the parent directory exists
rootDir.Create();
await blockBlob.DownloadToFileAsync(localPath,FileMode.Create);
//OR
using (var fs = new FileStream(localPath, FileMode.Create))
{
await blockBlob.DownloadToStreamAsync(fs);
}
Result:
Moreover, you could construct the CloudBlockBlob instance with the Uri contains the SAS token as follows:
https://brucchstorage.blob.core.windows.net/images/2017/11/28/lake.jpeg?st=2017-11-28T06%3A28%3A00Z&se=2017-11-29T06%3A28%3A00Z&sp=r&sv=2015-12-11&sr=b&sig=15NAaRB43C%2BniIZZe8gAvFl7LY%2BS6K7DNyjLflpvgBg%3D
More details, you could follow here.
I started encrypting my Azure blobs. Now I am occasionally getting 500 - The request timed out on blob operations (on both downloading and uploading blobs) after the request halts for about 30 seconds or so. After one of these timeouts, no other blob operation will work through the app, unless I restart my Azure website. Once restarted, everything runs as expected for a while.
Example: If I access an encrypted image through my application (I'm using the WebAPI to pull it and display it to the user) it shows up fine, but then if I try and access the same file hours later, the request halts and eventually times out. After that, I get the same issue while accessing any other file through my web app. However, if I access the direct url of the blob, then I can access the file (even though it's encrypted and therefore useless).
I cannot say with certainty what is causing this and when would the issue start occurring as I am not the only person accessing the app, so there's a good chance that the issue might have started before my failed request. Also, I never had issues as such before encryption was applied, nor did I have issues while testing encryption locally.
Any idea why this is happening, or maybe how can I prevent this? I am attaching my code below if it helps:
public async Task<Tuple<string, string>> UploadToStorage(CloudBlobContainer container, Stream stream, string reference, string contentType, byte[] byteArray = null) {
var blockBlob = container.GetBlockBlobReference(reference);
blockBlob.Properties.ContentType = contentType;
var cloudResolver = new KeyVaultKeyResolver(GetToken);
var rsa = await cloudResolver.ResolveKeyAsync(new BlobConfig().BlobKeyVault, CancellationToken.None);
var policy = new BlobEncryptionPolicy(rsa, null);
var options = new BlobRequestOptions { EncryptionPolicy = policy };
if (byteArray != null) blockBlob.UploadFromByteArray(byteArray, 0, byteArray.Length, null, options);
else blockBlob.UploadFromStream(stream, stream.Length, null, options);
return Tuple.Create(new Config().BaseUrl + "/api/blobs/" + container.Name + "/" + reference, blockBlob.Properties.ContentType);
}
public BlobDto DownloadBlob(string container, string filename) {
var account = CloudStorageAccount.Parse(new BlobConfig().StorageConnectionString);
var blobClient = account.CreateCloudBlobClient();
var blobContainer = blobClient.GetContainerReference(container);
var blob = blobContainer.GetBlockBlobReference(filename);
var cloudResolver = new KeyVaultKeyResolver(GetToken);
var policy = new BlobEncryptionPolicy(null, cloudResolver);
var options = new BlobRequestOptions { EncryptionPolicy = policy };
var m = new MemoryStream();
blob.DownloadToStream(m, null, options);
return new BlobDto { Blob = m.ToArray(), BlobContentType = blob.Properties.ContentType };
}
private async static Task<string> GetToken(string authority, string resource, string scope) {
var config = new BlobConfig();
var clientCredential = new ClientCredential(config.BlobClientId, config.BlobClientSecret);
var authContext = new AuthenticationContext(authority);
var result = await authContext.AcquireTokenAsync(resource, clientCredential);
if (result == null) throw new InvalidOperationException("Failed to obtain the access token");
return result.AccessToken;
}
I replaced all the synchronous blob related calls with asynchronous calls, particularly the downloading call. Also changed from stream to bytearray, but I don't think that's relevant:
await blob.DownloadToByteArrayAsync(byteArray, 0, null, options, null);