Develop Reference

Azure Storage Data Movement Library copy is much slower than AzCopy

I have next test setup:
One test Azure blob storage account
Local folder with ~3000 small files (200 bytes each)
When I execute azcopy command:
azcopy copy --recursive "c:\localDir\*" "https://BLOBConnectionString"
it takes ~2 seconds to copy data.
When I do next c# code:
ServicePointManager.Expect100Continue = false;
ServicePointManager.DefaultConnectionLimit = 32;
TransferManager.Configurations.ParallelOperations = 32;
var account = CloudStorageAccount.Parse("https://BLOBConnectionString");
CloudBlobClient client = account.CreateCloudBlobClient();
CloudBlobContainer container = client.GetContainerReference("container");
await container.CreateIfNotExistsAsync();
CloudBlobDirectory destinationBlob = container.GetDirectoryReference("data");
await TransferManager.UploadDirectoryAsync(#"c:\localDir\", destinationBlob);
It takes ~1 minute to copy same amount of data.
I expect to have approximately same latency results for c# code base.

I tried in my environment and got below results:
Code:
using Microsoft.Azure.Storage;
using Microsoft.Azure.Storage.Blob;
using Microsoft.Azure.Storage.DataMovement;
using System.ComponentModel;
namespace fastercpy
{
class program
{
public static void Main()
{
string storageConnectionString = "< Connection string >";
CloudStorageAccount account = CloudStorageAccount.Parse(storageConnectionString);
CloudBlobClient blobClient = account.CreateCloudBlobClient();
CloudBlobContainer blobContainer = blobClient.GetContainerReference("test");
blobContainer.CreateIfNotExists();
string sourcePath = "C:\\Users\\v-vsettu\\Documents\\Venkat";
CloudBlobDirectory destinationBlob = blobContainer.GetDirectoryReference("data");
TransferManager.Configurations.ParallelOperations = 64;
// Setup the transfer context and track the upoload progress
SingleTransferContext context = new SingleTransferContext();
context.ProgressHandler = new Progress<TransferStatus>((progress) =>
{
Console.WriteLine("Bytes uploaded: {0}", progress.BytesTransferred);
});
var task=TransferManager.UploadDirectoryAsync(sourcePath, destinationBlob);
task.Wait();
}
}
}
you have been used
TransferManager.Configurations.ParallelOperations = 32 ,So try to use TransferManager.Configurations.ParallelOperations = 64; in your code it will speed up process.
The Microsoft Azure Storage Data Movement Library was created for fast uploading, downloading, and copying of Azure Storage Blob and File.
Console:
Portal:
Reference:
Transfer data with the Data Movement library for .NET - Azure Storage | Microsoft Learn

Azure Blob Storage - how do I get the Blob Storage ID after uploading a blob?

I am using a C# Console Application (.NET Core 3.1) to read a load of image files from Azure Blob Storage and produce thumbnails of those images. The new images are to be saved back to Azure, and the Blob ID stored in our database. How do I find the ID of the items saved. Here is the command:
Azure.Response<BlobContentInfo> blobs = containerClient.UploadBlob(fileName, outStream);
I can't seem to find it in the returned object
https://learn.microsoft.com/en-us/dotnet/api/azure.storage.blobs.models.blobcontentinfo?view=azure-dotnet
My original pictures were created and save using PowerApps, where the control does indeed return the Blob ID - see below:
Set(
gblSentBlob,
AzureBlobStorage.CreateFile(
Text(gblAzureFileContainer),
GUID() & ".jpg",
camControl.Photo
)
);
If(
!IsEmpty(gblSentBlob),
Notify("Picture saved to Azure storage:" & gblSentBlob.DisplayName);
UpdateContext({locFileName: gblSentBlob.DisplayName});
UpdateContext({locAzureStorageID: Text(gblSentBlob.Id)}); // <- *** this is the Blob ID ***
UpdateContext({locSavedToAzure: true});
Here, the AzureBlobStorage.CreateFile function returns an object which contains the ID I am looking for.
How do I get this ID in my Console Application.
A typical Blob ID looks like this:
JTJmc2hpcmVibG9iY29udGFpbmVyJTJmNTk3MzQ4NGYtNGVhNy00NzJkLTkyMzQtYWIwNzM5NWNlOGRiLmpwZw==
I can then retrieve the images for display using the following (in PowerApps)
AzureBlobStorage.GetFileContent(ThisItem.BlobStorageID)
My full code:
var blobClient = containerClient.GetBlobClient(blobName);
using Stream stream = await blobClient.OpenReadAsync();
Image myImage = Image.FromStream(stream);
Image myThumbnail = PictureProcessor.returnThumbnail(myImage);
// now save this image
string guid = Guid.NewGuid().ToString();
string fileName = guid + ".jpg";
//create a memory stream ready for the rescaled image
Stream outStream = new MemoryStream();
myThumbnail.Save(outStream, System.Drawing.Imaging.ImageFormat.Jpeg);
Console.WriteLine(
"Length = {0}, Position = {1}\n",
outStream.Length.ToString(),
outStream.Position.ToString());
outStream.Position = 0;
Azure.Response<BlobContentInfo> blobs = containerClient.UploadBlob(fileName, outStream);
Console.WriteLine("blobs RETURN OBJECT: " + blobs.ToString());
Console.WriteLine("blobs GetRawResponse: " + blobs.GetRawResponse());
Console.ReadKey();

When I decoded this from Base64 and then into UTF-8:
JTJmc2hpcmVibG9iY29udGFpbmVyJTJmNTk3MzQ4NGYtNGVhNy00NzJkLTkyMzQtYWIwNzM5NWNlOGRiLmpwZw==
I got this:
%2fshireblobcontainer%2f5973484f-4ea7-472d-9234-ab07395ce8db.jpg
So your "Blob ID" appears to be the Base64-encoded representation of the UTF-8 (or 7-bit ASCII?) representation of the URL-encoded string value you're passing into AzureBlobStorage.CreateFile's second parameter.
So do this:
String powerAppsBlobId = #"JTJmc2hpcmVibG9iY29udGFpbmVyJTJmNTk3MzQ4NGYtNGVhNy00NzJkLTkyMzQtYWIwNzM5NWNlOGRiLmpwZw==";
Byte[] blobIdBytes = Convert.FromBase64String( powerAppsBlobId );
String urlEncodedBlobName = Encoding.UTF8.GetString( bytes );
String actualBlobName = Uri.UnescapeDataString( urlEncodedBlobName );
Console.WriteLine( actualBlobName )
This program will then print this:
/shireblobcontainer/5973484f-4ea7-472d-9234-ab07395ce8db.jpg

After posting my first answer (with the Base64 decoding) I took a look at the documentation for the Azure Blob connector for PowerApps and I see that the BlobMetadata.Name and/or BlobMetadata.Path values should contain the full blob-name too.
I don't know why you pointed to the documentation for Azure.Storage.Blobs.Models as that's not for use by PowerApps.
So a better idea is to store the Blob.MetadataPath value when you upload the blob from within PowerApps, so your Console application can get to it - this is just in case PowerApps changes their "algorithm" for generating those Base64 BlobIds (e.g. they could include the blob version date+time, or shared access signatures, for example).
So change your code to this:
Set(
gblSentBlob,
AzureBlobStorage.CreateFile(
Text(gblAzureFileContainer),
GUID() & ".jpg",
camControl.Photo
)
);
If(
!IsEmpty(gblSentBlob),
Notify("Picture saved to Azure storage:" & gblSentBlob.DisplayName);
UpdateContext({locFileName: gblSentBlob.DisplayName});
UpdateContext({locAzureStorageID: Text(gblSentBlob.Id)});
UpdateContext({locAzureStoragePath: Text(gblSentBlob.Path)}); // <--- Add this here
UpdateContext({locSavedToAzure: true});

blob stream Length property value always -1, no matter the file empty or not

I have a code that opens a stream for a blob on storage to read its content (steps described below), the blob is a Json file which is sometimes empty, it appears on azure portal with 0B, and can be downloaded and opened.
I check the blobStream.Length property to know if it's an empty blob or not for handling the file, for some reason no matter if the file empty or not the Length property is -1 and not 0.
I have read at some places that the -1 is the initialized value of the property, and that FetchAttributesAsync must be called in order to update the metadata .. and I read on Microsoft docs that calling OpenReadAsync automatically calls FetchAttributesAsync, mentioned in these docs under remarks:
OpenReadAsyncDocs
FetchAttributesAsyncDocs
not sure what's happening, whether the problem is with the FetchAttributes call or something with opening the stream, and why the Length is -1 always.
I use Azure.Storage.Blobs, to create a blob client with the connection string and return the blob stream to read its content
_blobServiceClient = new BlobServiceClient(connectionString);
blobStream = await _blobServiceClient.GetBlobContainerClient(container).GetBlobClient(blobName).OpenReadAsync(cancellationToken: cancellationToken);
on the stream returned from OpenReadAsync we check the length .. and handle the file:
if( blobStream.Length == 0 ) continue; < -- the length is -1 always

If you want to read the blob file as stream, BlobBaseClient.DownloadTo method will help.
BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
BlobContainerClient container = blobServiceClient.GetBlobContainerClient(containerName);
BlobClient blob = container.GetBlobClient(blobName);
MemoryStream s = new MemoryStream();
blob.DownloadTo(s);
Console.WriteLine(s.Length);
And there is another way to get the length of the blob's content with BlobProperties.ContentLength.
BlobClient blob = container.GetBlobClient(blobName);
var properties = blob.GetProperties();
Console.WriteLine(properties.Value.ContentLength);

Download Blob file into Memory Stream from Azure using C#

I am trying to read a blob from Azure using C# Console application. I can download the blob file to my local machine and read it but when I am trying to download it to memory stream it is throwing an error as attached.
I want to directly read my blob file instead of downloading it to local machine as I have to deploy it in Azure. Any suggestions on it?
var storageaccount = new CloudStorageAccount(new Microsoft.WindowsAzure.Storage.Auth.StorageCredentials(StorageAccountName, StorageAccountKey), true);
var blobclient = storageaccount.CreateCloudBlobClient();
var container = blobclient.GetContainerReference("");
// var blobs = container.ListBlobs();
var blockBlobReference = container.GetBlockBlobReference(sourceBlobFileName);
using (var memorystream = new MemoryStream())
{
blockBlobReference.DownloadToStream(memorystream);
}

I think the problem is because you're wrapping it in a using statement. Try the following:
var storageaccount = new CloudStorageAccount(new Microsoft.WindowsAzure.Storage.Auth.StorageCredentials(StorageAccountName, StorageAccountKey), true);
var blobclient = storageaccount.CreateCloudBlobClient();
var container = blobclient.GetContainerReference("");
// var blobs = container.ListBlobs();
var blockBlobReference = container.GetBlockBlobReference(sourceBlobFileName);
var memorystream = new MemoryStream();
blockBlobReference.DownloadToStream(memorystream);
byte[] content = memoryStream.ToArray();

I tried your code, but Blob downloaded successfully.
your screenshot just shows the variable in Visual Studio debugger, and it looks like the error doesn't actually occurred.
In screenshot, MemoryStream, CanTimeout property set to false.
So, it seems that ReadTimeout and WriteTimeout throw exceptions only because MemoryStream does not support them.
Reference:
Stream.ReadTimeout Property
Please note that the exception won't occurred until you use ReadTimeout and WriteTimeout properties.

Encrypting image data before uploading to azure blob storage

I have the following code that uploads an image to Azure blob storage. I would like to encrypt the image data before uploading to the blob. I already have a helper class for encrypting and decrypting that I can use by calling AESEncryption.Encrypt("plainText", "key", salt");
I'm just trying to figure out how tom integrate my encryption method into the code. Also, I'm guessing that once it's encrypted instead of calling blob.UploadFromFile() I will be calling blob.UploadFromByteArray().
public override Task ExecutePostProcessingAsync()
{
try
{
// Upload the files to azure blob storage and remove them from local disk
foreach (var fileData in this.FileData)
{
var filename = BuildFilename(Path.GetExtension(fileData.Headers.ContentDisposition.FileName.Trim('"')));
// Retrieve reference to a blob
var blob = _container.GetBlockBlobReference(filename);
blob.Properties.ContentType = fileData.Headers.ContentType.MediaType;
blob.UploadFromFile(fileData.LocalFileName, FileMode.Open);
File.Delete(fileData.LocalFileName);
Files.Add(new FileDetails
{
ContentType = blob.Properties.ContentType,
Name = blob.Name,
Size = blob.Properties.Length,
Location = blob.Uri.AbsoluteUri
});
}
}
catch (Exception ex)
{
throw ex;
}
return base.ExecutePostProcessingAsync();
}

As I see it, you could do it 3 ways:
Encrypt the file beforehand and then upload that encrypted file.
As you mentioned, you could read the file in byte array and then encrypt that byte array and upload it using UploadFromByteArray method.
Similar to #2 but instead of uploading byte array, you could rely on streams and upload using UploadFromStream method.