Develop Reference

Total progress for multiple async uploads via BlobClient.UploadAsync()

I tested and found that for uploading a folder full of files to an Azure Blob Storage container, the fastest method (with the v12 Azure Storage SDK) involves using a Queue (Queue<Task<Response<BlobContentInfo>>>) and firing off an UploadAsync() for each file, ending in a Task.WhenAll(). The blob client coordinates max concurrency, etc. as configured. However, I need to show total progress across all files, and when I use the same ProgressHandler for each blob client, the Progress<long>() event only returns total bytes uploaded per file, without any context for the underlying file whose progress is being reported. How can I get the cumulative progress of multiple async uploads?
using Azure;
using Azure.Storage;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;
public async Task UploadBlobs(string sourcePath)
{
var blobContainerClient = new BlobContainerClient(connectionString, containerName);
var tasks = new Queue<Task<Response<BlobContentInfo>>>();
var progressHandler = new Progress<long>();
progressHandler.ProgressChanged += UploadProgressChanged;
var options = new BlobUploadOptions()
{
ProgressHandler = progressHandler,
TransferOptions = new StorageTransferOptions()
{
MaximumConcurrency = Environment.ProcessorCount * 2,
MaximumTransferSize = 50 * 1024 * 1024
}
};
foreach (string filePath in Directory.GetFiles(sourcePath))
{
string fileName = Path.GetFileName(filePath);
Console.WriteLine($"Uploading {fileName}\r\n");
var blobClient = blobContainerClient.GetBlobClient(fileName);
tasks.Enqueue(blobClient.UploadAsync(filePath, options));
}
await Task.WhenAll(tasks);
}
private void UploadProgressChanged(object sender, long bytesUploaded)
{
// This handles every progress change for every file, but with no file context info!
}

One possible solution to track individual blob's upload progress is to create your own progress handler and pass BlobClient to it.
Here's a rudimentary implementation of this:
public class BlobUploadProgressChange : Progress<long>
{
private readonly BlobClient _blobClient;
public BlobUploadProgressChange(BlobClient blobClient) : base()
{
_blobClient = blobClient;
}
public BlobClient BlobClient
{
get { return _blobClient; }
}
}
This is how you would modify the UploadProgressChanged event handler:
void UploadProgressChanged(object? sender, long bytesUploaded)
{
BlobUploadProgressChange item = (BlobUploadProgressChange) sender;
Console.WriteLine(item.BlobClient.Name);
Console.WriteLine($"Bytes uploaded: {bytesUploaded}");
Console.WriteLine("====================================");
}
and this is how you can use it:
foreach (string filePath in Directory.GetFiles(sourcePath))
{
string fileName = Path.GetFileName(filePath);
Console.WriteLine($"Uploading {fileName}\r\n");
var blobClient = blobContainerClient.GetBlobClient(fileName);
var progressHandler = new BlobUploadProgressChange(blobClient);
progressHandler.ProgressChanged += UploadProgressChanged;
var options = new BlobUploadOptions()
{
ProgressHandler = progressHandler
};
tasks.Enqueue(blobClient.UploadAsync(filePath, options));
}

How to create zip file in memory?

I have to create a zip file from set of urls. and it should have a proper folder structure.
So i tried like
public async Task<byte[]> CreateZip(Guid ownerId)
{
try
{
string startPath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "zipFolder");//base folder
if (Directory.Exists(startPath))
{
DeleteAllFiles(startPath);
Directory.Delete(startPath);
}
Directory.CreateDirectory(startPath);
string zipPath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, $"{ownerId.ToString()}"); //folder based on ownerid
if (Directory.Exists(zipPath))
{
DeleteAllFiles(zipPath);
Directory.Delete(zipPath);
}
Directory.CreateDirectory(zipPath);
var attachemnts = await ReadByOwnerId(ownerId);
attachemnts.Data.ForEach(i =>
{
var fileLocalPath = $"{startPath}\\{i.Category}";
if (!Directory.Exists(fileLocalPath))
{
Directory.CreateDirectory(fileLocalPath);
}
using (var client = new WebClient())
{
client.DownloadFile(i.Url, $"{fileLocalPath}//{i.Flags ?? ""}_{i.FileName}");
}
});
var zipFilename = $"{zipPath}//result.zip";
if (File.Exists(zipFilename))
{
File.Delete(zipFilename);
}
ZipFile.CreateFromDirectory(startPath, zipFilename, CompressionLevel.Fastest, true);
var result = System.IO.File.ReadAllBytes(zipFilename);
return result;
}
catch (Exception ex)
{
var a = ex;
return null;
}
}
currently im writing all files in my base directory(may be not a good idea).corrently i have to manually delete all folders and files to avoid exception/unwanted files. Can everything be written in memory?
What changes required to write all files and folder structure in memory?

No you can't. Not with the built in Dotnet any way.
As per my comment I would recommend storing the files in a custom location based on a Guid or similar. Eg:
"/xxxx-xxxx-xxxx-xxxx/Folder-To-Zip/....".
This would ensure you could handle multiple requests with the same files or similar file / folder names.
Then you just have to cleanup and delete the folder again afterwards so you don't run out of space.

Hope the below code does the job.
public async Task<byte[]> CreateZip(Guid ownerId)
{
try
{
string startPath = Path.Combine(Path.GetTempPath(), $"{Guid.NewGuid()}_zipFolder");//folder to add
Directory.CreateDirectory(startPath);
var attachemnts = await ReadByOwnerId(ownerId);
attachemnts.Data = filterDuplicateAttachments(attachemnts.Data);
//filtering youtube urls
attachemnts.Data = attachemnts.Data.Where(i => !i.Flags.Equals("YoutubeUrl", StringComparison.OrdinalIgnoreCase)).ToList();
attachemnts.Data.ForEach(i =>
{
var fileLocalPath = $"{startPath}\\{i.Category}";
if (!Directory.Exists(fileLocalPath))
{
Directory.CreateDirectory(fileLocalPath);
}
using (var client = new WebClient())
{
client.DownloadFile(i.Url, $"{fileLocalPath}//{i.Flags ?? ""}_{i.FileName}");
}
});
using (var ms = new MemoryStream())
{
using (var zipArchive = new ZipArchive(ms, ZipArchiveMode.Create, true))
{
System.IO.DirectoryInfo di = new DirectoryInfo(startPath);
var allFiles = di.GetFiles("",SearchOption.AllDirectories);
foreach (var attachment in allFiles)
{
var file = File.OpenRead(attachment.FullName);
var type = attachemnts.Data.Where(i => $"{ i.Flags ?? ""}_{ i.FileName}".Equals(attachment.Name, StringComparison.OrdinalIgnoreCase)).FirstOrDefault();
var entry = zipArchive.CreateEntry($"{type.Category}/{attachment.Name}", CompressionLevel.Fastest);
using (var entryStream = entry.Open())
{
file.CopyTo(entryStream);
}
}
}
var result = ms.ToArray();
return result;
}
}
catch (Exception ex)
{
var a = ex;
return null;
}
}

Upload file into google drive folder using Xamarin.Android

I want to create file inside a specific folder in google drive ( not the default location ) using Xamarin.Andriod
I'm using the below code
MetadataChangeSet changeSetfile = new MetadataChangeSet.Builder()
.SetTitle("Test.jpg")
.SetMimeType("image/jpeg")
.Build();
DriveClass.DriveApi
.GetRootFolder(_googleApiClient)
.CreateFile(_googleApiClient, changeSetfile, contentResults.DriveContents);

Implement GoogleApiClient.IConnectionCallbacks
Obtain a GoogleApiClient with DriveClass.API and DriveClass.ScopeFile
GoogleApiClient Example:
if (_googleApiClient == null) // _googleApiClient is a class level variable
{
_googleApiClient = new GoogleApiClient.Builder(this)
.AddApi(DriveClass.API)
.AddScope(DriveClass.ScopeFile)
.AddConnectionCallbacks(this)
.AddOnConnectionFailedListener(onConnectionFailed)
.Build();
}
if (!_googleApiClient.IsConnected)
_googleApiClient.Connect();
Once connected, Query for folder, create if needed and then "write" a file to it.
Folder and File Example:
var folderName = "StackOverflow";
using (var driveId = DriveClass.DriveApi.GetRootFolder(_googleApiClient))
using (var query = new QueryClass.Builder().AddFilter(Filters.And(Filters.Eq(SearchableField.Title, folderName), Filters.Eq(SearchableField.Trashed, false))).Build())
using (var metaBufferResult = await driveId.QueryChildrenAsync(_googleApiClient, query))
{
if (metaBufferResult.Status.IsSuccess)
{
DriveId folderId = null;
foreach (var metaData in metaBufferResult.MetadataBuffer)
{
if (metaData.IsFolder && metaData.Title == folderName)
{
folderId = metaData.DriveId;
break;
}
}
IDriveFolder driveFolder = null;
switch (folderId)
{
case null: // if folder not found, create it and fall through to default
using (var folderChangeSet = new MetadataChangeSet.Builder().SetTitle(folderName).Build())
using (var folderResult = await driveId.CreateFolderAsync(_googleApiClient, folderChangeSet))
{
if (!folderResult.Status.IsSuccess)
{
Log.Error(TAG, folderResult.Status.StatusMessage);
break;
}
driveFolder = folderResult.DriveFolder;
}
goto default;
default:
driveFolder = driveFolder ?? folderId.AsDriveFolder();
// create your file in the IDriveFolder obtained,
using (var contentResults = await DriveClass.DriveApi.NewDriveContentsAsync(_googleApiClient))
{
if (contentResults.Status.IsSuccess)
{
using (var writer = new OutputStreamWriter(contentResults.DriveContents.OutputStream))
{
writer.Write("StackOverflow Rocks");
using (var changeSet = new MetadataChangeSet.Builder()
.SetTitle("StackOverflow Rocks")
.SetStarred(true)
.SetMimeType("text/plain")
.Build())
using (var driveFileResult = await driveFolder.CreateFileAsync(_googleApiClient, changeSet, contentResults.DriveContents))
{
if (driveFileResult.Status.IsSuccess)
Log.Debug(TAG, "File created, open https://drive.google.com to review it");
else
Log.Error(TAG, driveFileResult.Status.StatusMessage);
}
}
}
}
driveFolder.Dispose();
break;
}
folderId?.Dispose();
}
else
{
Log.Error(TAG, metaBufferResult.Status.StatusMessage);
}
}
Notes:
Do this on a background thread
Drive allows multiple files/folders with same name (Title)
Query for existing files if you want to replace one
Query for existing folders unless you really what multiple folders with the same Title
Folders and files in the Trash are returned queries unless excluded.
Make use of Using blocks and Dispose to avoid leaks

How to write the contents of a Onedrive<Item> to a local file

In my app, I am using OneDrive to keep data in sync. I am successfully writing the file to OneDrive, but am having no luck replacing the local outdated data with the newer OneDrive data.
My current method, which completes without throwing an exception, does not return the same text data that the file on OneDrive contains.
Goal of the method is to compare the datemodified to the OneDrive file to the local file, and if OneDrive is newer, write the contents of the OndeDrive file to the local StorageFile, and then return it to be de-serialized.
private async Task<string> GetSavedDataFileAsync(string filename)
{
string filepath = _appFolder + #"\" + KOWGame + #"\" + filename;
StorageFile localread;
BasicProperties localprops = null;
string txt;
try
{
localread = await local.GetFileAsync(filepath);
localprops = await localread.GetBasicPropertiesAsync();
}
catch (FileNotFoundException)
{ localread = null; }
if (_userDrive != null)
{
if (_userDrive.IsAuthenticated)
{
try
{
Item item = await _userDrive.Drive.Special.AppRoot.ItemWithPath(filepath).Request().GetAsync();
if (item != null)
{
DateTimeOffset drivemodified = (DateTimeOffset)item.FileSystemInfo.LastModifiedDateTime;
if (localprops != null)
{
if (drivemodified > localprops.DateModified)
{
Stream stream = await localread.OpenStreamForWriteAsync();
using (stream)
{ await _userDrive.Drive.Special.AppRoot.ItemWithPath(filepath).Request().GetAsync(); }
}
}
}
}
catch (OneDriveException e)
{
if (e.IsMatch(OneDriveErrorCode.ActivityLimitReached.ToString()))
{ string stop; }
}
}
}
if (localread == null) return string.Empty;
txt = await FileIO.ReadTextAsync(localread);
return txt;
}
I tried to reverse engineer another answer I found on Stack regarding writing a StorageFile to OneDrive, in that I needed to open the stream of the local file, but I doesn't appear to be working properly.

To get the content of a OneDrive item, we need use following method:
var contentStream = await _userDrive.Drive.Special.AppRoot.ItemWithPath(filepath).Content.Request().GetAsync();
While using
await _userDrive.Drive.Special.AppRoot.ItemWithPath(filepath).Request().GetAsync();
you are getting the OneDrive Item not its content.
So you can change your code like following to write the content of a Onedrive item to a local file:
if (drivemodified > localprops.DateModified)
{
using (var stream = await localread.OpenStreamForWriteAsync())
{
using (var contentStream = await _userDrive.Drive.Special.AppRoot.ItemWithPath(filepath).Content.Request().GetAsync())
{
contentStream.CopyTo(stream);
}
}
}

How to check if Azure Blob file Exists or Not

I want to check a particular file exist in Azure Blob Storage. Is it possible to check by specifying it's file name? Each time i got File Not Found Error.

var blob = client.GetContainerReference(containerName).GetBlockBlobReference(blobFileName);
if (blob.Exists())
//do your stuff

This extension method should help you:
public static class BlobExtensions
{
public static bool Exists(this CloudBlob blob)
{
try
{
blob.FetchAttributes();
return true;
}
catch (StorageClientException e)
{
if (e.ErrorCode == StorageErrorCode.ResourceNotFound)
{
return false;
}
else
{
throw;
}
}
}
}
Usage:
static void Main(string[] args)
{
var blob = CloudStorageAccount.DevelopmentStorageAccount
.CreateCloudBlobClient().GetBlobReference(args[0]);
// or CloudStorageAccount.Parse("<your connection string>")
if (blob.Exists())
{
Console.WriteLine("The blob exists!");
}
else
{
Console.WriteLine("The blob doesn't exist.");
}
}
http://blog.smarx.com/posts/testing-existence-of-a-windows-azure-blob

With the updated SDK, once you have the CloudBlobReference you can call Exists() on your reference.
UPDATE
The relevant documentation has been moved to https://learn.microsoft.com/en-us/dotnet/api/microsoft.windowsazure.storage.blob.cloudblob.exists?view=azurestorage-8.1.3#Microsoft_WindowsAzure_Storage_Blob_CloudBlob_Exists_Microsoft_WindowsAzure_Storage_Blob_BlobRequestOptions_Microsoft_WindowsAzure_Storage_OperationContext_
My implementation using WindowsAzure.Storage v2.0.6.1
private CloudBlockBlob GetBlobReference(string filePath, bool createContainerIfMissing = true)
{
CloudBlobClient client = _account.CreateCloudBlobClient();
CloudBlobContainer container = client.GetContainerReference("my-container");
if ( createContainerIfMissing && container.CreateIfNotExists())
{
//Public blobs allow for public access to the image via the URI
//But first, make sure the blob exists
container.SetPermissions(new BlobContainerPermissions { PublicAccess = BlobContainerPublicAccessType.Blob });
}
CloudBlockBlob blob = container.GetBlockBlobReference(filePath);
return blob;
}
public bool Exists(String filepath)
{
var blob = GetBlobReference(filepath, false);
return blob.Exists();
}

Use the ExistsAsync method of CloudBlockBlob.
bool blobExists = await cloudBlobContainer.GetBlockBlobReference("<name of blob>").ExistsAsync();

Using Microsoft.WindowsAzure.Storage.Blob version 4.3.0.0, the following code should work (there are a lot of breaking changes with older versions of this assembly):
Using container/blob name, and the given API (seems now Microsoft have actualy implemented this):
return _blobClient.GetContainerReference(containerName).GetBlockBlobReference(blobName).Exists();
Using blob URI (workaround):
try
{
CloudBlockBlob cb = (CloudBlockBlob) _blobClient.GetBlobReferenceFromServer(new Uri(url));
cb.FetchAttributes();
}
catch (StorageException se)
{
if (se.Message.Contains("404") || se.Message.Contains("Not Found"))
{
return false;
}
}
return true;
(Fetch attributes will fail if the blob is not existing. Dirty, I know :)

Using the new package Azure.Storage.Blobs
BlobServiceClient blobServiceClient = new BlobServiceClient("YourStorageConnectionString");
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient("YourContainerName");
BlobClient blobClient = containerClient.GetBlobClient("YourFileName");
then check if exists
if (blobClient.Exists()){
//your code
}

With the latest version of SDK, you need to use the ExistsAsync Method,
public async Task<bool> FileExists(string fileName)
{
return await directory.GetBlockBlobReference(fileName).ExistsAsync();
}
Here is the code sample.

This complete example is here to help.
public class TestBlobStorage
{
public bool BlobExists(string containerName, string blobName)
{
BlobServiceClient blobServiceClient = new BlobServiceClient(#"<connection string here>");
var container = blobServiceClient.GetBlobContainerClient(containerName);
var blob = container.GetBlobClient(blobName);
return blob.Exists();
}
}
then you can test in main
static void Main(string[] args)
{
TestBlobStorage t = new TestBlobStorage();
Console.WriteLine("blob exists: {0}", t.BlobExists("image-test", "AE665.jpg"));
Console.WriteLine("--done--");
Console.ReadLine();
}
Important I found the file names are case sensitive

## dbutils.widgets.get to call the key-value from data bricks job
storage_account_name= dbutils.widgets.get("storage_account_name")
container_name= dbutils.widgets.get("container_name")
transcripts_path_intent= dbutils.widgets.get("transcripts_path_intent")
# Read azure blob access key from dbutils
storage_account_access_key = dbutils.secrets.get(scope = "inteliserve-blob-storage-secret-scope", key = "storage-account-key")
from azure.storage.blob import BlockBlobService
block_blob_service = BlockBlobService(account_name=storage_account_name, account_key=storage_account_access_key)
def blob_exists():
container_name2 = container_name
blob_name = transcripts_path_intent
exists=(block_blob_service.exists(container_name2, blob_name))
return exists
blobstat = blob_exists()
print(blobstat)