I want to get a specific block from an azure block blob with the blockId, is this even possible?
something like
var blockBlob = new BlockBlobClient(connectionString, containerName, blobName);
var blocklist = await GetBlobBlockList(blobName, cancellationToken);
var firstBlock = blocklist.First();
var memStream = new MemoryStream();
await blockBlob.DownloadStreamingAsync(memStream, firstBlock.Name);
I want to get a specific block from an azure block blob with the
blockId, is this even possible?
It should be possible to do so however it won't be as simple as you mentioned in your sample code.
Here's what you would need to do:
Fetch list of blocks. Each element in the list will have a block id and the size of the block.
Assuming you want to get data for block "n", what you will do is iterate over the list from 0 to n - 1 block and add the size of each block.
Next you would need to call DownloadRangeToStreamAsync(Stream, Nullable<Int64>, Nullable<Int64>), where your offset value will be the sum of the size of each block calculated in step 2 and length value will be the size of the block you wish to download.
You need to create a block blob below you can find the procedure:
you need to create a container under storage account as follows:
inside the container you can find the block blob under properties as shown below
To Get the block Id you can follow the code below:
code:
using System;
using System.Threading.Tasks;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;
class Program
{
static async Task Main(string[] args)
{
string connectionString = "DefaultEndpointsProtocol=https;AccountName=<account-name>;AccountKey=<account-key>;EndpointSuffix=core.windows.net";
string containerName = "***";
string blobName = "***.txt";
BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(containerName);
BlobClient blobClient = containerClient.GetBlobClient(blobName);
BlockList blockList = await blobClient.GetBlockListAsync(BlockListType.All)
string firstBlockId = blockList.Value.First().Name;
Console.WriteLine("First block ID: " + firstBlockId);
}
}
After getting the Block Id now you can get the certain block using the code below:
code:
using System;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;
class Program
{
static async Task Main(string[] args)
{
string connectionString = "DefaultEndpointsProtocol=https;AccountName=<account-name>;AccountKey=<account-key>;EndpointSuffix=core.windows.net";
string containerName = "mycontainer---";
string blobName = "myblob**";
string blockId = "1234567**";
BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(containerName);
BlobClient blobClient = containerClient.GetBlobClient(blobName);
MemoryStream blockData = new MemoryStream();
await blobClient.DownloadStreamingAsync(blockId, blockData, cancellationToken: CancellationToken.None);
Console.WriteLine("Block data:");
Console.WriteLine(Convert.ToBase64String(blockData.ToArray()));
}
}
In the above code you need to replace connection string, container name, block Id, blob name as below:
By following the above procedure, I got successfully.
Related
I am in the process of moving applications to Azure using Azure Function Apps and Blob Storge. I’m trying to figure out a way to do Encrypting/Decrypting in Azure. When doing the same in our on-premise environment I used PGPCore package but that is when you can point to a local file. I’m struggling to figure out a way to do that in Azure Blob Storage. All examples seem to use local files.
Is see paid for apps like DidiSoft but I’m trying to stick with a free version if possible.
Currently I’m working on the decrypt side of things and the concept is that I will have a PGP file delivered to my Blob Storage. I have a Azure Function with a blob trigger looking for *.pgp files and that will call another Function App specifically for just PGP work. The idea is that the PGP Function App will, in this case, decrypt the file right back into the same blob storage but this time as a .txt file. I then have a blob trigger looking for .txt files and it will run code it needs to process that file. The .txt trigger is already done and working. I already have my PGP function app calling Key Vault to get any Public/Private keys needed but what I’m not finding is a good way to do the actual Encrypt/Decrypt in Azure.
Anyone have any suggestions or examples to go look at?
We follow a similar process: download a PGP file (SFTP in our case) and load it to Blob Storage. Then we call an Azure Function that uses PgpCore to Decrypt the PGP files in Blob Storage.
Here is the Decrypt code in the Azure Function:
using System;
using System.IO;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using PgpCore;
using Azure.Storage.Blobs;
namespace AF_Services
{
public static class PGPDecryptBlob
{
[FunctionName("PGPDecryptBlob")]
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequest req,
ILogger log)
{
log.LogInformation($"C# HTTP trigger function {nameof(PGPDecryptBlob)} processed a request.");
string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
dynamic data = JsonConvert.DeserializeObject(requestBody);
try
{
// Create TMP directory
Directory.CreateDirectory(IOHelper.TempAppDirectoryName);
string sourceContainerName = data?.sourceContainerName;
log.LogInformation($"Source container name: {sourceContainerName}");
string sourceBlobName = data?.sourceBlobName;
log.LogInformation($"Source blob name: {sourceBlobName}");
string targetContainerName = data?.targetContainerName;
log.LogInformation($"Target container name: {targetContainerName}");
string targetBlobName = data?.targetBlobName;
log.LogInformation($"Target blob name: {targetBlobName}");
if (sourceContainerName == null)
throw new ArgumentNullException("Required parameter sourceContainerName not provided");
if (sourceBlobName == null || targetContainerName == null || targetBlobName == null)
throw new ArgumentNullException("Required parameter sourceBlobName not provided");
if (targetContainerName == null)
throw new ArgumentNullException("Required parameter targetContainerName not provided");
if (targetBlobName == null)
throw new ArgumentNullException("Required parameter targetBlobName not provided");
string keyBlobName = SettingsHelper.Get(Settings.PrivateKeyBlobName);
string passPhrase = SettingsHelper.Get(Settings.PassPhrase);
string blobAccountConnStr = SettingsHelper.Get(Settings.BlobConnectionString);
// TMP file names
var temp_sourceFileName = IOHelper.BuildTempFileName(BlobHelper.StripPath(sourceBlobName));
var temp_targetFileName = IOHelper.BuildTempFileName(BlobHelper.StripPath(targetBlobName));
var temp_keyFileName = IOHelper.BuildTempFileName(BlobHelper.StripPath(keyBlobName));
// download Blob to TMP
using (var sourceStream = new FileStream(temp_sourceFileName, FileMode.Create))
{
var sourceBlobClient = new BlobClient(blobAccountConnStr, sourceContainerName, sourceBlobName);
await sourceBlobClient.DownloadToAsync(sourceStream);
}
// download key to TMP
using (var keyStream = new FileStream(temp_keyFileName, FileMode.Create))
{
var keyBlobClient = new BlobClient(blobAccountConnStr, sourceContainerName, keyBlobName);
await keyBlobClient.DownloadToAsync(keyStream);
}
// Decrypt
using (var pgp = new PGP())
{
using (FileStream inputFileStream = new FileStream(temp_sourceFileName, FileMode.Open))
{
using (Stream outputFileStream = File.Create(temp_targetFileName))
{
using (Stream privateKeyStream = new FileStream(temp_keyFileName, FileMode.Open))
{
await pgp.DecryptStreamAsync(inputFileStream, outputFileStream, privateKeyStream, passPhrase);
}
}
}
}
// write to target blob
using (var decryptStream = new FileStream(temp_targetFileName, FileMode.Open))
{
var targetBlobClient = new BlobClient(blobAccountConnStr, targetContainerName, targetBlobName);
await targetBlobClient.UploadAsync(decryptStream, true);
return new OkObjectResult(new { Status = "Decrypted", BlobInfo = targetBlobClient });
}
}
catch (Exception ex)
{
return new BadRequestObjectResult(new { RequestBody = data, Exception = ex });
}
finally
{
Directory.Delete(IOHelper.TempAppDirectoryName, true);
}
}
}
}
The Encrypt code is similar, just in the opposite direction.
UPDATE:
The sourceContainer, sourceBlobName, targetContainer, and targetBlobName are passed in the JSON body of the Azure Function call.
The PrivateKeyBlobName, PassPhrase, and BlobConnectionString are stored in the Azure Function App's Settings.
The PGP work has to happen locally, so the IOHelper class (code below) is for working with the Azure Function's local file system.
public static class IOHelper
{
private static string _TempAppDirectoryName;
public static string TempAppDirectoryName
{
get
{
if (String.IsNullOrWhiteSpace(_TempAppDirectoryName))
{
var ts = DateTime.UtcNow.Ticks.ToString();
_TempAppDirectoryName = Path.Combine(Path.GetTempPath(), "{YourLocalDirectoryName}", ts);
}
return _TempAppDirectoryName;
}
}
public static string BuildTempFileName(string fileName)
{
return Path.Combine(TempAppDirectoryName, fileName);
}
}
I have next test setup:
One test Azure blob storage account
Local folder with ~3000 small files (200 bytes each)
When I execute azcopy command:
azcopy copy --recursive "c:\localDir\*" "https://BLOBConnectionString"
it takes ~2 seconds to copy data.
When I do next c# code:
ServicePointManager.Expect100Continue = false;
ServicePointManager.DefaultConnectionLimit = 32;
TransferManager.Configurations.ParallelOperations = 32;
var account = CloudStorageAccount.Parse("https://BLOBConnectionString");
CloudBlobClient client = account.CreateCloudBlobClient();
CloudBlobContainer container = client.GetContainerReference("container");
await container.CreateIfNotExistsAsync();
CloudBlobDirectory destinationBlob = container.GetDirectoryReference("data");
await TransferManager.UploadDirectoryAsync(#"c:\localDir\", destinationBlob);
It takes ~1 minute to copy same amount of data.
I expect to have approximately same latency results for c# code base.
I tried in my environment and got below results:
Code:
using Microsoft.Azure.Storage;
using Microsoft.Azure.Storage.Blob;
using Microsoft.Azure.Storage.DataMovement;
using System.ComponentModel;
namespace fastercpy
{
class program
{
public static void Main()
{
string storageConnectionString = "< Connection string >";
CloudStorageAccount account = CloudStorageAccount.Parse(storageConnectionString);
CloudBlobClient blobClient = account.CreateCloudBlobClient();
CloudBlobContainer blobContainer = blobClient.GetContainerReference("test");
blobContainer.CreateIfNotExists();
string sourcePath = "C:\\Users\\v-vsettu\\Documents\\Venkat";
CloudBlobDirectory destinationBlob = blobContainer.GetDirectoryReference("data");
TransferManager.Configurations.ParallelOperations = 64;
// Setup the transfer context and track the upoload progress
SingleTransferContext context = new SingleTransferContext();
context.ProgressHandler = new Progress<TransferStatus>((progress) =>
{
Console.WriteLine("Bytes uploaded: {0}", progress.BytesTransferred);
});
var task=TransferManager.UploadDirectoryAsync(sourcePath, destinationBlob);
task.Wait();
}
}
}
you have been used
TransferManager.Configurations.ParallelOperations = 32 ,So try to use TransferManager.Configurations.ParallelOperations = 64; in your code it will speed up process.
The Microsoft Azure Storage Data Movement Library was created for fast uploading, downloading, and copying of Azure Storage Blob and File.
Console:
Portal:
Reference:
Transfer data with the Data Movement library for .NET - Azure Storage | Microsoft Learn
The app is set up on multiple on-premise services and uploads regularly some files to Azure Blob Storage placed in East US. But now it's necessary to place an instance of the app in the Australian region. As a result, upload time to the cloud increased drastically.
I have tested if Azure Front Door can help to improve it and I found that download from blob storage works 5 times faster if I use the Azure Front Door link. Now I struggle to change C# code to upload files via Azure Front Door. I have tried to use the suffix "azurefd.net" instead of "core.windows.net" in the connection string but it does not help. Could somebody give me a hint on how to upload files to Azure blob storage via Azure Front Door in C#?
As the Storage connection string uses only storage endpoint (core.windows.net), we cannot use front door endpoint (azurefd.net) in the connection string.
I integrated Azure Storage Account with Front Door. I am able to access the Blob Files in the Azure Storage Account with Front Door URL.
We cannot upload files to Blob Storage via Azure Front Door using C#
This is because Azure Storage in C# accepts connection string only from storage endpoint
Unfortunately, for upload, Azure Front Door does not provide any benefit. I used
PUT requests for the test described here: https://learn.microsoft.com/en-us/rest/api/storageservices/put-blob
PUT https://<entityName>.azurefd.net/<containerName>/<blobName>?<sharedAccessSignature>
x-ms-version: 2020-10-02
x-ms-blob-type: BlockBlob
< C:\Downloads\t1.txt
and compared times for storage account and Azure Front account. There is no difference in speed for upload.
Code that I used for test:
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
namespace SandboxV2
{
class Program
{
static async Task Main()
{
string frontDoorUrl = "https://<FRONT-DOOR>.azurefd.net";
string storageUrl = "https://{STORAGE-ACCOUNT}.blob.core.windows.net";
string sasString = "...";
Console.Write("File Path: ");
string filePath = Console.ReadLine();
await RunUploadTestAsync(filePath, frontDoorUrl, sasString, "-fd");
await RunUploadTestAsync(filePath, storageUrl, sasString, "-storage");
}
private static async Task RunUploadTestAsync(string filePath, string rootUrl, string sasString, string suffix)
{
string blobName = Path.GetFileNameWithoutExtension(filePath) + suffix + Path.GetExtension(filePath);
Console.WriteLine(rootUrl);
string containerName = "testaccess";
var speeds = new List<double>();
var times = new List<TimeSpan>();
for (int i = 0; i < 5; i++)
{
var t1 = DateTime.UtcNow;
var statusCode = await UploadAsSingleBlock(filePath, rootUrl, blobName, containerName, sasString);
var t2 = DateTime.UtcNow;
var time = t2 - t1;
var speed = new FileInfo(filePath).Length / time.TotalSeconds / 1024 / 1024 * 8;
speeds.Add(speed);
times.Add(time);
Console.WriteLine($"Code: {statusCode}. Time: {time}. Speed: {speed}");
}
Console.WriteLine($"Average time: {TimeSpan.FromTicks((long)times.Select(t => t.Ticks).Average())}. Average speed: {speeds.Average()}.");
}
private static async Task<HttpStatusCode> UploadAsSingleBlock(string filePath, string rootUrl, string blobName, string containerName, string sasString)
{
var request = new HttpRequestMessage(HttpMethod.Put, $"{rootUrl}/{containerName}/{blobName}?{sasString}");
request.Headers.Add("x-ms-version", "2020-10-02");
request.Headers.Add("x-ms-blob-type", "BlockBlob");
HttpResponseMessage response;
using (var fileStream = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.Read))
{
request.Content = new StreamContent(fileStream);
using (var client = new HttpClient())
response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
}
return response.StatusCode;
}
}
}
I am trying to read a blob from Azure using C# Console application. I can download the blob file to my local machine and read it but when I am trying to download it to memory stream it is throwing an error as attached.
I want to directly read my blob file instead of downloading it to local machine as I have to deploy it in Azure. Any suggestions on it?
var storageaccount = new CloudStorageAccount(new Microsoft.WindowsAzure.Storage.Auth.StorageCredentials(StorageAccountName, StorageAccountKey), true);
var blobclient = storageaccount.CreateCloudBlobClient();
var container = blobclient.GetContainerReference("");
// var blobs = container.ListBlobs();
var blockBlobReference = container.GetBlockBlobReference(sourceBlobFileName);
using (var memorystream = new MemoryStream())
{
blockBlobReference.DownloadToStream(memorystream);
}
I think the problem is because you're wrapping it in a using statement. Try the following:
var storageaccount = new CloudStorageAccount(new Microsoft.WindowsAzure.Storage.Auth.StorageCredentials(StorageAccountName, StorageAccountKey), true);
var blobclient = storageaccount.CreateCloudBlobClient();
var container = blobclient.GetContainerReference("");
// var blobs = container.ListBlobs();
var blockBlobReference = container.GetBlockBlobReference(sourceBlobFileName);
var memorystream = new MemoryStream();
blockBlobReference.DownloadToStream(memorystream);
byte[] content = memoryStream.ToArray();
I tried your code, but Blob downloaded successfully.
your screenshot just shows the variable in Visual Studio debugger, and it looks like the error doesn't actually occurred.
In screenshot, MemoryStream, CanTimeout property set to false.
So, it seems that ReadTimeout and WriteTimeout throw exceptions only because MemoryStream does not support them.
Reference:
Stream.ReadTimeout Property
Please note that the exception won't occurred until you use ReadTimeout and WriteTimeout properties.
As per the title, I have a program whereby I'm trying to add to an existing BlobkBlob using the PutBlock method:
private static void UploadNewText(string text)
{
string fileName = "test4.txt";
string containerString = "mycontainer";
CloudStorageAccount storage = CloudStorageAccount.Parse(connection);
CloudBlobClient client = storage.CreateCloudBlobClient();
CloudBlobContainer container = client.GetContainerReference(containerString);
CloudBlockBlob blob = container.GetBlockBlobReference(fileName);
using (MemoryStream stream = new MemoryStream())
using (StreamWriter sw = new StreamWriter(stream))
{
sw.Write(text);
sw.Flush();
stream.Position = 0;
string blockId = Convert.ToBase64String(
ASCIIEncoding.ASCII.GetBytes("0000005"));
Console.WriteLine(blockId);
blob.PutBlock(blockId, stream, null);
blob.PutBlockList(new string[] { blockId });
}
}
As I understand it, providing the BlockId increases (or at least differs), and is a consistent size, this should work; however, when I run it a second time for the same file (regardless of whether or not I increase the Block ID) it just overwrites the existing file with the new text.
I realise there are other options for appending to a blob (such as AppendBlob), but I'm curious if PutBlock, specifically can do this. Is what I am trying to do possible and, if so, what am I doing wrong?
Can PutBlock be used to append to an existing BlockBlob in Azure
Yes, it can. However in order to do that, you will need to work that in a little bit different way.
What you will need to do is:
First get the previously committed block list. The method you want to use is DownloadBlockList.
Upload new block. Note down its block id.
Append this block id to the list of block ids you downloaded in step #1.
Call put block list with this new list.