I am trying to read a blob from Azure using C# Console application. I can download the blob file to my local machine and read it but when I am trying to download it to memory stream it is throwing an error as attached.
I want to directly read my blob file instead of downloading it to local machine as I have to deploy it in Azure. Any suggestions on it?
var storageaccount = new CloudStorageAccount(new Microsoft.WindowsAzure.Storage.Auth.StorageCredentials(StorageAccountName, StorageAccountKey), true);
var blobclient = storageaccount.CreateCloudBlobClient();
var container = blobclient.GetContainerReference("");
// var blobs = container.ListBlobs();
var blockBlobReference = container.GetBlockBlobReference(sourceBlobFileName);
using (var memorystream = new MemoryStream())
{
blockBlobReference.DownloadToStream(memorystream);
}
I think the problem is because you're wrapping it in a using statement. Try the following:
var storageaccount = new CloudStorageAccount(new Microsoft.WindowsAzure.Storage.Auth.StorageCredentials(StorageAccountName, StorageAccountKey), true);
var blobclient = storageaccount.CreateCloudBlobClient();
var container = blobclient.GetContainerReference("");
// var blobs = container.ListBlobs();
var blockBlobReference = container.GetBlockBlobReference(sourceBlobFileName);
var memorystream = new MemoryStream();
blockBlobReference.DownloadToStream(memorystream);
byte[] content = memoryStream.ToArray();
I tried your code, but Blob downloaded successfully.
your screenshot just shows the variable in Visual Studio debugger, and it looks like the error doesn't actually occurred.
In screenshot, MemoryStream, CanTimeout property set to false.
So, it seems that ReadTimeout and WriteTimeout throw exceptions only because MemoryStream does not support them.
Reference:
Stream.ReadTimeout Property
Please note that the exception won't occurred until you use ReadTimeout and WriteTimeout properties.
Related
I am trying to download an image (.jpg) from an Azure storage blob using:
public static async Task DownloadToTemp(string path, string fileName)
{
string storageAccount_connectionString = "CONNECTION STRING";
CloudStorageAccount mycloudStorageAccount = CloudStorageAccount.Parse(storageAccount_connectionString);
CloudBlobClient blobClient = mycloudStorageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference(CONTAINER);
CloudBlockBlob cloudBlockBlob = container.GetBlockBlobReference(fileName);
// provide the file download location below
Stream file = File.OpenWrite(path);
await cloudBlockBlob.DownloadToStreamAsync(file);
file.Close();
return;
}
But when I try to open the image as a bit map Bitmap image = new Bitmap(path) I am getting the error System.ArgumentException: 'Parameter is not valid.'. I am using the call BlobHandler.DownloadToTemp(path, file).GetAwaiter() to ensure the file has been downloaded.
So the problem lies with not waiting fully for the image to be stored:
adding:
await BlobHandler.DownloadToTemp(path, file);
Ensures that the file is fully downloaded (This meant that the callerfunction had to be async).
https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-dotnet#list-directory-contents
I am trying to read Parquet file which is located in Azure DataLake Gen2 Container for automation framework. Able to connect using StorageAccount & AccessKey with information provided above URL. However I want to temporary read it to DataTable / SQL View / any other option which give me option to query and Assert records while the system is in test.
Getting System.IO. cannot convert to azure.storage.files.modesl error exception. Getting 200 response.
Unable to convert Parquet data from filestream.
Using NUnit 2019 Community Edition (c#).
DataLakeFileClient fileClient = fileSystemClient.GetFileClient(fileName);
Response<FileDownloadInfo> downloadResponse = fileClient.Read();
var reader = new StreamReader(downloadResponse.Value.Content);
Stream fileStream = GetSteamfromString(lines);
public static Stream GetSteamfromString(string lines)
{
var stream = new MemoryStream();
var writer = new StreamWriter(stream);
writer.Write(lines);
writer.Flush();
stream.Position = 0;
return stream;
}
Note - Test is to compare Raw Data (Blob Storage) to Cleaned data (DataLakeGen2). Storing data from blob into DataTable to assert the conditions/tests.
Thanks,
As per the title, I have a program whereby I'm trying to add to an existing BlobkBlob using the PutBlock method:
private static void UploadNewText(string text)
{
string fileName = "test4.txt";
string containerString = "mycontainer";
CloudStorageAccount storage = CloudStorageAccount.Parse(connection);
CloudBlobClient client = storage.CreateCloudBlobClient();
CloudBlobContainer container = client.GetContainerReference(containerString);
CloudBlockBlob blob = container.GetBlockBlobReference(fileName);
using (MemoryStream stream = new MemoryStream())
using (StreamWriter sw = new StreamWriter(stream))
{
sw.Write(text);
sw.Flush();
stream.Position = 0;
string blockId = Convert.ToBase64String(
ASCIIEncoding.ASCII.GetBytes("0000005"));
Console.WriteLine(blockId);
blob.PutBlock(blockId, stream, null);
blob.PutBlockList(new string[] { blockId });
}
}
As I understand it, providing the BlockId increases (or at least differs), and is a consistent size, this should work; however, when I run it a second time for the same file (regardless of whether or not I increase the Block ID) it just overwrites the existing file with the new text.
I realise there are other options for appending to a blob (such as AppendBlob), but I'm curious if PutBlock, specifically can do this. Is what I am trying to do possible and, if so, what am I doing wrong?
Can PutBlock be used to append to an existing BlockBlob in Azure
Yes, it can. However in order to do that, you will need to work that in a little bit different way.
What you will need to do is:
First get the previously committed block list. The method you want to use is DownloadBlockList.
Upload new block. Note down its block id.
Append this block id to the list of block ids you downloaded in step #1.
Call put block list with this new list.
I'm downloading file from blob container & saving in to a stream & trying to read pdf.
//creating a Cloud Storage instance
CloudStorageAccount StorageAccount = CloudStorageAccount.Parse(connectionstring);
//Creating a Client to operate on blob
CloudBlobClient blobClient = StorageAccount.CreateCloudBlobClient();
// fetching the container based on name
CloudBlobContainer container = blobClient.GetContainerReference(containerName);
//Get a reference to a blob within the container.
CloudBlockBlob blob = container.GetBlockBlobReference(fileName);
var memStream = new MemoryStream();
blob.DownloadToStream(memStream);
try
{
PdfReader reader = new PdfReader(memStream);
}
catch(Exception ex)
{
}
Exception : PDF header signature not found.
The reason, as was evident after troubleshooting through the comments was that this line:
blob.DownloadToStream(memStream);
positioned the stream right after the downloaded content.
Then, when constructing the pdf reader object it expected to find the Pdf file form the current position.
This is a common problem when dealing with streams that one first write something to and then afterwards tries to read that something, one must remember to reposition the stream if necessary.
In this case, assuming there is only the pdf in the stream, the solution is to reposition the stream back to the start before attempting to read the pdf file:
Add this line:
memStream.Position = 0;
after the download but before the reader is constructed to reposition.
Here's what the code can look like in this region:
blob.DownloadToStream(memStream);
memStream.Position = 0; // <----------------------------------- add this line
try
{
PdfReader reader = new PdfReader(memStream);
I have a method that returns an msi package using a filestream.
public FileStream DownloadMsiFileStream()
{
FileStream fs = new FileStream(#"C:\temp\test.msi", FileMode.Create, System.IO.FileAccess.ReadWrite);
// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference("deviceupdate");
// Retrieve reference to a blob named "KC.AttendanceManager.PrintServiceInstaller.msi".
CloudBlockBlob blockBlob = container.GetBlockBlobReference("test.msi");
//Retrive the memorystream
blockBlob.DownloadToStream(fs);
return fs;
}
This works just perfectly, I can download the filestream using a web api method, write the stream to a file and end up with a working msi package.
But now i want to avoid writing the file to disk on serverside as it results in concurrency issues. Instead i have tried changing the filestream to a Memorystream like this:
public MemoryStream DownloadMsi()
{
// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference("deviceupdate");
// Retrieve reference to a blob named "KC.AttendanceManager.PrintServiceInstaller.msi".
CloudBlockBlob blockBlob = container.GetBlockBlobReference("test.msi");
MemoryStream ms = new MemoryStream();
//Retrive the memorystream
blockBlob.DownloadToStream(ms);
return ms;
}
But when I am trying to write the stream to a file later on (just server side to get it working) like this:
MemoryStream ms = DeviceUpdateManager.GetClientUpdateMsi();
FileStream file = new FileStream(#"C:\temp\test2.msi", FileMode.OpenOrCreate);
byte[] bytes = new byte[file.Length];
file.Read(bytes, 0, (int)file.Length);
ms.Write(bytes, 0, (int)file.Length);
file.Close();
ms.Close();
The result is an invalid (empty) msi file. The memorystream is not empty as System.Text.Encoding.UTF8.GetString(ms.ToArray()) returns a bunch. How do I end up with a working msi? Any help appriciated.
As often the solution is all too simple. This was all I needed to retrieve my working msi package:
File.WriteAllBytes(#"C:\temp\test2.msi", ms.ToArray());