I have a controller in my dotnet core web application to fetch a resource from a Azure storage account in Azure and offer it to the user for download. The user can't directly access the Azure storage account, so my webapp works as a proxy and authenticates the user before service the file.
My doubt is if my implementation if efficient with large files? My concern is that DownloadToStreamAsync() actually fetches the entire file in the memory of the webapp before serving it.
public async Task<IActionResult> Serve(string path)
{
MemoryStream streamIn = null;
CloudFile file = null;
Stream fileStream = null;
var filename = Path.GetFileName(path);
// application-level permission checks checks
// fetching file from Azure Storage
try {
var storageConnectionString = _azureOptions.AzureStorageAccountConnectionString;
var storageAccount = CloudStorageAccount.Parse(storageConnectionString);
var fileClient = storageAccount.CreateCloudFileClient();
var share = fileClient.GetShareReference(_azureOptions.AzureStorageAccountContentShareName);
var root = share.GetRootDirectoryReference();
file = root.GetFileReference(path);
if (!await file.ExistsAsync())
{
return NotFound();
}
streamIn = new MemoryStream();
await file.DownloadToStreamAsync(streamIn);
fileStream = await file.OpenReadAsync();
} catch (StorageException e) {
_logger.LogError($"Error while retrieving content resource: {path}", e);
return NotFound();
}
return File(fileStream, _getContentType(filename));
}
You are right to have this concern because your code is downloading the entire blob to memory before uploading it to the client. This is very inefficient.
Furthermore, you are not even using the MemoryStream where you download the file. Just delete this code:
streamIn = new MemoryStream();
await file.DownloadToStreamAsync(streamIn);
The rest of the code should stream the file from Azure and stream it to the client in parallel.
Related
I'm a complete noob at c# and know very little about azure apis and a current cs student doing a project for work. I built some middleware with youtube tutorials that authenticates a with a storage account using a string connection and it enumerates, uploads, downloads, and deletes blobs within a container. The issue i'm having lies with ONLY the downloading functionality and ONLY when the storage account access is set to private. This function works fine with anon access. I suspect the issue is with appending the url, and I'm not sure how to fix it. The blobs are mainly csv data if that matters. Any help or direction to resources would be greatly appreciated 🙂 here is the relevant code:
url function
public async Task<string> GetBlob(string name, string containerName)
{
var containerClient = _blobClient.GetBlobContainerClient(containerName);
var blobClient = containerClient.GetBlobClient(name);
return blobClient.Uri.AbsoluteUri;
}
The config file
"AllowedHosts": "*",
"BlobConnection" : "<mystringconnection>******==;EndpointSuffix=core.windows.net"
action request
[HttpGet]
public async Task<IActionResult> ViewFile(string name)
{
var res = await _blobService.GetBlob(name, "<mystorageacc>");
return Redirect(res);
}
The reason you are not able to download the blobs from a private container is because you are simply returning the blob's URL from your method without any authorization information. Request to access blobs in a private container must be authorized.
What you would need to do is create a Shared Access Signature (SAS) with at least Read permission and then return that SAS URL. The method you would want to use is GenerateSasUri. Your code would be something like:
public async Task<string> GetBlob(string name, string containerName)
{
var containerClient = _blobClient.GetBlobContainerClient(containerName);
var blobClient = containerClient.GetBlobClient(name);
return blobClient.GenerateSasUri(BlobSasPermissions.Read, DateTime.UtcNow.AddMinutes(5)).Uri.AbsoluteUri;
}
This will give you a link which is valid for 5 minutes from the time of creation and has the permission to read (download) the blob.
If you want to download from the blob service;
public async Task<byte[]> ReadFileAsync(string path)
{
using var ms = new MemoryStream();
var blob = _client.GetBlobClient(path);
await blob.DownloadToAsync(ms);
return ms.ToArray();
}
If you want to download the file byte array from controllers, you can check this;
https://stackoverflow.com/a/3605510/3024129
If you want to set a blob file public access level;
https://learn.microsoft.com/en-us/azure/storage/blobs/anonymous-read-access-configure.
Pay attention to the images please;
Or you can connect with Azure Storage Explorer and choose the easy way.
I found the images on the Google, there may be differences. :)
This worked for me by returning a byte array:
byte[] base64ImageRepresentation = new byte[] { };
BlobClient blobClient = new BlobClient(blobConnectionString,
blobContainerUserDocs,+ "/" + fileName);
if (await blobClient.ExistsAsync())
{
using var ms = new MemoryStream();
await blobClient.DownloadToAsync(ms);
return ms.ToArray();
}
We have a parquet formatfile (500 mb) which is located in Azure blob.How to read the file directly from blob and save in memory of c# ,say eg:Datatable.
I am able to read parquet file which is physically located in folder using the below code.
public void ReadParqueFile()
{
using (Stream fileStream = System.IO.File.OpenRead("D:/../userdata1.parquet"))
{
using (var parquetReader = new ParquetReader(fileStream))
{
DataField[] dataFields = parquetReader.Schema.GetDataFields();
for (int i = 0; i < parquetReader.RowGroupCount; i++)
{
using (ParquetRowGroupReader groupReader = parquetReader.OpenRowGroupReader(i))
{
DataColumn[] columns = dataFields.Select(groupReader.ReadColumn).ToArray();
DataColumn firstColumn = columns[0];
Array data = firstColumn.Data;
//int[] ids = (int[])data;
}
}
}
}
}
}
(I am able to read csv file directly from blob using sourcestream).Please kindly suggest a fastest method to read the parquet file directly from blob
Per my experience, the solution to directly read the parquet file from blob is first to generate the blob url with sas token and then to get the stream of HttpClient from the url with sas and finally to read the http response stream via ParquetReader.
First, please refer to the sample code below of the section Create a service SAS for a blob of the offical document Create a service SAS for a container or blob with .NET using Azure Blob Storage SDK for .NET Core.
private static string GetBlobSasUri(CloudBlobContainer container, string blobName, string policyName = null)
{
string sasBlobToken;
// Get a reference to a blob within the container.
// Note that the blob may not exist yet, but a SAS can still be created for it.
CloudBlockBlob blob = container.GetBlockBlobReference(blobName);
if (policyName == null)
{
// Create a new access policy and define its constraints.
// Note that the SharedAccessBlobPolicy class is used both to define the parameters of an ad hoc SAS, and
// to construct a shared access policy that is saved to the container's shared access policies.
SharedAccessBlobPolicy adHocSAS = new SharedAccessBlobPolicy()
{
// When the start time for the SAS is omitted, the start time is assumed to be the time when the storage service receives the request.
// Omitting the start time for a SAS that is effective immediately helps to avoid clock skew.
SharedAccessExpiryTime = DateTime.UtcNow.AddHours(24),
Permissions = SharedAccessBlobPermissions.Read | SharedAccessBlobPermissions.Write | SharedAccessBlobPermissions.Create
};
// Generate the shared access signature on the blob, setting the constraints directly on the signature.
sasBlobToken = blob.GetSharedAccessSignature(adHocSAS);
Console.WriteLine("SAS for blob (ad hoc): {0}", sasBlobToken);
Console.WriteLine();
}
else
{
// Generate the shared access signature on the blob. In this case, all of the constraints for the
// shared access signature are specified on the container's stored access policy.
sasBlobToken = blob.GetSharedAccessSignature(null, policyName);
Console.WriteLine("SAS for blob (stored access policy): {0}", sasBlobToken);
Console.WriteLine();
}
// Return the URI string for the container, including the SAS token.
return blob.Uri + sasBlobToken;
}
Then to get the http response stream of HttpClient from the url with sas token .
var blobUrlWithSAS = GetBlobSasUri(container, blobName);
var client = new HttpClient();
var stream = await client.GetStreamAsync(blobUrlWithSAS);
Finally to read it via ParquetReader, the code comes from Reading Data of GitHub repo aloneguid/parquet-dotnet.
var options = new ParquetOptions { TreatByteArrayAsString = true };
var reader = new ParquetReader(stream, options);
I am trying to download files from azure to computer via an web app. It works when I run locally the project, but when uploaded to ftp server it does not download.
I have tried Environment.SpecialFolder.Peronal, Desktop, etc.
public async Task<bool> DownloadBlobAsync(string file, string fileExtension, string directory)
{
string downlaodPath = Environment.GetFolderPath(Environment.SpecialFolder.Personal);
_container = _client.GetContainerReference(containerName);
_directoy = _container.GetDirectoryReference(directory);
CloudBlockBlob blockBlob = _directoy.GetBlockBlobReference(file + "." + fileExtension);
using (var fileStream = File.OpenWrite(downlaodPath + "/"+ file + "." + fileExtension))
{
await blockBlob.DownloadToStreamAsync(fileStream);
return true;
}
}
The expected output should be on the documents or desktop.
The issue that you are seeing is due to the fact that your code is executing on the webserver, not on the clients (users) machine.
In other words, when you try to save to Environment.SpecialFolder.Personal, you're trying to save it to that folder on the web server, not the users desktop computer.
What you need to do is return the content of the blob in the request, and let the browser save the file - the user is likely to be prompted (depending on their browser settings) where exactly to save it. You should not be specifying this.
Here is an example of how to do this:
public async Task<HttpResponseMessage> DownloadBlobAsync(string file, string fileExtension, string directory)
{
_container = _client.GetContainerReference(containerName);
_directoy = _container.GetDirectoryReference(directory);
CloudBlockBlob blockBlob = _directoy.GetBlockBlobReference(file + "." + fileExtension);
using (var ms = new MemoryStream())
{
await blockBlob.DownloadToStreamAsync(ms);
var result = new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new ByteArrayContent(ms.ToArray())
};
result.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment")
{
FileName = "somefilename.ext"
};
result.Content.Headers.ContentType = new MediaTypeHeaderValue(blockBlob.Properties.ContentType);
return result;
}
}
Note that this is inefficient, as it will download the blob first to the webserver, and then return that to the client. It should be enough to get started.
When this endpoint is hit by the browser, the user will be prompted to save the file somewhere on their PC.
When I tried to download the image file from the online azure blob, it throws exception as "The given path's format is not supported". My code block is below:
StorageCredentials creds = new StorageCredentials(accountName, accountKey);
CloudStorageAccount account = new CloudStorageAccount(creds, useHttps: true);
CloudBlobClient client = account.CreateCloudBlobClient();
container = client.GetContainerReference(blobName);
CloudBlockBlob blockBlob = container.GetBlockBlobReference(MyPath);
await blockBlob.DownloadToFileAsync(Path, FileMode.OpenOrCreate);
using (var fileStream = System.IO.File.Create(Path))
{
await blockBlob.DownloadToStreamAsync(fileStream);
}
If I provide the local path as "c:\users\Joy\Downloads" in path,as like below:
var localPath = #"C:\Users\Joy \Downloads\user.jpg" ;
await blockBlob.DownloadToFileAsync(localPath, FileMode.OpenOrCreate);
using (var fileStream = System.IO.File.Create(localPath))
{
await blockBlob.DownloadToStreamAsync(fileStream);
}
It can be copied into corresponding location. But I couldn't download the file in my custom location.
According to your description, I enabled public read access to my blobs to check this issue. I created a console application and you could refer to the following code snippet for downloading the file and maintain the virtual directory in your local file system as follows:
CloudBlockBlob blockBlob = new CloudBlockBlob(new Uri("https://brucchstorage.blob.core.windows.net/images/2017/11/28/lake.jpeg"));
var localPath = Path.Combine(/*your custom root folder for storing file(s)*/AppDomain.CurrentDomain.BaseDirectory,$"downloads\\{blockBlob.Name}"); //blockBlob.Name =2017/11/28/lake.jpeg
var rootDir = new FileInfo(localPath).Directory;
if (!rootDir.Exists) //make sure the parent directory exists
rootDir.Create();
await blockBlob.DownloadToFileAsync(localPath,FileMode.Create);
//OR
using (var fs = new FileStream(localPath, FileMode.Create))
{
await blockBlob.DownloadToStreamAsync(fs);
}
Result:
Moreover, you could construct the CloudBlockBlob instance with the Uri contains the SAS token as follows:
https://brucchstorage.blob.core.windows.net/images/2017/11/28/lake.jpeg?st=2017-11-28T06%3A28%3A00Z&se=2017-11-29T06%3A28%3A00Z&sp=r&sv=2015-12-11&sr=b&sig=15NAaRB43C%2BniIZZe8gAvFl7LY%2BS6K7DNyjLflpvgBg%3D
More details, you could follow here.
I want to upload a file to Azure blob storage asynchronously. I have tried the way suggested in the official sdk:
This is how I get the container:
public static class BlobHelper
{
public static CloudBlobContainer GetBlobContainer()
{
// Pull these from config
var blobStorageConnectionString = ConfigurationManager.AppSettings["BlobStorageConnectionString"];
var blobStorageContainerName = ConfigurationManager.AppSettings["BlobStorageContainerName"];
// Create blob client and return reference to the container
var blobStorageAccount = CloudStorageAccount.Parse(blobStorageConnectionString);
var blobClient = blobStorageAccount.CreateCloudBlobClient();
var container = blobClient.GetContainerReference(blobStorageContainerName);
container.CreateIfNotExists();
container.SetPermissions(new BlobContainerPermissions { PublicAccess = BlobContainerPublicAccessType.Blob });
return container;
}
}
And this is how i try to upload the file:
var documentName = Guid.NewGuid().ToString();
CloudBlobContainer container = BlobHelper.GetBlobContainer();
CloudBlockBlob blockBlob = container.GetBlockBlobReference(documentName);
public class FilesService
{
public async Task<string> UploadFiles(HttpContent httpContent)
{
var documentName = Guid.NewGuid().ToString();
CloudBlobContainer container = BlobHelper.GetBlobContainer();
CloudBlockBlob blockBlob = container.GetBlockBlobReference(documentName);
using (var fileStream = System.IO.File.OpenRead(#"path\myfile"))
{
await blockBlob.UploadFromStreamAsync(fileStream);
}
return blockBlob.Uri.ToString();
}
}
The problem is that I do not know how to get the path to my file (it is uploaded by the user).
When I try this:
var rootpath = HttpContext.Current.Server.MapPath("~/App_Data");
var streamProvider = new MultipartFileStreamProvider(rootpath);
await httpContent.ReadAsMultipartAsync(streamProvider);
foreach (var file in streamProvider.FileData)
{
var localName = file.LocalFileName;
using (var fileStream = System.IO.File.OpenRead(file.LocalFileName))
{
await blockBlob.UploadFromStreamAsync(fileStream);
}
}
And when I try a post request. The request just crashes and does not return anything (even an exception);
Solution:
The issue was resolved in the following way. I used a service method in order to be able to upload a collection of files.
In the BlobHelper class I save the needed information about the container and then instantiate it, it is a static class. Using a collection makes it possible to upload a multiple files as a part of the same stream.
I think you are trying to get the path to the file that is being uploaded to the Blob Storage using standard ASP.NET methods and local context. Files uploaded to the blob will not be accessible that way.
Seems like you upload your blob properly. Now, if your file uploaded successfully, your method should return blockBlob.Uri.ToString(), which is the link to your file - you may store it somewhere in the database or anywhere else.