I'm a complete noob at c# and know very little about azure apis and a current cs student doing a project for work. I built some middleware with youtube tutorials that authenticates a with a storage account using a string connection and it enumerates, uploads, downloads, and deletes blobs within a container. The issue i'm having lies with ONLY the downloading functionality and ONLY when the storage account access is set to private. This function works fine with anon access. I suspect the issue is with appending the url, and I'm not sure how to fix it. The blobs are mainly csv data if that matters. Any help or direction to resources would be greatly appreciated 🙂 here is the relevant code:
url function
public async Task<string> GetBlob(string name, string containerName)
{
var containerClient = _blobClient.GetBlobContainerClient(containerName);
var blobClient = containerClient.GetBlobClient(name);
return blobClient.Uri.AbsoluteUri;
}
The config file
"AllowedHosts": "*",
"BlobConnection" : "<mystringconnection>******==;EndpointSuffix=core.windows.net"
action request
[HttpGet]
public async Task<IActionResult> ViewFile(string name)
{
var res = await _blobService.GetBlob(name, "<mystorageacc>");
return Redirect(res);
}
The reason you are not able to download the blobs from a private container is because you are simply returning the blob's URL from your method without any authorization information. Request to access blobs in a private container must be authorized.
What you would need to do is create a Shared Access Signature (SAS) with at least Read permission and then return that SAS URL. The method you would want to use is GenerateSasUri. Your code would be something like:
public async Task<string> GetBlob(string name, string containerName)
{
var containerClient = _blobClient.GetBlobContainerClient(containerName);
var blobClient = containerClient.GetBlobClient(name);
return blobClient.GenerateSasUri(BlobSasPermissions.Read, DateTime.UtcNow.AddMinutes(5)).Uri.AbsoluteUri;
}
This will give you a link which is valid for 5 minutes from the time of creation and has the permission to read (download) the blob.
If you want to download from the blob service;
public async Task<byte[]> ReadFileAsync(string path)
{
using var ms = new MemoryStream();
var blob = _client.GetBlobClient(path);
await blob.DownloadToAsync(ms);
return ms.ToArray();
}
If you want to download the file byte array from controllers, you can check this;
https://stackoverflow.com/a/3605510/3024129
If you want to set a blob file public access level;
https://learn.microsoft.com/en-us/azure/storage/blobs/anonymous-read-access-configure.
Pay attention to the images please;
Or you can connect with Azure Storage Explorer and choose the easy way.
I found the images on the Google, there may be differences. :)
This worked for me by returning a byte array:
byte[] base64ImageRepresentation = new byte[] { };
BlobClient blobClient = new BlobClient(blobConnectionString,
blobContainerUserDocs,+ "/" + fileName);
if (await blobClient.ExistsAsync())
{
using var ms = new MemoryStream();
await blobClient.DownloadToAsync(ms);
return ms.ToArray();
}
Related
We have a parquet formatfile (500 mb) which is located in Azure blob.How to read the file directly from blob and save in memory of c# ,say eg:Datatable.
I am able to read parquet file which is physically located in folder using the below code.
public void ReadParqueFile()
{
using (Stream fileStream = System.IO.File.OpenRead("D:/../userdata1.parquet"))
{
using (var parquetReader = new ParquetReader(fileStream))
{
DataField[] dataFields = parquetReader.Schema.GetDataFields();
for (int i = 0; i < parquetReader.RowGroupCount; i++)
{
using (ParquetRowGroupReader groupReader = parquetReader.OpenRowGroupReader(i))
{
DataColumn[] columns = dataFields.Select(groupReader.ReadColumn).ToArray();
DataColumn firstColumn = columns[0];
Array data = firstColumn.Data;
//int[] ids = (int[])data;
}
}
}
}
}
}
(I am able to read csv file directly from blob using sourcestream).Please kindly suggest a fastest method to read the parquet file directly from blob
Per my experience, the solution to directly read the parquet file from blob is first to generate the blob url with sas token and then to get the stream of HttpClient from the url with sas and finally to read the http response stream via ParquetReader.
First, please refer to the sample code below of the section Create a service SAS for a blob of the offical document Create a service SAS for a container or blob with .NET using Azure Blob Storage SDK for .NET Core.
private static string GetBlobSasUri(CloudBlobContainer container, string blobName, string policyName = null)
{
string sasBlobToken;
// Get a reference to a blob within the container.
// Note that the blob may not exist yet, but a SAS can still be created for it.
CloudBlockBlob blob = container.GetBlockBlobReference(blobName);
if (policyName == null)
{
// Create a new access policy and define its constraints.
// Note that the SharedAccessBlobPolicy class is used both to define the parameters of an ad hoc SAS, and
// to construct a shared access policy that is saved to the container's shared access policies.
SharedAccessBlobPolicy adHocSAS = new SharedAccessBlobPolicy()
{
// When the start time for the SAS is omitted, the start time is assumed to be the time when the storage service receives the request.
// Omitting the start time for a SAS that is effective immediately helps to avoid clock skew.
SharedAccessExpiryTime = DateTime.UtcNow.AddHours(24),
Permissions = SharedAccessBlobPermissions.Read | SharedAccessBlobPermissions.Write | SharedAccessBlobPermissions.Create
};
// Generate the shared access signature on the blob, setting the constraints directly on the signature.
sasBlobToken = blob.GetSharedAccessSignature(adHocSAS);
Console.WriteLine("SAS for blob (ad hoc): {0}", sasBlobToken);
Console.WriteLine();
}
else
{
// Generate the shared access signature on the blob. In this case, all of the constraints for the
// shared access signature are specified on the container's stored access policy.
sasBlobToken = blob.GetSharedAccessSignature(null, policyName);
Console.WriteLine("SAS for blob (stored access policy): {0}", sasBlobToken);
Console.WriteLine();
}
// Return the URI string for the container, including the SAS token.
return blob.Uri + sasBlobToken;
}
Then to get the http response stream of HttpClient from the url with sas token .
var blobUrlWithSAS = GetBlobSasUri(container, blobName);
var client = new HttpClient();
var stream = await client.GetStreamAsync(blobUrlWithSAS);
Finally to read it via ParquetReader, the code comes from Reading Data of GitHub repo aloneguid/parquet-dotnet.
var options = new ParquetOptions { TreatByteArrayAsString = true };
var reader = new ParquetReader(stream, options);
I have an anonymous function on Azure, successfully deployed (.netstandard 2.0), using Microsoft.NET.Sdk.Functions (1.0.13), but for some reason suddenly it stopped working and when I call it the response is:
<ApiErrorModel xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.datacontract.org/2004/07/Microsoft.Azure.WebJobs.Script.WebHost.Models">
<Arguments xmlns:d2p1="http://schemas.microsoft.com/2003/10/Serialization/Arrays" i:nil="true"/>
<ErrorCode>0</ErrorCode>
<ErrorDetails i:nil="true"/>
<Id>91fab400-3447-4913-878f-715d9d4ab46b</Id>
<Message>
An error has occurred. For more information, please check the logs for error ID 91fab400-3447-4913-878f-715d9d4ab46b
</Message>
<RequestId>0fb00298-733d-4d88-9e73-c328d024e1bb</RequestId>
<StatusCode>InternalServerError</StatusCode>
</ApiErrorModel>
How to figure that out?
EDIT: When I start AF environment locally and run the function it works as expected, without any issues, although what i see in console is a message in red:
and searching for it, stumbled across this GitHub post:
https://github.com/Azure/azure-functions-host/issues/2765
where I noticed this part:
#m-demydiuk noticed that azure function works with this error in the console. So this red error doesn't break function on the local machine. But I am afraid it may cause any problems in other environments.
and it bothers me. Can it be the problem?
I use a lib with a version that do not match my target framework, but again locally works fine, and also it was working fine before on Azure
My host version is "Version=2.0.11651.0"
This is the entire function:
public static class Function1
{
[FunctionName("HTML2IMG")]
public static async Task<HttpResponseMessage> Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)]HttpRequest req, TraceWriter log)
{
string url = req.Query["url"];
byte[] EncodedData = Convert.FromBase64String(url);
string DecodedURL = Encoding.UTF8.GetString(EncodedData);
string requestBody = new StreamReader(req.Body).ReadToEnd();
dynamic data = JsonConvert.DeserializeObject(requestBody);
DecodedURL = DecodedURL ?? data?.name;
var api = new HtmlToPdfOrImage.Api("9123314e-219c-342d-a763-0a3dsdf8ad21", "vmZ31vyg");
var ApiResult = api.Convert(new Uri($"{DecodedURL}"), new HtmlToPdfOrImage.GenerateSettings() { OutputType = HtmlToPdfOrImage.OutputType.Image });
string BlobName = Guid.NewGuid().ToString("n");
string ImageURL = await CreateBlob($"{BlobName}.png", (byte[])ApiResult.model, log);
var Result = new HttpResponseMessage(HttpStatusCode.OK);
var oJSON = new { url = ImageURL, hash = BlobName };
var jsonToReturn = JsonConvert.SerializeObject(oJSON);
Result.Content = new StringContent(jsonToReturn);
Result.Content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
return Result;
}
private async static Task<string> CreateBlob(string name, byte[] data, TraceWriter log)
{
string accessKey = "xxx";
string accountName = "xxx";
string connectionString = "DefaultEndpointsProtocol=https;AccountName=" + accountName + ";AccountKey=" + accessKey + ";EndpointSuffix=core.windows.net";
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(connectionString);
CloudBlobClient client = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = client.GetContainerReference("images");
await container.CreateIfNotExistsAsync();
BlobContainerPermissions permissions = await container.GetPermissionsAsync();
permissions.PublicAccess = BlobContainerPublicAccessType.Container;
await container.SetPermissionsAsync(permissions);
CloudBlockBlob blob = container.GetBlockBlobReference(name);
blob.Properties.ContentType = "image/png";
using (Stream stream = new MemoryStream(data))
{
await blob.UploadFromStreamAsync(stream);
}
return blob.Uri.AbsoluteUri;
}
DOUBLE EDIT:
I created empty V2 .net core AF in VS and published it straight away -> i get the same error... I even updated the Microsoft.NET.Sdk.Function to the latest 1.0.14 instead of 1.0.13 and still get the same error. Obviously something in Azure or Visual Studio (15.7.5) is broken?!?!
Solution
On Azure portal, go to Function app settings check your Runtime Version. It is probably Runtime version: 1.0.11913.0 (~1) on your side. This is the problem. Change FUNCTIONS_EXTENSION_VERSION to beta in Application settings and your code should work on Azure.
Explanation
You create a v2 function as your local host version is 2.0.11651.0. So the function runtime should also be beta 2.x(latest is 2.0.11933.0) online.
When you published functions from VS before, you probably saw this prompt
You may have chosen No so that you got the error.
Note that if we publish through CI/CD like VSTS or Git, such notification is not available. So we need to make sure those configurations are set correctly.
Suggestions
As you can see your local host version is 2.0.11651, which is lower than 2.0.11933 on Azure. I do recommend you to update Azure Functions and Web Jobs Tools(on VS menus, Tools->Extensions and Updates) to latest(15.0.40617.0) for VS to consume latest function runtime.
As for your code, I recommend you to create images container and set its public access level manually on portal since this process only requires executing once.
Then we can use blob output bindings.
Add StorageConnection to Application settings with storage connection string. If your images container is in the storage account used by function app (AzureWebJobsStorge in Application settings), ignore this step and delete Connection parameter below, because bindings use that storage account by default.
Add blob output bindings
public static async Task<HttpResponseMessage> Run(...,TraceWriter log,
[Blob("images", FileAccess.Read, Connection = "StorageConnection")] CloudBlobContainer container)
Change CreateBlob method
private async static Task<string> CreateBlob(string name, byte[] data, TraceWriter log, CloudBlobContainer container)
{
CloudBlockBlob blob = container.GetBlockBlobReference(name);
blob.Properties.ContentType = "image/png";
using (Stream stream = new MemoryStream(data))
{
await blob.UploadFromStreamAsync(stream);
}
return blob.Uri.AbsoluteUri;
}
I am currently trying to download a file from Azure blob storage using the DownloadToStream method to download the contents of a blob as a text string.
However I am not getting anything back but an empty string.
Here is my code that I use to connect to the azure blob container and retrieve the blob file.
public static string DownLoadFroalaImageAsString(string blobStorageName, string companyID)
{
// Retrieve storage account from connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
CloudConfigurationManager.GetSetting("StorageConnectionString"));
// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference(companyID.ToLower());
//retrieving the actual filename of the blob
string removeString = "BLOB/";
string trimmedString = blobStorageName.Remove(blobStorageName.IndexOf(removeString), removeString.Length);
// Retrieve reference to a blob named "trimmedString"
CloudBlockBlob blockBlob2 = container.GetBlockBlobReference(trimmedString);
string text;
using (var memoryStream = new MemoryStream())
{
blockBlob2.DownloadToStream(memoryStream);
text = System.Text.Encoding.UTF8.GetString(memoryStream.ToArray());
}
return text;
}
I was following along this documentation however I cannot seem to get it to work. Any help would be greatly appreciated.
However I am not getting anything back but an empty string.
I test your supplied code on my side, it works correctly. I assume that the test blob content is empty in your case. We could trouble shooting with following ways:
1.please have a try to check the Length of memoryStream. If length equal 0 we could know that the blob content is empty.
using (var memoryStream = new MemoryStream())
{
blockBlob2.DownloadToStream(memoryStream);
var length = memoryStream.Length;
text = System.Text.Encoding.UTF8.GetString(memoryStream.ToArray());
}
2.We could upload a blob with content to container, we could do that with Azure portal or Microsoft Azure storage explorer easily. And please have a try test it with uploaded blob.
If you want to get the text from the blob, you can use DownloadTextAsync()
var text = await blockBlob2.DownloadTextAsync();
If you want to return file stream back to an API respoinse, you can use FileStreamResult which is IActionResult.
var stream = await blockBlob2.OpenReadAsync();
return File(stream, blockBlob2.Properties.ContentType, "name");
I want to upload a file to Azure blob storage asynchronously. I have tried the way suggested in the official sdk:
This is how I get the container:
public static class BlobHelper
{
public static CloudBlobContainer GetBlobContainer()
{
// Pull these from config
var blobStorageConnectionString = ConfigurationManager.AppSettings["BlobStorageConnectionString"];
var blobStorageContainerName = ConfigurationManager.AppSettings["BlobStorageContainerName"];
// Create blob client and return reference to the container
var blobStorageAccount = CloudStorageAccount.Parse(blobStorageConnectionString);
var blobClient = blobStorageAccount.CreateCloudBlobClient();
var container = blobClient.GetContainerReference(blobStorageContainerName);
container.CreateIfNotExists();
container.SetPermissions(new BlobContainerPermissions { PublicAccess = BlobContainerPublicAccessType.Blob });
return container;
}
}
And this is how i try to upload the file:
var documentName = Guid.NewGuid().ToString();
CloudBlobContainer container = BlobHelper.GetBlobContainer();
CloudBlockBlob blockBlob = container.GetBlockBlobReference(documentName);
public class FilesService
{
public async Task<string> UploadFiles(HttpContent httpContent)
{
var documentName = Guid.NewGuid().ToString();
CloudBlobContainer container = BlobHelper.GetBlobContainer();
CloudBlockBlob blockBlob = container.GetBlockBlobReference(documentName);
using (var fileStream = System.IO.File.OpenRead(#"path\myfile"))
{
await blockBlob.UploadFromStreamAsync(fileStream);
}
return blockBlob.Uri.ToString();
}
}
The problem is that I do not know how to get the path to my file (it is uploaded by the user).
When I try this:
var rootpath = HttpContext.Current.Server.MapPath("~/App_Data");
var streamProvider = new MultipartFileStreamProvider(rootpath);
await httpContent.ReadAsMultipartAsync(streamProvider);
foreach (var file in streamProvider.FileData)
{
var localName = file.LocalFileName;
using (var fileStream = System.IO.File.OpenRead(file.LocalFileName))
{
await blockBlob.UploadFromStreamAsync(fileStream);
}
}
And when I try a post request. The request just crashes and does not return anything (even an exception);
Solution:
The issue was resolved in the following way. I used a service method in order to be able to upload a collection of files.
In the BlobHelper class I save the needed information about the container and then instantiate it, it is a static class. Using a collection makes it possible to upload a multiple files as a part of the same stream.
I think you are trying to get the path to the file that is being uploaded to the Blob Storage using standard ASP.NET methods and local context. Files uploaded to the blob will not be accessible that way.
Seems like you upload your blob properly. Now, if your file uploaded successfully, your method should return blockBlob.Uri.ToString(), which is the link to your file - you may store it somewhere in the database or anywhere else.
I'm using ASP MVC 5 Razor with Microsoft Azure Blob storage. I can successfully upload documents and images to the Blob Storage using MVC but I am struggling to find some MVC examples how to download and display the files.
It would be quite straightforward to do this if the blobs were stored as public files, but I need them to be private.
Can anyone give me any examples or guidance how to achieve this?
I've got some code below that seems to retrieve the Blob, but I'm not sure what to do with it in MVC to actually display it in a browser.
var fullFileName = "file1.pdf";
var containerName = "default";
// Retrieve storage account from connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(ConfigurationManager.ConnectionStrings["AttachmentStorageConnection"].ConnectionString);
// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference(containerName);
// Retrieve reference to a blob ie "picture.jpg".
CloudBlockBlob blockBlob = container.GetBlockBlobReference(fullFileName);
I'm making an assumption based on your comment
It would be quite straightforward to do this if the blobs were stored
as public files, but I need them to be private
that because the blobs are private you are attempting to return a byte array to the client via the mvc controller.
However, an alternate method would be to use a SharedAccessSignature to provide a client temporary access to the blob which you can then access as a public url. The period for which the url is valid can be specified in your controller. This also has the advantage of taking load away from your controller as the client will download the file directly from storage.
// view model
public class MyViewModel
{
string FileUrl {get; set;}
}
// controller
public ActionResult MyControllerAction
{
var readPolicy = new SharedAccessBlobPolicy()
{
Permissions = SharedAccessBlobPermissions.Read,
SharedAccessExpiryTime = DateTime.UtcNow + TimeSpan.FromMinutes(5)
};
// Your code ------
// Retrieve storage account from connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(ConfigurationManager.ConnectionStrings ["AttachmentStorageConnection"].ConnectionString);
// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference(containerName);
// Retrieve reference to a blob ie "picture.jpg".
CloudBlockBlob blockBlob = container.GetBlockBlobReference(fullFileName);
//------
var newUri = new Uri(blockBlob.Uri.AbsoluteUri + blockBlob.GetSharedAccessSignature(readPolicy));
var viewModel = new MyViewModel()
{
FileUrl = newUri.ToString()
};
return View("MyViewName", viewModel);
}
Then in your view you can use the view model value
//image
<img src="#Model.FileUrl" />
//in a new tab
`Open in new window`
I hope this answers your questions:
In order to download a file or open it in a new window/tab you need to specify the proper Content-Disposition in the header. There's an example here. Basically if you want to download a blob you execute the following. Keep in mind that if the mime type is set to application/octet-stream, the file will not be opened in a new tab. It will be downloaded. You need to set the correct ContentType when you save the blob in Azure.
//Downloads file
public ActionResult Index(string name)
{
Response.AddHeader("Content-Disposition", "attachment; filename=" + name);
var blob = _azureBlobContainer.DownloadData(); //Code that returns CloudBlockBlob
var memStream = new MemoryStream();
blob.DownloadToStream(memStream);
return File(memStream.ToArray(), blob.Properties.ContentType);
}
//Opens file if correct ContentType is passed
public ActionResult Index(string name)
{
Response.AddHeader("Content-Disposition", "inline; filename=" + name); //Set it as inline instead of attached.
var blob = _azureBlobContainer.DownloadData(); //Code that returns CloudBlockBlob
var memStream = new MemoryStream();
blob.DownloadToStream(memStream);
return File(memStream.ToArray(), blob.Properties.ContentType);
}
To open file in a new tab, make sure you specify the target in the view:
In regards to the blob being public/private, you should handle that in your interaction with Azure Storage. If you want to give users permission to access your blobs from outside your application, you should use a Shared Access Signature. Details here.
Hope this helps.
As an extension Alex S's excellent answer, if you're just adding say a 'Download' button or hyperlink then an alternative approach is to make the MyControllerAction method return a Redirect:
return Redirect(newUri.ToString());
And then in your view make the download link open the response from that controller action in a new window:
Download
That will trigger the download of the file to the user and without triggering navigation to a new page.