Develop Reference

Read Parquet file from Azure blob with out downloading it locally c# .net

We have a parquet formatfile (500 mb) which is located in Azure blob.How to read the file directly from blob and save in memory of c# ,say eg:Datatable.
I am able to read parquet file which is physically located in folder using the below code.
public void ReadParqueFile()
{
using (Stream fileStream = System.IO.File.OpenRead("D:/../userdata1.parquet"))
{
using (var parquetReader = new ParquetReader(fileStream))
{
DataField[] dataFields = parquetReader.Schema.GetDataFields();
for (int i = 0; i < parquetReader.RowGroupCount; i++)
{
using (ParquetRowGroupReader groupReader = parquetReader.OpenRowGroupReader(i))
{
DataColumn[] columns = dataFields.Select(groupReader.ReadColumn).ToArray();
DataColumn firstColumn = columns[0];
Array data = firstColumn.Data;
//int[] ids = (int[])data;
}
}
}
}
}
}
(I am able to read csv file directly from blob using sourcestream).Please kindly suggest a fastest method to read the parquet file directly from blob

Per my experience, the solution to directly read the parquet file from blob is first to generate the blob url with sas token and then to get the stream of HttpClient from the url with sas and finally to read the http response stream via ParquetReader.
First, please refer to the sample code below of the section Create a service SAS for a blob of the offical document Create a service SAS for a container or blob with .NET using Azure Blob Storage SDK for .NET Core.
private static string GetBlobSasUri(CloudBlobContainer container, string blobName, string policyName = null)
{
string sasBlobToken;
// Get a reference to a blob within the container.
// Note that the blob may not exist yet, but a SAS can still be created for it.
CloudBlockBlob blob = container.GetBlockBlobReference(blobName);
if (policyName == null)
{
// Create a new access policy and define its constraints.
// Note that the SharedAccessBlobPolicy class is used both to define the parameters of an ad hoc SAS, and
// to construct a shared access policy that is saved to the container's shared access policies.
SharedAccessBlobPolicy adHocSAS = new SharedAccessBlobPolicy()
{
// When the start time for the SAS is omitted, the start time is assumed to be the time when the storage service receives the request.
// Omitting the start time for a SAS that is effective immediately helps to avoid clock skew.
SharedAccessExpiryTime = DateTime.UtcNow.AddHours(24),
Permissions = SharedAccessBlobPermissions.Read | SharedAccessBlobPermissions.Write | SharedAccessBlobPermissions.Create
};
// Generate the shared access signature on the blob, setting the constraints directly on the signature.
sasBlobToken = blob.GetSharedAccessSignature(adHocSAS);
Console.WriteLine("SAS for blob (ad hoc): {0}", sasBlobToken);
Console.WriteLine();
}
else
{
// Generate the shared access signature on the blob. In this case, all of the constraints for the
// shared access signature are specified on the container's stored access policy.
sasBlobToken = blob.GetSharedAccessSignature(null, policyName);
Console.WriteLine("SAS for blob (stored access policy): {0}", sasBlobToken);
Console.WriteLine();
}
// Return the URI string for the container, including the SAS token.
return blob.Uri + sasBlobToken;
}
Then to get the http response stream of HttpClient from the url with sas token .
var blobUrlWithSAS = GetBlobSasUri(container, blobName);
var client = new HttpClient();
var stream = await client.GetStreamAsync(blobUrlWithSAS);
Finally to read it via ParquetReader, the code comes from Reading Data of GitHub repo aloneguid/parquet-dotnet.
var options = new ParquetOptions { TreatByteArrayAsString = true };
var reader = new ParquetReader(stream, options);

Azure File Share ListFilesAndDirectoriesSegmentedAsync() Fails Authentication

I am using the c#.net api to work with azure file storage but cannot successfully list all files in a fileshare. My code errors with:
Microsoft.WindowsAzure.Storage: Server failed to authenticate the
request. Make sure the value of Authorization header is formed
correctly including the signature.
The following code works perfectly, so my connection to the fileshare 'temp' is fine:
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(connectionString);
CloudFileClient fileClient = storageAccount.CreateCloudFileClient();
CloudFileShare share = fileClient.GetShareReference("temp");
CloudFile f = share.GetRootDirectoryReference().GetFileReference("Report-461fab0e-068e-42f0-b480-c5744272e103-8-14-2018.pdf");
log.Info("size " + f.StreamMinimumReadSizeInBytes.ToString());
The code below results in the discussed authentication error:
FileContinuationToken continuationToken = null;
do
{
var response = await share.GetRootDirectoryReference().ListFilesAndDirectoriesSegmentedAsync(continuationToken);
continuationToken = response.ContinuationToken;
}
while (continuationToken != null);
Any help would be appreciated.
Thanks.

Using key 1 instead of key resolved the issue.

Azure blob storage download to stream returning "" asp.net

I am currently trying to download a file from Azure blob storage using the DownloadToStream method to download the contents of a blob as a text string.
However I am not getting anything back but an empty string.
Here is my code that I use to connect to the azure blob container and retrieve the blob file.
public static string DownLoadFroalaImageAsString(string blobStorageName, string companyID)
{
// Retrieve storage account from connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
CloudConfigurationManager.GetSetting("StorageConnectionString"));
// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference(companyID.ToLower());
//retrieving the actual filename of the blob
string removeString = "BLOB/";
string trimmedString = blobStorageName.Remove(blobStorageName.IndexOf(removeString), removeString.Length);
// Retrieve reference to a blob named "trimmedString"
CloudBlockBlob blockBlob2 = container.GetBlockBlobReference(trimmedString);
string text;
using (var memoryStream = new MemoryStream())
{
blockBlob2.DownloadToStream(memoryStream);
text = System.Text.Encoding.UTF8.GetString(memoryStream.ToArray());
}
return text;
}
I was following along this documentation however I cannot seem to get it to work. Any help would be greatly appreciated.

However I am not getting anything back but an empty string.
I test your supplied code on my side, it works correctly. I assume that the test blob content is empty in your case. We could trouble shooting with following ways:
1.please have a try to check the Length of memoryStream. If length equal 0 we could know that the blob content is empty.
using (var memoryStream = new MemoryStream())
{
blockBlob2.DownloadToStream(memoryStream);
var length = memoryStream.Length;
text = System.Text.Encoding.UTF8.GetString(memoryStream.ToArray());
}
2.We could upload a blob with content to container, we could do that with Azure portal or Microsoft Azure storage explorer easily. And please have a try test it with uploaded blob.

If you want to get the text from the blob, you can use DownloadTextAsync()
var text = await blockBlob2.DownloadTextAsync();
If you want to return file stream back to an API respoinse, you can use FileStreamResult which is IActionResult.
var stream = await blockBlob2.OpenReadAsync();
return File(stream, blockBlob2.Properties.ContentType, "name");

Azure container permissions

Im reading this article.
I have an azure container called "test" that is set to private in azure.
That container has a scorm package in it "121/HEEDENNL/story.html"
I'm using the code below to set the permissions of the folder to read.
However that story.html file needs several other files to run properly.
The story page opens and doesn't return a 403 or 404.
but the files it trying to reference to to make the page run properly are not loading.
How can I get all the files needed for story.html to run properly, be set to read access also?
I thought changing the containers permissions would allow that file to access the files needed.
What am I missing here?
public ActionResult ViewContent(int id)
{
const string pageBlobName = "121/HEEDENNL/story.html";
CloudStorageAccount storageAccount = Common.Constants.Azure.ConnectionStringUrl;
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
//// Retrieve a reference to a container.
// CloudBlobContainer learningModulContainer = blobClient.GetContainerReference(Common.Constants.Azure.LearningModulesContainerName);
CloudBlobContainer learningModulContainer = blobClient.GetContainerReference("test");
PrintBlobs(learningModulContainer);
CloudBlockBlob myindexfile = learningModulContainer.GetBlockBlobReference(pageBlobName);
SharedAccessBlobPermissions permission = SharedAccessBlobPermissions.None;
permission = SharedAccessBlobPermissions.Read;
var token = GetSasForBlob(myindexfile, permission,30);
//this isn't finished.....must get learning module
var module = DataAccessService.Get<LearningModule>(id);
var url = $"{Common.Constants.Azure.StorageAccountUrl}{"test"}/{module.ScormPackage.Path.Replace("index_lms", "story")}{token}";
return Redirect(token);
}
public static string GetSasForBlob(CloudBlockBlob blob, SharedAccessBlobPermissions permission, int sasMinutesValid)
{
// var sasToken = blob.GetSharedAccessSignature(new SharedAccessBlobPolicy()
var sasToken = blob.Container.GetSharedAccessSignature(new SharedAccessBlobPolicy()
{
Permissions = permission,
SharedAccessStartTime = DateTime.UtcNow.AddMinutes(-15),
SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(sasMinutesValid),
});
return string.Format(CultureInfo.InvariantCulture, "{0}{1}", blob.Uri, sasToken);
}

How can I get all the files needed for story.html to run properly, be set to read access also?
Firstly, if possible, you could put these css&js&image etc files that your html page reference in a allow-public-access container.
Secondly, you could provide URL with SAS of the blob resource, and add reference like this in your html page.
<link href="https://{storageaccount}.blob.core.windows.net/styles/Style1.css?st=2017-06-15T02%3A27%3A00Z&se=2017-06-30T02%3A27%3A00Z&sp=r&sv=2015-04-05&sr=b&sig=%2FWwN0F4qyoIH97d7znRKo9lcp84S4oahU9RBwHTnlXk%3D" rel="stylesheet" />
Besides, if you’d like to host your web app, you could try to use Azure app service.

Download and display a private Azure Blob using ASP MVC

I'm using ASP MVC 5 Razor with Microsoft Azure Blob storage. I can successfully upload documents and images to the Blob Storage using MVC but I am struggling to find some MVC examples how to download and display the files.
It would be quite straightforward to do this if the blobs were stored as public files, but I need them to be private.
Can anyone give me any examples or guidance how to achieve this?
I've got some code below that seems to retrieve the Blob, but I'm not sure what to do with it in MVC to actually display it in a browser.
var fullFileName = "file1.pdf";
var containerName = "default";
// Retrieve storage account from connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(ConfigurationManager.ConnectionStrings["AttachmentStorageConnection"].ConnectionString);
// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference(containerName);
// Retrieve reference to a blob ie "picture.jpg".
CloudBlockBlob blockBlob = container.GetBlockBlobReference(fullFileName);

I'm making an assumption based on your comment
It would be quite straightforward to do this if the blobs were stored
as public files, but I need them to be private
that because the blobs are private you are attempting to return a byte array to the client via the mvc controller.
However, an alternate method would be to use a SharedAccessSignature to provide a client temporary access to the blob which you can then access as a public url. The period for which the url is valid can be specified in your controller. This also has the advantage of taking load away from your controller as the client will download the file directly from storage.
// view model
public class MyViewModel
{
string FileUrl {get; set;}
}
// controller
public ActionResult MyControllerAction
{
var readPolicy = new SharedAccessBlobPolicy()
{
Permissions = SharedAccessBlobPermissions.Read,
SharedAccessExpiryTime = DateTime.UtcNow + TimeSpan.FromMinutes(5)
};
// Your code ------
// Retrieve storage account from connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(ConfigurationManager.ConnectionStrings ["AttachmentStorageConnection"].ConnectionString);
// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference(containerName);
// Retrieve reference to a blob ie "picture.jpg".
CloudBlockBlob blockBlob = container.GetBlockBlobReference(fullFileName);
//------
var newUri = new Uri(blockBlob.Uri.AbsoluteUri + blockBlob.GetSharedAccessSignature(readPolicy));
var viewModel = new MyViewModel()
{
FileUrl = newUri.ToString()
};
return View("MyViewName", viewModel);
}
Then in your view you can use the view model value
//image
<img src="#Model.FileUrl" />
//in a new tab
`Open in new window`

I hope this answers your questions:
In order to download a file or open it in a new window/tab you need to specify the proper Content-Disposition in the header. There's an example here. Basically if you want to download a blob you execute the following. Keep in mind that if the mime type is set to application/octet-stream, the file will not be opened in a new tab. It will be downloaded. You need to set the correct ContentType when you save the blob in Azure.
//Downloads file
public ActionResult Index(string name)
{
Response.AddHeader("Content-Disposition", "attachment; filename=" + name);
var blob = _azureBlobContainer.DownloadData(); //Code that returns CloudBlockBlob
var memStream = new MemoryStream();
blob.DownloadToStream(memStream);
return File(memStream.ToArray(), blob.Properties.ContentType);
}
//Opens file if correct ContentType is passed
public ActionResult Index(string name)
{
Response.AddHeader("Content-Disposition", "inline; filename=" + name); //Set it as inline instead of attached.
var blob = _azureBlobContainer.DownloadData(); //Code that returns CloudBlockBlob
var memStream = new MemoryStream();
blob.DownloadToStream(memStream);
return File(memStream.ToArray(), blob.Properties.ContentType);
}
To open file in a new tab, make sure you specify the target in the view:
In regards to the blob being public/private, you should handle that in your interaction with Azure Storage. If you want to give users permission to access your blobs from outside your application, you should use a Shared Access Signature. Details here.
Hope this helps.

As an extension Alex S's excellent answer, if you're just adding say a 'Download' button or hyperlink then an alternative approach is to make the MyControllerAction method return a Redirect:
return Redirect(newUri.ToString());
And then in your view make the download link open the response from that controller action in a new window:
Download
That will trigger the download of the file to the user and without triggering navigation to a new page.