I have a ASP.NET core application that need to send a stream(the stream is posted by client) to Microsoft Cognitive Service to get the ID. And then send the same stream to azure blob for backup, the file name should be the ID received from Cognitive Service.
But it seems like the MemoryStream ms closed after used by faceServiceClient : an error accrued at the second "ms.Position = 0" statement saying "Cannot access a closed stream".
public static async Task CreatPerson(string _key, HttpRequest _req)
{
var faceServiceClient = new FaceServiceClient(_key);
using (MemoryStream ms = new MemoryStream())
{
_req.Body.CopyTo(ms);
ms.Position = 0;
var facesTask = faceServiceClient.AddFaceToFaceListAsync("himlens", ms);
//init azure blob
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(AZURE_STORE_CONN_STR);
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference("xxx");
var faces = await facesTask;
var blob = container.GetBlockBlobReference(faces.PersistedFaceId.ToString());
ms.Position = 0;//Error Here
await blob.UploadFromStreamAsync(ms);
}
}
I'm confused about it, can anybody help me solve this problem?
Thanks！
ms.Position = 0;//Error Here
To easily fix it, you could create a new instance of MemoryStream and copy the value from ms. Then you could upload it to your blob storage. Code below is for your reference.
using (MemoryStream ms = new MemoryStream())
{
_req.Body.CopyTo(ms);
ms.Position = 0;
//new code which I added
MemoryStream ms2 = new MemoryStream();
ms.CopyTo(ms2);
ms.Position = 0;
var facesTask = faceServiceClient.AddFaceToFaceListAsync("himlens", ms);
//init azure blob
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(AZURE_STORE_CONN_STR);
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference("xxx");
var faces = await facesTask;
var blob = container.GetBlockBlobReference(faces.PersistedFaceId.ToString());
//Code which I modified
ms2.Position = 0;
await blob.UploadFromStreamAsync(ms2);
}
Related
Context: Encrypt and Decrypt an audio file (.wav) in Azure Storage.
Issue: inputStream must be seek-able (when encrypting) "await pgp.EncryptStreamAsync(sourceStream, outputStream);"
I'm not a C# Developer :)
Thank you for your help,
Here is the code i'm using:
static async Task Main()
{
string connectionString = Environment.GetEnvironmentVariable("AZURE_STORAGE_CONNECTION_STRING");
////Create a unique name for the container
string containerName = "audioinput";
string filename = "abc.wav";
//// Create a BlobServiceClient object which will be used to create a container client
BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
BlobContainerClient sourcecontainer = blobServiceClient.GetBlobContainerClient(containerName);
BlobClient blobClient = sourcecontainer.GetBlobClient(filename);
if (sourcecontainer.Exists())
{
var sourceStream = new MemoryStream();
//Download blob to MemoryStream
await blobClient.DownloadToAsync(sourceStream);
sourceStream.Position = 0;
//OutputStream
await using var outputStream = new MemoryStream();
//Get encryptionkeys
EncryptionKeys encryptionKeys;
using (Stream publicKeyStream = new FileStream(#"...\public.asc", FileMode.Open))
encryptionKeys = new EncryptionKeys(publicKeyStream);
PGP pgp = new PGP(encryptionKeys);
await pgp.EncryptStreamAsync(sourceStream, outputStream);
}
else
{
Console.WriteLine("container doesn't exist");
}
}
I am trying to take a file and split it into piece and then push each new smaller file piece to azure. I have tried writing a MemoryStream to azure, but that causes the file to upload immediately, but the file is basically empty. I have tried using a BufferedStream which allows the data to be sent as i am writing to it, but i am not sure how to end the stream. I have tried to close each of the different streams i am using, but that does not work as it results in a stream closed exception. Any idea how to mark the stream as complete so the azure library will know to finish the file upload?
It does work to wait until the full file is build and then upload the memory stream, but i would like to be able to write to it while it is uploading if possible.
CloudBlobClient blobClient = StorageAccount.CreateCloudBlobClient();
CloudBlobContainer blobContainer = blobClient.GetContainerReference("containerName");
using (FileStream fileStream = File.Open(path)
{
int key = 0;
CsvWriter csvWriter = null;
MemoryStream memoryStream = null;
BufferedStream bufferedStream = null;
StreamWriter streamWriter = null;
Task<StorageUri> uploadTask = null;
using (var reader = new StreamReader(fileStream))
using (var csv = new CsvReader(reader, CultureInfo.InvariantCulture))
{
csv.Read();
csv.ReadHeader();
await foreach (model row in csv.GetRecordsAsync<MyModel>())
{
if (row.KeyColumn != key)
{
if (memoryStream != null)
{
//Wiat for the current upload to finish
await csvWriter.FlushAsync();
csvWriter.Dispose();
await uploadTask;
}
//Start New Upload
key = row.KeyColumn;
memoryStream = new MemoryStream();
bufferedStream = new BufferedStream(memoryStream)
streamWriter = new StreamWriter(bufferedStream);
csvWriter = new CsvWriter(streamWriter, CultureInfo.InvariantCulture);
csvWriter.WriteHeader<MyModel>();
await csvWriter.FlushAsync();
CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference($"file_{key}.csv");
uploadTask = blockBlob.UploadFromStreamAsync(bufferedStream);
}
csvWriter.WriteRecord(row);
await csvWriter.FlushAsync();
}
if (memoryStream != null)
{
await csvWriter.FlushAsync();
csvWriter.Dispose();
await uploadTask;
}
}
}
I'm using Azure Blob Storage to save some files. I'm having issues downloading this stream & am not sure why. I don't get any errors, just an empty stream. I've verified that the file exists in the container, and even ran code to list all files in the container. Any help would be greatly appreciated.
private async Task<MemoryStream> GetMemoryStreamAsync(string fileName)
{
var storageAccountName = Environment.GetEnvironmentVariable("storage_account_name");
var storageAccountKey = Environment.GetEnvironmentVariable("storage_access_key");
var storageContainerName = Environment.GetEnvironmentVariable("storage_container_name");
CloudStorageAccount storageAccount = new CloudStorageAccount(new Microsoft.WindowsAzure.Storage.Auth.StorageCredentials(storageAccountName, storageAccountKey), true);
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference(storageContainerName);
CloudBlockBlob blockBlob = container.GetBlockBlobReference(fileName);
MemoryStream stream = new MemoryStream();
await blockBlob.DownloadToStreamAsync(stream);
return stream;
}
You'll need to set the position to zero before returning the stream so that the consumer of the stream reads it from the beginning.
MemoryStream stream = new MemoryStream();
await blockBlob.DownloadToStreamAsync(stream);
stream.Position = 0;
return stream;
I'm rewriting my C# code to use Azure Blob storage instead of filesystem. So far no problems rewriting code for normal fileoperations. But I have some code that uses async write from a stream:
using (var stream = await Request.Content.ReadAsStreamAsync())
{
FileStream fileStream = new FileStream(#"c:\test.txt", FileMode.Create, FileAccess.Write, FileShare.None);
await stream.CopyToAsync(fileStream).ContinueWith(
(copyTask) =>
{
fileStream.Close();
});
}
I need to change the above to use Azure CloudBlockBlob or CloudBlobStream - but can't find a way to declare a stream object that copyToAsync can write to.
You would want to use UploadFromStreamAsync method on CloudBlockBlob. Here's a sample code to do so (I have not tried running this code though):
var cred = new StorageCredentials(accountName, accountKey);
var account = new CloudStorageAccount(cred, true);
var blobClient = account.CreateCloudBlobClient();
var container = blobClient.GetContainerReference("container-name");
var blob = container.GetBlockBlobReference("blob-name");
using (var stream = await Request.Content.ReadAsStreamAsync())
{
stream.Position = 0;
await blob.UploadFromStreamAsync(stream);
}
I have written a C# application to archive data from a SQL Server table into an Azure blob. The archiving is configured by a JSON file and the values retrieved from the JSON file dictate what data to retrieve and archive.
The data needs to be stored in a blob name in this format
year/month/day/hour/older-than-[query-date]
Where query-date is the current date minus a number of days specified in the JSON file.
The issue I am having is how to incorporate compression to the process.
We would like to compress the data being archived to save space.
Currently the JSON settings mean that any data only than 30 days should be archived, but this results in about 3.7 million rows of data, so sometimes I get out of memory exceptions.
Regardless, how can I compress using GZip each row of data to the Azure blob? Here is existing code.
using (SqlDataAdapter adr = new SqlDataAdapter(comm))
{
adr.Fill(data);
data.TableName = config.TargetTableName;
}
CloudStorageAccount storageAccount = CloudStorageAccount.Parse("blank");
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
blobClient.DefaultRequestOptions.ParallelOperationThreadCount = 20;
blobClient.DefaultRequestOptions.MaximumExecutionTime = TimeSpan.FromMinutes(20);
blobClient.DefaultRequestOptions.ServerTimeout = TimeSpan.FromMinutes(20);
CloudBlobContainer container = blobClient.GetContainerReference(config.AzureContainerName);
StringBuilder jsonData = new StringBuilder();
CloudBlockBlob blob = container.GetBlockBlobReference($"{config.TargetTableName}/{DateTime.Now.Year}/{DateTime.Now.Month}/{DateTime.Now.Day}/{DateTime.Now.Hour}/Older-Than-{queryParameter.Value}.log");
using (var writeStream = blob.OpenWrite())
{
using (var writer = new StreamWriter(writeStream))
{
data.WriteXml(writer, XmlWriteMode.WriteSchema);
}
}
I suggest you write your data to a MemoryStream. Then we can compress the memory stream and write it to Azure Blob Service. Code below is for your reference.
CloudBlockBlob blob = container.GetBlockBlobReference($"{config.TargetTableName}/{DateTime.Now.Year}/{DateTime.Now.Month}/{DateTime.Now.Day}/{DateTime.Now.Hour}/Older-Than-{queryParameter.Value}.log");
using (var writeStream = blob.OpenWrite())
{
MemoryStream memoryStream = new MemoryStream();
using (var writer = new StreamWriter(memoryStream))
{
data.WriteXml(writer, XmlWriteMode.WriteSchema);
}
using (GZipStream compressionStream = new GZipStream(writeStream,
CompressionMode.Compress))
{
memoryStream.Position = 0;
memoryStream.CopyTo(compressionStream);
}
}