I want to read the contents of a csv file that I have in my FileShare on my storageaccount in Azure, using Azure.Storage.Files.Shares library.
I am able to connect to the file using the ShareFileClient, but how can I then read the contents and process them (append a new line), in my code?
ShareFileClient file = ConnectToFile();
Steam content = await file.OpenReadAsync(true);
// gives a Stream object, that I cannot get to work to get the content.
What are the next steps to stream the contents of this file? I'm stuck at trying to get the read action to work with something like
using (Steam stream = new Stream() )
{
// The actions to read the stream go here
}
Any suggestions on how this could be achieved?
Regarding the issue, please refer to the following code
// read
using (var stream = await file.OpenReadAsync().ConfigureAwait(false))
using (var reader = new StreamReader(stream)) {
// read csv file one line by line
while (!reader.EndOfStream) {
var line =await reader.ReadLineAsync().ConfigureAwait(false);
Console.WriteLine(line);
}
}
//write
ShareFileProperties properties = await file.GetPropertiesAsync().ConfigureAwait(false);
var myPosition = properties.ContentLength;
var newData = "42,11,58, \"N\",85,12,45, \"W\", \"Worcester\", ND"+Environment.NewLine;
var bytes = Encoding.UTF8.GetBytes(newData);
await file.SetHttpHeadersAsync(myPosition + bytes.Length);
using (var writer = await file.OpenWriteAsync(overwrite: false, position:(myPosition -1)).ConfigureAwait(false)) {
await writer.WriteAsync(bytes, 0, bytes.Length);
await writer.FlushAsync();
}
Related
I have files (from 3rd parties) that are being FTP'd to a directory on our server. I download them and process them even 'x' minutes. Works great.
Now, some of the files are .zip files. Which means I can't process them. I need to unzip them first.
FTP has no concept of zip/unzipping - so I'll need to grab the zip file, unzip it, then process it.
Looking at the MSDN zip api, there seems to be no way i can unzip to a memory stream?
So is the only way to do this...
Unzip to a file (what directory? need some -very- temp location ...)
Read the file contents
Delete file.
NOTE: The contents of the file are small - say 4k <-> 1000k.
Zip compression support is built in:
using System.IO;
using System.IO.Compression;
// ^^^ requires a reference to System.IO.Compression.dll
static class Program
{
const string path = ...
static void Main()
{
using(var file = File.OpenRead(path))
using(var zip = new ZipArchive(file, ZipArchiveMode.Read))
{
foreach(var entry in zip.Entries)
{
using(var stream = entry.Open())
{
// do whatever we want with stream
// ...
}
}
}
}
}
Normally you should avoid copying it into another stream - just use it "as is", however, if you absolutely need it in a MemoryStream, you could do:
using(var ms = new MemoryStream())
{
stream.CopyTo(ms);
ms.Position = 0; // rewind
// do something with ms
}
You can use ZipArchiveEntry.Open to get a stream.
This code assumes the zip archive has one text file.
using (FileStream fs = new FileStream(path, FileMode.Open))
using (ZipArchive zip = new ZipArchive(fs) )
{
var entry = zip.Entries.First();
using (StreamReader sr = new StreamReader(entry.Open()))
{
Console.WriteLine(sr.ReadToEnd());
}
}
using (ZipArchive archive = new ZipArchive(webResponse.GetResponseStream()))
{
foreach (ZipArchiveEntry entry in archive.Entries)
{
Stream s = entry.Open();
var sr = new StreamReader(s);
var myStr = sr.ReadToEnd();
}
}
Looks like here is what you need:
using (var za = ZipFile.OpenRead(path))
{
foreach (var entry in za.Entries)
{
using (var r = new StreamReader(entry.Open()))
{
//your code here
}
}
}
You can use SharpZipLib among a variety of other libraries to achieve this.
You can use the following code example to unzip to a MemoryStream, as shown on their wiki:
using ICSharpCode.SharpZipLib.Zip;
// Compresses the supplied memory stream, naming it as zipEntryName, into a zip,
// which is returned as a memory stream or a byte array.
//
public MemoryStream CreateToMemoryStream(MemoryStream memStreamIn, string zipEntryName) {
MemoryStream outputMemStream = new MemoryStream();
ZipOutputStream zipStream = new ZipOutputStream(outputMemStream);
zipStream.SetLevel(3); //0-9, 9 being the highest level of compression
ZipEntry newEntry = new ZipEntry(zipEntryName);
newEntry.DateTime = DateTime.Now;
zipStream.PutNextEntry(newEntry);
StreamUtils.Copy(memStreamIn, zipStream, new byte[4096]);
zipStream.CloseEntry();
zipStream.IsStreamOwner = false; // False stops the Close also Closing the underlying stream.
zipStream.Close(); // Must finish the ZipOutputStream before using outputMemStream.
outputMemStream.Position = 0;
return outputMemStream;
// Alternative outputs:
// ToArray is the cleaner and easiest to use correctly with the penalty of duplicating allocated memory.
byte[] byteArrayOut = outputMemStream.ToArray();
// GetBuffer returns a raw buffer raw and so you need to account for the true length yourself.
byte[] byteArrayOut = outputMemStream.GetBuffer();
long len = outputMemStream.Length;
}
Ok so combining all of the above, suppose you want to in a very simple way take a zip file called
"file.zip" and extract it to "C:\temp" folder. (Note: This example was only tested for compress text files) You may need to do some modifications for binary files.
using System.IO;
using System.IO.Compression;
static void Main(string[] args)
{
//Call it like this:
Unzip("file.zip",#"C:\temp");
}
static void Unzip(string sourceZip, string targetPath)
{
using (var z = ZipFile.OpenRead(sourceZip))
{
foreach (var entry in z.Entries)
{
using (var r = new StreamReader(entry.Open()))
{
string uncompressedFile = Path.Combine(targetPath, entry.Name);
File.WriteAllText(uncompressedFile,r.ReadToEnd());
}
}
}
}
I'm trying to implement a StreamSocket communication and the server is now sending me a ".zip" file in byte[] chuncks. It looks like my byte[] in memory is fine (amount of bytes are the same as original .zip file in the server) but when I save those bytes in a file the unzip program says it's 'corrupted'.
I've tried several options but none of them worked for me:
1
var file = await StorageFile.CreateFileAsync();
using(var stream = await file.OpenStreamForWriteAsync())
{
stream.Write(myBytes, ...);
}
2
var file = await StorageFile.CreateFileAsync();
await FileIO.WriteBytesAsync(file, myBytes);
3
var file = await StorageFile.CreateFileAsync();
using (var fileStream = await file.OpenStreamForWriteAsync())
{
var sessionData = new MemoryStream(myBytes);
sessionData.Seek(0, SeekOrigin.Begin);
await sessionData.CopyToAsync(fileStream);
}
4
var file = await StorageFile.CreateFileAsync();
using (var writer = new BinaryWriter(await file.OpenStreamForWriteAsync())
{
writer.WriteBytes(myBytes);
}
If anyone has some new ideas please share.
Regards.
I am attempting to download a bunch of files that I am zipping up(archiving) via ZipoutputStream.
using (var zipStream = new ZipOutputStream(outputMemStream))
{
foreach (var documentIdString in documentUniqueIdentifiers)
{
...
var blockBlob = container.GetBlockBlobReference(documentId.ToString());
var fileMemoryStream = new MemoryStream();
blockBlob.DownloadToStream(fileMemoryStream);
zipStream.SetLevel(3);
fileMemoryStream.Position = 0;
ZipEntry newEntry = new ZipEntry(document.FileName);
newEntry.DateTime = DateTime.Now;
zipStream.PutNextEntry(newEntry);
fileMemoryStream.Seek(0, SeekOrigin.Begin);
StreamUtils.Copy(fileMemoryStream, zipStream, new byte[4096]);
zipStream.IsStreamOwner = false; // False stops the Close also Closing the underlying stream.
}
outputMemStream.Seek(0, SeekOrigin.Begin);
return outputMemStream;
}
In my controller I am returning the following code that should download the Zip file i created in the previous example. The controller actions downloads the file as it should in the browser, but the Archived File is empty. I can see the content length populated returning from the method above...
file.Seek(0, SeekOrigin.Begin);
return File(file, "application/octet-stream", "Archive.zip");
Does anyone have any idea why my file that is returned by my controller is empty or corrupt?
I believe you need to close your entries and your final zip stream. You should also using and dispose all of your streams. Try this:
using (var zipStream = new ZipOutputStream(outputMemStream))
{
zipStream.IsStreamOwner = false;
// Set compression level
zipStream.SetLevel(3);
foreach (var documentIdString in documentUniqueIdentifiers)
{
...
var blockBlob = container.GetBlockBlobReference(documentId.ToString());
using (var fileMemoryStream = new MemoryStream())
{
// Populate stream with bytes
blockBlob.DownloadToStream(fileMemoryStream);
// Create zip entry and set date
ZipEntry newEntry = new ZipEntry(document.FileName);
newEntry.DateTime = DateTime.Now;
// Put entry RECORD, not actual data
zipStream.PutNextEntry(newEntry);
// Copy date to zip RECORD
StreamUtils.Copy(fileMemoryStream, zipStream, new byte[4096]);
// Mark this RECORD closed in the zip
zipStream.CloseEntry();
}
}
// Close the zip stream, parent stays open due to !IsStreamOwner
zipStream.Close();
outputMemStream.Seek(0, SeekOrigin.Begin);
return outputMemStream;
}
EDIT - you should remove:
// Reset position of stream
fileMemoryStream.Position = 0;
Pretty sure that's the problem.
I have files (from 3rd parties) that are being FTP'd to a directory on our server. I download them and process them even 'x' minutes. Works great.
Now, some of the files are .zip files. Which means I can't process them. I need to unzip them first.
FTP has no concept of zip/unzipping - so I'll need to grab the zip file, unzip it, then process it.
Looking at the MSDN zip api, there seems to be no way i can unzip to a memory stream?
So is the only way to do this...
Unzip to a file (what directory? need some -very- temp location ...)
Read the file contents
Delete file.
NOTE: The contents of the file are small - say 4k <-> 1000k.
Zip compression support is built in:
using System.IO;
using System.IO.Compression;
// ^^^ requires a reference to System.IO.Compression.dll
static class Program
{
const string path = ...
static void Main()
{
using(var file = File.OpenRead(path))
using(var zip = new ZipArchive(file, ZipArchiveMode.Read))
{
foreach(var entry in zip.Entries)
{
using(var stream = entry.Open())
{
// do whatever we want with stream
// ...
}
}
}
}
}
Normally you should avoid copying it into another stream - just use it "as is", however, if you absolutely need it in a MemoryStream, you could do:
using(var ms = new MemoryStream())
{
stream.CopyTo(ms);
ms.Position = 0; // rewind
// do something with ms
}
You can use ZipArchiveEntry.Open to get a stream.
This code assumes the zip archive has one text file.
using (FileStream fs = new FileStream(path, FileMode.Open))
using (ZipArchive zip = new ZipArchive(fs) )
{
var entry = zip.Entries.First();
using (StreamReader sr = new StreamReader(entry.Open()))
{
Console.WriteLine(sr.ReadToEnd());
}
}
using (ZipArchive archive = new ZipArchive(webResponse.GetResponseStream()))
{
foreach (ZipArchiveEntry entry in archive.Entries)
{
Stream s = entry.Open();
var sr = new StreamReader(s);
var myStr = sr.ReadToEnd();
}
}
Looks like here is what you need:
using (var za = ZipFile.OpenRead(path))
{
foreach (var entry in za.Entries)
{
using (var r = new StreamReader(entry.Open()))
{
//your code here
}
}
}
You can use SharpZipLib among a variety of other libraries to achieve this.
You can use the following code example to unzip to a MemoryStream, as shown on their wiki:
using ICSharpCode.SharpZipLib.Zip;
// Compresses the supplied memory stream, naming it as zipEntryName, into a zip,
// which is returned as a memory stream or a byte array.
//
public MemoryStream CreateToMemoryStream(MemoryStream memStreamIn, string zipEntryName) {
MemoryStream outputMemStream = new MemoryStream();
ZipOutputStream zipStream = new ZipOutputStream(outputMemStream);
zipStream.SetLevel(3); //0-9, 9 being the highest level of compression
ZipEntry newEntry = new ZipEntry(zipEntryName);
newEntry.DateTime = DateTime.Now;
zipStream.PutNextEntry(newEntry);
StreamUtils.Copy(memStreamIn, zipStream, new byte[4096]);
zipStream.CloseEntry();
zipStream.IsStreamOwner = false; // False stops the Close also Closing the underlying stream.
zipStream.Close(); // Must finish the ZipOutputStream before using outputMemStream.
outputMemStream.Position = 0;
return outputMemStream;
// Alternative outputs:
// ToArray is the cleaner and easiest to use correctly with the penalty of duplicating allocated memory.
byte[] byteArrayOut = outputMemStream.ToArray();
// GetBuffer returns a raw buffer raw and so you need to account for the true length yourself.
byte[] byteArrayOut = outputMemStream.GetBuffer();
long len = outputMemStream.Length;
}
Ok so combining all of the above, suppose you want to in a very simple way take a zip file called
"file.zip" and extract it to "C:\temp" folder. (Note: This example was only tested for compress text files) You may need to do some modifications for binary files.
using System.IO;
using System.IO.Compression;
static void Main(string[] args)
{
//Call it like this:
Unzip("file.zip",#"C:\temp");
}
static void Unzip(string sourceZip, string targetPath)
{
using (var z = ZipFile.OpenRead(sourceZip))
{
foreach (var entry in z.Entries)
{
using (var r = new StreamReader(entry.Open()))
{
string uncompressedFile = Path.Combine(targetPath, entry.Name);
File.WriteAllText(uncompressedFile,r.ReadToEnd());
}
}
}
}
I'm trying to create a resx file and write it to a stream so that I might return it as a string instead of immediately saving it to a file. However, when I try to read that stream, it is empty. What am I doing wrong here? i did verify that the entries are not null. I can actually use the ResXResourceWriter constructor that saves it to disk successfully, but I'm trying to avoid using temp files. Also, I can see the stream is 0k before the loop and about 8k in length after the loop.
using (var stream = new MemoryStream())
{
using (var resx = new ResXResourceWriter(stream))
{
// build the resx and write to memory
foreach (var entry in InputFile.Entries.Values)
{
resx.AddResource(new ResXDataNode(entry.Key, entry.Value) { Comment = entry.Comment });
}
var reader = new StreamReader(stream);
var text = reader.ReadToEnd(); // text is an empty string here!
return null;
}
}
You need to flush and reset the output/stream before trying to read it. This should work, using Generate and Position:
resx.Generate();
stream.Position = 0;
var reader = new StreamReader(stream);
var text = reader.ReadToEnd();
return text;