I have an application that processes file streams based on a list of strings, and the string can either be a file on disk, or a file inside a Zip file. To clean up the code, I'd like to refactor out the process of opening the file.
I've created a method that returns a Stream of the file contents, but because the stream depends on the ZipFile IDisposable, by the time I read the stream, the ZipFile is disposed an throws an exception.
void Main()
{
using (var stream = OpenFileForImport("zipfile.zip;insidefile.txt"))
new StreamReader(stream).ReadToEnd(); // Exception
using (var stream = OpenFileForImport("outside.txt"))
new StreamReader(stream).ReadToEnd(); // Works
}
public static Stream OpenFileForImport(string filePath)
{
var path = Path.Combine(basefolder, filePath);
if (path.Contains(";"))
{
var parts = path.Split(';');
var zipPath = parts[0];
//Error checking logic to ensure zip file exists and is valid...
using (var zip = ZipFile.OpenRead(zipPath))
using (var entry = zip.GetEntry(parts[1]))
{
//Error checking logic to ensure inside file exists within zip file.
return entry.Open();
}
}
var file = new FileInfo(path);
if (file != null)
return file.OpenRead();
return null;
}
I could remove the using clause from the zip and entry declarations, but I doubt they'd ever get disposed. Is there an appropriate pattern to return a disposable, when it depends on other disposables?
Don't return the stream directly, instead return a disposable object which can provide the stream you want to dispose, but that cleans up that stream and the other dependant resources when it is disposed of:
public class NameToBeDetermined : IDisposable
{
private ZipFile zip;
public Stream Stream { get; }
public NameToBeDetermined(ZipFile zip, Stream stream)
{
this.zip = zip;
Stream = stream;
}
public void Dispose()
{
zip.Dispose();
Stream.Dispose();
}
}
Then return that, rather than the stream itself. If it's worth spending the time, you could turn that wrapper into a Stream itself, that just forwards all Stream methods into the composed stream, but that does the extra work when disposing. Whether it's worth the time to create that more involved wrapper rather than having a caller access a Stream property is up to you.
You likely should copy the file from the ZipEntry into a MemoryStream so that you have a copy to work with.
//Error checking logic to ensure zip file exists and is valid...
using (var zip = ZipFile.OpenRead(zipPath))
using (var entry = zip.GetEntry(parts[1]))
{
//Error checking logic to ensure inside file exists within zip file.
MemoryStream stream = new MemoryStream();
entry.Open().CopyTo(stream);
stream.Seek(0, SeekOrigin.Begin);
return stream;
}
Related
I'm trying to serve a file to the users that is packed inside a zip archive on the server.
The project is ASP.NET Core 5.0 MVC project.
I managed to do it by using ZipArchiveEntry.Open() and copying that to a memory stream.
string zipFile = #"D:\all_installs.zip";
using (FileStream fs = new FileStream(zipFile, FileMode.Open))
{
using (ZipArchive zip = new ZipArchive(fs))
{
ZipArchiveEntry entry = zip.Entries.FirstOrDefault(x => x.FullName == "downloadable file.iso");
string name = entry.FullName;
string baseName = Path.GetFileName(name);
//open a stream to the zip entry
Stream stream = entry.Open();
//copy stream to memory
MemoryStream memoryStream = new MemoryStream();
stream.CopyTo(memoryStream); //big memory usage?
memoryStream.Position = 0;
return this.File(memoryStream, "application/octet-stream", baseName);
}
}
This would require a lot of RAM if there are many simultaneous downloads, so instead I would like to serve it directly from the archive, which I know will require the CPU while unpacking it, but that's fine since the download speed will be very limited anyways.
I tried serving stream directly, but I get the following error:
NotSupportedException: Stream does not support reading.
How can I serve the entry-stream directly?
Problem is both FileStream fs and ZipArchive zip are disposed here, so when it's time to write response and asp.net tries to read your zip entry (stream) - it's not available any more, since everything has been disposed.
You need to not dispose them right away but instead tell asp.net to dispose them when it's done writing the response. For that, HttpResponse has method RegisterForDispose, so you need to do something like that:
string zipFile = #"C:\tmp\record.zip";
FileStream fs = null;
ZipArchive zip = null;
Stream stream = null;
try {
fs = new FileStream(zipFile, FileMode.Open);
zip = new ZipArchive(fs);
ZipArchiveEntry entry = zip.Entries.First(x => x.FullName == "24fa535b-2fc9-4ce5-96f4-2ff1ef0d9b64.json");
string name = entry.FullName;
string baseName = Path.GetFileName(name);
//open a stream to the zip entry
stream = entry.Open();
return this.File(stream, "application/octet-stream", baseName);
}
finally {
if (stream != null)
this.Response.RegisterForDispose(stream);
if (zip != null)
this.Response.RegisterForDispose(zip);
if (fs != null)
this.Response.RegisterForDispose(fs);
}
Now asp.net will first write the response, then dispose all your disposables for you.
I have a service that downloads a *.tgz file from a remote endpoint. I use SharpZipLib to extract and write the content of that compressed archive to disk. But now I want to prevent writing the files to disk (because that process doesn't have write permissions on that disk) and keep them in memory.
How can I access the decompressed files from memory? (Let's assume the archive holds simple text files)
Here is what I have so far:
public void Decompress(byte[] byteArray)
{
Stream inStream = new MemoryStream(byteArray);
Stream gzipStream = new GZipInputStream(inStream);
TarArchive tarArchive = TarArchive.CreateInputTarArchive(gzipStream);
tarArchive.ExtractContents(#".");
tarArchive.Close();
gzipStream.Close();
inStream.Close();
}
Check this and this out.
Turns out, ExtractContents() works by iterating over TarInputStream. When you create your TarArchive like this:
TarArchive.CreateInputTarArchive(gzipStream);
it actually wraps the stream you're passing into a TarInputStream. Thus, if you want more fine-grained control over how you extract files, you must use TarInputStream directly.
See, if you can iterate over files, directories and actual file contents like this:
Stream inStream = new MemoryStream(byteArray);
Stream gzipStream = new GZipInputStream(inStream);
using (var tarInputStream = new TarInputStream(gzipStream))
{
TarEntry entry;
while ((entry = tarInputStream.GetNextEntry()) != null)
{
var fileName = entry.Name;
using (var fileContents = new MemoryStream())
{
tarInputStream.CopyEntryContents(fileContents);
// use entry, fileName or fileContents here
}
}
}
I have a WCF service which works in raw format, using streams:
[ServiceContract]
public interface IEncryptingService
{
[WebInvoke(UriTemplate = "/")]
[OperationContract]
Stream SignDocument(Stream requestStream);
}
public class EncryptingService : IEncryptingService
{
public Stream SignDocument(Stream requestStream)
{
string originalFileName = Path.GetTempFileName();
string signedFileName = Path.GetTempFileName();
using (var originalFileStream = File.Open(originalFileName, FileMode.Create, FileAccess.Write))
{
requestStream.CopyTo(originalFileStream);
}
XmlDocumentSigner.SignFile(originalFileName, signedFileName);
return File.Open(signedFileName, FileMode.Open, FileAccess.Read);
}
}
Now, how can I remove this file after WCF ends returning the file?
I have tried to use finally block, but it gets called right after return, and throws exception, since the file is still used by a process.
Of course, these is a workaround like a background worker waiting for a file to be available for deletion, but, in my opinion, it is not like how web-services should be implemented.
I haven't tried it but you could open the file and write that stream to another stream. Something like:
MemoryStream ms = new MemoryStream();
using (FileStream fs = File.OpenRead(signedFileName))
{
//Read from fs and write to ms
}
Then all you will need to do is call delete on the file and return ms:
File.Delete(signedFileName);
return ms;
The solution came to mind inexcusably quickly, and it is absolutely logical: I can simply read file contents to memory and remove the file.
public Stream SignDocument(Stream requestStream)
{
string originalFileName = Path.GetTempFileName();
string signedFileName = Path.GetTempFileName();
using (var originalFileStream = File.Open(originalFileName, FileMode.Create, FileAccess.Write))
{
requestStream.CopyTo(originalFileStream);
}
XmlDocumentSigner.SignFile(originalFileName, signedFileName);
byte[] signedFileBytes = File.ReadAllBytes(signedFileName);
File.Delete(signedFileName);
return new MemoryStream(signedFileBytes);
}
Note that using statement makes this code fail as well:
using (var ms = new MemoryStream(signedFileBytes))
{
return ms;
}
I'm writing a Windows Phone Silverlight app. I want to save an object to a JSON file. I've written the following piece of code.
string jsonFile = JsonConvert.SerializeObject(usr);
IsolatedStorageFile isoStore = IsolatedStorageFile.GetUserStoreForApplication();
IsolatedStorageFileStream isoStream = new IsolatedStorageFileStream("users.json", FileMode.Create, isoStore);
StreamWriter str = new StreamWriter(isoStream);
str.Write(jsonFile);
This is enough to create a JSON file but it is empty. Am I doing something wrong? Wasn't this supposed to write the object to the file?
The problem is that you're not closing the stream.
File I/O in Windows have buffers at the operating system level, and .NET might even implement buffers at the API level, which means that unless you tell the class "Now I'm done", it will never know when to ensure those buffers are propagated all the way down to the platter.
You should rewrite your code just slightly, like this:
using (StreamWriter str = new StreamWriter(isoStream))
{
str.Write(jsonFile);
}
using (...) { ... } will ensure that when the code leaves the block, the { ... } part, it will call IDisposable.Dispose on the object, which in this case will flush the buffers and close the underlying file.
I use these. Shoud work for you as well.
public async Task SaveFile(string fileName, string data)
{
System.IO.IsolatedStorage.IsolatedStorageFile local =
System.IO.IsolatedStorage.IsolatedStorageFile.GetUserStoreForApplication();
if (!local.DirectoryExists("MyDirectory"))
local.CreateDirectory("MyDirectory");
using (var isoFileStream =
new System.IO.IsolatedStorage.IsolatedStorageFileStream(
string.Format("MyDirectory\\{0}.txt", fileName),
System.IO.FileMode.Create, System.IO.FileAccess.ReadWrite, System.IO.FileShare.ReadWrite,
local))
{
using (var isoFileWriter = new System.IO.StreamWriter(isoFileStream))
{
await isoFileWriter.WriteAsync(data);
}
}
}
public async Task<string> LoadFile(string fileName)
{
string data;
System.IO.IsolatedStorage.IsolatedStorageFile local =
System.IO.IsolatedStorage.IsolatedStorageFile.GetUserStoreForApplication();
using (var isoFileStream =
new System.IO.IsolatedStorage.IsolatedStorageFileStream
(string.Format("MyDirectory\\{0}.txt", fileName),
System.IO.FileMode.Open, System.IO.FileAccess.Read, System.IO.FileShare.Read,
local))
{
using (var isoFileReader = new System.IO.StreamReader(isoFileStream))
{
data = await isoFileReader.ReadToEndAsync();
}
}
return data;
}
Is there a way to create a Zip archive that contains multiple files, when the files are currently in memory? The files I want to save are really just text only and are stored in a string class in my application. But I would like to save multiple files in a single self-contained archive. They can all be in the root of the archive.
It would be nice to be able to do this using SharpZipLib.
Use ZipEntry and PutNextEntry() for this. The following shows how to do it for a file, but for an in-memory object just use a MemoryStream
FileStream fZip = File.Create(compressedOutputFile);
ZipOutputStream zipOStream = new ZipOutputStream(fZip);
foreach (FileInfo fi in allfiles)
{
ZipEntry entry = new ZipEntry((fi.Name));
zipOStream.PutNextEntry(entry);
FileStream fs = File.OpenRead(fi.FullName);
try
{
byte[] transferBuffer[1024];
do
{
bytesRead = fs.Read(transferBuffer, 0, transferBuffer.Length);
zipOStream.Write(transferBuffer, 0, bytesRead);
}
while (bytesRead > 0);
}
finally
{
fs.Close();
}
}
zipOStream.Finish();
zipOStream.Close();
Using SharpZipLib for this seems pretty complicated. This is so much easier in DotNetZip. In v1.9, the code looks like this:
using (ZipFile zip = new ZipFile())
{
zip.AddEntry("Readme.txt", stringContent1);
zip.AddEntry("readings/Data.csv", stringContent2);
zip.AddEntry("readings/Index.xml", stringContent3);
zip.Save("Archive1.zip");
}
The code above assumes stringContent{1,2,3} contains the data to be stored in the files (or entries) in the zip archive. The first entry is "Readme.txt" and it is stored in the top level "Directory" in the zip archive. The next two entries are stored in the "readings" directory in the zip archive.
The strings are encoded in the default encoding. There is an overload of AddEntry(), not shown here, that allows you to explicitly specify the encoding to use.
If you have the content in a stream or byte array, not a string, there are overloads for AddEntry() that accept those types. There are also overloads that accept a Write delegate, a method of yours that is invoked to write data into the zip. This works for easily saving a DataSet into a zip file, for example.
DotNetZip is free and open source.
This function should create a byte array from a stream of data: I've created a simple interface for handling files for simplicity
public interface IHasDocumentProperties
{
byte[] Content { get; set; }
string Name { get; set; }
}
public void CreateZipFileContent(string filePath, IEnumerable<IHasDocumentProperties> fileInfos)
{
using (var memoryStream = new MemoryStream())
{
using (var zipArchive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true))
{
foreach(var fileInfo in fileInfos)
{
var entry = zipArchive.CreateEntry(fileInfo.Name);
using (var entryStream = entry.Open())
{
entryStream.Write(fileInfo.Content, 0, fileInfo.Content.Length);
}
}
}
using (var fileStream = new FileStream(filePath, FileMode.OpenOrCreate, System.IO.FileAccess.Write))
{
memoryStream.Position = 0;
memoryStream.CopyTo(fileStream);
}
}
}
Yes, you can use SharpZipLib to do this - when you need to supply a stream to write to, use a MemoryStream.
I come across this problem, using the MSDN example I created this class:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO.Packaging;
using System.IO;
public class ZipSticle
{
Package package;
public ZipSticle(Stream s)
{
package = ZipPackage.Open(s, FileMode.Create);
}
public void Add(Stream stream, string Name)
{
Uri partUriDocument = PackUriHelper.CreatePartUri(new Uri(Name, UriKind.Relative));
PackagePart packagePartDocument = package.CreatePart(partUriDocument, "");
CopyStream(stream, packagePartDocument.GetStream());
stream.Close();
}
private static void CopyStream(Stream source, Stream target)
{
const int bufSize = 0x1000;
byte[] buf = new byte[bufSize];
int bytesRead = 0;
while ((bytesRead = source.Read(buf, 0, bufSize)) > 0)
target.Write(buf, 0, bytesRead);
}
public void Close()
{
package.Close();
}
}
You can then use it like this:
FileStream str = File.Open("MyAwesomeZip.zip", FileMode.Create);
ZipSticle zip = new ZipSticle(str);
zip.Add(File.OpenRead("C:/Users/C0BRA/SimpleFile.txt"), "Some directory/SimpleFile.txt");
zip.Add(File.OpenRead("C:/Users/C0BRA/Hurp.derp"), "hurp.Derp");
zip.Close();
str.Close();
You can pass a MemoryStream (or any Stream) to ZipSticle.Add such as:
FileStream str = File.Open("MyAwesomeZip.zip", FileMode.Create);
ZipSticle zip = new ZipSticle(str);
byte[] fileinmem = new byte[1000];
// Do stuff to FileInMemory
MemoryStream memstr = new MemoryStream(fileinmem);
zip.Add(memstr, "Some directory/SimpleFile.txt");
memstr.Close();
zip.Close();
str.Close();
Note this answer is outdated; since .Net 4.5, the ZipArchive class allows zipping files in-memory. See johnny 5's answer below for how to use it.
You could also do it a bit differently, using a Serializable object to store all strings
[Serializable]
public class MyStrings {
public string Foo { get; set; }
public string Bar { get; set; }
}
Then, you could serialize it into a stream to save it.
To save on space you could use GZipStream (From System.IO.Compression) to compress it. (note: GZip is stream compression, not an archive of multiple files).
That is, of course if what you need is actually to save data, and not zip a few files in a specific format for other software.
Also, this would allow you to save many more types of data except strings.
I was utilizing Cheeso's answer by adding MemoryStreams as the source of the different Excel files. When I downloaded the zip, the files had nothing in them. This could be the way we were getting around trying to create and download a file over AJAX.
To get the contents of the different Excel files to be included in the Zip, I had to add each of the files as a byte[].
using (var memoryStream = new MemoryStream())
using (var zip = new ZipFile())
{
zip.AddEntry("Excel File 1.xlsx", excelFileStream1.ToArray());
zip.AddEntry("Excel File 2.xlsx", excelFileStream2.ToArray());
// Keep the file off of disk, and in memory.
zip.Save(memoryStream);
}
Use a StringReader to read from your string objects and expose them as Stream s.
That should make it easy to feed them to your zip-building code.