I've been downloading files from an FTP server via the WebClient object that the .NET namespace provides and then write the bytes to a actual file via a BinaryWriter. All is good. However, now, the files have dramatically increased in size and I'm worried about memory constraints so I'd like to create a download stream, create an file stream, and line by line read from the download and write to the file.
I'm nervous since I couldn't find a nice example of this. Here's my end result:
var request = new WebClient();
// Omitted code to add credentials, etc..
var downloadStream = new StreamReader(request.OpenRead(ftpFilePathUri.ToString()));
using (var writeStream = File.Open(toLocation, FileMode.CreateNew))
{
using (var writer = new StreamWriter(writeStream))
{
while (!downloadStream.EndOfStream)
{
writer.Write(downloadStream.ReadLine());
}
}
}
Am I going about this incorrect/better way/etc?
Have you tried the following usage of WebClient class?
using (WebClient webClient = new WebClient())
{
webClient.DownloadFile("url", "filePath");
}
Update
using (var client = new WebClient())
using (var stream = client.OpenRead("..."))
using (var file = File.Create("..."))
{
stream.CopyTo(file);
}
If you want to download file explicitly using customized buffer size:
public static void DownloadFile(Uri address, string filePath)
{
using (var client = new WebClient())
using (var stream = client.OpenRead(address))
using (var file = File.Create(filePath))
{
var buffer = new byte[4096];
int bytesReceived;
while ((bytesReceived = stream.Read(buffer, 0, buffer.Length)) != 0)
{
file.Write(buffer, 0, bytesReceived);
}
}
}
Related
I'm trying to create a zip stream on the fly with some byte array data and make it download via my MVC action.
But the downloaded file always gives the following corrupted error when opened in windows.
And this error when I try to xtract from 7z
But note that the files extracted from the 7z is not corrupted.
I'm using ZipArchive and the below is my code.
private byte[] GetZippedPods(IEnumerable<POD> pods, long consignmentID)
{
using (var zipStream = new MemoryStream())
{
//Create an archive and store the stream in memory.
using (var zipArchive = new ZipArchive(zipStream, ZipArchiveMode.Create, true))
{
int index = 1;
foreach (var pod in pods)
{
var zipEntry = zipArchive.CreateEntry($"POD{consignmentID}{index++}.png", CompressionLevel.NoCompression);
using (var originalFileStream = new MemoryStream(pod.ByteData))
{
using (var zipEntryStream = zipEntry.Open())
{
originalFileStream.CopyTo(zipEntryStream);
}
}
}
return zipStream.ToArray();
}
}
}
public ActionResult DownloadPOD(long consignmentID)
{
var pods = _consignmentService.GetPODs(consignmentID);
var fileBytes = GetZippedPods(pods, consignmentID);
return File(fileBytes, MediaTypeNames.Application.Octet, $"PODS{consignmentID}.zip");
}
What am I doing wrong here.
Any help would be highly appreciated as I'm struggling with this for a whole day.
Thanks in advance
Move zipStream.ToArray() outside of the zipArchive using.
The reason for your problem is that the stream is buffered. There's a few ways to deal wtih it:
You can set the stream's AutoFlush property to true.
You can manually call .Flush() on the stream.
Or, since it's MemoryStream and you're using .ToArray(), you can simply allow the stream to be Closed/Disposed first (which we've done by moving it outside the using).
I Dispose ZipArchive And error solved
public static byte[] GetZipFile(Dictionary<string, List<FileInformation>> allFileInformations)
{
MemoryStream compressedFileStream = new MemoryStream();
//Create an archive and store the stream in memory.
using (var zipArchive = new ZipArchive(compressedFileStream, ZipArchiveMode.Create, true))
{
foreach (var fInformation in allFileInformations)
{
var files = allFileInformations.Where(x => x.Key == fInformation.Key).SelectMany(x => x.Value).ToList();
for (var i = 0; i < files.Count; i++)
{
ZipArchiveEntry zipEntry = zipArchive.CreateEntry(fInformation.Key + "/" + files[i].FileName);
var caseAttachmentModel = Encoding.UTF8.GetBytes(files[i].Content);
//Get the stream of the attachment
using (var originalFileStream = new MemoryStream(caseAttachmentModel))
using (var zipEntryStream = zipEntry.Open())
{
//Copy the attachment stream to the zip entry stream
originalFileStream.CopyTo(zipEntryStream);
}
}
}
//i added this line
zipArchive.Dispose();
return compressedFileStream.ToArray();
}
}
public void SaveZipFile(){
var zipFileArray = Global.GetZipFile(allFileInformations);
var zipFile = new MemoryStream(zipFileArray);
FileStream fs = new FileStream(path + "\\111.zip",
FileMode.Create,FileAccess.Write);
zipFile.CopyTo(fs);
zipFile.Flush();
fs.Close();
zipFile.Close();
}
I was also having problems with this and I found my issue was not the generation of the archive itself but rather how I was handing my GET request in AngularJS.
This post helped me: how to download a zip file using angular
The key was adding responseType: 'arraybuffer' to my $http call.
factory.serverConfigExportZIP = function () {
return $http({
url: dataServiceBase + 'serverConfigExport',
method: "GET",
responseType: 'arraybuffer'
})
};
you can remove "using" and use Dispose and Close methods
it's work for me
...
zip.Dispose();
zipStream.Close();
return zipStream.ToArray();
I know this is a C# question but for managed C++, delete the ZipArchive^ after you're done with it to fix the error.
ZipArchive^ zar = ZipFile::Open(starget, ZipArchiveMode::Create);
ZipFileExtensions::CreateEntryFromFile(zar, sfile1, "file.txt");
ZipFileExtensions::CreateEntryFromFile(zar, sfile2, "file2.txt");
delete zar;
when i wanted to create zip file directly from MemoryStream which i used for ZipArchive i was getting error ( "unexpected end of data" or zero length file )
there are three points to get ride of this error
set the last parameter of ZipArchive constructor to true ( it leaves to leave stream open after ZipArchive disposed )
call dispose() on ZipArchive and dispose it manually.
create another MemoryStream based on which you set in ZipArchive constructor, by calling ToArray() method.
here is sample code :
using (var memoryStream = new MemoryStream())
{
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create,))
{
foreach (var s3Object in objectList.S3Objects)
{
var entry = archive.CreateEntry(s3Object.Key, CompressionLevel.NoCompression);
using (var entryStream = entry.Open())
{
var request = new GetObjectRequest { BucketName = command.BucketName, Key = s3Object.Key };
using (var getObjectResponse = await client.GetObjectAsync(request))
{
await getObjectResponse.ResponseStream.CopyToAsync(entryStream);
}
}
}
archive.Dispose();
using (var fileStream = new FileStream(outputFileName, FileMode.Create, FileAccess.Write))
{
var zipFileMemoryStream = new MemoryStream(memoryStream.ToArray());
zipFileMemoryStream.CopyTo(fileStream);
zipFileMemoryStream.Flush();
fileStream.Close();
zipFileMemoryStream.Close();
}
}
}
I had the same problem... In this case I just needed to move the ToArray() (byte[]) from MemoryStream outside the using (var zipArchive = new ZipArchive...
I think it is necessary for using related to ZipArchive to completely close and dispose of the file before converting it into a byte array
I'm trying on c# to download a zip file from a webservice and extract an entry in the memory but when I try to read the stream how is in the documentation of the dotnetzip I get the exception "This stream does not support seek operations” in the "ZipFile.Read(stream)" part.
Somebody could tell me what I'm doing wrong? Thanks in advance
urlAuthentication="https://someurl/?login=foo&token=faa"
var request = (HttpWebRequest)WebRequest.Create(urlAuthentication);
request.Proxy = WebRequest.DefaultWebProxy;
request.Credentials = System.Net.CredentialCache.DefaultCredentials; ;
request.Proxy.Credentials = System.Net.CredentialCache.DefaultCredentials;
using (var ms = new MemoryStream())
{
using (var response = (HttpWebResponse)request.GetResponse())
{
using (var stream =response.GetResponseStream())
{
using (ZipFile zipout = ZipFile.Read(stream))
{
ZipEntry entry = zipout["file1.xml"];
entry.Extract(ms);
}
}
}
}
Apparently dotnetzip requires a stream to support seek operations and the response stream of a HttpWebResponse does not support seeking.
You can solve this issue by first downloading the entire file in memory, and then accessing it:
using (var ms = new MemoryStream())
{
using (MemoryStream seekable = new MemoryStream())
{
using (var stream = response.GetResponseStream())
{
int bytes;
byte[] buffer = new byte[1024];
while ((bytes = stream.Read(buffer, 0, buffer.Length)) > 0)
{
seekable.Write(buffer, 0, bytes);
}
}
seekable.Position = 0;
using (ZipFile zipout = ZipFile.Read(seekable))
{
ZipEntry entry = zipout["file1.xml"];
entry.Extract(ms);
}
}
// access ms
}
I have files (from 3rd parties) that are being FTP'd to a directory on our server. I download them and process them even 'x' minutes. Works great.
Now, some of the files are .zip files. Which means I can't process them. I need to unzip them first.
FTP has no concept of zip/unzipping - so I'll need to grab the zip file, unzip it, then process it.
Looking at the MSDN zip api, there seems to be no way i can unzip to a memory stream?
So is the only way to do this...
Unzip to a file (what directory? need some -very- temp location ...)
Read the file contents
Delete file.
NOTE: The contents of the file are small - say 4k <-> 1000k.
Zip compression support is built in:
using System.IO;
using System.IO.Compression;
// ^^^ requires a reference to System.IO.Compression.dll
static class Program
{
const string path = ...
static void Main()
{
using(var file = File.OpenRead(path))
using(var zip = new ZipArchive(file, ZipArchiveMode.Read))
{
foreach(var entry in zip.Entries)
{
using(var stream = entry.Open())
{
// do whatever we want with stream
// ...
}
}
}
}
}
Normally you should avoid copying it into another stream - just use it "as is", however, if you absolutely need it in a MemoryStream, you could do:
using(var ms = new MemoryStream())
{
stream.CopyTo(ms);
ms.Position = 0; // rewind
// do something with ms
}
You can use ZipArchiveEntry.Open to get a stream.
This code assumes the zip archive has one text file.
using (FileStream fs = new FileStream(path, FileMode.Open))
using (ZipArchive zip = new ZipArchive(fs) )
{
var entry = zip.Entries.First();
using (StreamReader sr = new StreamReader(entry.Open()))
{
Console.WriteLine(sr.ReadToEnd());
}
}
using (ZipArchive archive = new ZipArchive(webResponse.GetResponseStream()))
{
foreach (ZipArchiveEntry entry in archive.Entries)
{
Stream s = entry.Open();
var sr = new StreamReader(s);
var myStr = sr.ReadToEnd();
}
}
Looks like here is what you need:
using (var za = ZipFile.OpenRead(path))
{
foreach (var entry in za.Entries)
{
using (var r = new StreamReader(entry.Open()))
{
//your code here
}
}
}
You can use SharpZipLib among a variety of other libraries to achieve this.
You can use the following code example to unzip to a MemoryStream, as shown on their wiki:
using ICSharpCode.SharpZipLib.Zip;
// Compresses the supplied memory stream, naming it as zipEntryName, into a zip,
// which is returned as a memory stream or a byte array.
//
public MemoryStream CreateToMemoryStream(MemoryStream memStreamIn, string zipEntryName) {
MemoryStream outputMemStream = new MemoryStream();
ZipOutputStream zipStream = new ZipOutputStream(outputMemStream);
zipStream.SetLevel(3); //0-9, 9 being the highest level of compression
ZipEntry newEntry = new ZipEntry(zipEntryName);
newEntry.DateTime = DateTime.Now;
zipStream.PutNextEntry(newEntry);
StreamUtils.Copy(memStreamIn, zipStream, new byte[4096]);
zipStream.CloseEntry();
zipStream.IsStreamOwner = false; // False stops the Close also Closing the underlying stream.
zipStream.Close(); // Must finish the ZipOutputStream before using outputMemStream.
outputMemStream.Position = 0;
return outputMemStream;
// Alternative outputs:
// ToArray is the cleaner and easiest to use correctly with the penalty of duplicating allocated memory.
byte[] byteArrayOut = outputMemStream.ToArray();
// GetBuffer returns a raw buffer raw and so you need to account for the true length yourself.
byte[] byteArrayOut = outputMemStream.GetBuffer();
long len = outputMemStream.Length;
}
Ok so combining all of the above, suppose you want to in a very simple way take a zip file called
"file.zip" and extract it to "C:\temp" folder. (Note: This example was only tested for compress text files) You may need to do some modifications for binary files.
using System.IO;
using System.IO.Compression;
static void Main(string[] args)
{
//Call it like this:
Unzip("file.zip",#"C:\temp");
}
static void Unzip(string sourceZip, string targetPath)
{
using (var z = ZipFile.OpenRead(sourceZip))
{
foreach (var entry in z.Entries)
{
using (var r = new StreamReader(entry.Open()))
{
string uncompressedFile = Path.Combine(targetPath, entry.Name);
File.WriteAllText(uncompressedFile,r.ReadToEnd());
}
}
}
}
I'm working in C#, and I'm downloading for the internet a zip file with one XML file in it. and I wish to load this XML file. This is what I have so far:
byte[] data;
WebClient webClient = new WebClient();
try {
data = webClient.DownloadData(downloadUrl);
}
catch (Exception ex) {
Console.WriteLine("Error in DownloadData (Ex:{0})", ex.Message);
throw;
}
if (data == null) {
Console.WriteLine("Bulk data is null");
throw new Exception("Bulk data is null");
}
//Create the stream
MemoryStream stream = new MemoryStream(data);
XmlDocument document = new XmlDocument();
//Gzip
GZipStream gzipStream = new GZipStream(stream, CompressionMode.Decompress);
//Load report straight from the gzip stream
try {
document.Load(gzipStream);
}
catch (Exception ex) {
Console.WriteLine("Error in Load (Ex:{0})", ex.Message);
throw;
}
in document.Load I'm always getting the following exception:
The magic number in GZip header is not correct. Make sure you are passing in a GZip stream.
What I'm doing wrong?
Apparently SharpZipLib is now unmaintained and you probably want to avoid it:
https://stackoverflow.com/a/593030
In .NET 4.5 there is now built in support for zip files, so for your example it would be:
var data = new WebClient().DownloadData(downloadUrl);
//Create the stream
var stream = new MemoryStream(data);
var document = new XmlDocument();
//zip
var zipArchive = new ZipArchive(stream);
//Load report straight from the zip stream
document.Load(zipArchive.Entries[0].Open());
If you have a byte array that contains a zip archive with a single file, you can use the ZipArchive class to get an unzipped byte array with the file's data.
ZipArchive is contained in .NET 4.5, in the assembly System.IO.Compression.FileSystem (you need to reference it explicitly).
The following function, adapted from this answer, works for me:
public static byte[] UnzipSingleEntry(byte[] zipped)
{
using (var memoryStream = new MemoryStream(zipped))
{
using (var archive = new ZipArchive(memoryStream))
{
foreach (ZipArchiveEntry entry in archive.Entries)
{
using (var entryStream = entry.Open())
{
using (var reader = new BinaryReader(entryStream))
{
return reader.ReadBytes((int)entry.Length);
}
}
}
}
}
return null; // To quiet my compiler
}
I am using SharpZipLib and it's working great !
Below is a function that encapsulate the library
public static void Compress(FileInfo sourceFile, string destinationFileName,string destinationTempFileName)
{
Crc32 crc = new Crc32();
string zipFile = Path.Combine(sourceFile.Directory.FullName, destinationTempFileName);
zipFile = Path.ChangeExtension(zipFile, ZIP_EXTENSION);
using (FileStream fs = File.Create(zipFile))
{
using (ZipOutputStream zOut = new ZipOutputStream(fs))
{
zOut.SetLevel(9);
ZipEntry entry = new ZipEntry(ZipEntry.CleanName(destinationFileName));
entry.DateTime = DateTime.Now;
entry.ZipFileIndex = 1;
entry.Size = sourceFile.Length;
using (FileStream sourceStream = sourceFile.OpenRead())
{
crc.Reset();
long len = sourceFile.Length;
byte[] buffer = new byte[bufferSize];
while (len > 0)
{
int readSoFar = sourceStream.Read(buffer, 0, buffer.Length);
crc.Update(buffer, 0, readSoFar);
len -= readSoFar;
}
entry.Crc = crc.Value;
zOut.PutNextEntry(entry);
len = sourceStream.Length;
sourceStream.Seek(0, SeekOrigin.Begin);
while (len > 0)
{
int readSoFar = sourceStream.Read(buffer, 0, buffer.Length);
zOut.Write(buffer, 0, readSoFar);
len -= readSoFar;
}
}
zOut.Finish();
zOut.Close();
}
fs.Close();
}
}
As the others have mentioned GZip and Zip are not the same so you might need to use a zip library. I use a library called: DotNetZip - available from the below site:
http://dotnetzip.codeplex.com/
From GZipStream Class description:
Compressed GZipStream objects written to a file with an extension of .gz can be decompressed using many common compression tools; however, this class does not inherently provide functionality for adding files to or extracting files from .zip archives
So unless you control server-side files, I'd suggest looking for specific zip-targeted library (SharpZipLib for example).
Behold the code:
using (var client = new WebClient())
{
using (var stream = client.OpenWrite("http://localhost/", "POST"))
{
stream.Write(post, 0, post.Length);
}
}
Now, how do I read the HTTP output?
It looks like you have a byte[] of data to post; in which case I expect you'll find it easier to use:
byte[] response = client.UploadData(address, post);
And if the response is text, something like:
string s = client.Encoding.GetString(response);
(or your choice of Encoding - perhaps Encoding.UTF8)
If you want to keep streams everywhere and avoid allocating huge arrays of bytes, which is good practise (for example, if you plan to post big files), you still can do it with a derived version of WebClient. Here is a sample code that does it.
using (var client = new WebClientWithResponse())
{
using (var stream = client.OpenWrite(myUrl))
{
// open a huge local file and send it
using (var file = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
file.CopyTo(stream);
}
}
// get response as an array of bytes. You'll need some encoding to convert to string, etc.
var bytes = client.Response;
}
And here is the customized WebClient:
public class WebClientWithResponse : WebClient
{
// we will store the response here. We could store it elsewhere if needed.
// This presumes the response is not a huge array...
public byte[] Response { get; private set; }
protected override WebResponse GetWebResponse(WebRequest request)
{
var response = base.GetWebResponse(request);
var httpResponse = response as HttpWebResponse;
if (httpResponse != null)
{
using (var stream = httpResponse.GetResponseStream())
{
using (var ms = new MemoryStream())
{
stream.CopyTo(ms);
Response = ms.ToArray();
}
}
}
return response;
}
}