Develop Reference

c# files downloaded with httpwebrequest and cookies get corrupted

I am trying to make a program which is able to download files with URI(URL) using httpwebrequest and cookies(for credential information to keep login status).
I can download files with following code but files get corrupted after being downloaded.
when I download xlsx file(on the web page) into text file at local drive, I see some part of numbers and words from an original file in a corrupted file, therefore I assume I have reached to the right file.
however, when I download xlsx file(on the web page) in xlsx file at local drive, it seems like it fails to open saying
excel cannot open the file 'filename.xlsx' because the file format or
file extension is not valid. Verify that the file has not been
corrupted and that the file extension matches the format of the file.
Is there any way I can keep fully original file content after I download?
I attach a part of result content as well.
private void btsDownload_Click(object sender, EventArgs e)
{
try
{
string filepath1 = #"PathAndNameofFile.txt";
string sTmpCookieString = GetGlobalCookies(webBrowser1.Url.AbsoluteUri);
HttpWebRequest fstRequest = (HttpWebRequest)WebRequest.Create(sLinkDwPage);
fstRequest.Method = "GET";
fstRequest.CookieContainer = new System.Net.CookieContainer();
fstRequest.CookieContainer.SetCookies(webBrowser1.Document.Url, sTmpCookieString);
HttpWebResponse fstResponse = (HttpWebResponse)fstRequest.GetResponse();
StreamReader sr = new StreamReader(fstResponse.GetResponseStream());
string sPageData = sr.ReadToEnd();
sr.Close();
string sViewState = ExtractInputHidden(sPageData, "__VIEWSTATE");
string sEventValidation = this.ExtractInputHidden(sPageData, "__EVENTVALIDATION");
string sUrl = ssItemLinkDwPage;
HttpWebRequest hwrRequest = (HttpWebRequest)WebRequest.Create(sUrl);
hwrRequest.Method = "POST";
string sPostData = "__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE=" + sViewState + "&__EVENTVALIDATION=" + sEventValidation + "&Name=test" + "&Button1=Button";
ASCIIEncoding encoding = new ASCIIEncoding();
byte[] bByteArray = encoding.GetBytes(sPostData);
hwrRequest.ContentType = "application/x-www-form-urlencoded";
Uri convertedURI = new Uri(ssDwPage);
hwrRequest.CookieContainer = new System.Net.CookieContainer();
hwrRequest.CookieContainer.SetCookies(convertedURI, sTmpCookieString);
hwrRequest.ContentLength = bByteArray.Length;
Stream sDataStream = hwrRequest.GetRequestStream();
sDataStream.Write(bByteArray, 0, bByteArray.Length);
sDataStream.Close();
using (WebResponse response = hwrRequest.GetResponse())
{
using (sDataStream = response.GetResponseStream())
{
StreamReader reader = new StreamReader(sDataStream);
{
string sResponseFromServer = reader.ReadToEnd();
FileStream fs = File.Open(filepath1, FileMode.OpenOrCreate, FileAccess.Write);
Byte[] info = encoding.GetBytes(sResponseFromServer);
fs.Write(info, 0, info.Length);
fs.Close();
reader.Close();
sDataStream.Close();
response.Close();
}
}
}
}
catch
{
MessageBox.Show("Error");
}
}

StreamReader is for dealing with text data. Using it corrupts your binary data(excel file).
Write sDataStream directly to file. For ex.
sDataStream.CopyTo(fs)
PS: I prepared a test case (using similar logic) to show how your code doesn't work
var binaryData = new byte[] { 128,255 };
var sr = new StreamReader(new MemoryStream(binaryData));
var str3 = sr.ReadToEnd();
var newData = new ASCIIEncoding().GetBytes(str3); //<-- 63,63
Just compare binaryData with newData

When creating a Zip file in memory, why do I have to parse the MemoryStream for the Zip to be created correctly?

I've looked at other questions on here but nobody answers my question as to why I can't use the already existing MemoryStream that exists to create the zip.
Here is a working API method to get a zipped folder containing files. The files already exist on the server and the method simply looks up there location and, all in memory, retrieves all the files and sends the user a HttpResponMessage object with the Zipped folder containing all the files. There is one line of code below that doesn't seem to make sense however. I have to parse the MemoryStream to a byte array then back to a MemoryStream in order for the Zip to be created and sent back correctly. This seems wrong, but I don't know how to correct it. (See below "//Works:" and "//Does Not work:")
public HttpResponseMessage GetEvalByCompany([FromUri]Guid id, [FromUri] int year)
{
try
{
string companyName = _portalDB.Companies.FirstOrDefault(c => c.Id == id).FirmName;
//Get all evaluation file paths for a given company for a given year
var files = _evaluationDB.Evaluations.Where(j => j.CompanyId == id).Select(j => #"C:\Path\To\File" + j.RelativePath.Replace("~", #"").Replace(#"/", #"\")).Distinct().ToList<string>();
using (var outStream = new MemoryStream())
{
using (var archive = new ZipArchive(outStream, ZipArchiveMode.Create, true))
{
foreach (var record in files)
{
var fileInArchive = archive.CreateEntryFromFile(record, Path.GetFileName(record), CompressionLevel.Optimal);
}
}
// Create a response
HttpResponseMessage response = Request.CreateResponse(HttpStatusCode.OK);
//Works:
response.Content = new StreamContent(new MemoryStream(outStream.ToArray())); //Why does this need to happen?
//Does Not work:
//response.Content = new StreamContent(outStream); //Why doesn't this work?
// Add appropriate headers to make browser recognize response as a download
response.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment");
response.Content.Headers.ContentDisposition.FileName = "Evaluations for Company - " + companyName + ".zip";
response.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/zip");
return response;
}
}
catch (Exception ex)
{
return ErrorHandler.HandleException(ex, Request);
}
}
Based on the answers I received I was able to fix the code so it doesn't have to parse the stream unnecessarily. Here's the updated code for future devs looking for the solution simply. Removing the using doesn't seem to cause any GC issues either. I found a link to explain that a bit better as well. (MemoryStream.Close() or MemoryStream.Dispose())
public HttpResponseMessage GetEvalByCompany([FromUri]Guid id, [FromUri] int year)
{
try
{
string companyName = _portalDB.Companies.FirstOrDefault(c => c.Id == id).FirmName;
//Get all evaluation file paths for a given company for a given year
var files = _evaluationDB.Evaluations.Where(j => j.CompanyId == id).Select(j => #"C:\Path\To\File" + j.RelativePath.Replace("~", #"").Replace(#"/", #"\")).Distinct().ToList<string>();
var outStream = new MemoryStream();
using (var archive = new ZipArchive(outStream, ZipArchiveMode.Create, true))
{
foreach (var record in files)
{
var fileInArchive = archive.CreateEntryFromFile(record, Path.GetFileName(record), CompressionLevel.Optimal);
}
}
// Create a response
HttpResponseMessage response = Request.CreateResponse(HttpStatusCode.OK);
//Updated code
outStream.Seek(0, SeekOrigin.Begin);
response.Content = new StreamContent(outStream); //works now because outStream isn't in a using block
// Add appropriate headers to make browser recognize response as a download
response.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment");
response.Content.Headers.ContentDisposition.FileName = "Evaluations for Company - " + companyName + ".zip";
response.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/zip");
return response;
}
catch (Exception ex)
{
return ErrorHandler.HandleException(ex, Request);
}
}

Remove
using (var outStream = new MemoryStream()),
leave instead only:
var outStream = new MemoryStream();
Then later :
outStream.Seek(0, SeekOrigin.Begin)
response.Content = new StreamContent(outStream);

The problem here is that your original MemoryStream is getting disposed before it is used. StreamContent is keeping a reference to an object that is disposed. You need to remove the using block for outStream and somehow arrange to dispose of it later.

Display PDF file from C# to Angular2 view

I'm trying to display a PDF that was previously uploaded into the server. The PDF resides inside the App_Data folder. I want to fetch it using C# Web API 2 and display it in my Angular 2 frontend view.
C# code:
HttpResponseMessage response = Request.CreateResponse(HttpStatusCode.BadRequest);
string filename = "somefilename.pdf";
byte[] buffer = new byte[0];
MemoryStream memStream = new MemoryStream();
if (filename != "") {
PdfReader reader = new PdfReader(Path.Combine(HttpContext.Current.Request.PhysicalApplicationPath) + "/App_Data/Uploads/PDFs/" + filename);
PdfStamper stamper = new PdfStamper(reader, memStream);
response = Request.CreateResponse(HttpStatusCode.OK);
buffer = memStream.ToArray();
response.Content = new StreamContent(new MemoryStream(buffer));
response.Content.Headers.ContentType = new MediaTypeHeaderValue("application/pdf");
response.Content.Headers.ContentLength = buffer.Length;
ContentDispositionHeaderValue contentDisposition = null;
if (ContentDispositionHeaderValue.TryParse("inline; filename=" + filename, out contentDisposition))
{
response.Content.Headers.ContentDisposition = contentDisposition;
}
}
return response;
Angular 2 code:
this.documentsService.getFile()
.subscribe((response: any) => {
let file = new Blob([response], { type: 'application/pdf' });
let url = URL.createObjectURL(file);
window.open(url);
});
I think my Angular code is wrong. But nevertheless, my first goal here is to fetch it from the MVC side. Right now the file gets downloaded and it's corrupted or there's not fetch correctly. It always give me a 15bytes size PDF file, so I know there's something wrong. I'm using iTextSharp in my C# backend.
Thanks in advance!

From my comment (providing an answer is easier for code):
if (filename != "") {
var fs = System.IO.File.OpenRead(filename);
_myDisposable = fs; // see further down in this answer.
var response = Request.CreateResponse(HttpStatusCode.OK);
response.Content = new StreamContent(fs);
response.Content.Headers.ContentType = new MediaTypeHeaderValue("application/pdf");
return response;
}
return response;
Edit: The code I posted above works for me. The only problem is disposing of the stream. The best way to probably do that is by creating a private IDisposable member on your controller class, and then adding a dispose override, sort of like this:
private IDisposable _myDisposable;
public override void Dispose(bool disposing)
{
if (disposing && _myDisposable != null)
_myDisposable.Dispose();
}
Please note that this assumes ASP.NET is done with the request (which seems reasonable, because it's disposing of your controller), which should be correct.

Creating a dynamic zip of a bunch of URLs on the fly

I am trying to create a zip file of any size on the fly. The source of the zip archive is a bunch of URLs and could be potentially large (500 4MB JPGs in the list). I want to be able to do everything inside the request and have the download start right away and have the zip created and streamed as it is built. It should not have to reside in memory or on disk on the server.
The closest I have come is this:
Note: urls is a keyvaluepair of URLs to the file names as they should exist in the created zip
Response.ClearContent();
Response.ClearHeaders();
Response.ContentType = "application/zip";
Response.AddHeader("Content-Disposition", "attachment; filename=DyanmicZipFile.zip");
using (var memoryStream = new MemoryStream())
{
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true))
{
foreach (KeyValuePair<string, string> fileNamePair in urls)
{
var zipEntry = archive.CreateEntry(fileNamePair.Key);
using (var entryStream = zipEntry.Open())
using (WebClient wc = new WebClient())
wc.OpenRead(GetUrlForEntryName(fileNamePair.Key)).CopyTo(entryStream);
//this doesn't work either
//using (var streamWriter = new StreamWriter(entryStream))
// using (WebClient wc = new WebClient())
// streamWriter.Write(wc.OpenRead(GetUrlForEntryName(fileNamePair.Key)));
}
}
memoryStream.WriteTo(Response.OutputStream);
}
HttpContext.Current.ApplicationInstance.CompleteRequest();
This code gives me a zip file, but each JPG file inside the zip is just a text file that says "System.Net.ConnectStream" I have other attempts on this that do build a zip file with the proper files inside, but you can tell by the delay at the beginning that the server is completely building the zip in memory and then blasting it down at the end. It doesn't respond at all when the file count gets near 50. The part in comments gives me the same result I have tried Ionic.Zip as well.
This is .NET 4.5 on IIS8. I am building with VS2013 and trying to run this on AWS Elastic Beanstalk.

So to answer my own question - here is the solution that works for me:
private void ProcessWithSharpZipLib()
{
byte[] buffer = new byte[4096];
ICSharpCode.SharpZipLib.Zip.ZipOutputStream zipOutputStream = new ICSharpCode.SharpZipLib.Zip.ZipOutputStream(Response.OutputStream);
zipOutputStream.SetLevel(0); //0-9, 9 being the highest level of compression
zipOutputStream.UseZip64 = ICSharpCode.SharpZipLib.Zip.UseZip64.Off;
foreach (KeyValuePair<string, string> fileNamePair in urls)
{
using (WebClient wc = new WebClient())
{
using (Stream wcStream = wc.OpenRead(GetUrlForEntryName(fileNamePair.Key)))
{
ICSharpCode.SharpZipLib.Zip.ZipEntry entry = new ICSharpCode.SharpZipLib.Zip.ZipEntry(ICSharpCode.SharpZipLib.Zip.ZipEntry.CleanName(fileNamePair.Key));
zipOutputStream.PutNextEntry(entry);
int count = wcStream.Read(buffer, 0, buffer.Length);
while (count > 0)
{
zipOutputStream.Write(buffer, 0, count);
count = wcStream.Read(buffer, 0, buffer.Length);
if (!Response.IsClientConnected)
{
break;
}
Response.Flush();
}
}
}
}
zipOutputStream.Close();
Response.Flush();
Response.End();
}

You're trying to create a zip file and have it stream while it's being created. This turns out to be very difficult.
You need to understand the Zip file format. In particular, notice that a local file entry has header fields that can't be updated (CRC, compressed and uncompressed file sizes) until the entire file has been compressed. So at minimum you'll have to buffer at least one entire file before sending it to the response stream.
So at best you could do something like:
open archive
for each file
create entry
write file to entry
read entry raw data and send to the response output stream
The problem you'll run into is that there's no documented way (and no undocumented way that I'm aware of) to read the raw data. The only read method ends up decompressing the data and throwing away the headers.
There might be some other zip library available that can do what you need. I wouldn't suggest trying to do it with ZipArchive.

There must be a way in the zip component you are using that allows for delayed addition of entries to the archive, ie. adding them after the zip.Save() is called. I am using IonicZip using the delayed technique, The code to download flickr albums looks like this:
protected void Page_Load(object sender, EventArgs e)
{
if (!IsLoggedIn())
Response.Redirect("/login.aspx");
else
{
// this is dco album id, find out what photosetId it maps to
string albumId = Request.Params["id"];
Album album = findAlbum(new Guid(albumId));
Flickr flickr = FlickrInstance();
PhotosetPhotoCollection photos = flickr.PhotosetsGetPhotos(album.PhotosetId, PhotoSearchExtras.OriginalUrl | PhotoSearchExtras.Large2048Url | PhotoSearchExtras.Large1600Url);
Response.Clear();
Response.BufferOutput = false;
// ascii only
//string archiveName = album.Title + ".zip";
string archiveName = "photos.zip";
Response.ContentType = "application/zip";
Response.AddHeader("content-disposition", "attachment; filename=" + archiveName);
int picCount = 0;
string picNamePref = album.PhotosetId.Substring(album.PhotosetId.Length - 6);
using (ZipFile zip = new ZipFile())
{
zip.CompressionMethod = CompressionMethod.None;
zip.CompressionLevel = Ionic.Zlib.CompressionLevel.None;
zip.ParallelDeflateThreshold = -1;
_map = new Dictionary<string, string>();
foreach (Photo p in photos)
{
string pictureUrl = p.Large2048Url;
if (string.IsNullOrEmpty(pictureUrl))
pictureUrl = p.Large1600Url;
if (string.IsNullOrEmpty(pictureUrl))
pictureUrl = p.LargeUrl;
string pictureName = picNamePref + "_" + (++picCount).ToString("000") + ".jpg";
_map.Add(pictureName, pictureUrl);
zip.AddEntry(pictureName, processPicture);
}
zip.Save(Response.OutputStream);
}
Response.Close();
}
}
private volatile Dictionary<string, string> _map;
protected void processPicture(string pictureName, Stream output)
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(_map[pictureName]);
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream input = response.GetResponseStream())
{
byte[] buf = new byte[8092];
int len;
while ( (len = input.Read(buf, 0, buf.Length)) > 0)
output.Write(buf, 0, len);
}
output.Flush();
}
}
This ways the code in Page_Load gets to zip.Save() immediately, the download starts (the client is presented with the "Save As" box, and only then the images are pulled from flickr.

This code working fine but when I host my code on windows azure as cloud service it corrupts my zip file throwing message invalid file
private void ProcessWithSharpZipLib(){
byte[] buffer = new byte[4096];
ICSharpCode.SharpZipLib.Zip.ZipOutputStream zipOutputStream = new ICSharpCode.SharpZipLib.Zip.ZipOutputStream(Response.OutputStream);
zipOutputStream.SetLevel(0); //0-9, 9 being the highest level of compression
zipOutputStream.UseZip64 = ICSharpCode.SharpZipLib.Zip.UseZip64.Off;
foreach (KeyValuePair<string, string> fileNamePair in urls)
{
using (WebClient wc = new WebClient())
{
using (Stream wcStream = wc.OpenRead(GetUrlForEntryName(fileNamePair.Key)))
{
ICSharpCode.SharpZipLib.Zip.ZipEntry entry = new ICSharpCode.SharpZipLib.Zip.ZipEntry(ICSharpCode.SharpZipLib.Zip.ZipEntry.CleanName(fileNamePair.Key));
zipOutputStream.PutNextEntry(entry);
int count = wcStream.Read(buffer, 0, buffer.Length);
while (count > 0)
{
zipOutputStream.Write(buffer, 0, count);
count = wcStream.Read(buffer, 0, buffer.Length);
if (!Response.IsClientConnected)
{
break;
}
Response.Flush();
}
}
}
}
zipOutputStream.Close();
Response.Flush();
Response.End();
}
This code is working fine on local machine but not after deployed on server. It corrupts my zip file if its large in size.

WP8 Gzip compression

WP8 does not support Gzip compression, but there is 3rd party libraries that will allow you do so, i have tried many but i am not able to make it work. this is my latest try:
var handler = new HttpClientHandler();
if (handler.SupportsAutomaticDecompression)
{
handler.AutomaticDecompression = DecompressionMethods.GZip |
DecompressionMethods.Deflate;
}
Uri myUri = new Uri("http://www.web.com");
HttpClient client = new HttpClient(handler);
client.BaseAddress = myUri;
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.Add("ubq-compression", "gzip");
HttpRequestMessage req = new HttpRequestMessage(HttpMethod.Post, myUri);
req.Content = new StringContent(finalURL, Encoding.UTF8);
HttpResponseMessage rm = client.SendAsync(req).Result;
string rst = await rm.Content.ReadAsStringAsync();
the API return to me an array of bytes but the the first 300 are not Gziped but everything else is
i need to unzip everything that comes after the 300 bytes.
i am using the http://www.nuget.org/packages/Microsoft.Net.Http
// i am splitting the array
byte[] hJ = res.Take(300).ToArray();
byte[] bJ = res.Skip(300).ToArray();
bj is what need to be decompressed.
i am trying this
MemoryStream stream = new MemoryStream();
stream.Write(bj, 0, bj.Length);
using (var inStream = new MemoryStream(bj))
{
var bigStreamsss = new GZipStream(inStream, CompressionMode.Decompress, true);
using (var bigStreamOut = new MemoryStream())
{
bigStreamsss.CopyTo(bigStreamOut);
output = Encoding.UTF8.GetString(bigStreamOut.ToArray(), 0, bigStreamOut.ToArray().Length);
}
}
but it is always failing on this line "
var bigStreamsss = new GZipStream(inStream, CompressionMode.Decompress, true);
"
Any help would be much appreciated

If you are using the compression header there's nothing you need to do. The server compresses and the client decompresses automatically and you don't have to worry about anything. However it sounds like you're using some proprietary content compression standard where you only compress some of it. If that's the case don't mess with any compression settings on the http client, and instead use a 3rd party uncompress library. Just seek 300 bytes on the response stream, then pass that stream to the library. You should be able to use my inflater from my gzip library, that you can find here: https://github.com/dotMorten/SharpGIS.GZipWebClient/blob/master/src/SharpGIS.GZipWebClient/GZipDeflateStream.cs
It's extremely light-weight (it's just this one file). First call
myResultStream.Seek(300, SeekOrigin.Begin);
If the stream isn't seekable, you will need to read the first 300 bytes first though.
Then use my class to decompress the rest:
Stream gzipStream = new SharpGIS.GZipInflateStream(myResultStream);
You can now read the gzipStream as if it was an uncompressed stream.
However I really don't understand why you don't use standard http compression and let the server compress everything including the first 300 bytes. It's much easier and better for the all the kittens out there.

You can use http://www.componentace.com/zlib_.NET.htm library (available through nuget):
part of request handler:
try {
var request = (HttpWebRequest)callbackResult.AsyncState;
using (var response = request.EndGetResponse(callbackResult))
using (var stream = response.GetResponseStream()) {
bool zip = false;
if (response.Headers.AllKeys.Contains("Content-Encoding") &&
response.Headers[HttpRequestHeader.ContentEncoding].ToLower() == "gzip") zip = true;
using (var reader = zip ?
#if NETFX_CORE
new StreamReader(new GZipStream(stream, CompressionMode.Decompress))
#else
new StreamReader(new zlib.ZOutputStream(stream), false)
#endif
: new StreamReader(stream)) {
var str = reader.ReadToEnd();
result = new ObjectResult(str);
}
}
}
catch (WebException e) {...}