My requirement is to transfer a zip file of size 400MB or more; The following code works for at least 40MB; But for more I would have to change byte[] bytes = new byte[50000000]; to byte[] bytes = new byte[400000000]; and maxRequestLength to maxRequestLength="409600";
The problem is byte[] bytes = new byte[100000000]; returns an error regarding insufficient space. So how can I transfer large files using WebClient??
WebClient client = new WebClient();
client.AllowWriteStreamBuffering = true;
UriBuilder ub = new UriBuilder("http://localhost:57596/UploadImages.ashx");
ub.Query = "ImageName=" + "DataSet" + DataSetId + ".zip";
client.OpenWriteCompleted += (InputStream, eArguments) =>
{
try
{
using (Stream output = eArguments.Result)
{
output.Write(ImagesAux, 0, (int)ImagesAux.Length);
//numeroimagem++;
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
//throw;
}
};
client.OpenWriteAsync(ub.Uri);
in UploadImages.ashx
public void ProcessRequest(HttpContext context)
{
//context.Response.ContentType = "text/plain";
//context.Response.Write("Hello World");
string ImageName = context.Request.QueryString["ImageName"];
string UploadPath = context.Server.MapPath("~/ServerImages/");
using (FileStream stream = File.Create(UploadPath + ImageName))
{
byte[] bytes = new byte[50000000]; //
int bytesToRead = 0;
while ((bytesToRead =
context.Request.InputStream.Read(bytes, 0, bytes.Length)) != 0)
{
stream.Write(bytes, 0, bytesToRead);
stream.Close();
}
}
}
in Web.config
<httpRuntime targetFramework="4.5" maxRequestLength="40960"/>
You should never load everything in memory then write all back to disk, but instead you should load pieces and write them to disk while you are reading them.
When you've done reading you close the stream you are writing to.
Otherwise as soon as you reach sizes as GB you can get an OutOfMemory really quick.
So I would change the writing bytes to disk from this:
using (FileStream stream = File.Create(UploadPath + ImageName))
{
byte[] bytes = new byte[50000000]; //
int bytesToRead = 0;
while ((bytesToRead = context.Request.InputStream.Read(bytes, 0, bytes.Length)) != 0)
{
stream.Write(bytes, 0, bytesToRead);
stream.Close();
}
}
to this:
using (FileStream stream = File.Create(UploadPath + ImageName))
{
byte[] bytes = new byte[1024];
long totalBytes = context.Request.InputStream.Length;
long bytesRead = 0;
int bytesToRead = bytes.Length;
if (totalBytes - bytesRead < bytes.Length)
bytesToRead = (int)(totalBytes - bytesRead);
bytes = new byte[bytesToRead];
while ((bytesToRead = context.Request.InputStream.Read(bytes, bytesRead, bytes.Length)) != 0)
{
stream.Write(bytes, bytesRead, bytes.Length);
bytesRead += bytes.Length;
if (totalBytes - bytesRead < bytes.Length)
bytesToRead = (int)(totalBytes - bytesRead);
bytes = new byte[bytesToRead];
}
stream.Close();
}
1024 would be the buffer size.
Related
I need help to improve this code but I do not understand where I made a mistake. The file is downloaded but in a corrupted format. I am using cookies also which is a required part of this method.
/*Downlod task */
public static async Task<int> CreateDownloadTask(string urlToDownload,string sDestinationPath, string cookiedstr)
{
int BufferSize = 4096;
int receivedBytes = 0;
int totalBytes = 0;
WebClient client = new WebClient();
client.Headers.Add(HttpRequestHeader.Cookie, cookiedstr);
using (var stream = await client.OpenReadTaskAsync(urlToDownload))
{
using (MemoryStream ms = new MemoryStream())
{
int read = 0;
totalBytes = Int32.Parse(client.ResponseHeaders[HttpResponseHeader.ContentLength]);
var buffer = new byte[BufferSize];
while ((read = await stream.ReadAsync(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
FileStream file = new FileStream(sDestinationPath, FileMode.Append, FileAccess.Write);
ms.Position = 0;
ms.WriteTo(file);
file.Flush();
file.Close();
buffer = new byte[BufferSize];
receivedBytes += read;
Console.WriteLine(receivedBytes + " " + totalBytes);
// DownloadBytesProgress args = new DownloadBytesProgress(urlToDownload, receivedBytes, totalBytes);
}
ms.Close();
}
return receivedBytes;
}
}
I try to Zip some .pdf files in C#. My code works fine but when the size of one of the pdfs is big, it is going to overwrite that pdf on the rest of pdfs. I am not sure what is happening. I tried to increase the size of buffer or zip file but still same issue. Do you have any suggestion?
This is my code:
public void ProcessZipRequest(string strQueueID, string strBatchID, string strFtpPath)
{
int intReportCnt = 0;
string strZipFileName = "Order-" + strBatchID + "-" + strQueueID + "-" + DateTime.Now.ToString("MM-dd-yyyy-HH-mm") + ".zip";
strZipFileName = SafeFileName(strZipFileName);
//MemoryStream ms = new MemoryStream();
FileStream ms = new FileStream(#"c:\surf\nikoo.zip", FileMode.Create);
ZipOutputStream oZipStream = new ZipOutputStream(ms); // create zip stream
oZipStream.SetLevel(9); // maximum compression
intReportCnt += 1;
string strRptFilename=string.Empty;
MemoryStream outputStream = new MemoryStream();
if (strQueueID != null)
{
String[] filenames = Directory.GetFiles(#"C:\uploaded");
// setting Report name to path given for Report name
foreach (String filename in filenames)
{
strRptFilename = filename.Substring(filename.LastIndexOf("\\") + 1);
FileStream fs = File.OpenRead(#"C:\uploaded\" + strRptFilename);
int bufferSize = 2048;
int readCount;
byte[] buffer = new byte[bufferSize];
readCount = fs.Read(buffer, 0, bufferSize);
while (readCount>0)
{
outputStream.Write(buffer, 0, readCount);
readCount = fs.Read(buffer, 0, bufferSize);
}
fs.Close();
outputStream.Position = 0;
ZipFile(ref outputStream, strRptFilename, ref oZipStream);
}
}
outputStream.Close();
oZipStream.Finish();
oZipStream.Flush();
oZipStream.IsStreamOwner = false; // False stops the Close also Closing the underlying stream.
oZipStream.Close(); // Must finish the ZipOutputStream before using outputMemStream.
ms.Close();
}
And this is Zipfile Method:
public void ZipFile(ref MemoryStream msFile, string strFilename, ref ZipOutputStream oZipStream)
{
ZipEntry oZipEntry = new ZipEntry(strFilename);
oZipEntry.DateTime = DateTime.Now;
oZipEntry.Size = msFile.Length;
oZipStream.PutNextEntry(oZipEntry);
StreamUtils.Copy(msFile, oZipStream, new byte[4096]);
oZipStream.CloseEntry();
}
I found the issue. I have to create a new MemoyStream in for loop and close it at the end of the loop.
foreach (String filename in filenames)
{
strRptFilename = filename.Substring(filename.LastIndexOf("\\") + 1);
outputStream = new MemoryStream();
FileStream fs = File.OpenRead(#"C:\uploaded\" + strRptFilename);
int bufferSize = 2048;
int readCount;
byte[] buffer = new byte[bufferSize];
readCount = fs.Read(buffer, 0, bufferSize);
while (readCount>0)
{
outputStream.Write(buffer, 0, readCount);
readCount = fs.Read(buffer, 0, bufferSize);
}
fs.Close();
outputStream.Position = 0;
ZipFile(ref outputStream, strRptFilename, ref oZipStream);
fs.Close();
outputStream.Close();
}
I am developing an application which downloads videos from a given URL. The problem is that I do not receive the entire file content, then cannot play the file. For example, trying to download a video of size ~2.23 MB, gives me only ~2.11 MB. When I use the URL in a browser it shows me a dialog to save the video and the file is downloaded successfully.
I have tried using WebClient class and it works, but I want to download the file in chunks in order to be able to report the status(percentage completed). Here is the code I use:
private void backgroundWorker_DoWork(object sender, DoWorkEventArgs e)
{
int bufferSize = 1024 * 300;
string filePath = saveFileDialog.FileName;
if (File.Exists(filePath))
File.Delete(filePath);
int totalBytes = 0;
HttpWebRequest webRequest = (HttpWebRequest)HttpWebRequest.Create(DownloadUrl);
long contentLength = webRequest.GetResponse().ContentLength;
using (Stream webStream = webRequest.GetResponse().GetResponseStream())
using (StreamReader reader = new StreamReader(webStream))
using (BinaryWriter fileWriter = new BinaryWriter(File.Create(filePath)))
{
do
{
char[] buffer = new char[bufferSize];
bytesRead = reader.ReadBlock(buffer, 0, bufferSize); // also tried with Read(buffer, 0, bufferSize);
totalBytes += bytesRead;
Console.WriteLine("Bytes read: " + bytesRead + " Total Bytes: " + totalBytes + " Content length: " + contentLength);
if (bytesRead > 0)
fileWriter.Write(buffer, 0, bytesRead);
} while (!reader.EndOfStream);
}
}
I have also tried to read until bytesRead = 0, but it's the same result.
Is there something that I am missing?
I would recommend that you use DownloadFileAsync instead. This provides you with two events that makes it much easier to track the progress of your download.
DownloadProgressChangedEventHandler
DownloadFileCompleted
A very simple implementation would look something like this.
WebClient client = new WebClient();
client.DownloadProgressChanged +=
new DownloadProgressChangedEventHandler(DownloadProgressCallback);
client.DownloadFileAsync(DownloadUrl, filePath);
Then have a function that updates your progress bar.
private void DownloadProgressCallback(object sender,
DownloadProgressChangedEventArgs e)
{
myProgressBar.Value = e.ProgressPercentage;
}
You also have access to data like e.BytesReceived and e.TotalBytesToReceive.
Edit:
The main change I had to do was to change your buffer from char[] to byte[], and then use a Stream instead of a StreamReader.
We also check the end of file by checking to see if there are any bytes to write to our hard-drive. If there are none left to write, we know that we are done.
private static void download()
{
int bufferSize = 1024 * 300;
string filePath = "Test.exe";
if (File.Exists(filePath))
File.Delete(filePath);
int totalBytes = 0;
HttpWebRequest webRequest =
(HttpWebRequest)
HttpWebRequest.Create(
#"http://www.rarlab.com/rar/wrar420.exe");
long contentLength = webRequest.GetResponse().ContentLength;
Console.WriteLine(totalBytes);
using (WebResponse webResponse = webRequest.GetResponse())
using (Stream reader = webResponse.GetResponseStream())
using (BinaryWriter fileWriter = new BinaryWriter(File.Create(filePath)))
{
int bytesRead = 0;
byte[] buffer = new byte[bufferSize];
do
{
bytesRead = reader.Read(buffer, 0, buffer.Length);
totalBytes += bytesRead;
fileWriter.Write(buffer, 0, bytesRead);
Console.WriteLine("BytesRead: " + bytesRead + " -- TotalBytes: " + totalBytes);
} while (bytesRead > 0);
}
}
Try
private void backgroundWorker_DoWork(object sender, DoWorkEventArgs e)
{
int bufferSize = 1024 * 300;
string filePath = saveFileDialog.FileName;
if (File.Exists(filePath))
File.Delete(filePath);
int totalBytes = 0;
HttpWebRequest webRequest = (HttpWebRequest)HttpWebRequest.Create(DownloadUrl);
long contentLength = webRequest.GetResponse().ContentLength;
using (Stream webStream = webRequest.GetResponse().GetResponseStream())
using (StreamReader reader = new StreamReader(webStream))
using (BinaryWriter fileWriter = new BinaryWriter(File.Create(filePath)))
{
do
{
char[] buffer = new char[bufferSize];
bytesRead = reader.ReadBlock(buffer, 0, bufferSize); // also tried with Read(buffer, 0, bufferSize);
totalBytes += bytesRead;
Console.WriteLine("Bytes read: " + bytesRead + " Total Bytes: " + totalBytes + " Content length: " + contentLength);
if (bytesRead > 0)
fileWriter.Write(buffer, 0, bytesRead);
} while (!reader.EndOfStream);
fileWriter.flush();
}
}
Hi, that was my bad, I just added the fileWriter.flush(); outside the loop
I have several .gz files, and I want to decompress them one by one.
I have writen a simple code using GzipStream in C#, but got failed. I wonder a correct and useful method to achieve what I want. Thanks a lot.
private string Extrgz(string infile)
{
string dir = Path.GetDirectoryName(infile);
string decompressionFileName = dir + Path.GetFileNameWithoutExtension(infile) + "_decompression.bin";
using (GZipStream instream = new GZipStream(File.OpenRead(infile), CompressionMode.Compress))// ArgumentException...
{
using (FileStream outputStream = new FileStream(decompressionFileName, FileMode.Append, FileAccess.Write))
{
int bufferSize = 8192, bytesRead = 0;
byte[] buffer = new byte[bufferSize];
while ((bytesRead = instream.Read(buffer, 0, bufferSize)) > 0)
{
outputStream.Write(buffer, 0, bytesRead);
}
}
}
return decompressionFileName;
}
You need to decompress but you set CompressionMode.Compress, replace it with CompressionMode.Decompress.
Example here.
Here:
public static void DeCompressFile(string CompressedFile, string DeCompressedFile)
{
byte[] buffer = new byte[1024 * 1024];
using (System.IO.FileStream fstrmCompressedFile = System.IO.File.OpenRead(CompressedFile)) // fi.OpenRead())
{
using (System.IO.FileStream fstrmDecompressedFile = System.IO.File.Create(DeCompressedFile))
{
using (System.IO.Compression.GZipStream strmUncompress = new System.IO.Compression.GZipStream(fstrmCompressedFile,
System.IO.Compression.CompressionMode.Decompress))
{
int numRead = strmUncompress.Read(buffer, 0, buffer.Length);
while (numRead != 0)
{
fstrmDecompressedFile.Write(buffer, 0, numRead);
fstrmDecompressedFile.Flush();
numRead = strmUncompress.Read(buffer, 0, buffer.Length);
} // Whend
//int numRead = 0;
//while ((numRead = strmUncompress.Read(buffer, 0, buffer.Length)) != 0)
//{
// fstrmDecompressedFile.Write(buffer, 0, numRead);
// fstrmDecompressedFile.Flush();
//} // Whend
strmUncompress.Close();
} // End Using System.IO.Compression.GZipStream strmUncompress
fstrmDecompressedFile.Flush();
fstrmDecompressedFile.Close();
} // End Using System.IO.FileStream fstrmCompressedFile
fstrmCompressedFile.Close();
} // End Using System.IO.FileStream fstrmCompressedFile
} // End Sub DeCompressFile
// http://www.dotnetperls.com/decompress
public static byte[] Decompress(byte[] gzip)
{
byte[] baRetVal = null;
using (System.IO.MemoryStream ByteStream = new System.IO.MemoryStream(gzip))
{
// Create a GZIP stream with decompression mode.
// ... Then create a buffer and write into while reading from the GZIP stream.
using (System.IO.Compression.GZipStream stream = new System.IO.Compression.GZipStream(ByteStream
, System.IO.Compression.CompressionMode.Decompress))
{
const int size = 4096;
byte[] buffer = new byte[size];
using (System.IO.MemoryStream memory = new System.IO.MemoryStream())
{
int count = 0;
count = stream.Read(buffer, 0, size);
while (count > 0)
{
memory.Write(buffer, 0, count);
memory.Flush();
count = stream.Read(buffer, 0, size);
}
baRetVal = memory.ToArray();
memory.Close();
}
stream.Close();
} // End Using System.IO.Compression.GZipStream stream
ByteStream.Close();
} // End Using System.IO.MemoryStream ByteStream
return baRetVal;
} // End Sub Decompress
I'm trying to securely transfer files between 2 devices, so I'm using an SslStream attached to a TcpClient. Documents and text files come across just fine, but image files don't show up correctly. The following is the server code:
listener = new TcpListener(IPAddress.Any, 1337);
listener.Start();
while (true)
{
TcpClient client = listener.AcceptTcpClient();
SslStream sslStream = new SslStream(client.GetStream(), false, new RemoteCertificateValidationCallback(CertificateValidationCallback), new LocalCertificateSelectionCallback(CertificateSelectionCallback));
var certificate = Connection.GetClientCertificate(((IPEndPoint)client.Client.RemoteEndPoint).Address.ToString());
try
{
sslStream.AuthenticateAsServer(certificate, true, SslProtocols.Default, true);
sslStream.ReadTimeout = 5000;
sslStream.WriteTimeout = 5000;
var messageData = ReadMessage(sslStream);
var mode = messageData[0];
var tokenBytes = messageData.Splice(1, 16);
var fileNameBytes = messageData.Splice(17, 128);
var fileBytes = messageData.Splice(146);
var fileName = Encoding.ASCII.GetString(fileNameBytes).TrimEnd('\0');
using (var tempFile = new FileStream(fileName, FileMode.OpenOrCreate, FileAccess.Write))
{
tempFile.Write(fileBytes, 0, fileBytes.Length);
tempFile.Flush();
}
if (mode == 0)
tempFiles.Add(fileName);
Process.Start(fileName);
}
catch (AuthenticationException e)
{
MessageBox.Show("The other side failed to authenticate.");
}
finally
{
sslStream.Close();
client.Close();
}
}
And ReadMessage is defined as follows:
private static byte[] ReadMessage(SslStream sslStream)
{
byte[] buffer = new byte[2048];
MemoryStream stream = new MemoryStream();
int bytes = -1;
while (bytes != 0)
{
bytes = sslStream.Read(buffer, 0, buffer.Length);
stream.Write(buffer, 0, bytes);
}
return stream.ToArray();
}
And then the client code is this:
TcpClient client = new TcpClient();
client.Connect(new IPEndPoint(IPAddress.Parse(ip), 1337));
SslStream sslStream = new SslStream(client.GetStream(), false, new RemoteCertificateValidationCallback(CertificateValidationCallback), new LocalCertificateSelectionCallback(CertificateSelectionCallback));
var certificate = Connection.GetClientCertificate(ip);
try
{
sslStream.AuthenticateAsClient(ip, new X509CertificateCollection() { certificate }, SslProtocols.Default, false);
sslStream.ReadTimeout = 5000;
sslStream.WriteTimeout = 5000;
sslStream.Write(data);
}
catch (AuthenticationException e)
{
MessageBox.Show("The other side failed to authenticate.");
}
finally
{
sslStream.Close();
client.Close();
}
And the code that calls into it just does:
var fileBytes = File.ReadAllBytes(file);
var tokenBytes = Encoding.UTF8.GetBytes(token);
var fileNameBytes = Encoding.UTF8.GetBytes(Path.GetFileName(file));
var buffer = new byte[145 + fileBytes.Length];
buffer[0] = 1;
for (int i = 0; i < 16; i++)
{
buffer[i + 1] = tokenBytes[i];
}
for (int i = 0; i < fileNameBytes.Length; i++)
{
buffer[i + 17] = fileNameBytes[i];
}
for (int i = 0; i < fileBytes.Length; i++)
{
buffer[i + 145] = fileBytes[i];
}
SocketConnection.Send(ip, buffer);
Is there anything inherently wrong with what I'm doing, or do I need to do something different for images?
EDIT: I have changed it to reflect the current code, and also, after doing a dump of the raw bytes on both ends, it looks like for some reason the bytes are getting rearranged when they come over the wire. Is there any way to ensure that the bytes come across in the original order?
In ReadMessage you're writing bytes.Length bytes to the stream, regardless of the number of bytes that were actually read. Try:
private static byte[] ReadMessage(SslStream sslStream)
{
byte[] buffer = new byte[2048];
MemoryStream stream = new MemoryStream();
int bytes = -1;
while (bytes != 0)
{
bytes = sslStream.Read(buffer, 0, buffer.Length);
// Use "bytes" instead of "buffer.Length" here
stream.Write(buffer, 0, bytes);
}
return stream.ToArray();
}
Based on your follow-up, you're also taking the file data from the wrong point in the buffer, and so you're losing the first byte of the file.
Your code should be:
var fileBytes = messageData.Splice(145); // File data starts at 145, not 146
Is this possibly a conflict between endianness? If the bytes from the server are ABCDEF and the client is seeing the image bytes as BADCFE then that's the issue.
I haven't worked with image files, but when I read a short or an int instead of a String from the bytes coming in over the wire, I do something like this:
int intFromStream = IPAddress.NetworkToHostOrder(BitConverter.ToInt32(byteArrayWithLength4, 0));