Develop Reference

How to fix download issues for some users using WebClient DownloadFileAsync?

I created a Windows Form Application in C# that users of my game modification can use to download updates automatically.
The 'Launcher', as I call it, uses WebClient to download the updates. But the first release of the mod is very big (2,7 GB zipped). The launcher works perfect for me and most users, but for some users the extraction of the zip file logs an error where the file is corrupted and not readable.
I searched already on stack, and it is possible that the file might be corrupted or truncated due to bad internet connection. But how do I build in a method that fix that problem?
//Start downloading file
using (WebClient webClient = new WebClient())
{
webClient.DownloadFileCompleted += new
AsyncCompletedEventHandler(Client_DownloadFileCompleted);
webClient.DownloadProgressChanged += new
DownloadProgressChangedEventHandler(Client_DownloadProgressChanged);
webClient.DownloadFileAsync(new Uri("http://www.dagovaxgames.com/api/downloads/+ patch.path), downloadPath);
}
private void Client_DownloadFileCompleted(object sender, AsyncCompletedEventArgs e)
{
//install the update
InstallUpdate();
}
private void InstallUpdate()
{
var file = currentPatchPath;
//get the size of the zip file
fileInfo = new FileInfo(file);
_fileSize = fileInfo.Length;
installBackgroundWorker = new BackgroundWorker();
installBackgroundWorker.DoWork += ExtractFile_DoWork;
installBackgroundWorker.ProgressChanged += ExtractFile_ProgressChanged;
installBackgroundWorker.RunWorkerCompleted += ExtractFile_RunWorkerCompleted;
installBackgroundWorker.WorkerReportsProgress = true;
installBackgroundWorker.RunWorkerAsync();
}
EDIT, just showing install code so that you know I am using a backgroundworker to extract the zip.

Common approach here is to download large file in small chunks and put them together on client after completion. Using this approach you can: 1. run few downloads in parallel and 2. in case of problem with network you don't have to download entire file again, just download incomplete chunks.

I faced a similar issue many years ago and created a subclass of WebClient that uses the DownloadProgressChanged event and a timer to abort downloads that get hung up and cancels the download smoother than the underlying internet transport layer does. The code also supports a callback to notify the calling code of progress. I found that sufficient to smoothly handle occasional hiccups downloading 1GB-ish files.
The idea of breaking your download into multiple pieces also has merit. You could leverage a library such as 7-Zip to both create the chunks and piece them back together (many compression libraries have that feature; I'm personally most familiar with 7-Zip).
Here's the code I wrote. Feel free to use and/or modify in any way that's helpful to you.
public class JWebClient : WebClient, IDisposable
{
public int Timeout { get; set; }
public int TimeUntilFirstByte { get; set; }
public int TimeBetweenProgressChanges { get; set; }
public long PreviousBytesReceived { get; private set; }
public long BytesNotNotified { get; private set; }
public string Error { get; private set; }
public bool HasError { get { return Error != null; } }
private bool firstByteReceived = false;
private bool success = true;
private bool cancelDueToError = false;
private EventWaitHandle asyncWait = new ManualResetEvent(false);
private Timer abortTimer = null;
private bool isDisposed = false;
const long ONE_MB = 1024 * 1024;
public delegate void PerMbHandler(long totalMb);
public delegate void TaggedPerMbHandler(string tag, long totalMb);
public event PerMbHandler NotifyMegabyteIncrement;
public event TaggedPerMbHandler NotifyTaggedMegabyteIncrement;
public JWebClient(int timeout = 60000, int timeUntilFirstByte = 30000, int timeBetweenProgressChanges = 15000)
{
this.Timeout = timeout;
this.TimeUntilFirstByte = timeUntilFirstByte;
this.TimeBetweenProgressChanges = timeBetweenProgressChanges;
this.DownloadFileCompleted += new System.ComponentModel.AsyncCompletedEventHandler(MyWebClient_DownloadFileCompleted);
this.DownloadProgressChanged += new DownloadProgressChangedEventHandler(MyWebClient_DownloadProgressChanged);
abortTimer = new Timer(AbortDownload, null, TimeUntilFirstByte, System.Threading.Timeout.Infinite);
}
protected void OnNotifyMegabyteIncrement(long totalMb)
{
NotifyMegabyteIncrement?.Invoke(totalMb);
}
protected void OnNotifyTaggedMegabyteIncrement(string tag, long totalMb)
{
NotifyTaggedMegabyteIncrement?.Invoke(tag, totalMb);
}
void AbortDownload(object state)
{
cancelDueToError = true;
this.CancelAsync();
success = false;
Error = firstByteReceived ? "Download aborted due to >" + TimeBetweenProgressChanges + "ms between progress change updates." : "No data was received in " + TimeUntilFirstByte + "ms";
asyncWait.Set();
}
private object disposeLock = new object();
void MyWebClient_DownloadProgressChanged(object sender, DownloadProgressChangedEventArgs e)
{
if (cancelDueToError || isDisposed) return;
long additionalBytesReceived = e.BytesReceived - PreviousBytesReceived;
PreviousBytesReceived = e.BytesReceived;
BytesNotNotified += additionalBytesReceived;
if (BytesNotNotified > ONE_MB)
{
OnNotifyMegabyteIncrement(e.BytesReceived);
OnNotifyTaggedMegabyteIncrement(Tag, e.BytesReceived);
BytesNotNotified = 0;
}
firstByteReceived = true;
try
{
lock (disposeLock)
{
if (!isDisposed) abortTimer.Change(TimeBetweenProgressChanges, System.Threading.Timeout.Infinite);
}
}
catch (ObjectDisposedException) { } // Some strange timing issue causes this to throw now and then
}
public string Tag { get; private set; }
public bool DownloadFileWithEvents(string url, string outputPath, string tag = null)
{
Tag = tag;
asyncWait.Reset();
Uri uri = new Uri(url);
this.DownloadFileAsync(uri, outputPath);
asyncWait.WaitOne();
return success;
}
void MyWebClient_DownloadFileCompleted(object sender, System.ComponentModel.AsyncCompletedEventArgs e)
{
if (e.Error != null) success = false;
if (cancelDueToError || isDisposed) return;
asyncWait.Set();
}
protected override WebRequest GetWebRequest(Uri address)
{
var result = base.GetWebRequest(address);
result.Timeout = this.Timeout;
return result;
}
void IDisposable.Dispose()
{
lock (disposeLock)
{
isDisposed = true;
if (asyncWait != null) asyncWait.Dispose();
if (abortTimer != null) abortTimer.Dispose();
base.Dispose();
}
}
}

I fixed it using a combination of a BackgroundWorker and downloading in small chunks (following up mtkachenko's solution). It also checks if the length of the downloaded file is the same as the one on the server. Using that, it can continue the download where the connection was interupted.
private void DownloadPatch(Patch patch){
//using background worker now!
downloadBackgroundWorker = new BackgroundWorker();
downloadBackgroundWorker.DoWork += (sender, e) => DownloadFile_DoWork(sender, e, patch);
downloadBackgroundWorker.ProgressChanged += DownloadFile_ProgressChanged;
downloadBackgroundWorker.RunWorkerCompleted += DownloadFile_RunWorkerCompleted;
downloadBackgroundWorker.WorkerReportsProgress = true;
downloadBackgroundWorker.RunWorkerAsync();
}
private void DownloadFile_DoWork(object sender, DoWorkEventArgs e, Patch patch)
{
string startupPath = Application.StartupPath;
string downloadPath = Path.Combine(Application.StartupPath, patch.path);
string path = ("http://www.dagovaxgames.com/api/downloads/" + patch.path);
long iFileSize = 0;
int iBufferSize = 1024;
iBufferSize *= 1000;
long iExistLen = 0;
System.IO.FileStream saveFileStream;
// Check if file exists. If true, then check amount of bytes
if (System.IO.File.Exists(downloadPath))
{
System.IO.FileInfo fINfo =
new System.IO.FileInfo(downloadPath);
iExistLen = fINfo.Length;
}
if (iExistLen > 0)
saveFileStream = new System.IO.FileStream(downloadPath,
System.IO.FileMode.Append, System.IO.FileAccess.Write,
System.IO.FileShare.ReadWrite);
else
saveFileStream = new System.IO.FileStream(downloadPath,
System.IO.FileMode.Create, System.IO.FileAccess.Write,
System.IO.FileShare.ReadWrite);
System.Net.HttpWebRequest hwRq;
System.Net.HttpWebResponse hwRes;
hwRq = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(path);
hwRq.AddRange((int)iExistLen);
System.IO.Stream smRespStream;
hwRes = (System.Net.HttpWebResponse)hwRq.GetResponse();
smRespStream = hwRes.GetResponseStream();
iFileSize = hwRes.ContentLength;
//using webclient to receive file size
WebClient webClient = new WebClient();
webClient.OpenRead(path);
long totalSizeBytes = Convert.ToInt64(webClient.ResponseHeaders["Content-Length"]);
int iByteSize;
byte[] downBuffer = new byte[iBufferSize];
while ((iByteSize = smRespStream.Read(downBuffer, 0, downBuffer.Length)) > 0)
{
if (stopDownloadWorker == true)
{
autoDownloadReset.WaitOne();
}
saveFileStream.Write(downBuffer, 0, iByteSize);
long downloadedBytes = new System.IO.FileInfo(downloadPath).Length;
// Report progress, hint: sender is your worker
int percentage = Convert.ToInt32(100.0 / totalSizeBytes * downloadedBytes);
(sender as BackgroundWorker).ReportProgress(percentage, null);
}
}
As you can see, I report the progress, so that I have a working progress bar as well.
private void DownloadFile_ProgressChanged(object sender, ProgressChangedEventArgs e)
{
statusTextLabel.Text = "Downloading updates for version " + currentDownloadingPatch + " (" + e.ProgressPercentage + "%)";
progressBar.Value = e.ProgressPercentage;
}

BackgroundWorker's progress bar is not working

The BackgroundWorker's progressbar is not updated while doing some tasks. What I would like to reach is progressbar moving while iterating through each file in DirectoryInfo. Suppose we have 20 files of ".sql" while first file completed it should be 5%, 10% and etc.
Here is my code.
private void CSV_Click(object sender, RoutedEventArgs e)
{
try
{
btnExtract.IsEnabled = false;
workerextract.RunWorkerAsync();
}
catch (Exception ex)
{
System.Windows.MessageBox.Show(ex.Message);
}
}
private void workerextract_DoWork(object sender, System.ComponentModel.DoWorkEventArgs e)
{
try
{
this.Dispatcher.Invoke(() =>
{
DirectoryInfo di = new DirectoryInfo(txtQueryfolder.Text);
files = di.GetFiles("*.sql").Count();
currentfile = 0;
foreach (FileInfo fi in di.GetFiles("*.sql"))
{
// Open the text file using a stream reader.
using (StreamReader sr = new StreamReader(fi.FullName))
{
// Read the stream to a string, and write the string to the console.
string line = sr.ReadToEnd();
//System.Windows.MessageBox.Show(line);
ExtractToCSV(line, System.IO.Path.GetFileNameWithoutExtension(fi.Name));
currentfile++;
}
int percentage = (currentfile + 1) * 100 / files;
workerextract.ReportProgress(percentage);
}
});
}
catch(Exception ex)
{
System.Windows.MessageBox.Show(ex.Message);
}
}
private void workerextract_ProgressChanged(object sender, System.ComponentModel.ProgressChangedEventArgs e)
{
progressBarExtract.Value = e.ProgressPercentage;
}
private void workerextract_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e)
{
btnExtract.IsEnabled = true;
System.Windows.MessageBox.Show("CSV Data extraction finished!");
}
I found that
private void workerextract_ProgressChanged(object sender,
System.ComponentModel.ProgressChangedEventArgs e)
is called once at the end when 100%.
Also,
private void workerextract_RunWorkerCompleted(object sender,
RunWorkerCompletedEventArgs e)
never called as I do not see Message Box at the end.
So, I think I am doing something wrong here, could you please direct me on right way?

The problem was in wrapping whole DoWork inside Dispatcher.Invoke.
I need to wrap only those code where it is interacting with UI.
So I changed the code appropriately and it works.
private void workerextract_DoWork(object sender, System.ComponentModel.DoWorkEventArgs e)
{
try
{
this.Dispatcher.Invoke(() =>
{
di = new DirectoryInfo(txtQueryfolder.Text);
});
files = di.GetFiles("*.sql").Count();
currentfile = 0;
foreach (FileInfo fi in di.GetFiles("*.sql"))
{
// Open the text file using a stream reader.
using (StreamReader sr = new StreamReader(fi.FullName))
{
// Read the stream to a string, and write the string to the console.
string line = sr.ReadToEnd();
this.Dispatcher.Invoke(() =>
{
//System.Windows.MessageBox.Show(line);
ExtractToCSV(line, System.IO.Path.GetFileNameWithoutExtension(fi.Name));
});
currentfile++;
}
int percentage = (currentfile + 1) * 100 / files;
workerextract.ReportProgress(percentage);
}
}
catch(Exception ex)
{
System.Windows.MessageBox.Show(ex.Message);
}
}
Thanks to all for showing the direction.

Using this.Dispatcher.Invoke in the BackgroundWorker's DoWork event you are executing the whole operation in the UI thread; which is what BackgroundWorker born to avoid to do.
Also, you get an error unwrapping your code from the dispatcher because you are accessing an UI object, which is txtQueryfolder.
Just use:
private void workerextract_DoWork(object sender, System.ComponentModel.DoWorkEventArgs e)
{
string queryFolder = e.Argument.ToString();
try
{
DirectoryInfo di = new DirectoryInfo(queryFolder);
files = di.GetFiles("*.sql").Count();
currentfile = 0;
foreach (FileInfo fi in di.GetFiles("*.sql"))
{
// Open the text file using a stream reader.
using (StreamReader sr = new StreamReader(fi.FullName))
{
// Read the stream to a string, and write the string to the console.
string line = sr.ReadToEnd();
//System.Windows.MessageBox.Show(line);
// ExtractToCSV shouldn't access to a UI object.
ExtractToCSV(line, System.IO.Path.GetFileNameWithoutExtension(fi.Name));
currentfile++;
}
int percentage = (currentfile + 1) * 100 / files;
workerextract.ReportProgress(percentage);
}
}
catch (Exception ex)
{
// Don't use MessageBox in a thread different from the UI one. Just set the result (e.Result) and get that in the RunWorkerCompleted event.
// System.Windows.MessageBox.Show(ex.Message);
}
}
When you call the RunWorkerAsync method just add the parameter like below:
workerextrac.RunWorkerAsync(txtQueryfolder.Text);

FileSystem Watcher- Checking if copied file is an Image

I'm monitoring a Folder for File creation(Copied) event using FileSystem Watcher. I only want the program to process image Files.
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Filter = "*.*";
watcher.Created += new FileSystemEventHandler(watcher_FileCreated);
watcher.Path = path;
So I try to create a Bitmap and avoid the file if an exception is thrown
private static void watcher_FileCreated(object sender, FileSystemEventArgs e)
{
try
{
using (Bitmap test = new Bitmap(Bitmap.FromFile(e.FullPath)))
{
mytoprocesslist.add(e.FullPath);
}
//do my processing with image
Console.WriteLine(e.FullPath);
}
catch (Exception error)
{
Console.WriteLine("File Error");
}
}
This throws Out of Memory exception even when a valid image file is copied, which I think happens because the event was raised before the file was copied completely. How can I get over this? I only want to add the valid image files to a to do list and I will process these images one by one later.

A bit cleaner solution than a Try-Catch might be this one.
Im using this code without any exceptions raised.
private static bool IsImage(string path) {
try {
var result = false;
using (var stream = new FileStream(path, FileMode.Open)) {
stream.Seek(0, SeekOrigin.Begin);
var jpg = new List<string> { "FF", "D8" };
var bmp = new List<string> { "42", "4D" };
var gif = new List<string> { "47", "49", "46" };
var png = new List<string> { "89", "50", "4E", "47", "0D", "0A", "1A", "0A" };
var imgTypes = new List<List<string>> { jpg, bmp, gif, png };
var bytesIterated = new List<string>();
for (var i = 0; i < 8; i++) {
var bit = stream.ReadByte().ToString("X2");
bytesIterated.Add(bit);
var isImage = imgTypes.Any(img => !img.Except(bytesIterated).Any());
if (isImage) {
result = true;
break;
}
}
}
return result;
} catch (UnauthorizedAccessException) {
return false;
}
}
Usage of code
foreach (var file in Directory.EnumerateFiles(#"pathToFlowersFolder"))
{
Console.WriteLine($"File: {file} Result:{IsImage(file)}");
}
Edit
After playing around i got an IO-Exception (File already in use)
After reading this i'd offer you the following solution:
private void button1_Click(object sender, EventArgs e)
{
var watcher = new FileSystemWatcher();
watcher.Created += new FileSystemEventHandler(fileSystemWatcher1_Changed);
watcher.Path = #"c:\temp";
watcher.NotifyFilter = NotifyFilters.FileName | NotifyFilters.Size;
watcher.EnableRaisingEvents = true;
}
private void fileSystemWatcher1_Changed(object sender, System.IO.FileSystemEventArgs e)
{
Thread.Sleep(100); // <- give the Creator some time. Increase value for greate pause
if (IsImage(e.FullPath))
{
Console.WriteLine("success----------->" + e.FullPath);
}
}
Note
This piece of code properly works on my machine. My HDD is an SSD, so you might need to increase the thread-sleeping time. It properly works for all images (jpg, bmp, gif, png) up to a size of 7 Mb (im quite sure and greater).
If this code doesnt works for you, please post the exception rather than uploading your code.

For the first requirement: "I only want the program to process image files"
private static void fileSystemWatcher1_Changed(object sender, FileSystemEventArgs e)
{
string strFileExt = getFileExt(e.FullPath);
// filter file types
if (Regex.IsMatch(strFileExt, #"\.png|\.jpg", RegexOptions.IgnoreCase))
{
//here Process the image file
}
}
For the second requirement: "Out of Memory Exception"
Here what happens is, when the file is created (only file name and some attributes) the system is calling the created event. Then the file changed event is also called
So you have to do the processing in the changed event. Also to prevent duplicate calling you have to add a filter to your watcher.
The following is the complete code.
private void fileSystemWatcher1_Changed(object sender, System.IO.FileSystemEventArgs e)
{
FileInfo fileInfo = new FileInfo(e.FullPath);
string strFileExt = fileInfo.Extension;
// filter file types
if (Regex.IsMatch(strFileExt, #"\.png|\.jpg", RegexOptions.IgnoreCase))
{
//here Process the image file
try
{
using (Bitmap test = new Bitmap(Bitmap.FromFile(e.FullPath)))
{
//Do your code here.
}
}
catch (Exception error)
{
Console.WriteLine("File Error");
}
}
}
private void Form1_Load(object sender, EventArgs e)
{
fileSystemWatcher1.Path = #"C:\Users\Christlin\Desktop\res";
//To Prevent duplicated calling of changed event
fileSystemWatcher1.NotifyFilter = NotifyFilters.FileName | NotifyFilters.Size;
}

Dotnetzip, Event Handler not Fired after Save

I have a small problem, which might me a stupid mistake on my side.
Here is my code to create a zipfile when needed and the method to add a file to the archive.
Adding a file works without problem but for some reason the Event is not fired after saving.
I set a breakpoint at zipFile_SaveProgress, the event is not fired.
class CoverArchive
{
private ZipFile zipFile;
private String coverArchivePath;
public CoverArchive()
{
coverArchivePath = "Archive\\Covers";
if (!File.Exists("Archive\\Covers"))
{
CreateZipFile();
}
using (zipFile = ZipFile.Read(coverArchivePath))
{
//zipFile.AddProgress += zipFile_AddProgress;
//zipFile.ExtractProgress += zipFile_ExtractProgress;
//zipFile.ZipError += zipFile_ZipError;
zipFile.SaveProgress += zipFile_SaveProgress;
}
}
private void CreateZipFile()
{
zipFile = new ZipFile();
zipFile.CompressionLevel = Ionic.Zlib.CompressionLevel.BestCompression;
zipFile.UseUnicodeAsNecessary = true;
if (!Directory.Exists("Archive"))
Directory.CreateDirectory("Archive");
zipFile.Save(coverArchivePath);
}
public void AddCover(List<String> directories, String coverName, Stream fileStream)
{
try
{
using (zipFile = ZipFile.Read(coverArchivePath))
{
String filePath = createPath(directories, coverName);
ZipEntry entry = zipFile.AddEntry(filePath, fileStream);
zipFile.Save();
}
}
catch (Exception ex)
{
Console.WriteLine("Error adding File" + ex);
}
}
private void zipFile_SaveProgress(Object sender, SaveProgressEventArgs e)
{
if (e.EventType == ZipProgressEventType.Saving_Completed)
Console.WriteLine("Save Done");
}
}
Thanks a lot in advance :)

You're adding the event handling after calling Save().
Therefore, when it raised the SaveProgress event, there weren't any handlers.

FileSystemWatcher triggers for filestream open

I have a filesystemwatcher that will trigger an event when a file is modified. I want to read from that file once the lock has been removed. At the moment I am just trying to open the file once the event is triggered, when A large file is being copied the file lock stays on for a while after the events have been sent, preventing the file from being opened for read access.
Any suggestions?

This one's actually a bit of a doozie, unless the problem space has changed significantly since I last had to deal with it.
The easiest way is to simply try to open the file, catch the resulting IOException, and if the file is locked, add it to a queue to be checked later. You can't just try to process every file that comes in because there are all kinds of cases where multiple events will be generated for the same file, so setting up a retry loop on every single received event can turn into a disaster, fast. You need to queue them up instead and check the queue at a regular interval.
Here is a basic class template that should help you out with this problem:
public class FileMonitor : IDisposable
{
private const int PollInterval = 5000;
private FileSystemWatcher watcher;
private HashSet<string> filesToProcess = new HashSet<string>();
private Timer fileTimer; // System.Threading.Timer
public FileMonitor(string path)
{
if (path == null)
throw new ArgumentNullException("path");
watcher = new FileSystemWatcher();
watcher.Path = path;
watcher.NotifyFilter = NotifyFilters.FileName;
watcher.Created += new FileSystemEventHandler(FileCreated);
watcher.EnableRaisingEvents = true;
fileTimer = new Timer(new TimerCallback(ProcessFilesTimer),
null, PollInterval, Timeout.Infinite);
}
public void Dispose()
{
fileTimer.Dispose();
watcher.Dispose();
}
private void FileCreated(object source, FileSystemEventArgs e)
{
lock (filesToProcess)
{
filesToProcess.Add(e.FullPath);
}
}
private void ProcessFile(FileStream fs)
{
// Your code here...
}
private void ProcessFilesTimer(object state)
{
string[] currentFiles;
lock (filesToProcess)
{
currentFiles = filesToProcess.ToArray();
}
foreach (string fileName in currentFiles)
{
TryProcessFile(fileName);
}
fileTimer.Change(PollInterval, Timeout.Infinite);
}
private void TryProcessFile(string fileName)
{
FileStream fs = null;
try
{
FileInfo fi = new FileInfo(fileName);
fs = fi.OpenRead();
}
catch (IOException)
{
// Possibly log this error
return;
}
using (fs)
{
ProcessFile(fs);
}
lock (filesToProcess)
{
filesToProcess.Remove(fileName);
}
}
}
(Note - I'm recalling this from memory here so it might not be perfect - let me know if it's buggy.)