Develop Reference

What is the best way to check a url is valid every second (or less), with Task await or ContinueWith in a C# Window Forms Application (.NET)

I'm new to C# .Net and Visual Studio 2022 - What I'm trying to achieve is to have a timer running every second to check that a website url is valid/is up. If the url IS reachable and the current WebView2 is not showing that website, then it should navigate to it. If it's already showing that website, it should do nothing else. If it was showing that website, but now it's no longer valid, the WebView should navigate to my custom error page. If whilst on the custom error page the website becomes available again, it should (re)load the website.
In my particular scenario I'm making a webView load localhost (127.0.0.1) for now. I want to continuously check the website is ip, and if it goes down, show custom error, if it comes back, show the website.
Not sure I'm explaining that very well. From the research I have done, I believe I need Task and also await using async method.
Here's my current timer and checkurl code as well as navigtionstarted and navigationcompeted:
private void webView_NavigationStarting(object sender, CoreWebView2NavigationStartingEventArgs e)
{
timerCheckRSLCDURL.Enabled = false;
}
private void webView_NavigationCompleted(object sender, Microsoft.Web.WebView2.Core.CoreWebView2NavigationCompletedEventArgs e)
{
if (e.IsSuccess)
{
Debug.WriteLine("JT:IsSuccess");
((Microsoft.Web.WebView2.WinForms.WebView2) sender).ExecuteScriptAsync("document.querySelector('body').style.overflow='hidden'");
}
else if (!e.IsSuccess)
{
Debug.WriteLine("JT:IsNOTSuccess");
webView.DefaultBackgroundColor = Color.Blue;
//webView.CoreWebView2.NavigateToString(Program.htmlString);
}
timerCheckRSLCDURL.Enabled = true;
}
private void timerCheckRSLCDURL_Tick(object sender, EventArgs e)
{
Debug.WriteLine("Timer Fired! Timer.Enabled = " + timerCheckRSLCDURL.Enabled);
CheckURL(Properties.Settings.Default.URL, Properties.Settings.Default.Port);
}
private async void CheckURL(string url, decimal port)
{
timerCheckRSLCDURL = false;
Program.isWebSiteUp = false;
string webViewURL = BuildURL();
Debug.WriteLine("Checking URL: " + webViewURL);
try
{
var request = WebRequest.Create(webViewURL);
request.Method = "HEAD";
var response = (HttpWebResponse) await Task.Factory.FromAsync < WebResponse > (request.BeginGetResponse, request.EndGetResponse, null);
if (response.StatusCode == HttpStatusCode.OK)
{
Program.isWebSiteUp = true;
}
}
catch (System.Net.WebException exception)
{
Debug.WriteLine("WebException: " + exception.Message);
if (exception.Message.Contains("(401) Unauthorized"))
{
Program.isWebSiteUp = false;
}
else
{
Program.isWebSiteUp = false;
} // This little block is unfinished atm as it doesn't really affect me right now
}
catch (Exception exception)
{
Debug.WriteLine("Exception: " + exception.Message);
Program.isWebSiteUp = false;
}
if (Program.isWebSiteUp == true && webView.Source.ToString().Equals("about:blank"))
{
Debug.WriteLine("JT:1");
Debug.WriteLine("isWebSiteUp = true, webView.Source = about:blank");
webView.CoreWebView2.Navigate(webViewURL);
}
else if (Program.isWebSiteUp == true && !webView.Source.ToString().Equals(webViewURL))
{
Debug.WriteLine("JT:2");
Debug.WriteLine("isWebSiteUp = true\nwebView.Source = " + webView.Source.ToString() + "\nwebViewURL = " + webViewURL + "\nWebView Source == webViewURL: " + webView.Source.ToString().Equals(webViewURL) + "\n");
webView.CoreWebView2.Navigate(webViewURL);
}
else if (Program.isWebSiteUp == false && !webView.Source.ToString().Equals("about:blank"))
{
Debug.WriteLine("JT:3");
Debug.WriteLine("This SHOULD be reloading the BSOD page!");
webView.CoreWebView2.NavigateToString(Program.htmlString);
}
}
private string BuildURL()
{
string webViewURL;
string stringURL = Properties.Settings.Default.URL;
string stringPort = Properties.Settings.Default.Port.ToString();
string stringURLPORT = $ "{stringURL}:{stringPort}";
if (stringPort.Equals("80"))
{
webViewURL = stringURL;
}
else
{
webViewURL = stringURLPORT;
}
if (!webViewURL.EndsWith("/"))
{
webViewURL += "/";
}
//For now, the URL will always be at root, so don't need to worry about accidentally
//making an invalid url like http://example.com/subfolder/:port
//although potentially will need to address this at a later stage
Debug.WriteLine("BuildURL returns: " + webViewURL);
return webViewURL;
}
So the timer is fired every 1000ms (1 second) because I need to actively check the URL is still alive. I think the way I'm controlling the timer is wrong - and I imagine there's a better way of doing it, but what I want to do is this...
Check website URL every 1 second
To avoid repeating the same async task, I'm trying to disable the timer so it does not fire a second time whilst the async checkurl is running
Once the async/await task of checking the url has finished, the timer should be re-enabled to continue monitoring is the website url is still up
If the website is down, it should show my custom error page (referred to as BSOD) which is some super basic html loaded from resources and 'stored' in Program.htmlString
if the the website is down, and the webview is already showing the BSOD, the webview should do nothing. The timer should continue to monitor the URL.
if the website is up and the webview is showing the BSOD, then it should navigate to the checked url that is up. If the website is up, and the webview is already showing the website, then the webview should do nothing. The timer should continue to monitor the URL.
From other research, I'm aware I shouldn't be using private async void - eg shouldn't be using it as a void. But I've not yet figured out / understood the correct way to do this
In the Immediate Window, it appears that webView_NavigationCompleted is being fired twice (or sometimes even a few times) instantly as the immediate window output will show JT:IsSuccess or JT:IsNOTSuccess a few times repeated in quick succession. Is that normal? I'm assuming something isn't correct there.
The main problem appears to be due to the timer being only 1 second. If I change the timer to fire every 30 seconds for example, it seems to work ok, but when it's every second (I may even need it less than that at some point) it's not really working as expected. Sometimes the BSOD doesn't load at all for example, as well as the webView_NavigationCompleted being fire multiple times in quick succession etc.
Could someone pretty please help me make this code better and correct.
I've searched countless websites etc and whilst there is some good info, some of it seems overwhelming / too technical so to speak. I had to lookup what "antecedent" meant earlier as it's a completely new word to me! :facepalm:
Many thanks inadvance

This answer will focus on the Task timer loop to answer the specific part of your question "check a url is valid every second". There are lots of answers about how to perform the actual Ping (like How do you check if a website is online in C#) and here's the Microsoft documentation for Ping if you choose to go that route.
Since it's not uncommon to set a timeout value of 120 seconds for a ping request, it calls into question whether it would have any value to do this on a steady tick of one second. My suggestion is that it would make more sense to:
Make a background thread
Perform a synchronous ping (wait for the result) on the background thread.
Marshal the ping result onto the UI thread to perform the other tasks you have laid out.
Synchronously wait a Task.Delay on the background thread before performing the next ping.
Here is how I personally go about doing that in my own production code:
void execPing()
{
Task.Run(() =>
{
while (!DisposePing.IsCancellationRequested)
{
var pingSender = new Ping();
var pingOptions = new PingOptions
{
DontFragment = true,
};
// https://learn.microsoft.com/en-us/dotnet/api/system.net.networkinformation.ping?view=net-6.0#examples
// Create a buffer of 32 bytes of data to be transmitted.
string data = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
byte[] buffer = Encoding.ASCII.GetBytes(data);
int timeout = 120;
try
{
// https://stackoverflow.com/a/25654227/5438626
if (Uri.TryCreate(textBoxUri.Text, UriKind.Absolute, out Uri? uri)
&& (uri.Scheme == Uri.UriSchemeHttp ||
uri.Scheme == Uri.UriSchemeHttps))
{
PingReply reply = pingSender.Send(
uri.Host,
timeout, buffer,
pingOptions);
switch (reply.Status)
{
case IPStatus.Success:
Invoke(() => onPingSuccess());
break;
default:
Invoke(() => onPingFailed(reply.Status));
break;
}
}
else
{
Invoke(() => labelStatus.Text =
$"{DateTime.Now}: Invalid URI: try 'http://");
}
}
catch (Exception ex)
{
// https://stackoverflow.com/a/60827505/5438626
if (ex.InnerException == null)
{
Invoke(() => labelStatus.Text = ex.Message);
}
else
{
Invoke(() => labelStatus.Text = ex.InnerException.Message);
}
}
Task.Delay(1000).Wait();
}
});
}
What works for me is initializing it when the main window handle is created:
protected override void OnHandleCreated(EventArgs e)
{
base.OnHandleCreated(e);
if (!(DesignMode || _isHandleInitialized))
{
_isHandleInitialized = true;
execPing();
}
}
bool _isHandleInitialized = false;
Where:
private void onPingSuccess()
{
labelStatus.Text = $"{DateTime.Now}: {IPStatus.Success}";
// Up to you what you do here
}
private void onPingFailed(IPStatus status)
{
labelStatus.Text = $"{DateTime.Now}: {status}";
// Up to you what you do here
}
public CancellationTokenSource DisposePing { get; } = new CancellationTokenSource();
Example 404:

limit # of web service requests simultaneously

I have an Excel Add-In written in C#, .NET 4.5. It will send many web service requests to a web server to get data. E.g. it sends 30,000 requests to web service server. When data of a request comes back, the addin will plot the data in Excel.
Originally I did all the requests asynchronously, but sometime I will get OutOfMemoryException
So I changed, sent the requests one by one, but it is too slow, takes long time to finish all requests.
I wonder if there is a way that I can do 100 requests at a time asynchronously, once the data of all the 100 requests come back and plot in Excel, then send the next 100 requests.
Thanks
Edit
On my addin, there is a ribbon button "Refresh", when it is clicked, refresh process starts.
On main UI thread, ribbon/button is clicked, it will call web service BuildMetaData,
once it is returned back, in its callback MetaDataCompleteCallback, another web service call is sent
Once it is returned back, in its callback DataRequestJobFinished, it will call plot to plot data on Excel. see below
RefreshBtn_Click()
{
if (cells == null) return;
Range firstOccurence = null;
firstOccurence = cells.Find(functionPattern, null,
null, null,
XlSearchOrder.xlByRows,
XlSearchDirection.xlNext,
null, null, null);
DataRequest request = null;
_reportObj = null;
Range currentOccurence = null;
while (!Helper.RefreshCancelled)
{
if(firstOccurence == null ||IsRangeEqual(firstOccurence, currentOccurence)) break;
found = true;
currentOccurence = cells.FindNext(currentOccurence ?? firstOccurence);
try
{
var excelFormulaCell = new ExcelFormulaCell(currentOccurence);
if (excelFormulaCell.HasValidFormulaCell)
{
request = new DataRequest(_unityContainer, XLApp, excelFormulaCell);
request.IsRefreshClicked = true;
request.Workbook = Workbook;
request.Worksheets = Worksheets;
_reportObj = new ReportBuilder(_unityContainer, XLApp, request, index, false);
_reportObj.ParseParameters();
_reportObj.GenerateReport();
//this is necessary b/c error message is wrapped in valid object DataResponse
//if (!string.IsNullOrEmpty(_reportObj.ErrorMessage)) //Clear previous error message
{
ErrorMessage = _reportObj.ErrorMessage;
Errors.Add(ErrorMessage);
AddCommentToCell(_reportObj);
Errors.Remove(ErrorMessage);
}
}
}
catch (Exception ex)
{
ErrorMessage = ex.Message;
Errors.Add(ErrorMessage);
_reportObj.ErrorMessage = ErrorMessage;
AddCommentToCell(_reportObj);
Errors.Remove(ErrorMessage);
Helper.LogError(ex);
}
}
}
on Class to GenerateReport
public void GenerateReport()
{
Request.ParseFunction();
Request.MetacompleteCallBack = MetaDataCompleteCallback;
Request.BuildMetaData();
}
public void MetaDataCompleteCallback(int id)
{
try
{
if (Request.IsRequestCancelled)
{
Request.FormulaCell.Dispose();
return;
}
ErrorMessage = Request.ErrorMessage;
if (string.IsNullOrEmpty(Request.ErrorMessage))
{
_queryJob = new DataQueryJob(UnityContainer, Request.BuildQueryString(), DataRequestJobFinished, Request);
}
else
{
ModifyCommentOnFormulaCellPublishRefreshEvent();
}
}
catch (Exception ex)
{
ErrorMessage = ex.Message;
ModifyCommentOnFormulaCellPublishRefreshEvent();
}
finally
{
Request.MetacompleteCallBack = null;
}
}
public void DataRequestJobFinished(DataRequestResponse response)
{
Dispatcher.Invoke(new Action<DataRequestResponse>(DataRequestJobFinishedUI), response);
}
public void DataRequestJobFinished(DataRequestResponse response)
{
try
{
if (Request.IsRequestCancelled)
{
return;
}
if (response.status != Status.COMPLETE)
{
ErrorMessage = ManipulateStatusMsg(response);
}
else // COMPLETE
{
var tmpReq = Request as DataRequest;
if (tmpReq == null) return;
new VerticalTemplate(tmpReq, response).Plot();
}
}
catch (Exception e)
{
ErrorMessage = e.Message;
Helper.LogError(e);
}
finally
{
//if (token != null)
// this.UnityContainer.Resolve<IEventAggregator>().GetEvent<DataQueryJobComplete>().Unsubscribe(token);
ModifyCommentOnFormulaCellPublishRefreshEvent();
Request.FormulaCell.Dispose();
}
}
on plot class
public void Plot()
{
...
attributeRange.Value2 = headerArray;
DataRange.Value2 = ....
DataRange.NumberFormat = ...
}

OutOfMemoryException is not about the too many requests sent simultaneously. It is about freeing your resources right way. In my practice there are two main problems when you are getting such exception:
Wrong working with immutable structures or System.String class
Not disposing your disposable resources, especially graphic objects and WCF requests.
In case of reporting, for my opinion, you got a second one type of a problem. DataRequest and DataRequestResponse are good point to start the investigation for the such objects.
If this doesn't help, try to use the Tasks library with async/await pattern, you can find good examples here:
// Signature specifies Task<TResult>
async Task<int> TaskOfTResult_MethodAsync()
{
int hours;
// . . .
// Return statement specifies an integer result.
return hours;
}
// Calls to TaskOfTResult_MethodAsync
Task<int> returnedTaskTResult = TaskOfTResult_MethodAsync();
int intResult = await returnedTaskTResult;
// or, in a single statement
int intResult = await TaskOfTResult_MethodAsync();
// Signature specifies Task
async Task Task_MethodAsync()
{
// . . .
// The method has no return statement.
}
// Calls to Task_MethodAsync
Task returnedTask = Task_MethodAsync();
await returnedTask;
// or, in a single statement
await Task_MethodAsync();
In your code I see a while loop, in which you can store your Task[] of size of 100, for which you can use the WaitAll method, and the problem should be solved. Sorry, but your code is huge enough, and I can't provide you a more straight example.

I'm having a lot of trouble parsing your code to figure out is being iterated for your request but the basic template for batching asynchronously is going to be something like this:
static const int batchSize = 100;
public async Task<IEnumerable<Results>> GetDataInBatches(IEnumerable<RequestParameters> parameters) {
if(!parameters.Any())
return Enumerable.Empty<Result>();
var batchResults = await Task.WhenAll(parameters.Take(batchSize).Select(doQuery));
return batchResults.Concat(await GetDataInBatches(parameters.Skip(batchSize));
}
where doQuery is something with the signature
Task<Results> async doQuery(RequestParameters parameters) {
//.. however you do the query
}
I wouldn't use this for a million requests since its recursive, but your case should would generate a callstack only 300 deep so you'll be fine.
Note that this also assumes that your data request stuff is done asynchronously and returns a Task. Most libraries have been updated to do this (look for methods with the Async suffix). If it doesn't expose that api you might want to create a separate question for how to specifically get your library to play nice with the TPL.

Serially process ConcurrentQueue and limit to one message processor. Correct pattern?

I'm building a multithreaded app in .net.
I have a thread that listens to a connection (abstract, serial, tcp...).
When it receives a new message, it adds it to via AddMessage. Which then call startSpool. startSpool checks to see if the spool is already running and if it is, returns, otherwise, starts it in a new thread. The reason for this is, the messages HAVE to be processed serially, FIFO.
So, my questions are...
Am I going about this the right way?
Are there better, faster, cheaper patterns out there?
My apologies if there is a typo in my code, I was having problems copying and pasting.
ConcurrentQueue<IMyMessage > messages = new ConcurrentQueue<IMyMessage>();
const int maxSpoolInstances = 1;
object lcurrentSpoolInstances;
int currentSpoolInstances = 0;
Thread spoolThread;
public void AddMessage(IMyMessage message)
{
this.messages.Add(message);
this.startSpool();
}
private void startSpool()
{
bool run = false;
lock (lcurrentSpoolInstances)
{
if (currentSpoolInstances <= maxSpoolInstances)
{
this.currentSpoolInstances++;
run = true;
}
else
{
return;
}
}
if (run)
{
this.spoolThread = new Thread(new ThreadStart(spool));
this.spoolThread.Start();
}
}
private void spool()
{
Message.ITimingMessage message;
while (this.messages.Count > 0)
{
// TODO: Is this below line necessary or does the TryDequeue cover this?
message = null;
this.messages.TryDequeue(out message);
if (message != null)
{
// My long running thing that does something with this message.
}
}
lock (lcurrentSpoolInstances)
{
this.currentSpoolInstances--;
}
}

This would be easier using BlockingCollection<T> instead of ConcurrentQueue<T>.
Something like this should work:
class MessageProcessor : IDisposable
{
BlockingCollection<IMyMessage> messages = new BlockingCollection<IMyMessage>();
public MessageProcessor()
{
// Move this to constructor to prevent race condition in existing code (you could start multiple threads...
Task.Factory.StartNew(this.spool, TaskCreationOptions.LongRunning);
}
public void AddMessage(IMyMessage message)
{
this.messages.Add(message);
}
private void Spool()
{
foreach(IMyMessage message in this.messages.GetConsumingEnumerable())
{
// long running thing that does something with this message.
}
}
public void FinishProcessing()
{
// This will tell the spooling you're done adding, so it shuts down
this.messages.CompleteAdding();
}
void IDisposable.Dispose()
{
this.FinishProcessing();
}
}
Edit: If you wanted to support multiple consumers, you could handle that via a separate constructor. I'd refactor this to:
public MessageProcessor(int numberOfConsumers = 1)
{
for (int i=0;i<numberOfConsumers;++i)
StartConsumer();
}
private void StartConsumer()
{
// Move this to constructor to prevent race condition in existing code (you could start multiple threads...
Task.Factory.StartNew(this.spool, TaskCreationOptions.LongRunning);
}
This would allow you to start any number of consumers. Note that this breaks the rule of having it be strictly FIFO - the processing will potentially process "numberOfConsumer" elements in blocks with this change.
Multiple producers are already supported. The above is thread safe, so any number of threads can call Add(message) in parallel, with no changes.

I think that Reed's answer is the best way to go, but for the sake of academics, here is an example using the concurrent queue -- you had some races in the code that you posted (depending upon how you handle incrementing currnetSpoolInstances)
The changes I made (below) were:
Switched to a Task instead of a Thread (uses thread pool instead of incurring the cost of creating a new thread)
added the code to increment/decrement your spool instance count
changed the "if currentSpoolInstances <= max ... to just < to avoid having one too many workers (probably just a typo)
changed the way that empty queues were handled to avoid a race: I think you had a race, where your while loop could have tested false, (you thread begins to exit), but at that moment, a new item is added (so your spool thread is exiting, but your spool count > 0, so your queue stalls).
private ConcurrentQueue<IMyMessage> messages = new ConcurrentQueue<IMyMessage>();
const int maxSpoolInstances = 1;
object lcurrentSpoolInstances = new object();
int currentSpoolInstances = 0;
public void AddMessage(IMyMessage message)
{
this.messages.Enqueue(message);
this.startSpool();
}
private void startSpool()
{
lock (lcurrentSpoolInstances)
{
if (currentSpoolInstances < maxSpoolInstances)
{
this.currentSpoolInstances++;
Task.Factory.StartNew(spool, TaskCreationOptions.LongRunning);
}
}
}
private void spool()
{
IMyMessage message;
while (true)
{
// you do not need to null message because it is an "out" parameter, had it been a "ref" parameter, you would want to null it.
if(this.messages.TryDequeue(out message))
{
// My long running thing that does something with this message.
}
else
{
lock (lcurrentSpoolInstances)
{
if (this.messages.IsEmpty)
{
this.currentSpoolInstances--;
return;
}
}
}
}
}

Check 'Pipelines pattern': http://msdn.microsoft.com/en-us/library/ff963548.aspx
Use BlockingCollection for the 'buffers'.
Each Processor (e.g. ReadStrings, CorrectCase, ..), should run in a Task.
HTH..

handling database connection issues

I have some code that needs to retry whenever it cannot make a database connection, for example 20 times for each login. The website has dozens of users, what would be the best way to handle this? I was thinking of using a counter but as multiple users will be logged on at the same time perhaps some threading would be required, which I am unfamiliar with.
EDIT :
I get the error 'method name expected' on the line where I am instantiating the Thread object. Here is my code :
private static List<Field.Info> FromDatabase(this Int32 _campId)
{
List<Field.Info> lstFields = new List<Field.Info>();
Field.List.Response response = new Field.List.Ticket
{
campId = _campId
}.Commit();
if (response.status == Field.List.Status.success)
{
lstFields = response.fields;
lock (campIdLock)
{
loadedCampIds.Add(_campId);
}
}
if (response.status == Field.List.Status.retry)
{
Thread th1 = new Thread(new ThreadStart(FromDatabase(_campId)));
FromDatabase(_campId);
}
return lstFields;
}

My approach would be an event, which increments the counter and calls an asynchronous methode if counter <= 20.
Then i would block the current thread and wait for the connection result.
To thread-safe increment your counter use
Interlocked.Increment(ref int counter);

Running multiple delegates asynchronously and waiting for response in C#

I have a table of urls which I need to loop through, download each file, update the table and return the results. I want to run up to 10 downloads at a time so am thinking about using delegates as follows:
DataTable photos;
bool scanning = false,
complete = false;
int rowCount = 0;
public delegate int downloadFileDelegate();
public void page_load(){
photos = Database.getData...
downloadFileDelegate d = downloadFile;
d.BeginInvoke(downloadFileComplete, d);
d.BeginInvoke(downloadFileComplete, d);
d.BeginInvoke(downloadFileComplete, d);
d.BeginInvoke(downloadFileComplete, d);
d.BeginInvoke(downloadFileComplete, d);
while(!complete){}
//handle results...
}
int downloadFile(){
while(scanning){} scanning = true;
DataRow r;
for (int ii = 0; ii < rowCount; ii++) {
r = photos.Rows[ii];
if ((string)r["status"] == "ready"){
r["status"] = "running";
scanning = false; return ii;
}
if ((string)r["status"] == "running"){
scanning = false; return -2;
}
}
scanning = false; return -1;
}
void downloadFileComplete(IAsyncResult ar){
if (ar == null){ return; }
downloadFileDelegate d = (downloadFileDelegate)ar.AsyncState;
int i = d.EndInvoke(ar);
if (i == -1){ complete = true; return; }
//download file...
//update row
DataRow r = photos.Rows[i];
r["status"] = "complete";
//invoke delegate again
d.BeginInvoke(downloadFileComplete, d);
}
However when I run this it takes the same amount of time to run 5 as it does 1. I was expecting it to take 5 times faster.
Any ideas?

You look like you're trying to use lockless syncronization (using while(scanning) to check a boolean that's set at the start of the function and reset at the end), but all this succeeds in doing is only running one retrieval at a time.
Is there a reason that you don't want them to run concurrently? This seems like the entire point of your exercise. I can't see a reason for this, so I'd lose the scanning flag (and associated logic) entirely.
If you're going to take this approach, your boolean flags need to be declared as volatile (otherwise their reads could be cached and you could wait endlessly)
Your data update operations (updating the value in the DataRow) must take place on the UI thread. You'll have to wrap these operations up in a Control.Invoke or Control.BeginInvoke call, otherwise you'll be interacting with the control across thread boundaries.
BeginInvoke returns an AsyncWaitHandle. Use this for your logic that will take place when the operations are all complete. Something like this
-
WaitHandle[] handles = new WaitHandle[]
{
d.BeginInvoke(...),
d.BeginInvoke(...),
d.BeginInvoke(...),
d.BeginInvoke(...),
d.BeginInvoke(...)
}
WaitHandle.WaitAll(handles);
This will cause the calling thread to block until all of the operations are complete.

It will take the same amount of time if you are limited by network bandwidth. If you are downloading all 10 files from the same site then it won't be any quicker. Multi threading is useful when you either want the User Interface to respond or you have something processor intensive and multiple cores

WaitHandle[] handles = new WaitHandle[5];
handles[0] = d.BeginInvoke(downloadFileComplete, d).AsyncWaitHandle;
handles[1] = d.BeginInvoke(downloadFileComplete, d).AsyncWaitHandle;
handles[2] = d.BeginInvoke(downloadFileComplete, d).AsyncWaitHandle;
handles[3] = d.BeginInvoke(downloadFileComplete, d).AsyncWaitHandle;
handles[4] = d.BeginInvoke(downloadFileComplete, d).AsyncWaitHandle;
WaitHandle.WaitAll(handles);

There are numerous things in this implementation that are not safe for concurrent use.
But the one that is likely causing the effect you describe is the fact that downloadFile() has a while() loop that examines the scanning variable. This variable is shared by all instances of the running delegate. This while loop prevents the delegates from running concurrently.
This "scanning loop" is not a proper threading construct - it is possible for two threads to both read the variable and both set it at the same time. You should really use a Semaphore or lock() statement to to protected the DataTable from concurrent access. Since each thread is spending most of it's time waiting for scanning to be false, the downloadFile method cannot run concurrently.
You should reconsider how you've structured this code so that downloading the data and updating the structures in your application are not in contention.

Something like this would work better I think.
public class PhotoDownload
{
public ManualResetEvent Complete { get; private set; }
public Object RequireData { get; private set; }
public Object Result { get; private set; }
}
public void DownloadPhotos()
{
var photos = new List<PhotoDownload>();
// build photo download list
foreach (var photo in photos)
{
ThreadPool.QueueUserWorkItem(DownloadPhoto, photo);
}
// wait for the downloads to complete
foreach (var photo in photos)
{
photo.Complete.WaitOne();
}
// make sure everything happened correctly
}
public void DownloadPhoto(object state)
{
var photo = state as PhotoDownload;
try
{
// do not access fields in this class
// everything should be inside the photo object
}
finally
{
photo.Complete.Set();
}
}