Develop Reference

C# UI becomes unresponsive during processing in background thread

I am testing the validity of a large list of proxy servers concurrently. During this testing, many exceptions are being raised and caught. Although I am doing the testing in a background thread, my UI becomes unresponsive unless I use a SemaphoreSlim object to control the concurrency.
I know this is a self imposed bottle neck, and when scaling with an even larger list of proxies to test, I was hoping there might be a better way to solve the problem.
private void ValidateProxiesButton_Click(object sender, EventArgs e)
{
new Thread(async () =>
{
Thread.CurrentThread.IsBackground = true;
await ValidateProxiesAsync(proxies, judges, tests, 10);
}).Start();
}
public async Task ValidateProxiesAsync(IEnumerable<Proxy> proxies, IEnumerable<ProxyJudge> judges, IEnumerable<ProxyTest> tests = null, int maxConcurrency = 20)
{
if (proxies.Count() == 0)
{
throw new ArgumentException("Proxy list empty.");
}
foreach (var proxy in proxies)
{
proxy.Status = ProxyStatus.Queued;
}
//Get external IP to check if proxy is anonymous.
var publicIp = await WebUtility.GetPublicIP();
foreach (var judge in judges)
{
judge.Invalidation = publicIp;
}
await ValidateTestsAsync(judges.ToList<IProxyTest>());
var validJudges = judges.ToList<IProxyTest>().GetValidTests();
if (validJudges.Count == 0)
{
throw new ArgumentException("No valid judges found.");
}
if (tests != null)
{
await ValidateTestsAsync(tests.ToList<IProxyTest>());
}
var semaphore = new SemaphoreSlim(maxConcurrency);
var tasks = new List<Task>();
foreach (var proxy in proxies)
{
tasks.Add(Task.Run(async () =>
{
await semaphore.WaitAsync();
proxy.Status = ProxyStatus.Testing;
var isValid = await proxy.TestValidityAsync((IProxyTest)validJudges.GetRandomItem());
proxy.Status = isValid ? ProxyStatus.Valid : ProxyStatus.Invalid;
semaphore.Release();
}));
}
await Task.WhenAll(tasks);
}
Inside proxy.TestValidityAsync method
public async Task<bool> TestValidityAsync(IProxyTest test, int timeoutSeconds = 30)
{
try
{
var req = WebRequest.Create(test.URL);
req.Proxy = new WebProxy(this.ToString());
var respBody = await WebUtility.GetResponseStringAsync(req).TimeoutAfter(new TimeSpan(0, 0, timeoutSeconds));
if (respBody.Contains(test.Validation))
{
return true;
}
else
{
return false;
}
}
catch (Exception)
{
return false;
}
}

So I found a working solution, it is to add the TPL Dataflow NuGet package to my project and then use the TransformBlock class. When I do this, my UI stays very responsive even if I am processing tons of concurrent requests that often throw exceptions. The code below is proof of concept, I will update it when I translate it to work with my project.
Source: Throttling asynchronous tasks
private async void button1_Click(object sender, EventArgs e)
{
var downloader = new TransformBlock<string, WebResponse>(
url => Download(url),
new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 200 }
);
var buffer = new BufferBlock<WebResponse>();
downloader.LinkTo(buffer);
var urls = new List<string>();
for (int i = 0; i < 100000; i++)
{
urls.Add($"http://example{i}.com");
}
foreach (var url in urls)
downloader.Post(url);
//or await downloader.SendAsync(url);
downloader.Complete();
await downloader.Completion;
IList<WebResponse> responses;
if (buffer.TryReceiveAll(out responses))
{
//process responses
}
}
private WebResponse Download(string url)
{
WebResponse resp = null;
try
{
var req = WebRequest.Create(url);
resp = req.GetResponse();
}
catch (Exception)
{
}
return resp;
}
}

C# Download Speed Asynchronously

I'm trying to get the current user's network download speed. After hitting a dead end with NetworkInterfaces and all I tried a solution I found online. I edited it a bit and it works great but it's not asynchronous.
public static void GetDownloadSpeed(this Label lbl)
{
double[] speeds = new double[5];
for (int i = 0; i < 5; i++)
{
int fileSize = 407; //Size of File in KB.
WebClient client = new WebClient();
DateTime startTime = DateTime.Now;
if (!Directory.Exists($"{CurrentDir}/tmp/speedtest"))
Directory.CreateDirectory($"{CurrentDir}/tmp/speedtest");
client.DownloadFile(new Uri("https://ajax.googleapis.com/ajax/libs/threejs/r69/three.min.js"), "/tmp/speedtest/three.min.js");
DateTime endTime = DateTime.Now;
speeds[i] = Math.Round((fileSize / (endTime - startTime).TotalSeconds));
}
lbl.Text = string.Format("{0}KB/s", speeds.Average());
}
That function is called within a timer at an interval of 2 minutes.
MyLbl.GetDownloadSpeed()
I've tried using WebClient.DownloadFileAsync but that just shows the unlimited symbol.My next try would be to use HttpClient but before I go on does anyone have a recommended way of getting the current users download speed asynchronously (without lagging the main GUI thread)?

As it was suggested you could make an async version of GetDownloadSpeed():
async void GetDownloadSpeedAsync(this Label lbl, Uri address, int numberOfTests)
{
string directoryName = #"C:\Work\Test\speedTest";
string fileName = "tmp.dat";
if (!Directory.Exists(directoryName))
Directory.CreateDirectory(directoryName);
Stopwatch timer = new Stopwatch();
timer.Start();
for (int i = 0; i < numberOfTests; ++i)
{
using (WebClient client = new WebClient())
{
await client.DownloadFileTaskAsync(address, Path.Combine(directoryName, fileName), CancellationToken.None);
}
}
lbl.Text == Convert.ToString(timer.Elapsed.TotalSeconds / numberOfTests);
}
WebClient class being relatively old does not have awaitable DownloadFileAsync().
EDITED
As it was correctly pointed out WebClient in fact has a task-based async method DownloadFileTaskAsync(), which i advise to use. The code below can still help addressing the case when async method returning Task is not provided.
We can fix it with the help of TaskCompletionSource<T>:
public static class WebClientExtensions
{
public static Task DownloadFileAwaitableAsync(this WebClient instance, Uri address,
string fileName, CancellationToken cancellationToken)
{
TaskCompletionSource<object> tcs = new TaskCompletionSource<object>();
// Subscribe for completion event
instance.DownloadFileCompleted += instance_DownloadFileCompleted;
// Setup cancellation
var cancellationRegistration = cancellationToken.CanBeCanceled ? (IDisposable)cancellationToken.Register(() => { instance.CancelAsync(); }) : null;
// Initiate asyncronous download
instance.DownloadFileAsync(address, fileName, Tuple.Create(tcs, cancellationRegistration));
return tcs.Task;
}
static void instance_DownloadFileCompleted(object sender, System.ComponentModel.AsyncCompletedEventArgs e)
{
((WebClient)sender).DownloadDataCompleted -= instance_DownloadFileCompleted;
var data = (Tuple<TaskCompletionSource<object>, IDisposable>)e.UserState;
if (data.Item2 != null) data.Item2.Dispose();
var tcs = data.Item1;
if (e.Cancelled)
{
tcs.TrySetCanceled();
}
else if (e.Error != null)
{
tcs.TrySetException(e.Error);
}
else
{
tcs.TrySetResult(null);
}
}
}

Try `await Task.Run(()=> { //your code });
Edit: #JustDevInc I still think you should use DownloadAsync. Task.Run(delegate) creates a new thread and you might want to avoid that. If you want, post some of your old code so we can try to fix it.
Edit: The first solution turned out to be the only one of the two working. DownloadFileAsync doesn't return task, so can't it awaited.

backgroundWorker with foreach

The program does not follow the logic, I need it to do the request according to the order of the items of the listbox
private void backgroundWorker1_DoWork(object sender, DoWorkEventArgs e)
{
foreach (var listBoxItem in listBox1.Items)
{
for (int j = 0; j < listBox1.Items.Count; j++)
{
string lista = listBox1.Items[j].ToString();
string[] split = lista.Split(';');
num.Text = split[0];
v1.Text = split[1];
v2.Text = split[2];
c.Text = split[3];
WebClient client = new WebClient();
client.Proxy = null;
client.DownloadDataCompleted += new DownloadDataCompletedEventHandler(this.asyncWebRequest_DownloadDataCompleted);
client.DownloadDataAsync(new Uri("http://127.0.0.1/sv/" + num.Text));
j++;
}
}
}
private void asyncWebRequest_DownloadDataCompleted(object sender, DownloadDataCompletedEventArgs e)
{
string #string = Encoding.Default.GetString(e.Result);
if (#string.Contains("uva"))
{
this.listBox2.Items.Add(numero.Text);
}
}
The program should make the download request of the string with EACH item in the listbox
example:
DownloadString http://127.0.0.1/sv/ + num.text and check if it contains a particular string
I need it to do the following
DownloadString http://127.0.0.1/sv/ + num.text
if
DownloadedString contains uva
then
listbox2.items.add(num.text)
else
next item from listbox1

There's so much wrong with this code, it's no wonder it's going wrong.
You're using Background Workers, don't, you should be using Tasks and async/await
You're looping over every item in the listbox twice, why?
You're spinning up multiple web clients with event subscriptions that are never disposed - probably gonna be leaking some memory here.
Here's how I'd do it:
public async void ButtonThatStartsEverything_Click(object sender, EventArgs e)
{
await DoTheDownloadStuff();
}
public async Task DoTheDownloadStuff()
{
var client = new WebClient();
foreach(var item in ListBox1.Items)
{
var expanded = item.Split(';');
var num = expanded[0];
var result = await client.DownloadDataAsyncTask(new Uri("http://127.0.0.1/sv/" + num));
if (result.Contains("uva"))
{
listBox2.Items.Add(num);
}
}
}
Please be aware that the code was written outside of Visual Studio, it may not be 100% accurate, and may not represent best practice overall e.g. You may want to download the data in parallel, which would require a change.
This code has all the basic stuff you need though.

multiple webrequest through a string array

I have over 2000 url calls to make and with the code below it is taking almost 2 minutes to complete. Could someone help me to speed the process up?
private void button4_Click(object sender, EventArgs e)
{
WebRequest req;
WebResponse res;
string[] lines = File.ReadAllLines(#"c:\data\temp.txt");
for (int i = 0; i < lines.Count(); i++)
{
req = WebRequest.Create(lines[i]);
res = req.GetResponse();
StreamReader rd = new StreamReader(res.GetResponseStream(), Encoding.ASCII);
rd.Close();
res.Close();
textBox1.Text += ".";
}
}
Many thanks

I'm going to suggest that you use Microsoft's Reactive Framework for this. NuGet "Rx-Main", "Rx-WinForms"/"Rx-WPF".
Here's what the code would look like:
private void button4_Click(object sender, EventArgs e)
{
var query =
from line in File.ReadAllLines(#"c:\data\temp.txt").ToObservable()
from result in Observable.Defer(() =>
{
var req = WebRequest.Create(line);
return
Observable.Using(
() => req.GetResponse(),
res => Observable.Using(
() => new StreamReader(res.GetResponseStream(), Encoding.ASCII),
st => Observable.Start(() => st.ReadToEnd())));
})
select new { line, result };
query
.ObserveOn(textBox1)
.Subscribe(x => textBox1.Text += ".");
}
I have assumed that you are trying to read a string from the stream.
This code nicely disposes of all intermediate objects. It also correctly multithreads the requests and it the marshalls the results to the UI thread and updates the text box text.
A slightly cleaner version of this code is this:
private void button4_Click(object sender, EventArgs e)
{
var query =
from line in File.ReadAllLines(#"c:\data\temp.txt").ToObservable()
from result in Observable.Using(
() => new WebClient(),
wc => Observable.Start(() => wc.DownloadString(new Uri(line))))
select new { line, result };
query
.ObserveOn(textBox1)
.Subscribe(x => textBox1.Text += ".");
}
It uses WebClient for the download. It still multithreads as required.

You can't speed-up things much because bottleneck is your Internet connection. However there is something you can do:
1) Do not LINQ count lines, it's an array and its size is known (micro optimization, you won't ever notice this change).
2) Use using to release disposable objects (nothing to do with speed, better error handling: if something went wrong with your code you'll release resources with GC).
3) Make them parallel. This will speed-up things little bit:
private void button4_Click(object sender, EventArgs e)
{
var lines = File.ReadAllLines(#"c:\data\temp.txt");
var options = new ParallelOptions { MaxDegreeOfParallelism = 4 };
Parallel.ForEach(lines, options, line =>
{
var request = WebRequest.Create(line);
using (var response = request.GetResponse())
{
var reader = new StreamReader(response.GetResponseStream(), Encoding.ASCII);
// Do your stuff
BeginInvoke(new MethodInvoker(delegate
{
textBox1.Text += ".";
}));
}
});
}
Few more notes:
MaxDegreeOfParallelism sets maximum number of concurrent requests. Multiple active concurrent connections won't speed-up things indefinitely and they may even slow things down. Some trials will help you to set this value to a reasonable value.
There is not any error checking but network things may temporary go wrong but after a short delay they may work as expected. I suggest to also read System.Net.WebException: The remote name could not be resolved and this for I/O operations.
To make it a more complete example, your click even handler will be:
private void button4_Click(object sender, EventArgs e)
{
var options = new ParallelOptions { MaxDegreeOfParallelism = 4 };
Parallel.ForEach(ReadUrlList(#"c:\data\temp.txt"), options, ProcessUrl);
}
Actual code to process each URL and to read URL list:
private static string[] ReadUrlList(string path)
{
return File.ReadAllLines(#"c:\data\temp.txt");
}
private void ProcessUrl(string url)
{
ProcessResponse(response =>
{
using (var reader = new StreamReader(response.GetResponseStream(), Encoding.ASCII))
{
// Do your stuff
// We're working on separate threads, to access UI we
// have to dispatch the call to UI thread. Note that
// code will be executed asynchronously then local
// objects may have been disposed!
BeginInvoke(new MethodInvoker(delegate
{
textBox1.Text += ".";
}));
}
});
}
With this helper method to hide try/wait pattern for network operations:
private static void ProcessResponse(string url, Action<WebResponse> action)
{
for (int i=1; i <= NumberOfRetries; ++i)
{
try
{
var request = WebRequest.Create(line);
using (var response = request.GetResponse())
{
action(response);
}
break;
}
catch (Exception e)
{
if (i == NumberOfRetries)
throw;
Thread.Sleep(DelayOnRetry);
}
}
}
private const int NumberOfRetries = 3;
private const int DelayOnRetry = 1000;

Since you don't specify a framework version I'll assume you are using at least 4.5.
You can use ActionBlock to easily execute multiple calls concurrently. An ActionBlock executes its action method in a single thread and multiple executions can be performed concurrently.
You could use something like this:
var options=new ExecutionDataflowBlockOptions
{
MaxDegreeOfParallelism = 10
}
var block=new ActionBlock<string>(url=>
{
using(var req = WebRequest.Create(url))
using(var res = req.GetResponse())
{
//Process the response here
}
});
string[] lines = File.ReadAllLines(#"c:\data\temp.txt");
foreach(var line in lines)
{
block.Post(line);
}
block.Complete();
await block.Completion;
You can control how many requests are made concurrently by changing the MaxDegreeOfParallelism method.
You can also call GetResponseAsync to execute the request asynchronously. This won't make them go faster but it will reduce the number of ThreadPool threads used to serve the same number of requests. This means that less CPU is wasted while blocking and context switching.
var block=new ActionBlock<string>(url=>async
{
using(var req = WebRequest.Create(url))
using(var res = await req.GetResponseAsync())
{
//Process the response here
}
});
Disposing requests and responses is important. Unless you dispose the response, the connection to the server remains active. .NET enforces a 2 concurrent request per domain (ie URL) limit so orphaned responses can cause delays until the garbage collector runs and collects them. While you can override the limit, it's best to always dispose of the responses.

Handle code after task are done

I'm trying to show a waiting symbol while while a ASYNC task are doing.
I'm really new to this, so if there are better ways to implement this code, please enlighten me :)
But, everything works except the hiding of the pictureBox1 after the code are done and there are now result found. In other words, when there are a result, the pictureBox1 are hidden
Here are the method that runs every time a outlook item are opened
private void FormRegion1_FormRegionShowing(object sender, System.EventArgs e)
{
if (this.OutlookItem is Microsoft.Office.Interop.Outlook.MailItem)
{
Microsoft.Office.Interop.Outlook.MailItem item = (Microsoft.Office.Interop.Outlook.MailItem)this.OutlookItem;
getContactByEmail(item);
}
}
This is the method that I implement the wait stuff
public async Task getContactByEmail(Microsoft.Office.Interop.Outlook.MailItem item)
{
pictureBox1.Visible = true;
using (var client = new System.Net.Http.HttpClient())
{
client.BaseAddress = new Uri("http://api.....");
client.DefaultRequestHeaders.Accept.Clear();
HttpResponseMessage response = await client.GetAsync("tools/getContactByEmail?email=" + item.SenderEmailAddress + "&key=1232");
if (response.IsSuccessStatusCode)
{
SimpleContact contact = await response.Content.ReadAsAsync<SimpleContact>();
lblName.Text = contact.Name;
lblMobile.Text = contact.Phone;
}
pictureBox1.Visible = false;
}
}

Posting the code that fixes this so the exception are not raised
if (response.IsSuccessStatusCode)
{
SimpleContact contact = await response.Content.ReadAsAsync<SimpleContact>();
if (contact != null)
{
lblName.Text = contact.Name;
lblMobile.Text = contact.Phone;
}
pictureBox1.Visible = false;
}

In C# method names are always CamelCase and asynchronous methods are always suffixed Async. Just conventions.
You might want to extract the non UI code to another asynchronous method to avoid going back and forth to the UI thread:
private async void FormRegion1_FormRegionShowing(object sender, System.EventArgs e)
{
if (this.OutlookItem is Microsoft.Office.Interop.Outlook.MailItem)
{
Microsoft.Office.Interop.Outlook.MailItem item = (Microsoft.Office.Interop.Outlook.MailItem)this.OutlookItem;
pictureBox1.Visible = true;
var contact = GetContactByEmailAsync(item);
if (contact != null)
{
lblName.Text = contact.Name;
lblMobile.Text = contact.Phone;
}
pictureBox1.Visible = false;
}
}
public async Task<SimpleContact> GetContactByEmailAsync(Microsoft.Office.Interop.Outlook.MailItem item)
{
using (var client = new System.Net.Http.HttpClient())
{
client.BaseAddress = new Uri("http://api.....");
client.DefaultRequestHeaders.Accept.Clear();
HttpResponseMessage response = await client.GetAsync(
"tools/getContactByEmail?email=" + item.SenderEmailAddress + "&key=1232")
.ConfigureAwait(false);
return (response.IsSuccessStatusCode)
? await response.Content.ReadAsAsync<SimpleContact>();
: null;
}
}
Note: Don't forget proper exception handling!!!