Requirement:
A Silverlight client that loads a file from the user's computer.
For each row in the file it needs to issue a GET to a url and either approve or reject the row based on the JSON validation message returned.
When all rows have completed, show the user a summary of how many rows passed and how many failed.
I would have preferred doing this "synchronously" in the BackgroundWorker, but that breaks the SL async thinking.
My code currently passes the result from the OpenFileDialog to a BackgroundWorker which reads the file into a list of strongly typed objects (client side validation). I create a WebClient and call DownloadStringAsync multiple times as I loop over the lines. More often than not, the background worker completes and runs the completed event method long before the WebClient is completed. The UI thread can run as async as it wants, but I need to either wait with completing the backgroundworker thread until it is actually finished getting the data, or have some additional event handler that works when all (hundreds) of rows have been validated.
What is best practice when wanting to handle the last Completed event of X total events? Any use of auto reset events here that could help?
You shouldn't need to go to a background thread for that, nor you need to use an AutoResetEvent here. After reading the file, count the number of lines (= number of requests you'll send) and store it in an instance variable; then fire all the WebRequest.DownloadAsync (or some other WebClient async call to download the data). On each callback for the async method, you'd Interlock.Decrement the instance variable, and when it reaches 0, you know that you have all the results and you can display the summary to the client.
You can choose between random response sequence (but parallel execution)
or ordered response but in sequential mode.
There is different cons and pros.
using System;
using System.Net;
using System.Reactive.Linq;//Rx libriary
using System.Threading;
using System.Windows;
using System.Windows.Controls;
public partial class MainPage : UserControl
{
private int count = 0;
private int error = 0;
public MainPage()
{
InitializeComponent();
}
private void Button_Click(object sender, RoutedEventArgs e)
{
ThreadPool.QueueUserWorkItem(StartParallel);
//ThreadPool.QueueUserWorkItem(StartSequential);
}
private void Update(Exception exception)
{
if (exception == null)
Interlocked.Increment(ref count);
else
Interlocked.Increment(ref error);
if ((count%100) == 0)
{
int count1 = count;
Dispatcher.BeginInvoke(() => { textBox.Text = count1.ToString(); });
}
}
private void StartSequential(object o)
{
//single instance of WebClient
WebClient wc = new WebClient();
var observer = Observable.FromEventPattern<DownloadStringCompletedEventArgs>(wc, "DownloadStringCompleted")
.Select(newResult => new {newResult.EventArgs.Error, newResult.EventArgs.Result});
wc.DownloadStringAsync(new Uri("http://localhost:7123/SilverlightApplication2TestPage.aspx"));
int i = 0;
foreach (var nextValue in observer.Next())
{
if (i == 10000) break;
wc.DownloadStringAsync(new Uri("http://localhost:7123/SilverlightApplication2TestPage.aspx"));
Update(nextValue.Error);
}
}
private void StartParallel(object o)
{
for (int i = 0; i < 10000; i++)
{
//multiple instance of WebClient
WebClient t = new WebClient();
t.DownloadStringCompleted +=
(x, nextValue) => Update(nextValue.Error);//order of result sequence is not guaranteed
t.DownloadStringAsync(new Uri("http://localhost:7123/SilverlightApplication2TestPage.aspx"));
}
}
}
Related
I have a .NET framework Windows Forms application with a form that has this code:
using System;
using System.Collections.Generic;
using System.IO;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using System.Windows.Forms;
namespace test
{
public partial class Main : Form
{
public int exitCode = 1;
private Options opts;
CancellationTokenSource cancellationSource = new CancellationTokenSource();
public Main(Options opts)
{
InitializeComponent();
this.opts = opts;
}
private void btnCancel_Click(object sender, EventArgs e)
{
exitCode = 1;
cancellationSource.Cancel();
Close();
}
async Task doUpload()
{
using (var content = new MultipartFormDataContent())
{
List<FileStream> streams = new List<FileStream>();
try
{
foreach (string fPath in opts.InputFiles)
{
FileStream stream = new FileStream(fPath, FileMode.Open, FileAccess.Read);
streams.Add(stream);
content.Add(new StreamContent(stream), fPath);
}
var progressContent = new ProgressableStreamContent(
content,
4096,
(sent, total) =>
{
double percent = 100 * sent / total;
progressBar.Value = (int)percent;
});
using (var client = new HttpClient())
{
using (var response = await client.PostAsync(opts.URL, progressContent, cancellationSource.Token))
{
if (response.IsSuccessStatusCode)
{
exitCode = 0;
}
else
{
MessageBox.Show(
response.Content.ToString(),
"Error " + response.StatusCode,
MessageBoxButtons.OK, MessageBoxIcon.Error
);
}
Close();
}
}
}
finally
{
foreach (FileStream stream in streams)
{
stream.Close();
}
}
}
}
private void Main_Load(object sender, EventArgs e)
{
}
private void Main_FormClosing(object sender, FormClosingEventArgs e)
{
e.Cancel = !cancellationSource.IsCancellationRequested;
}
private void Main_Shown(object sender, EventArgs e)
{
doUpload();
}
}
}
The ProgressableStreamContent is the same that was given here: C#: HttpClient, File upload progress when uploading multiple file as MultipartFormDataContent
The problem is that the response is never returned. In other words: await for postAsync never completes. Also, the progress callback is never called back. Even if I try to use a POST URL that contains a non-exsitent domain, nothing happens. I guess it is a deadlock, but I don't see how? The async Task's result is never used anywhere and it is not awaited for.
It is different from An async/await example that causes a deadlock because .Result is not used and the method is never awaited for, and also it seems that calling ConfigureAwait(false) ha no effect.
UPDATE: I have created a new github repo for this question, so anyone can test it:
https://github.com/nagylzs/csharp_http_post_example
UPDATE: Finally it works. ConfigureAwait is not needed. All UI update operations must be placed inside Invoke. I have updated the test repo to the working version. Also added TLSv1.2 support (which is disabled by default).
PostAsync in the code you've posted doesn't block (but it really never returns though!). It throws an exception:
System.InvalidOperationException: Cross-thread operation not valid: Control 'progressBar' accessed from a thread other than the thread it was created on.
That's the reason for the breakpoints that didn't worked for you. The right solution would be:
var progressContent = new ProgressableStreamContent(
content,
4096,
(sent, total) =>
{
Invoke((Action) (() => {
double percent = 100 * sent / total;
progressBar.Value = (int) percent;
}));
});
(either add Invoke or BeginInvoke to the callback)
The callbacks of the HTTP client are called on a background thread, and you have to put them into your window's even queue if you want them to access your UI controls.
.ConfigureAwait(false) has nothing to do with this issue, you shouldn't use it in UI context (quite the opposite: you want it to put the continuation onto the UI thread, so you shouldn't use it).
You need to change this:
client.PostAsync(opts.URL, progressContent, cancellationSource.Token)
to
client.PostAsync(opts.URL, progressContent, cancellationSource.Token).ConfigureAwait(false)
This is already discussed so you can find additional resources on the net, but this should be good starting point.
I am doing some TCP programming and I want to simulate some latency.
Each "message" (a byte[] representing a serialized object) must be delayed by some time t. I thought that I can have one function to collect raw messages:
private Queue<byte[]> rawMessages = new Queue<byte[]>();
private void OnDataReceived(object sender, byte[] data)
{
rawMessages.Enqueue(data);
}
Another method with a while-loop to continuously read from rawMessages and then delay each one:
private Queue<byte[]> delayedRawMessages = new Queue<byte[]>();
delayMessagesInTask = Task.Factory.StartNew(() =>
{
while (true) //TODO: Swap a cancellation token
{
if(rawMessages.Count > 0){
var rawMessage = rawMessages.Dequeue();
//???
//Once the message has been delayed, enqueue it into another buffer.
delayedRawMessages.Enqueue(rawMessage);
}
}
});
I thought that to delay each message, I could have another method to spawn a thread which uses Thread.Sleep(t) to wait for time t and then enqueue delayedRawMessages. I'm sure this will work, but I think there must be a better way. The issue with the Thread.Sleep approach is that message 2 might finish being delayed before message 1... I obviously need the messages to be delayed and done so in the correct order, otherwise I would not be using TCP.
I'm looking for a way to do this that will delay for as close to time t as possible and will not impact the rest of the application by slowing it down.
Does anyone here know of a better way to do this?
I decided to go the producer/multiple consumer approach.
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
public class LatencySimulator {
public enum SimulatorType { UP, DOWN };
public SimulatorType type;
public int latency = 0;
public int maxConsumers = 50;
public BlockingCollection<byte[]> inputQueue = new BlockingCollection<byte[]>(new ConcurrentQueue<byte[]>());
public Queue<byte[]> delayedMessagesQueue = new Queue<byte[]>();
void CreateConsumers()
{
for (int i = 0; i < maxConsumers; i++)
{
Task.Factory.StartNew(() => Consumer(),TaskCreationOptions.LongRunning);
}
}
private void Consumer()
{
foreach (var item in inputQueue.GetConsumingEnumerable())
{
Thread.Sleep(latency);
delayedMessagesQueue.Enqueue(item);
}
}
}
To use it, create a new LatencySimulator, set its 'type', max consumers and latency to simulate. Call CreateConsmers() and then populate the inputQueue. When the messages have been delayed, they will appear in the delayedMessagesQueue.
I do not really know if this is an ideal way to achieve my goal, but it works... for now.
How should the reading of bulk data from a device in C# be handled in .NET 4.0? Specifically I need to read quickly from a USB HID device that emits reports over 26 packets where order must be preserved.
I've tried doing this in a BackgroundWorker thread. It reads one packet from the device at a time, and process it, before reading more. This gives reasonably good response times, but it is liable to lose a packet here and there, and the overhead costs of a single packet read adds up.
while (!( sender as BackgroundWorker ).CancellationPending) {
//read a single packet
//check for header or footer
//process packet data
}
}
What is the best practice in C# for reading a device like this?
Background:
My USB HID device continuously reports a large amount of data. The data is split over 26 packets and I must preserver the order. Unfortunately the device only marks the first the last packets in each report, so I need to be able to catch every other packet in between.
For .Net 4 you can use a BlockingCollection to provide a threadsafe queue that can be used by a producer and a consumer. The BlockingCollection.GetConsumingEnumerable() method provides an enumerator which automatically terminates when the queue has been marked as completed using CompleteAdding() and is empty.
Here's some sample code. The payload is an array of ints in this example, but of course you would use whatever data type you need.
Note that for your specific example, you can use the overload of GetConsumingEnumerable() which accepts an argument of type CancellationToken.
using System;
using System.Collections.Concurrent;
using System.Threading;
using System.Threading.Tasks;
namespace Demo
{
public static class Program
{
private static void Main()
{
var queue = new BlockingCollection<int[]>();
Task.Factory.StartNew(() => produce(queue));
consume(queue);
Console.WriteLine("Finished.");
}
private static void consume(BlockingCollection<int[]> queue)
{
foreach (var item in queue.GetConsumingEnumerable())
{
Console.WriteLine("Consuming " + item[0]);
Thread.Sleep(25);
}
}
private static void produce(BlockingCollection<int[]> queue)
{
for (int i = 0; i < 1000; ++i)
{
Console.WriteLine("Producing " + i);
var payload = new int[100];
payload[0] = i;
queue.Add(payload);
Thread.Sleep(20);
}
queue.CompleteAdding();
}
}
}
For .Net 4.5 and later, you could use the higher-level classes from Microsoft's Task Parallel Library, which has a wealth of functionality (and can be somewhat daunting at first sight).
Here's the same example using TPL DataFlow:
using System;
using System.Threading;
using System.Threading.Tasks;
using System.Threading.Tasks.Dataflow;
namespace Demo
{
public static class Program
{
private static void Main()
{
var queue = new BufferBlock<int[]>();
Task.Factory.StartNew(() => produce(queue));
consume(queue).Wait();
Console.WriteLine("Finished.");
}
private static async Task consume(BufferBlock<int[]> queue)
{
while (await queue.OutputAvailableAsync())
{
var payload = await queue.ReceiveAsync();
Console.WriteLine("Consuming " + payload[0]);
await Task.Delay(25);
}
}
private static void produce(BufferBlock<int[]> queue)
{
for (int i = 0; i < 1000; ++i)
{
Console.WriteLine("Producing " + i);
var payload = new int[100];
payload[0] = i;
queue.Post(payload);
Thread.Sleep(20);
}
queue.Complete();
}
}
}
If missing packets is a concern do not do your processing and your reading on the same thread. Starting with .NET 4.0 they added the System.Collections.Concurrent namespace which makes this very easy to do. All you need is a BlockingCollection which behaves as a queue for your incoming packets.
BlockingCollection<Packet> _queuedPackets = new BlockingCollection<Packet>(new ConcurrentQueue<Packet>());
void readingBackgroundWorker_DoWork(object sender, DoWorkEventArgs e)
{
while (!( sender as BackgroundWorker ).CancellationPending)
{
Packet packet = GetPacket();
_queuedPackets.Add(packet);
}
_queuedPackets.CompleteAdding();
}
void processingBackgroundWorker_DoWork(object sender, DoWorkEventArgs e)
{
List<Packet> report = new List<Packet>();
foreach(var packet in _queuedPackets.GetConsumingEnumerable())
{
report.Add(packet);
if(packet.IsLastPacket)
{
ProcessReport(report);
report = new List<Packet>();
}
}
}
What will happen is while _queuedPackets is empty _queuedPackets.GetConsumingEnumerable() will block the thread not consuming any resources. As soon as a packet arrives it will unblock and do the next iteration of the foreach.
When you call _queuedPackets.CompleteAdding(); the foreach on your processing thread will run till the collection is empty then exit the foreach loop. If you don't want it to "finish up the queue" when you cancel you can easily change it up to quit early. I also am going to switch to using Tasks instead of Background workers because it makes the passing in parameters much easier to do.
void ReadingLoop(BlockingCollection<Packet> queue, CancellationToken token)
{
while (!token.IsCancellationRequested)
{
Packet packet = GetPacket();
queue.Add(packet);
}
queue.CompleteAdding();
}
void ProcessingLoop(BlockingCollection<Packet> queue, CancellationToken token)
{
List<Packet> report = new List<Packet>();
try
{
foreach(var packet in queue.GetConsumingEnumerable(token))
{
report.Add(packet);
if(packet.IsLastPacket)
{
ProcessReport(report);
report = new List<Packet>();
}
}
}
catch(OperationCanceledException)
{
//Do nothing, we don't care that it happened.
}
}
//This would replace your backgroundWorker.RunWorkerAsync() calls;
private void StartUpLoops()
{
var queue = new BlockingCollection<Packet>(new ConcurrentQueue<Packet>());
var cancelRead = new CancellationTokenSource();
var cancelProcess = new CancellationTokenSource();
Task.Factory.StartNew(() => ReadingLoop(queue, cancelRead.Token));
Task.Factory.StartNew(() => ProcessingLoop(queue, cancelProcess.Token));
//You can stop each loop indpendantly by calling cancelRead.Cancel() or cancelProcess.Cancel()
}
I'm currently using WebClient to open a few websites but after a period of time I start receiving Error 403 messages.
I'm assuming it's because I'm hitting their servers to frequently/quickly. I'm assuming all I need to do is add a Thread.Sleep time-frame between requests.
Since I'm having to do this a large number of times is there a suggestion on how to handle the throttling issue without have to take a tremendous amount of time?
For instance 3 secs between requests will end up taking me like 3 hrs to do.
So the question is, is Thread.Sleep really the right solution for this? and if it is, what is a good time-frame for it?
As a side note I have also used HttpWebRequest and ran into the same problem. I do still use it in other code projects and technically am hoping to utilize the same solution (or close to it) for these other code projects utilizing the HttpWebRequest
try running the requests in parallel
public static void RunRequest(Uri uri, Action<string> onCompleted)
{
var client = new WebClient();
client.DownloadStringCompleted += (sender, e) => onCompleted(e.Result);
client.DownloadStringAsync(uri);
};
warning: code is not testet and i have never used WebClient
private const int _maxParallelRequest = 10;
private int _requestCount = 0;
private readonly object _sync = new object();
private ManualResetEvent _ev = new ManualResetEvent(false);
while(true)
{
foreach (var uri in _allYourUris)
{
var wait = false;
lock (_sync)
{
if (_requestCount >= _maxParallelRequest)
wait = true;
}
if (!wait)
{
lock (_sync) { ++_requestCount; }
RunRequest(uri, r => {
lock (_sync)
{
--_requestCount;
_ev.Set();
}
// handle r
});
continue;
}
_ev.WaitOne();
}
Thread.Sleep(3000);
}
I want to make 10 asynchronous http requests at once and only process the results when all have completed and in a single callback function. I also do not want to block any threads using WaitAll (it is my understanding that WaitAll blocks until all are complete). I think I want to make a custom IAsyncResult which will handle multiple calls. Am I on the right track? Are there any good resources or examples out there that describe handling this?
I like Darin's solution. But, if you want something more traditional, you can try this.
I would definitely use an array of wait handles and the WaitAll mechanism:
static void Main(string[] args)
{
WaitCallback del = state =>
{
ManualResetEvent[] resetEvents = new ManualResetEvent[10];
WebClient[] clients = new WebClient[10];
Console.WriteLine("Starting requests");
for (int index = 0; index < 10; index++)
{
resetEvents[index] = new ManualResetEvent(false);
clients[index] = new WebClient();
clients[index].OpenReadCompleted += new OpenReadCompletedEventHandler(client_OpenReadCompleted);
clients[index].OpenReadAsync(new Uri(#"http:\\www.google.com"), resetEvents[index]);
}
bool succeeded = ManualResetEvent.WaitAll(resetEvents, 10000);
Complete(succeeded);
for (int index = 0; index < 10; index++)
{
resetEvents[index].Dispose();
clients[index].Dispose();
}
};
ThreadPool.QueueUserWorkItem(del);
Console.WriteLine("Waiting...");
Console.ReadKey();
}
static void client_OpenReadCompleted(object sender, OpenReadCompletedEventArgs e)
{
// Do something with data...Then close the stream
e.Result.Close();
ManualResetEvent readCompletedEvent = (ManualResetEvent)e.UserState;
readCompletedEvent.Set();
Console.WriteLine("Received callback");
}
static void Complete(bool succeeded)
{
if (succeeded)
{
Console.WriteLine("Yeah!");
}
else
{
Console.WriteLine("Boohoo!");
}
}
In .NET 4.0 there's a nice parallel Task library that allows you to do things like:
using System;
using System.Linq;
using System.Net;
using System.Threading.Tasks;
class Program
{
public static void Main()
{
var urls = new[] { "http://www.google.com", "http://www.yahoo.com" };
Task.Factory.ContinueWhenAll(
urls.Select(url => Task.Factory.StartNew(u =>
{
using (var client = new WebClient())
{
return client.DownloadString((string)u);
}
}, url)).ToArray(),
tasks =>
{
var results = tasks.Select(t => t.Result);
foreach (var html in results)
{
Console.WriteLine(html);
}
});
Console.ReadLine();
}
}
As you can see for each url in the list a different task is started and once all tasks are completed the callback is invoked and passed the result of all tasks.
I think you are better off using the WaitAll approach. Otherwise you will be processing 10 IAsyncResult callbacks, and using a semaphore to determine that all 10 are finally complete.
Keep in mind that WaitAll is very efficient; it is not like the silliness of having a thread "sleep." When a thread sleeps, it continues to use processing time. When a thread is "de-scheduled" because it hit a WaitAll, then the thread no longer consumes any processor time. It is very efficient.