Develop Reference

RestSharp - Asynchronous Request Reply Pattern

The following situation is given:
A new job is sent to an API via Post Request. This API returns a JobID and the HTTP ResponseCode 202.
This JobID is then used to request a status endpoint. If the end point has a "Finished" property set in the response body, you can continue with step 3.
The results are queried via a result endpoint using the JobID and can be processed.
My question is how I can solve this elegantly and cleanly. Are there perhaps already ready-to-use libraries that implement exactly this functionality? I could not find such functionality for RestSharp or another HttpClient.
The current solution looks like this:
async Task<string> PostNewJob()
{
var restClient = new RestClient("https://baseUrl/");
var restRequest = new RestRequest("jobs");
//add headers
var response = await restClient.ExecutePostTaskAsync(restRequest);
string jobId = JsonConvert.DeserializeObject<string>(response.Content);
return jobId;
}
async Task WaitTillJobIsReady(string jobId)
{
string jobStatus = string.Empty;
var request= new RestRequest(jobId) { Method = Method.GET };
do
{
if (!String.IsNullOrEmpty(jobStatus))
Thread.Sleep(5000); //wait for next status update
var response = await restClient.ExecuteGetTaskAsync(request, CancellationToken.None);
jobStatus = JsonConvert.DeserializeObject<string>(response.Content);
} while (jobStatus != "finished");
}
async Task<List<dynamic>> GetJobResponse(string jobID)
{
var restClient = new RestClient(#"Url/bulk/" + jobID);
var restRequest = new RestRequest(){Method = Method.GET};
var response = await restClient.ExecuteGetTaskAsync(restRequest, CancellationToken.None);
dynamic downloadResponse = JsonConvert.DeserializeObject(response.Content);
var responseResult = new List<dynamic>() { downloadResponse?.ToList() };
return responseResult;
}
async main()
{
var jobId = await PostNewJob();
WaitTillJobIsReady(jobID).Wait();
var responseResult = await GetJobResponse(jobID);
//handle result
}
As #Paulo Morgado said, I should not use Thread.Sleep / Task Delay in production code. But in my opinion I have to use it in the method WaitTillJobIsReady() ? Otherwise I would overwhelm the API with Get Requests in the loop?
What is the best practice for this type of problem?

Long Polling
There are multiple ways you can handle this type of problem, but as others have already pointed out no library such as RestSharp currently has this built in. In my opinion, the preferred way of overcoming this would be to modify the API to support some type of long-polling like Nikita suggested. This is where:
The server holds the request open until new data is available. Once
available, the server responds and sends the new information. When the
client receives the new information, it immediately sends another
request, and the operation is repeated. This effectively emulates a
server push feature.
Using a scheduler
Unfortunately this isn't always possible. Another more elegant solution would be to create a service that checks the status, and then using a scheduler such as Quartz.NET or HangFire to schedule the service at reoccurring intervals such as 500ms to 3s until it is successful. Once it gets back the "Finished" property you can then mark the task as complete to stop the process from continuing to poll. This would arguably be better than your current solution and offer much more control and feedback over whats going on.
Using Timers
Aside from using Thread.Sleep a better choice would be to use a Timer. This would allow you to continuously call a delegate at specified intervals, which seems to be what you are wanting to do here.
Below is an example usage of a timer that will run every 2 seconds until it hits 10 runs. (Taken from the Microsoft documentation)
using System;
using System.Threading;
using System.Threading.Tasks;
class Program
{
private static Timer timer;
static void Main(string[] args)
{
var timerState = new TimerState { Counter = 0 };
timer = new Timer(
callback: new TimerCallback(TimerTask),
state: timerState,
dueTime: 1000,
period: 2000);
while (timerState.Counter <= 10)
{
Task.Delay(1000).Wait();
}
timer.Dispose();
Console.WriteLine($"{DateTime.Now:HH:mm:ss.fff}: done.");
}
private static void TimerTask(object timerState)
{
Console.WriteLine($"{DateTime.Now:HH:mm:ss.fff}: starting a new callback.");
var state = timerState as TimerState;
Interlocked.Increment(ref state.Counter);
}
class TimerState
{
public int Counter;
}
}
Why you don't want to use Thread.Sleep
The reason that you don't want to use Thread.Sleep for operations that you want on a reoccurring schedule is because Thread.Sleep actually relinquishes control and ultimately when it regains control is not up to the thread. It's simply saying it wants to relinquish control of it's remaining time for a least x milliseconds, but in reality it could take much longer for it to regain it.
Per the Microsoft documentation:
The system clock ticks at a specific rate called the clock resolution.
The actual timeout might not be exactly the specified timeout, because
the specified timeout will be adjusted to coincide with clock ticks.
For more information on clock resolution and the waiting time, see the
Sleep function from the Windows system APIs.
Peter Ritchie actually wrote an entire blog post on why you shouldn't use Thread.Sleep.
EndNote
Overall I would say your current approach has the appropriate idea on how this should be handled however, you may want to 'future proof' it by doing some refactoring to utilize on of the methods mentioned above.

Proper way to stop listening UDP socket with RxExtensions

I need to listen UDP socket and after 10 seconds or 100 items in buffer some logic should be invoke. Generally it works ok, but i don't know how to proper stop listening socket.
var ip = new IPEndPoint(IPAddress.Parse("127.0.0.1"), 1234);
var socket = new UdpClient(ip);
var cancellationTokenSource = new CancellationTokenSource();
var observable =
Observable
.FromAsync(socket.ReceiveAsync)
.DoWhile(() => !cancellationTokenSource.IsCancellationRequested)
.Buffer(TimeSpan.FromSeconds(10), 100);
var subscribtion = observable.Subscribe(o =>
{
//logic
});
//simulate close method from another thread
Task.Factory.StartNew(() =>
{
Task.Delay(TimeSpan.FromSeconds(12)).Wait();
cancellationTokenSource.Cancel();
socket.Close();
subscribtion.Dispose();
});
When i simulate closing socket, there exists a situation when exists some data in buffer that can't be processed - is any way to avoid this behavior?
When i send some messages from another process with 500ms delay, it will be works like example below:
20 messages will income
Some logic will be invoke - subscriber logic
4 messages will income
Simulate close method will be invoke
When "close method" will be invoke i need immediately process all data in buffer and close application without waiting even for buffer timeout. Buffer delay time is defined by user, so i don't want wait for invoke subscriber logic, because it can be a quite long time.

Welcome to StackOverflow!
Existing overloads of the buffer method don't support time, count and a gate. But there is an overload which triggers buffer closing when a sequence produces a value. So we will simply create a sequence by merging observables of all the conditions for buffer closing.
Have a look at this demo.
onTime will produce a value after the period specified.
onCount will produce its first value after x items have passed.
onClose will immediately produce a value on subscribe - but we won't connect to it until we decide to.
var producer = Observable.Interval(TimeSpan.FromSeconds(0.2));
var source = producer.Publish().RefCount();
var onClose = Observable.Return(0L).Publish();
var onTime = Observable.Timer(TimeSpan.FromSeconds(2));
var onCount = source.Skip(10);
var bufferClose = Observable.Merge(onClose, onTime, onCount);
var subscription =
source
.Buffer(() => bufferClose)
.Subscribe(list => Console.WriteLine(string.Join(",", list)));
Console.WriteLine("Waiting for close");
Console.ReadLine();
onClose.Connect(); //signal
subscription.Dispose();
Console.WriteLine("Closed");
Console.ReadLine();
This will produce an output based on count or time, until return is pressed, when it immediately closes with what's available in the buffer.

How can I restart communication with an FTP server that sends back reset packets without restarting our process?

We have a (long-running) Windows service that among other things periodically communicates with an FTP server embedded on a third-party device using FtpWebRequest. This works great most of the time, but sometimes our service stops communicating with the device, but as soon as you restart our service everything starts working again.
I've spent some time debugging this with an MCVE (included below) and discovered via Wireshark that once communication starts failing there is no network traffic going to the external FTP server (no packets at all show up going to this IP in Wireshark). If I try to connect to the same FTP from another application on the same machine like Windows explorer everything works fine.
Looking at the packets just before everything stops working I see packets with the reset (RST) flag set coming from the device, so I suspect this may be the issue. Once some part of the network stack on the computer our service in running on receives the reset packet it does what's described in the TCP resets section of this article and blocks all further communication from our process to the device.
As far as I can tell there's nothing wrong with the way we're communicating with the device, and most of the time the exact same code works just fine. The easiest way to reproduce the issue (see MCVE below) seems to be to make a lot of separate connections to the FTP at the same time, so I suspect the issue may occur when there are a lot of connections being made to the FTP (not all by us) at the same time.
The thing is that if we do restart our process everything works fine, and we do need to re-establish communication with the device. Is there a way to re-establish communication (after a suitable amount of time has passed) without having to restart the entire process?
Unfortunately the FTP server is running embedded on a fairly old third-party device that's not likely to be updated to address this issue, and even if it were we'd still want/need to communicate with all the ones already out in the field without requiring our customers to update them if possible.
Options we are aware of:
Using a command line FTP client such as the one built into Windows.
One downside to this is that we need to list all the files in a directory and then download only some of them, so we'd have to write logic to parse the response to this.
We'd also have to download the files to a temp file instead of to a stream like we do now.
Creating another application that handles the FTP communication part that we tear down after each request completes.
The main downside here is that inter-process communication is a bit of a pain.
MCVE
This runs in LINQPad and reproduces the issue fairly reliably. Typically the first several tasks succeed and then the issue occurs, and after that all tasks start timing out. In Wireshark I can see that no communication between my computer and the device is happening.
If I run the script again then all tasks fail until I restart LINQPad or do "Cancel All Threads and Reset" which restarts the process LINQPad uses to run the query. If I do either of those things then we're back to the first several tasks succeeding.
async Task Main() {
var tasks = new List<Task>();
var numberOfBatches = 3;
var numberOfTasksPerBatch = 10;
foreach (var batchNumber in Enumerable.Range(1, numberOfBatches)) {
$"Starting tasks in batch {batchNumber}".Dump();
tasks.AddRange(Enumerable.Range(1, numberOfTasksPerBatch).Select(taskNumber => Connect(batchNumber, taskNumber)));
await Task.Delay(TimeSpan.FromSeconds(5));
}
await Task.WhenAll(tasks);
}
async Task Connect(int batchNumber, int taskNumber) {
try {
var client = new FtpClient();
var result = await client.GetFileAsync(new Uri("ftp://192.168.0.191/logging/20140620.csv"), TimeSpan.FromSeconds(10));
result.Count.Dump($"Task {taskNumber} in batch {batchNumber} succeeded");
} catch (Exception e) {
e.Dump($"Task {taskNumber} in batch {batchNumber} failed");
}
}
public class FtpClient {
public virtual async Task<ImmutableList<Byte>> GetFileAsync(Uri fileUri, TimeSpan timeout) {
if (fileUri == null) {
throw new ArgumentNullException(nameof(fileUri));
}
FtpWebRequest ftpWebRequest = (FtpWebRequest)WebRequest.Create(fileUri);
ftpWebRequest.Method = WebRequestMethods.Ftp.DownloadFile;
ftpWebRequest.UseBinary = true;
ftpWebRequest.KeepAlive = false;
using (var source = new CancellationTokenSource(timeout)) {
try {
using (var response = (FtpWebResponse)await ftpWebRequest.GetResponseAsync()
.WithWaitCancellation(source.Token)) {
using (Stream ftpStream = response.GetResponseStream()) {
if (ftpStream == null) {
throw new InvalidOperationException("No response stream");
}
using (var dataStream = new MemoryStream()) {
await ftpStream.CopyToAsync(dataStream, 4096, source.Token)
.WithWaitCancellation(source.Token);
return dataStream.ToArray().ToImmutableList();
}
}
}
} catch (OperationCanceledException) {
throw new WebException(
String.Format("Operation timed out after {0} seconds.", timeout.TotalSeconds),
WebExceptionStatus.Timeout);
} finally {
ftpWebRequest.Abort();
}
}
}
}
public static class TaskCancellationExtensions {
/// http://stackoverflow.com/a/14524565/1512
public static async Task<T> WithWaitCancellation<T>(
this Task<T> task,
CancellationToken cancellationToken) {
// The task completion source.
var tcs = new TaskCompletionSource<Boolean>();
// Register with the cancellation token.
using (cancellationToken.Register(
s => ((TaskCompletionSource<Boolean>)s).TrySetResult(true),
tcs)) {
// If the task waited on is the cancellation token...
if (task != await Task.WhenAny(task, tcs.Task)) {
throw new OperationCanceledException(cancellationToken);
}
}
// Wait for one or the other to complete.
return await task;
}
/// http://stackoverflow.com/a/14524565/1512
public static async Task WithWaitCancellation(
this Task task,
CancellationToken cancellationToken) {
// The task completion source.
var tcs = new TaskCompletionSource<Boolean>();
// Register with the cancellation token.
using (cancellationToken.Register(
s => ((TaskCompletionSource<Boolean>)s).TrySetResult(true),
tcs)) {
// If the task waited on is the cancellation token...
if (task != await Task.WhenAny(task, tcs.Task)) {
throw new OperationCanceledException(cancellationToken);
}
}
// Wait for one or the other to complete.
await task;
}
}

This reminds me of old(?) IE behaviour of no reload of pages even when the network came back after N unsuccessful tries.
You should try setting the FtpWebRequest's cache policy to BypassCache.
HttpRequestCachePolicy bypassPolicy = new HttpRequestCachePolicy(
HttpRequestCacheLevel.BypassCache
);
ftpWebRequest.CachePolicy = bypassPolicy;
after setting KeepAlive.

I had the same issue, when trying to connect to an ftps server without the EnableSsl = true. The connection would fail twice, Wireshark shows the RST command, and then no more requests would leave the network resulting in the timeout exception, even after setting the EnableSsl = true.
I found setting the ConnectionGroupName allows the connection to reset and use a new port.
eg:
request.ConnectionGroupName = Guid.NewGuid();
Beware of port exhaustion using this method however, see https://learn.microsoft.com/en-us/troubleshoot/dotnet/framework/ports-run-out-use-connectiongroupname

Stopping threads in C#

I need to make TcpClient event driven rather than polling for messages all the time, so I thought: I will create a thread that would wait for a message to come and fire an event once it does. Here is a general idea:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using System.Net.Sockets;
using System.Net;
namespace ThreadsTesting
{
class Program
{
static void Main(string[] args)
{
Program p = new Program();
//imitate a remote client connecting
TcpClient remoteClient = new TcpClient();
remoteClient.Connect(IPAddress.Parse("127.0.0.1"), 80);
//start listening to messages
p.startMessageListener();
//send some fake messages from the remote client to our server
for (int i = 0; i < 5; i++)
{
remoteClient.GetStream().Write(new byte[] { 0x80 }, 0, 1);
Thread.Sleep(200);
}
//sleep for a while to make sure the cpu is not used
Console.WriteLine("Sleeping for 2sec");
Thread.Sleep(2000);
//attempt to stop the server
p.stopMessageListener();
Console.ReadKey();
}
private CancellationTokenSource cSource;
private Task listener;
private TcpListener server;
private TcpClient client;
public Program()
{
server = new TcpListener(IPAddress.Parse("127.0.0.1"), 80);
server.Start();
}
private void startMessageListener()
{
client = server.AcceptTcpClient();
//start listening to the messages
cSource = new CancellationTokenSource();
listener = Task.Factory.StartNew(() => listenToMessages(cSource.Token), cSource.Token);
}
private void stopMessageListener()
{
Console.Out.WriteLine("Close requested");
//send cancelation signal and wait for the thread to finish
cSource.Cancel();
listener.Wait();
Console.WriteLine("Closed");
}
private void listenToMessages(CancellationToken token)
{
NetworkStream stream = client.GetStream();
//check if cancelation requested
while (!token.IsCancellationRequested)
{
//wait for the data to arrive
while (!stream.DataAvailable)
{ }
//read the data (always 1 byte - the message will always be 1 byte)
byte[] bytes = new byte[1];
stream.Read(bytes, 0, 1);
Console.WriteLine("Got Data");
//fire the event
}
}
}
}
This for obvious reasons doesn't work correctly:
while (!stream.DataAvailable) blocks the thread and uses always 25% CPU (on 4-core CPU), even if no data is there.
listener.Wait(); will wait for ever since the while loop doesn't pick up that cancel has been called.
My alternative solution would be using async calls within the listenToMessages method:
private async Task listenToMessages(CancellationToken token)
{
NetworkStream stream = client.GetStream();
//check if cancelation requested
while (!token.IsCancellationRequested)
{
//read the data
byte[] bytes = new byte[1];
await stream.ReadAsync(bytes, 0, 1, token);
Console.WriteLine("Got Data");
//fire the event
}
}
This works exactly as I expected:
The CPU is not blocked if there are no messages in the queue, but we are still waiting for them
Cancelation request is picked up correctly and thread finished as expected
I wanted to go further though. Since listenToMessages now returns a Task itself, I thought there is no need of starting a task that would execute that method. Here is what I did:
private void startMessageListener()
{
client = server.AcceptTcpClient();
//start listening to the messages
cSource = new CancellationTokenSource();
listener = listenToMessages(cSource.Token);
}
This doesn't work as I have expected in the sence that when Cancel() is called, the ReadAsync() method doesn't seem to pick up the cancelation message from the token, and the thread doesn't stop, instead it is stuck on the ReadAsync() line.
Any idea why is this happening? I would think the ReadAsync will still pick up the token, as it did before...
Thanks for all your time and help.
-- EDIT --
Ok so after more in depth evaluation my solution no.2 doesn't really work as expected:
the thread itself ends to the caller and so the caller can continue. However, the thread is not "dead", so if we send some data it will execute once more!
Here is an example:
//send some fake messages from the remote client to our server
for (int i = 0; i < 5; i++)
{
remoteClient.GetStream().Write(new byte[] { 0x80 }, 0, 1);
Thread.Sleep(200);
}
Console.WriteLine("Sleeping for 2sec");
Thread.Sleep(2000);
//attempt to stop the server
p.stopListeners();
//check what will happen if we try to write now
remoteClient.GetStream().Write(new byte[] { 0x80 }, 0, 1);
Thread.Sleep(200);
Console.ReadKey();
This will output the message "Got Data" even though in theory we stopped! I will investigate further and report on my findings.

With modern libraries, any time you type new Thread, you've already got legacy code.
The core solution for your situation is asynchronous socket methods. There are a few ways to approach your API design, though: Rx, TPL Dataflow, and plain TAP come to mind. If you truly want events then EAP is an option.
I have a library of EAP sockets here. It does require a synchronizing context, so you'd have to use something like ActionDispatcher (included in the same library) if you need to use it from a Console application (you don't need this if you're using it from WinForms/WPF).

ReadAsync doesn't seem to support cancellation on a NetworkStream - have a look at the answers in this thread:
NetworkStream.ReadAsync with a cancellation token never cancels

How do i wait for a load of asyncronous pings to all complete in C#?

I am using the code below to start a load of asyncronous pings. I also have a function which is called by PingCompletedEventHandler, this function adds the address of a successfull ping to a list for use later.
I would like to know how It would be possible to call a function once all of the ping threads have completed?
public static void PingRange(IPRange range)
{
foreach (IPAddress ip in range.GetAllIP())
{
// Create Pinger
AutoResetEvent waiter = new AutoResetEvent(false);
Ping pingSender = new Ping();
pingSender.PingCompleted += new PingCompletedEventHandler(pingSender_PingCompleted);
// Create a 32 byte data buffer
string data = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
byte[] buffer = Encoding.ASCII.GetBytes(data);
// Wait 4 seconds for a reply
int timeout = 4000;
PingOptions options = new PingOptions(64, true);
pingSender.SendAsync(ip, timeout, buffer, options, waiter);
PingStartCount++;
}
}

This is fairly easy if you are using .NET 4.0. You can use the CountdownEvent class instead of the PingStartCount counter and waiter event. CountdownEvent will perform the job of both.
public static void PingRange(IPRange range)
{
var finished = new CountdownEvent(1);
foreach (IPAddress ip in range.GetAllIP())
{
finished.AddCount(); // Indicate that a new ping is pending
var pingSender = new Ping();
pingSender.PingCompleted +=
(sender, e) =>
{
finished.Signal(); // Indicate that this ping is complete
};
string data = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
byte[] buffer = Encoding.ASCII.GetBytes(data);
PingOptions options = new PingOptions(64, true);
pingSender.SendAsync(ip, 4000, buffer, options, waiter);
}
finished.Signal(); // Indicate that all pings have been submitted
finished.Wait(); // Wait for all pings to complete
}

A simple solution would be to use the PingStartCount variable in the pingSender_PingCompleted event handler to track how much pings are still waiting for completion (deduct 1 every time pingSender_PingCompleted is called). Once the PingStartCount reaches 0 you would have recieved all pings back.
However, this is not full proof as you may still be in the process of starting pings, when the counter reaches 0. To make sure this is not the case, a simple bool variable that you set to false at the start of PingRange and to true at the end of PingRange allows you to check if pings are still being send out.