Develop Reference

How to properly use Dapper SQL Connections in async methods

I have a controller method that has multiple tasks running simultaneously and I use await Task.WhenAll() to ensure all tasks finish before returning the response in my controller. Each tasks runs a SQL query and does some other stuff but I run into an error:
An unhandled exception has occurred while executing the request.
System.InvalidOperationException: ExecuteReader requires an open and available Connection. The connection's current state is open.
My code is setup as follows:
using (var connection = new SqlConnection(connectionString))
{
List<Task> tasks = new List<Task>();
tasks.Add(Task.Run(async () =>
{
connection.Query("exec dbo.DOCFUNCTION #TAB_ID, #DriveItemID, #WebURL, #EditLink", new
{
TAB_ID = queryResult.TAB_ID,
DriveItemID = UploadResponse.Id,
WebURL = UploadResponse.WebUrl,
EditLink = SharedLinkResponse.Link.WebUrl,
}); //run other code inside task
}));
tasks.Add(Task.Run(async () =>
{
connection.Query("exec dbo.DOCFUNCTION #TAB_ID, #DriveItemID, #WebURL, #EditLink", new
{
TAB_ID = queryResult.TAB_ID,
DriveItemID = UploadResponse.Id,
WebURL = UploadResponse.WebUrl,
EditLink = SharedLinkResponse.Link.WebUrl,
});//run other code inside task
}));
tasks.Add(Task.Run(async () =>
{
var queryResult = connection.QuerySingle(query, new { jsonData = jsonData, certificationJSON = certificationJSON, UserID = UserID });//run other code inside task
}));
await Task.WhenAll(tasks);
}
If I remove the tasks.add() and just use await Task.Run() for each tasks then everything works fine. Its when I try to do this asynchronously it causes this error. I am not sure what the issue could be. Am I not allowed to make multiple queries in separate threads that could occur close in time?

Listening to Firestore Document Changes Through RPC API

I'm currently trying to consume the RPC API provided by Google Firestore through an Unity Application via the following Code Snippet. The snippet is based on the Protobuffer Definitions and the official gRPC Tutorial showing how to implement and consume bidirectional streams:
public async void ListenTest()
{
// create cancellation token:
CancellationTokenSource source = new CancellationTokenSource();
CancellationToken cancellationToken = source.Token;
using (AsyncDuplexStreamingCall<ListenRequest, ListenResponse> call = client.Listen())
{
// create listenRequest with corresponding properties:
var listenRequest = new ListenRequest();
listenRequest.Database = database;
listenRequest.AddTarget = new Target();
listenRequest.AddTarget.Documents = new Target.Types.DocumentsTarget
{
Documents = { database + "/documents/MyCollection/MyDocument" }
};
// task to handle response stream messages
var responseReaderTask = Task.Run(async () =>
{
while (await call.ResponseStream.MoveNext(cancellationToken))
{
ListenResponse loc = call.ResponseStream.Current;
Debug.Log("Received " + loc);
}
});
Debug.Log("Listening for " + database + "/documents/MyCollection/MyDocument");
// 1. asynchronously send listenRequest
await call.RequestStream.WriteAsync(listenRequest);
// 2. wait until listenRequest is sent
await call.RequestStream.CompleteAsync();
// 3. handle responses...
await responseReaderTask;
}
}
Executing this immediately terminates responseReaderTask, no changes made within Firestore reach the application.
Is there something wrong with this code? I read a yet unanswered similar question, which explains the same problem for GoLang. May be the same issue.
Retrieving a single document via client.GetDocument(getDocumentRequest) works.

Ok, I found the error I made, maybe this helps future generations, I just removed the following line to make it work:
// 2. wait until listenRequest is sent
await call.RequestStream.CompleteAsync();
It seems in an AsyncDuplexStreamingCall the streams are not independent from each other (unlike the two bidirectional streams in the example I used as a basis). On completion of the RequestStream the ResponseStream also terminated, so I wasn't able to observe any changes. The Duplex in AsyncDuplexStreamingCall kind of implies this behaviour.

Async TcpClient disables the Internet after 30 seconds of sending data

I wrote a small library for sending simple HTTP requests for the sake of interest based on asynchronous TcpClient. I went to the tests and faced a problem. When I run about 200 tasks to send a request, the program sends and receives data in about 30 seconds, after which Internet access is cut off and requests are stopped.
var tasks = new List<Task>();
for (int i = 0; i < 200; i++)
{
tasks.Add(Task.Run(async () =>
{
while (true)
{
var message = new GetHttpMessage("www.stackoverflow.com")
{
Headers = { {"User-agent","Test-agent-1"} }
};
var req = new Request(message);
var resp = await req.SendAsync();
}
}));
}
await Task.WhenAll(tasks);
The Internet disappears completely, the browser does not work too. Having decided to look at the activity of the network, I saw that at the beginning of work the connections actively appear and disappear, and at the time of stopping they simply hang in a certain state.
Screenshot of network activity
I use a router, maybe the problem is in it?

How can I restart communication with an FTP server that sends back reset packets without restarting our process?

We have a (long-running) Windows service that among other things periodically communicates with an FTP server embedded on a third-party device using FtpWebRequest. This works great most of the time, but sometimes our service stops communicating with the device, but as soon as you restart our service everything starts working again.
I've spent some time debugging this with an MCVE (included below) and discovered via Wireshark that once communication starts failing there is no network traffic going to the external FTP server (no packets at all show up going to this IP in Wireshark). If I try to connect to the same FTP from another application on the same machine like Windows explorer everything works fine.
Looking at the packets just before everything stops working I see packets with the reset (RST) flag set coming from the device, so I suspect this may be the issue. Once some part of the network stack on the computer our service in running on receives the reset packet it does what's described in the TCP resets section of this article and blocks all further communication from our process to the device.
As far as I can tell there's nothing wrong with the way we're communicating with the device, and most of the time the exact same code works just fine. The easiest way to reproduce the issue (see MCVE below) seems to be to make a lot of separate connections to the FTP at the same time, so I suspect the issue may occur when there are a lot of connections being made to the FTP (not all by us) at the same time.
The thing is that if we do restart our process everything works fine, and we do need to re-establish communication with the device. Is there a way to re-establish communication (after a suitable amount of time has passed) without having to restart the entire process?
Unfortunately the FTP server is running embedded on a fairly old third-party device that's not likely to be updated to address this issue, and even if it were we'd still want/need to communicate with all the ones already out in the field without requiring our customers to update them if possible.
Options we are aware of:
Using a command line FTP client such as the one built into Windows.
One downside to this is that we need to list all the files in a directory and then download only some of them, so we'd have to write logic to parse the response to this.
We'd also have to download the files to a temp file instead of to a stream like we do now.
Creating another application that handles the FTP communication part that we tear down after each request completes.
The main downside here is that inter-process communication is a bit of a pain.
MCVE
This runs in LINQPad and reproduces the issue fairly reliably. Typically the first several tasks succeed and then the issue occurs, and after that all tasks start timing out. In Wireshark I can see that no communication between my computer and the device is happening.
If I run the script again then all tasks fail until I restart LINQPad or do "Cancel All Threads and Reset" which restarts the process LINQPad uses to run the query. If I do either of those things then we're back to the first several tasks succeeding.
async Task Main() {
var tasks = new List<Task>();
var numberOfBatches = 3;
var numberOfTasksPerBatch = 10;
foreach (var batchNumber in Enumerable.Range(1, numberOfBatches)) {
$"Starting tasks in batch {batchNumber}".Dump();
tasks.AddRange(Enumerable.Range(1, numberOfTasksPerBatch).Select(taskNumber => Connect(batchNumber, taskNumber)));
await Task.Delay(TimeSpan.FromSeconds(5));
}
await Task.WhenAll(tasks);
}
async Task Connect(int batchNumber, int taskNumber) {
try {
var client = new FtpClient();
var result = await client.GetFileAsync(new Uri("ftp://192.168.0.191/logging/20140620.csv"), TimeSpan.FromSeconds(10));
result.Count.Dump($"Task {taskNumber} in batch {batchNumber} succeeded");
} catch (Exception e) {
e.Dump($"Task {taskNumber} in batch {batchNumber} failed");
}
}
public class FtpClient {
public virtual async Task<ImmutableList<Byte>> GetFileAsync(Uri fileUri, TimeSpan timeout) {
if (fileUri == null) {
throw new ArgumentNullException(nameof(fileUri));
}
FtpWebRequest ftpWebRequest = (FtpWebRequest)WebRequest.Create(fileUri);
ftpWebRequest.Method = WebRequestMethods.Ftp.DownloadFile;
ftpWebRequest.UseBinary = true;
ftpWebRequest.KeepAlive = false;
using (var source = new CancellationTokenSource(timeout)) {
try {
using (var response = (FtpWebResponse)await ftpWebRequest.GetResponseAsync()
.WithWaitCancellation(source.Token)) {
using (Stream ftpStream = response.GetResponseStream()) {
if (ftpStream == null) {
throw new InvalidOperationException("No response stream");
}
using (var dataStream = new MemoryStream()) {
await ftpStream.CopyToAsync(dataStream, 4096, source.Token)
.WithWaitCancellation(source.Token);
return dataStream.ToArray().ToImmutableList();
}
}
}
} catch (OperationCanceledException) {
throw new WebException(
String.Format("Operation timed out after {0} seconds.", timeout.TotalSeconds),
WebExceptionStatus.Timeout);
} finally {
ftpWebRequest.Abort();
}
}
}
}
public static class TaskCancellationExtensions {
/// http://stackoverflow.com/a/14524565/1512
public static async Task<T> WithWaitCancellation<T>(
this Task<T> task,
CancellationToken cancellationToken) {
// The task completion source.
var tcs = new TaskCompletionSource<Boolean>();
// Register with the cancellation token.
using (cancellationToken.Register(
s => ((TaskCompletionSource<Boolean>)s).TrySetResult(true),
tcs)) {
// If the task waited on is the cancellation token...
if (task != await Task.WhenAny(task, tcs.Task)) {
throw new OperationCanceledException(cancellationToken);
}
}
// Wait for one or the other to complete.
return await task;
}
/// http://stackoverflow.com/a/14524565/1512
public static async Task WithWaitCancellation(
this Task task,
CancellationToken cancellationToken) {
// The task completion source.
var tcs = new TaskCompletionSource<Boolean>();
// Register with the cancellation token.
using (cancellationToken.Register(
s => ((TaskCompletionSource<Boolean>)s).TrySetResult(true),
tcs)) {
// If the task waited on is the cancellation token...
if (task != await Task.WhenAny(task, tcs.Task)) {
throw new OperationCanceledException(cancellationToken);
}
}
// Wait for one or the other to complete.
await task;
}
}

This reminds me of old(?) IE behaviour of no reload of pages even when the network came back after N unsuccessful tries.
You should try setting the FtpWebRequest's cache policy to BypassCache.
HttpRequestCachePolicy bypassPolicy = new HttpRequestCachePolicy(
HttpRequestCacheLevel.BypassCache
);
ftpWebRequest.CachePolicy = bypassPolicy;
after setting KeepAlive.

I had the same issue, when trying to connect to an ftps server without the EnableSsl = true. The connection would fail twice, Wireshark shows the RST command, and then no more requests would leave the network resulting in the timeout exception, even after setting the EnableSsl = true.
I found setting the ConnectionGroupName allows the connection to reset and use a new port.
eg:
request.ConnectionGroupName = Guid.NewGuid();
Beware of port exhaustion using this method however, see https://learn.microsoft.com/en-us/troubleshoot/dotnet/framework/ports-run-out-use-connectiongroupname

file writing using blockingcollection

I have a tcp listener which listens and writes data from the server. I used a BlockingCollection to store data. Here I don't know when the file ends. So, my filestream is always open.
Part of my code is:
private static BlockingCollection<string> Buffer = new BlockingCollection<string>();
Process()
{
var consumer = Task.Factory.StartNew(() =>WriteData());
while()
{
string request = await reader.ReadLineAsync();
Buffer.Add(request);
}
}
WriteData()
{
FileStream fStream = new FileStream(filename,FileMode.Append,FileAccess.Write,FileShare.Write, 16392);
foreach(var val in Buffer.GetConsumingEnumerable(token))
{
fStream.Write(Encoding.UTF8.GetBytes(val), 0, val.Length);
fStream.Flush();
}
}
The problem is I cannot dispose filestream within loop otherwise I have to create filestream for each line and the loop may never end.

This would be much easier in .NET 4.5 if you used a DataFlow ActionBlock. An ActionBlock accepts and buffers incoming messages and processes them asynchronously using one or more Tasks.
You could write something like this:
public static async Task ProcessFile(string sourceFileName,string targetFileName)
{
//Pass the target stream as part of the message to avoid globals
var block = new ActionBlock<Tuple<string, FileStream>>(async tuple =>
{
var line = tuple.Item1;
var stream = tuple.Item2;
await stream.WriteAsync(Encoding.UTF8.GetBytes(line), 0, line.Length);
});
//Post lines to block
using (var targetStream = new FileStream(targetFileName, FileMode.Append,
FileAccess.Write, FileShare.Write, 16392))
{
using (var sourceStream = File.OpenRead(sourceFileName))
{
await PostLines(sourceStream, targetStream, block);
}
//Tell the block we are done
block.Complete();
//And wait fo it to finish
await block.Completion;
}
}
private static async Task PostLines(FileStream sourceStream, FileStream targetStream,
ActionBlock<Tuple<string, FileStream>> block)
{
using (var reader = new StreamReader(sourceStream))
{
while (true)
{
var line = await reader.ReadLineAsync();
if (line == null)
break;
var tuple = Tuple.Create(line, targetStream);
block.Post(tuple);
}
}
}
Most of the code deals with reading each line and posting it to the block. By default, an ActionBlock uses only a single Task to process one message at a time, which is fine in this scenario. More tasks can be used if needed to process data in parallel.
Once all lines are read, we notify the block with a call to Complete and await for it to finish processing with await block.Completion.
Once the block's Completion task finishes we can close the target stream.
The beauty of the DataFlow library is that you can link multiple blocks together, to create a pipeline of processing steps. ActionBlock is typically the final step in such a chain. The library takes care to pass data from one block to the next and propagate completion down the chain.
For example, one step can read files from a log, a second can parse them with a regex to find specific patterns (eg error messages) and pass them on, a third can receive the error messages and write them to another file. Each step will execute on a different thread, with intermediate messages buffered at each step.