Develop Reference

gRPC C#: how to cancel upload correctly

For uploading images from a client to the server I use chunking.
Here is the client code:
private async Task UploadPersonImage(int personId, string fileName, CancellationToken cancellationToken)
{
var stream = Client.UploadPersonImage();
PersonImageMessage personImageMessage = new PersonImageMessage();
personImageMessage.PersonId = personId;
personImageMessage.ImageType = ImageType.Jpg;
byte[] image = File.ReadAllBytes(fileName);
int imageOffset = 0;
byte[] imageChunk = new byte[imageChunkSize];
while (imageOffset < image.Length && !cancellationToken.IsCancellationRequested)
{
int length = Math.Min(imageChunkSize, image.Length - imageOffset);
Buffer.BlockCopy(image, imageOffset, imageChunk, 0, length);
imageOffset += length;
ByteString byteString = ByteString.CopyFrom(imageChunk);
personImageMessage.ImageChunk = byteString;
await stream.RequestStream.WriteAsync(personImageMessage).ConfigureAwait(false);
}
await stream.RequestStream.CompleteAsync().ConfigureAwait(false);
if (!cancellationToken.IsCancellationRequested)
{
var uploadPersonImageResult = await stream.ResponseAsync.ConfigureAwait(false);
// Process answer...
}
}
And this is the server code:
public override async Task<TransferStatusMessage> UploadPersonImage(
IAsyncStreamReader<PersonImageMessage> requestStream, ServerCallContext context)
{
TransferStatusMessage transferStatusMessage = new TransferStatusMessage();
transferStatusMessage.Status = TransferStatus.Success;
try
{
await Task.Run(
async () =>
{
CancellationToken cancellationToken = context.CancellationToken;
await using (Stream fs = File.OpenWrite(ImageFileName))
{
await foreach (PersonImageMessage personImageMessage in
requestStream.ReadAllAsync(cancellationToken).ConfigureAwait(false))
{
fs.Write(personImageMessage.ImageChunk.ToByteArray());
}
}
}).ConfigureAwait(false);
}
// Is thrown on cancellation -> ignore...
catch (OperationCanceledException)
{
transferStatusMessage.Status = TransferStatus.Cancelled;
}
catch (RpcException rpcEx)
{
if (rpcEx.StatusCode == StatusCode.Cancelled)
{
transferStatusMessage.Status = TransferStatus.Cancelled;
}
else
{
_logger.LogError($"Exception while processing image file '{ImageFileName}'. Exception: '{requestStream}'");
transferStatusMessage.Status = TransferStatus.Failure;
}
}
// Delete incomplete file
if (transferStatusMessage.Status != TransferStatus.Success)
{
File.Delete(ImageFileName);
}
return transferStatusMessage;
}
Everything works fine. I want to cancel the upload in between sending the chunks. Now, CompleteAsync() is called and the server thinks the data transfer ended successfully. I'm looking for a way to trigger the cancellation in the server (i.e. the CancellationToken in the ServerCallContext) via the client.
As a workaround I could add a flag to PersonImageMessage, something like 'upload_cancelled', to tell the server that the transfer is aborted. But there must be a built-in mechanism.
Does somebody know the trick?

You can pass cancellationToken to your Client stream RPC method. Then the server can receive it as context.cancellationToken.
var stream = Client.UploadPersonImage(cancellationToken: cancellationToken);
If the client cancels, stream.RequestStream.CompleteAsync() should not be called.

Return response to caller when one of the async call is done

I need to save file in two different server using .net api.Its asp.net Core web api. I want to return the response to the caller when saving the file in any of the server is succeeded. Could this be achieved using async programming? Can we return the response from api and let it save at other location?
For now I am using Parallel option:
var imageName = $"{Guid.NewGuid().ToString()}.{extension}";
var locations = new ConcurrentDictionary<string, string>();
Parallel.ForEach(destinationFolders, folder =>
{
try
{
var fullName = $#"{folder.Value}\{imageName}";
if (!Directory.Exists(folder.Value))
Directory.CreateDirectory(folder.Value);
var bytes = Convert.FromBase64String(content);
using (var imageFile = new FileStream(fullName, FileMode.Create))
{
imageFile.Write(bytes, 0, bytes.Length);
imageFile.Flush();
}
locations.TryAdd(folder.Key, Regex.Replace(folder.Value, #"\\", #"\"));
}
catch (Exception ex)
{
Logging.Log.Error($"{Constants.ExceptionOccurred} : {ex.Message} against {folder.Key} Value : {Regex.Replace(folder.Value, #"\\", #"\")}");
}
});
However the problem here is I am making it to wait while it finishes saving in both the location and return the response.

You do not appear to need Task Parallel for this. Consider the following truly asynchronous approach. Also note, you only need to do the Base64 decode once.
public Task WriteFiles(IEnumerable<string> destinationFolders, string content)
{
byte[] bytes = Convert.FromBase64String(content);
IList<Task> tasks = new List<Task>();
foreach (var folder in destinationFolders)
{
// <snipped>
Func<Task> taskFactory = async () =>
{
using (var imageFile = new FileStream(fullName, FileMode.Create))
{
await imageFile.WriteAsync(bytes, 0, bytes.Length);
await imageFile.FlushAsync();
}
};
tasks.Add(taskFactory());
// <snipped>
}
return Task.WhenAny(tasks);
}

ASP.NET Core with WebSockets - WebSocket handshake never occurs

I am very new to C# programming, having previously only worked with Java. This project I am building should be very straightforward - we have a web page with a selection of foreign currency pairs. The element chosen is sent to the server, which responds with a hardcoded value of their exchange rate. The requirement is that both actions are implemented through the use of WebSockets. Here is the JS code on my page:
var protocol;
var wsUri;
var socket;
window.onload = function(e) {
e.preventDefault();
protocol = location.protocol === "https:" ? "wss:" : "ws:";
wsUri = protocol + "//" + window.location.host;
socket = new WebSocket(wsUri);
socket.onopen = e => {
console.log("socket opened", e);
};
document.getElementById("currencypair").onchange = function()
{
var selector = document.getElementById("currencypair");
var text = selector.options[selector.selectedIndex].text;
socket.send(text);
};
socket.onmessage = function (evt) {
var receivedMessage = evt.data;
document.getElementById("output").html(receivedMessage);
};
};
Here is a snippet of the Startup.cs class Configure method:
app.UseWebSockets();
app.UseMiddleware<WebSocketMiddleware>();
And here is the middleware class to process requests.
public class WebSocketMiddleware
{
private readonly RequestDelegate _next;
public WebSocketMiddleware(RequestDelegate next)
{
_next = next;
}
public async Task Invoke(HttpContext context)
{
if (!context.WebSockets.IsWebSocketRequest)
{
await _next.Invoke(context);
return;
}
var ct = context.RequestAborted;
using (var socket = await context.WebSockets.AcceptWebSocketAsync())
{
while (true)
{
var stringReceived = await ReceiveStringAsync(socket, ct);
if (CurrencyPairCollection.CurrencyPairs.TryGetValue(stringReceived, out var value))
{
await SendStringAsync(socket, value.ToString(), ct);
}
else
{
throw new Exception("Unexpected value");
}
await Task.Delay(1000, ct);
}
}
}
private static async Task<string> ReceiveStringAsync(WebSocket socket, CancellationToken ct = default(CancellationToken))
{
var buffer = new ArraySegment<byte>();
using (var ms = new MemoryStream())
{
WebSocketReceiveResult result;
do
{
ct.ThrowIfCancellationRequested();
result = await socket.ReceiveAsync(buffer, ct);
ms.Write(buffer.Array, buffer.Offset, result.Count);
}
while (!result.EndOfMessage);
ms.Seek(0, SeekOrigin.Begin);
if (result.MessageType != WebSocketMessageType.Text || result.Count.Equals(0))
{
throw new Exception("Unexpected message");
}
using (var reader = new StreamReader(ms, Encoding.UTF8))
{
return await reader.ReadToEndAsync();
}
}
}
private static Task SendStringAsync(WebSocket socket, string data, CancellationToken ct = default(CancellationToken))
{
var segment = new ArraySegment<byte>(Encoding.UTF8.GetBytes(data));
return socket.SendAsync(segment, WebSocketMessageType.Text, true, ct);
}
}
Please mind I was working with the following example which contains mistakes listed by people in the comment section. I did my best to resolve them, however due to my limited experience, that may be where the fault lies.
https://www.softfluent.com/blog/dev/Using-Web-Sockets-with-ASP-NET-Core
Basically, upon running the app the browser console immediately reports this:
WebSocket connection to 'ws://localhost:51017/' failed: Error during WebSocket handshake: Unexpected response code: 200

I have been able to answer my own question. So in Startup.cs of which I provided only a snippet, a call to app.UseMvc() is made right before the lines I have already shared. This is generated by the default template. The trick was to move this call to below the following:
app.UseWebSockets();
app.UseMiddleware<WebSocketMiddleware>();
as otherwise the request pipeline is disrupted.
This will allow our socket to open, however without changing the following line in async Task ReceiveStringAsync(...)
var buffer = new ArraySegment<byte>();
to
var buffer = new ArraySegment<byte>(new byte[8192]);
it will still close prematurely. Next, just needed to correct JS syntax error. Changed
document.getElementById("output").html(receivedMessage);
to
document.getElementById("output").value = receivedMessage;
That's it, it works.

Parallel HTTP requests using System.Net.Http.HttpClient

I'm trying to figure out the correct way to parallelize HTTP requests using Task and async/await. I'm using the HttpClient class which already has async methods for retrieving data. If I just call it in a foreach loop and await the response, only one request gets sent at a time (which makes sense because during the await, control is returning to our event loop, not to the next iteration of the foreach loop).
My wrapper around HttpClient looks as such
public sealed class RestClient
{
private readonly HttpClient client;
public RestClient(string baseUrl)
{
var baseUri = new Uri(baseUrl);
client = new HttpClient
{
BaseAddress = baseUri
};
}
public async Task<Stream> GetResponseStreamAsync(string uri)
{
var resp = await GetResponseAsync(uri);
return await resp.Content.ReadAsStreamAsync();
}
public async Task<HttpResponseMessage> GetResponseAsync(string uri)
{
var resp = await client.GetAsync(uri);
if (!resp.IsSuccessStatusCode)
{
// ...
}
return resp;
}
public async Task<T> GetResponseObjectAsync<T>(string uri)
{
using (var responseStream = await GetResponseStreamAsync(uri))
using (var sr = new StreamReader(responseStream))
using (var jr = new JsonTextReader(sr))
{
var serializer = new JsonSerializer {NullValueHandling = NullValueHandling.Ignore};
return serializer.Deserialize<T>(jr);
}
}
public async Task<string> GetResponseString(string uri)
{
using (var resp = await GetResponseStreamAsync(uri))
using (var sr = new StreamReader(resp))
{
return sr.ReadToEnd();
}
}
}
And the code invoked by our event loop is
public async void DoWork(Action<bool> onComplete)
{
try
{
var restClient = new RestClient("https://example.com");
var ids = await restClient.GetResponseObjectAsync<IdListResponse>("/ids").Ids;
Log.Info("Downloading {0:D} items", ids.Count);
using (var fs = new FileStream(#"C:\test.json", FileMode.Create, FileAccess.Write, FileShare.Read))
using (var sw = new StreamWriter(fs))
{
sw.Write("[");
var first = true;
var numCompleted = 0;
foreach (var id in ids)
{
Log.Info("Downloading item {0:D}, completed {1:D}", id, numCompleted);
numCompleted += 1;
try
{
var str = await restClient.GetResponseString($"/info/{id}");
if (!first)
{
sw.Write(",");
}
sw.Write(str);
first = false;
}
catch (HttpException e)
{
if (e.StatusCode == HttpStatusCode.Forbidden)
{
Log.Warn(e.ResponseMessage);
}
else
{
throw;
}
}
}
sw.Write("]");
}
onComplete(true);
}
catch (Exception e)
{
Log.Error(e);
onComplete(false);
}
}
I've tried a handful of different approaches involving Parallel.ForEach, Linq.AsParallel, and wrapping the entire contents of the loop in a Task.

The basic idea is to keep of track of all the asynchronous tasks, and awaiting them at once. The simplest way to do this is to extract the body of your foreach to a separate asynchronous method, and do something like this:
var tasks = ids.Select(i => DoWorkAsync(i));
await Task.WhenAll(tasks);
This way, the individual tasks are issued separately (still in sequence, but without waiting for the I/O to complete), and you await them all at the same time.
Do note that you will also need to do some configuration - HTTP is throttled by default to only allow two simultaneous connections to the same server.

Error checking when using HttpClient for asynchronous file downloads

This question is a followup to Threading issues when using HttpClient for asynchronous file downloads.
To get a file transfer to complete asynchronously using HttpClient, you need to add HttpCompletionOption.ResponseHeadersRead to the SendAsync request. Thus, when that call completes, you will be able to determine that all was well with the request and the response headers by adding a call to EnsureSuccessStatusCode. However the data is possibly still being transferred at this point.
How can you detect errors which happen after the headers are returned but before the data transfer is complete? How would said errors manifest themselves?
Some example code follows, with the point of the question marked at line 109)with the comment: "// *****WANT TO DO MORE ERROR CHECKING HERE**"
using System;
using System.Collections.Generic;
using System.IO;
using System.Net.Http;
using System.Threading.Tasks;
namespace TestHttpClient2
{
class Program
{
/* Use Yahoo portal to access quotes for stocks - perform asynchronous operations. */
static string baseUrl = "http://real-chart.finance.yahoo.com/";
static string requestUrlFormat = "/table.csv?s={0}&d=0&e=1&f=2016&g=d&a=0&b=1&c=1901&ignore=.csv";
static void Main(string[] args)
{
var activeTaskList = new List<Task>();
string outputDirectory = "StockQuotes";
if (!Directory.Exists(outputDirectory))
{
Directory.CreateDirectory(outputDirectory);
}
while (true)
{
Console.WriteLine("Enter symbol or [ENTER] to exit:");
string symbol = Console.ReadLine();
if (string.IsNullOrEmpty(symbol))
{
break;
}
Task downloadTask = DownloadDataForStockAsync(outputDirectory, symbol);
if (TaskIsActive(downloadTask))
{
// This is an asynchronous world - lock the list before updating it!
lock (activeTaskList)
{
activeTaskList.Add(downloadTask);
}
}
else
{
Console.WriteLine("task completed already?!??!?");
}
CleanupTasks(activeTaskList);
}
Console.WriteLine("Cleaning up");
while (CleanupTasks(activeTaskList))
{
Task.Delay(1).Wait();
}
}
private static bool CleanupTasks(List<Task> activeTaskList)
{
// reverse loop to allow list item deletions
// This is an asynchronous world - lock the list before updating it!
lock (activeTaskList)
{
for (int i = activeTaskList.Count - 1; i >= 0; i--)
{
if (!TaskIsActive(activeTaskList[i]))
{
activeTaskList.RemoveAt(i);
}
}
return activeTaskList.Count > 0;
}
}
private static bool TaskIsActive(Task task)
{
return task != null
&& task.Status != TaskStatus.Canceled
&& task.Status != TaskStatus.Faulted
&& task.Status != TaskStatus.RanToCompletion;
}
static async Task DownloadDataForStockAsync(string outputDirectory, string symbol)
{
try
{
using (var client = new HttpClient())
{
client.BaseAddress = new Uri(baseUrl);
client.Timeout = TimeSpan.FromMinutes(5);
string requestUrl = string.Format(requestUrlFormat, symbol);
var request = new HttpRequestMessage(HttpMethod.Post, requestUrl);
var response = await client.SendAsync(request,
HttpCompletionOption.ResponseHeadersRead);
response.EnsureSuccessStatusCode();
using (var httpStream = await response.Content.ReadAsStreamAsync())
{
var timestampedName = FormatTimestampedString(symbol, true);
var filePath = Path.Combine(outputDirectory, timestampedName + ".csv");
using (var fileStream = File.Create(filePath))
{
await httpStream.CopyToAsync(fileStream);
}
}
// *****WANT TO DO MORE ERROR CHECKING HERE*****
}
}
catch (HttpRequestException ex)
{
Console.WriteLine("Exception on thread: {0}: {1}\r\n",
System.Threading.Thread.CurrentThread.ManagedThreadId,
ex.Message,
ex.StackTrace);
}
catch (Exception ex)
{
Console.WriteLine("Exception on thread: {0}: {1}\r\n",
System.Threading.Thread.CurrentThread.ManagedThreadId,
ex.Message,
ex.StackTrace);
}
}
static volatile string lastTimestampedString = string.Empty;
static volatile string dummy = string.Empty;
static string FormatTimestampedString(string message, bool uniquify = false)
{
// This is an asynchronous world - lock the shared resource before using it!
lock (dummy)
//lock (lastTimestampedString)
{
Console.WriteLine("IN - Thread: {0:D2} lastTimestampedString: {1}",
System.Threading.Thread.CurrentThread.ManagedThreadId,
lastTimestampedString);
string newTimestampedString;
while (true)
{
DateTime lastDateTime = DateTime.Now;
newTimestampedString = string.Format(
"{1:D4}_{2:D2}_{3:D2}_{4:D2}_{5:D2}_{6:D2}_{7:D3}_{0}",
message,
lastDateTime.Year, lastDateTime.Month, lastDateTime.Day,
lastDateTime.Hour, lastDateTime.Minute, lastDateTime.Second,
lastDateTime.Millisecond
);
if (!uniquify)
{
break;
}
if (newTimestampedString != lastTimestampedString)
{
break;
}
//Task.Delay(1).Wait();
};
lastTimestampedString = newTimestampedString;
Console.WriteLine("OUT - Thread: {0:D2} lastTimestampedString: {1}",
System.Threading.Thread.CurrentThread.ManagedThreadId,
lastTimestampedString);
return lastTimestampedString;
}
}
}
}

I have copied and slightly cleaned up the relevant code.
var request = new HttpRequestMessage(HttpMethod.Post, requestUrl);
var response = await client.SendAsync(request,
HttpCompletionOption.ResponseHeadersRead);
response.EnsureSuccessStatusCode();
using (var httpStream = await response.Content.ReadAsStreamAsync())
{
var timestampedName = FormatTimestampedString(symbol, true);
var filePath = Path.Combine(outputDirectory, timestampedName + ".csv");
using (var fileStream = File.Create(filePath))
{
await httpStream.CopyToAsync(fileStream);
}
}
The question is, what if something goes wrong during reading the stream and copying it into your file?
All logical errors have already been addressed as part of the HTTP request and response cycle: the server has received your request, it has decided it is valid, it has responded with success (header portion of response), and it is now sending you the result (body portion of response).
The only errors that could occur now are things like the server crashing, the connection being lost, etc. My understanding is that these will manifest as HttpRequestException, meaning you can write code like this:
try
{
using (var httpStream = await response.Content.ReadAsStreamAsync())
{
var timestampedName = FormatTimestampedString(symbol, true);
var filePath = Path.Combine(outputDirectory, timestampedName + ".csv");
using (var fileStream = File.Create(filePath))
{
await httpStream.CopyToAsync(fileStream);
}
}
}
catch (HttpRequestException e)
{
...
}
The documenation doesn't say much, unfortunately. The reference source doesn't either. So your best bet is to start with this and maybe log all exceptions that are not HttpRequestException in case there is another exception type that could be thrown during the download of the response body.

If you want to narrow it down to the part which is between the header read and the content read, you actually leave yourself with the asynchronous buffer read:
var httpStream = await response.Content.ReadAsStreamAsync();
If you look whats going on inside the method, you'll see:
public Task<Stream> ReadAsStreamAsync()
{
this.CheckDisposed();
TaskCompletionSource<Stream> tcs = new TaskCompletionSource<Stream>();
if (this.contentReadStream == null && this.IsBuffered)
{
this.contentReadStream = new MemoryStream(this.bufferedContent.GetBuffer(),
0, (int)this.bufferedContent.Length,
false, false);
}
if (this.contentReadStream != null)
{
tcs.TrySetResult(this.contentReadStream);
return tcs.Task;
}
this.CreateContentReadStreamAsync().ContinueWithStandard(delegate(Task<Stream> task)
{
if (!HttpUtilities.HandleFaultsAndCancelation<Stream>(task, tcs))
{
this.contentReadStream = task.Result;
tcs.TrySetResult(this.contentReadStream);
}
});
return tcs.Task;
}
CreateContentReadStreamAsync is the one doing all the reading, internally, it will call LoadIntoBufferAsync, which you can find here.
Basically, you can see the it encapsulates IOException and ObjectDisposedException, or an ArgumentOutOfRangeException is the buffer is larger than 2GB (although i think that will be highly rare).