Develop Reference

How to pass on the content of a HttpInputStream and send it to another Rest Api in c# .net

I have a REST API Web Service that acts as middleware that redirects calls onto another REST API Service.
For example in this case I upload a file on the webpage. And this Web Service receives the files and sends them to another API that actually processes the files.
It is something like this:
public async Task<IHttpActionResult> ExecuteFileUpload()
{
IHttpActionResult res;
try
{
var httpRequest = HttpContext.Current.Request;
var requestedFiles = new List<System.IO.Stream>();
var url = "http://localhost:2288" + "/api/v1/templates/upload";
if (httpRequest.Files.Count > 0)
{
HttpFileCollection files = httpRequest.Files;
using (var content = new MultipartFormDataContent())
{
int index = 0;
foreach (var file in httpRequest.Files)
{
var postedFile = httpRequest.Files[index];
var fileName = postedFile.FileName;
var fileInMemory = postedFile.InputStream;
content.Add(new StreamContent(fileInMemory), "f" + index, fileName);
index++;
}
res = await ForwardPost(url, content);
}
}
else
res = BadRequest();
}
catch (Exception ex)
{
res = InternalServerError(ex);
}
return res;
}
The forward post function is simple just like this:
protected async Task<IHttpActionResult> ForwardPost(string url, MultipartFormDataContent forwardContent)
{
IHttpActionResult res;
using (var client = CreateClient())
{
using (var response = await client.PostAsync(url, forwardContent))
{
if (response.StatusCode == HttpStatusCode.OK)
{
var content = await response.Content.ReadAsAsync<JToken>();
res = Ok(content);
}
else
{
res = InternalServerError();
}
}
return res;
}
}
As you can see I just want to forward whatever is passed to me from the webpage and forward it to the actual REST API that handles this.
However it throws an exception on this line:
response = await client.PostAsync(url, forwardContent)
It throws a System.IO.IOException
Cannot close stream until all bytes are written.
Why is this the case?
Is there a way to solve this problem?

The using in the ForwardPost function will dispose the forwardedContent from the calling method. This dispose will attempt to dispose the "postedFile.InputStream" reference from the request object of the origin method. This is likely deeply tied to the httprequest object.
using (var response = await client.PostAsync(url, forwardContent))
The solution is to copy the postedFile.InputStream to a new memorystream such that it can be disposed separately.

Return response to caller when one of the async call is done

I need to save file in two different server using .net api.Its asp.net Core web api. I want to return the response to the caller when saving the file in any of the server is succeeded. Could this be achieved using async programming? Can we return the response from api and let it save at other location?
For now I am using Parallel option:
var imageName = $"{Guid.NewGuid().ToString()}.{extension}";
var locations = new ConcurrentDictionary<string, string>();
Parallel.ForEach(destinationFolders, folder =>
{
try
{
var fullName = $#"{folder.Value}\{imageName}";
if (!Directory.Exists(folder.Value))
Directory.CreateDirectory(folder.Value);
var bytes = Convert.FromBase64String(content);
using (var imageFile = new FileStream(fullName, FileMode.Create))
{
imageFile.Write(bytes, 0, bytes.Length);
imageFile.Flush();
}
locations.TryAdd(folder.Key, Regex.Replace(folder.Value, #"\\", #"\"));
}
catch (Exception ex)
{
Logging.Log.Error($"{Constants.ExceptionOccurred} : {ex.Message} against {folder.Key} Value : {Regex.Replace(folder.Value, #"\\", #"\")}");
}
});
However the problem here is I am making it to wait while it finishes saving in both the location and return the response.

You do not appear to need Task Parallel for this. Consider the following truly asynchronous approach. Also note, you only need to do the Base64 decode once.
public Task WriteFiles(IEnumerable<string> destinationFolders, string content)
{
byte[] bytes = Convert.FromBase64String(content);
IList<Task> tasks = new List<Task>();
foreach (var folder in destinationFolders)
{
// <snipped>
Func<Task> taskFactory = async () =>
{
using (var imageFile = new FileStream(fullName, FileMode.Create))
{
await imageFile.WriteAsync(bytes, 0, bytes.Length);
await imageFile.FlushAsync();
}
};
tasks.Add(taskFactory());
// <snipped>
}
return Task.WhenAny(tasks);
}

Windows Store Apps, download multiple files with threads

I have this following method that I use to download a file's content:
public async Task<String> DownloadFileService(String filePath, string id)
{
string resposta = string.Empty;
try
{
using (var httpClient = new HttpClient { BaseAddress = Constants.baseAddress })
{
string token = App.Current.Resources["token"] as string;
httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
string fname = Path.GetFileName(filePath);
string path = Path.GetDirectoryName(filePath);
path = path.Replace(fname, "");
StorageFolder folder = await ApplicationData.Current.LocalFolder.CreateFolderAsync(Constants.DataDirectory + "\\" + path, CreationCollisionOption.OpenIfExists);
StorageFile imgFile = await folder.CreateFileAsync(fname, CreationCollisionOption.ReplaceExisting);
using (var response2 = await httpClient.GetAsync("file?fileId=" + id))
{
Stream imageStream = await response2.Content.ReadAsStreamAsync();
byte[] bytes = new byte[imageStream.Length];
imageStream.Read(bytes, 0, (int)imageStream.Length);
await FileIO.WriteBytesAsync(imgFile, bytes);
resposta = Convert.ToBase64String(bytes);
}
}
return resposta;
}
catch (Exception e)
{
Debug.WriteLine(e.Message);
}
}
I would like to know how I can call this multiple times to download several files at same time and wait until all files are downloaded, then do other stuff.
EDIT
After this suggestion I tried creating the following method:
public async void checkFilesExist(JsonArray array, string path)
{
List<Document> list = new List<Document>();
ObjectsService obj = new ObjectsService();
List<Task> ts = new List<Task>();
foreach (var item in array)
{
JsonObject newDoc;
JsonObject.TryParse(item.Stringify(), out newDoc);
if (newDoc.ContainsKey("libraryType") || !newDoc.ContainsKey("fileName"))
continue;
string name = newDoc["fileName"].GetString();
string id = newDoc["_id"].GetString();
File file = new File(name);
file.id = id;
Document doc = file;
doc.Parent = Document.FromPath(path);
path = path.Replace("/", "\\");
StorageFolder folder = await ApplicationData.Current.LocalFolder.CreateFolderAsync(Constants.DataDirectory + "\\" + path, CreationCollisionOption.OpenIfExists);
try
{
await folder.GetFileAsync(file.Name);
}
catch (Exception e)
{
list.Add(doc);
Task x = obj.DownloadFileService(doc.GetFullPath(), file.id);
ts.Add(x);
Debug.WriteLine(" Ex: " + e.Message);
}
}
try
{
Task.WaitAll(ts.ToArray());
Debug.WriteLine("AFTER THrEADS");
}
catch (Exception e)
{
Debug.WriteLine("Ex2: " + e.Message);
}
}
What this does is, with a response in json I get from a webservice listing some files, I check if they already exist in localfolder.
If they don't I call the method I had at start of the question.
I then have a list of tasks, and I add the call of the DownloadFileService() as a new task in the list, after that I do the Task.WaitAll() to wait for the downloads to finish.
Using fiddler I see the downloads all start, but for some reason my code doesn't stop at Task.WaitAll(), it just keeps going and it starts to use the files that are still being downloaded, creating a bunch of problems :D

you can use Task.WaitAll. It waits for all of the provided Task objects to complete execution.
var t1 = DownloadFileService("file1", "1");
var t2 = DownloadFileService("file2", "2");
var t3 = DownloadFileService("file3", "3");
Tasks.WaitAll(t1, t2, t3);

Or you can use :
await DownloadFileService("Path", "id");
await DownloadFileService("Path", "id");
await DownloadFileService("Path", "id");
await DownloadFileService("Path", "id");

Error checking when using HttpClient for asynchronous file downloads

This question is a followup to Threading issues when using HttpClient for asynchronous file downloads.
To get a file transfer to complete asynchronously using HttpClient, you need to add HttpCompletionOption.ResponseHeadersRead to the SendAsync request. Thus, when that call completes, you will be able to determine that all was well with the request and the response headers by adding a call to EnsureSuccessStatusCode. However the data is possibly still being transferred at this point.
How can you detect errors which happen after the headers are returned but before the data transfer is complete? How would said errors manifest themselves?
Some example code follows, with the point of the question marked at line 109)with the comment: "// *****WANT TO DO MORE ERROR CHECKING HERE**"
using System;
using System.Collections.Generic;
using System.IO;
using System.Net.Http;
using System.Threading.Tasks;
namespace TestHttpClient2
{
class Program
{
/* Use Yahoo portal to access quotes for stocks - perform asynchronous operations. */
static string baseUrl = "http://real-chart.finance.yahoo.com/";
static string requestUrlFormat = "/table.csv?s={0}&d=0&e=1&f=2016&g=d&a=0&b=1&c=1901&ignore=.csv";
static void Main(string[] args)
{
var activeTaskList = new List<Task>();
string outputDirectory = "StockQuotes";
if (!Directory.Exists(outputDirectory))
{
Directory.CreateDirectory(outputDirectory);
}
while (true)
{
Console.WriteLine("Enter symbol or [ENTER] to exit:");
string symbol = Console.ReadLine();
if (string.IsNullOrEmpty(symbol))
{
break;
}
Task downloadTask = DownloadDataForStockAsync(outputDirectory, symbol);
if (TaskIsActive(downloadTask))
{
// This is an asynchronous world - lock the list before updating it!
lock (activeTaskList)
{
activeTaskList.Add(downloadTask);
}
}
else
{
Console.WriteLine("task completed already?!??!?");
}
CleanupTasks(activeTaskList);
}
Console.WriteLine("Cleaning up");
while (CleanupTasks(activeTaskList))
{
Task.Delay(1).Wait();
}
}
private static bool CleanupTasks(List<Task> activeTaskList)
{
// reverse loop to allow list item deletions
// This is an asynchronous world - lock the list before updating it!
lock (activeTaskList)
{
for (int i = activeTaskList.Count - 1; i >= 0; i--)
{
if (!TaskIsActive(activeTaskList[i]))
{
activeTaskList.RemoveAt(i);
}
}
return activeTaskList.Count > 0;
}
}
private static bool TaskIsActive(Task task)
{
return task != null
&& task.Status != TaskStatus.Canceled
&& task.Status != TaskStatus.Faulted
&& task.Status != TaskStatus.RanToCompletion;
}
static async Task DownloadDataForStockAsync(string outputDirectory, string symbol)
{
try
{
using (var client = new HttpClient())
{
client.BaseAddress = new Uri(baseUrl);
client.Timeout = TimeSpan.FromMinutes(5);
string requestUrl = string.Format(requestUrlFormat, symbol);
var request = new HttpRequestMessage(HttpMethod.Post, requestUrl);
var response = await client.SendAsync(request,
HttpCompletionOption.ResponseHeadersRead);
response.EnsureSuccessStatusCode();
using (var httpStream = await response.Content.ReadAsStreamAsync())
{
var timestampedName = FormatTimestampedString(symbol, true);
var filePath = Path.Combine(outputDirectory, timestampedName + ".csv");
using (var fileStream = File.Create(filePath))
{
await httpStream.CopyToAsync(fileStream);
}
}
// *****WANT TO DO MORE ERROR CHECKING HERE*****
}
}
catch (HttpRequestException ex)
{
Console.WriteLine("Exception on thread: {0}: {1}\r\n",
System.Threading.Thread.CurrentThread.ManagedThreadId,
ex.Message,
ex.StackTrace);
}
catch (Exception ex)
{
Console.WriteLine("Exception on thread: {0}: {1}\r\n",
System.Threading.Thread.CurrentThread.ManagedThreadId,
ex.Message,
ex.StackTrace);
}
}
static volatile string lastTimestampedString = string.Empty;
static volatile string dummy = string.Empty;
static string FormatTimestampedString(string message, bool uniquify = false)
{
// This is an asynchronous world - lock the shared resource before using it!
lock (dummy)
//lock (lastTimestampedString)
{
Console.WriteLine("IN - Thread: {0:D2} lastTimestampedString: {1}",
System.Threading.Thread.CurrentThread.ManagedThreadId,
lastTimestampedString);
string newTimestampedString;
while (true)
{
DateTime lastDateTime = DateTime.Now;
newTimestampedString = string.Format(
"{1:D4}_{2:D2}_{3:D2}_{4:D2}_{5:D2}_{6:D2}_{7:D3}_{0}",
message,
lastDateTime.Year, lastDateTime.Month, lastDateTime.Day,
lastDateTime.Hour, lastDateTime.Minute, lastDateTime.Second,
lastDateTime.Millisecond
);
if (!uniquify)
{
break;
}
if (newTimestampedString != lastTimestampedString)
{
break;
}
//Task.Delay(1).Wait();
};
lastTimestampedString = newTimestampedString;
Console.WriteLine("OUT - Thread: {0:D2} lastTimestampedString: {1}",
System.Threading.Thread.CurrentThread.ManagedThreadId,
lastTimestampedString);
return lastTimestampedString;
}
}
}
}

I have copied and slightly cleaned up the relevant code.
var request = new HttpRequestMessage(HttpMethod.Post, requestUrl);
var response = await client.SendAsync(request,
HttpCompletionOption.ResponseHeadersRead);
response.EnsureSuccessStatusCode();
using (var httpStream = await response.Content.ReadAsStreamAsync())
{
var timestampedName = FormatTimestampedString(symbol, true);
var filePath = Path.Combine(outputDirectory, timestampedName + ".csv");
using (var fileStream = File.Create(filePath))
{
await httpStream.CopyToAsync(fileStream);
}
}
The question is, what if something goes wrong during reading the stream and copying it into your file?
All logical errors have already been addressed as part of the HTTP request and response cycle: the server has received your request, it has decided it is valid, it has responded with success (header portion of response), and it is now sending you the result (body portion of response).
The only errors that could occur now are things like the server crashing, the connection being lost, etc. My understanding is that these will manifest as HttpRequestException, meaning you can write code like this:
try
{
using (var httpStream = await response.Content.ReadAsStreamAsync())
{
var timestampedName = FormatTimestampedString(symbol, true);
var filePath = Path.Combine(outputDirectory, timestampedName + ".csv");
using (var fileStream = File.Create(filePath))
{
await httpStream.CopyToAsync(fileStream);
}
}
}
catch (HttpRequestException e)
{
...
}
The documenation doesn't say much, unfortunately. The reference source doesn't either. So your best bet is to start with this and maybe log all exceptions that are not HttpRequestException in case there is another exception type that could be thrown during the download of the response body.

If you want to narrow it down to the part which is between the header read and the content read, you actually leave yourself with the asynchronous buffer read:
var httpStream = await response.Content.ReadAsStreamAsync();
If you look whats going on inside the method, you'll see:
public Task<Stream> ReadAsStreamAsync()
{
this.CheckDisposed();
TaskCompletionSource<Stream> tcs = new TaskCompletionSource<Stream>();
if (this.contentReadStream == null && this.IsBuffered)
{
this.contentReadStream = new MemoryStream(this.bufferedContent.GetBuffer(),
0, (int)this.bufferedContent.Length,
false, false);
}
if (this.contentReadStream != null)
{
tcs.TrySetResult(this.contentReadStream);
return tcs.Task;
}
this.CreateContentReadStreamAsync().ContinueWithStandard(delegate(Task<Stream> task)
{
if (!HttpUtilities.HandleFaultsAndCancelation<Stream>(task, tcs))
{
this.contentReadStream = task.Result;
tcs.TrySetResult(this.contentReadStream);
}
});
return tcs.Task;
}
CreateContentReadStreamAsync is the one doing all the reading, internally, it will call LoadIntoBufferAsync, which you can find here.
Basically, you can see the it encapsulates IOException and ObjectDisposedException, or an ArgumentOutOfRangeException is the buffer is larger than 2GB (although i think that will be highly rare).

Can't get Task.Result to return string unless I add a Thread.Sleep

First time asking a question here I'll try to be as specific as possible.
I am working with a WebBased Api at work and I am having a difficult time getting the Task to return a result.
I have a trade class as such with a Buy method and a GetOrderNumber method. On my MainForm I am using a Task to execute the Trade and then retrieve the ordernumber associated with that trade.
NewTrade Test = new NewTrade();
Task<string> tmp = Task.Factory.StartNew(() => Test.Buy("JPM.NY", 100, 59.50));
string reqid = tmp.Result;
Task<string> tmp2 = Task.Factory.StartNew(() => Test.GetOrderNumber(reqid));
Console.WriteLine(tmp2.Result);
The First Task returns a REQID number. The second Task uses the REQID number and returns the actual ORDER NUMBER for the Trade Placed.
The problem I'm running into is I get the ReqID number from the first task but I return blank on the second task.
IF HOWEVER I add a Thread.sleep() between the first and second task I get the ordernumbers.
I'm using Tasks becasue eventually there will be hundreds of orders being placed it's critical to have the ordernumber for every order.
class NewTrade
{
public string Buy(string stkName, int stkShares, double limitPrice)
{
try
{
using (MyWebClient client = new MyWebClient())
{
Stream data = client.OpenRead("http://localhost:8080/ExecuteOrder?symbol=" + stkName + "&limitprice=" + limitPrice + "&ordername=ARCA%20Buy%20ARCX%20Limit%20DAY&shares=" + stkShares);
StreamReader reader = new StreamReader(data);
string s = reader.ReadToEnd();
s = getBetween(s, "<Content>", "</Content>"); //this is a util function to parse the result
// MessageBox.Show(s.ToString());
data.Close();
reader.Close();
return s;
}
}
catch (WebException e)
{
Console.WriteLine(e.Message);
Console.WriteLine(e.StackTrace);
return "Nothing";
}
}
public string GetOrderNumber(string id)
{
string s = "";
using (MyWebClient client = new MyWebClient())
{
try
{
WebRequest request = WebRequest.Create("http://localhost:8080/GetOrderNumber?requestid=" + id);
WebResponse response = request.GetResponse();
var reader = new StreamReader(response.GetResponseStream());
s = reader.ReadToEnd();
Console.WriteLine(s);
return s;
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
return "";
}
}
}
}

Using Tasks can make your application more scalable as you are enabling threads to handle other tasks while the asynchronous task is in progress. That being said, using Task.Run or Task.Factory.StartNew to do IO tasks in the background defeats that purpose. You need to make your methods truly async by calling the async API of the IO constructs you are using. Try changing your code to something like this:
public Task<string> MakeTradeAsync()
{
NewTrade Test = new NewTrade();
string reqid = await Test.Buy("JPM.NY", 100, 59.50);
string orderNumber = await Test.GetOrderNumber(reqid);
Console.WriteLine(orderNumber);
return orderNumber;
}
That will require you to make Test.Buy and Test.GetOrderNumber async as well. Which you can do by using async APIs to access your server data.
public async Task<string> Buy(string stkName, int stkShares, double limitPrice)
{
try
{
using (MyWebClient client = new MyWebClient())
{
// use async API here to get the data, assuming existance of OpenReadAsync
Stream data = await client.OpenReadAsync("http://localhost:8080/ExecuteOrder?symbol=" + stkName + "&limitprice=" + limitPrice + "&ordername=ARCA%20Buy%20ARCX%20Limit%20DAY&shares=" + stkShares);
StreamReader reader = new StreamReader(data);
string s = reader.ReadToEnd();
s = getBetween(s, "<Content>", "</Content>"); //this is a util function to parse the result
// MessageBox.Show(s.ToString());
data.Close();
reader.Close();
return s;
}
}
catch (WebException e)
{
Console.WriteLine(e.Message);
Console.WriteLine(e.StackTrace);
return "Nothing";
}
}
public async Task<string> GetOrderNumber(string id)
{
string s = "";
using (MyWebClient client = new MyWebClient())
{
try
{
WebRequest request = WebRequest.Create("http://localhost:8080/GetOrderNumber?requestid=" + id);
WebResponse response = await request.GetResponseAsync();
var reader = new StreamReader(response.GetResponseStream());
s = await reader.ReadToEndAsync();
Console.WriteLine(s);
return s;
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
return "";
}
}
}