I was using a method like below on the previous versions of WCF Web API:
// grab the posted stream
Stream stream = request.Content.ContentReadStream;
// write it to
using (FileStream fileStream = File.Create(fullFileName, (int)stream.Length)) {
byte[] bytesInStream = new byte[stream.Length];
stream.Read(bytesInStream, 0, (int)bytesInStream.Length);
fileStream.Write(bytesInStream, 0, bytesInStream.Length);
}
But on the preview 6, HttpRequestMessage.Content.ContentReadStream property is gone. I believe that it now should look like this one:
// grab the posted stream
System.Threading.Tasks.Task<Stream> stream = request.Content.ReadAsStreamAsync();
But I couldn't figure out what the rest of the code should be like inside the using statement. Can anyone provide me a way of doing it?
You might have to adjust this depending on what code is happening before/after, and there's no error handling, but something like this:
Task task = request.Content.ReadAsStreamAsync().ContinueWith(t =>
{
var stream = t.Result;
using (FileStream fileStream = File.Create(fullFileName, (int) stream.Length))
{
byte[] bytesInStream = new byte[stream.Length];
stream.Read(bytesInStream, 0, (int) bytesInStream.Length);
fileStream.Write(bytesInStream, 0, bytesInStream.Length);
}
});
If, later in your code, you need to ensure that this has completed, you can call task.Wait() and it will block until this has completed (or thrown an exception).
I highly recommend Stephen Toub's Patterns of Parallel Programming to get up to speed on some of the new async patterns (tasks, data parallelism etc) in .NET 4.
Quick and dirty fix:
// grab the posted stream
Task<Stream> streamTask = request.Content.ReadAsStreamAsync();
Stream stream = streamTask.Result; //blocks until Task is completed
Be aware that the fact that the sync version has been removed from the API suggests that you should really be attempting to learn the new async paradigms to avoid gobbling up many threads under high load.
You could for instance:
streamTask.ContinueWith( _ => {
var stream = streamTask.Result; //result already available, so no blocking
//work with stream here
} )
or with new async await features:
//async wait until task is complete
var stream = await request.Content.ReadAsStreamAsync();
Take time to learn async/await. It's pretty handy.
Here is how you can do this better with async and await:
private async void WhatEverMethod()
{
var stream = await response.Content.ReadAsStreamAsync();
using (FileStream fileStream = File.Create(fullFileName, (int)stream.Length))
{
byte[] bytesInStream = new byte[stream.Length];
stream.Read(bytesInStream, 0, (int)bytesInStream.Length);
fileStream.Write(bytesInStream, 0, bytesInStream.Length);
}
});
Related
I have a three async function that I want to call from multiple threads in parallel at the same time. Till now I have tried the following approach -
int numOfThreads = 4;
var taskList = List<Task>();
using(fs = new FileStream(inputFilePath, FileMode.OpenOrCreate,FileAccess.ReadWrite,FileShare.ReadWrite))
{
for(int i=1; i<= numOfThreads ; i++)
{
taskList.Add(Task.Run( async() => {
byte[] buffer = new byte[length]; // length could be upto a few thousand
await Function1Async(); // Reads from the file into a byte array
long result = await Function2Aync(); // Does some async operation with that byte array data
await Function3Async(result); // Writes the result into the file
}
}
}
Task.WaitAll(taskList.toArray());
However, not all of the tasks complete before the execution reaches an end. I have limited experience with threading in c#. What am I doing wrong in my code? Or should I take an alternative approach?
EDIT -
So I made some changes to my approach. I got rid of the Function3Async for now -
for(int i=1;i<=numOfThreads; i++)
{
using(fs = new FileStream(----))
{
taskList.Add(Task.Run( async() => {
byte[] buffer = new byte[length]; // length could be upto a few thousand
await Function1Async(buffer); // Reads from the file into a byte array
Stream data = new MemoryStream(buffer);
/** Write the Stream into a file and return
* the offset at which the write operation was done
*/
long blockStartOffset = await Function2Aync(data);
Console.WriteLine($"Block written at - {blockStartOffset}");
}
}
}
Task.WaitAll(taskList.toArray());
Now all threads seem to proceed to completion but the Function2Async seems to randomly write some Japanese characters to the output file. I guess it is some threading issue perhaps?
Here is the implementation of the Function2Async ->
public async Task<long> Function2Async(Stream data)
{
long offset = getBlockOffset();
using(var outputFs = new FileStream(fileName,
FileMode.OpenOrCreate,
FileAccess.ReadWrite,
FileShare.ReadWrite))
{
outputFs.Seek(offset, SeekOrigin.Begin);
await data.CopyToAsync(outputFs);
}
return offset;
}
In your example you have passed neither fs nor buffer into Function1Async but your comment says it reads from fs into buffer, so I will assume that is what happens.
You cannot read from a stream in parallel. It does not support that. If you find one that supports it, it will be horribly inefficient, because that is how hard disk storage works. Even worse if it is a network drive.
Read from the stream into your buffers first and in sequence, then let your threads loose and run your logic. In parallel, on the already existing buffers in memory.
Writing by the way would have the same problem if you wrote to the same file. If you write to one file per buffer, that's fine, otherwise, do it sequentially.
I'm using .net 4.0 and have following code:
var stream = new FileStream(filename, FileMode.Open, FileAccess.Read, FileShare.Read, bufferSize,
FileOptions.Asynchronous | FileOptions.SequentialScan);
var buffer = new byte[bufferSize];
Debug.Assert(stream.IsAsync, "stream.IsAsync");
var ia = stream.BeginRead(buffer, 0, buffer.Length, t =>
{
var ms = new MemoryStream(buffer);
using (TextReader rdr = new StreamReader(ms, Encoding.ASCII))
{
for (uint iEpoch = 0; item < FileHeader.NUMBER_OF_ITEMS; item++)
{
dataList.Add(epochData);
}
}
}, null);
return Task<int>.Factory.FromAsync(ia, t =>
{
var st = stream;
var bytes1 = st.EndRead(t);
var a = EpochDataList.Count;
var b = FileHeader.NUMBER_OF_EPOCHS;
Debug.Assert(a == b);
st.Dispose();
return bytes1;
});
And it seems that there are race conditions between execution of async callback and end method lambda function(assert is raising). But according to msdn it is explicitly stated that end method should be executing after async callback is finished:
Creates a Task that executes an end method function when a specified IAsyncResult completes.
Am I right that I'm confusing fact of completion of IO operation which triggering end method and fact of completion of async callback, so they both can potentially execute in the same time?
Meanwhile this code works great:
return Task<int>.Factory.FromAsync(stream.BeginRead, (ai) =>
{
var ms = new MemoryStream(buffer);
using (TextReader rdr = new StreamReader(ms, Encoding.ASCII))
{
using (TextReader rdr = new StreamReader(ms, Encoding.ASCII))
{
for (uint iEpoch = 0; item < FileHeader.NUMBER_OF_ITEMS; item++)
{
dataList.Add(epochData);
}
}
}
stream.Dispose();
return stream.EndRead(ai);
}, buffer, 0, buffer.Length, null);
Also I need to mention that returned task is used within continuation.
Thanks in advance.
You're doing this so wrong, I'm almost inclined not to answer - you're going to hurt someone with that code. But since this isn't Code Review...
Your most immediate problem is that the callback you provide to BeginRead isn't part of the IAsyncResult at all. Thus, when a specified IAsyncResult completes doesn't talk about your callback, it only talks about the underlying asynchronous operation - you get two separate callbacks launched by the same event.
Now, for the other problems:
You need to keep issuing BeginReads over and over again, until EndRead returns 0. Otherwise, you're only ever reading the whole buffer at most - if your file is longer than that, you're not going to read the whole file.
You're combining old-school asynchronous API callbacks with Task-based asynchrony. This is bound to give you trouble. Just learn to use Tasks properly, and you'll find the callbacks are 100% unnecessary.
EndRead is telling you how many bytes were actually read in the preceding BeginRead operation - you're ignoring that information.
Doing this correctly isn't all that easy - if possible, I'd suggest upgrading to .NET 4.5, and taking advantage of the await keyword. If that's not possible, you can install the async targetting pack, which adds await to 4.0 as a simple NuGet package.
With await, reading the whole file is as simple as
using (var sr = new StreamReader(fs))
{
string line;
while ((line = await sr.ReadLineAsync(buffer, 0, buffer.Length)) > 0)
{
// Do whatever
}
}
Currently I am implementing a way to report Progress with the HttpClient, since we share code with a .NET4 WPF and a Windows Universal App we use the Microsoft HTTP Client Libraries from NuGet. The idea was to wrap the target file stream in a CountingInputStream and report progress there:
public override void Write(byte[] buffer, int offset, int count)
{
_stream.Write(buffer, offset, count);
_bytesRead += count;
_progress.Report(_bytesRead);
if (_cancellationToken.IsCancellationRequested)
{
_cancellationToken.ThrowIfCancellationRequested();
}
}
Then I send my request with: HttpResponseMessage httpResponseMessage = AsyncHelpers.RunSync(() => _httpClient.SendAsync(httpRequestMessage, HttpCompletionOption.ResponseHeadersRead, cancellationToken));
After that I open the file stream and then copy the content stream. The response has correct headers:
Content-Length: 213334
Content-Type: application/octet-stream; charset=UTF-8
Content-Disposition: attachment; filename="Bondi Beach.jpg"; filename*=UTF-8''Bondi%20Beach.jpg
using(Stream fileStream = new CountingInputStream(storage.Open(downloadRequest.TargetPath, FileMode.Create), downloadRequest.Progress, cancellationToken )) {
await HttpHeaderResponseMessage.Content.CopyToAsync(fileStream);
}
The problem is that the StreamContent only starts writing to the file stream after the download has finished. When it started writing progress reporting just works fine.
I already tried different approaches like:
ReadAsStreamAsync and then copy the response stream to file stream
ReadAsStreamAsync manually read to buffer and then write to file stream
_httpClient = new System.Net.Http.HttpClient(){MaxResponseContentBufferSize = 4096};to restrict the BufferSize
Any ideas how I could force the ContentStream to write to the file stream while it is still downloading?
UPDATE:
Following Luaans advice I tried to override the WriteAsync and implemented a StreamContent Extensions method:
//CountingInputStream
public Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken)
{
_bytesRead += count;
_progress.Report(_bytesRead);
if (_cancellationToken.IsCancellationRequested)
{
_cancellationToken.ThrowIfCancellationRequested();
}
return _stream.WriteAsync(buffer, offset, count, cancellationToken);
}
//static Extensions Class
public static async Task CopyToAs(this StreamContent source, Stream targetStream)
{
int read;
byte[] buffer = new byte[4096];
using(Stream responseStream = await source.ReadAsStreamAsync()) {
while ((read = await responseStream.ReadAsync(buffer,0,buffer.Length))>0) {
await targetStream.WriteAsync(buffer, 0, read);
}
}
}
It still waits till the download is finish until it calls ReadAsync the first time. Any hints what I have done wrong?
The fact that ReadAsStreamAsync is, well, async, makes this rather suspicious. Why would you asynchronously wait to get the stream? You're supposed to read it asynchronously, but you should have the stream itself ready right away.
Reading the documentation makes this blatantly obvious:
This operation will not block. The returned task object will complete after the whole response (including content) is read.
However, there's overloads you can use to have it return after the headers have been read. This still means you need to wait for the server to process the request, before starting to get progress, but for the download itself, you are in luck.
Sample code:
var response =
await
(
new HttpClient()
.GetAsync("http://www.microsoft.com/", HttpCompletionOption.ResponseHeadersRead)
);
var stream = await response.Content.ReadAsStreamAsync();
var buffer = new byte[2048];
while (await stream.ReadAsync(buffer, 0, buffer.Length) > 0)
{
// Report progress and write to a different stream
}
EDIT:
It sounds like you should be using Windows.Web.Http.HttpClient instead of System.Net.Http.HttpClient:
async Task DownloadWithProgress()
{
var awaitable = httpClient.GetAsync(yourUrl)
awaitable.Progress = (res, progress) =>
{
// Report progress
}
await awaitable;
}
I trying to copy an incoming HTTP stream into another stream and use that instead.
When I use Request.Content.ReadAsStreamAsync().Result directly, everything works fine.
The problem that I am facing is how to "channel" the original stream thru another one.
I wrote the flowing method, but the problem is that it either blocks until all the stream data has arrived (when I use Wait()), or it returns immediately and exists without capturing any data.
private Task<Stream> GetAudioStream(RecordingSession recordingSession)
{
Task<Stream> task = Task<Stream>.Factory.StartNew(() =>
{
Request.Content.ReadAsStreamAsync().Result.CopyToAsync(recordingSession.AudioStream).Wait();
return recordingSession.AudioStream;
});
return task;
}
Perhaps, you tackle the problem from the wrong side. CopyToAsync gives you a complete copy of the stream, the Task returned by CopyToAsync is completed when all data has been copied.
If you don't need the whole complete copy of the stream, or don't want to wait for all data, then copy it manually yourself and process each chunk:
static async Task<Stream> CopyAndProcessAsync(RecordingSession recordingSession, CancellationToken token)
{
var srcStream = await Request.Content.ReadAsStreamAsync();
var dstStream = recordingSession.AudioStream;
var buffer = new byte[4096];
int bytesRead;
while ((bytesRead = await srcStream.ReadAsync(buffer, 0, buffer.Length, token).ConfigureAwait(false)) != 0)
{
await dstStream.WriteAsync(buffer, 0, bytesRead, token).ConfigureAwait(false);
// do whatever you want with the chunk
PlaybackChunk(buffer);
}
return dstStream;
}
Do this wherever you'd otherwise be consuming the result of your GetAudioStream. It well may turn out that you don't need a copy at all, when you've processed each chunk.
I think you mean something like this:
private async Task<Stream> GetAudioStream(RecordingSession recordingSession)
{
var result = await Request.Content.ReadAsStreamAsync();
await result.CopyToAsync(recordingSession.AudioStream);
return recordingSession.AudioStream;
}
async methods return Tasks. Very rarely should a non-async method return a Task.
I'm trying to read from a file asynchronously, and safely (minimum level of permissions sought). I'm using .NET 3.5 and cannot find a good example for this (all uses async and await).
public string GetLines()
{
var encoding = new UnicodeEncoding();
byte[] allText;
using (FileStream stream =File.Open(_path, FileMode.Open))
{
allText = new byte[stream.Length];
//something like this, but does not compile in .net 3.5
stream.ReadAsync(allText, 0, (int) allText.Length);
}
return encoding.GetString(allText);
}
Question is, how do I do this asynchronously in .net 3.5, wait till the operation is finished and send back all lines to the caller?
The caller can wait till the operation is complete, but the read has to happen in a background thread.
The caller is a UI thread, and I'm using .NET 3.5
There are several options, but the simplest would be to have this method accept a callback, and then call it when it has computed the given value. The caller than needs to pass in the callback method to process the results rather than blocking on the method call:
public static void GetLines(Action<string> callback)
{
var encoding = new UnicodeEncoding();
byte[] allText;
FileStream stream = File.Open(_path, FileMode.Open);
allText = new byte[stream.Length];
//something like this, but does not compile in .net 3.5
stream.ReadAsync(allText, 0, (int)allText.Length);
stream.BeginRead(allText, 0, allText.Length, result =>
{
callback(encoding.GetString(allText));
stream.Dispose();
}, null);
}
If you want to wait until the operation is complete, why do you need to do it asynchronously?
return File.ReadAllText(_path, new UnicodeEncoding());
Would do the trick
Maybe something like this:
GetLines(string path, ()=>
{
// here your code...
});
public void GetLines(string _path, Action<string> callback)
{
var result = string.Empty;
new Action(() =>
{
var encoding = new UnicodeEncoding();
byte[] allText;
using (FileStream stream = File.Open(_path, FileMode.Open))
{
allText = new byte[stream.Length];
//something like this, but does not compile in .net 3.5
stream.Read(allText, 0, (int)allText.Length);
}
result = encoding.GetString(allText);
}).BeginInvoke(x => callback(result), null);
}