I'm simply trying to have a basic SendAsync method write it's progress to the console, and even after a bunch of SO answers for GET, I'm not really seeing how this works for POST.
The file I'm testing with is ~ 100Mb and when it hits the SendAsync method, it sit until complete.
What might I be missing?
var request = new HttpRequestMessage
{
RequestUri = endpoint,
Method = httpMethod,
Content = data
};
var result = string.Empty;
using (var response = await _httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead))
using (var responseStream = await response.Content.ReadAsStreamAsync())
{
int read;
var offset = 0;
var responseBuffer = new byte[1024];
do
{
read = await responseStream.ReadAsync(responseBuffer, 0, responseBuffer.Length);
result += Encoding.UTF8.GetString(responseBuffer, 0, read);
offset += read;
Console.WriteLine($"Offset: {offset}");
} while (read != 0);
response.EnsureSuccessStatusCode();
}
return JsonConvert.DeserializeObject<T>(result);
Related
I am sending multiples files from my web api but I want to read each part of the stream to convert him into a byte array , then at the end I have a list of byte[], and I can save each files:
[Route("GetFiles")]
public HttpResponseMessage GetFile([FromUri] List<string> filesNames)
{
HttpResponseMessage response = new HttpResponseMessage(HttpStatusCode.BadRequest);
if (filesNames.Count == 0)
return Request.CreateResponse(HttpStatusCode.BadRequest);
var content = new MultipartContent();
filesNames.ForEach(delegate (string fileName)
{
string filePath = System.Web.Hosting.HostingEnvironment.MapPath("~/Uploads/" + fileName);
byte[] pdf = File.ReadAllBytes(filePath);
content.Add(new ByteArrayContent(pdf));
response.Headers.Add(fileName, fileName);
});
var files = JsonConvert.SerializeObject(content);
response.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment");
response.Content.Headers.ContentType = new MediaTypeHeaderValue("application/pdf");
response = Request.CreateResponse(HttpStatusCode.OK, content);
return response;
}
Here is how I get one file into a stream, then convert him into a byte array to report the process percentage :
public static async Task<byte[]> CreateDownloadTaskForFile(string urlToDownload, IProgress<DownloadBytesProgress> progessReporter)
{
int receivedBytes = 0;
int totalBytes = 0;
WebClient client = new WebClient();
using (var stream = await client.OpenReadTaskAsync(urlToDownload))
{
byte[] buffer = new byte[BufferSize];
totalBytes = Int32.Parse(client.ResponseHeaders[HttpResponseHeader.ContentLength]);
using (MemoryStream memoryStream = new MemoryStream())
{
for (; ; )
{
int bytesRead = await stream.ReadAsync(buffer, 0, buffer.Length);
memoryStream.Write(buffer, 0, buffer.Length);
if (bytesRead == 0)
{
await Task.Yield();
break;
}
receivedBytes += bytesRead;
if (progessReporter != null)
{
DownloadBytesProgress args = new DownloadBytesProgress(urlToDownload, receivedBytes, totalBytes);
progessReporter.Report(args);
}
}
return memoryStream.ToArray();
}
}
}
How do I get the position of a stream for each files send ?
Update :
I made a HttpResponseMessage like this :
[Route("GetFiles")]
public HttpResponseMessage GetFiles([FromUri] List<string> filesNames)
{
HttpResponseMessage response = new HttpResponseMessage(HttpStatusCode.BadRequest);
if (filesNames.Count == 0)
return Request.CreateResponse(HttpStatusCode.BadRequest);
var content = new MultipartFormDataContent();
filesNames.ForEach(delegate (string fileName)
{
string filePath = System.Web.Hosting.HostingEnvironment.MapPath("~/Uploads/" + fileName);
byte[] pdf = File.ReadAllBytes(filePath);
content.Add(new ByteArrayContent(pdf), fileName);
});
response = Request.CreateResponse(HttpStatusCode.OK, content);
response.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment");
response.Content.Headers.ContentType = new MediaTypeHeaderValue("application/pdf");
return response;
}
But from my device side : When I am trying to run the request But there is nothing on the response content :
using (var httpResponseMessage = await httpClient.GetAsync(urlToDownload + filesNamesArg))
{
var streamProvider = new MultipartMemoryStreamProvider();
streamProvider = httpResponseMessage.Content.ReadAsMultipartAsync().Result;
}
Could you show me some docs or advice ?
What?
This answer provides a 100% working example for:
Serving multiple files as a single response from a web API using multipart/mixed content type,
Reading the file contents on the client by parsing the response of the web API implemented in 1
I hope this helps.
Server:
The server application is a .Net 4.7.2 MVC project with web API support.
The following method is implemented in an ApiController and returns all the files under the ~/Uploads folder in a single response.
Please make note of the use of Request.RegisterForDispose extension to register the FileStreams for later disposal.
public async Task<HttpResponseMessage> GetFiles()
{
string filesPath = System.Web.Hosting.HostingEnvironment.MapPath("~/Uploads");
List<string> fileNames = new List<string>(Directory.GetFiles(filesPath));
var content = new MultipartContent();
fileNames.ForEach(delegate(string fileName)
{
var fileContent = new StreamContent(File.OpenRead(fileName));
Request.RegisterForDispose(fileContent);
fileContent.Headers.ContentType = System.Net.Http.Headers.MediaTypeHeaderValue.Parse("image/jpeg");
content.Add(fileContent);
});
var response = new HttpResponseMessage();
response.Content = content;
return response;
}
The response's Content-Type header shows as Content-Type: multipart/mixed; boundary="7aeff3b4-2e97-41b2-b06f-29a8c23a7aa7" and each file is packed in different blocks separated by the boundary.
Client:
The client application is a .Net Core 3.0.1 console application.
Please note the synchronous usage of the async methods. This can be easily changed to asynchronous using await, but implemented like this for simplicity:
using System;
using System.IO;
using System.Net.Http;
namespace console
{
class Program
{
static void Main(string[] args)
{
using (HttpClient httpClient = new HttpClient())
{
using (HttpResponseMessage httpResponseMessage = httpClient.GetAsync("http://localhost:60604/api/GetImage/GetFiles").Result)
{
var content = (HttpContent)new StreamContent(httpResponseMessage.Content.ReadAsStreamAsync().Result);
content.Headers.ContentType = httpResponseMessage.Content.Headers.ContentType;
MultipartMemoryStreamProvider multipartResponse = new MultipartMemoryStreamProvider();
content.ReadAsMultipartAsync(multipartResponse);
for(int i = 0; i< multipartResponse.Contents.Count;i++)
{
Stream contentStream = multipartResponse.Contents[i].ReadAsStreamAsync().Result;
Console.WriteLine("Content {0}, length {1}", i, contentStream.Length);
}
}
}
}
}
}
The example shown on the following page doesn't work:
Using c# ClientWebSocket with streams
It hangs on the following line:
await ws.ConnectAsync(serverUri, CancellationToken.None);
It appears the connection is not made.
Please indicate the simplest modification to make the following code work. I do not wish to use any 3rd party tools or libraries.
private static async Task DoClientWebSocket()
{
using (ClientWebSocket ws = new ClientWebSocket())
{
Uri serverUri = new Uri("wss://echo.websocket.org/");
await ws.ConnectAsync(serverUri, CancellationToken.None);
while (ws.State == WebSocketState.Open)
{
string msg = "hello123";
ArraySegment<byte> bytesToSend = new ArraySegment<byte>(Encoding.UTF8.GetBytes(msg));
await ws.SendAsync(bytesToSend, WebSocketMessageType.Text, true, CancellationToken.None);
ArraySegment<byte> bytesReceived = new ArraySegment<byte>(new byte[1024]);
WebSocketReceiveResult result = await ws.ReceiveAsync(bytesReceived, CancellationToken.None);
Console.WriteLine(Encoding.UTF8.GetString(bytesReceived.Array, 0, result.Count));
}
}
}
You are correct. You don't need to add any header in order to use wss://echo.websocket.org/. Your code run just fine at my end. But I'll suggest one improvement to include timeout for your ConnectAsync, SendAsync and ReceiveAsync calls so that it do not get stuck for long.
I have restricted code to call SendAsync to just 5 times so that its easier to verify output.
[Edited:] Include logic to receive complete response by calling `ReceiveAsync multiple times.
private static async Task DoClientWebSocket()
{
using (ClientWebSocket ws = new ClientWebSocket())
{
Uri serverUri = new Uri("wss://echo.websocket.org/");
//Implementation of timeout of 5000 ms
var source = new CancellationTokenSource();
source.CancelAfter(5000);
await ws.ConnectAsync(serverUri, source.Token);
var iterationNo = 0;
// restricted to 5 iteration only
while (ws.State == WebSocketState.Open && iterationNo++ < 5)
{
string msg = "hello0123456789123456789123456789123456789123456789123456789";
ArraySegment<byte> bytesToSend =
new ArraySegment<byte>(Encoding.UTF8.GetBytes(msg));
await ws.SendAsync(bytesToSend, WebSocketMessageType.Text,
true, source.Token);
//Receive buffer
var receiveBuffer = new byte[200];
//Multipacket response
var offset = 0;
var dataPerPacket = 10; //Just for example
while (true)
{
ArraySegment<byte> bytesReceived =
new ArraySegment<byte>(receiveBuffer, offset, dataPerPacket);
WebSocketReceiveResult result = await ws.ReceiveAsync(bytesReceived,
source.Token);
//Partial data received
Console.WriteLine("Data:{0}",
Encoding.UTF8.GetString(receiveBuffer, offset,
result.Count));
offset += result.Count;
if (result.EndOfMessage)
break;
}
Console.WriteLine("Complete response: {0}",
Encoding.UTF8.GetString(receiveBuffer, 0,
offset));
}
}
}
static void Main(string[] args)
{
var taskWebConnect = Task.Run(() => DoClientWebSocket());
taskWebConnect.Wait();
}
Output on command prompt:
Data:hello01234
Data:5678912345
Data:6789123456
Data:7891234567
Data:8912345678
Data:9123456789
Complete response: hello0123456789123456789123456789123456789123456789123456789
I am receiving JSON through a websocket. At least: I am partially. Using an online websocket service I receive the full JSON response (all the HTML markup is ignored). When I look at the JSON that I receive in my console I can see the HTML markup (viewing it with the HTML viewer during debugging removes the HTML) but it ends abruptly (incomplete data).
My buffer has plenty of space and I am using async-await to (supposedly) wait for the entire response to come in before continuing.
private async Task Receive()
{
var buffer = new byte[4096 * 20];
while (_socket.State == WebSocketState.Open)
{
var response = await _socket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
if (response.MessageType == WebSocketMessageType.Close)
{
await
_socket.CloseAsync(WebSocketCloseStatus.NormalClosure, "Close response received",
CancellationToken.None);
}
else
{
var result = Encoding.UTF8.GetString(buffer);
var a = buffer[1000];
var b = buffer[10000];
var c = buffer[50000];
var d = buffer[81000];
Console.WriteLine(result);
var responseObject = JsonConvert.DeserializeObject<Response>(result, _requestParameters.ResponseDataType);
OnSocketReceive.Invoke(this, new SocketEventArgs {Response = responseObject });
buffer = new byte[4096 * 20];
}
}
}
Things to note: The buffer is perfectly big enough and b, c and d are never filled. I should also note that this only happens for the 1-questions-newest-tag-java request, 155-questions-active works perfectly fine.
After doing some digging I have found that response.CloseStatus and response.CloseStatusDescription are always null, response.Count is always 1396 (copy-pasting the result in Word does show that there are always 1396 characters) and response.EndOfMessage is false.
Digging through some source code I have found that the DefaultReceiveBufferSize is 16 * 1024 (big enough) and the WebSocketGetDefaultKeepAliveInterval() refers to an external implementation (but the debugger shows 00:00:30).
It is not a matter of timeout since the debugger halts at the same moment the online service receives its response.
Why is my method continuing to execute when the socket has not yet received all data?
Just to complete #Noseratio response, the code would be something like this:
ArraySegment<Byte> buffer = new ArraySegment<byte>(new Byte[8192]);
WebSocketReceiveResult result= null;
using (var ms = new MemoryStream())
{
do
{
result = await socket.ReceiveAsync(buffer, CancellationToken.None);
ms.Write(buffer.Array, buffer.Offset, result.Count);
}
while (!result.EndOfMessage);
ms.Seek(0, SeekOrigin.Begin);
if (result.MessageType == WebSocketMessageType.Text)
{
using (var reader = new StreamReader(ms, Encoding.UTF8))
{
// do stuff
}
}
}
Cheers.
I might be wrong, but I don't think you're always supposed to receive a complete WebSocket message at once. The server may be sending the message in chunks (that'd correspond to calling SendAsync with endOfMessage: false).
So, do await _socket.ReceiveAsync() in a loop and accumulate the received chunks, until WebSocketReceiveResult.EndOfMessage is true or an error has occured.
On a side note, you probably should be using WebSocket.CreateClientBuffer instead of new ArraySegment<byte>(buffer).
// Read the bytes from the web socket and accumulate all into a list.
var buffer = new ArraySegment<byte>(new byte[1024]);
WebSocketReceiveResult result = null;
var allBytes = new List<byte>();
do
{
result = await webSocket.ReceiveAsync(buffer, CancellationToken.None);
for (int i = 0; i < result.Count; i++)
{
allBytes.Add(buffer.Array[i]);
}
}
while (!result.EndOfMessage);
// Optional step to convert to a string (UTF-8 encoding).
var text = Encoding.UTF8.GetString(allBytes.ToArray(), 0, allBytes.Count);
Following Noseratio's answer I have implemented a temporary buffer that will construct the data of the entire message.
var temporaryBuffer = new byte[BufferSize];
var buffer = new byte[BufferSize * 20];
int offset = 0;
WebSocketReceiveResult response;
while (true)
{
response = await _socket.ReceiveAsync(
new ArraySegment<byte>(temporaryBuffer),
CancellationToken.None);
temporaryBuffer.CopyTo(buffer, offset);
offset += response.Count;
temporaryBuffer = new byte[BufferSize];
if (response.EndOfMessage)
{
break;
}
}
Full implementation here
Try this:
try
{
WebSocketReceiveResult result;
string receivedMessage = "";
var message = new ArraySegment<byte>(new byte[4096]);
do
{
result = await WebSocket.ReceiveAsync(message, DisconectToken);
if (result.MessageType != WebSocketMessageType.Text)
break;
var messageBytes = message.Skip(message.Offset).Take(result.Count).ToArray();
receivedMessage += Encoding.UTF8.GetString(messageBytes);
}
while (!result.EndOfMessage);
if (receivedMessage != "{}" && !string.IsNullOrEmpty(receivedMessage))
{
ResolveWebSocketResponse.Invoke(receivedMessage, Connection);
Console.WriteLine("Received: {0}", receivedMessage);
}
}
catch (Exception ex)
{
var mes = ex.Message;
}
I have the following Invoke-method in a middleware I'm playing with:
public async Task Invoke(HttpContext context)
{
var sw = new Stopwatch();
sw.Start();
context.Response.Headers.Add("X-SW-Start", "GO");
var stream = context.Response.Body;
var buffer = new MemoryStream();
context.Response.Body = buffer;
context.Response.Headers.Add("X-CanSeek", context.Response.Body.CanSeek.ToString());
context.Response.Headers.Add("X-CanRead", context.Response.Body.CanRead.ToString());
context.Response.Headers.Add("X-CanTimeout", context.Response.Body.CanTimeout.ToString());
context.Response.Headers.Add("X-CanWrite", context.Response.Body.CanWrite.ToString());
await next.Invoke(context);
context.Response.Headers.Add("X-SW-Stoppp", $"FINISH {sw.Elapsed.TotalSeconds}");
buffer.Seek(0, SeekOrigin.Begin);
var bufferString = new StreamReader(buffer).ReadToEnd();
bufferString = bufferString.Replace("[[[", "***");
var newBuffer = GenerateStreamFromString(bufferString);
await newBuffer.CopyToAsync(stream);
await stream.FlushAsync();
//context.Response.Body.Seek(0, SeekOrigin.Begin);
//var swr = new StreamWriter(buffer);
//swr.WriteLine("<!-- bob -->");
//await swr.FlushAsync();
}
Ignore all the response headers, writeline-debugging ftw...
This actually works, once, then all subsequent requests are empty except for the headers. All I really want to do is go to the top of the response-pipeline, and then run some global string-replace on the entire document body before it's sent to the browser. If anyone has any better way to do this then by all means...
I am receiving JSON through a websocket. At least: I am partially. Using an online websocket service I receive the full JSON response (all the HTML markup is ignored). When I look at the JSON that I receive in my console I can see the HTML markup (viewing it with the HTML viewer during debugging removes the HTML) but it ends abruptly (incomplete data).
My buffer has plenty of space and I am using async-await to (supposedly) wait for the entire response to come in before continuing.
private async Task Receive()
{
var buffer = new byte[4096 * 20];
while (_socket.State == WebSocketState.Open)
{
var response = await _socket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
if (response.MessageType == WebSocketMessageType.Close)
{
await
_socket.CloseAsync(WebSocketCloseStatus.NormalClosure, "Close response received",
CancellationToken.None);
}
else
{
var result = Encoding.UTF8.GetString(buffer);
var a = buffer[1000];
var b = buffer[10000];
var c = buffer[50000];
var d = buffer[81000];
Console.WriteLine(result);
var responseObject = JsonConvert.DeserializeObject<Response>(result, _requestParameters.ResponseDataType);
OnSocketReceive.Invoke(this, new SocketEventArgs {Response = responseObject });
buffer = new byte[4096 * 20];
}
}
}
Things to note: The buffer is perfectly big enough and b, c and d are never filled. I should also note that this only happens for the 1-questions-newest-tag-java request, 155-questions-active works perfectly fine.
After doing some digging I have found that response.CloseStatus and response.CloseStatusDescription are always null, response.Count is always 1396 (copy-pasting the result in Word does show that there are always 1396 characters) and response.EndOfMessage is false.
Digging through some source code I have found that the DefaultReceiveBufferSize is 16 * 1024 (big enough) and the WebSocketGetDefaultKeepAliveInterval() refers to an external implementation (but the debugger shows 00:00:30).
It is not a matter of timeout since the debugger halts at the same moment the online service receives its response.
Why is my method continuing to execute when the socket has not yet received all data?
Just to complete #Noseratio response, the code would be something like this:
ArraySegment<Byte> buffer = new ArraySegment<byte>(new Byte[8192]);
WebSocketReceiveResult result= null;
using (var ms = new MemoryStream())
{
do
{
result = await socket.ReceiveAsync(buffer, CancellationToken.None);
ms.Write(buffer.Array, buffer.Offset, result.Count);
}
while (!result.EndOfMessage);
ms.Seek(0, SeekOrigin.Begin);
if (result.MessageType == WebSocketMessageType.Text)
{
using (var reader = new StreamReader(ms, Encoding.UTF8))
{
// do stuff
}
}
}
Cheers.
I might be wrong, but I don't think you're always supposed to receive a complete WebSocket message at once. The server may be sending the message in chunks (that'd correspond to calling SendAsync with endOfMessage: false).
So, do await _socket.ReceiveAsync() in a loop and accumulate the received chunks, until WebSocketReceiveResult.EndOfMessage is true or an error has occured.
On a side note, you probably should be using WebSocket.CreateClientBuffer instead of new ArraySegment<byte>(buffer).
// Read the bytes from the web socket and accumulate all into a list.
var buffer = new ArraySegment<byte>(new byte[1024]);
WebSocketReceiveResult result = null;
var allBytes = new List<byte>();
do
{
result = await webSocket.ReceiveAsync(buffer, CancellationToken.None);
for (int i = 0; i < result.Count; i++)
{
allBytes.Add(buffer.Array[i]);
}
}
while (!result.EndOfMessage);
// Optional step to convert to a string (UTF-8 encoding).
var text = Encoding.UTF8.GetString(allBytes.ToArray(), 0, allBytes.Count);
Following Noseratio's answer I have implemented a temporary buffer that will construct the data of the entire message.
var temporaryBuffer = new byte[BufferSize];
var buffer = new byte[BufferSize * 20];
int offset = 0;
WebSocketReceiveResult response;
while (true)
{
response = await _socket.ReceiveAsync(
new ArraySegment<byte>(temporaryBuffer),
CancellationToken.None);
temporaryBuffer.CopyTo(buffer, offset);
offset += response.Count;
temporaryBuffer = new byte[BufferSize];
if (response.EndOfMessage)
{
break;
}
}
Full implementation here
Try this:
try
{
WebSocketReceiveResult result;
string receivedMessage = "";
var message = new ArraySegment<byte>(new byte[4096]);
do
{
result = await WebSocket.ReceiveAsync(message, DisconectToken);
if (result.MessageType != WebSocketMessageType.Text)
break;
var messageBytes = message.Skip(message.Offset).Take(result.Count).ToArray();
receivedMessage += Encoding.UTF8.GetString(messageBytes);
}
while (!result.EndOfMessage);
if (receivedMessage != "{}" && !string.IsNullOrEmpty(receivedMessage))
{
ResolveWebSocketResponse.Invoke(receivedMessage, Connection);
Console.WriteLine("Received: {0}", receivedMessage);
}
}
catch (Exception ex)
{
var mes = ex.Message;
}