Develop Reference

Couldn't fetch response more than 4096 byte in Chat (Socket) [duplicate]

I am receiving JSON through a websocket. At least: I am partially. Using an online websocket service I receive the full JSON response (all the HTML markup is ignored). When I look at the JSON that I receive in my console I can see the HTML markup (viewing it with the HTML viewer during debugging removes the HTML) but it ends abruptly (incomplete data).
My buffer has plenty of space and I am using async-await to (supposedly) wait for the entire response to come in before continuing.
private async Task Receive()
{
var buffer = new byte[4096 * 20];
while (_socket.State == WebSocketState.Open)
{
var response = await _socket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
if (response.MessageType == WebSocketMessageType.Close)
{
await
_socket.CloseAsync(WebSocketCloseStatus.NormalClosure, "Close response received",
CancellationToken.None);
}
else
{
var result = Encoding.UTF8.GetString(buffer);
var a = buffer[1000];
var b = buffer[10000];
var c = buffer[50000];
var d = buffer[81000];
Console.WriteLine(result);
var responseObject = JsonConvert.DeserializeObject<Response>(result, _requestParameters.ResponseDataType);
OnSocketReceive.Invoke(this, new SocketEventArgs {Response = responseObject });
buffer = new byte[4096 * 20];
}
}
}
Things to note: The buffer is perfectly big enough and b, c and d are never filled. I should also note that this only happens for the 1-questions-newest-tag-java request, 155-questions-active works perfectly fine.
After doing some digging I have found that response.CloseStatus and response.CloseStatusDescription are always null, response.Count is always 1396 (copy-pasting the result in Word does show that there are always 1396 characters) and response.EndOfMessage is false.
Digging through some source code I have found that the DefaultReceiveBufferSize is 16 * 1024 (big enough) and the WebSocketGetDefaultKeepAliveInterval() refers to an external implementation (but the debugger shows 00:00:30).
It is not a matter of timeout since the debugger halts at the same moment the online service receives its response.
Why is my method continuing to execute when the socket has not yet received all data?

Just to complete #Noseratio response, the code would be something like this:
ArraySegment<Byte> buffer = new ArraySegment<byte>(new Byte[8192]);
WebSocketReceiveResult result= null;
using (var ms = new MemoryStream())
{
do
{
result = await socket.ReceiveAsync(buffer, CancellationToken.None);
ms.Write(buffer.Array, buffer.Offset, result.Count);
}
while (!result.EndOfMessage);
ms.Seek(0, SeekOrigin.Begin);
if (result.MessageType == WebSocketMessageType.Text)
{
using (var reader = new StreamReader(ms, Encoding.UTF8))
{
// do stuff
}
}
}
Cheers.

I might be wrong, but I don't think you're always supposed to receive a complete WebSocket message at once. The server may be sending the message in chunks (that'd correspond to calling SendAsync with endOfMessage: false).
So, do await _socket.ReceiveAsync() in a loop and accumulate the received chunks, until WebSocketReceiveResult.EndOfMessage is true or an error has occured.
On a side note, you probably should be using WebSocket.CreateClientBuffer instead of new ArraySegment<byte>(buffer).

// Read the bytes from the web socket and accumulate all into a list.
var buffer = new ArraySegment<byte>(new byte[1024]);
WebSocketReceiveResult result = null;
var allBytes = new List<byte>();
do
{
result = await webSocket.ReceiveAsync(buffer, CancellationToken.None);
for (int i = 0; i < result.Count; i++)
{
allBytes.Add(buffer.Array[i]);
}
}
while (!result.EndOfMessage);
// Optional step to convert to a string (UTF-8 encoding).
var text = Encoding.UTF8.GetString(allBytes.ToArray(), 0, allBytes.Count);

Following Noseratio's answer I have implemented a temporary buffer that will construct the data of the entire message.
var temporaryBuffer = new byte[BufferSize];
var buffer = new byte[BufferSize * 20];
int offset = 0;
WebSocketReceiveResult response;
while (true)
{
response = await _socket.ReceiveAsync(
new ArraySegment<byte>(temporaryBuffer),
CancellationToken.None);
temporaryBuffer.CopyTo(buffer, offset);
offset += response.Count;
temporaryBuffer = new byte[BufferSize];
if (response.EndOfMessage)
{
break;
}
}
Full implementation here

Try this:
try
{
WebSocketReceiveResult result;
string receivedMessage = "";
var message = new ArraySegment<byte>(new byte[4096]);
do
{
result = await WebSocket.ReceiveAsync(message, DisconectToken);
if (result.MessageType != WebSocketMessageType.Text)
break;
var messageBytes = message.Skip(message.Offset).Take(result.Count).ToArray();
receivedMessage += Encoding.UTF8.GetString(messageBytes);
}
while (!result.EndOfMessage);
if (receivedMessage != "{}" && !string.IsNullOrEmpty(receivedMessage))
{
ResolveWebSocketResponse.Invoke(receivedMessage, Connection);
Console.WriteLine("Received: {0}", receivedMessage);
}
}
catch (Exception ex)
{
var mes = ex.Message;
}

How can I send audio to Nexmo Voice through websocket

I am trying to implement Nexmo's Voice api, with websockets, in a .Net Core 2 web api.
This api needs to :
receive audio from phone call, through Nexmo
use Microsoft Cognitive Speech to text api
send the text to a bot
use Microsoft Cognitive text to speech on the reply of the bot
send back the speech to nexmo, through their voice api websocket
For now, I'm bypassing the bot steps, as I am first trying to connect to the websocket.
When trying an echo method (send back to the websocket the audio received), it works without any issue.
But when I try to send the speech from Microsoft text to speech, the phone call ends.
I am not finding any documentation implementing something different than just an echo.
The TextToSpeech and SpeechToText methods work as expected when used outside of the websocket.
Here's the websocket with the speech-to-text :
public static async Task Echo(HttpContext context, WebSocket webSocket)
{
var buffer = new byte[1024 * 4];
WebSocketReceiveResult result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
while (!result.CloseStatus.HasValue)
{
while(!result.EndOfMessage)
{
result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
}
var text = SpeechToText.RecognizeSpeechFromBytesAsync(buffer).Result;
Console.WriteLine(text);
}
await webSocket.CloseAsync(result.CloseStatus.Value, result.CloseStatusDescription, CancellationToken.None);
}
And here's the websocket with the text-to-speech :
public static async Task Echo(HttpContext context, WebSocket webSocket)
{
var buffer = new byte[1024 * 4];
WebSocketReceiveResult result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
while (!result.CloseStatus.HasValue)
{
var ttsAudio = await TextToSpeech.TransformTextToSpeechAsync("Hello, this is a test", "en-US");
await webSocket.SendAsync(new ArraySegment<byte>(ttsAudio, 0, ttsAudio.Length), WebSocketMessageType.Binary, true, CancellationToken.None);
result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
}
await webSocket.CloseAsync(result.CloseStatus.Value, result.CloseStatusDescription, CancellationToken.None);
}
Update March 1st 2019
in reply to Sam Machin's comment
I tried splitting the array into chunks of 640 bytes each (I'm using 16000khz sample rate), but nexmo still hangs up the call, and I still don't hear anything.
public static async Task NexmoTextToSpeech(HttpContext context, WebSocket webSocket)
{
var ttsAudio = await TextToSpeech.TransformTextToSpeechAsync("This is a test", "en-US");
var buffer = new byte[1024 * 4];
WebSocketReceiveResult result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
while (!result.CloseStatus.HasValue)
{
await SendSpeech(context, webSocket, ttsAudio);
result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
}
await webSocket.CloseAsync(WebSocketCloseStatus.NormalClosure, "Closing Socket", CancellationToken.None);
}
private static async Task SendSpeech(HttpContext context, WebSocket webSocket, byte[] ttsAudio)
{
const int chunkSize = 640;
var chunkCount = 1;
var offset = 0;
var lastFullChunck = ttsAudio.Length < (offset + chunkSize);
try
{
while(!lastFullChunck)
{
await webSocket.SendAsync(new ArraySegment<byte>(ttsAudio, offset, chunkSize), WebSocketMessageType.Binary, false, CancellationToken.None);
offset = chunkSize * chunkCount;
lastFullChunck = ttsAudio.Length < (offset + chunkSize);
chunkCount++;
}
var lastMessageSize = ttsAudio.Length - offset;
await webSocket.SendAsync(new ArraySegment<byte>(ttsAudio, offset, lastMessageSize), WebSocketMessageType.Binary, true, CancellationToken.None);
}
catch (Exception ex)
{
}
}
Here's the exception that sometimes appears in the logs :
System.Net.WebSockets.WebSocketException (0x80004005): The remote
party closed the WebSocket connection without completing the close
handshake.

It looks like you're writing the whole audio clip to the websocket, the Nexmo interface requires the audio to be in 20ms frames one per message, this means that you need to break your clip up into 320 or 640 byte (depending on if you're using 8Khz or 16Khz) chunks and write each one to the socket. If you try and write too larger file to the socket it will close as you are seeing.
See https://developer.nexmo.com/voice/voice-api/guides/websockets#writing-audio-to-the-websocket for the details.

c# - Unexpected end of Stream, the content may have already been read by another component

I've been trying to get this to work, but no luck.
I have middleware class where used to verify uploaded files.
After middleware debug goes to the Controller class, but not to the action. Context Request ending after controller's constructor.
Middleware class:
public class UploadMiddleware
{
public UploadMiddleware(RequestDelegate next)
{
_next = next;
}
[DisableFormValueModelBinding]
[ValidateAntiForgeryToken]
public async Task Invoke(HttpContext context)
{
var authToken = context.Request.Headers["id"].ToString();
var path = context.Request.Path.ToString();
var streamBody = context.Request.Body;
if (authToken != String.Empty)
{
if (!IsMultipartContentType(context.Request.ContentType))
{
await context.Response.WriteAsync("Unexpected error.");
await _next(context);
return;
}
var boundary = GetBoundary(context.Request.ContentType);
var reader = new MultipartReader(boundary, context.Request.Body);
// reader.HeadersLengthLimit = int.MaxValue;
// reader.BodyLengthLimit = long.MaxValue;
var section = await reader.ReadNextSectionAsync();
var buffer = new byte[256];
while (section != null)
{
var fileName = GetFileName(section.ContentDisposition);
var bytesRead = 0;
using (var stream = new FileStream(fileName,FileMode.Append))
{
do{
bytesRead = await section.Body.ReadAsync(buffer, 0, buffer.Length);
stream.Write(buffer, 0, bytesRead);
buffer = Convert.FromBase64String(System.Text.Encoding.UTF8.GetString(buffer));
} while (bytesRead < 0);
}
if (!verifyUpcomingFiles(buffer))
{
await context.Response.WriteAsync("Undefined file type detected.");
return;
}
else
{
context.Request.Headers.Add("isVerified","verified");
}
section = await reader.ReadNextSectionAsync();
}
}
else
{
await context.Response.WriteAsync("You are not allowed to complete this process.");
return;
}
await _next(context);
}
I am stuck in this problem. I really need someone point me in a direction, that would be greatly appreciated.

The error message is pretty clear, as well as the docs are and warns you about it in big red box:
Warning
Do not call next.Invoke after the response has been sent to the client. Changes to HttpResponse after the response has started will throw an exception. For example, changes such as setting headers, status code, etc, will throw an exception. Writing to the response body after calling next:
May cause a protocol violation. For example, writing more than the stated content-length.
May corrupt the body format. For example, writing an HTML footer to a CSS file.
HttpResponse.HasStarted is a useful hint to indicate if headers have been sent and/or the body has been written to.
if (!IsMultipartContentType(context.Request.ContentType))
{
await context.Response.WriteAsync("Unexpected error.");
// Here you're calling the next middleware after writing to the response stream!
await _next(context);
return;
}

HttpClient - Size of downloading file before download

I'm implementing download of files with progress bar. I'm using IAsyncOperationWithProgress for this issue, concretely this code. It is working nice, but I'm receiving only count of bytes that were received/downloaded. But I need calculate percentual part for showing progress. That means I need to know total count of bytes at the start of downloading and I didn't find way how to do this effectively.
Following code resolves progress reporting. I tried to get stream length with responseStream.Length but error "This stream does not support seek operations." was thrown.
static async Task<byte[]> GetByteArratTaskProvider(Task<HttpResponseMessage> httpOperation, CancellationToken token, IProgress<int> progressCallback)
{
int offset = 0;
int streamLength = 0;
var result = new List<byte>();
var responseBuffer = new byte[500];
// Execute the http request and get the initial response
// NOTE: We might receive a network error here
var httpInitialResponse = await httpOperation;
using (var responseStream = await httpInitialResponse.Content.ReadAsStreamAsync())
{
int read;
do
{
if (token.IsCancellationRequested)
{
token.ThrowIfCancellationRequested();
}
read = await responseStream.ReadAsync(responseBuffer, 0, responseBuffer.Length);
result.AddRange(responseBuffer);
offset += read;
// here I want to send percents of downloaded data
// offset / (totalSize / 100)
progressCallback.Report(offset);
} while (read != 0);
}
return result.ToArray();
}
Do you have any idea how to deal with this issue? Or do you have some another way how to download files with progress reporting through HttpClient? I tried to use BackgroundDownloader as well but it was not sufficient for me. Thank you.

You can look at the value of the Content-Length header returned by the server, which is stored in your case in httpInitialResponse.Content.Headers. You'll have to find the header in the collection with the corresponding key (i.e. "Content-Length")
You can do it for example like this:
int length = int.Parse(httpInitialResponse.Content.Headers.First(h => h.Key.Equals("Content-Length")).Value.First());
(You have to make sure first that a Content-Length header has been sent by the server, otherwise the line above will fail with an exception)
Your code would look like:
static async Task<byte[]> GetByteArrayTaskProvider(Task<HttpResponseMessage> httpOperation, CancellationToken token, IProgress<int> progressCallback)
{
int offset = 0;
int streamLength = 0;
var result = new List<byte>();
var responseBuffer = new byte[500];
// Execute the http request and get the initial response
// NOTE: We might receive a network error here
var httpInitialResponse = await httpOperation;
var totalValueAsString = httpInitialResponse.Content.Headers.SingleOrDefault(h => h.Key.Equals("Content-Length"))?.Value?.First());
int? totalValue = totalValueAsString != null ? int.Parse(totalValueAsString) : null;
using (var responseStream = await httpInitialResponse.Content.ReadAsStreamAsync())
{
int read;
do
{
if (token.IsCancellationRequested)
{
token.ThrowIfCancellationRequested();
}
read = await responseStream.ReadAsync(responseBuffer, 0, responseBuffer.Length);
result.AddRange(responseBuffer);
offset += read;
if (totalSize.HasValue)
{
progressCallback.Report(offset * 100 / totalSize);
}
//for else you could send back the offset, but the code would become to complex in this method and outside of it. The logic for "supports progress reporting" should be somewhere else, to keep methods simple and non-multi-purpose (I would create a method for with bytes progress and another for percentage progress)
} while (read != 0);
}
return result.ToArray();
}

Read continous bytestream from Stream using TcpClient and Reactive Extensions

Consider the following code:
internal class Program
{
private static void Main(string[] args)
{
var client = new TcpClient();
client.ConnectAsync("localhost", 7105).Wait();
var stream = client.GetStream();
var observable = stream.ReadDataObservable().Repeat();
var s = from d in observable.Buffer(4)
let headerLength = IPAddress.NetworkToHostOrder(BitConverter.ToInt16(d.ToArray(), 2))
let b = observable.Take(headerLength)
select b.ToEnumerable().ToArray();
s.Subscribe(a => Console.WriteLine("{0}", a));
Console.ReadLine();
}
}
public static class Extensions
{
public static IObservable<byte> ReadDataObservable(this Stream stream)
{
return Observable.Defer(async () =>
{
var buffer = new byte[1024];
var readBytes = await stream.ReadAsync(buffer, 0, buffer.Length);
return buffer.Take(readBytes).ToObservable();
});
}
}
Basically I want to parse the messages I receive with Reactive Extensions. The header of the message is parsed correctly using the Buffer(4) and I get the length of the remainder of the message. The problem that arises is that when I do stream.Take(headerLength), the code reevaluates the whole "chain" and tries to get a new message from the stream instead of returning the rest of the bytes which already has been read from the stream. To be more exact, the first ReadAsync(...) returns 38 bytes, the Buffer(4) returns the first 4 of those, the observable.Take(headerLength) does not return the remainding 34 bytes but instead tries to read a new message with ReadAsync.
The question is, how can I make sure the observable.Take(headerLength) receives the already read 34 bytes and not try to read a new message from the stream? I've searched around for a solution, but I can't really figure out how to achieve this.
Edit: This solution (Using Reactive Extensions (Rx) for socket programming practical?) is not what I'm looking for. This isn't reading everything available in the stream (up to buffersize) and makes a continous bytestream out of it. To me this solution doesn't seem like a very efficient way to read from a stream, hence my question.

This approach isn't going to work. The problem is the way you are using the observable. Buffer will not read 4 bytes and quit, it will continually read 4 byte chunks. The Take forms a second subscription that will read overlapping bytes. You'll find it much easier to parse the stream directly into messages.
The following code makes a good deal of effort to clean up properly as well.
Assuming your Message is just this, (ToString added for testing):
public class Message
{
public byte[] PayLoad;
public override string ToString()
{
return Encoding.UTF8.GetString(PayLoad);
}
}
And you have acquired a Stream then you can parse it as follows. First, a method to read an exact number of bytes from a stream:
public async static Task ReadExactBytesAsync(
Stream stream, byte[] buffer, CancellationToken ct)
{
var count = buffer.Length;
var totalBytesRemaining = count;
var totalBytesRead = 0;
while (totalBytesRemaining != 0)
{
var bytesRead = await stream.ReadAsync(
buffer, totalBytesRead, totalBytesRemaining, ct);
ct.ThrowIfCancellationRequested();
totalBytesRead += bytesRead;
totalBytesRemaining -= bytesRead;
}
}
Then the conversion of a stream to IObservable<Message>:
public static IObservable<Message> ReadMessages(
Stream sourceStream,
IScheduler scheduler = null)
{
int subscribed = 0;
scheduler = scheduler ?? Scheduler.Default;
return Observable.Create<Message>(o =>
{
// first check there is only one subscriber
// (multiple stream readers would cause havoc)
int previous = Interlocked.CompareExchange(ref subscribed, 1, 0);
if (previous != 0)
o.OnError(new Exception(
"Only one subscriber is allowed for each stream."));
// we will return a disposable that cleans
// up both the scheduled task below and
// the source stream
var dispose = new CompositeDisposable
{
Disposable.Create(sourceStream.Dispose)
};
// use async scheduling to get nice imperative code
var schedule = scheduler.ScheduleAsync(async (ctrl, ct) =>
{
// store the header here each time
var header = new byte[4];
// loop until cancellation requested
while (!ct.IsCancellationRequested)
{
try
{
// read the exact number of bytes for a header
await ReadExactBytesAsync(sourceStream, header, ct);
}
catch (OperationCanceledException)
{
throw;
}
catch (Exception ex)
{
// pass through any problem in the stream and quit
o.OnError(new InvalidDataException("Error in stream.", ex));
return;
}
ct.ThrowIfCancellationRequested();
var bodyLength = IPAddress.NetworkToHostOrder(
BitConverter.ToInt16(header, 2));
// create buffer to read the message
var payload = new byte[bodyLength];
// read exact bytes as before
try
{
await ReadExactBytesAsync(sourceStream, payload, ct);
}
catch (OperationCanceledException)
{
throw;
}
catch (Exception ex)
{
o.OnError(new InvalidDataException("Error in stream.", ex));
return;
}
// create a new message and send it to client
var message = new Message { PayLoad = payload };
o.OnNext(message);
}
// wrap things up
ct.ThrowIfCancellationRequested();
o.OnCompleted();
});
// return the suscription handle
dispose.Add(schedule);
return dispose;
});
}
EDIT - Very hacky test code I used:
private static void Main(string[] args)
{
var listener = new TcpListener(IPAddress.Any, 12873);
listener.Start();
var listenTask = listener.AcceptTcpClientAsync();
listenTask.ContinueWith((Task<TcpClient> t) =>
{
var client = t.Result;
var stream = client.GetStream();
const string messageText = "Hello World!";
var body = Encoding.UTF8.GetBytes(messageText);
var header = BitConverter.GetBytes(
IPAddress.HostToNetworkOrder(body.Length));
for (int i = 0; i < 5; i++)
{
stream.Write(header, 0, 4);
stream.Write(body, 0, 4);
stream.Flush();
// deliberate nasty delay
Thread.Sleep(2000);
stream.Write(body, 4, body.Length - 4);
stream.Flush();
}
stream.Close();
listener.Stop();
});
var tcpClient = new TcpClient();
tcpClient.Connect(new IPEndPoint(IPAddress.Loopback, 12873));
var clientStream = tcpClient.GetStream();
ReadMessages(clientStream).Subscribe(
Console.WriteLine,
ex => Console.WriteLine("Error: " + ex.Message),
() => Console.WriteLine("Done!"));
Console.ReadLine();
}
Wrapping up
You need to think about setting a timeout for reads, in case the server dies, and some kind of "end message" should be sent by the server. Currently this method will just continually tries to receive bytes. As you haven't specced it, I haven't included anything like this - but if you do, then as I've written it just breaking out of the while loop will cause OnCompleted to be sent.

I guess what is needed here is Qactive: A Rx.Net based queryable reactive tcp server provider
Server
Observable
.Interval(TimeSpan.FromSeconds(1))
.ServeQbservableTcp(new IPEndPoint(IPAddress.Loopback, 3205))
.Subscribe();
Client
var datasourceAddress = new IPEndPoint(IPAddress.Loopback, 3205);
var datasource = new TcpQbservableClient<long>(datasourceAddress);
(
from value in datasource.Query()
//The code below is actually executed on the server
where value <= 5 || value >= 8
select value
)
.Subscribe(Console.WriteLine);
What´s mind blowing about this is that clients can say what and how frequently they want the data they receive and the server can still limit and control when, how frequent and how much data it returns.
For more info on this https://github.com/RxDave/Qactive
Another blog.sample
https://sachabarbs.wordpress.com/2016/12/23/rx-over-the-wire/