How can I send audio to Nexmo Voice through websocket - c#

I am trying to implement Nexmo's Voice api, with websockets, in a .Net Core 2 web api.
This api needs to :
receive audio from phone call, through Nexmo
use Microsoft Cognitive Speech to text api
send the text to a bot
use Microsoft Cognitive text to speech on the reply of the bot
send back the speech to nexmo, through their voice api websocket
For now, I'm bypassing the bot steps, as I am first trying to connect to the websocket.
When trying an echo method (send back to the websocket the audio received), it works without any issue.
But when I try to send the speech from Microsoft text to speech, the phone call ends.
I am not finding any documentation implementing something different than just an echo.
The TextToSpeech and SpeechToText methods work as expected when used outside of the websocket.
Here's the websocket with the speech-to-text :
public static async Task Echo(HttpContext context, WebSocket webSocket)
{
var buffer = new byte[1024 * 4];
WebSocketReceiveResult result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
while (!result.CloseStatus.HasValue)
{
while(!result.EndOfMessage)
{
result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
}
var text = SpeechToText.RecognizeSpeechFromBytesAsync(buffer).Result;
Console.WriteLine(text);
}
await webSocket.CloseAsync(result.CloseStatus.Value, result.CloseStatusDescription, CancellationToken.None);
}
And here's the websocket with the text-to-speech :
public static async Task Echo(HttpContext context, WebSocket webSocket)
{
var buffer = new byte[1024 * 4];
WebSocketReceiveResult result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
while (!result.CloseStatus.HasValue)
{
var ttsAudio = await TextToSpeech.TransformTextToSpeechAsync("Hello, this is a test", "en-US");
await webSocket.SendAsync(new ArraySegment<byte>(ttsAudio, 0, ttsAudio.Length), WebSocketMessageType.Binary, true, CancellationToken.None);
result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
}
await webSocket.CloseAsync(result.CloseStatus.Value, result.CloseStatusDescription, CancellationToken.None);
}
Update March 1st 2019
in reply to Sam Machin's comment
I tried splitting the array into chunks of 640 bytes each (I'm using 16000khz sample rate), but nexmo still hangs up the call, and I still don't hear anything.
public static async Task NexmoTextToSpeech(HttpContext context, WebSocket webSocket)
{
var ttsAudio = await TextToSpeech.TransformTextToSpeechAsync("This is a test", "en-US");
var buffer = new byte[1024 * 4];
WebSocketReceiveResult result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
while (!result.CloseStatus.HasValue)
{
await SendSpeech(context, webSocket, ttsAudio);
result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
}
await webSocket.CloseAsync(WebSocketCloseStatus.NormalClosure, "Closing Socket", CancellationToken.None);
}
private static async Task SendSpeech(HttpContext context, WebSocket webSocket, byte[] ttsAudio)
{
const int chunkSize = 640;
var chunkCount = 1;
var offset = 0;
var lastFullChunck = ttsAudio.Length < (offset + chunkSize);
try
{
while(!lastFullChunck)
{
await webSocket.SendAsync(new ArraySegment<byte>(ttsAudio, offset, chunkSize), WebSocketMessageType.Binary, false, CancellationToken.None);
offset = chunkSize * chunkCount;
lastFullChunck = ttsAudio.Length < (offset + chunkSize);
chunkCount++;
}
var lastMessageSize = ttsAudio.Length - offset;
await webSocket.SendAsync(new ArraySegment<byte>(ttsAudio, offset, lastMessageSize), WebSocketMessageType.Binary, true, CancellationToken.None);
}
catch (Exception ex)
{
}
}
Here's the exception that sometimes appears in the logs :
System.Net.WebSockets.WebSocketException (0x80004005): The remote
party closed the WebSocket connection without completing the close
handshake.

It looks like you're writing the whole audio clip to the websocket, the Nexmo interface requires the audio to be in 20ms frames one per message, this means that you need to break your clip up into 320 or 640 byte (depending on if you're using 8Khz or 16Khz) chunks and write each one to the socket. If you try and write too larger file to the socket it will close as you are seeing.
See https://developer.nexmo.com/voice/voice-api/guides/websockets#writing-audio-to-the-websocket for the details.

Related

ispossible create websocket server connected to another websocket?

I need WebSocket code for implement structure of my client. I will create WebSocket server for my client with receive by client from ex: binance websocket
Just part of connect to binance websocket need implement.
ASP.NET Core 5 C#
private async Task Echo(HttpContext context, WebSocket webSocket)
{
var buffer = new byte[1024 * 4];
WebSocketReceiveResult result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
while (!result.CloseStatus.HasValue)
{
await webSocket.SendAsync(new ArraySegment<byte>(buffer, 0, result.Count), result.MessageType, result.EndOfMessage, CancellationToken.None);
result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
}
await webSocket.CloseAsync(result.CloseStatus.Value, result.CloseStatusDescription, CancellationToken.None);
}
I did something like that. I created 2 WebSocket servers, one acting as a proxy to the other. You can check it out, maybe it works for you or at least you can find some answers there. It is a bit disordered because I did it as a research, so ask me if you need more info.
https://github.com/RenanDiaz/WebSockets
public override async Task OnConnected(WebSocket socket)
{
await base.OnConnected(socket);
if (_client == null)
{
_client = new ClientWebSocket();
await _client.ConnectAsync(new Uri("ws://localhost:5001/chat"), CancellationToken.None);
var thread = new Thread(new ThreadStart(ReceiveMessageFromAPIServer));
thread.Start();
}
}
private async void ReceiveMessageFromAPIServer()
{
var buffer = new byte[1024 * 4];
while (_client != null)
{
var result = await _client.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
if (result.MessageType == WebSocketMessageType.Close)
{
await _client.CloseOutputAsync(WebSocketCloseStatus.NormalClosure, "", CancellationToken.None);
break;
}
var messageString = Encoding.UTF8.GetString(buffer, 0, result.Count);
var message = JsonConvert.DeserializeObject<IncomingAPIServerMessage>(messageString);
await SendMessageToAll(message);
}
}

ClientWebSocket example hangs

The example shown on the following page doesn't work:
Using c# ClientWebSocket with streams
It hangs on the following line:
await ws.ConnectAsync(serverUri, CancellationToken.None);
It appears the connection is not made.
Please indicate the simplest modification to make the following code work. I do not wish to use any 3rd party tools or libraries.
private static async Task DoClientWebSocket()
{
using (ClientWebSocket ws = new ClientWebSocket())
{
Uri serverUri = new Uri("wss://echo.websocket.org/");
await ws.ConnectAsync(serverUri, CancellationToken.None);
while (ws.State == WebSocketState.Open)
{
string msg = "hello123";
ArraySegment<byte> bytesToSend = new ArraySegment<byte>(Encoding.UTF8.GetBytes(msg));
await ws.SendAsync(bytesToSend, WebSocketMessageType.Text, true, CancellationToken.None);
ArraySegment<byte> bytesReceived = new ArraySegment<byte>(new byte[1024]);
WebSocketReceiveResult result = await ws.ReceiveAsync(bytesReceived, CancellationToken.None);
Console.WriteLine(Encoding.UTF8.GetString(bytesReceived.Array, 0, result.Count));
}
}
}
You are correct. You don't need to add any header in order to use wss://echo.websocket.org/. Your code run just fine at my end. But I'll suggest one improvement to include timeout for your ConnectAsync, SendAsync and ReceiveAsync calls so that it do not get stuck for long.
I have restricted code to call SendAsync to just 5 times so that its easier to verify output.
[Edited:] Include logic to receive complete response by calling `ReceiveAsync multiple times.
private static async Task DoClientWebSocket()
{
using (ClientWebSocket ws = new ClientWebSocket())
{
Uri serverUri = new Uri("wss://echo.websocket.org/");
//Implementation of timeout of 5000 ms
var source = new CancellationTokenSource();
source.CancelAfter(5000);
await ws.ConnectAsync(serverUri, source.Token);
var iterationNo = 0;
// restricted to 5 iteration only
while (ws.State == WebSocketState.Open && iterationNo++ < 5)
{
string msg = "hello0123456789123456789123456789123456789123456789123456789";
ArraySegment<byte> bytesToSend =
new ArraySegment<byte>(Encoding.UTF8.GetBytes(msg));
await ws.SendAsync(bytesToSend, WebSocketMessageType.Text,
true, source.Token);
//Receive buffer
var receiveBuffer = new byte[200];
//Multipacket response
var offset = 0;
var dataPerPacket = 10; //Just for example
while (true)
{
ArraySegment<byte> bytesReceived =
new ArraySegment<byte>(receiveBuffer, offset, dataPerPacket);
WebSocketReceiveResult result = await ws.ReceiveAsync(bytesReceived,
source.Token);
//Partial data received
Console.WriteLine("Data:{0}",
Encoding.UTF8.GetString(receiveBuffer, offset,
result.Count));
offset += result.Count;
if (result.EndOfMessage)
break;
}
Console.WriteLine("Complete response: {0}",
Encoding.UTF8.GetString(receiveBuffer, 0,
offset));
}
}
}
static void Main(string[] args)
{
var taskWebConnect = Task.Run(() => DoClientWebSocket());
taskWebConnect.Wait();
}
Output on command prompt:
Data:hello01234
Data:5678912345
Data:6789123456
Data:7891234567
Data:8912345678
Data:9123456789
Complete response: hello0123456789123456789123456789123456789123456789123456789

Basic WebSocket Chat

My objective is to create a basic chat application using web sockets.
I've got a client setup, however I am getting a specific exception every time I try to receive text on the client's side specifically.
The exception I am getting is the following:
The buffer type '166' is invalid. Valid buffer types are: 'Close', 'BinaryFragment', 'BinaryMessage', 'UTF8Fragment', 'UTF8Message'
I get this exception on the following line of code var socketResult = await _socket.ReceiveAsync(segment, CancellationToken.None); in the below code:
private readonly ClientWebSocket_socket;
private async Task ListenAsync(Action<string> textReceived)
{
while (_socket.State == WebSocketState.Open || _socket.State == WebSocketState.CloseSent)
{
var buffer = new byte[1024];
var segment = new ArraySegment<byte>(buffer, 0, buffer.Length);
var socketResult = await _socket.ReceiveAsync(segment, CancellationToken.None); // problem occurs here
if (socketResult.MessageType == WebSocketMessageType.Close)
{
await _socket.CloseAsync(WebSocketCloseStatus.NormalClosure, string.Empty, CancellationToken.None);
break;
}
var text = Encoding.UTF8.GetString(buffer, 0, socketResult.Count);
textReceived(text);
}
}
what am I doing wrong? similar code works fine to receive the text on the server side.
EDIT:
The server listens fine and the _socket.ReceiveAsync method runs as expected. It waits until a message has been published and only then does it move on to reply. If I don't listen on the client side, Everything works. However when I start the server to listen for the client messages then start the client to listen, the client breaks. I have tried sending a message first from the client and then starting to listen for the response afterwards and I still experience the same thing.
Below is the method I use to listen on the server side.
private static WebSocket _socket;
public async Task ListenAsync()
{
while (_socket.State == WebSocketState.Open || _socket.State == WebSocketState.CloseSent)
{
var buffer = new byte[1024];
var segment = new ArraySegment<byte>(buffer, 0, buffer.Length);
var socketResult = await _socket.ReceiveAsync(segment, CancellationToken.None);
if (socketResult.MessageType == WebSocketMessageType.Close)
{
await _socket.CloseOutputAsync(WebSocketCloseStatus.NormalClosure, string.Empty, CancellationToken.None);
break;
}
var text = Encoding.UTF8.GetString(buffer, 0, socketResult.Count);
await PublishAsync($"Server received the following text: {text} # {DateTime.Now:g}");
}
}
This is the method I use to publish messages from the server.
public async Task PublishAsync(string text)
{
var bytes = Encoding.UTF8.GetBytes(text);
var buffer = new ArraySegment<byte>(bytes, 0, bytes.Length);
await _socket.SendAsync(buffer, WebSocketMessageType.Text, true, CancellationToken.None);
}

collect output from Task.Whenall to a list/data-type

I have the following c# code where i am sending a series of requests and reponses
public static async Task AuthenticateQvpx2()
{
var handshake = new Handshake();
foreach (var request in handshake.AutheticateStrings)
{
var buffer = _encoder.GetBytes(request);
await Task.WhenAll(Receive(_webSocket), Send(_webSocket, buffer));
}
}
The async functions Send and Receive, has the following code.
await webSocket.SendAsync(new ArraySegment<byte>(buffer), WebSocketMessageType.Text, true, CancellationToken.None);
var result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
I wish to collect the requests and responses into an array/ any form of data type.
I having trouble as I am not particularly sure of what i should do next?
I wish to collect the requests and responses
It's kind of odd to collect the requests, as that data is already right there (in the buffer variable).
Assuming you meant that you just need the response, you can do that using await:
public static async Task AuthenticateQvpx2()
{
var handshake = new Handshake();
foreach (var request in handshake.AutheticateStrings)
{
var buffer = _encoder.GetBytes(request);
var receiveTask = Receive(_webSocket);
await Task.WhenAll(receiveTask, Send(_webSocket, buffer));
var response = await receiveTask;
}
}
Not sure based on your snippets what type of Tasks your Send and Receive return but generally you can get your results from multiple tasks after using Task.WhenAll using LINQ this way:
var handshake = new Handshake();
List<Task<WebsocketReceiveResult>> tasks = newList<Task<WebsocketReceiveResult>>();
foreach (var request in handshake.AutheticateStrings)
{
var buffer = _encoder.GetBytes(request);
tasks.Add(webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None));
}
await Task.WhenAll(tasks);
var resultArray = tasks.Select(t => t.Result).ToArray();
Try something like
ConcurrentDictionary<Guid, (System.Byte[], WebSocketReceiveResult)> x = new ConcurrentDictionary<Guid, (byte[], WebSocketReceiveResult)>();
Generate GUID in your foreach a pass it to your methods:
foreach (var request in handshake.AutheticateStrings)
{
var buffer = _encoder.GetBytes(request);
var guid = Guid.NewGuid();
await Task.WhenAll(Receive(_webSocket, guid), Send(_webSocket, buffer, guid));
}
Then you can work with dictionary from within your Receive and Send methods in natural way.
void Send(WebSocket webSocket, byte[] buffer, Guid guid)
{
x.GetOrAdd(guid, new ValueTuple<System.Byte[], WebSocketReceiveResult>(buffer, null));
await webSocket.SendAsync(new ArraySegment<byte>(buffer), WebSocketMessageType.Text, true, CancellationToken.None)
}
void Receive(WebSocket webSocket, Guid guid)
{
var result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
x[guid].Item2 = result;
}

How to implement progress reporting for Portable HttpClient

I'm writing a library with intentions to use it in desktop (.Net 4.0 and up), phone (WP 7.5 and up) and Windows Store (Windows 8 and up) apps.
The library has the capability to download files from the Internet using Portable HttpClient library, and report the progress of the download.
I search around here and the rest of the internet for documentations and code sample/guidelines on how to implement the progress reporting, and this search led me to nowhere.
Does anyone has an article, documentation, guideline, code sample or whatever to help me achieve this?
I wrote the following code to implement progress reporting. The code supports all the platforms I wanted; however, you need to reference the following NuGet packages:
Microsoft.Net.Http
Microsoft.Bcl.Async
Here is the code:
public async Task DownloadFileAsync(string url, IProgress<double> progress, CancellationToken token)
{
var response = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead, token);
if (!response.IsSuccessStatusCode)
{
throw new Exception(string.Format("The request returned with HTTP status code {0}", response.StatusCode));
}
var total = response.Content.Headers.ContentLength.HasValue ? response.Content.Headers.ContentLength.Value : -1L;
var canReportProgress = total != -1 && progress != null;
using (var stream = await response.Content.ReadAsStreamAsync())
{
var totalRead = 0L;
var buffer = new byte[4096];
var isMoreToRead = true;
do
{
token.ThrowIfCancellationRequested();
var read = await stream.ReadAsync(buffer, 0, buffer.Length, token);
if (read == 0)
{
isMoreToRead = false;
}
else
{
var data = new byte[read];
buffer.ToList().CopyTo(0, data, 0, read);
// TODO: put here the code to write the file to disk
totalRead += read;
if (canReportProgress)
{
progress.Report((totalRead * 1d) / (total * 1d) * 100);
}
}
} while (isMoreToRead);
}
}
The using it is as simple as:
var progress = new Microsoft.Progress<double>();
progress.ProgressChanged += (sender, value) => System.Console.Write("\r%{0:N0}", value);
var cancellationToken = new CancellationTokenSource();
await DownloadFileAsync("http://www.dotpdn.com/files/Paint.NET.3.5.11.Install.zip", progress, cancellationToken.Token);
You can specify HttpCompletionOption.ResponseHeadersRead and then get the stream and report progress while you read from the stream. See this similar question.

Categories

Resources