Develop Reference

How to hook real-time audio stream endpoint to Direct Line Speech Endpoint?

I am trying to hook-up my real time audio endpoint which produces continuous audio stream with Direct Line Speech (DLS) endpoint which eventually interacts with my Azure bot api.
I have a websocket API that continuously receives audio stream in binary format and this is what I intend to forward it to the DLS endpoint for continuous Speech2Text with my bot.
Based on the feedback and answer here, I have been able to hook up my Direct Line speech endpoint with a real-time stream.
I've tried a sample wav file which correctly gets transcribed by DLS and my bot is correctly able to retrieve the text to operate on it.
I have used the ListenOnce() API and am using a PushAudioInputStream method to push the audio stream to the DLS speech endpoint.
The below code is internals of ListenOnce() method
// Create a push stream
using (var pushStream = AudioInputStream.CreatePushStream())
{
using (var audioInput = AudioConfig.FromStreamInput(pushStream))
{
// Create a new Dialog Service Connector
this.connector = new DialogServiceConnector(dialogServiceConfig, audioInput);
// ... also subscribe to events for this.connector
// Open a connection to Direct Line Speech channel
this.connector.ConnectAsync();
Debug.WriteLine("Connecting to DLS");
pushStream.Write(dataBuffer, dataBuffer.Length);
try
{
this.connector.ListenOnceAsync();
System.Diagnostics.Debug.WriteLine("Started ListenOnceAsync");
}
}
}
dataBuffer in above code is the 'chunk' of binary data I've received on my websocket.
const int maxMessageSize = 1024 * 4; // 4 bytes
var dataBuffer = new byte[maxMessageSize];
while (webSocket.State == WebSocketState.Open)
{
var result = await webSocket.ReceiveAsync(new ArraySegment<byte>(dataBuffer), CancellationToken.None);
if (result.MessageType == WebSocketMessageType.Close)
{
Trace.WriteLine($"Received websocket close message: {result.CloseStatus.Value}, {result.CloseStatusDescription}");
await webSocket.CloseAsync(result.CloseStatus.Value, result.CloseStatusDescription, CancellationToken.None);
}
else if (result.MessageType == WebSocketMessageType.Text)
{
var message = Encoding.UTF8.GetString(dataBuffer);
Trace.WriteLine($"Received websocket text message: {message}");
}
else // binary
{
Trace.WriteLine("Received websocket binary message");
ListenOnce(dataBuffer); //calls the above
}
}
But the above code doesn't work. I believe I have couple of issues/questions with this approach -
I believe I am not correctly chunking the data to Direct Line Speech to ensure that it receives full audio for correct S2T conversion.
I know DLS API supports ListenOnceAsync() but not sure if this supports ASR (it knows when the speaker on other side stopped talking)
Can I just get the websocket url for the Direct Line Speech endpoint and assume DLS correctly consumes the direct websocket stream?

I believe I am not correctly chunking the data to Direct Line Speech to ensure that it receives full audio for correct S2T conversion.
DialogServiceConnector.ListenOnceAsync will listen until the stream is closed (or enough silence is detected). You are not closing your stream except for when you dispose of it at the end of your using block. You could await ListenOnceAsync but you'd have to make sure you close the stream first. If you don't await ListenOnceAsync then you can close the stream whenever you want, but you should probably do it as soon as you finish writing to the stream and you have to make sure you don't dispose of the stream (or the config) before ListenOnceAsync has had a chance to complete.
You also want to make sure ListenOnceAsync gets the full utterance. If you're only receiving 4 bytes at a time then that's certainly not a full utterance. If you want to keep your chunks to 4 bytes then it may be a good idea to keep ListenOnceAsync running during multiple iterations of that loop rather than calling it over and over for every 4 bytes you get.
I know DLS API supports ListenOnceAsync() but not sure if this supports ASR (it knows when the speaker on other side stopped talking)
I think you will have to determine when the speaker stops talking on the client side and then receive a message from your WebSocket indicating that you should close the audio stream for ListenOnceAsync.
It looks like ListenOnceAsync does support ASR.
Can I just get the websocket url for the Direct Line Speech endpoint and assume DLS correctly consumes the direct websocket stream?
You could try it, but I would not assume that myself. Direct Line Speech is still in preview and I don't expect compatibility to come easy.

Sending screen capture from a C# Unity project to a Python socket

I'm trying to do a project in which I have a virtual environment with cameras and want to send screenshots of selected cameras through a socket to a python client that will be running on a raspberry pi zero. In the raspberry I will have additional processing but that is not my current issue.
I've looked into screencapturing in unity, there was a project that appeared to do the capturing part quite right although I'm having some trouble adapting the code from https://assetstore.unity.com/packages/tools/camera/screenshot-helper-102472
I have
ScreenshortHelper.iCaptureWithCamera(camera, (texture2D) => {//Code}};
Would I be able to insert the "send" code in the "//Code" block?
Also that part for me is a bit tricky, sending the "texture" through the socket, I was thinking about sending some parameters like size and dimensions first so that I know what I would be receiving on the python end. Does this make sense?
I appreciate any help that you can give me!
Thanks in advance.

First, you need to configure the Transport Layer API in your unity code, and put references to your hostId and your channelId somewhere where that function can access it.
Then, that anonymous function there can be:
(texture2D) => {
// Encode it to PNG bytes and put it into a buffer:
byte[] buffer = texture2D.EncodeToPNG();
// Connect to the python listener:
byte error;
connectionId = NetworkTransport.Connect(hostId, "192.168.1.42", 8888, 0, out error);
// Send:
int bufferLength = bytes.Length;
NetworkTransport.Send(hostId, connectionId, channelId, buffer, bufferLength, out error);
// Disconnect:
NetworkTransport.Disconnect(hostId, connectionId, out error);
}
And then on the other side, you can use Python to listen on a binary stream, and then do whatever you need to do with it. There is some useful information on how to read an image over a socket in python here: Receive Image using socket programming in Python

StreamSocket : datareader.LoadAsync waits infinte even when data is avaialbe

I have the following code in my Windows phone 8 app.
//connection code, done during app start
socket = new StreamSocket();
await socket.ConnectAsync(serverHostName, serviceName);
dataReader = new DataReader(socket.InputStream);
dataReader.InputStreamOptions = InputStreamOptions.Partial;
dataWriter = new DataWriter(socket.OutputStream);
After the connection is established, I have another thread which checks for incoming network packets
await dataReader.LoadAsync(2048);
dataReader.ReadBytes(buffer);
----------
Workflow is as follows
Phone connects to server using socket.ConnectAsync
Server responds with initial message (Phone receives this properly in dataReader.LoadAsync function)
Phone now sends the 'business specific' request
Server now replies with 'business specific' response (Problem lies here. Phone doesn't receive the reply from server at some of the times).
There is no scenario difference between working state' and 'non working state'.
So I tried to debug this. I put a breakpoint for dataReader.LoadAsync and saw that execution waits infinitely at the call.
To make sure that the server is sending data properly, I ran the app in Windows phone emulator and ran the WireShark network analyzer in the PC. I could see that packets are being received for the IP Address of the phone.
Anyone has any hints on why the dataReader.LoadAsync function call doesn't return at all, when there is data ready to be read in the socket?

I faced the same problem. It is especially bad for Bluetooth RFCOMM SPP serial port devices, because the underlying Rfcomm-object does not provide capabilities for setting ReadTimeout values.
Edit: The InputStreamOptions.Partial option seems to be working UWP Win10 platform, but it is only useful when you are already know much data you are expecting. Otherwise it will wait indefinitely on the last call.
I almost gave up, when I found in references below these lines to solve the problem by using a CancellationTokenSource
//connect your Windows.Devices.Bluetooth.Rfcomm.RfcommDeviceService
// see the Bluetooth chat example
[...]
StreamSocket streamSocket = new StreamSocket();
await streamSocket.ConnectAsync(...); //connect to Bluetooth device
DataReader dataReader = new DataReader(inputStream); // to read from the stream
try
{
var timeoutSource = new CancellationTokenSource(1000); // 1000 ms
uint numberBytesToRead = 256;
var data = await dataReader.LoadAsync(numberBytesToRead).AsTask(timeoutSource.Token);
}
catch (TaskCanceledException)
{
// we will get here, and everything looks fine, but the problem is:
// The underlying streamSocket is also closed!
// we can not re-use the streamSocket. Any more calls to LoadAsync results in exceptions (something about the object being not assigned...)
// we need to do a full await streamSocket.ConnectAsync(...) again, but that takes ~3 seconds.
}
So this method is only a brute-force, last-resort attempt at a time-out.
The method from #mayu works very good (serialDevice.ReadTimeout), but only on devices of class Windows.Devices.SerialCommunication.Serial​Device, but not on
Windows.Devices.Bluetooth.Rfcomm.RfcommDeviceService. I don't know how the situation is for TCP/IP sockets.
In short, is there any usable time-out for RFCOMM SPP Bluetooth connections?
Or any method to know ahead of time if .LoadAsync(1) will block, because no new data is available?
This fellow over at MSDN has the exact same problem, but MS don't know an answer either: https://social.msdn.microsoft.com/Forums/vstudio/en-US/71ea17d4-ca16-43c2-ab43-02d5301def3f/chow-to-set-timeout-on-streamsocketreadasync?forum=wpdevelop
References:
In UWP StreamSocket, can I read data with timeout and leave the connection open if timeout elapses
https://social.msdn.microsoft.com/Forums/en-US/8a5c4fdc-28d6-4a22-8df6-bc519efeaa4d/how-to-control-the-timeout-for-reading-from-streamsocket?forum=winappswithcsharp
DataReader of SocketStream for UWP App

"According to documentation when using InputStreamOptions.Partial, you should use UnconsummedBufferLength instead of an hardcoded value"
That sample seems to be broken.
"await reader.LoadAsync(reader.UnconsumedBufferLength);" is equivalent to
await reader.LoadAsync(0); and then it's not possible to read any data, since you have no buffer to read from.
I'm testing this now and it seems like "reader.InputStreamOptions = Partial;" has no effect at all. My only workaround is to lower the read timeout.

According to documentation when using InputStreamOptions.Partial, you should use UnconsummedBufferLength instead of an hardcoded value :
DataReader reader = new DataReader(clientSocket.InputStream);
// Set inputstream options so that we don't have to know the data size
reader.InputStreamOptions = Partial;
await reader.LoadAsync(reader.UnconsumedBufferLength);
Sample is there

I had a similar problem using Windows Remote Arduino library and SerialUSB stream. I had to change this library and call LoadAsync(1) instead of original LoadAsync(100). Now the code is working fine.
see: https://github.com/ms-iot/remote-wiring/issues/111

For serial devices you need to set
device.ReadTimeout = TimeSpan.FromMilliseconds(100);
for the LoadAsync to return before the buffer is full.

The only way I found to get round not knowing the size of the data before reading was to read one byte at a time until I got a timeout. Feels horrid, but works. Is there a better way yet?
private async Task ReadData(CancellationToken cancellationToken)
{
cancellationToken.ThrowIfCancellationRequested();
DataReaderObject.InputStreamOptions = InputStreamOptions.Partial;
uint data = 0;
uint bufferLength = DataReaderObject.UnconsumedBufferLength;
var timeoutSource = new CancellationTokenSource(100); // 100 ms
try
{
while (true)
{
data = await DataReaderObject.LoadAsync(1).AsTask(timeoutSource.Token);
if (data > 0)
{
String temp = DataReaderObject.ReadString(data);
TemperatureValue.Text += temp.Trim();
}
}
}
catch (Exception)
{
;
}
}

How can I read from a socket repeatedly?

To start I am coding in C#. I am writing data of varying sizes to a device through a socket. After writing the data I want to read from the socket because the device will write back an error code/completion message once it has finished processing all of the data. Currently I have something like this:
byte[] resultErrorCode = new byte[1];
resultErrorCode[0] = 255;
while (resultErrorCode[0] == 255)
{
try
{
ReadFromSocket(ref resultErrorCode);
}
catch (Exception)
{
}
}
Console.WriteLine(ErrorList[resultErrorCode[0] - 48]);
I use ReadFromSocket in other places, so I know that it is working correctly. What ends up happening is that the port I am connecting from (on my machine) changes to random ports. I think that this causes the firmware on the other side to have a bad connection. So when I write data on the other side, it tries to write data to the original port that I connected through, but after trying to read several times, the connection port changes on my side.
How can I read from the socket continuously until I receive a completion command? If I know that something is wrong with the loop because for my smallest test file it takes 1 min and 13 seconds pretty consistently. I have tested the code by removing the loop and putting the code to sleep for 1 min and 15 seconds. When it resumes, it successfully reads the completion command that I am expecting. Does anyone have any advice?

What you should have is a separate thread which will act like a driver of your external hardware. This thread will receive all data, parse it and transmit the appropriate messages to the rest of your application. This portion of code will give you an idea of how receive and parse data from your hardware.
public void ContinuousReceive(){
byte[] buffer = new byte[1024];
bool terminationCodeReceived = false;
while(!terminationCodeReceived){
try{
if(server.Receive(buffer)>0){
// We got something
// Parse the received data and check if the termination code
// is received or not
}
}catch (SocketException e){
Console.WriteLine("Oops! Something bad happened:" + e.Message);
}
}
}
Notes:
If you want to open a specific port on your machine (some external hardware are configured to talk to a predefined port) then you should specify that when you create your socket
Never close your socket until you want to stop your application or the external hardware API requires that. Keeping your socket open will resolve the random port change
using Thread.Sleep when dealing with external hardware is not a good idea. When possible, you should either use events (in case of RS232 connections) or blocking calls on separate threads as it is the case in the code above.

Image not sending over UDP due to less buffer size C#

I am sending Bitmap image encoded in the form jpeg *over UDP socket*.The issue is there is variable size of images each time.Container is Image Packet which consists of Multiple or Single Image with a packet Identifier Information.
Server Side
MemoryStream Ms = new MemoryStream();
bformatter.Serialize(Ms, container);
byte[] TestingFlow = Ms.GetBuffer();
ServerSocket.SendTo(TestingFlow, 54000,
SocketFlags.None, RemoteEndpoint);
Client Side
byte[] Recievedbytes = UdpListener.Receive(ref RemoteEndPoint);
ImageStream = new MemoryStream(Recievedbytes, 0, Recievedbytes.Length);
imagecontainer = (ImageContainer)bformater.Deserialize(ImageStream);

I'm pretty sure you have to use
ServerSocket.SendTo(TestingFlow,TestingFlow.Length,
SocketFlags.None, RemoteEndpoint);
in server side

UDP for sending an image file? No way.
If you're after data integrity then forget it. TCP is the better protocol.
UDP is for sending small data packets where speed is the issue rather than data integrity, hence its use in internet gaming. Datagrams may arrive out of order, be duplicated, or vanish completely. UDP has no intrinsic error checking or correction. This is left to the application if required. Hence its speed over reliability.
Unless you wish to write all of that error checking with resend requests and datagram handling (to ensure that you build your file back up in the right order) then just use TCP over sockets.
At least with TCP you can split your image up into manageable blocks, send them and be safe in the knowledge that they'll arrive in the right order and complete.