Continuous Speech Recognition using Microsoft Cognitive Speech + Websocket - Xamarin - c#

I am trying to build a continuous speech recognition from microphone using the Microsoft Cognitive Speech for Xamarin Android. I don't think there is library for Xamarin so I modified the "Xamarin.Cognitive.BingSpeech" library a little bit (the endpoint, etc) to get it work. I have some problem
I want to connect to the microsoft web socket by following tutorial from https://learn.microsoft.com/en-us/azure/cognitive-services/speech/api-reference-rest/websocketprotocol.
I tried sending HTTPREQUEST using basic HttpClient and got the 101 switch protocol result (I guess I succeed this part?).
UPDATE : My HTTP Request is :
System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Ssl3;
var request = new HttpWebRequest(uriBuilder.Uri);
request.Headers.Add("Authorization", new System.Net.Http.Headers.AuthenticationHeaderValue(Bearer, AuthClient.Token).ToString());
request.Accept=MimeTypes.Json;
request.Host = SpeechEndpoint.Host;
request.Connection = "Upgrade";
request.Headers.Add("Upgrade", "Websocket");
request.KeepAlive = true;
request.Method = "GET";
request.CookieContainer = new CookieContainer();
request.AllowAutoRedirect = true;
request.Date = DateTime.Now;
request.CachePolicy = new System.Net.Cache.RequestCachePolicy(System.Net.Cache.RequestCacheLevel.CacheIfAvailable);
request.Headers.Add("Sec-WebSocket-Key", "dGhlIHNhbXBsZSBub25jZQ==");
request.Headers.Add("Sec-WebSocket-Version", "13");
request.Headers.Add("Sec-WebSocket-Protocol", "chat, superchat");
request.Headers.Add("X-ConnectionId",xConnectionId = Guid.NewGuid().ToString().ToUpper());
After making a HTTPRequest, I am trying to connect to the websocket,
but I always get "Unable to connect to remote server" without any error code or anything. (wss://xxxxxxxx).
Uri wsuri = new Uri(AppConfig.BINGWSSURI);
await _socketclient.ConnectAsync(wsuri, CancellationToken.None);
Log.Info("WSOCKETFINISH", _socketclient.State.ToString());
The second thing I want to achieve is to stream the audio from microphone to the websocket using binary message, so I have to
Record from Microphone (I am using Plugin.AudioRecorder)
Cut it into small chunk pieces
Stream the small pieces asynchronously using the websocket
What I want to achieve : Speech to Text using Microphone with Microsoft Cognitive Speech, dictation mode, so I need partial result instead of waiting the recording to be completed.

I think you want to convert the speech to text. Since the Xamarin.Cognitive.BingSpeech needs you to record the speech and send them as file or stream to the server. I think you could try to use Android speech. And it could also convert text to speech. Here is an example.
If you want to use Xamarin.Cognitive.BingSpeech, you could use Audio Recorder plugin to record the speech and use BingSpeechApiClient to send to server. For example:
BingSpeechApiClient bingSpeechClient = new BingSpeechApiClient ("My Bing Speech API Subscription Key");
var audioFile = "/a/path/to/my/audio/file/in/WAV/format.wav";
var simpleResult = await bingSpeechClient.SpeechToTextSimple (audioFile);
Or
var simpleResult = await bingSpeechClient.SpeechToTextSimple (stream, <sample rate>, <audio record Task>);
Here is the example for Xamarin.Cognitive.BingSpeech.
Update:
I always get "Unable to connect to remote server" without any error code or anything.
You are missing something value in header.
X-ConnectionIdYou need to generate a UUID and add it to the header. For example: client.Options.SetRequestHeader("X-ConnectionId", System.Guid.NewGuid().ToString());
Authorization You need post your subscription key to https://api.cognitive.microsoft.com/sts/v1.0/issueToken. You could use Postman to do this. Then add the return value in the header.
client.Options.SetRequestHeader("Authorization", "eyJ0eXAiOiJKV1Q....uW72PAOBRcUvqY");
so I need partial result instead of waiting the recording to be completed
You could use the GetAudioFileStream() method.For example:
var audioRecordTask = await recorder.StartRecording();
using (var stream = recorder.GetAudioFileStream ())
{
//this will get the recording audio data as it continues to record
}
Update2:
The websoket part code:
var client = new ClientWebSocket();
client.Options.UseDefaultCredentials = true;
client.Options.SetRequestHeader("X-ConnectionId", System.Guid.NewGuid().ToString());
client.Options.SetRequestHeader("Authorization", "eyJ0eXAiOiJKV1QiL....16pbFPOWT3VHXot8");
var a = client.ConnectAsync(new Uri("wss://speech.platform.bing.com/speech/recognition/Dictation/cognitiveservices/v1"), CancellationToken.None);
a.Wait();
Note: Keep your Authorization's value up-to-date.

Related

How to get server side events (onmessage) in C# in Unity?

Im not experienced at all with SSE (or web development in general) so please forgive my ignorance on display. Im trying to get onmessage events from an SSE stream in C# in Unity. I need this to run continuously (preferably async) as these events just keep coming.
For context I'm trying to get events from: https://www.blaseball.com/events/streamGameData
The behaviour I'm trying to replicate in Javascript is:
const evtSource = new EventSource(https://www.blaseball.com/events/streamGameData, {});
evtSource.onmessage = function(event) {
var data = JSON.parse(event.data)["value"];
// Use data here
}
So far I have tried:
Web Request
I found some sample code on a Github jist that uses WebRequest (code below). This works, and gives me the right data back, but I dont understand how (if at all) I can only get the onmessage events, like the supplied JS, this way. Obviously this code just locks the application in its current state.
void Main()
{
var request = WebRequest.Create(new Uri(url));
var response = request.GetResponse();
var stream = response.GetResponseStream();
var encoder = new UTF8Encoding();
var buffer = new byte[2048];
while(true)
{
if (!stream.CanRead)
return;
int length = stream.Read(buffer, 0, 2048);
if (length > 0)
{
var text = encoder.GetString(buffer, 0, length);
// Parse...
}
}
}
Service Stack
I tried using the ServiceStack ServerEventsClient (Link) to get events. I built the NuGet package in VSCode and placed the .netstandard2.0 builds in Unity's Plugins folder. The packages compile fine in Unity and I can access their APIs with no issue.
However, await client.Connect() never returns. I'll admit that I dont really understand these APIs. I used the code verbatim from the link above using Using C# Async/Await friendly API’s and I also tried with client.Start() for good measure. Nothing. However one line in the ServiceStack documentation suggests that this only works with ServiceStack servers? (I do not know enough about SSE to understand the difference)
async void Connect()
{
var client = new ServerEventsClient("https://www.blaseball.com/events/streamGameData");
client.OnMessage += (evt) =>
{
Debug.Log("Message!");
};
client.Start();
ServerEventConnect connectMsg = await client.Connect();
// Never gets here...
}
Can anyone explain what I'm doing wrong with ServerStack, or if that route is even possible? Or perhaps help with a Web Request method? Recommendations for other methods or libraries are also very welcome...

Confirm firebase message was received

I have a c# project sending firebase messages via http post to clients having ios and android.
When clients uninstall my app their firebase device IDs are not deleted from my database unfortunately.
The next time I send a message to the device id witch corresponds to an user who uninstalled my app, of course the message is not delivered.
Is there any way to know if the message was not delivered ?
Unfortunately the response is always successful even if the message is not delivered.
My current code:
var firebaseMessage = new FirebaseMessage();
firebaseMessage.data = notificationMessages;
firebaseMessage.to = device.DeviceRegistrationId; <-- maybe this device is no longer valid
firebaseMessage.priority = "high";
firebaseMessage.notification = new ExpandoObject();
firebaseMessage.notification.title = "myApp";
firebaseMessage.notification.body = "testMessage";
firebaseMessage.notification.sound = "default";
firebaseMessage.notification.click_action = "FCM_PLUGIN_ACTIVITY";
firebaseMessage.notification.icon = "fcm_push_icon";
firebaseMessage.notification.delivery_receipt_requested= true;
var client = new HttpClient();
var appKey = "key=" + ApplicationConfig.FirebasKey;
client.DefaultRequestHeaders.TryAddWithoutValidation("Authorization", appKey);
var response = await client.PostAsJsonAsync("https://fcm.googleapis.com/fcm/send", message);
return response;
When your app is uninstalled from a device, the corresponding registration token would then be invalidated by the FCM server, so any messages sent to that specific token would result to a NotRegistered response (also see my post here). In that event, you could proceed with deleting the token (or archiving it).
If your use-case intentionally wants to know if the message was received on the client side, you're gonna have to implement Delivery receipts.

Bing Speech API and bots Frameworks

I am trying to use Bing's speech API within the Bot Framework (I am familiarizing myself with both of these technologies). Specifically, I am trying to use the DataClientWithIntent that it supports. I was able to look at this example in GitHub, but unfortunately this is seems to be using DataClient only and I am not able to identify where this is specified. The API is being called in the following manner:
using (var client = new HttpClient())
{
var token = Authentication.Instance.GetAccessToken();
client.DefaultRequestHeaders.Add("Authorization", "Bearer " + token.access_token);
using (var binaryContent = new ByteArrayContent(StreamToBytes(audiostream)))
{
binaryContent.Headers.TryAddWithoutValidation("content-type", "audio/wav; codec=\"audio/pcm\"; samplerate=16000");
var response = await client.PostAsync(requestUri, binaryContent);
var responseString = await response.Content.ReadAsStringAsync();
dynamic data = JsonConvert.DeserializeObject(responseString);
return data.header.name;
}
As you can see a stream is passed in, but unfortunately this only writes back what the user wrote.
I have already developed a test bot that uses a Luis application for what I want, but I want to add the ability for the user to either talk to it or type and achieve the same results. I did find this other example, but this is implementing it directly through the Skype framework, which is something I am not interested in at the moment.
Any ideas, documentation, or clarification would be appreciated.

Upload Image to Twitter

I am trying to upload an image to Twitter using Twitter API Version 1.1 and the update_with_media.json method.
https://dev.twitter.com/docs/api/1.1/post/statuses/update_with_media
This is the code I have so far, yet despite numerous variations I can not get a successful upload.
public TwitterResponse UpdateStatus(string message, String fileName, String contentType, byte[] image)
{
RestClient client = new RestClient
{
Authority = TwitterConstants.Authority,
VersionPath = TwitterConstants.Version
};
message = HttpUtility.HtmlEncode(message);
client.AddHeader("content-type", "multipart/form-data");
client.AddField("status", message);
client.AddField("media[]", Convert.ToBase64String(image) + ";filename=" + fileName + ";type=" + contentType);
RestRequest request = new RestRequest
{
Credentials = this.Credentials,
Path = "statuses/update_with_media.json",
Method = Hammock.Web.WebMethod.Post
};
return new TwitterResponse(client.Request(request));
}
I am using Hammock to perform these requests.
Just to rule out possible other issues, I can successfully post a status update to Twitter using the update.json method.
I have also tried using the client.AddFile method and using Fiddler it looks like everything is in place. But the error message I keep getting back is
{"errors":[{"code":195,"message":"Missing or invalid url parameter"}]}
Instead of using native Twitter API, you can use TweeterSharp plugin available at Nuget.
Sample with description is written at this article by me Post message with image on twitter using C#
In particular this is the code snippet
using (var stream = new FileStream(imagePath, FileMode.Open))
{
var result = service.SendTweetWithMedia(new SendTweetWithMediaOptions
{
Status = message,
Images = new Dictionary<string, Stream> { { "john", stream } }
});
lblResult.Text = result.Text.ToString();
}
The complete demo is downloadable attached with the article, feel free to download.
Thanks
I've never used Hammock or or c#, but I know that we had a similar issue...
Our core twitter library worked for everything, but we couldn't get image uploads to work. It turns out that the OAuth library that our twitter lib depended on didn't calculate the signature properly when posting files. We had to update our oauth to get it work.
In our case the exact code we were trying to use worked fine once I substituted an updated OAuth.
If you are using an older version of OAuth, I would suggest looking for a more recent version, and pulling together a quick script to try with that.
Regarding that error message, it may be more of a red herring than a valid message - especially because it's not even listed on their error page:
https://dev.twitter.com/docs/error-codes-responses

Facebook Retrive Data using Graph API using c#

i have created desktop Facebook application using c# .net. i want to retrieve users message,post and chat history. which is convenient way to retrieve users all information.i have started with Facebook Graph API but i am not getting any example.
can any one help me ?
A bit late to the party but anyway:
Add a reference to System.Net.Http and Newtonsoft.Json
string userToken = "theusertokentogiveyoumagicalpowers";
using (var client = new HttpClient())
{
client.BaseAddress = new Uri("https://graph.facebook.com");
HttpResponseMessage response = client.GetAsync($"me?fields=name,email&access_token={userToken}").Result;
response.EnsureSuccessStatusCode();
string result = response.Content.ReadAsStringAsync().Result;
var jsonRes = JsonConvert.DeserializeObject<dynamic>(result);
var email = jsonRes["email"].ToString();
}
Go to developer.facebook.com -> Tools & Support -> Select Graph API Explorer
Here U get FQL Query, Access Token
Then write code in C#.....
var client = new FacebookClient();
client.AccessToken = Your Access Token;
//show user's profile picture
dynamic me = client.Get("me?fields=picture");
pictureBoxProfile.Load(me.picture.data.url);
//show user's birthday
me = client.Get("me/?fields=birthday");
labelBirthday.Text = Convert.ToString(me.birthday);
http://www.codeproject.com/Articles/380635/Csharp-Application-Integration-with-Facebook-Twitt
I hope this will help you.!!!
you can check the Graph explorer tool on Developer.facebook.com , go to Tools and select graph explorer, its a nice tool which gives you exact idea about what you can fetch by sending "GET" and "POST" method on FB Graph APis
From what i see the app now only uses webhooks to post data to a data endpoint (in your app) at which point you can parse and use this. (FQL is deprecated). This is used for things like messaging.
A get request can be send to the API to get info - like the amt. of likes on your page.
The docs of FB explain the string you have to send pretty nicely. Sending requests can be done with the webclient, or your own webrequests.
https://msdn.microsoft.com/en-us/library/bay1b5dh(v=vs.110).aspx
Then once you have a string of the JSON formatted page you can parse this using JSON.NET library. It's available as a NUGEt package.

Categories

Resources