Develop Reference

Why is my grpc so much slower then my minimal api?

Hey there a bit of an odd question
I've downloaded the code for this :
https://medium.com/geekculture/build-high-performant-microservices-using-grpc-and-net-6-adde158c5ac
Actual Code:
https://github.com/csehammad/gRPCDemoUsingNET6
basically is streams 1 line at a time to a client~---takes about 1 min to stream 1.5 mil lines.
Tried replicating the same behavior with Minimal Api and not sure how,
But it takes 13 seconds to read and stream all 5 million records.
Shouldnt it be slower?
I think my code is badly messed up and probably rotter logic all over the place 😄.
Any Help would be swell
My Ui Code:
`using HttpClient client = new();
using HttpResponseMessage response = await client.GetAsync(
"http://localhost:5247/test",
HttpCompletionOption.ResponseHeadersRead
).ConfigureAwait(false);
IAsyncEnumerable<Sales> Sales= await response.Content.ReadFromJsonAsync<IAsyncEnumerable<Sales>>().ConfigureAwait(false);
var count = 0;
var watch = System.Diagnostics.Stopwatch.StartNew();
await foreach (var each in Sales)
{
Console.WriteLine(String.Format("New Order Receieved from {0}-{1},Order ID = {2}, ", each.Country, each.Region, each.TotalRevenue));
}`
The Backend:
`app.MapGet("/test", async () =>
{
return MakeHttpCall();
});
{
var watch = System.Diagnostics.Stopwatch.StartNew();
int Count = 0;
using (var reader = new StreamReader("path"))
{
while (watch.Elapsed < TimeSpan.FromSeconds(60) && !reader.EndOfStream)
{
var line = reader.ReadLine();
var pieces = line.Split(',');
var _model = new Sales();
_model.Region = pieces[0];
_model.Country = pieces[1];
yield return _model;
}
}
}`
Also the ui seems to start printing the text only when its all done, meaning not rly streaming..
(removed some code regarding model,watch,and model building --non relevant stuff)
Excpeting it to work after hours of debugging etc but getting nowhere..

The write operation doesn't update the characteristic value for iOS in xamarin forms

I'm creating a BLE application using xamarin forms. Everything is working fine in Android I'm able to read and write GATT characteristics. In iOS I'm able to read successfully but the write operation doesn't update the characteristics value. There is no error in the write operation as well it is executing but the characteristics value is not changing. I tried iOS native application called light blue there its working fine the characteristic value is updated I'm facing issue only in Xamarin forms app. This is my code
private async Task<string> ProcessDeviceInformationService(IService deviceInfoService)
{
try
{
await adapter.ConnectToDeviceAsync(device);
var sb = new StringBuilder("Getting information from Device Information service: \n");
var characteristics = deviceInfoService.GetCharacteristicsAsync();
var characteristic = await deviceInfoService.GetCharacteristicAsync(Guid.Parse("00002a2b-0000-1000-8000-00805F9B34FB"));
try
{
if (characteristic != null)
{
var sbnew = new StringBuilder("BLE Characteristics\n");
byte[] senddata = Encoding.UTF8.GetBytes(string.IsNullOrEmpty(SendMessageLabel.Text) ? "12" : SendMessageLabel.Text);
characteristic.ValueUpdated += (o, args) =>
{
var bytes = characteristic.Value;
};
await characteristic.WriteAsync(senddata);
string str = Encoding.UTF8.GetString(senddata);
sbnew.AppendLine($"Characteristics found on this device: {string.Join(", ", str.ToString())}");
CharactericsLabel.Text = sbnew.ToString();
}
}
catch (Exception ex)
{
//return ex.Message;
DisplayAlert("Notice", ex.Message.ToString(), "OK");
}
I tried delay and I also tried to get write without response from peripheral but it doesn't work. This is my peripheral code
// Current Time characteristic
BluetoothGattCharacteristic currentTime = new BluetoothGattCharacteristic(CURRENT_TIME,
//Read-only characteristic, supports notifications
BluetoothGattCharacteristic.PROPERTY_READ | BluetoothGattCharacteristic.PROPERTY_NOTIFY | BluetoothGattCharacteristic.PROPERTY_WRITE_NO_RESPONSE,
BluetoothGattCharacteristic.PERMISSION_READ | BluetoothGattCharacteristic.PROPERTY_WRITE);
BluetoothGattDescriptor configDescriptor = new BluetoothGattDescriptor(CLIENT_CONFIG,
//Read/write descriptor
BluetoothGattDescriptor.PERMISSION_READ | BluetoothGattDescriptor.PERMISSION_WRITE);
currentTime.addDescriptor(configDescriptor);
// Local Time Information characteristic
BluetoothGattCharacteristic localTime = new BluetoothGattCharacteristic(LOCAL_TIME_INFO,
//Read-only characteristic
BluetoothGattCharacteristic.PROPERTY_READ,
BluetoothGattCharacteristic.PERMISSION_READ);
BluetoothGattCharacteristic sampleText = new BluetoothGattCharacteristic sampleText = new BluetoothGattCharacteristic(SAMPLE_TEXT,
//Read-only characteristic
BluetoothGattCharacteristic.PROPERTY_WRITE_NO_RESPONSE | BluetoothGattCharacteristic.PROPERTY_READ | BluetoothGattCharacteristic.PROPERTY_NOTIFY,
BluetoothGattCharacteristic.PERMISSION_WRITE | BluetoothGattCharacteristic.PERMISSION_READ);
I have no clue how to fix this any suggestions.I tried even Semaphore but it didn't help as you can see in my code
private static async Task<string> WriteAndWaitForResponseAsync(
ICharacteristic characteristic,
byte[] senddata)
{
var semaphore = new SemaphoreSlim(0, 1);
string result = null;
characteristic.ValueUpdated += (o, args) =>
{
var bytes = characteristic.Value;
result = Encoding.UTF8.GetString(bytes); // Note I don't know if this is your intended behaviour with the values you get back, you can decide what to actually do with the response.
// Notify a value has been received.
semaphore.Release();
};
await characteristic.WriteAsync(senddata,new CancellationToken(true)).ConfigureAwait(false);
// Wait until we receive a notification.
await semaphore.WaitAsync(); // I strongly suggest you look in to CancellationTokens but I am not going in to that now.
return result;
}

I suspect that threading is part of the problem here and that you subscribe for the event, call WriteAsync but then you ultimately leave the method before the data/event has been received. I suggest you need to try blocking the method from leaving before you receive your data.
Try something like:
private static async Task<string> WriteAndWaitForResponseAsync(
ICharacteristic characteristic,
byte[] senddata)
{
var semaphore = new SemaphoreSlim(0, 1);
string result = null;
characteristic.ValueUpdated += (o, args) =>
{
// Make sure you read from the new value in event not the existing characteristic.
var bytes = args.Characteristic.Value;
result = Encoding.UTF8.GetString(bytes); // Note I don't know if this is your intended behaviour with the values you get back, you can decide what to actually do with the response.
// Notify a value has been received.
semaphore.Release();
};
await characteristic.WriteAsync(senddata);
// Wait until we receive a notification.
await semaphore.WaitAsync(); // I strongly suggest you look in to CancellationTokens but I am not going in to that now.
return result;
}
Then you can call this method from within your current one like:
string str = await WriteAndWaitForResponseAsync(characteristic, senddata);

How to optimize this code and call Api every 40ms

I want to interrogate one sensor that returns a JSON Rest Api response. I make Api call every 40 milliseconds but it gave me this error :
in System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) in System.Threading.Tasks.Task1.get_Result()
I have timer where interval = 40. And this is the code how I call tha Api :
private void Timer(object sender, EventArgs e)
{
tmrPollingSick.Stop();
string strJson = "";
HttpClient client = new HttpClient();
string baseUrl = "http://9999.99999.99999.8";
client.BaseAddress = new Uri(baseUrl);
var contentType = new MediaTypeWithQualityHeaderValue("application/json");
client.DefaultRequestHeaders.Accept.Add(contentType);
string strAltezza = string.Empty;
try
{
strJson = "Here I set HEADERS... DATA ect " + Convert.ToChar(34) +
"header" + Convert.ToChar(34) + ": {............"
var contentData = new StringContent(strJson, System.Text.Encoding.UTF8, "application/json");
using (var responseMessage = client.PostAsync("/bla/bla/bla", contentData).Result)
{
if (responseMessage.IsSuccessStatusCode)
{
string strContext = responseMessage.Content.ReadAsStringAsync().Result;
Object dec = JsonConvert.DeserializeObject(strContext); // deserializing Json string (it will deserialize Json string)
JObject obj = JObject.Parse(strContext);
//Process Data In
JObject obj1 = JObject.Parse(obj["bla"].ToString());
JObject obj2 = JObject.Parse(obj1["processDataIn"].ToString());
strAltezza = obj2["1"].ToString();
textBox1.Text = strAltezza;
}
}
}
catch(WebException ex1)
{
MessageBox.Show("web: "+ex1.StackTrace.ToString() + " - " + ex1.Message);
}
catch (Exception ex)
{
MessageBox.Show(ex.StackTrace.ToString() + " - " + ex.Message);
}
tmrPollingSick.Start();
}
Everything works fine but after a while it gives me that error.
I allready read this (How to implement real time data for a web page and this) but I haven't tried them yet.
Any suggestions how to fix this?
Is there another way how to get the result in real-time without crashing?

May I baptize this as stubborn pooling?
You don't want to use a timer. What you want is x time between request-response cycles. (This will solve the socket exhaustion).
Split your code into phases (client init, request fetch, response processing). See #Oliver answer.
Make a function to execute everything. And run some sort of infinite foreach loop where you can sleep for x time after calling the fetch function (and the process, but you could defer this to another thread or do it async).

When you call this method all 40ms you'll run out of send sockets, because you create every time a new HttpClient. Even putting this into a using statement (cause HttpClient implements IDisposable) wouldn't solve this problem, cause the underlying socket will be blocked for 3 minutes from the OS (take a look at this answer for further explanations).
You should split this stuff into some initialization phase where you setup the client, build up the request as far as possible and within this timer method just call the PostAsync() method and check the response.

Forcing an asynchronous parallel loop to stop immediately- C#

Here is my method for requesting information from Azures FaceAPI.
I finally realized that in order to make my application work best, a bunch of my security camera frames must be grabbed in parallel, then sent away to the Neural Network to be analyzed.
(Note: it's an Alexa custom skill, so it times out around 8-10 seconds)
Because I grab a bunch of frames in parallel, and asynchronously, there is no way to know which of the images will return a decent face detection. However, when a good detection is found, and the request return Face Data, there is no way to stop the rest of the information from returning.
This happens because the Security camera images were sent in Parallel, they are gone, the info is coming back no matter what.
You'll see that I'm able to use "thread local variables" to capture information and send it back to the function scoped "imageAnalysis" variable to serialize and allow Alexa to describe people in the security image. BUT, because the loop is in Parallel, it doesn't break right away.
It may only take a second or two, but I'm on a time limit thanks to Alexas strict time-out policies.
There doesn't seem to be a way to break the parallel loop immediately...
Or is there?
The more time is spent collecting the "imageAnalysis" Data, the longer Alexa has to wait for a response. She doesn't wait long, and it's important to try and send as many possible images for analysis as possible before Alexa times-out, and also keeping under the Azure FaceAPI limits.
public static async Task<List<Person>> DetectPersonAsync()
{
ConfigurationDto config = Configuration.Configuration.GetSettings();
string imageAnalysis;
using (var cameraApi = new SecurityCameraApi())
{
byte[] imageData = cameraApi.GetImageAsByte(config.SecurityCameraUrl +
config.SecurityCameraStaticImage +
DateTime.Now);
//Unable to get an image from the Security Camera
if (!imageData.Any())
{
Logger.LogInfo(Logger.LogType.Info, "Unable to aquire image from Security Camera \n");
return null;
}
Logger.LogInfo(Logger.LogType.Info, "Attempting Image Face Detection...\n");
Func<string, bool> isEmptyOrErrorAnalysis = s => s.Equals("[]") || s.Contains("error");
imageAnalysis = "[]";
List<byte[]> savedImageList = cameraApi.GetListOfSavedImagesAsByte();
if (savedImageList.Any())
{
Parallel.ForEach(savedImageList, new ParallelOptions
{
MaxDegreeOfParallelism = 50
},
async (image, loopState) =>
{
string threadLocalImageAnalysis = "[]";
if (!loopState.IsStopped)
threadLocalImageAnalysis = await GetImageAnalysisAsync(image, config);
if (!isEmptyOrErrorAnalysis(threadLocalImageAnalysis))
{
imageAnalysis = threadLocalImageAnalysis;
loopState.Break();
}
});
}
// Don't do to many image analysis - or Alexa will time-out.
Func<List<byte[]>, int> detectionCount =
savedImageListDetections => savedImageListDetections.Count > 5 ? 0 : 16;
//Continue with detections of current Camera image frames
if (isEmptyOrErrorAnalysis(imageAnalysis))
{
Parallel.For(0, detectionCount(savedImageList), new ParallelOptions
{
MaxDegreeOfParallelism = 50
},
async (i, loopState) =>
{
imageData = cameraApi.GetImageAsByte(config.SecurityCameraUrl +
config.SecurityCameraStaticImage +
DateTime.Now);
string threadLocalImageAnalysis = "[]";
if (!loopState.IsStopped)
threadLocalImageAnalysis = await GetImageAnalysisAsync(imageData, config);
if (!isEmptyOrErrorAnalysis(threadLocalImageAnalysis))
{
imageAnalysis = threadLocalImageAnalysis;
loopState.Break();
}
});
}
}
try
{
//Cognitive sense has found elements(people) in the image
return new NewtonsoftJsonSerializer().DeserializeFromString<List<Person>>(imageAnalysis);
}
catch (Exception ex)
{
//No elements(people) detected from the camera stream
Logger.LogInfo(Logger.LogType.Info,
string.Format(
"Microsoft Cognitive Sense Face Api Reports: \n{0} \nNo people in the CCTV Camera Image.\n",
ex.Message));
return new List<Person>(); //Empty
}
}
private static async Task<string> GetImageAnalysisAsync(byte[] image, ConfigurationDto config)
{
string json;
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key",
config.MicrosoftCognitiveSenseApiKey);
// Request parameters.
const string queryString =
"returnFaceId=true" +
"&returnFaceLandmarks=false" +
"&returnFaceAttributes=age,gender,accessories,hair";
const string uri =
"https://westus.api.cognitive.microsoft.com/face/v1.0/detect?" + queryString;
using (var content = new ByteArrayContent(image))
{
content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
HttpResponseMessage response = await client.PostAsync(uri, content);
json = await response.Content.ReadAsStringAsync();
try
{
Logger.LogInfo(Logger.LogType.Info, json);
}
catch
{
}
}
}
return json;
}

How to know which links I have visited when making calls in parallel foreach?

I am bulding a web-scraping project.
I have two lists:
private ConcurrentQueue<string> links = new ConcurrentQueue<string>();
private ConcurrentQueue<string> Visitedlinks = new ConcurrentQueue<string>();
On for all the links that I find on a page and one which will hold all the links I have scrapped.
Method that handels the business:
public async Task GetUrlContent(string url)
{
var page = string.Empty;
try
{
page = await service.Get(url);
if (page != string.Empty)
{
Regex regex = new Regex(#"<a[^>]*?href\s*=\s*[""']?([^'"" >]+?)[ '""][^>]*?>",
RegexOptions.Singleline | RegexOptions.CultureInvariant);
if (regex.IsMatch(page))
{
Console.WriteLine("Downloading url: " + url);
for (int i = 0; i < regex.Matches(page).Count; i++)
{
if (regex.Matches(page)[i].Groups[1].Value.StartsWith("/"))
{
if (!links.Contains(BaseUrl + regex.Matches(page)[i].Groups[1].Value.ToLower().Replace(".html", "")) &&
!Visitedlinks.Contains(BaseUrl + regex.Matches(page)[i].Groups[1].Value.ToLower()))
{
Uri ValidUri = GetUrl(regex.Matches(page)[i].Groups[1].Value);
if (ValidUri != null && HostUrls.Contains(ValidUri.Host))
links.Enqueue(regex.Matches(page)[i].Groups[1].Value.ToLower().Replace(".html", ""));
else
links.Enqueue(BaseUrl + regex.Matches(page)[i].Groups[1].Value.ToLower().Replace(".html", ""));
}
}
}
}
var results = links.Where(m => !Visitedlinks.Contains(m)); // problkem here, get multiple values
if (!results.Any())
{
// do nothing
}
else
{
Parallel.ForEach(results, new ParallelOptions { MaxDegreeOfParallelism = 4 },
webpage =>
{
if (ValidUrl(webpage))
{
if (!Visitedlinks.Contains(webpage))
{
Visitedlinks.Enqueue(webpage);
GetUrlContent(webpage).Wait();
}
}
});
}
}
}
catch (Exception e)
{
throw;
}
}
Problem is here:
var results = links.Where(m => !Visitedlinks.Contains(m));
The first iteration I might get:
Link1, link2, link3, link4,
Second iteration:
Link2 link3 link4, link5, link6 ,link 7
Third:
Link 3, link4, link 5, link 6, etc
This means that I will get the same links multiple times since this is a parallel foreach which does several operations at once. I can't figure out how to make sure that I dont get multiple values.
Anyone that can lend a helping hand?

If I understand correctly, the first queue contains the links you want to scrape, and the second queue contains the ones you have scraped.
The problem is that you're trying to iterate over the contents of your ConcurrentQueue:
var results = links.Where(m => !Visitedlinks.Contains(m));
This won't work predictably if you're accessing these queues from multiple threads.
What you should do is take items out of the queue and process them. What stands out is that TryDequeue doesn't appear anywhere in your code. Items are going into the queue but never coming out. The whole purpose of a queue is that we put things in and take them out. ConcurrentQueue makes it safe for multiple threads to put items in and take them out without stepping all over each other.
If you dequeue a link that you want to process:
string linkToProcess = null;
if(links.TryDequeue(out linkToProcess)) // if this returns false, the queue was empty
{
// process it
}
Then as soon as you've taken an item out of the queue to process it, it won't be in the queue anymore. Other threads don't have to check to see if an item has been processed. They just take the next item out of the queue, if there is one. Two threads won't ever take the same item out of the queue. Only one thread can take a given item out of the queue, because as soon as it does, the item isn't in the queue anymore.

Thanks to #Scott Hannen
The final solution is as follows:
Parallel.ForEach(links, new ParallelOptions { MaxDegreeOfParallelism = 25 },
webpage =>
{
try
{
if (WebPageValidator.ValidUrl(webpage))
{
string linkToProcess = webpage;
if (links.TryDequeue(out linkToProcess) && !Visitedlinks.Contains(linkToProcess))
{
Task obj = Scrape(linkToProcess);
Visitedlinks.Enqueue(linkToProcess);
}
}
}
catch (Exception e)
{
log.Error("Error occured: " + e.Message);
Console.WriteLine("Error occured, check log for further details.");
}