Develop Reference

How to get parallel access to the method for multiple users

My telegram bot is necessary so that the user can answer questions in order and save these answers in the same order for a specific user in parallel.
static readonly ConcurrentDictionary<int, string[]> Answers = new ConcurrentDictionary<int, string[]>();
static void Main(string[] args)
{
try
{
Task t1 = CreateHostBuilder(args).Build().RunAsync();
Task t2 = BotOnMessage();
await Task.WhenAll(t1, t2);
}
catch (Exception ex)
{
Console.WriteLine("Error" + ex);
}
}
here is my BotOnMessage() method to receive and process messages from users
async static Task BotOnMessage()
{
int offset = 0;
int timeout = 0;
try
{
await bot.SetWebhookAsync("");
while (true)
{
var updates = await bot.GetUpdatesAsync(offset, timeout);
foreach (var update in updates)
{
var message = update.Message;
if (message.Text == "/start")
{
Registration(message.Chat.Id.ToString(), message.Chat.FirstName.ToString(), createdDateNoTime.ToString("yyyy-MM-dd HH:mm:ss.fff", CultureInfo.InvariantCulture));
var replyKeyboard = new ReplyKeyboardMarkup
{
Keyboard = new[]
{
new[]
{
new KeyboardButton("eng"),
new KeyboardButton("ger")
},
}
};
replyKeyboard.OneTimeKeyboard = true;
await bot.SendTextMessageAsync(message.Chat.Id, "choose language", replyMarkup: replyKeyboard);
}
switch (message.Text)
{
case "eng":
var replyKeyboardEN = new ReplyKeyboardMarkup
{
Keyboard = new[]
{
new[]
{
new KeyboardButton("choice1"),
new KeyboardButton("choice2")
},
}
};
replyKeyboardEN.OneTimeKeyboard = true;
await bot.SendTextMessageAsync(message.Chat.Id, "Enter choice", replyMarkup: replyKeyboardEN);
await AnonymEN();
break;
case "ger":
var replyKeyboardGR = new ReplyKeyboardMarkup
{
Keyboard = new[]
{
new[]
{
new KeyboardButton("choice1.1"),
new KeyboardButton("choice2.2")
},
}
};
replyKeyboardGR.OneTimeKeyboard = true;
await bot.SendTextMessageAsync(message.Chat.Id, "Enter choice", replyMarkup: replyKeyboardGR);
await AnonymGR();
break;
}
offset = update.Id + 1;
}
}
}
catch (Exception ex)
{
Console.WriteLine("Error" + ex);
}
}
and AnonymEN() method for eng case in switch. The problem appears here when I call this method from switch case in BotOnMessage(). Until switch (message.Text) multiple users can asynchronously send messages and get response. When first user enters AnonymEN() second user can't get response from this method until first user will finish it till the end. Also I call BotOnMessage() in the end of AnonymEN() to get back for initial point with possibility to start bot again. For the ordered structure of questions and answers I used ConcurrentDictionary way from here Save user messages sent to bot and send finished form to other user. Any suggestion and solution how to edit code to make this bot available for multiple users at one time?
async static Task AnonymEN()
{
int offset = 0;
int timeout = 0;
try
{
await bot.SetWebhookAsync("");
while (true)
{
var updates = await bot.GetUpdatesAsync(offset, timeout);
foreach (var update in updates)
{
var message = update.Message;
int userId = (int)message.From.Id;
if (message.Type == MessageType.Text)
{
if (Answers.TryGetValue(userId, out string[] answers))
{
var title = message.Text;
if (answers[0] == null)
{
answers[0] = message.Text;
await bot.SendTextMessageAsync(message.Chat, "Enter age");
}
else
{
SaveMessage(message.Chat.Id.ToString(), "anonym", "anonym", "anonym", answers[0].ToString(), title.ToString(), createdDateNoTime.ToString("yyyy-MM-dd HH:mm:ss.fff", CultureInfo.InvariantCulture));
Answers.TryRemove(userId, out string[] _);
await bot.SendTextMessageAsync(message.Chat.Id, "ty for request click /start");
await BotOnMessage();
}
}
else if (message.Text == "choice1")
{
Answers.TryAdd(userId, new string[1]);
await bot.SendTextMessageAsync(message.Chat.Id, "Enter name");
}
}
offset = update.Id + 1;
}
}
}
catch (Exception ex)
{
Console.WriteLine("Error" + ex);
}
}

I can see multiple issues with your code:
It is hard to read. While this is a personal preference I strongly advise to write short concise methods that have 1 responsibility. This will make it easier to understand and maintain your code. https://en.wikipedia.org/wiki/Single-responsibility_principle
Everything is static. This makes it very hard to keep track of any state such as language that should be tracked per user.
Using infinite loops and recursion with no escape. I highly doubt this was intended but you could get an infinite chain of calls like this BotOnMessage -> AnonymEN -> BotOnMessage -> AnonymEN. I think you want to exit the AnonymEN function using either a return, break or while(someVar) approach instead of calling the BotOnMessage function.
If two users are sending messages you get mixed responses. Example message flow user1: /start, user1: eng, user2: hello. The bot will now give an english response to user2. I'm sure this is not intended
The code below is a minimal example that addresses the issues I mentioned. It is not perfect code but should help you get started.
private Dictionaty<string, UserSession> userSessions = new ();
async Task BotOnMessage()
{
try
{
while(true)
{
var message = await GetMessage(timeout);
var userSession = GetUserSession(message.user);
userSession.ProcessMessage(message);
}
}
catch(){}
}
async void GetUserSession(string user)
{
if(!userSessions.HasKey(user))
{
userSessions[user](new Session());
}
return userSessions[user];
}
public class UserSession
{
public async Task ProcessMessage(message)
{
// Existing message processing code goes here.
// Do not use a loop or recursion.
// Instead track the state (e.g. languge) using fields.
}
}

How does WhenAll works with Tasks

I have the following code inside my asp.net console application, to call a method named gettingCustomerInfo in parallel of 20 requests, as follow:
class Program
{
static SemaphoreSlim throttler = new SemaphoreSlim(initialCount: 20);
private static async Task<ScanInfo> gettingCustomerInfo(string website,
string phone, long companyId)
{
await throttler.WaitAsync();
ScanInfo si = new ScanInfo() { companyId = companyId };
try
{
//code goes here...
}
finally
{
throttler.Release();
}
}
static async Task Main(string[] args)
{
bool hasmore = true;
string offset = string.Empty;
bool first = true;
try
{
while (hasmore || first)
{
Marketing ipfd = new Marketing();
first = false;
try
{
// call the PM API to get the account id
using (WebClient wc = new WebClient())
{
wc.Encoding = Encoding.UTF8;
string url = "https://api.hubapi.com/companies/v2/companies/" +
"paged?hapikey=******properties=website" +
"&properties=mse_scan&properties=phone" +
"&limit=" + 100 + "&offset=" + offset;
string tempurl = url.Trim();
var json = wc.DownloadString(tempurl);
ipfd = JsonConvert.DeserializeObject<Marketing>(json);
int rrr = ipfd.companies.Count();
offset = ipfd.offset;
hasmore = ipfd.hasmore;
}
}
catch (Exception e)
{
}
var tasks = ipfd.companies
.Select(c => Task.Run(() => gettingCustomerInfo
(
c.properties.website.value,
(
c.properties.phone != null &&
!String.IsNullOrEmpty(c.properties.phone.value) ?
c.properties.phone.value : null
),
c.companyId
)
));
var results = await Task.WhenAll(tasks);
}
}
}
}
Now I got lost of how the logic and the flow of my code will run? Now inside the API call I am getting 100 items, then inside the gettingCustomerInfo I have defined a SemaphoreSlim = 20. but I am not sure how the flow of my code will be? Can anyone advice?
Edit-1
now i added the following inside my gettingCustomerInfo method:-
private static async Task<ScanInfo> gettingCustomerInfo(string website,string phone, long compnayId)
{
Console.WriteLine("a");
await throttler.WaitAsync();
ScanInfo si = new ScanInfo() { companyId = compnayId };
Console.WriteLine("b");
and i got this pattern:-
a
a
a
a
a
a
a
b
b
a
b
a
b
b
b
a
b
a
b
b
a
b
b
b
a
b
a
b
a
b
a
b
a
b

now inside the api call i am getting 100 items, then inside the gettingCustomerInfo i have defined a SemaphoreSlim = 20. but i am not sure how the flow of my code will be?
So companies has 100 items, and those items are each Selected into tasks.
When this code calls Task.WhenAll, all 100 tasks are created, and gettingCustomerInfo is invoked 100 times.
The first 20 calls to await throttler.WaitAsync will complete immediately and continue executing, running //code goes here. As each one completes, it will call Release, which completes one of the 80 calls waiting at await throttler.WaitAsync.
Eventually, all 100 calls will finish running //code goes here (no more than 20 at a time), and then the await Task.WhenAll completes.

Google Cloud Speech API streaming limit at 65 seconds, not 5 minutes

I am using Google's Cloud Speech API in C# to stream input from a microphone and output text. It works well up to the point where it crashes at 65 seconds, with the error saying that there's a 65 minute request limit. This website https://cloud.google.com/speech-to-text/quotas says that my limit should be about 5 minutes. The function I used is below, and the seconds parameter is how long the stream is requested to last. Am I using old methods or could it be because I'm on the free trial version for the API key?
I've seen other Stack Overflow questions similar to this, and they link to the same website saying the streaming limit is supposed to be about one minute, yet the site has updated within the last year and a half but my limit is still one minute.
public static async Task<object> StreamingMicRecognizeAsync(int seconds)
{
if (NAudio.Wave.WaveIn.DeviceCount < 1)
{
Console.WriteLine("No microphone!");
return -1;
}
var speech = SpeechClient.Create();
var streamingCall = speech.StreamingRecognize();
// Write the initial request with the config.
await streamingCall.WriteAsync(
new StreamingRecognizeRequest()
{
StreamingConfig = new StreamingRecognitionConfig()
{
Config = new RecognitionConfig()
{
Encoding =
RecognitionConfig.Types.AudioEncoding.Linear16,
SampleRateHertz = 16000,
LanguageCode = "en",
},
InterimResults = true,
}
});
// Print responses as they arrive.
Task printResponses = Task.Run(async () =>
{
while (await streamingCall.ResponseStream.MoveNext(
default(CancellationToken)))
{
foreach (var result in streamingCall.ResponseStream
.Current.Results)
{
foreach (var alternative in result.Alternatives)
{
if (alternative.Confidence != 0)
{
Console.WriteLine(alternative.Transcript);
}
}
}
}
});
// Read from the microphone and stream to API.
object writeLock = new object();
bool writeMore = true;
var waveIn = new NAudio.Wave.WaveInEvent();
waveIn.DeviceNumber = 0;
waveIn.WaveFormat = new NAudio.Wave.WaveFormat(16000, 1); //(hertz, channels)
waveIn.DataAvailable +=
(object sender, NAudio.Wave.WaveInEventArgs args) =>
{
lock (writeLock)
{
if (!writeMore) return;
streamingCall.WriteAsync(
new StreamingRecognizeRequest()
{
AudioContent = Google.Protobuf.ByteString
.CopyFrom(args.Buffer, 0, args.BytesRecorded)
}).Wait();
}
};
waveIn.StartRecording();
Console.WriteLine("Speak now.");
await Task.Delay(TimeSpan.FromSeconds(seconds));
waveIn.StopRecording();
lock (writeLock) writeMore = false;
await streamingCall.WriteCompleteAsync();
await printResponses;
return 0;
}
EDIT: A product manager from Google replied to my post on a Google discussion page. He said that the 5 minutes advertised is a mistake, yet they are planning on extending the streaming limit soon. Is there a clean way to make multiple streaming requests that are within the 65 second limit yet also doesn't miss the user's speaking or cut words in half between requests?

How do I optimize calculating the hash of thousands of files?

I have a .Net program that runs through a directory containing tens of thousands of relatively small files (around 10MB each), calculates their MD5 hash and stores that data in an SQLite database. The whole process works fine, however it takes a relatively long time (1094353ms with around 60 thousand files) and I'm looking for ways to optimize it. Here are the solutions I've thought of:
Use additional threads and calculate the hash of more than one file simultaneously. Not sure how I/O speed would limit me with this one.
Use a better hashing algorithm. I've looked around and the one I'm currently using seems to be the fastest one (on C# at least).
Which would be the best approach, and are there any better ones?
Here's my current code:
private async Task<string> CalculateHash(string file, System.Security.Cryptography.MD5 md5) {
Task<string> MD5 = Task.Run(() =>
{
{
using (var stream = new BufferedStream(System.IO.File.OpenRead(file), 1200000))
{
var hash = md5.ComputeHash(stream);
var fileMD5 = string.Concat(Array.ConvertAll(hash, x => x.ToString("X2")));
return fileMD5;
}
};
});
return await MD5;
}
public async Main() {
using (var md5 = System.Security.Cryptography.MD5.Create()) {
foreach (var file in Directory.GetFiles(path)) {
var hash = await CalculateHash(file, md5);
// Adds `hash` to the database
}
}
}

Create a pipeline of work, the easiest way I know how to create a pipeline that uses both parts of the code that must be single threaded and parts that must be multi-threaded is to use TPL Dataflow
public static class Example
{
private class Dto
{
public Dto(string filePath, byte[] data)
{
FilePath = filePath;
Data = data;
}
public string FilePath { get; }
public byte[] Data { get; }
}
public static async Task ProcessFiles(string path)
{
var getFilesBlock = new TransformBlock<string, Dto>(filePath => new Dto(filePath, File.ReadAllBytes(filePath))); //Only lets one thread do this at a time.
var hashFilesBlock = new TransformBlock<Dto, Dto>(dto => HashFile(dto),
new ExecutionDataflowBlockOptions{MaxDegreeOfParallelism = Environment.ProcessorCount, //We can multi-thread this part.
BoundedCapacity = 50}); //Only allow 50 byte[]'s to be waiting in the queue. It will unblock getFilesBlock once there is room.
var writeToDatabaseBlock = new ActionBlock<Dto>(WriteToDatabase,
new ExecutionDataflowBlockOptions {BoundedCapacity = 50});//MaxDegreeOfParallelism defaults to 1 so we don't need to specifiy it.
//Link the blocks together.
getFilesBlock.LinkTo(hashFilesBlock, new DataflowLinkOptions {PropagateCompletion = true});
hashFilesBlock.LinkTo(writeToDatabaseBlock, new DataflowLinkOptions {PropagateCompletion = true});
//Queue the work for the first block.
foreach (var filePath in Directory.EnumerateFiles(path))
{
await getFilesBlock.SendAsync(filePath).ConfigureAwait(false);
}
//Tell the first block we are done adding files.
getFilesBlock.Complete();
//Wait for the last block to finish processing its last item.
await writeToDatabaseBlock.Completion.ConfigureAwait(false);
}
private static Dto HashFile(Dto dto)
{
using (var md5 = System.Security.Cryptography.MD5.Create())
{
return new Dto(dto.FilePath, md5.ComputeHash(dto.Data));
}
}
private static async Task WriteToDatabase(Dto arg)
{
//Write to the database here.
}
}
This creates a pipeline with 3 segments.
One that is single threaded that reads the files from the hard drive in to memory and stored as a byte[].
A second one that can use up to Enviorement.ProcessorCount threads to hash the files, it will only allow 50 items to be sitting on it's inbound queue, when the first block tries to add it will stop processing new items until the next block is ready to accept new items.
And a third one that is single threaded and adds the data to the database, it allows only 50 items in it's inbound queue at a time.
Because of the two 50 limits there will be at most 100 byte[] in memory (50 in hashFilesBlock queue, 50 in the writeToDatabaseBlock queue, items currently being processed count toward the BoundedCapacity limit.
Update: for fun I wrote a version that reports progress too, it's untested though and uses C# 7 features.
using System;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using System.Threading.Tasks.Dataflow;
public static class Example
{
private class Dto
{
public Dto(string filePath, byte[] data)
{
FilePath = filePath;
Data = data;
}
public string FilePath { get; }
public byte[] Data { get; }
}
public static async Task ProcessFiles(string path, IProgress<ProgressReport> progress)
{
int totalFilesFound = 0;
int totalFilesRead = 0;
int totalFilesHashed = 0;
int totalFilesUploaded = 0;
DateTime lastReported = DateTime.UtcNow;
void ReportProgress()
{
if (DateTime.UtcNow - lastReported < TimeSpan.FromSeconds(1)) //Try to fire only once a second, but this code is not perfect so you may get a few rapid fire.
{
return;
}
lastReported = DateTime.UtcNow;
var report = new ProgressReport(totalFilesFound, totalFilesRead, totalFilesHashed, totalFilesUploaded);
progress.Report(report);
}
var getFilesBlock = new TransformBlock<string, Dto>(filePath =>
{
var dto = new Dto(filePath, File.ReadAllBytes(filePath));
totalFilesRead++; //safe because single threaded.
return dto;
});
var hashFilesBlock = new TransformBlock<Dto, Dto>(inDto =>
{
using (var md5 = System.Security.Cryptography.MD5.Create())
{
var outDto = new Dto(inDto.FilePath, md5.ComputeHash(inDto.Data));
Interlocked.Increment(ref totalFilesHashed); //Need the interlocked due to multithreaded.
ReportProgress();
return outDto;
}
},
new ExecutionDataflowBlockOptions{MaxDegreeOfParallelism = Environment.ProcessorCount, BoundedCapacity = 50});
var writeToDatabaseBlock = new ActionBlock<Dto>(arg =>
{
//Write to database here.
totalFilesUploaded++;
ReportProgress();
},
new ExecutionDataflowBlockOptions {BoundedCapacity = 50});
getFilesBlock.LinkTo(hashFilesBlock, new DataflowLinkOptions {PropagateCompletion = true});
hashFilesBlock.LinkTo(writeToDatabaseBlock, new DataflowLinkOptions {PropagateCompletion = true});
foreach (var filePath in Directory.EnumerateFiles(path))
{
await getFilesBlock.SendAsync(filePath).ConfigureAwait(false);
totalFilesFound++;
ReportProgress();
}
getFilesBlock.Complete();
await writeToDatabaseBlock.Completion.ConfigureAwait(false);
ReportProgress();
}
}
public class ProgressReport
{
public ProgressReport(int totalFilesFound, int totalFilesRead, int totalFilesHashed, int totalFilesUploaded)
{
TotalFilesFound = totalFilesFound;
TotalFilesRead = totalFilesRead;
TotalFilesHashed = totalFilesHashed;
TotalFilesUploaded = totalFilesUploaded;
}
public int TotalFilesFound { get; }
public int TotalFilesRead{ get; }
public int TotalFilesHashed{ get; }
public int TotalFilesUploaded{ get; }
}

As far as I understand, Task.Run will instantiate a new thread for every file you have there, which leads to lots of threads and context switching between them. The case like you describe, sounds like a good case for using Parallel.For or Parallel.Foreach, something like this:
public void CalcHashes(string path)
{
string GetFileHash(System.Security.Cryptography.MD5 md5, string fileName)
{
using (var stream = new BufferedStream(System.IO.File.OpenRead(fileName), 1200000))
{
var hash = md5.ComputeHash(stream);
var fileMD5 = string.Concat(Array.ConvertAll(hash, x => x.ToString("X2")));
return fileMD5;
}
}
ParallelOptions options = new ParallelOptions();
options.MaxDegreeOfParallelism = 8;
Parallel.ForEach(filenames, options, fileName =>
{
using (var md5 = System.Security.Cryptography.MD5.Create())
{
GetFileHash(md5, fileName);
}
});
}
EDIT: Seems Parallel.ForEach does not actually do the partitioning automatically. Added max degree of parallelism limit to 8. As a result:
107005 files
46628 ms

Calculate loading time of multiple websites using AsyncTask in Xamarin Android and C#.Net

I have a string array which contains addresses of websites:
string[] arr = new string[]
{
"https://www/google.com",
"https://www.yahoo.com",
"https://www.microsoft.com"
};
I have to send these URLs as argument to the asynctask method so that I will be able to calculate the loading time of each website. I don't have to show the website pages, so I am not using webview.
I can use stopwatch or httprequest to calculate the loading time and my ultimate goal is that all the websites need to start loading at the same time asynchronously, and output has to look like the following
Loading time
google - 00:00:04:092345 (hr:min:sec:millisec) yahoo - 00:00:06:028458
How can I send an array to asynctask and how I can generate loading time without using await?

Here is a brief solution of what you could do.
This is not complete nor perfect. It will will give you the loading time of one URL. Also there is a suggestion of how you could extend this to multiple URLs.
You will need a WebView, either in code or from UI.
Load the URL into the WebView using webview.LoadUrl("https://www/google.com");.
Create a new class by extending it from WebViewClient as follows:
public class myWebViewClient : WebViewClient
{
public override void OnPageFinished(WebView view, string url)
{
base.OnPageFinished(view, url);
Console.WriteLine("OnPageFinished for url : " + url + " at : " + DateTime.Now);
}
}
In your OnCreate() method add the following line of code :
webview.SetWebViewClient(new myWebViewClient());
So from here what you have to do is, Create a Dictionary with URL as key and Loading time as value. Set all the loading time to 0 initially. Update the value corresponding to each URL in the OnPageFinished(). Create an async Task function which would return you the populated dictionary.
public async Task<Dictionary<string, double>> myAsyncFunction()
{
await Task.Delay(5); //to make it async
//Wait till all the OnPageFinished events have fired.
while (myDictionary.Any(x=>x.Value == 0) == true)
{
//there are still websites which have not fully loaded.
await Task.Delay(1); //wait a millisecond before checking again
}
return myDictionary;
}
You can call myAsyncFunction() in a seprate thread than your UI and implement the ContinueWith() or just let it run in a separate thread and write that output into somewhere that you can check when required.
eg : Task.Run(async () => await myAsyncFunction());
UPDATE : based on OP's comments
In the UI thread :
var myClassList = new List<myClass>
{
new myClass{URL = "https://www/google.com", TimeTaken = null},
new myClass{URL = "https://www.yahoo.com", TimeTaken = null},
new myClass{URL = "https://www.microsoft.com", TimeTaken = null}
};
Console.WriteLine("Started at : " + DateTime.Now.ToShortTimeString());
var business = new BusinessLogic();
var loadtimetask = business.GetLoadTimeTakenAsync(myClassList);
await loadtimetask;
Console.WriteLine("Completed at : " + DateTime.Now.ToShortTimeString());
And implementation class :
public async Task<List<myClass>> GetLoadTimeTakenAsync(List<myClass> myClassList)
{
Parallel.ForEach(myClassList, myClassObj =>
{
using (var client = new HttpClient())
{
myClassObj.StartTime = DateTime.Now;
var stream = client.GetStreamAsync(myClassObj.URL)
.ContinueWith((s) =>
{
if (s.IsCompleted)
{
var myClassObjCompleted = myClassList.Where(x => x.URL == myClassObj.URL).First();
myClassObjCompleted.EndTime = DateTime.Now;
myClassObjCompleted.TimeTaken = myClassObj.EndTime - myClassObj.StartTime;
}
});
Task.Run(async () => await stream);
}
});
while (myClassList.Any(x => x.TimeTaken == null))
{
await Task.Delay(1);
}
return myClassList;
}

//Create TextView to display status of Wifi
TextView wifitext = FindViewById<TextView>(Resource.Id.WifiTextView);
//Configuring Wifi connection
var connectivityManager = (ConnectivityManager)GetSystemService(ConnectivityService);
var activeConnection = connectivityManager.ActiveNetworkInfo;
if (activeConnection != null && activeConnection.IsConnected)
{
wifitext.Text = "WIFI AVAILABLE";
string[] urladdress = new string[] { "https://www.google.com/", "https://www.yahoo.com/"};
for (int i = 0; i < urladdress.Length; i++)
{
string url = urladdress[i];
//Call async method
Task returnedTask = Task_MethodAsync(url);
}
}
else
wifitext.Text = "WIFI UNAVAILABLE";
}
public async Task Task_MethodAsync(string url)
{
LinearLayout ll = FindViewById<LinearLayout>(Resource.Id.linearLayout1);
WebClient client = new WebClient();
Stopwatch stopwatch = new Stopwatch();
stopwatch.Start();
Stream listurl = client.OpenRead(url);
StreamReader reader = new StreamReader(listurl);
stopwatch.Stop();
// listurl.Close();
var time = Convert.ToString(stopwatch.Elapsed);