Develop Reference

How to create a task that watches the size of a file in a c# console application

I'm working on a C# project that will need to create text files and fill them with guids using an array of 25 tasks. I want to use another task separate from the writing tasks to monitor the size of the file at 0.5 second intervals.
I'm struggling to figure out a way to do this. I know with a C# WPF application I could use the FILEIO class but I don't think I have access to that using a console app.
What could I use instead of the FILEIO class to create this task and how would I create it?

you can create a new Thread and use the Length property of a System.IO.FileInfo object to get the size of a file.
See this link for more details on Threading.

How about something like this. Everything is asynchronous. The only thread created monitors the count (so that you can see what's going on - which is the point of this, I think).
First I create a class like this:
internal class GuidWriter
{
public string Folder { get; }
public string FileNamePattern { get; }
public int CountOfFiles { get; }
public int CountOfGuids { get; }
public int TotalGuidsToWrite { get; }
private int _currentGuidCount = 0;
public int CurrentGuidCount { get { return _currentGuidCount; } }
public GuidWriter(string folder, string fileNamePattern, int countOfFiles, int countOfGuids)
{
Folder = folder;
FileNamePattern = fileNamePattern;
CountOfFiles = countOfFiles;
CountOfGuids = countOfGuids;
TotalGuidsToWrite = countOfFiles * countOfGuids;
}
public async Task WriteAllFiles()
{
var writeTaskList = new List<Task>(CountOfFiles);
for (int i = 0; i < CountOfFiles; i++)
{
var writeTask = WriteGuidsToOneFile(i + 1);
writeTaskList.Add(writeTask);
}
await Task.WhenAll(writeTaskList);
}
private async Task WriteGuidsToOneFile(int fileNumber)
{
var fileName = string.Format(FileNamePattern, fileNumber);
var filePathName = Path.Combine(Folder, fileName);
using (StreamWriter writer = File.CreateText(filePathName))
{
for (var i = 0; i < CountOfGuids; ++i)
{
var guidString = Guid.NewGuid().ToString();
await writer.WriteLineAsync(guidString);
Interlocked.Increment(ref _currentGuidCount);
}
}
}
}
You create one of these things, tell it how it should work (in the constructor) and then let it rip in the WriteAllFiles (which is async, and which you should await).
It uses Interlocked.Increment on the Guid count - if the framework doesn't anything in threads, I want to make sure that I don't mess up the count.
Then in the main program, I create a Monitor function that counts the number of Guids created and outputs them to the console. It runs in a separate thread, and will go away when it's CancellationToken is canceled:
public static async Task MonitorGuidProduction (GuidWriter guidWriter, CancellationToken cancellationToken)
{
while (!cancellationToken.IsCancellationRequested)
{
var monitorString = $"{guidWriter.CurrentGuidCount}/{guidWriter.TotalGuidsToWrite}";
Console.WriteLine(monitorString);
await Task.Delay(500);
}
}
Then, in the main program, I put it all together:
static async Task Main(string[] args)
{
using (var cancelTokenSource = new CancellationTokenSource())
{
var cancelToken = cancelTokenSource.Token;
var folder = #"C:\temp\guids";
var pattern = "GuidFile{0}.txt";
var guidWriter = new GuidWriter(folder, pattern, CountOfFiles, CountOfGuids);
var monitorTask = Task.Run(() => MonitorGuidProduction(guidWriter, cancelToken));
cancelToken));
await guidWriter.WriteAllFiles();
cancelTokenSource.Cancel();
await monitorTask;
Console.WriteLine($"{guidWriter.CurrentGuidCount} Guids written to {CountOfFiles} files");
Console.ReadLine();
}
}
I create an instance of a GuidWriter and, just before I let it go (by calling guidWriter.WriteAllFiles, I start up my monitor function in another thread. I do that by running Task.Run, but not awaiting its result until everything else is done (getting the cleanup just right).
Then I await the call to guidWriter.WriteAllFiles, and when it completes, I signal my cancellation token to stop the monitor task/thread.

Parallel.For and httpclient crash the application C#

I want to avoid application crashing problem due to parallel for loop and httpclient but I am unable to apply solutions that are provided elsewhere on the web due to my limited knowledge of programming. My code is pasted below.
class Program
{
public static List<string> words = new List<string>();
public static int count = 0;
public static string output = "";
private static HttpClient Client = new HttpClient();
public static void Main(string[] args)
{
//input path strings...
List<string> links = new List<string>();
links.AddRange(File.ReadAllLines(input));
List<string> longList = new List<string>(File.ReadAllLines(#"a.txt"));
words.AddRange(File.ReadAllLines(output1));
System.Net.ServicePointManager.DefaultConnectionLimit = 8;
count = longList.Count;
//for (int i = 0; i < longList.Count; i++)
Task.Run(() => Parallel.For(0, longList.Count, new ParallelOptions { MaxDegreeOfParallelism = 5 }, (i, loopState) =>
{
Console.WriteLine(i);
string link = #"some link" + longList[i] + "/";
try
{
if (!links.Contains(link))
{
Task.Run(async () => { await Download(link); }).Wait();
}
}
catch (System.Exception e)
{
}
}));
//}
}
public static async Task Download(string link)
{
HtmlAgilityPack.HtmlDocument document = new HtmlDocument();
document.LoadHtml(await getURL(link));
//...stuff with html agility pack
}
public static async Task<string> getURL(string link)
{
string result = "";
HttpResponseMessage response = await Client.GetAsync(link);
Console.WriteLine(response.StatusCode);
if(response.IsSuccessStatusCode)
{
HttpContent content = response.Content;
var bytes = await response.Content.ReadAsByteArrayAsync();
result = Encoding.UTF8.GetString(bytes);
}
return result;
}
}
There are solutions for example this one, but I don't know how to put await keyword in my main method, and currently the program simply exits due to its absence before Task.Run(). As you can see I have already applied a workaround regarding async Download() method to call it in main method.
I have also doubts regarding the use of same instance of httpclient in different parallel threads. Please advise me whether I should create new instance of httpclient each time.

You're right that you have to block tasks somewhere in a console application, otherwise the program will just exit before it's complete. But you're doing this more than you need to. Aim for just blocking the main thread and delegating the rest to an async method. A good practice is to create a method with a signature like private async Task MainAsyc(args), put the "guts" of your program logic there, call it from Main like this:
MainAsync(args).Wait();
In your example, move everything from Main to MainAsync. Then you're free to use await as much as you want. Task.Run and Parallel.For are explicitly consuming new threads for I/O bound work, which is unnecessary in the async world. Use Task.WhenAll instead. The last part of your MainAsync method should end up looking something like this:
await Task.WhenAll(longList.Select(async s => {
Console.WriteLine(i);
string link = #"some link" + s + "/";
try
{
if (!links.Contains(link))
{
await Download(link);
}
}
catch (System.Exception e)
{
}
}));
There is one little wrinkle here though. Your example is throttling the parallelism at 5. If you find you still need this, TPL Dataflow is a great library for throttled parallelism in the async world. Here's a simple example.
Regarding HttpClient, using a single instance across threads is completely safe and highly encouraged.

C# Async Tasks wait vs await

There is something that is bugging me for a while, something that happen in my code I can't understand.
I've defined a workflow to extract information from the Facebook API, basically I have 3 different async tasks.
Execute Reports
private static async Task<List<DataTypes.ReportRun>> ExecuteMarketingReports(FacebookClient fb, List<DataTypes.Request.Batch> batch_requests)
{
//Do stuff - Make async POST requests to
//FB to execute reports on FB server side
}
Monitorize Reports
private static async Task<List<DataTypes.AdReportRun>> MonitorizeMarketingReports(FacebookClient fb, List<DataTypes.ReportRun> tListReportRun)
{
//Do stuff -- Check if the reports are ready or not, and return a list with status
}
GetReportData
private static async Task GetReportData(FacebookClient fb, List<DataTypes.AdReportRun> tListReportRun, DateTime since, DateTime until, string breakdown)
{
//Do stuff - Gets report thata once the reports are finish and create the files
}
This is the main Task where all the other methods are called
private static async Task PullInsightsData(FacebookClient fb, List<DataTypes.Request.Batch> batchRequests, DateTime since, DateTime until, string breakdown)
{
var tResult = new List<DataTypes.AdReportRun>();
int retry = 0;
List<DataTypes.AdReportRun> tReportCompleteList = new List<DataTypes.AdReportRun>();
List<DataTypes.AdReportRun> tReportIncompleteList = new List<DataTypes.AdReportRun>();
var report_ids = await ExecuteMarketingReports(fb, batchRequests);
Thread.Sleep(20000); // Waits 20 seconds before get the info.
do
{
/*Start monitorizing the reports*/
var tReport_info = await MonitorizeMarketingReports(fb, report_ids);
/*Get the reports that are complete*/
tReportCompleteList = tReport_info.Where(x => x.async_percent_completion == 100).ToList();
if (tReportCompleteList.Count > 0)
await GetReportData(fb, tReportCompleteList, since, until, breakdown);
tReportIncompleteList = tReport_info.Where(x => x.async_percent_completion < 100).ToList();
report_ids = (from x in tReportIncompleteList
select new DataTypes.ReportRun { report_run_id = x.id }).ToList();
var sleepTime = TimeSpan.FromSeconds(Math.Pow(2, retry + 1));
Thread.Sleep(sleepTime);
retry++;
} while (report_ids.Count > 0 && retry < 8);
}
I call my Main task in this foreach loop, and this is where the problem occurs.
for (int i = 0; i < ActiveAdAccounts.Count; i = i + 50)
{
var AdAccountsSubList = ActiveAdAccounts.Skip(i).Take(50).ToList();
var batchRequests = ....
await PullInsightsData(fb, batchRequests, (DateTime)since, (DateTime)until, breakdown.Replace(",", "_"));
//tTaskList.Add(PullInsightsData(fb, batchRequests, (DateTime)since, (DateTime)until, breakdown.Replace(",", "_")));
}
//Task.WaitAll(tTaskList);
I don't understand why the foreach loop does't continue using the await the console application closes suddenly, shouldn't await "wait" until the task is finish and so then proceed to the next line of code?
I've solved the problem putting all the tasks into a list and waiting for all, but I would like an explanation for this.
[EDITED] Question was edited to create a minimal reproducible example.

Give code limited amount of time to execute within

I have a line of code in an app that looks like this:
string botMessage = chatterBotSession.Think(input);
It's querying a chat bot service and stores the response in botMessage. Sometimes however, the chat bot can take too long to think of a response.
Is there a way I can run the code like normal, but if it doesn't complete in one second cancel it and then run some handler that can alert the user that the service took too long?
Like, normally I might do this:
string botMessage = chatterBotSession.Think(input);
Console.WriteLine("The bot responded with: " + botMessage);
but if the bot is slow the second line doesn't get executed (fast enough). How can I limit the bot's "thinking" time to one second, and run the rest of the code (that would normally run) as soon as it completes if it was successful or run a separate bit of code to display an error message if it hasn't completed.

You can wrap the bot's service call in a Task.Run call and wait for it a predefined amount of time. It would look something like this
static void Main(string[] args)
{
var task = Task.Run(() => chatterBotSession.Think(input));
if (task.Wait(1000))
{
Console.WriteLine(task.Result);
}
else
{
Console.WriteLine("Couldn't get an answer in a timely manner");
}
Console.ReadLine();
}

It is very easy to limit the task execution by using CancellationTokenSource with timeout:
var cancellationToken = new CancellationTokenSource();
var task = chatterBotSession.Think(cancellationToken.Token);
cancellationToken.CancelAfter(TimeSpan.FromMilliseconds(1000)); // cancel after 1sec
await task;
In Think method you should add call to CancellationToken.ThrowIfCancellationRequested

Calling Bot:
static void Main(string[] args)
{
TalkingBot bot = new TalkingBot();
try
{
Console.WriteLine("Thinking started...");
Console.WriteLine(bot.Think("Some input...", 2000));
}
catch (Exception ex)
{
Console.WriteLine("Error: {0}", ex.Message);
}
Console.ReadLine();
}
And Bot itself:
class TalkingBot
{
string input = null;
int timeout = 0;
string asyncThnikResult = null;
public string Think(string input, int timeout)
{
DateTime timeLimit = DateTime.Now.AddMilliseconds(timeout);
this.input = input;
this.timeout = timeout;
System.Threading.Thread thread = new System.Threading.Thread(new System.Threading.ThreadStart(AsyncThnik));
thread.Start();
//wait for result, in this case
while (string.IsNullOrEmpty(asyncThnikResult))
{
if (timeLimit <= DateTime.Now)
{
throw new Exception("Timeout occured!");
}
System.Threading.Thread.Sleep(10);
}
//return result...
return this.asyncThnikResult;
}
/// <summary>
/// Do your thing async...
/// </summary>
void AsyncThnik()
{
string temp = "This value will never be returned due to timeout limit...";
System.Threading.Thread.Sleep(timeout + 1000); //add second to produce timeout...
this.asyncThnikResult = temp;
}
}

Why is this TAP async/await code slower than the TPL version?

I had to write a console application that called Microsoft Dynamics CRM web service to perform an action on over eight thousand CRM objects. The details of the web service call are irrelevant and not shown here but I needed a multi-threaded client so that I could make calls in parallel. I wanted to be able to control the number of threads used from a config setting and also for the application to cancel the whole operation if the number of service errors reached a config-defined threshold.
I wrote it using Task Parallel Library Task.Run and ContinueWith, keeping track of how many calls (threads) were in progress, how many errors we'd received, and whether the user had cancelled from the keyboard. Everything worked fine and I had extensive logging to assure myself that threads were finishing cleanly and that everything was tidy at the end of the run. I could see that the program was using the maximum number of threads in parallel and, if our maximum limit was reached, waiting until a running task completed before starting another one.
During my code review, my colleague suggested that it would be better to do it with async/await instead of tasks and continuations, so I created a branch and rewrote it that way. The results were interesting - the async/await version was almost twice as slow, and it never reached the maximum number of allowed parallel operations/threads. The TPL one always got to 10 threads in parallel whereas the async/await version never got beyond 5.
My question is: have I made a mistake in the way I have written the async/await code (or the TPL code even)? If I have not coded it wrong, can you explain why the async/await is less efficient, and does that mean it is better to carry on using TPL for multi-threaded code.
Note that the code I tested with did not actually call CRM - the CrmClient class simply thread-sleeps for a duration specified in the config (five seconds) and then throws an exception. This meant that there were no external variables that could affect the performance.
For the purposes of this question I created a stripped down program that combines both versions; which one is called is determined by a config setting. Each of them starts with a bootstrap runner that sets up the environment, creates the queue class, then uses a TaskCompletionSource to wait for completion. A CancellationTokenSource is used to signal a cancellation from the user. The list of ids to process is read from an embedded file and pushed onto a ConcurrentQueue. They both start off calling StartCrmRequest as many times as max-threads; subsequently, every time a result is processed, the ProcessResult method calls StartCrmRequest again, keeping going until all of our ids are processed.
You can clone/download the complete program from here: https://bitbucket.org/kentrob/pmgfixso/
Here is the relevant configuration:
<appSettings>
<add key="TellUserAfterNCalls" value="5"/>
<add key="CrmErrorsBeforeQuitting" value="20"/>
<add key="MaxThreads" value="10"/>
<add key="CallIntervalMsecs" value="5000"/>
<add key="UseAsyncAwait" value="True" />
</appSettings>
Starting with the TPL version, here is the bootstrap runner that kicks off the queue manager:
public static class TplRunner
{
private static readonly CancellationTokenSource CancellationTokenSource = new CancellationTokenSource();
public static void StartQueue(RuntimeParameters parameters, IEnumerable<string> idList)
{
Console.CancelKeyPress += (s, args) =>
{
CancelCrmClient();
args.Cancel = true;
};
var start = DateTime.Now;
Program.TellUser("Start: " + start);
var taskCompletionSource = new TplQueue(parameters)
.Start(CancellationTokenSource.Token, idList);
while (!taskCompletionSource.Task.IsCompleted)
{
if (Console.KeyAvailable)
{
if (Console.ReadKey().Key != ConsoleKey.Q) continue;
Console.WriteLine("When all threads are complete, press any key to continue.");
CancelCrmClient();
}
}
var end = DateTime.Now;
Program.TellUser("End: {0}. Elapsed = {1} secs.", end, (end - start).TotalSeconds);
}
private static void CancelCrmClient()
{
CancellationTokenSource.Cancel();
Console.WriteLine("Cancelling Crm client. Web service calls in operation will have to run to completion.");
}
}
Here is the TPL queue manager itself:
public class TplQueue
{
private readonly RuntimeParameters parameters;
private readonly object locker = new object();
private ConcurrentQueue<string> idQueue = new ConcurrentQueue<string>();
private readonly CrmClient crmClient;
private readonly TaskCompletionSource<bool> taskCompletionSource = new TaskCompletionSource<bool>();
private int threadCount;
private int crmErrorCount;
private int processedCount;
private CancellationToken cancelToken;
public TplQueue(RuntimeParameters parameters)
{
this.parameters = parameters;
crmClient = new CrmClient();
}
public TaskCompletionSource<bool> Start(CancellationToken cancellationToken, IEnumerable<string> ids)
{
cancelToken = cancellationToken;
foreach (var id in ids)
{
idQueue.Enqueue(id);
}
threadCount = 0;
// Prime our thread pump with max threads.
for (var i = 0; i < parameters.MaxThreads; i++)
{
Task.Run((Action) StartCrmRequest, cancellationToken);
}
return taskCompletionSource;
}
private void StartCrmRequest()
{
if (taskCompletionSource.Task.IsCompleted)
{
return;
}
if (cancelToken.IsCancellationRequested)
{
Program.TellUser("Crm client cancelling...");
ClearQueue();
return;
}
var count = GetThreadCount();
if (count >= parameters.MaxThreads)
{
return;
}
string id;
if (!idQueue.TryDequeue(out id)) return;
IncrementThreadCount();
crmClient.CompleteActivityAsync(new Guid(id), parameters.CallIntervalMsecs).ContinueWith(ProcessResult);
processedCount += 1;
if (parameters.TellUserAfterNCalls > 0 && processedCount%parameters.TellUserAfterNCalls == 0)
{
ShowProgress(processedCount);
}
}
private void ProcessResult(Task<CrmResultMessage> response)
{
if (response.Result.CrmResult == CrmResult.Failed && ++crmErrorCount == parameters.CrmErrorsBeforeQuitting)
{
Program.TellUser(
"Quitting because CRM error count is equal to {0}. Already queued web service calls will have to run to completion.",
crmErrorCount);
ClearQueue();
}
var count = DecrementThreadCount();
if (idQueue.Count == 0 && count == 0)
{
taskCompletionSource.SetResult(true);
}
else
{
StartCrmRequest();
}
}
private int GetThreadCount()
{
lock (locker)
{
return threadCount;
}
}
private void IncrementThreadCount()
{
lock (locker)
{
threadCount = threadCount + 1;
}
}
private int DecrementThreadCount()
{
lock (locker)
{
threadCount = threadCount - 1;
return threadCount;
}
}
private void ClearQueue()
{
idQueue = new ConcurrentQueue<string>();
}
private static void ShowProgress(int processedCount)
{
Program.TellUser("{0} activities processed.", processedCount);
}
}
Note that I am aware that a couple of the counters are not thread safe but they are not critical; the threadCount variable is the only critical one.
Here is the dummy CRM client method:
public Task<CrmResultMessage> CompleteActivityAsync(Guid activityId, int callIntervalMsecs)
{
// Here we would normally call a CRM web service.
return Task.Run(() =>
{
try
{
if (callIntervalMsecs > 0)
{
Thread.Sleep(callIntervalMsecs);
}
throw new ApplicationException("Crm web service not available at the moment.");
}
catch
{
return new CrmResultMessage(activityId, CrmResult.Failed);
}
});
}
And here are the same async/await classes (with common methods removed for the sake of brevity):
public static class AsyncRunner
{
private static readonly CancellationTokenSource CancellationTokenSource = new CancellationTokenSource();
public static void StartQueue(RuntimeParameters parameters, IEnumerable<string> idList)
{
var start = DateTime.Now;
Program.TellUser("Start: " + start);
var taskCompletionSource = new AsyncQueue(parameters)
.StartAsync(CancellationTokenSource.Token, idList).Result;
while (!taskCompletionSource.Task.IsCompleted)
{
...
}
var end = DateTime.Now;
Program.TellUser("End: {0}. Elapsed = {1} secs.", end, (end - start).TotalSeconds);
}
}
The async/await queue manager:
public class AsyncQueue
{
private readonly RuntimeParameters parameters;
private readonly object locker = new object();
private readonly CrmClient crmClient;
private readonly TaskCompletionSource<bool> taskCompletionSource = new TaskCompletionSource<bool>();
private CancellationToken cancelToken;
private ConcurrentQueue<string> idQueue = new ConcurrentQueue<string>();
private int threadCount;
private int crmErrorCount;
private int processedCount;
public AsyncQueue(RuntimeParameters parameters)
{
this.parameters = parameters;
crmClient = new CrmClient();
}
public async Task<TaskCompletionSource<bool>> StartAsync(CancellationToken cancellationToken,
IEnumerable<string> ids)
{
cancelToken = cancellationToken;
foreach (var id in ids)
{
idQueue.Enqueue(id);
}
threadCount = 0;
// Prime our thread pump with max threads.
for (var i = 0; i < parameters.MaxThreads; i++)
{
await StartCrmRequest();
}
return taskCompletionSource;
}
private async Task StartCrmRequest()
{
if (taskCompletionSource.Task.IsCompleted)
{
return;
}
if (cancelToken.IsCancellationRequested)
{
...
return;
}
var count = GetThreadCount();
if (count >= parameters.MaxThreads)
{
return;
}
string id;
if (!idQueue.TryDequeue(out id)) return;
IncrementThreadCount();
var crmMessage = await crmClient.CompleteActivityAsync(new Guid(id), parameters.CallIntervalMsecs);
ProcessResult(crmMessage);
processedCount += 1;
if (parameters.TellUserAfterNCalls > 0 && processedCount%parameters.TellUserAfterNCalls == 0)
{
ShowProgress(processedCount);
}
}
private async void ProcessResult(CrmResultMessage response)
{
if (response.CrmResult == CrmResult.Failed && ++crmErrorCount == parameters.CrmErrorsBeforeQuitting)
{
Program.TellUser(
"Quitting because CRM error count is equal to {0}. Already queued web service calls will have to run to completion.",
crmErrorCount);
ClearQueue();
}
var count = DecrementThreadCount();
if (idQueue.Count == 0 && count == 0)
{
taskCompletionSource.SetResult(true);
}
else
{
await StartCrmRequest();
}
}
}
So, setting MaxThreads to 10 and CrmErrorsBeforeQuitting to 20, the TPL version on my machine completes in 19 seconds and the async/await version takes 35 seconds. Given that I have over 8000 calls to make this is a significant difference. Any ideas?

I think I'm seeing the problem here, or at least a part of it. Look closely at the two bits of code below; they are not equivalent.
// Prime our thread pump with max threads.
for (var i = 0; i < parameters.MaxThreads; i++)
{
Task.Run((Action) StartCrmRequest, cancellationToken);
}
And:
// Prime our thread pump with max threads.
for (var i = 0; i < parameters.MaxThreads; i++)
{
await StartCrmRequest();
}
In the original code (I am taking it as a given that it is functionally sound) there is a single call to ContinueWith. That is exactly how many await statements I would expect to see in a trivial rewrite if it is meant to preserve the original behaviour.
Not a hard and fast rule and only applicable in simple cases, but nevertheless a good thing to keep an eye out for.

I think you over complicated your solution and ended up not getting where you wanted in either implementation.
First of all, connections to any HTTP host are limited by the service point manager. The default limit for client environments is 2, but you can increase it yourself.
No matter how much threads you spawn, there won't be more active requests than those allwed.
Then, as someone pointed out, await logically blocks the execution flow.
And finally, you spent your time creating an AsyncQueue when you should have used TPL data flows.

When implemented with async/await, I would expect the I/O bound algorithm to run on a single thread. Unlike #KirillShlenskiy, I believe that the bit responsible for "bringing back" to caller's context is not responsible for the slow-down. I think you overrun the thread pool by trying to use it for I/O-bound operations. It's designed primarily for compute-bound ops.
Have a look at ForEachAsync. I feel that's what you're looking for (Stephen Toub's discussion, you'll find Wischik's videos meaningful too):
http://blogs.msdn.com/b/pfxteam/archive/2012/03/05/10278165.aspx
(Use degree of concurrency to reduce memory footprint)
http://vimeo.com/43808831
http://vimeo.com/43808833