Develop Reference

Adding an ETA to an embedded loop sequence

UPDATE 1:
Some of the solutions offered below seem good. However, I only know the amount of times a loop will iterate after it's parent loop's iterations have been determined. So I can't count all the iterations beforehand.
ORIGINAL QUESTION:
I have embedded loops in a program similar to this:
Prog1:
using System;
using System.Threading;
namespace time_remaining_loop_strip
{
class Program
{
static void Main(string[] args)
{
var random = new Random();
Console.Clear();
// Simulate initiation delay
Console.WriteLine("initiate");
Thread.Sleep(random.Next(100, 1000));
int intCount = random.Next(1, 10);
for (int loop1 = 0; loop1 <= intCount; loop1++)
{
// Simulate loop1 delay
Console.WriteLine("\tloop1");
Thread.Sleep(random.Next(100, 1000));
for (int loop2 = 0; loop2 <= random.Next(1, 10); loop2++)
{
// Simulate loop2 delay
Console.WriteLine("\t\tloop2");
Thread.Sleep(random.Next(100, 1000));
for (int loop3 = 0; loop3 <= random.Next(1, 10); loop3++)
{
// Simulate loop3 delay
Console.WriteLine("\t\t\tloop3");
Thread.Sleep(random.Next(100, 1000));
for (int loop4 = 0; loop4 <= random.Next(1, 10); loop4++)
{
// Simulate loop4 delay
Console.WriteLine("\t\t\t\tloop4");
Thread.Sleep(random.Next(100, 1000));
}
}
}
}
}
}
}
I am trying to display Processing Time Remaining (ETA), so I can see a rough estimate of the amount of time remaining before the loop sequence above completes
I now have another bit of code which does display an ETA which works fine when the loop is very simplistic:
Prog2:
using System;
using System.Threading;
namespace time_remaining
{
class Program
{
public static TimeSpan ComputeRemaining((int count, DateTime time) start, (int count, DateTime time) current, int end) =>
current.count - start.count == 0
? TimeSpan.MaxValue
: TimeSpan.FromSeconds((end - current.count) * current.time.Subtract(start.time).TotalSeconds / (current.count - start.count));
static void Main(string[] args)
{
Console.Clear();
var random = new Random();
int Count = random.Next(10, 60);
DateTime startTime = DateTime.Now;
for (int i = 0; i <= Count; i++)
{
Thread.Sleep(random.Next(100, 2000));
TimeSpan timeRemaining = ComputeRemaining((0, startTime), (i, DateTime.Now), Count);
Console.SetCursorPosition(0,0);
Console.Write("ETA: ");
Console.Write(String.Format("{0} Days, {1} Hours, {2} Minutes, {3} Seconds", timeRemaining.Days.ToString().PadLeft(3,'0'), timeRemaining.Hours.ToString().PadLeft(2,'0'), timeRemaining.Minutes.ToString().PadLeft(2,'0'), timeRemaining.Seconds.ToString().PadLeft(2,'0')));
}
}
}
}
When I try to combine the ETA aspect of Prog1 into Prog2, it does not seem to work well:
Prog3 = Prog1+Prog2:
using System;
using System.Threading;
namespace time_remaining_loop_strip
{
class Program
{
public static TimeSpan ComputeRemaining((int count, DateTime time) start, (int count, DateTime time) current, int end) =>
current.count - start.count == 0
? TimeSpan.MaxValue
: TimeSpan.FromSeconds((end - current.count) * current.time.Subtract(start.time).TotalSeconds / (current.count - start.count));
static void Main(string[] args)
{
DateTime startTime = DateTime.Now;
var random = new Random();
Console.Clear();
// Simulate initiation delay
//Console.WriteLine("initiate");
Thread.Sleep(random.Next(100, 1000));
int intCount = random.Next(1, 10);
for (int loop1 = 0; loop1 <= intCount; loop1++)
{
// Simulate loop1 delay
//Console.WriteLine("\tloop1");
Thread.Sleep(random.Next(100, 1000));
for (int loop2 = 0; loop2 <= random.Next(1, 10); loop2++)
{
// Simulate loop2 delay
//Console.WriteLine("\t\tloop2");
Thread.Sleep(random.Next(100, 1000));
for (int loop3 = 0; loop3 <= random.Next(1, 10); loop3++)
{
// Simulate loop3 delay
//Console.WriteLine("\t\t\tloop3");
Thread.Sleep(random.Next(100, 1000));
for (int loop4 = 0; loop4 <= random.Next(1, 10); loop4++)
{
// Simulate loop4 delay
//Console.WriteLine("\t\t\t\tloop4");
Thread.Sleep(random.Next(100, 1000));
}
}
}
TimeSpan timeRemaining = ComputeRemaining((0, startTime), (loop1, DateTime.Now), intCount);
Console.SetCursorPosition(0,0);
Console.Write("ETA: ");
Console.Write(String.Format("{0} Days, {1} Hours, {2} Minutes, {3} Seconds", timeRemaining.Days.ToString().PadLeft(3,'0'), timeRemaining.Hours.ToString().PadLeft(2,'0'), timeRemaining.Minutes.ToString().PadLeft(2,'0'), timeRemaining.Seconds.ToString().PadLeft(2,'0')));
}
}
}
}
This doesn't seem to work very well at all. It does display an ETA, but it has long delays before it shows anything, because of the way the loop is structured.
How can I update this so the ETA code displays an ETA more accurately and at more predictive intervals, as in every second?

With the concession that you've just built a simple model of what's actually happening (you have some series of nested process of variable delay and count, which cannot even be determined until runtime), what you're asking for currently is a random number predictor.
There will be between n & m cycles (1 & 10 in your example) of between o & p duration (100 & 1000ms), but still random.. well, random quantized into bands. The way you've written it, this is dice-roll random (a dice has no memory), although in practice it seems more likely the duration of one cycle must somewhat imply the duration of the next, (which is how you've written ComputeRemaining) and within one band the count of one loop must help with the count of the next.
So despite its apparent simplicity Prog2 covers our examples.. given a known loop count, where each cycle takes a random duration (which is in fact pick(n,m)^3*pick(o,p).. but this is still just a random number) - predict the end. For reporting purposes, you'll want to refactor this to consider the inner loops as well too, but it's effectively the same process. (^3 is a simplification, it's actually a series of independent picks multiplied)
Ok, so we don't need to talk about time/delays (I mean.. you clearly want that, but it's just some number that represents the future - a TimeSpan is an long count of x ticks since some time... a finish time is just Now + x*tick). So we can simplify this into a long predictor.
Setup
interface IAvger
{
public double Avg { get; }
}
interface IAdder
{
void Add(long value);
}
class Mean
{
private int _count = 0;
public double Total { get; private set; } = 0;
public double? Avg => _count == 0 ? null : (double?)(Total / _count);
public void Add(double val)
{
Total += val;
_count++;
}
}
You can ignore the interfaces (I used them when switching out potential solutions). Class Mean should be familiar... it calculates the mean average of a number of values, scaling/adapting as more values are found.
/// <summary>
/// Equivalent to your ComputeRemaining
/// </summary>
class RunningAvg : IAvger, IAdder
{
private Mean _mean = new Mean();
private readonly double _guess;
public RunningAvg(double guess)
{
_guess = guess;
}
public double Avg => _mean.Avg ?? _guess;
public void Add(long value) => _mean.Add(value);
}
Here's an equivalent to your ComputeRemaining. The value of guess helps an early prediction when nothing else is known (vs the equivalent of TimeSpan.Max)
/// <summary>
/// Drop the lowest and highest value
/// - Fairly typical in stats, however note this is only one biggest/smallest
/// (will work best when the standard devation is low, and outliers
/// are rare)
/// </summary>
class IgnoreExtremes : IAvger, IAdder
{
private long? _worst;
private long? _best;
private Mean _mean = new Mean();
private readonly int _wt;
private readonly double _guess;
public IgnoreExtremes(double guess, int weight = 4)
{
_wt = weight;
_guess = guess;
}
public long Best => _best ?? (long)Math.Round(_guess);
public long Worst => _worst ?? (long)Math.Round(_guess);
public double Avg
{
get
{
var avg = _mean.Avg;
if (!avg.HasValue) return _guess;
return (Best + _wt * avg.Value + Worst) / (2 + _wt);
}
}
public void Add(long value)
{
if (!_best.HasValue)
{
_best = value;
}
else if (value < _best)
{
_mean.Add(_best.Value);
_best = value;
}
else if (!_worst.HasValue)
{
_worst = value;
}
else if (value > _worst)
{
_mean.Add(_worst.Value);
_worst = value;
}
else
{
_mean.Add(value);
}
}
}
Finally some stats! IgnoreExtremes suppresses the highest and lowest (single) value. In scientific sampling it's fairly typical to ignore these, however with a real random distribution of numbers (eg dice roll, or random.Next) only one extreme will be discarded. This should predict better numbers than RunningAvg. Note this is a form of weighted average, you can tune it (slightly) by supplying a weight value at construction (wt=4 is fairly common), or tying _wt to _mean.count (some code changes required)
class IgnoreStdDevOutlier : IAvger, IAdder
{
private const int AT_LEAST = 5;
private Mean _mean = new Mean();
private readonly List<long> _vals = new List<long>();
private readonly double _guess;
//private long _tot;
private readonly double _outlierStdDevMulti;
public IgnoreStdDevOutlier(double guess, double outlierStdDevMulti = 2)
{
_guess = guess;
_outlierStdDevMulti = outlierStdDevMulti;
}
private double StdDev()
{
var avg = Avg;
double tot = 0;
foreach (var item in _vals)
tot += (item - avg) * (item - avg);
return Math.Sqrt(tot / (_vals.Count - 1));
}
public void Add(long value)
{
_vals.Add(value);
if (_vals.Count > AT_LEAST)
{
var avg = Avg;
var sd = StdDev();
var min = avg - _outlierStdDevMulti * sd;
var max = avg + _outlierStdDevMulti * sd;
//Ignore outliers
if (value < min || value > max) return;
}
_mean.Add(value);
}
public double Avg => _mean.Avg ?? 0;
}
Another statistics approach is to ignore values more than n*StandardDeviation from average, where n is often 2 or 3 (you'll find conflicting opinions). All values seen are part of the standard deviation, but only those that aren't outliers are considered part of the average. This ends up acting like a suppression factor, preventing the estimate swinging too much.
Alright, so to run a test we need some sort of measuring class:
class Performance
{
private readonly List<long> _set = new List<long>();
private long _actual;
public void Add(long item) => _set.Add(item);
public void SetFinal(long final) => _actual = final;
public void Report()
{
foreach (var item in _set)
{
Console.WriteLine("{0} {1} = {2}", item, _actual, (item / (double)_actual - 1) * 100);
}
}
}
A real guess can't know the final (_actual) value, but this class allows us to see how the guess is doing so far.
Finally the program class:
class Program
{
const int MIN_MSEC = 100;
const int MAX_MSEC = 1000;
const int MIN_LOOP = 10;
const int MAX_LOOP = 50;
static void C(Random random)
{
int nm = random.Next(MAX_LOOP, MAX_LOOP);
var guess = (double)((MAX_LOOP + MIN_LOOP) / 2 * (MAX_MSEC + MIN_MSEC) / 2);
var predict = new RunningAvg(guess);
//var predict = new IgnoreExtremes(guess);
//var predict = new IgnoreStdDevOutlier(guess,3);
var per = new Performance();
long tot = 0;
for (int i = 0; i <= nm; i++)
{
var op = random.Next(MIN_MSEC, MAX_MSEC);
predict.Add(op);
per.Add((long)Math.Round(predict.Avg * nm));
tot += op;
}
per.SetFinal(tot);
per.Report();
}
static void Main(string[] args)
{
var random = new Random();
C(random);
}
}
You can ignore the work is done in a method called C (just a side effect of the code - A was your Prog1, while B was Prog2). Within C try changing which of RunningAvg, IgnoreExtremes or IgnoreStdDevOutlier is uncommented. Because, again, what's written is dice-roll random, you cannot take a single run as a good benchmark. The next stage is to wrap this in repeat-runs and take the average of the standard deviation of the predictions (or perhaps only the late predictions - a user probably doesn't mind if an early estimate is far off, as long as by the end it isn't jumping around) - but I ran out of time. I find IgnoreStdDevOutlier, on average, converges on the right answer fairly swiftly being off by 0-1% by the end. IgnoreExtremes suffers from only ignoring one extreme (in each direction), so it's sort of a light version of IgnoreStdDevOutlier. If your data isn't random, and there are only occasionally extreme cases - it'll do just fine. RunningAvg doesn't actually perform terribly some of the time, other times it's off by double digit percentages all the way through. Random numbers, if only they were easy to predict.
Note on usage
Timespan.Ticks is a long. All of this is written to predict a long, which can be considered the difference between then and now. To directly switch use new Timespan(long ticks) to build the spans, and DateTime.Now.Subtract(startTime).Ticks to get a long from a span. Obviously all these classes could be re-written for TimeSpan instead of long.. unfortunately there isn't an easy generic where constraint that includes both long and TimeSpan

You can know how many times(t) loop 4 (L4) will execute multiplying L1tL2tL3t*L4t = total. Now at the very beginning you declare
dt = DateTime.Now;
count = 0L;
Now inside L4 you increase count and calculates how much time is elapsed
et= DateTime.Now - dt;
count++;
And there using rule of three you calculate how many seconds are left to reach the total iterations.
remainingSeconds = (total*et.TotalSeconds/count) - et.TotalSeconds;
Now the ETA is
DateTime.Now.AddSeconds(remainingSeconds);

I used below logic to solve this.
Before first foreach each loop I calculated the all different loops count.
After that I created 4 different variables to hold sleep time for each loop which is settings inside them.
Now before first for loop I calculated the total time in milliseconds that will be taken be all loops using below logic:
var totalTime = (firstLoopCount * firstLoopSleepTime) +
(firstLoopCount * secondLoopCount * secondLoopSleepTime) +
(firstLoopCount * secondLoopCount * thirdLoopCount * thirdLoopSleepTime) +
(firstLoopCount * secondLoopCount * thirdLoopCount * fourthLoopCount * fourthLoopSleepTime);
Note: I added 1 in function call (GetTotoalTimeinMilliSecond) where loops count is passing because loop is starting from 0 and ending with different loops count including itself.
Now before loop start, printed total time taken
Inside each loop, after thread sleep, subtract thread sleep time from total time and print that. And set new calculated time as total time.
Repeat this inside every loop.
Below is the code:
class Program
{
static void Main(string[] args)
{
DateTime startTime = DateTime.Now;
var random = new Random();
Console.Clear();
// Simulate initiation delay
//Console.WriteLine("initiate");
Thread.Sleep(random.Next(100, 1000));
int firstLoopCount = random.Next(1, 10);
int secondLoopCount = random.Next(1, 10);
int thirdLoopCount = random.Next(1, 10);
int fourthLoopCount = random.Next(1, 10);
int firstLoopSleepTime = random.Next(100, 1000);
int secondLoopSleepTime =random.Next(100, 1000);
int thirdLoopSleepTime = random.Next(100, 1000);
int fourthLoopSleepTime = random.Next(100, 1000);
//**************Added 1 because loop is starting from 0 and ending with different loops count including itself.
var totalTimeinMillSec = GetTotoalTimeinMilliSecond(firstLoopCount + 1, secondLoopCount + 1, thirdLoopCount + 1, fourthLoopCount + 1, firstLoopSleepTime, secondLoopSleepTime, thirdLoopSleepTime, fourthLoopSleepTime);
PrintAndGetTimeRemaining(totalTimeinMillSec);
for (int loop1 = 0; loop1 <= firstLoopCount; loop1++)
{
// Simulate loop1 delay
//Console.WriteLine("\tloop1");
Thread.Sleep(firstLoopSleepTime);
totalTimeinMillSec = PrintAndGetTimeRemaining(totalTimeinMillSec - firstLoopSleepTime);
for (int loop2 = 0; loop2 <= secondLoopCount; loop2++)
{
// Simulate loop2 delay
//Console.WriteLine("\t\tloop2");
Thread.Sleep(secondLoopSleepTime);
totalTimeinMillSec = PrintAndGetTimeRemaining(totalTimeinMillSec - secondLoopSleepTime);
for (int loop3 = 0; loop3 <= thirdLoopCount; loop3++)
{
// Simulate loop3 delay
//Console.WriteLine("\t\t\tloop3");
Thread.Sleep(thirdLoopSleepTime);
totalTimeinMillSec = PrintAndGetTimeRemaining(totalTimeinMillSec - thirdLoopSleepTime);
for (int loop4 = 0; loop4 <= fourthLoopCount; loop4++)
{
// Simulate loop4 delay
//Console.WriteLine("\t\t\t\tloop4");
Thread.Sleep(fourthLoopSleepTime);
totalTimeinMillSec = PrintAndGetTimeRemaining(totalTimeinMillSec - fourthLoopSleepTime);
}
}
}
}
}
private static int PrintAndGetTimeRemaining(int totalTimeinMillSec)
{
TimeSpan timeRemaining = TimeSpan.FromMilliseconds(totalTimeinMillSec);
Console.SetCursorPosition(0, 0);
Console.WriteLine("ETA: ");
Console.WriteLine(String.Format("{0} Days, {1} Hours, {2} Minutes, {3} Seconds, {4} Milliseconds", timeRemaining.Days.ToString().PadLeft(3, '0'), timeRemaining.Hours.ToString().PadLeft(2, '0'), timeRemaining.Minutes.ToString().PadLeft(2, '0'), timeRemaining.Seconds.ToString().PadLeft(2, '0'), timeRemaining.Milliseconds.ToString().PadLeft(2, '0')));
return totalTimeinMillSec;
}
private static int GetTotoalTimeinMilliSecond(int firstLoopCount, int secondLoopCount, int thirdLoopCount, int fourthLoopCount, int firstLoopSleepTime, int secondLoopSleepTime, int thirdLoopSleepTime, int fourthLoopSleepTime)
{
var totalTime = (firstLoopCount * firstLoopSleepTime +
firstLoopCount * secondLoopCount * secondLoopSleepTime +
firstLoopCount * secondLoopCount * thirdLoopCount * thirdLoopSleepTime +
firstLoopCount * secondLoopCount * thirdLoopCount * fourthLoopCount * fourthLoopSleepTime);
return totalTime;
}
}
}
Please let me know If I missed something.

I think this may work for you; in order to accomplish the solution I created a few classs to help.
The code itself has some comments.
First an enum, to know in which loop we are; it is not fully necessary, but it can be worth for a refactor later on.
public enum LoopEnum
{
loop1,
loop2,
loop3,
loop4
}
Then i created a class called EtaLoop which will contain the entire loop information/logic to know how long a single iteration of a loop takes, I use StopWatch:
public class EtaLoop
{
public readonly int TotalIterations;
private readonly List<long> _loopsTimesInMiliseconds;
private readonly Stopwatch _stopwatch;
public EtaLoop(int totalIterations)
{
//+1 as the loops starts on 0
TotalIterations = totalIterations+1;
_loopsTimesInMiliseconds = new List<long>();
_stopwatch = new Stopwatch();
}
public double AvgExecution()
{
return _loopsTimesInMiliseconds.Any() ? _loopsTimesInMiliseconds.Average(a => a) : 0;
}
public void Start()
{
if(!_stopwatch.IsRunning)
_stopwatch.Start();
}
public void Stop()
{
_stopwatch.Stop();
_loopsTimesInMiliseconds.Add(_stopwatch.ElapsedMilliseconds);
ResetStopWatch();
}
public int CurrentIteration()
{
return _loopsTimesInMiliseconds.Count();
}
public long EstimatedCurrentIteration()
{
return Convert.ToInt64(_loopsTimesInMiliseconds.Average(a => a) * TotalIterations);
}
private void ResetStopWatch()
{
_stopwatch.Reset();
}
}
the methods I think are clear enough without more explanation.
another class called EtaTime which will contain more logic, this class is a wrapper for EtaLoop for example, if a for has 5 iterations, it will contain 5 elements on the list.
each element (EtaLoop) is added to the list, once we finish (all the iterations are finished) It has some more methods, one of them, the "hard one" it is explained
public class EtaTime
{
public readonly List<EtaLoop> Loops;
public readonly LoopEnum Loop;
private EtaLoop _currentLoop;
public EtaTime(LoopEnum loop)
{
Loops = new List<EtaLoop>();
Loop = loop;
}
public void SetUpTotal(int totalIterations)
{
_currentLoop = new EtaLoop(totalIterations);
}
public void StartLoop()
{
_currentLoop.Start();
}
public void EndLoop()
{
_currentLoop.Stop();
}
public void RegisterLoop()
{
Loops.Add(_currentLoop);
}
/// <summary>
/// Get the average time per execution, and the average number of loops per parent loop.
/// The idea is to know how many times (and how long) the loop x is executed per each x-1
/// </summary>
/// <returns></returns>
public (double, double) GetAvgTimeAndAvgExTimes()
{
double avgTimePerLoop = Loops.Any() ? Loops.Average(a => a.AvgExecution()) : _currentLoop.AvgExecution();
double avgTotalLoopsIteration = Loops.Any() ? Loops.Average(a => a.TotalIterations) : _currentLoop.TotalIterations;
return (avgTimePerLoop, avgTotalLoopsIteration);
}
public int GetCurrentIteration()
{
return _currentLoop.CurrentIteration();
}
public int TotalIterations()
{
return _currentLoop.TotalIterations;
}
}
Finally the wrapper for the EtaTimeHelper which will contain all the EtaTimes; originally i was thinking to make a list, thats why the enum was for, but, i think like this is more clear.
Note: this class can be split/moved to extension methods.
The main points here are RegisterLoop every time we finish a loop we have to call that method; and it needs to be here because i need the information from the other loops.
public class EtaTimeHelper
{
//This part can be done in a list, but i think it is easier to see like this.
public readonly EtaTime Loop1;
public readonly EtaTime Loop2;
public readonly EtaTime Loop3;
public readonly EtaTime Loop4;
public readonly DateTime StartTime;
private DateTime _lastPrintTime;
private const int TimeBetweenPrintsInSeconds = 10;
public EtaTimeHelper()
{
Loop1 = new EtaTime(LoopEnum.loop1);
Loop2 = new EtaTime(LoopEnum.loop2);
Loop3 = new EtaTime(LoopEnum.loop3);
Loop4 = new EtaTime(LoopEnum.loop4);
StartTime = DateTime.Now;
_lastPrintTime = DateTime.MinValue;
}
public void RegisterLoop(LoopEnum loopNumber)
{
switch (loopNumber)
{
case LoopEnum.loop1:
Loop1.RegisterLoop();
break;
case LoopEnum.loop2:
Loop2.RegisterLoop();
break;
case LoopEnum.loop3:
Loop3.RegisterLoop();
break;
case LoopEnum.loop4:
Loop4.RegisterLoop();
break;
default:
throw new NotImplementedException("please register the loop");
}
PrintCompletionTime(DateTime.Now, loopNumber);
}
public void PrintCompletionTime(DateTime printTime, LoopEnum loopNumber)
{
if(_lastPrintTime.AddSeconds(TimeBetweenPrintsInSeconds) < printTime)
{
var time = CalculatePredictionTime(loopNumber);
Print(time);
_lastPrintTime = printTime;
}
}
private long CalculatePredictionTime(LoopEnum loopNumber)
{
switch (loopNumber)
{
case LoopEnum.loop1:
return LoopPrediction(Loop1.GetAvgTimeAndAvgExTimes());
case LoopEnum.loop2:
return Loop2Prediction(Loop1, Loop2);
case LoopEnum.loop3:
return Loop3Prediction(Loop1, Loop2, Loop3);
case LoopEnum.loop4:
return Loop4Prediction(Loop1, Loop2, Loop3, Loop4);
default:
throw new NotImplementedException("please register the loop");
}
//If all loops in #1 are finished, all sub loops are also finished. which means, it is the "end of the loop"
long LoopPrediction((double, double) avgTimeAndAvgExTimes)
{
double avgTimePerLoop = avgTimeAndAvgExTimes.Item1;
double avgIterations = avgTimeAndAvgExTimes.Item2;
return Convert.ToInt64(avgTimePerLoop * avgIterations);
}
long Loop2Prediction(EtaTime loop1, EtaTime loop2)
{
var loop1Prediction = LoopPrediction(loop1.GetAvgTimeAndAvgExTimes());
var loop2Values = loop2.GetAvgTimeAndAvgExTimes();
long avgPerLoop = LoopPrediction(loop2Values);
var loop1AvgIterations = loop1.GetAvgTimeAndAvgExTimes().Item2;
var expectedLoop2Iterations = loop1AvgIterations;
double loop2Predictions = avgPerLoop * expectedLoop2Iterations;
if (loop1Prediction == 0)
{
return Convert.ToInt64(loop2Predictions);
}
else
{
//1+current iteration
return loop1Prediction + loop2.GetCurrentIteration();
}
}
long Loop3Prediction(EtaTime loop1, EtaTime loop2, EtaTime loop3)
{
var loop1_2Prediction = Loop2Prediction(loop1, loop2);
var loop3Values = loop3.GetAvgTimeAndAvgExTimes();
long avgPerLoop = LoopPrediction(loop3Values);
var loop2AvgIterations = loop2.GetAvgTimeAndAvgExTimes().Item2;
var loop1AvgIterations = loop1.GetAvgTimeAndAvgExTimes().Item2;
var expectedLoop3Iterations = loop2AvgIterations * loop1AvgIterations;
double loop3Predictions = avgPerLoop * expectedLoop3Iterations;
if (loop1_2Prediction == 0)
{
return Convert.ToInt64(loop3Predictions);
}
else
{
//1-2+current iteration
return loop1_2Prediction+ loop3.GetCurrentIteration();
}
}
long Loop4Prediction(EtaTime loop1, EtaTime loop2, EtaTime loop3, EtaTime loop4)
{
var loop1_2_3Prediction = Loop3Prediction(loop1, loop2, loop3);
var loop4Values = loop4.GetAvgTimeAndAvgExTimes();
long avgPerLoop = LoopPrediction(loop4Values);
var loop2AvgIterations = loop2.GetAvgTimeAndAvgExTimes().Item2;
var loop1AvgIterations = loop1.GetAvgTimeAndAvgExTimes().Item2;
var loop3AvgIterations = loop3.GetAvgTimeAndAvgExTimes().Item2;
var expectedLoop4Iterations = loop2AvgIterations * loop1AvgIterations* loop3AvgIterations;
double loop4Predictions = avgPerLoop * expectedLoop4Iterations;
if (loop1_2_3Prediction == 0)
{
return Convert.ToInt64(loop4Predictions);
}
else
{
//1-2-3+current iteration
return loop1_2_3Prediction + loop4.GetCurrentIteration();
}
}
}
private void Print(long ms)
{
DateTime estimatedCompletionTime = StartTime.AddMilliseconds(ms);
TimeSpan leftTime = (estimatedCompletionTime - DateTime.Now);
Console.WriteLine("ETA: ");
Console.WriteLine($"{leftTime.Days} Days, {leftTime.Hours} Hours, {leftTime.Minutes} Minutes, {leftTime.Seconds} Seconds");//, leftTime.Days.ToString().PadLeft(3, '0'), leftTime.Hours.ToString().PadLeft(2, '0'), leftTime.Minutes.ToString().PadLeft(2, '0'), leftTime.Seconds.ToString().PadLeft(2, '0')));
Console.WriteLine($"on {estimatedCompletionTime.ToString("yyyy/MM/dd HH:mm:ss")}");
Console.WriteLine($"Current Time: {DateTime.Now.ToString("yyyy/MM/dd HH:mm:ss")}");
}
}
and the print. as you mention you want some mecanisim to print, but you did not specify if it is a consol app a web or a winforms app. Probably it is winforms if it is a desktop app you probably can do something like myLabel.SetPropertyThreadSafe to modify the label you want to update.
The algorithim which calculates the time it is on CalculatePredictionTime I tried to make it a single method for n number of loops but I couldnt.
Now the main.
I did a few changes on the Thread.Sleep as it is easier for me to figure it out if makes sense while i was testing. also, a range 10-100 (10x) I thought it was very big, i reduced it for testing reasons.
the code follows a pattern, first you need to instanciate the EtaTimeHelper class.
before each loop you have to setUpTotal which means you have to pass the number of iterations are going to be.
The first line of the loop will be .StartLoop() this will start the StopWatch and the last line of the loop (before the }) will be a .EndLoop();
after the closing bracket } you have to indicate the etaTimeHelper.RegisterLoop(loopNumber); passing as parameter the loop number this will print if necessary the time.
static void Main(string[] args)
{
var random = new Random();
Console.Clear();
// Simulate initiation delay
//Console.WriteLine("initiate");
EtaTimeHelper etaTimeHelper = new EtaTimeHelper();
int intCount = random.Next(1, 10);
etaTimeHelper.Loop1.SetUpTotal(intCount);
for (int loop1 = 0; loop1 <= intCount; loop1++)
{
etaTimeHelper.Loop1.StartLoop();
// Simulate loop1 delay
Console.WriteLine("\tloop1");
Thread.Sleep(random.Next(40, 50));
//number of times the loop 2 is going to execute inside this loop 1;
int loop2times = random.Next(1, 10);
etaTimeHelper.Loop2.SetUpTotal(loop2times);
for (int loop2 = 0; loop2 <= loop2times; loop2++)
{
etaTimeHelper.Loop2.StartLoop();
// Simulate loop2 delay
//Console.WriteLine("\t\tloop2");
Thread.Sleep(random.Next(30, 40));
//number of times the loop 3 is going to execute inside this loop 3;
int loop3times = random.Next(1, 10);
etaTimeHelper.Loop3.SetUpTotal(loop3times);
for (int loop3 = 0; loop3 <= loop3times; loop3++)
{
etaTimeHelper.Loop3.StartLoop();
// Simulate loop3 delay
//Console.WriteLine("\t\t\tloop3");
Thread.Sleep(random.Next(10, 20));
var loop4Times = random.Next(1, 10);
etaTimeHelper.Loop4.SetUpTotal(loop4Times);
for (int loop4 = 0; loop4 <= loop4Times; loop4++)
{
etaTimeHelper.Loop4.StartLoop();
// Simulate loop4 delay
//Console.WriteLine("\t\t\t\tloop4");
Thread.Sleep(random.Next(20, 30));
etaTimeHelper.Loop4.EndLoop();
}
etaTimeHelper.RegisterLoop(LoopEnum.loop4);
etaTimeHelper.Loop3.EndLoop();
}
etaTimeHelper.RegisterLoop(LoopEnum.loop3);
etaTimeHelper.Loop2.EndLoop();
}
etaTimeHelper.RegisterLoop(LoopEnum.loop2);
etaTimeHelper.Loop1.EndLoop();
}
etaTimeHelper.RegisterLoop(LoopEnum.loop1);
}
Here is a "working" fiddle https://dotnetfiddle.net/Z06W4g
note: it times out after a few seconds, in your local machine works fine.

I think you will have to benchmark it a few times for the operation that you need and then extrapolate. There are variables here. The compute power and storage type (if involved) and network (if involved) of the target will all contribute to the ETA and every execution will have a different execution time. But ETA can be closely predicted after some benchmarking. And if you want to be really clever, you can show a warning that it may take more time than the displayed ETA because of x, y and z e.g. CPU utilisation by other processes running on the target.
Jon Skeet is an authority on this topic and he has a good resource here:
link
Also read here for how the types in your operation and bitness of the CPU will affect your ETA:
link

In order to assert the ETA every second , what I have done is attaching an event to a System.Timers.Timer object. The event will be triggered every second in order to continuously providing the user with the ETA feedback.
System.Timers.Timer aTimer = new System.Timers.Timer(1000);
aTimer.Elapsed += (sender, e) => ATimer_Elapsed(sender, e, new MyEventArguments
{
Loop1TotalIterations = intCount,
CurrentIndex = loop1,
Start = startTime
});
aTimer.AutoReset = true;
aTimer.Enabled = true;
In your example we are using the Thread.Sleep in order simulate a delay.
Using Thread.Sleep will put everything to sleep - therefor the time logic will be inaccurate.
What I have done is creating a Multi-thread application - in order to run the logic on one thread and the time on another. Therefor when we pause the time checking and ETA will continue to run.
Putting it all together we have the following:
namespace ConsoleApp10
{
#region Usings
using System;
using System.Threading;
using System.Threading.Tasks;
using System.Timers;
#endregion
/// <summary>
/// Event class to be passed to ElapsedEventHandler
/// </summary>
public class MyEventArguments
{
public int Loop1TotalIterations { get; set; }
public DateTime Start { get; set; }
public DateTime CurrentDate => DateTime.Now;
public int CurrentIndex { get; set; }
}
class Program
{
/// <summary>
/// Compute the remaining time
/// </summary>
public static void ComputeRemaining((int count, DateTime time) start, (int count, DateTime time) current, int end)
{
var T = current.count - start.count == 0
? TimeSpan.MaxValue
: TimeSpan.FromSeconds((end - current.count) * current.time.Subtract(start.time).TotalSeconds / (current.count - start.count));
Console.Clear();
Console.SetCursorPosition(0, 0);
Console.ForegroundColor = (ConsoleColor)15;
Console.WriteLine(String.Format("ETA: {0} Days, {1} Hours, {2} Minutes, {3} Seconds",
T.Days.ToString().PadLeft(3, '0'),
T.Hours.ToString().PadLeft(2, '0'),
T.Minutes.ToString().PadLeft(2, '0'),
T.Seconds.ToString().PadLeft(2, '0')));
Console.WriteLine();
}
static void Main(string[] args)
{
DateTime startTime = DateTime.Now;
Random random = new Random();
int intCount = random.Next(1, 10);
int loop1 = 0;
var time = Task.Factory.StartNew(() =>
{
System.Timers.Timer aTimer = new System.Timers.Timer(1000);
aTimer.Elapsed += (sender, e) => ATimer_Elapsed(sender, e, new MyEventArguments
{
Loop1TotalIterations = intCount,
CurrentIndex = loop1,
Start = startTime
});
aTimer.AutoReset = true;
aTimer.Enabled = true;
});
var logic = Task.Factory.StartNew(() =>
{
PrintAndSimulate("Initiate");
for (loop1 = 0; loop1 <= intCount; loop1++)
{
PrintAndSimulate("\tloop1");
for (int loop2 = 0; loop2 <= random.Next(1, 10); loop2++)
{
PrintAndSimulate("\t\tloop2");
for (int loop3 = 0; loop3 <= random.Next(1, 10); loop3++)
{
PrintAndSimulate("\t\t\tloop3");
for (int loop4 = 0; loop4 <= random.Next(1, 10); loop4++)
{
PrintAndSimulate("\t\t\t\tloop4");
}
}
}
}
});
Task.WaitAll(new[] { time, logic });
}
/// <summary>
/// Display the info string and simulates a delay
/// </summary>
private static void PrintAndSimulate(string info)
{
int time = new Random().Next(100, 1000);
Console.SetCursorPosition(5, 5);
Console.ForegroundColor = (ConsoleColor)new Random().Next(15);
Console.WriteLine(info);
Thread.Sleep(time);
}
/// <summary>
/// ElapsedEventHandler
/// </summary>
private static void ATimer_Elapsed(object sender, ElapsedEventArgs e, MyEventArguments myEventArguments)
{
ComputeRemaining((0, myEventArguments.Start), (myEventArguments.CurrentIndex, myEventArguments.CurrentDate), myEventArguments.Loop1TotalIterations);
}
}
}
I have added some colours to the console feedback - making a change more evident.

This may not be ideal solution but has potential to solve the issue.
First you build a Tree of Action containing what you are going to execute and then execute the Actions by iterating the tree.
The tree acts as index of what you are going to execute. Even you can track what is being executed. Whats done and whats pending.
I have nested the actions in single level in the tree for demo purpose but you can nest the tree as much as possible, inside loops etc.(pseudo code ahead but poc builds and executes well)
class Program
{
static void Main(string[] args)
{
var random = new Random();
TreeNode<Action> root = new TreeNode<Action>(() => { });
var loop1 = root.AddChild(() =>
{
int Count = random.Next(1, 3);
for (int i = 0; i <= Count; i++)
Thread.Sleep(random.Next(100, 2000));
});
var loop2 = loop1.AddChild(() =>
{
int Count = random.Next(1, 3);
for (int i = 0; i <= Count; i++)
Thread.Sleep(random.Next(100, 2000));
});
var loop3 = loop2.AddChild(() =>
{
int Count = random.Next(1, 3);
for (int i = 0; i <= Count; i++)
Thread.Sleep(random.Next(100, 2000));
});
var loop4 = loop3.AddChild(() =>
{
int Count = random.Next(1, 3);
for (int i = 0; i <= Count; i++)
Thread.Sleep(random.Next(100, 2000));
});
var loop5 = loop4.AddChild(() =>
{
int Count = random.Next(1, 3);
for (int i = 0; i <= Count; i++)
Thread.Sleep(random.Next(100, 2000));
});
var loop6 = loop5.AddChild(() =>
{
int Count = random.Next(1, 3);
for (int i = 0; i <= Count; i++)
Thread.Sleep(random.Next(100, 2000));
});
root.Execute(DateTime.Now);
Console.ReadLine();
}
}
public static class Extensions
{
public static string Humanize(this TimeSpan timeRemaining)
{
return String.Format("{0} Days, {1} Hours, {2} Minutes, {3} Seconds", timeRemaining.Days.ToString().PadLeft(3, '0'), timeRemaining.Hours.ToString().PadLeft(2, '0'), timeRemaining.Minutes.ToString().PadLeft(2, '0'), timeRemaining.Seconds.ToString().PadLeft(2, '0'));
}
public static void PrintETA(this TimeSpan timeRemaining)
{
//Console.SetCursorPosition(0, 0);
Console.Write("ETA: ");
Console.Write(timeRemaining.Humanize());
}
public static TimeSpan ComputeRemaining((int count, DateTime time) start, (int count, DateTime time) current, int end) =>
current.count - start.count == 0
? TimeSpan.MaxValue
: TimeSpan.FromSeconds((end - current.count) * current.time.Subtract(start.time).TotalSeconds / (current.count - start.count));
public static void Execute(this TreeNode<Action> root, DateTime startTime)
{
var current = root;
var end = current.Count();
var currentCount = 1;
Iterate:
var Count = current.Children.Count();
for (int i = 0; i < Count; i++)
{
TreeNode<Action> node = current.Children.ElementAt(i);
node.Data();
TimeSpan timeRemaining = ComputeRemaining((0, startTime), (currentCount++, DateTime.Now), end-1);
timeRemaining.PrintETA();
Console.WriteLine();
Console.WriteLine("Processed {0} of {1}", currentCount - 1, end-1);
if (node.Children.Count() > 0)
{
current = node;
goto Iterate;
}
}
}
}
Ref:
TreeNode.cs
Yet another tree structure: https://github.com/gt4dev/yet-another-tree-structure

Updated.
The first version of the answer calculated the values of RemainingTime and TotalTime at the end of every iteration. Given the way the for loops are nested, it could cause long delays between updates. In order to read those values at given intervals, some changes are made.
Let's start with the Loop class. It is used to keep track of the details of every for loop, like the total amount of iterations, the current iteration and the time consumed in every iteration. To acomplish the later, two System.Diagnostic Stopwatch are used. One is kept running free, not reseted, to ease the calculation of the average time for an iteration. The other clock is reseted on every iteration to provide a value for LoopTime, used in calculations on the fly of RemainingTime and TotalTime, when accesed via the property getters. When the iteration ends, signaled by the method StopClock(), the average loop time and the related properties are updated. The values obtained here are more accurate than the ones calculated on the fly.
Its parent, LoopTimer class, is in charge of creating and storing references of Loop instances, start and stop clocks and calculate the global ETA. The method EnterLoop() is used at the beginning of the for loop. It creates a new Loop. The overload EnterLoop() with a single parameter is used in the rest of iterations to retrieve Loop instances stored in a Dictionary. The method ExitLoop(), at the end of the loop, stops the clock and updates calculations.
Functions ExitLoopRetETA() and ExitLoopRetTotalEta() are provided as replacements of ExitLoop(), to print data at the end of the loop, for testing or debugging. Method Bonus() shows how to use it. Method ShowStatus() and function GetStatus provide internal information on the Loop objects.
To show the values periodically, a Task is used, to run DoUpdate() (or DoUpdateTotal() or DoUpdateStatus())in a separate thread.
The target framework is .Net 4.0
The working classes:
using System;
using System.Collections.Generic;
using System.Diagnostics;
namespace LoopTimer
{
public struct TotalEta
{
public TimeSpan Eta;
public TimeSpan Total;
}
internal class LoopTimer
{
// Private helper class
private class Loop
{
// Declarations
private Stopwatch _clock;
private Stopwatch _interval_clock;
// Constructor
public Loop(int index, int count)
{
Index = index;
Count = count;
_clock = new Stopwatch();
_interval_clock = new Stopwatch();
}
// Properties
internal int Index { get; set; }
internal int Count { get; private set; }
private double _loopTimeMs;
internal double LoopTimeMs
{
get
{
if (_clock.IsRunning)
{
return _interval_clock.Elapsed.TotalMilliseconds;
}
else
return _loopTimeMs; ;
}
}
private double _remainingTimeMs;
internal double RemainingTimeMs
{
get
{
if (_clock.IsRunning)
return CalculateRemainingTime();
else
return _remainingTimeMs;
}
}
private double _totalTimeMs;
internal double TotalTimeMs
{
get
{
if (_clock.IsRunning)
return CalculateTotalTime();
else
return _totalTimeMs;
}
}
internal TimeSpan LoopTime
{
get { return TimeSpan.FromMilliseconds(LoopTimeMs); }
}
internal TimeSpan TotalTime
{
get { return TimeSpan.FromMilliseconds(TotalTimeMs); }
}
internal TimeSpan RemainingTime
{
get { return TimeSpan.FromMilliseconds(RemainingTimeMs); }
}
// Methods
internal void StartClock()
{
_clock.Start();
_interval_clock.Start();
}
internal void StopClock()
{
_clock.Stop();
_interval_clock.Stop();
UpdateTimes();
_interval_clock.Reset();
}
private void UpdateTimes()
{
// reading clock
double elapsed = _clock.Elapsed.TotalMilliseconds;
// Calculating average loop time. The Stopwatch is not reseted between iterations.
_loopTimeMs = elapsed / (Index + 1);
// Calculating estimated remaining time = average loop time * remaining iterations.
_remainingTimeMs = CalculateRemainingTime();
// Calculating estimated total time = average loop time * iterations.
_totalTimeMs = CalculateTotalTime();
}
private double CalculateRemainingTime()
{
// Calculating estimated remaining time = average loop time * remaining iterations.
double time;
int countt = Count - Index;
if (countt > 1)
time = LoopTimeMs * countt;
else if (countt == 1)
time = LoopTimeMs;
else
time = 0;
return time;
}
private double CalculateTotalTime()
{
return LoopTimeMs * Count;
}
}
// End Private helper class
// Declarations
private Dictionary<int, Loop> _loopDict;
private int _loopIndex;
// Constructor
public LoopTimer()
{
_loopDict = new Dictionary<int, Loop>();
_loopIndex = -1;
}
// Properties
internal TimeSpan TotalTime
{
get { return TimeSpan.FromMilliseconds(TotalTimeMs); }
}
internal TimeSpan RemainingTime
{
get { return TimeSpan.FromMilliseconds(RemainingTimeMs); }
}
private double TotalTimeMs
{ get { return CalculateTotalTime(); } }
private double RemainingTimeMs
{ get { return CalculateRemainingTime(); } }
// Methods
internal void EnterLoop(int index, int count)
{
Loop newLoop;
// increase index
_loopIndex++;
if (!_loopDict.ContainsKey(_loopIndex))
{
// create new Loop
newLoop = new Loop(index, count);
_loopDict[_loopIndex] = newLoop;
}
else
{ // retrieve Loop from Dict
newLoop = _loopDict[_loopIndex];
}
newLoop.StartClock();
}
internal void EnterLoop(int index)
{
// increase index
_loopIndex++;
// retrive loop & start clock
_loopDict[_loopIndex].Index = index;
_loopDict[_loopIndex].StartClock();
}
internal void ExitLoop()
{ // retrive loop & stop clock
_loopDict[_loopIndex].StopClock();
// decrease index
_loopIndex--;
}
// bonus method
internal TimeSpan ExitLoopRetETA()
{ // retrive loop & stop clock
_loopDict[_loopIndex].StopClock();
// decrease index
_loopIndex--;
return RemainingTime;
}
// bonus method
internal TotalEta ExitLoopRetTotalEta()
{
TotalEta retval;
retval.Eta = ExitLoopRetETA();
retval.Total = TotalTime;
return retval;
}
// debug method
internal void ShowStatus()
{
Console.WriteLine("Status:");
Console.WriteLine(" RemainingTime:");
for (int i = 0; i < _loopDict.Count; i++)
{
TimeSpan time = _loopDict[i].RemainingTime;
Console.WriteLine(string.Format(" Loop: {0} Value: {1}", i, time.ToString()));
}
Console.WriteLine();
}
// debug method
internal TotalEta[] GetStatus()
{
TotalEta[] retArr = new TotalEta[_loopDict.Count];
TotalEta retval;
for (int i = 0; i < _loopDict.Count; i++)
{
retval = new TotalEta();
retval.Eta = _loopDict[i].RemainingTime;
retval.Total = _loopDict[i].TotalTime;
retArr[i] = retval;
}
return retArr;
}
private double CalculateRemainingTime()
{
double max, time;
max = 0;
// Remaining Time, the greater of all
for (int i = 0; i < _loopDict.Count; i++)
{
time = _loopDict[i].RemainingTimeMs;
if (time > max)
max = time;
}
return max;
}
// Total Time, bonus
private double CalculateTotalTime()
{
double max, time;
max = 0;
// Total Time, the greater of all
for (int i = 0; i < _loopDict.Count; i++)
{
time = _loopDict[i].TotalTimeMs;
if (time > max)
max = time;
}
return max;
}
}
}
The sample program:
using System;
using System.Threading;
using System.Threading.Tasks;
namespace LoopTimer
{
class Solution
{
static CancellationTokenSource ts;
static void Main(string[] args)
{
Console.Clear();
LoopTimer lm = new LoopTimer();
var random = new Random();
// For easy change test parameters
int minRndCount = 1;
int maxRndCount = 10;
int minRndSleep = 100;
int maxRndSleep = 1000;
// A task to update console, with cancellation token
ts = new CancellationTokenSource();
Task updater = new Task(() => DoUpdate(lm), ts.Token);
// Uncomment to show estimated total time.
//Task updater = new Task(() => DoUpdateTotal(lm), ts.Token);
// Uncomment to show estimated total time and internal values of every loop.
//Task updater = new Task(() => DoUpdateStatus(lm), ts.Token);
// Simulate initiation delay
Thread.Sleep(random.Next(minRndSleep, maxRndSleep));
// Console.WriteLine("initiate");
updater.Start();
int intCountL1 = random.Next(minRndCount, maxRndCount);
for (int loop1 = 0; loop1 <= intCountL1; loop1++)
{
// Entering loop1
if (loop1 == 0)
lm.EnterLoop(loop1, intCountL1);
else
lm.EnterLoop(loop1);
// Simulate loop1 delay
//Console.WriteLine("\tloop1");
Thread.Sleep(random.Next(minRndSleep, maxRndSleep));
int intCountL2 = random.Next(minRndCount, maxRndCount);
for (int loop2 = 0; loop2 <= intCountL2; loop2++)
{
// Entering loop2
if (loop2 == 0)
lm.EnterLoop(loop2, intCountL2);
else
lm.EnterLoop(loop2);
// Simulate loop2 delay
//Console.WriteLine("\t\tloop2");
Thread.Sleep(random.Next(minRndSleep, maxRndSleep));
int intCountL3 = random.Next(minRndCount, maxRndCount);
for (int loop3 = 0; loop3 <= intCountL3; loop3++)
{
// Entering loop3
if (loop3 == 0)
lm.EnterLoop(loop3, intCountL3);
else
lm.EnterLoop(loop3);
// Simulate loop3 delay
//Console.WriteLine("\t\t\tloop3");
Thread.Sleep(random.Next(minRndSleep, maxRndSleep));
int intCountL4 = random.Next(minRndCount, maxRndCount);
for (int loop4 = 0; loop4 <= intCountL4; loop4++)
{
// Entering loop4
if (loop4 == 0)
lm.EnterLoop(loop4, intCountL4);
else
lm.EnterLoop(loop4);
// Simulate loop4 delay
//Console.WriteLine("\t\t\t\tloop4");
Thread.Sleep(random.Next(minRndSleep, maxRndSleep));
// Exiting loop4
lm.ExitLoop();
}
// Exiting loop3
lm.ExitLoop();
}
// Exiting loop2
lm.ExitLoop();
}
// Exiting loop1
lm.ExitLoop();
}
ts.Cancel();
}
static private void DoUpdate(LoopTimer lm)
{
char[] animchar = { '|', '/', '-', '\\' };
int index = 0;
Thread.Sleep(100);
while (true)
{
TimeSpan eta = lm.RemainingTime;
Console.SetCursorPosition(0, 0);
Console.Write(string.Format(" {4} ETA: {0} Days, {1} Hours, {2} Minutes, {3} Seconds", eta.Days.ToString().PadLeft(3, '0'), eta.Hours.ToString().PadLeft(2, '0'), eta.Minutes.ToString().PadLeft(2, '0'), eta.Seconds.ToString().PadLeft(2, '0'), animchar[index].ToString()));
if (++index > 3)
index = 0;
Thread.Sleep(1000);
ts.Token.ThrowIfCancellationRequested();
}
}
/*
This method is provided as a sample on displaying the estimated total time.
*/
static private void DoUpdateTotal(LoopTimer lm)
{
char[] animchar = { '|', '/', '-', '\\' };
int index = 0;
Thread.Sleep(100);
while (true)
{
TimeSpan eta = lm.RemainingTime;
TimeSpan total = lm.TotalTime;
Console.SetCursorPosition(0, 0);
Console.Write(string.Format(" {4} ETA: {0} Days, {1} Hours, {2} Minutes, {3} Seconds", eta.Days.ToString().PadLeft(3, '0'), eta.Hours.ToString().PadLeft(2, '0'), eta.Minutes.ToString().PadLeft(2, '0'), eta.Seconds.ToString().PadLeft(2, '0'), animchar[index].ToString()));
Console.Write(string.Format("\n Total: {0} Days, {1} Hours, {2} Minutes, {3} Seconds", total.Days.ToString().PadLeft(3, '0'), total.Hours.ToString().PadLeft(2, '0'), total.Minutes.ToString().PadLeft(2, '0'), total.Seconds.ToString().PadLeft(2, '0')));
if (++index > 3)
index = 0;
Thread.Sleep(1000);
ts.Token.ThrowIfCancellationRequested();
}
}
/*
This method is provided as a sample on displaying the estimated total time, and
the internal values of every loop.
*/
static private void DoUpdateStatus(LoopTimer lm)
{
char[] animchar = { '|', '/', '-', '\\' };
int index = 0;
Thread.Sleep(100);
while (true)
{
TimeSpan eta = lm.RemainingTime;
TimeSpan total = lm.TotalTime;
TotalEta[] status = lm.GetStatus();
Console.SetCursorPosition(0, 0);
Console.Write(string.Format(" {4} ETA: {0} Days, {1} Hours, {2} Minutes, {3} Seconds", eta.Days.ToString().PadLeft(3, '0'), eta.Hours.ToString().PadLeft(2, '0'), eta.Minutes.ToString().PadLeft(2, '0'), eta.Seconds.ToString().PadLeft(2, '0'), animchar[index].ToString()));
Console.Write(string.Format("\n Total: {0} Days, {1} Hours, {2} Minutes, {3} Seconds", total.Days.ToString().PadLeft(3, '0'), total.Hours.ToString().PadLeft(2, '0'), total.Minutes.ToString().PadLeft(2, '0'), total.Seconds.ToString().PadLeft(2, '0')));
Console.WriteLine();
int loop = 0;
foreach (var item in status)
{
Console.Write(string.Format("\n Loop: {0} ETA: {1} \tTotal: {2}", loop, item.Eta.ToString(#"hh\:mm\:ss\.FFFF"), item.Total.ToString(#"hh\:mm\:ss\.FFFF")));
loop++;
}
if (++index > 3)
index = 0;
Thread.Sleep(1000);
ts.Token.ThrowIfCancellationRequested();
}
}
/*
This method is provided as a sample for variations on
the ExitLoopRet method. Uses in-place calls.
*/
static internal void Bonus()
{
TotalEta remVal;
TimeSpan remTime;
LoopTimer lm = new LoopTimer();
Console.Clear();
// easy change test parameters
var random = new Random();
int minRndCount = 1;
int maxRndCount = 5;
int maxRndSleep = 1000;
// First, outer loop
int intCountL1 = random.Next(minRndCount, maxRndCount);
for (int i = 0; i < intCountL1; i++)
{
if (i == 0)
lm.EnterLoop(i, intCountL1);
else
lm.EnterLoop(i);
Console.WriteLine(string.Format("\nLoop1 begin iteration: {0} of {1}. Will work(sleep) for {2} ms.", i, intCountL1 - 1, maxRndSleep));
Thread.Sleep(maxRndSleep);
// Second, middle loop
int intCountL2 = random.Next(minRndCount, maxRndCount);
for (int j = 0; j < intCountL2; j++)
{
if (j == 0)
lm.EnterLoop(j, intCountL2);
else
lm.EnterLoop(j);
Console.WriteLine(string.Format("\n\tLoop2 begin iteration: {0} of {1}. Will work(sleep) for {2} ms.", j, intCountL2 - 1, maxRndSleep));
Thread.Sleep(maxRndSleep);
// Third, inner loop
int intCountL3 = random.Next(minRndCount, maxRndCount);
for (int k = 0; k < intCountL3; k++)
{
if (k == 0)
lm.EnterLoop(k, intCountL3);
else
lm.EnterLoop(k);
Console.WriteLine(string.Format("\n\t\tLoop3 begin iteration: {0} of {1}. Will work(sleep) for {2} ms.", k, intCountL3 - 1, maxRndSleep));
Thread.Sleep(maxRndSleep);
lm.ExitLoop();
Console.WriteLine(string.Format("\n\t\tLoop3 end iteration: {0} of {1}", k, intCountL3 - 1));
lm.ShowStatus();
}
remTime = lm.ExitLoopRetETA();
Console.WriteLine(string.Format("\n\tLoop2 end iteration: {0} of {1}", j, intCountL2 - 1));
Console.WriteLine("\t\tRem: " + remTime.ToString());
}
remVal = lm.ExitLoopRetTotalEta();
Console.WriteLine(string.Format("\nLoop1 end iteration: {0} of {1}", i, intCountL1 - 1));
Console.WriteLine("\t\tTot: " + remVal.Total.ToString());
Console.WriteLine("\t\tRem: " + remVal.Eta.ToString());
}
}
}
}

custom threads in my code?

I need to create a program scraping a website.
And I did use Thread to solve.
Example:
I have 100 pages and I need divide it, instead of get each page I need custom Thread number to get page:
2 threads - 50 pages/thread
4 threads - 25 pages/thread
I tried my code below, however when to the last page of each thread that very slow.
Before I ask I used to find the way to solve but I can't, therefore I need help.
int so_thread = 10;//thread number
int page_du = 0;
List<NameValueCollection> List_item = new List<NameValueCollection>();
Thread[] threads = new Thread[so_thread];
int dem = 0;
await Task.Run(() =>
{
for (int i = 1; i <= so_thread; i++)
{
if ((Int32.Parse(o_sopage.Text) % so_thread) != 0 && i == so_thread)
{
page_du = Int32.Parse(o_sopage.Text) % so_thread;//Int32.Parse(o_sopage.Text) == page number need get
}
threads[i - 1] = new Thread((object data) =>
{
Array New_Data = new object[2];
New_Data = (Array)data;
int _i = (int)New_Data.GetValue(0);
int _pagedu = (int)New_Data.GetValue(1);
int page_per_thread = Int32.Parse(o_sopage.Text) / so_thread;//Int32.Parse(o_sopage.Text) == page number need get
for (int j = ((page_per_thread * _i) - page_per_thread) + 1; j <= ((page_per_thread * _i) + _pagedu); j++)
{
//MessageBox.Show(j.ToString());
var TG = ebay.GetPage(j);
lock (List_item)
{
List_item.AddRange(TG);
dem++;
progressBar1.Invoke((MethodInvoker)delegate
{
progressBar1.Value = dem;
});
}
}
});
object DATA = new object[2] { i, page_du };
threads[i - 1].Start(DATA);
}
});

Use Parallel.ForEach instead of creating the threads on your own.
Parallel.ForEach(yourCollection, () => { /* your code here */});

producer and consumer issue in operating system

Producer consumer problem using c# from book Abraham Silberschatz-Operating System Concepts.
I have wrote code of this pseudocode in C# ,,, but a warning occurs "Unreachable code detected in line 43"...... i am new to programming ..little guide is needed to solve this problem !
pseudo-code given in book:
#define BUFFER_SIZE 5
typedef struct {
. . .
} item;
item buffer[BUFFER_SIZE];
int in = 0;
int out = 0;
Producer:
item next_produced;
while (true) {
/* produce an item in next produced */
while (((in + 1) % BUFFER_SIZE) == out)
; /* do nothing */
buffer[in] = next_produced;
in = (in + 1) % BUFFER_SIZE;
}
Consumer:
item next_consumed;
while (true) { while (in == out)
; /* do nothing */ next_consumed = buffer[out];
out = (out + 1) % BUFFER_SIZE;
/* consume the item in next consumed */
}
My Code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace producer_consumer_problem_Csharp
{
struct item
{
private int iData;
public item(int x)
{
this.iData = x;
}
};
class Program
{
static void Main(string[] args)
{
int Buffer_Size = 5;
item[] buffer = new item[Buffer_Size];
int inp = 0;
int outp = 0;
item next_produced;
while(true)
{
Console.WriteLine("Enter the next produced item");
int x = Convert.ToInt32( Console.ReadLine() );
next_produced = new item(x);
while ((inp + 1) % Buffer_Size == outp) ;
// do nothing
buffer[inp] = next_produced;
inp = (inp + 1) % Buffer_Size;
}
item next_consumed = new item();
while (true)
{
while (inp == outp);
/*donothing*/
next_consumed = buffer[outp];
outp = (outp +1) % Buffer_Size; /* consume the item in next consumed */
Console.WriteLine("Next consuumed item is: {0} ", next_consumed);
}
}
}
}

As Soner mentioned, the first infinite loop prevents second one from running. To get your "consumer-producer" pair to work you have to run both parts in concurrent threads. Here's an example:
struct item
{
private int iData;
public item(int x)
{
this.iData = x;
}
public override string ToString()
{
return iData.ToString();
}
};
class Program
{
static void Main(string[] args)
{
int Buffer_Size = 5;
item[] buffer = new item[Buffer_Size];
int inp = 0;
int outp = 0;
var console_lock = new object();
new Thread(() =>
{
item next_produced;
while (true)
{
lock (console_lock)
{
Console.WriteLine("Enter the next produced item");
int x = Convert.ToInt32(Console.ReadLine());
next_produced = new item(x);
}
while ((inp + 1) % Buffer_Size == outp) ;
// do nothing
buffer[inp] = next_produced;
inp = (inp + 1) % Buffer_Size;
}
}).Start();
new Thread(() =>
{
item next_consumed = new item();
while (true)
{
while (inp == outp) ;
/*donothing*/
next_consumed = buffer[outp];
outp = (outp + 1) % Buffer_Size; /* consume the item in next consumed */
lock (console_lock) Console.WriteLine("Next consuumed item is: {0} ", next_consumed);
}
}).Start();
Thread.Sleep(Timeout.Infinite);
}
}
Please keep in mind that this is a way unefficient implementation and should be considered as just a primitive example of "producer-consumer" pattern, bun not a guide. This code will needlessly consume a lot of your computing power for a /*donothing*/ which is basically a SpinWait.

Thread safety Parallel.For c#

im frenchi so sorry first sorry for my english .
I have an error on visual studio (index out of range) i have this problem only with a Parallel.For not with classic for.
I think one thread want acces on my array[i] and another thread want too ..
It's a code for calcul Kmeans clustering for building link between document (with cosine similarity).
more information :
IndexOutOfRange is on similarityMeasure[i]=.....
I have a computer with 2 Processor (12logical)
with classic for , cpu usage is 9-14% , time for 1 iteration=9min..
with parallel.for , cpu usage is 70-90% =p, time for 1 iteration =~1min30
Sometimes it works longer before generating an error
My function is :
private static int FindClosestClusterCenter(List<Centroid> clustercenter, DocumentVector obj)
{
float[] similarityMeasure = new float[clustercenter.Count()];
float[] copy = similarityMeasure;
object sync = new Object();
Parallel.For(0, clustercenter.Count(), (i) => //for(int i = 0; i < clustercenter.Count(); i++) Parallel.For(0, clustercenter.Count(), (i) => //
{
similarityMeasure[i] = SimilarityMatrics.FindCosineSimilarity(clustercenter[i].GroupedDocument[0].VectorSpace, obj.VectorSpace);
});
int index = 0;
float maxValue = similarityMeasure[0];
for (int i = 0; i < similarityMeasure.Count(); i++)
{
if (similarityMeasure[i] > maxValue)
{
maxValue = similarityMeasure[i];
index = i;
}
}
return index;
}
My function is call here :
do
{
prevClusterCenter = centroidCollection;
DateTime starttime = DateTime.Now;
foreach (DocumentVector obj in documentCollection)//Parallel.ForEach(documentCollection, parallelOptions, obj =>//foreach (DocumentVector obj in documentCollection)
{
int ind = FindClosestClusterCenter(centroidCollection, obj);
resultSet[ind].GroupedDocument.Add(obj);
}
TimeSpan tempsecoule = DateTime.Now.Subtract(starttime);
Console.WriteLine(tempsecoule);
//Console.ReadKey();
InitializeClusterCentroid(out centroidCollection, centroidCollection.Count());
centroidCollection = CalculMeanPoints(resultSet);
stoppingCriteria = CheckStoppingCriteria(prevClusterCenter, centroidCollection);
if (!stoppingCriteria)
{
//initialisation du resultat pour la prochaine itération
InitializeClusterCentroid(out resultSet, centroidCollection.Count);
}
} while (stoppingCriteria == false);
_counter = counter;
return resultSet;
FindCosSimilarity :
public static float FindCosineSimilarity(float[] vecA, float[] vecB)
{
var dotProduct = DotProduct(vecA, vecB);
var magnitudeOfA = Magnitude(vecA);
var magnitudeOfB = Magnitude(vecB);
float result = dotProduct / (float)Math.Pow((magnitudeOfA * magnitudeOfB),2);
//when 0 is divided by 0 it shows result NaN so return 0 in such case.
if (float.IsNaN(result))
return 0;
else
return (float)result;
}
CalculMeansPoint :
private static List<Centroid> CalculMeanPoints(List<Centroid> _clust)
{
for (int i = 0; i < _clust.Count(); i++)
{
if (_clust[i].GroupedDocument.Count() > 0)
{
for (int j = 0; j < _clust[i].GroupedDocument[0].VectorSpace.Count(); j++)
{
float total = 0;
foreach (DocumentVector vspace in _clust[i].GroupedDocument)
{
total += vspace.VectorSpace[j];
}
_clust[i].GroupedDocument[0].VectorSpace[j] = total / _clust[i].GroupedDocument.Count();
}
}
}
return _clust;
}

You may have some side effects in FindCosineSimilarity, make sure it does not modify any field or input parameter. Example: resultSet[ind].GroupedDocument.Add(obj);. If resultSet is not a reference to locally instantiated array, then that is a side effect.
That may fix it. But FYI you could use AsParallel for this rather than Parallel.For:
similarityMeasure = clustercenter
.AsParallel().AsOrdered()
.Select(c=> SimilarityMatrics.FindCosineSimilarity(c.GroupedDocument[0].VectorSpace, obj.VectorSpace))
.ToArray();

You realize that if you synchronize the whole Content of the Parallel-For, it's just the same as having a normal synchrone for-loop, right? Meaning the code as it is doesnt do anything in parallel, so I dont think you'll have any Problems with concurrency. My guess from what I can tell is clustercenter[i].GroupedDocument is propably an empty Array.

c# How to run a application faster

I am creating a word list of possible uppercase letters to prove how insecure 8 digit passwords are this code will write aaaaaaaa to aaaaaaab to aaaaaaac etc. until zzzzzzzz using this code:
class Program
{
static string path;
static int file = 0;
static void Main(string[] args)
{
new_file();
var alphabet = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ123456789+-*_!$£^=<>§°ÖÄÜöäü.;:,?{}[]";
var q = alphabet.Select(x => x.ToString());
int size = 3;
int counter = 0;
for (int i = 0; i < size - 1; i++)
{
q = q.SelectMany(x => alphabet, (x, y) => x + y);
}
foreach (var item in q)
{
if (counter >= 20000000)
{
new_file();
counter = 0;
}
if (File.Exists(path))
{
using (StreamWriter sw = File.AppendText(path))
{
sw.WriteLine(item);
Console.WriteLine(item);
/*if (!(Regex.IsMatch(item, #"(.)\1")))
{
sw.WriteLine(item);
counter++;
}
else
{
Console.WriteLine(item);
}*/
}
}
else
{
new_file();
}
}
}
static void new_file()
{
path = #"C:\" + "list" + file + ".txt";
if (!File.Exists(path))
{
using (StreamWriter sw = File.CreateText(path))
{
}
}
file++;
}
}
The Code is working fine but it takes Weeks to run it. Does anyone know a way to speed it up or do I have to wait? If anyone has a idea please tell me.

Performance:
size 3: 0.02s
size 4: 1.61s
size 5: 144.76s
Hints:
removed LINQ for combination generation
removed Console.WriteLine for each password
removed StreamWriter
large buffer (128k) for file writing
const string alphabet = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ123456789+-*_!$£^=<>§°ÖÄÜöäü.;:,?{}[]";
var byteAlphabet = alphabet.Select(ch => (byte)ch).ToArray();
var alphabetLength = alphabet.Length;
var newLine = new[] { (byte)'\r', (byte)'\n' };
const int size = 4;
var number = new byte[size];
var password = Enumerable.Range(0, size).Select(i => byteAlphabet[0]).Concat(newLine).ToArray();
var watcher = new System.Diagnostics.Stopwatch();
watcher.Start();
var isRunning = true;
for (var counter = 0; isRunning; counter++)
{
Console.Write("{0}: ", counter);
Console.Write(password.Select(b => (char)b).ToArray());
using (var file = System.IO.File.Create(string.Format(#"list.{0:D5}.txt", counter), 2 << 16))
{
for (var i = 0; i < 2000000; ++i)
{
file.Write(password, 0, password.Length);
var j = size - 1;
for (; j >= 0; j--)
{
if (number[j] < alphabetLength - 1)
{
password[j] = byteAlphabet[++number[j]];
break;
}
else
{
number[j] = 0;
password[j] = byteAlphabet[0];
}
}
if (j < 0)
{
isRunning = false;
break;
}
}
}
}
watcher.Stop();
Console.WriteLine(watcher.Elapsed);
}

Try the following modified code. In LINQPad it runs in < 1 second. With your original code I gave up after 40 seconds. It removes the overhead of opening and closing the file for every WriteLine operation. You'll need to test and ensure it gives the same results because I'm not willing to run your original code for 24 hours to ensure the output is the same.
class Program
{
static string path;
static int file = 0;
static void Main(string[] args)
{
new_file();
var alphabet = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ123456789+-*_!$£^=<>§°ÖÄÜöäü.;:,?{}[]";
var q = alphabet.Select(x => x.ToString());
int size = 3;
int counter = 0;
for (int i = 0; i < size - 1; i++)
{
q = q.SelectMany(x => alphabet, (x, y) => x + y);
}
StreamWriter sw = File.AppendText(path);
try
{
foreach (var item in q)
{
if (counter >= 20000000)
{
sw.Dispose();
new_file();
counter = 0;
}
sw.WriteLine(item);
Console.WriteLine(item);
}
}
finally
{
if(sw != null)
{
sw.Dispose();
}
}
}
static void new_file()
{
path = #"C:\temp\list" + file + ".txt";
if (!File.Exists(path))
{
using (StreamWriter sw = File.CreateText(path))
{
}
}
file++;
}
}

your alphabet is missing 0
With that fixed there would be 89 chars in your set. Let's call it 100 for simplicity. The set you are looking for is all the 8 character length strings drawn from that set. There are 100^8 of these, i.e. 10,000,000,000,000,000.
The disk space they will take up depends on how you encode them, lets be generous - assume you use some 8 bit char set that contains the these characters, and you don't put in carriage returns, so one byte per char, so 10,000,000,000,000,000 bytes =~ 10 peta byes?
Do you have 10 petabytes of disk? (10000 TB)?
[EDIT] In response to 'this is not an answer':
The original motivation is to create the list? The shows how large the list would be. Its hard to see what could be DONE with the list if it was actualised, i.e. it would always be quicker to reproduce it than to load it. Surely whatever point could be made by producing the list can also be made by simply knowing it's size, which the above shows how to work it out.
There are LOTS of inefficiencies in you code, but if your questions is 'how can i quickly produce this list and write it to disk' the answer is 'you literally cannot'.
[/EDIT]