Develop Reference

Multiple consumers update result array inconsistently

I have a large file, each row can be process separately, so I launch one reader, and multiple parsers.
The each parser will write result back to a result holder array for further process.
I found if I launch more parser, the result holder array gives different content each time, no matter if I use ConcurrentQueue or BlockingCollection or some other things
I repeatedly run the program and output the result array many times, each time will give different if I use more than 1 parsers.
string[] result = new string[nRow];
static BlockingCollection<queueItem> myBlk = new BlockingCollection<queueItem>();
static void Main()
{
Reader();
}
static void parserThread()
{
while (myBlk.IsCompleted == false)
{
queueItem one;
if (myBlk.TryTake(out one) == false)
{
System.Threading.Thread.Sleep(tSleep);
}
else
{
oneDataRow(one.seqIndex, one.line);
}
}
}
static void oneDataRow(int rowIndex, string line)
{
result[rowIndex] = // some process with line
}
static void Reader()
{
for (int i = 0; i < 10; i++)
{
Task t = new Task(() => parserThread());
t.Start();
}
StreamReader sr = new StreamReader(path);
string line;
int nRead=0;
while((line = sr.ReadLine()) != null)
{
string innerLine = line;
int innerN = nRead;
myBlk.Add(new queueItem(innerN, innerLine));
nRead++;
}
siteBlk.CompleteAdding();
sw.close();
while (myBlk.IsCompleted == false)
{
System.Threading.Thread.Sleep(tSleep);
}
}
class queueItem
{
public int seqIndex = 0;
public string line = "";
public queueItem(int RowOrder, string content)
{
seqIndex = RowOrder;
line = content;
}
}

The way you are waiting for the process to complete is problematic:
while (myBlk.IsCompleted == false)
{
System.Threading.Thread.Sleep(tSleep);
}
Here is the description of the IsCompleted property:
Gets whether this BlockingCollection<T> has been marked as complete for adding and is empty.
In your case the completion of the BlockingCollection should not signal the completion of the whole operation, because the last lines taken from the collection may not be processed yet.
Instead you should store the worker tasks into an array (or list), and wait them to complete.
Task.WaitAll(tasks);
In general you should rarely use the IsCompleted property for anything other than for logging debug information. Using it for controlling the execution flow introduces race conditions in most cases.

Worker Threads Blocking When ConcurrentQueue has too many items

This is a weird one, I have a Thread[] of worker threads which each process items in a ConcurrentQueue<string> until the queue is empty, at which point the rest of the program continues.
This works until about ~1500 items at which point all threads stay blocked in the WaitSleepJoin state and never process any of the items in the queue.
I've tried stepping through my code and it appears that the threads are still created, still started and are alive but get blocked immediately and never run their relevant function.
I'm completely flummoxed so any help would be appreciated!
The relevant sections of code are below:
Main Thread Segment:
ConcurrentQueue<string> convertionQueue = new ConcurrentQueue<string>();
List<Thread> converterThreads = new List<Thread>();
Directory.GetFiles(_folderOne, "*.fdf", SearchOption.AllDirectories).ToList().ForEach(file => convertionQueue.Enqueue(file));
Directory.GetFiles(_folderTwo, "*.fdf", SearchOption.AllDirectories).ToList().ForEach(file => convertionQueue.Enqueue(file));
int filesDone = 0;
int totalFiles = convertionQueue.Count;
progressBar.Maximum = totalFiles;
panel1.Visible = true;
for (int i = 0; i < Environment.ProcessorCount; i++)
{
converterThreads.Add(new Thread(() => ConvThreadWorker(convertionQueue, ref filesDone)));
}
converterThreads.ForEach(thread => thread.Start());
DateTime lastTick = DateTime.Now;
int lastFilesDone = 0;
int[] valuesSpeed = { 1, 1, 1, 1, 1 };
int[] valuesTime = { 1, 1, 1, 1, 1 };
int counter = 0;
while (converterThreads.Any(thread => thread.IsAlive))
{
TimeSpan t = DateTime.Now - lastTick;
int deltaFiles = filesDone - lastFilesDone;
double speed = (float)t.TotalMilliseconds <= 0.0 ? 0.0 : deltaFiles / (float)t.TotalMilliseconds;
double tMinus = speed <= 0 ? 0.0 : (totalFiles - filesDone) / speed;
int currentSpeed = (int)(speed * 1000);
int currentTime = (int)(tMinus / 1000);
valuesSpeed[counter] = currentSpeed;
valuesTime[counter] = currentTime;
lblFilesLeft.Text = string.Format("{0}/{1}", filesDone, totalFiles);
lblSpeed.Text = valuesSpeed.Sum() / 5 + " /s";
lblTime.Text = valuesTime.Sum() / 5 + " s";
lblFilesLeft.Update();
lblSpeed.Update();
lblTime.Update();
progressBar.Value = filesDone;
progressBar.Update();
lastTick = DateTime.Now;
lastFilesDone = filesDone;
counter = ++counter % 5;
Thread.Sleep(500);
}
Worker Function:
private void ConvThreadWorker(ConcurrentQueue<string> queue, ref int fileCounter)
{
while (!queue.IsEmpty)
{
string file;
if (queue.TryDequeue(out file))
{
ConvToG(file);
fileCounter++;
}
}
}
Convertion Function:
private void ConvToG(string file)
{
MessageBox.Show("Entering Convertion Function");
if (!_fileCreationDictionary.ContainsKey(file))
{
DateTime lastTimeModified = File.GetLastWriteTime(file);
_fileCreationDictionary.AddOrUpdate(file, lastTimeModified, (key,oldvalue)=>lastTimeModified);
}
ProcessStartInfo procStart = new ProcessStartInfo
{
Arguments = file,
UseShellExecute = true,
FileName = Fdfg,
WindowStyle = ProcessWindowStyle.Hidden
};
Process process = new Process {StartInfo = procStart};
MessageBox.Show("Starting convertion process");
process.Start();
process.WaitForExit();
MessageBox.Show("Finished");
}
The confusing part appears to be how this all revolves around the number of items in the queue, yet there appears to be no overflow.
UPDATE: Adding the mbox's shows that it freezes on the process.Start() section of code, with no errors and will not proceed past that point.
UPDATE 2: If UseShellExecute = false the code works. Which is very confusing to say the least.

I have done something similar with threads spawning processes to collate data. I had issues around the actual process starting and hanging. What I did to get my program working was something like this:
using (Process process = Process.Start(startInfo)) {
if(process.WaitForExit(timeOutMilliseconds)) {
MessageBox.Show("Process exited ok");
//...snip
} else {
MessageBox.Show("Process did not exit in time!");
//...snip
process.Kill();
}
}
There is a bit more going on in the background, regarding limiting the number of running process, etc, but I found that occasionally, for an unknown reason, that I would see several process in the task manager just hanging around forever.
Hope that helps?

for loop keeps making my program crash c#

I have a application which programs firmware to a circuit board. In the application you can program a single board, or a tray. When programming a tray you can only load 14 at a time.
The user may want to program say 30 boards, so I want the program to program the 14 boards and then tell the user they need to reload a tray.
At the moment I only have one board to practice on, so I have just been reprogramming the same one pretending its a tray.
I have tried to resolve this using loops but when I press the start button it all freezes and stops responding.
The following is my code:
private void setFirmwareMultiple()
{
clearTicksandCrosses();
string firmwareLocation = Firmware(productComboBox.Text); //get the firmware location
string STPath = #"C:\Users\Falconex\Documents\FalconexTest\FalconexTest\ST-LINK Utility\ST-LINK_CLI.exe"; //file location
string result; //set string
string result2; //set string
int counter = 0;
int numberOfBoards = int.Parse(numberOfBoardsTextBox.Text);
while (numberOfBoards > counter) {
for (int i = 0; i > 14; i = i + 1) {
ProcessStartInfo start = new ProcessStartInfo(); //new process start info
start.FileName = STPath; //set file name
start.Arguments = "-C -ME -p " + firmwareLocation + " -v -Run"; //set arguments
start.UseShellExecute = false; //set shell execute (need this to redirect output)
start.RedirectStandardOutput = true; //redirect output
start.RedirectStandardInput = true; //redirect input
start.WindowStyle = ProcessWindowStyle.Hidden; //hide window
start.CreateNoWindow = true; //create no window
string picNumber = i.ToString();
using (Process process = Process.Start(start)) //create process
{
programmingTextBlock.Text = "Board Programming...";
System.Windows.Application.Current.Dispatcher.Invoke(DispatcherPriority.Background,
new Action(delegate { }));
try
{
while (process.HasExited == false) //while open
{
process.StandardInput.WriteLine(); //send enter key
}
using (StreamReader reader = process.StandardOutput) //create stream reader
{
result = reader.ReadToEnd(); //read till end of process
File.WriteAllText("File.txt", result); //write to file
}
saveReport();
}
catch { } //so doesn't blow up
finally
{
int code = process.ExitCode; //get exit code
codee = code.ToString(); //set code to string
File.WriteAllText("Code.txt", codee); //save code
if (code == 0)
{
tick1.Visibility = Visibility.Visible;
counter = counter + 1;
}
else
{
cross1.Visibility = Visibility.Visible;
}
programmingTextBlock.Text = "";
}
}
System.Windows.MessageBox.Show("Load new boards");
}
}
}
I have put the total amount of boards the user wants in the for loop.
I think it maybe to do with the for loop. Because at first, in the for loop, I accidently put (i<14) and it caused it to run fine, however it then didn't stop.
Any help would be massively appreciated!
Thank you in advance,
Lucy

As the code stands now, your for loop's content never gets executed. The condition in the for loop is a continue condition. Since i is initialized with 0 the condition i > 14 is never met. So the result is an infinite outer while loop.
Your first "accident" with i < 14 was correct. But then the loop did not stop, because your inner while loop never finishes:
while (process.HasExited == false) //while open
{
process.StandardInput.WriteLine(); //send enter key
}
At first, please don't compare a bool to true or false. A simple while (!process.HasExited) is enough.
Secondly, you have to refresh your process instance to update the HasExited property correctly:
while (!process.HasExited) //while open
{
process.StandardInput.WriteLine(); //send enter key
process.Refresh(); // update the process's properties!
}
You may also consider to add a Thread.Sleep(...) in that loop.

The answer is simple:
while (numberOfBoards > counter) {
for (int i = 0; i > 14; i = i + 1) {
In the code above, the for loop will never be executed, because i will be always less than 14.
Because this, counter will never increment, and than, the while will never finish.
But besides this, your approach to the looping is wrong. The following example (fully test program) is something you should do instead:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
int counter = 0;
int i = 0;
int numberOfBoards = 35;
for (; numberOfBoards > counter; i++, counter++)
{
Console.WriteLine("Counter {0}/i {1}", counter, i);
//call your thread here.
//make sure that he exists.
//use somekind of timeout to finish
//alert the user in case of failure but move to the next anyway to avoid an infinite looping.
if (i == 13) i = 0;
}
Console.WriteLine("Press any key");
Console.ReadKey();
}
}
}

Lock input and output data

I have main and working threads. These working threads must not stopped before main task is over (i.e. i cant re-creating them with Thread.Start()). These working threads reading data from main thread, then writing other data and then main thread writing it in file.
I must use NET 3.5, not including ThreadPool and BackgroundWorker
InFile -> RawData -> Working -> ReadyData -> OutFile.
public func(string InputFile,string outputFile) {
...
for(int i=0;i<ThreadCount;i++)
{
CompressorThreads[i] = new Thread(new ParameterizedThreadStart(CompressBlock));
ThreadParam thr = new ThreadParam(i);
CompressorThreads[i].Start(thr);
}
int count;
while (InputFile.Position < InputFile.Length)
{
for (int i = 0; i < ThreadCount; i++)
{
count = InputFile.Read(buffer, 0, BLOCK_SIZE);
RawData[i] = new byte[count];
/*here activating(start) thread*/
}
for (int i = 0; (i < ThreadCount) && (CompressorThreads[i] != null);)
{
/*prevent(pause) working thread from changing CompressedData*/
OutputFile.Write(CompressedData[i], 0, CompressedData[i].Length);
}
}
...
}
private static void CompressBlock(object BlockProperties)
{
ThreadParam Block = BlockProperties as ThreadParam;
int ThreadId = Block.ThreadId;
while (true)
{
if(RawData[ThreadId].Length == 0)
{
break;
}
...
/*working here*/
...
/*then save data to CompressedData*/
/*pause thread*/
}
}
I tried AutoResetEvent, but it seems inappropriate for this problem. What i'm looking for?

Task.Factory.StartNew is not working in console c# application

I am having a console application which reads the messages from Console.OpenStandardInput();
I am doing this in a task. but it seems to be not working.
static void Main(string[] args)
{
wtoken = new CancellationTokenSource();
readInputStream = Task.Factory.StartNew(() =>
{
wtoken.Token.ThrowIfCancellationRequested();
while (true)
{
if (wtoken.Token.IsCancellationRequested)
{
wtoken.Token.ThrowIfCancellationRequested();
}
else
{
OpenStandardStreamIn();
}
}
}, wtoken.Token
);
Console.ReadLine();
}
Here is my OpenStandardStreamIn function
public static void OpenStandardStreamIn()
{
Stream stdin = Console.OpenStandardInput();
int length = 0;
byte[] bytes = new byte[4];
stdin.Read(bytes, 0, 4);
length = System.BitConverter.ToInt32(bytes, 0);
string input = "";
for (int i = 0; i < length; i++)
{
input += (char)stdin.ReadByte();
}
Console.Write(input);
}
Any help? why it is not working in a continous loop

You basically have a race condition between Console.ReadLine and your task. Both of them are trying to read from standard input - and I certainly don't know what you should expect when reading from standard input from two threads at the same time, but it seems like something worth avoiding.
You can easily test this by changing the task to do something other than reading from standard input. For example:
using System;
using System.Threading;
using System.Threading.Tasks;
class Test
{
static void Main()
{
var wtoken = new CancellationTokenSource();
var readInputStream = Task.Factory.StartNew(() =>
{
for (int i = 0; i < 10; i++)
{
Console.WriteLine(i);
Thread.Sleep(200);
}
}, wtoken.Token);
Console.ReadLine();
}
}
If your real code needs to read from standard input, then I suggest you change Console.ReadLine() into readInputStream.Wait(). I'd also suggest you use Task.Run instead of Task.Factory.StartNew() if you're using .NET 4.5, just for readability - assuming you don't need any of the more esoteric behaviour of TaskFactory.StartNew.