Develop Reference

Read and process 100 text files in c# in parallel

I have project that reads 100 text file with 5000 words in it.
I insert the words into a list. I have a second list that contains english stop words. I compare the two lists and delete the stop words from first list.
It takes 1 hour to run the application. I want to be parallelize it. How can I do that?
Heres my code:
private void button1_Click(object sender, EventArgs e)
{
List<string> listt1 = new List<string>();
string line;
for (int ii = 1; ii <= 49; ii++)
{
string d = ii.ToString();
using (StreamReader reader = new StreamReader(#"D" + d.ToString() + ".txt"))
while ((line = reader.ReadLine()) != null)
{
string[] words = line.Split(' ');
for (int i = 0; i < words.Length; i++)
{
listt1.Add(words[i].ToString());
}
}
listt1 = listt1.ConvertAll(d1 => d1.ToLower());
StreamReader reader2 = new StreamReader("stopword.txt");
List<string> listt2 = new List<string>();
string line2;
while ((line2 = reader2.ReadLine()) != null)
{
string[] words2 = line2.Split('\n');
for (int i = 0; i < words2.Length; i++)
{
listt2.Add(words2[i]);
}
listt2 = listt2.ConvertAll(d1 => d1.ToLower());
}
for (int i = 0; i < listt1.Count(); i++)
{
for (int j = 0; j < listt2.Count(); j++)
{
listt1.RemoveAll(d1 => d1.Equals(listt2[j]));
}
}
listt1=listt1.Distinct().ToList();
textBox1.Text = listt1.Count().ToString();
}
}
}
}

I fixed many things up with your code. I don't think you need multi-threading:
private void RemoveStopWords()
{
HashSet<string> stopWords = new HashSet<string>();
using (var stopWordReader = new StreamReader("stopword.txt"))
{
string line2;
while ((line2 = stopWordReader.ReadLine()) != null)
{
string[] words2 = line2.Split('\n');
for (int i = 0; i < words2.Length; i++)
{
stopWords.Add(words2[i].ToLower());
}
}
}
var fileWords = new HashSet<string>();
for (int fileNumber = 1; fileNumber <= 49; fileNumber++)
{
using (var reader = new StreamReader("D" + fileNumber.ToString() + ".txt"))
{
string line;
while ((line = reader.ReadLine()) != null)
{
foreach(var word in line.Split(' '))
{
fileWords.Add(word.ToLower());
}
}
}
}
fileWords.ExceptWith(stopWords);
textBox1.Text = fileWords.Count().ToString();
}
You are reading through the list of stopwords many times as well as continually adding to the list and re-attempting to remove the same stopwords over and again due to the way your code is structured. Your needs are also better matched to a HashSet than to a List, as it has set based operations and uniqueness already handled.
If you still wanted to make this parallel, you could do it by reading the stopword list once and passing it to an async method that will read the input file, remove the stopwords and return the resulting list, then you would need to merge the resulting lists after the asynchronous calls came back, but you had better test before deciding you need that, because that is quite a bit more work and complexity than this code already has.

If I understand you correctly, you want to:
Read all words from a file into a List
Remove all "stop words" from the List
Repeat for 99 more files, saving only the unique words
If this is correct, the code is pretty simple:
// The list of words to delete ("stop words")
var stopWords = new List<string> { "remove", "these", "words" };
// The list of files to check - you can get this list in other ways
var filesToCheck = new List<string>
{
#"f:\public\temp\temp1.txt",
#"f:\public\temp\temp2.txt",
#"f:\public\temp\temp3.txt"
};
// This list will contain all the unique words from all
// the files, except the ones in the "stopWords" list
var uniqueFilteredWords = new List<string>();
// Loop through all our files
foreach (var fileToCheck in filesToCheck)
{
// Read all the file text into a varaible
var fileText = File.ReadAllText(fileToCheck);
// Split the text into distinct words (splitting on null
// splits on all whitespace) and ignore empty lines
var fileWords = fileText.Split(null)
.Where(line => !string.IsNullOrWhiteSpace(line))
.Distinct();
// Add all the words from the file, except the ones in
// your "stop list" and those that are already in the list
uniqueFilteredWords.AddRange(fileWords.Except(stopWords)
.Where(word => !uniqueFilteredWords.Contains(word)));
}
This can be condensed into a single line with no explicit loop:
// This list will contain all the unique words from all
// the files, except the ones in the "stopWords" list
var uniqueFilteredWords = filesToCheck.SelectMany(fileToCheck =>
File.ReadAllText(fileToCheck)
.Split(null)
.Where(word => !string.IsNullOrWhiteSpace(word) &&
!stopWords.Any(stopWord => stopWord.Equals(word,
StringComparison.OrdinalIgnoreCase)))
.Distinct());
This code processed over 100 files with more than 12000 words each in less than a second (WAY less than a second... 0.0001782 seconds)

One issue I see here that can help improve performance is listt1.ConvertAll() will run in O(n) on the list. You are already looping to add the items to the list, why not convert them to lower case there. Also why not store the words in a hash set, so you can do look up and insertion in O(1). You could store the list of stop words in a hash set and when you are reading your text input see if the word is a stop word and if its not add it to the hash set to output the user.

trouble reading and writing to a file c#

I am currently trying to take a file of words that are not in alphabetical, re-order the words so that they are in alphabetical order (I am trying to use a non-built in sort method), and then write the newly ordered list into a new txt file(one that must be created). For example, lets say there is only five words in the txt file that are as follows "dog bat apple rabbit cat". I would want the program to resort these in alphabetical order, and then create a txt file that saves that order. As of right now, the program will iterate through the txt file, but will not save the re-ordered list into the new txt file. What is saved into the new file is this... "System.Collections.Generic.List`1[System.String]"
Truth be told, I am not very savvy with c# yet, so i apologize if my structuring or coding is not very well. The original file that is un-ordered is called "jumbled english FILTERED.ALL.txt", and the file I am trying to write to is called "english FILTERED.ALL.txt".
static void Main(string[] args)
{
// declaring integer for minimum.
int min = 0;
// declare the list for the original file
List<string> LinuxWords = new List<string>();
List<string> lexicalOrder = new List<string>();
// read the text from the file
string[] lines = System.IO.File.ReadAllLines("jumbled english FILTERED.ALL.txt");
string line = string.Empty;
// seperate each word into a string
//foreach (string line in lines)
//{
//add each word into the list.
//LinuxWords.Add(line);
//}
for (int i = 0; i < lines.Length - 1; i++)
{
for (int j = i + 1; j < lines.Length; j++)
{
if (lines[i].Length < lines[j].Length)
{
min = lines[i].Length;
}
else
{
min = lines[j].Length;
}
for (int k = 0; k < min; k++)
{
if (lines[i][k] > lines[j][k])
{
line = lines[i].ToString();
lines[i] = lines[j];
lines[j] = line;
break;
}
else if (lines[i][k] == lines[j][k])
{
continue;
}
else
{
break;
}
}
}
}
for (int i = 0; i < lines.Length; i++)
{
Console.WriteLine("The program is formatting the correct order");
lexicalOrder.Add(lines[i]);
}
//lexicalOrder.ForEach(Console.WriteLine);
//}
//LinuxWords.ForEach(Console.WriteLine);
File.WriteAllText(AppDomain.CurrentDomain.BaseDirectory + "english FILTERED.ALL.txt",
lexicalOrder.ToString());
// write the ordered list back into another .txt file named "english FILTERED.ALL.txt"
// System.IO.File.WriteAllLines("english FILTERED.ALL.txt", lexicalOrder);
Console.WriteLine("Finished");
}

Assuming you mean that you don't get the list saved (if that's not the problem - please be more specific) - you need to change
lexicalOrder.ToString()
to something like
lexicalOrder.Aggregate((s1, s2) => s1 + " " + s2)

Count the number of words within an Array or List

I need to count the number of words within an array or a list. The reason I say array or list is because I am not sure which would be the best to use in this situation. The data is static and in a .txt file (It's actually a book). I was able to create an array and break down words from the array but for the life of me I can not count! I have tried many different ways to do this and I'm thinking since it is a string it is unable to count. I have even teetered on the edge of just printing the whole book to a listbox and counting from the listbox but, that's ridiculous.
public partial class mainForm : Form
{
//------------------------
//GLOBAL VARIABLES:
//------------------------
List<string> countWords;
string[] fileWords;
string[] fileLines;
char[] delim = new char[] { ' ', ',','.','?','!' };
string path;
public mainForm()
{
InitializeComponent();
}
private void BookTitle() // TiTleAndAuthor Method will pull the Book Title and display it.
{
for (int i = 0; i < 1; i++)
{
bookTitleLabel.Text = fileLines[i];
}
}
private void BookAuthor() // TiTleAndAuthor Method will pull the Book Author and display it.
{
for (int i = 1; i < 2; i++)
{
bookAuthorLabel.Text = fileLines[i];
}
}
private void FirstLines() // FirstTenWords Method pulls the first ten words of any text file and prints the to a ListBox
{
for (int i = 0; i <= 499; i++)
{
wordsListBox.Items.Add(fileWords[i]);
}
}
private void WordCount() // Count all the words in the file.
{
}
private void openFileButton_Click(object sender, EventArgs e)
{
OpenFileDialog inputFile = new OpenFileDialog();
if (inputFile.ShowDialog() == DialogResult.OK) // check the file the user selected
{
path = inputFile.FileName; // save that path of the file to a string variable for later use
StreamReader fileRead = new StreamReader(path); // read a file at the path outlined in the path variable
fileWords = fileRead.ReadToEnd().Split(delim); // Breakdown the text into lines of text to call them at a later date
fileLines = File.ReadAllLines(path);
countWords = File.ReadLines(path).ToList();
wordsListBox.Items.Clear();
BookTitle();
BookAuthor();
FirstLines();
WordCount();
}
else
{
MessageBox.Show("Not a valid file, please select a text file");
}
}
}

Maybe this is useful:
static void Main(string[] args)
{
string[] lines = File_ReadAllLines();
List<string> words = new List<string>();
foreach(var line in lines)
{
words.AddRange(line.Split(' '));
}
Console.WriteLine(words.Count);
}
private static string[] File_ReadAllLines()
{
return new[] {
"The one book",
"written by gnarf",
"once upon a time ther werent any grammer",
"iso 1-12122-445",
"(c) 2012 under the hills"
};
}

Before I get to the answer, a quick observation on some of the loops:
for (int i = 1; i < 2; i++)
{
bookAuthorLabel.Text = fileLines[i];
}
This'll only run once, so it's pointless to have it in a loop (unless you intended this to actually loop through the whole list, in which case it's a bug). If this is the expected behavior, you might as well just do
bookAuthorLabel.Text = fileLines[1];
You have something similar here:
for (int i = 0; i < 1; i++)
{
bookTitleLabel.Text = fileLines[i];
}
Again, this is pointless.
Now for the answer itself. I'm not sure if you're trying to get total word count or count of individual words, so here's a code sample for doing both:
private static void CountWords()
{
const string fileName = "CountWords.txt";
// Create a dummy file
using (var sw = new StreamWriter(fileName))
{
sw.WriteLine("This is a short sentence");
sw.WriteLine("This is a long sentence");
}
string text = File.ReadAllText(fileName);
string[] result = text.Split(new[] { " ", Environment.NewLine }, StringSplitOptions.RemoveEmptyEntries);
// Total word count
Console.WriteLine("Total count: " + result.Count().ToString());
// Now to illustrate getting the count of individual words
var dictionary = new Dictionary<string, int>();
foreach (string word in result)
{
if (dictionary.ContainsKey(word))
{
dictionary[word]++;
}
else
{
dictionary[word] = 1;
}
}
foreach (string key in dictionary.Keys)
{
Console.WriteLine(key + ": " + dictionary[key].ToString());
}
}
This should be easy to adapt to your particular needs in this case.

Read text file line by line. split by empty character and remove unnecessary spaces. sum this count to total.
var totalWords = 0;
using (StreamReader sr = new StreamReader("abc.txt"))
{
while (!sr.EndOfStream)
{
int count = sr
.ReadLine()
.Split(new char[] {' '}, StringSplitOptions.RemoveEmptyEntries).Count();
totalWords += count;
}
You can also use the below code:
totalWords = fileRead.ReadToEnd().Split(delim, StringSplitOptions.RemoveEmptyEntries).Length;

name is not incrementing by 1 after number 45

i am trying to add a location name to my output text files.
As you can see my numbers are incrementing properly. But i have coded like after number 45 i need to reset the number to 1, also the Carousel:45 should change to ** Carousel1:1**. But it is not happening... why it is not happening. any help please!!!!
My code snippet:
public void just_create_text()
{
//Here we are exporting header
string[] strLines = System.IO.File.ReadAllLines(textBox1.Text);
string CarouselName = enter.Text;
int[] cols = new int[] { 15, 15, 25, 15, 15 };
StringBuilder sb = new StringBuilder();
string line = RemoveWhiteSpace(strLines[0]).Trim();
string[] cells = line.Replace("\"", "").Split('\t');
for (int c = 0; c < cells.Length; c++)
sb.Append(cells[c].Replace(" ", "_").PadRight(cols[c]));
sb.AppendLine("Location".PadRight(15));
sb.AppendLine();
int tmpCarousel = 0;
int carouselNumber = 0;
Dictionary<string, int> namesForCarousels = new Dictionary<string, int>();
for (int i = 0; i < textfile.Count; i++)
{
for (int c = 0; c < cells.Length; c++)
sb.Append(textfile[i].Cells[c].Replace(" ", "_").PadRight(cols[c]));
string name = textfile[i].Cells[1];
if (namesForCarousels.TryGetValue(name, out tmpCarousel) == false)
{
carouselNumber++;
if (carouselNumber > 45)
carouselNumber = 1;//resetting to number1, but name is
//not changing to Carousel1..
namesForCarousels[name] = carouselNumber;
}
var strCorousel = lstMX.Find(x => x.MAX_PN.Equals(name)).Carousel;
strCorousel = (String.IsNullOrEmpty(strCorousel)) ? CarouselName : strCorousel;
sb.Append(String.Format("{0}:{1}", strCorousel, carouselNumber).PadRight(15));
sb.Append("\r\n");
}
System.IO.File.WriteAllText(#"Z:\Desktop\output.TXT", sb.ToString());
}
OUTPUT i need
I need after Carousel:45 >>> i need Carousel1:1. How can i do this..?

You never use the numbers stored in your dictionary namesForCarousels after setting them. Probably you want
sb.Append(String.Format("{0}:{1}", strCorousel, namesForCarousels[name]).PadRight(15));
Also, you should rename carouselNumber to something like carouselNumberCounter. It's not the number of the current carousel, it's a counter used to assign a number to the next carousel. And for additional clarity, get rid of the local variable tmpCarousel and do:
if (!namesForCarousels.ContainsKey(name))
{

You might find it easier to follow your code if you use more descriptive variable names. It's not entirely clear what you're trying to do, but I assume you want to re-use the same carousel number for a given "Max Pn" if you've already allocated it - at the moment, you populate that mapping but you don't use it, you are reliant on max pn being in order. I don't actually see why carousel number wouldn't reset, but if you tidy it up perhaps you have a better chance of seeing what is happening.
Also given your null reference exception from your other question, this protects against that - though it probably indicates another problem elsewhere in the population of your "lstMx".
public static void just_create_text()
{
//Here we are exporting header
string[] strLines = System.IO.File.ReadAllLines(textBox1.Text);
string defaultCarouselName = enter.Text;
int[] columnPaddings = new int[] { 15, 15, 25, 15, 15 };
StringBuilder completedOutputBuilder = new StringBuilder();
string line = RemoveWhiteSpace(strLines[0]).Trim();
string[] cells = line.Replace("\"", "").Split('\t');
for (int c = 0; c < cells.Length; c++)
completedOutputBuilder.Append(cells[c].Replace(" ", "_").PadRight(columnPaddings[c]));
completedOutputBuilder.AppendLine("Location".PadRight(15));
completedOutputBuilder.AppendLine();
int carouselNumberForEntry = 0;
Dictionary<string, int> maxPnToCarouselNumber = new Dictionary<string, int>(StringComparer.OrdinalIgnoreCase);
for (int i = 0; i < textfile.Count; i++)
{
for (int c = 0; c < cells.Length; c++)
completedOutputBuilder.Append(_textfile[i].Cells[c].Replace(" ", "_").PadRight(columnPaddings[c]));
string maxPnForEntry = textfile[i].Cells[1];
int previouslyAllocatedCarouselNumberForMaxPn = 0;
if (maxPnToCarouselNumber.TryGetValue(maxPnForEntry, out previouslyAllocatedCarouselNumberForMaxPn) == false)
{
// assign a new courousel number for this max pn
carouselNumberForEntry++;
if (carouselNumberForEntry > 45)
carouselNumberForEntry = 1;
// for better clarity use add
maxPnToCarouselNumber.Add(maxPnForEntry, carouselNumberForEntry);
}
else
{
// use the carousel number previous assigned for this maxPn
carouselNumberForEntry = previouslyAllocatedCarouselNumberForMaxPn;
}
// find the related max pn carousel entry (if relatedPn is not found this suggests a problem elsewhere)
MAX_PN_Carousel relatedPn = lstMx.Find(x => x.MAX_PN != null && x.MAX_PN.Equals(maxPnForEntry, StringComparison.OrdinalIgnoreCase));
// assign the name from the entry, or use the default carousel name if unavailable
string carouselNameForMaxPn = (relatedPn == null || String.IsNullOrWhiteSpace(relatedPn.Carousel)) ? defaultCarouselName : relatedPn.Carousel;
// add the new column in the output
completedOutputBuilder.Append(String.Format("{0}:{1}", carouselNameForMaxPn, carouselNumberForEntry).PadRight(15));
completedOutputBuilder.Append("\r\n");
}
System.IO.File.WriteAllText(#"c:\dev\output.TXT", completedOutputBuilder.ToString());
}

Check for existing file

I need a way to check whether or not a picture already is in the lbl_Dias. If not add picture, if it is, move to next picture.
This is to make a list of images that is in a random order, but without dublicates.
What I got so far is this
protected void DiasShow()
{
string[] getFiles = Directory.GetFiles(HttpContext.Current.Server.MapPath("/CSS/Design/Page_Design/Dias/1920x1080/"));
for (int i = 0; i <= GetFiles.Length; i++)
{
Random FindRandom = new Random();
string RandomFileName = getFiles[FindRandom.Next(getFiles.Length)];
FileInfo ImageName = new FileInfo(RandomFileName);
string FileType = ImageName.Name.Substring(ImageName.Name.Length - 4);
if ((FileType.ToUpper() == ".JPG") || (FileType.ToUpper() == "JPEG"))
{
lbl_Dias.Text += "<img src=\"CSS/Design/Page_Design/Dias/1920x1080/" + ImageName.Name + "\" />";
}
}
}
I hope you guys can help, i'm kinda stock ^^

First of all, get rid of NumberOfImages, as it's pointless.
The whole foreach-loop is horrible since you're iterating through the collection where it isn't needed.
Secondly, you can use the Extension property of FileInfo to check for the extension string of a file - no need to substring, etc.
Thirdly, what exactly are you trying to do here?
You do realize that you will probably get random duplicates at the end of the loop, since you are not removing used images from the collection.
In the end, you don't need to check if a file exists at all, since you got it from a function that returns files that exist.
protected void DiasShow()
{
var mapPath = HttpContext.Current.Server.MapPath("/CSS/Design/Page_Design/Dias/1920x1080/");
var images =
Directory.GetFiles(mapPath).Select(
file => new FileInfo(file)).Where(fi =>
fi.Extension.EndsWith("jpg", StringComparison.OrdinalIgnoreCase) ||
fi.Extension.EndsWith("jpeg", StringComparison.OrdinalIgnoreCase)).ToList();
var rand = new Random();
while (images.Count > 0)
{
var i = rand.Next(images.Count);
lbl_Dias.Text += "<img src=\"CSS/Design/Page_Design/Dias/1920x1080/" + images[i].Name + "\"/>";
images.RemoveAt(i);
}
}

I presume you just want to list the files in a random order:
protected void DiasShow()
{
var getFiles = Directory.GetFiles(HttpContext.Current.Server.MapPath("/CSS/Design/Page_Design/Dias/1920x1080/")); //Find alle filer I en mappe
var random = RandomiseList(getFiles);
var txt = new StringBuilder();
foreach (var randomFileName in random)
{
var fileType = System.IO.Path.GetExtension(randomFileName).ToUpper();
if ((fileType == ".JPG") || (fileType == ".JPEG"))
{
var imageName = System.IO.Path.GetFileName(randomFileName);
txt.Append("<img src=\"CSS/Design/Page_Design/Dias/1920x1080/" + imageName+ "\" />");
}
}
lbl_Dias.Text += txt.ToString();
}
public static T[] RandomiseList<T>(T[] source)
{
var rand = new Random(); //no need for own seed
var list = new List<T>(source); //copy to a new list which we can remove from
var result = new T[source.Length];
for (int i = 0; i < source.Length; i++)
{
var listIndex = rand.Next(list.Count());
result[i]= list[listIndex];
list.RemoveAt(listIndex);
}
return result;
}

Since there is no code where you are checking for the file, I assume you are checking file names.
use Contains() Or Regex match after extracting out text from lbl_Dias.

You can make a list of filenames.
each time you assigned a filename you check if it's in the list, if not add it to the list, if it exists look for a different file.

To verify something is not in the text of your label have a corresponding dictionary and check it before adding it to the label such as
Dictionary<string, int> _filenames = new Dictionary<string,int>();
....
if (_filenames.ContainsKey( ImageName.Name ) == false)
{
_filenames.Add(ImageName.Name, 0);
lbl_Dias.Text += "<img src=\"CSS/Design/Page_Design/Dias/1920x1080/" + ImageName.Name + "\" />";
}