Develop Reference

Fastest way to fuzzy match two csv files

I have written a very simple program using a nuget package in c# to read in 2 csv files and fuzzy match them and output a new csv file with all the matches. The problem is i need the program to be able to read and compare files up to 700k and comparw it to 100k. I havent been able to find a way to speed up the process. Is there any way i can do this? I will even use another language if need be.
you can ignore all the commented code its just there for when i was using it for testing purposes. sorry im a newer programmer.
the read csv funciton is for reading in the csv. the rest is code inside another function where i pass in the string arrays to pass them through fuzzymatch
static string[] ReadCSV(string path)
{
List<string> name = new List<string>();
List<string> address = new List<string>();
List<string> city = new List<string>();
List<string> state = new List<string>();
List<string> zip = new List<string>();
using (var reader = new StreamReader(path))
{
reader.ReadLine();
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
var values = line.Split(',');
name.Add(values[0] +", "+ values[1]);
//address.Add(values[1]);
//city.Add(values[2]);
//state.Add(values[3]);
//zip.Add(values[4]);
}
}
string[] name1 = name.ToArray();
return name1;
//foreach (var item in name)
//{
// Console.WriteLine(item.ToString());
//}
}
StringBuilder csvcontent = new StringBuilder();
string csvpath = #"C:\Users\bigel\Documents\outputtest.csv";
csvcontent.AppendLine("Name,Address,Match");
//Console.WriteLine("Levenshtein Edit Distance:");
int x = 1;
foreach (var name in string1)
{
for (int i = 0; i < length; i++)
{
int leven = match[i].LevenshteinDistance(name);
//Console.WriteLine(match[i] + "\t{0} against {1}", leven, name);
if (leven <= 7)
{
output[i] = input[i] + ",match";
csvcontent.AppendLine(output[i]);
//Console.WriteLine(match[i] + " " + leven + " against " + name + " is a Match");
//Console.WriteLine(output[i]);
}
else
{
if (i == 500)
{
Console.WriteLine(x);
x++;
}
}
}
}
File.AppendAllText(csvpath, csvcontent.ToString());

use streamreader find the specific user entry in the text file and display the 5th component in label

Data was passed from another form which containing a user entry, using Streamreader find the specific user entry in the text file and display the 5th component in the array.
in the login form, I passed the user entry: seow into this form, how do I make sure it displays 20, in label3?
in account.txt
seow 1111 wen 12345 20 50
user 1234 user1 12345 70 80
C# Code:
List<string> user = new List<string>();
private void Balance_Load(object sender, EventArgs e)
{
string username;
username = login.accountname;
StreamReader sr = new StreamReader("account.txt");
string line = "";
if (user.Contains(username))
{
while ((line = sr.ReadLine()) != null)
{
string[] components = line.Split(" ".ToCharArray(), StringSplitOptions.RemoveEmptyEntries);
label3.Text = string.Join(" ", components[5]);
}
sr.Close();
}
}

here is an example of I have tried
string path1 = AppDomainAppPath + "\\Subscribtion\\Subscribe.txt";
System.IO.StreamReader file = new System.IO.StreamReader(path1);
var lines = File.ReadAllLines(path1);
for (var i = 0; i < lines.Length; i += 1)
{
// Process line
txtcollege.Value = lines[1];
txtplace.Value = lines[3];
}
file.Close();
ouput

ive madeit by changing the condition, thanks for the ideas
while ((line = sr.ReadLine()) != null)
{
string[] components = line.Split(" ".ToCharArray(), StringSplitOptions.RemoveEmptyEntries);
if( components.Contains(username))
{
curbal = string.Join(" ", components[5]);
label3.Text = curbal;
savbal = string.Join(" ", components[6]);
label5.Text = savbal;
}
}
sr.Close();

Comparing rows in a csv file

I am currently using the below code to compare two csv files with each other. This code gives an output with all the rows that are not the same. But when a row is missing everything after that row is not the same. How can I fix this? Thanks in advance.
List<string> lines = new List<string>();
List<string> lines2 = new List<string>();
try
{
StreamReader reader = new StreamReader(System.IO.File.OpenRead(file1));
StreamReader read = new StreamReader(System.IO.File.OpenRead(file2));
List<string> differences = new List<string>();
string line;
string line2;
int i = 0;
while ((line = reader.ReadLine()) != null && (line2 = read.ReadLine()) != null)
{
string[] split = line.Split(Convert.ToChar("\t"));
string[] split2 = line2.Split(Convert.ToChar("\t"));
if (split[i] != split2[i])
{
differences.Add("this row is not the same:, " + line);
}
else
{
}
i++;
}
System.IO.File.WriteAllLines(differencesFile, differences);
reader.Dispose();
read.Dispose();
}
catch
{
}

After help from a friend I made it work with this code:
List<string> file1 = new List<string>();
List<string> output = new List<string>();
string differencesFile = path;
File.WriteAllText(differencesFile, "");
try
{
StreamReader readFile1 = new StreamReader(System.IO.File.OpenRead(pathfile1));
string lineFile1;
while ((lineFile1 = readFile1.ReadLine()) != null)
{
bool match = false;
string[] colums = lineFile1.Split('\t');
StreamReader readFile2 = new StreamReader(System.IO.File.OpenRead(pathfile2));
string line2;
while ((line2 = readFile2.ReadLine()) != null)
{
string[] columsFile2 = line2.Split('\t');
if (colums[0] == columsFile2[0])
{
match = true;
}
}
if (!match)
{
output.Add(colums[0] + "; doesnt exist in pathfile2");
}
}
System.IO.File.WriteAllLines(differencesFile, output);
}
catch { }

Read text file and split it over

So, I know my headline is a bit confusing, I will explain.
My code looks like this:
string filename = "C:\\C#\\maplist.txt"; // please put the text file path.
string filename2 = "C:\\C#\\zemaplist.txt";
string map;
StreamReader sr = new StreamReader(filename);
StreamWriter sw = new StreamWriter(filename2);
List<string> maps = new List<string> { };
while ((map = sr.ReadLine()) != null)
{
maps.Add(map);
}
sr.Close();
for (int i = 0; i < maps.Count; i++)
{
Console.WriteLine(maps[i]);
sw.WriteLine(maps[i]);
}
sw.Close();
and what i need to do is when the code read a new line, in my line there is
"Hey,Hey"
I want to split the , from each other so I can take both of them as other parameters, so that the first Hey will be added to maps and the other hey will be maps2,
How can I do that?

You can use Split() function to Split the given String based on delimiter.
Try This:
while ((map = sr.ReadLine()) != null)
{
maps.Add(map.Split(',')[0].Trim());
maps2.Add(map.Split(',')[1].Trim());
}
Simple Code:
using System.IO;
string filename = "C:\\C#\\maplist.txt"; // please put the text file path.
string filename2 = "C:\\C#\\zemaplist.txt";
string map;
StreamWriter sw = new StreamWriter(filename2);
List<string> maps = new List<string> { };
List<string> maps2 = new List<string> { };
String [] allLines = File.ReadAllLines(filename);
foreach(String line in allLines)
{
maps.Add(line.Split(',')[0].Trim());
maps2.Add(line.Split(',')[1].Trim());
}
for (int i = 0; i < maps.Count; i++)
{
Console.WriteLine(maps[i]);
sw.WriteLine(maps[i]);
}
sw.Close();
Solution 2:
String mapItem1="";
String mapItem2="";
if(maps.Count == maps2.Count)
{
for(int i=0;i<maps.Count;i++)
{
mapItem1=maps[i];
mapItem2=maps2[i];
}
}

while ((map = sr.ReadLine()) != null)
{
string[] split = map.Split(',');
//First Hey would be split[0], second Hey would be split[1]
maps.Add(split[0].Trim());
maps2.Add(split[1].Trim());
}
The Split method should help you out with that.
If you want to trim leading whitespace characters, you can use the .Trim() method on a string.

Use Split().
string heys = "Hey,Hey";
string[] splitArray = heys.Split(',');
Then you have:
splitArray[0] = "Hey";
splitArray[1] = "Hey";

Why even bother reading line by line? Read the entire file, replace the new line chars for a "," (to prevent last and first elements from different lines to be treated as one), and loop through a clean string.
string fileContent = Regex.Replace(File.ReadAllText("test.txt"), #"\r", ",");
List<string> mapList = new List<string>();
foreach (string map in Regex.Split(fileContent.Replace(#"\s+", ""), ","))
{
mapList.Add(map.Trim());
}

File name from StreamReader C# - asp.net MVC3 to array

my application is MVC3 C#; I am populating two dropdownlists using json using the following:
public ActionResult CheckWord(string cword)
{
try
{
List<string[]> arrayList = new List<string[]>();
List<string[]> stateList = new List<string[]>();
//
List<string[]> fileList = new List<string[]>();
//
string[] filePaths = Directory.GetFiles(System.Web.HttpContext.Current.Server.MapPath("/Video"), "*.srt");
string[] fnList = new string[filePaths.Length];
for (int i = 0; i < fnList.Length; ++i)
{
FileInfo fi = new FileInfo(filePaths[i]);
fnList[i] = fi.Name.Substring(0, fi.Name.LastIndexOf(".srt"));
}
int nFiles = filePaths.Length;
string cacheline = "";
string line;
for (int i = 0; i < nFiles; ++i)
{
StreamReader file = new StreamReader(System.Web.HttpContext.Current.Server.MapPath("/Video/" + fnList[i] + ".srt"));
List<string> lines = new List<string>();
List<string> statments = new List<string>();
//
List<string> fnames = new List<string>();
//
while ((line = file.ReadLine()) != null)
{
if (line.Contains(cword))
{
statments.Add(line);
// fnames.Add(file);
lines.Add(cacheline);
}
cacheline = line;
}
file.Close();
var array = lines.ToArray();
arrayList.Add(array);
stateList.Add(statments.ToArray());
}
return Json(new { success = true, fnList = fnList, arrayList = arrayList.ToArray(), stateList = stateList.ToArray() });
}
catch { }
return Json(new { success = false });
}
I am checking if a word exists in a group of files; then display the names of files in one dropdownlist and the lines from each file in the other dropdownlist. It works fine, however it gives me a list of all files becasue I am sending back fnlist. However I am trying to display only the files that contain that word; I could not get the file name from the StreamReader and add it to an array fileList. I would appreciate your suggestions, thanks in advance.

Already so many lists! Why not another? You already open the file with fnList[i] within the context of the loop, so...
List<string[]> results = new List<string[]>();
....
while ((line = file.ReadLine()) != null) {
if (line.Contains(cword)) {
results.Add(fnList[i]);
break; // optional, if possible, but if you need to continue check for dupes
}
}
....
return Json(new {
success = true,
fnList = results.ToArray(),
arrayList = arrayList.ToArray(),
stateList = stateList.ToArray()
});

System.IO.StreamReader file = new System.IO.StreamReader("setup.txt");
Later on, we would like to print the name of the file being used by stream reader.
eg, if there is an error, I would like a message box that displays "error reading file: 'filename'"
MessageBox.Show("Error loading " + ((FileStream)file.BaseStream).Name);

Not sure what exactly you are looking for but since you are creating StreamReader from a file name why not have file name in a separate variable and use it later:
var fileName = System.Web.HttpContext.Current.Server.MapPath(
"/Video/" + fnList[i] + ".srt");
StreamReader file = new StreamReader(fileName);