Develop Reference

faster way to iterate through and compare two lists of classes

i need to iterate through and compare 2 lists of classes and compare the 2 and output the matching records. this is taking hours and i cant figure out a way to speed up the process. the lists are roughly 600k records a piece. here is my code for the class and code for iterating through and comparing.
class Person
{
string NPI;
string address;
string zip5;
string lname;
string lsk;
string state;
string fname;
string zipfull;
string seqNo;
public Person(string npi, string Address, string Zip5, string Lname, string LSK, string st, string Fname, string zipFull, string seqno)
{
this.NPI = npi;
this.address = Address;
this.zip5 = Zip5;
this.lname = Lname;
this.lsk = LSK;
this.state = st;
this.fname = Fname;
this.zipfull = zipFull;
this.seqNo = seqno;
}
public string getNPI()
{
return NPI;
}
public string getzip5()
{
return zip5;
}
public string getaddress()
{
return address;
}
public string Full()
{
string full = NPI + "," + address + "," + zip5 + "," + lname + "," + lsk + "," + state + "," + fname + "," + zipfull + "," + seqNo;
return full;
}
}
here is the code for iterating through. the fuzz.ratio is a fuzzy matching nuget package i downloaded and i know that isnt the problem as i have done speed tests with it and it is very fast
string inputfile = #"C:\Input_File_150k.csv";
string blacklist = #"C:\Blacklist1.csv";
List<Person> input = Readcsv(inputfile);
List<Person> BL = Readcsv(blacklist);
string outputtest = #"C:\outputtest.csv";
StringBuilder csvcontent = new StringBuilder();
int lengthinput = input.Count();
for(int i = 0; i <lengthinput; i++)
{
int lengthbl = BL.Count();
for(int x = 0; x < lengthbl; x++)
{
if(input[i].getzip5() == BL[x].getzip5())
{
if(input[i].getNPI() == BL[x].getNPI())
{
if(Fuzz.Ratio(input[i].getaddress(),BL[x].getaddress()) > 90)
{
csvcontent.AppendLine(input[i].Full());
}
}
}
}
}
File.AppendAllText(outputtest, csvcontent.ToString());

Try creating a dictionary from one list to use as a lookup while iterating the other. That will change the complexity from polynomial to linear.
string inputfile = #"C:\Input_File_150k.csv";
string blacklist = #"C:\Blacklist1.csv";
List<Person> input = Readcsv(inputfile);
var blAddresses = Readcsv(blacklist).ToDictionary(
x => (Zip : x.getzip5(), NPI : x.getNPI()),
x => x.getaddress());
string outputtest = #"C:\outputtest.csv";
StringBuilder csvcontent = new StringBuilder();
int lengthinput = input.Count();
for(int i = 0; i <lengthinput; i++)
{
var zip = input[i].getzip5();
var npi = input[i].getNPI();
if(blAddresses.TryGetValue((zip,npi), out var blAddress)
{
if(Fuzz.Ratio(input[i].getaddress(),blAddress) > 90)
{
csvcontent.AppendLine(input[i].Full());
}
}
}
File.AppendAllText(outputtest, csvcontent.ToString());
Specifically I've created a dictionary that keys on the Zip and NPI and gets the Address which is all that's needed. I'm using some C# 7 stuff like the value tuples, but that can be changed to reference tuples, anonymous class, or a custom class if needed.
Edit
Here's a change to make this work just as your current code does assuming that you have duplicate Zip/NPI values
string inputfile = #"C:\Input_File_150k.csv";
string blacklist = #"C:\Blacklist1.csv";
List<Person> input = Readcsv(inputfile);
var blAddresses = Readcsv(blacklist)
.GroupBy(x => (Zip : x.getzip5(), NPI : x.getNPI()))
.ToDictionary(
grp => grp.Key,
grp => grp.Select(y => y.getAddress()).ToList());
string outputtest = #"C:\outputtest.csv";
StringBuilder csvcontent = new StringBuilder();
int lengthinput = input.Count();
for(int i = 0; i <lengthinput; i++)
{
var zip = input[i].getzip5();
var npi = input[i].getNPI();
if(blAddresses.TryGetValue((zip,npi), out var blAddressList)
{
foreach(var blAddress in blAddressList)
{
if(Fuzz.Ratio(input[i].getaddress(),blAddress) > 90)
{
csvcontent.AppendLine(input[i].Full());
}
}
}
}
File.AppendAllText(outputtest, csvcontent.ToString());
Alternatively if you just need to filter out anything in the black list where the NPI is blank to have unique keys you can do this instead
var blAddresses = Readcsv(blacklist)
.Whree(x => x.getNPI().Length > 0)
.ToDictionary(
x => (Zip : x.getzip5(), NPI : x.getNPI()),
x => x.getaddress());

You could iterate through each list once to put all the values in a hash map. For instance, you can use the zip code as the key, and the value is an array of people who match that zip code.
Then you can iterate through the hashmap one record at a time, and only have to compare the people inside each hashmap bucket with eachother.
Unless all your people are in the same zip code (if so, hopefully one of your keys would work for this), this should be faster than N^2 comparison. Should be closer to O(N) depending on how many people are in each bucket.

in addition to above comments ( https://stackoverflow.com/a/60886352/5334191 and using Dictionaries),
in your loops
int lengthinput = input.Count();
**int lengthbl = BL.Count();** //move out of the loops
for(int i = 0; i <lengthinput; i++)
{
**var inputi = input[i];** //move out of the inner loop
for(int x = 0; x < lengthbl; x++)
{
**var blx = BL[x];**
if(inputi.getzip5() == blx.getzip5())
{
if(inputi.getNPI() == blx.getNPI())
{
if(Fuzz.Ratio(inputi.getaddress(),blx.getaddress()) > 90)
{
csvcontent.AppendLine(inputi.Full());
}
}
}
}
}

I would definity implement the IEqualityComparer interface.
That allows for much cleaner code and to also benefit from some of the Linq extension methods available on collections.
Those also support things like parallelism.
https://marcofranssen.nl/delegate-your-equality-comparisons/

I think this part of code may be slow
string full = NPI + "," + address + "," + zip5 + "," + lname + "," + lsk + "," + state + "," + fname + "," + zipfull + "," + seqNo;
What if you perform csvcontent.Append on those properties and , "manually" instead of using Full()? ps: it would require to add an possibility to read it from the outside, so public get / private set

ArgumentOutOfRangeException: startIndex cannot be longer than the length of string

When I run the compiler it throws an exception:
ArgumentOutOfRangeException: startIndex cannot be longer than the
length of string.
I do not understand why, the string which I write into the compiler consists of 10 numbers, and i want the substring to pick the 9th number, but obviously I'm doing something wrong.
Example of string: 9303140456
Code:
public string kollaKön()
{
string siffraAsString = form.textBox3.Text.Substring(8, 1);
int siffraAsNum = int.Parse(siffraAsString);
int result = (siffraAsNum % 2);
if (result == 1)
{
return form.textBox5.Text = ("Är en Man");
}
else
{
return form.textBox5.Text = (" Är en Kvinna");
}
}
Complete code, maybe makes my mistake obvious:
namespace WindowsFormsApplication3
{
public class Person
{
Form1 form = new Form1();
string förnamn;
string efternamn;
string personnummer;
string Kön;
public Person(string förNamn, string efterNamn, string personNummer)
{
string förnamn = förNamn;
string efternamn = efterNamn;
string personnummer = personNummer;
}
static bool luhn(string personNummer)
{
int sum = 0;
for (int i = 0; i < personNummer.Length; i++)
{
int temp = (personNummer[i] - '0') << (1 - (i & 1));
if (temp > 9) temp -= 9;
sum += temp;
}
return (sum % 10) == 0;
}
public string kollaKön()
{
string debuggerIsMyFriend = form.textBox3.Text;
string siffraAsString = form.textBox3.Text.Substring(8, 1);
int siffraAsNum = int.Parse(siffraAsString);
int result = (siffraAsNum % 2);
if (result == 1)
{
return form.textBox5.Text = ("Är en Man");
}
else
{
return form.textBox5.Text = (" Är en Kvinna");
}
}
}
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}
private void button2_Click(object sender, EventArgs e)
{
string förnamn = textBox1.Text;
string efternamn = textBox2.Text;
string personnummer = textBox3.Text;
string persnr = "";
if (personnummer.Length != 10)
{
persnr = " Personnummer är i fel format. ";
}
if (personnummer.Length == 10)
{
persnr = " Personnummers är i korrekt format.";
}
Person p1 = new Person(förnamn, efternamn, personnummer);
p1.kollaKön();
textBox4.Text = förnamn + " " + efternamn + " " + personnummer + " " + " " + persnr;
}
}
}

This line in your Person class
Form1 form = new Form1();
declares and initialize a new Instance of Form1. This instance is not the one that you have shown on your monitor. When you type your strings you type them on the instance that is shown on monitor not into the textboxes of this hidden instance.
Of course this means that when you write
string siffraAsString = form.textBox3.Text.Substring(8, 1);
you are taking the text of the hidden textbox that is empty and the following Substring code fails with the Index Out of Range Exception
You need to change you Person class and add properties that receive the content of the textboxes required for your calculations.
For example
public class Person
{
// Sorry but I have no idea of what Siffra means...
// change the name as you wish
public string Siffra {get;set;}
...... other properties
public string kollaKön()
{
// A defensive attitude is always a good practice.
if(string.IsNullOrEmpty(this.Siffra) ||
this.Siffra.Length < 10)
return "Invalid Siffra";
string siffraAsString = this.Siffra.Substring(8, 1);
int siffraAsNum = int.Parse(siffraAsString);
int result = (siffraAsNum % 2);
return (result == 1 ? "Är en Man" : " Är en Kvinna");
}
}
and in your button click on the form you run
.....
Person p1 = new Person(förnamn, efternamn, personnummer);
p1.Siffra = this.textBox3.Text;
string result = p1.kollaKön();
this.textBox5.Text = result;

Before all your code, add this line
string debuggerIsMyFriend = form.textBox3.Text;
Set a breakpoint on that line, run the code and hover the mouse over the variable to see what it actually contains, I bet it's not the 10 characters string you are expecting

You should handle this more gracefully. This will handle your edge cases properly.
The string you're getting in that TextBox is most certainly not 9 characters long or more, and unless you have 100% certainty that your input is 9 characters, you should handle it gracefully.
Rule of thumb: Don't try to access the index of an array without checking its length first.
...Second rule of thumb: Also make sure you're accessing the correct UI control if you think something's fishy.
public string kollaKön()
{
//string siffraAsString = form.textBox3.Text.Substring(8, 1);
int siffraAsNum = GetNinthNumber(form.textBox3.Text);
int result = (siffraAsNum % 2);
if (result == 1)
{
return form.textBox5.Text = ("Är en Man");
}
else
{
return form.textBox5.Text = (" Är en Kvinna");
}
}
public static int GetNinthNumber(string str)
{
string numberField = str.Trim();
if (numberField.Length < 9)
{
// Handle less than 9 cases
return 0;
}
else
{
int number;
bool isNumber = Int32.TryParse(str[8].ToString(), out number);
if (!isNumber)
{
throw new Exception("Value is not a number");
}
return number;
}
}

I'm very sure that your line string siffraAsString = form.textBox3.Text.Substring(8, 1); doesn't have 10 numbers.
Checks the value of form.textBox3.Text as say Jacopo.

c# regular expression getting specific string from string[CLOSE]

help i want to get the specific string from my string x="Glass 1 1000"; i want to get the string "Glass" only and save it to my string type.
int[] quanvalue = new int[2];
int x1 = 0;
string type = "";
string x="Glass 1 1000";
string[] numbers = Regex.Split(x, #"\D+");
foreach (string value in numbers)
{
if (!string.IsNullOrEmpty(value))
{
int ib = int.Parse(value);
quanvalue[x1] = ib;
MessageBox.Show(quanvalue[0].ToString() + " " + quanvalue[1].ToString());
x1++;
}
else
{
// i want to get string from here
}

string sub = x.Substring(0, 5);

You can use a substring function to fetch the first 5 characters from x.
And save it in x itself

Split numeric and Non-Numeric in a String

If I have a string value like this "1234-", then I need to split till the non-numeric character that is - and add numeric value 1 after the non-numeric char. later I have to update the value to "1234-1". Then the program will check with the last updated value 1234-1 then it will increment by 1 every time and store it for future use. If no non-numeric in a string then the program will increment by 1 with the numeric string.
Below are some examples of String and Output Value
Ex Str1 Output
2014- 2014-1
2014-1 2014-2
AAA AAA1
ABC-ABC ABC-ABC1
12345 12346
1234AA 1234AA1
I have used the below code before.
Code
var SiteFile = (from site in db.SiteFiles where site.Code == "ACQPONUM" select site.Line2).FirstOrDefault(); // Get Input string to generate AUTO number.
int Count = (from Porders in db.Porders where Porders.No.StartsWith(SiteFile) select Porders.No).ToList().Count; // Get the count of matching values in db.
var PONo = (from Porders in db.Porders where Porders.No.StartsWith(SiteFile) select Porders.No).ToList(); // Get list of Matching existing values.
if (Count != 0)
{
if (PONo != null)
{
int Val = (from PONos in PONo let value = Regex.Match(PONos, #"\d+").Value select Convert.ToInt32(value == string.Empty ? "0" : Regex.Match(PONos, #"\d+").Value) + 1).Concat(new[] { 0 }).Max(); // Fiind the maximum value in the matched list nd Increment value by if same type exists in the db.
porder.No = SiteFile + Val.ToString();
}
}
else
{
porder.No = SiteFile + "1";
}
Any help to this will be appreciated.

Maybe something like this:
string s = "123419";
string res = null;
char ch = s[s.Length - 1];
if(char.IsDigit(ch)) // handle numbers
{
res = s.Substring(0,s.Length - 1);
string suffix = null;
// special case
if(ch == '9'){
suffix = "10";
}
else
{
suffix = (++ch).ToString();
}
res += suffix;
}
else
{
res = string.Format("{0}1", s);
}

Try this code:
private string Incrementvalue(string str)
{
string retVal;
if (str.Contains(DELIMITER))
{
string[] parts = str.Split(new char[] { DELIMITER }, 2);
string origSuffix = parts[1];
string newSuffix;
int intSuffix;
if (int.TryParse(origSuffix, out intSuffix))
//Delimiter exists and suffix is already a number: Increment!
newSuffix = (intSuffix + 1).ToString();
else
//Delimiter exists and suffix is NTO number: Add a "1" suffix.
newSuffix = origSuffix + 1;
retVal = parts[0] + DELIMITER + newSuffix;
}
else
{
int temp;
if (int.TryParse(str, out temp))
{
//Delimiter does not exists and the input is a number: Increment last digit!
string newSuffix = (int.Parse(str[str.Length - 1].ToString()) + 1).ToString();
retVal = str.Substring(0, str.Length - 1) + newSuffix;
retVal = str.Substring(0, str.Length - 1) + newSuffix;
}
else
{
//Delimiter does not exists and the input is NOT a number: Add a "1" suffix.
retVal = str + "1";
}
}
return retVal;
}
The code could be written in a much more compact manner, but think this will be more readable and it will work...

Replacing character after single quote

I am using the following C# code to modify a lowercase letter to uppercase after a single quote:
public virtual string FirstName
{
get { return _firstName; }
set
{
if (value != null)
{
int pos = value.IndexOf("'", 0);
int strlength = value.Length - 1;
if (pos >= 0 && pos != strlength)
{
string temp = value[pos + 1].ToString();
temp = temp.ToUpper();
value = value.Remove(pos + 1, 1);
value = value.Insert(pos + 1, temp);
}
}
}
}
To me this looks like overkill. Is there an easier way to achieve the desired result:
Value: Mc'donald
Expected: Mc'Donald

here is without regex
int pos = data.IndexOf("'");
if (pos >= 0 && pos < data.Length - 1)
{
StringBuilder sbl = new StringBuilder(data);
sbl[pos + 1] = char.ToUpper(sbl[pos + 1]);
data = sbl.ToString();
}

Since you're open to Regex, would this overload of the Regex.Replace do what you need?
Regex.Replace Method (String, MatchEvaluator)
Here's a modified version of the example given at the link above. I've changed it to use the '\w pattern and to return the match in upper case.
using System;
using System.Text.RegularExpressions;
class RegExSample
{
static string CapText(Match m)
{
// Return the match in upper case
return m.ToString().ToUpperInvariant();
}
static void Main()
{
string text = "Mc'donald";
System.Console.WriteLine("text=[" + text + "]");
Regex rx = new Regex(#"'\w");
string result = rx.Replace(text, new MatchEvaluator(RegExSample.CapText));
System.Console.WriteLine("result=[" + result + "]");
}
}

Perhaps regular expressions?
string value = "Mc'donald";
string found = Regex.Match(value, "'[\\w]").Value;
string result = value.Replace(found, found.ToUpper());
Console.WriteLine(result); // Mc'Donald