In a list with some hundred thousand entries, how does one go about comparing each entry with the rest of the list for duplicates?
For example, List fileNames contains both "00012345.pdf" and "12345.pdf" and are considered duplicte. What is the best strategy to flagging this kind of a duplicate?
Thanks
Update: The naming of files is restricted to numbers. They are padded with zeros. Duplicates are where the padding is missing. Thus, "123.pdf" & "000123.pdf" are duplicates.
You probably want to implement your own substring comparer to test equality based on whether a substring is contained within another string.
This isn't necessarily optimised, but it will work. You could also possibly consider using Parallel Linq if you are using .NET 4.0.
EDIT: Answer updated to reflect refined question after it was edited
void Main()
{
List<string> stringList = new List<string> { "00012345.pdf","12345.pdf","notaduplicate.jpg","3453456363234.jpg"};
IEqualityComparer<string> comparer = new NumericFilenameEqualityComparer ();
var duplicates = stringList.GroupBy (s => s, comparer).Where(grp => grp.Count() > 1);
// do something with grouped duplicates...
}
// Not safe for null's !
// NB do you own parameter / null checks / string-case options etc !
public class NumericFilenameEqualityComparer : IEqualityComparer<string> {
private static Regex digitFilenameRegex = new Regex(#"\d+", RegexOptions.Compiled);
public bool Equals(string left, string right) {
Match leftDigitsMatch = digitFilenameRegex.Match(left);
Match rightDigitsMatch = digitFilenameRegex.Match(right);
long leftValue = leftDigitsMatch.Success ? long.Parse(leftDigitsMatch.Value) : long.MaxValue;
long rightValue = rightDigitsMatch.Success ? long.Parse(rightDigitsMatch.Value) : long.MaxValue;
return leftValue == rightValue;
}
public int GetHashCode(string value) {
return base.GetHashCode();
}
}
I understand you are looking for duplicates in order to remove them?
One way to go about it could be the following:
Create a class MyString which takes care of duplication rules. That is, overrides Equals and GetHashCode to recreate exactly the duplication rules you are considering. (I'm understanding from your question that 00012345.pdf and 12345.pdf should be considered duplicates?)
Make this class explicitly or implictly convertible to string (or override ToString() for that matter).
Create a HashCode<MyString> and fill it up iterating through your original List<String> checking for duplicates.
Might be dirty but it will do the trick. The only "hard" part here is correctly implementing your duplication rules.
I have a simple solution for everyone to find a duplicate string word and cahracter
For word
public class Test {
public static void main(String[] args) {
findDuplicateWords("i am am a a learner learner learner");
}
private static void findDuplicateWords(String string) {
HashMap<String,Integer> hm=new HashMap<>();
String[] s=string.split(" ");
for(String tempString:s){
if(hm.get(tempString)!=null){
hm.put(tempString, hm.get(tempString)+1);
}
else{
hm.put(tempString,1);
}
}
System.out.println(hm);
}
}
for character use for loop, get array length and use charAt()
Maybe somthing like this:
List<string> theList = new List<string>() { "00012345.pdf", "00012345.pdf", "12345.pdf", "1234567.pdf", "12.pdf" };
theList.GroupBy(txt => txt)
.Where(grouping => grouping.Count() > 1)
.ToList()
.ForEach(groupItem => Console.WriteLine("{0} duplicated {1} times with these values {2}",
groupItem.Key,
groupItem.Count(),
string.Join(" ", groupItem.ToArray())));
Related
Sorry if the title, is confusing, I had some trouble putting my problem into words.
I have a List, where every string is composed of 2 words, delimited by space.
For example:
{ "word1 word2", "wordA wordB", "dog cat", "mouse cat" }
I want to use OrderBy to sort the list by the 2nd word, if any words are equal, I then want to sort those by the 1st word. I'm having trouble figuring out how to handle the 2nd condition for this (sorting by 1st word only if 2nd words are equal).
I originally tried:
public List<string> SpecialSort(List<string> text)
{
return text.OrderBy(x => x.Split(' ')[1]).ThenBy(x => x.Split(' ')[0]);
}
but this seems to just sort first by the 2nd word, and then re-sort everything by the 1st word. Is there a way for me to do this where I only sort by 1st word if the 2nd words are equal?
Thanks!
My advice would be to split the text into words, while keeping the original text in a Select. Then sort the sequence and finally remove the split words.
Requirement
Input: a sequence of strings, every string has exactly one space.
This space is neither the first nor the last character.
The characters before this one and only space are defined as the first word.
The characters after the space are defined as the second word.
Output: Sort the sequence by 2nd word, then by 1st word.
IEnumerable<string> inputTexts = ...
const string splitChar = ' ';
// first add the split words
var sortedSequence = inputTexts.Select(txt => new
{
Original = txt,
Split = txt.Split(splitChar, StringSplitOptions.None),
})
// then sort by the split words
.OrderBy(splitTxt => splitTxt.Split[1])
.ThenBy(splitTxt => splitTxt.Split[0])
// finally remove the split words
.Select(splitTxt => splitTxt.Original);
Create intermediate results within an .OrderBy() statement can be painful, cause the comparer needs to possibly call them multiple times on each object. Also to make it better maintainable I would write a class that gets the original value, creates the desired elements and feeding these intermediate objects into a specific comparer that can sort them. At the end just get the original value out of the intermediate class and you're done.
A rough sketch for your example would look something like this:
using System;
using System.Collections.Generic;
using System.Linq;
public static class Program
{
private static void Main(string[] args)
{
var words = new List<string>{"word1 word2", "wordA wordB", "dog cat", "mouse cat"};
var ordered = words
.Select(SpecialComparerInstance.Create)
.OrderBy(special => special, SpecialComparer.Default)
.Select(special => special.Value);
foreach (var item in ordered)
{
Console.WriteLine(item);
}
}
}
public class SpecialComparerInstance
{
public static SpecialComparerInstance Create(string value) => new SpecialComparerInstance(value);
public SpecialComparerInstance(string value)
{
if (string.IsNullOrEmpty(value))
throw new ArgumentNullException(nameof(value));
var elements = value.Split(' ');
if (elements.Length != 2)
throw new ArgumentException("Must contain exactly one space character", nameof(value));
Value = value;
FirstOrderValue = elements[1];
SecondOrderValue = elements[0];
}
public string Value { get; }
public string FirstOrderValue { get; }
public string SecondOrderValue { get; }
}
public class SpecialComparer : IComparer<SpecialComparerInstance>
{
public static readonly IComparer<SpecialComparerInstance> Default = new SpecialComparer(StringComparer.Ordinal);
private readonly StringComparer _comparer;
public SpecialComparer(StringComparer comparer)
{
_comparer = comparer;
}
public int Compare(SpecialComparerInstance x, SpecialComparerInstance y)
{
if (ReferenceEquals(x, y))
return 0;
if (ReferenceEquals(x, null))
return 1;
if (ReferenceEquals(y, null))
return -1;
var result = _comparer.Compare(x.FirstOrderValue, y.FirstOrderValue);
if (result == 0)
result = _comparer.Compare(x.SecondOrderValue, y.SecondOrderValue);
return result;
}
}
I use System.Linq.Dynamic to order an items list.
items = items.AsQueryable().OrderBy("Name ASC");
To my surprise, lowercase names gets ordered after the capital cased names, so the items are returned something like this.
Ape
Cat
Dog
alligator
ant
beetle
I expected this order:
alligator
ant
Ape
beetle
Cat
Dog
Is there a way to get the correct order? Checked all method signatures for OrderBy and googled around, but nada.
You do not need to create a custom comparer because there's already a StringComparer class which derives from IComparer.
words.OrderBy (x => x, StringComparer.OrdinalIgnoreCase)
This way, you do not need to create different IComparer implementations if you wanted to use other string comparison methods, like StringComparer.InvariantCultureIgnoreCase.
However, this might be desirable depending on your situation. For example, I do have multiple extension methods defined in LINQPad, like OrderBySelfInvariantCultureIgnoreCase, because it is convenient to use this with code completion rather than typing out the equivalent code by hand:
public static IEnumerable<string> OrderBySelfInvariantCultureIgnoreCase(this IEnumerable<string> source)
{
return source.OrderBy (x => x, StringComparer.InvariantCultureIgnoreCase);
}
You must create a custom comparer, such as:
public void Main()
{
String[] words = { "aPPLE", "AbAcUs", "bRaNcH", "BlUeBeRrY", "ClOvEr", "cHeRry" };
var sortedWords = words.OrderBy(a => a, new CaseInsensitiveComparer());
ObjectDumper.Write(sortedWords);
}
public class CaseInsensitiveComparer : IComparer<string>
{
public int Compare(string x, string y)
{
return string.Compare(x, y, StringComparison.OrdinalIgnoreCase);
}
}
Found # https://code.msdn.microsoft.com/SQL-Ordering-Operators-050af19e
I have faced the same issue and found no easy solution over the internet. Then I was trying in many ways and finally got a very simple way. It completely worked for me. My solution is
string sort = "Name ASC";
string[] data = sort.Split(" ");
items.OrderBy($"{data[0].ToUpper() data[1]}");
Now the output is alligator,
ant,
Ape,
beetle,
Cat,
Dog
I have a list of string[].
List<string[]> cardDataBase;
I need to sort that list by each list-item's second string value (item[1]) in custom order.
The custom order is a bit complicated, order by those starting characters:
"MW1"
"FW"
"DN"
"MWSTX1CK"
"MWSTX2FF"
then order by these letters following above starting letters:
"A"
"Q"
"J"
"C"
"E"
"I"
"A"
and then by the numbers following above.
a sample, unordered list left, ordered right:
MW1E10 MW1Q04
MWSTX2FFI06 MW1Q05
FWQ02 MW1E10
MW1Q04 MW1I06
MW1Q05 FWQ02
FWI01 FWI01
MWSTX2FFA01 DNC03
DNC03 MWSTX1CKC02
MWSTX1CKC02 MWSTX2FFI03
MWSTX2FFI03 MWSTX2FFI06
MW1I06 MWSTX2FFA01
I tried Linq but I am not that good in it right now and cannot solve this on my own. Do I need a dictionary, regex or a dictionary with regex in it? What would be the best approach?
I think you're approaching this incorrectly. You're not sorting strings, you're sorting structured objects that are misrepresented as strings (somebody aptly named this antipattern "stringly typed"). Your requirements show that you know this structure, yet it's not represented in the datastructure List<string[]>, and that's making your life hard. You should parse that structure into a real type (struct or class), and then sort that.
enum PrefixCode { MW1, FW, DN, MWSTX1CK, MWSTX2FF, }
enum TheseLetters { Q, J, C, E, I, A, }
struct CardRecord : IComparable<CardRecord> {
public readonly PrefixCode Code;
public readonly TheseLetters Letter;
public readonly uint Number;
public CardRecord(string input) {
Code = ParseEnum<PrefixCode>(ref input);
Letter = ParseEnum<TheseLetters>(ref input);
Number = uint.Parse(input);
}
static T ParseEnum<T>(ref string input) { //assumes non-overlapping prefixes
foreach(T val in Enum.GetValues(typeof(T))) {
if(input.StartsWith(val.ToString())) {
input = input.Substring(val.ToString().Length);
return val;
}
}
throw new InvalidOperationException("Failed to parse: "+input);
}
public int CompareTo(CardRecord other) {
var codeCmp = Code.CompareTo(other.Code);
if (codeCmp!=0) return codeCmp;
var letterCmp = Letter.CompareTo(other.Letter);
if (letterCmp!=0) return letterCmp;
return Number.CompareTo(other.Number);
}
public override string ToString() {
return Code.ToString() + Letter + Number.ToString("00");
}
}
A program using the above to process your example might then be:
static class Program {
static void Main() {
var inputStrings = new []{ "MW1E10", "MWSTX2FFI06", "FWQ02", "MW1Q04", "MW1Q05",
"FWI01", "MWSTX2FFA01", "DNC03", "MWSTX1CKC02", "MWSTX2FFI03", "MW1I06" };
var outputStrings = inputStrings
.Select(s => new CardRecord(s))
.OrderBy(c => c)
.Select(c => c.ToString());
Console.WriteLine(string.Join("\n", outputStrings));
}
}
This generates the same ordering as in your example. In real code, I'd recommend you name the types according to what they represent, and not, for example, TheseLetters.
This solution - with a real parse step - is superior because it's almost certain that you'll want to do more with this data at some point, and this allows you to actually access the components of the data easily. Furthermore, it's comprehensible to a future maintainer since the reason behind the ordering is somewhat clear. By contrast, if you chose to do complex string-based processing it's often very hard to understand what's going on (especially if it's part of a larger program, and not a tiny example as here).
Making new types is cheap. If your method's return value doesn't quite "fit" in an existing type, just make a new one, even if that means 1000's of types.
A bit spoonfeeding, but I found this question pretty interesting and perhaps it will be useful for others, also added some comments to explain:
void Main()
{
var cardDatabase = new List<string>{
"MW1E10",
"MWSTX2FFI06",
"FWQ02",
"MW1Q04",
"MW1Q05",
"FWI01",
"MWSTX2FFA01",
"DNC03",
"MWSTX1CKC02",
"MWSTX2FFI03",
"MW1I06",
};
var orderTable = new List<string>[]{
new List<string>
{
"MW1",
"FW",
"DN",
"MWSTX1CK",
"MWSTX2FF"
},
new List<string>
{
"Q",
"J",
"C",
"E",
"I",
"A"
}
};
var test = cardDatabase.Select(input => {
var r = Regex.Match(input, "^(MW1|FW|DN|MWSTX1CK|MWSTX2FF)(A|Q|J|C|E|I|A)([0-9]+)$");
if(!r.Success) throw new Exception("Invalid data!");
// for each input string,
// we are going to split it into "substrings",
// eg: MWSTX1CKC02 will be
// [MWSTX1CK, C, 02]
// after that, we use IndexOf on each component
// to calculate "real" order,
// note that thirdComponent(aka number component)
// does not need IndexOf because it is already representing the real order,
// we still want to convert string to integer though, because we don't like
// "string ordering" for numbers.
return new
{
input = input,
firstComponent = orderTable[0].IndexOf(r.Groups[1].Value),
secondComponent = orderTable[1].IndexOf(r.Groups[2].Value),
thirdComponent = int.Parse(r.Groups[3].Value)
};
// and after it's done,
// we start using LINQ OrderBy and ThenBy functions
// to have our custom sorting.
})
.OrderBy(calculatedInput => calculatedInput.firstComponent)
.ThenBy(calculatedInput => calculatedInput.secondComponent)
.ThenBy(calculatedInput => calculatedInput.thirdComponent)
.Select(calculatedInput => calculatedInput.input)
.ToList();
Console.WriteLine(test);
}
You can use the Array.Sort() method. Where your first parameter is the string[] you're sorting and the second parameter contains the complicated logic of determining the order.
You can use the IEnumerable.OrderBy method provided by the System.Linq namespace.
Ok,
We have a lot of where clauses in our code. We have just as many ways to generate a string to represent the in condition. I am trying to come up with a clean way as follows:
public static string Join<T>(this IEnumerable<T> items, string separator)
{
var strings = from item in items select item.ToString();
return string.Join(separator, strings.ToArray());
}
it can be used as follows:
var values = new []{1, 2, 3, 4, 5, 6};
values.StringJoin(",");
// result should be:
// "1,2,3,4,5,6"
So this is a nice extension method that does a very basic job. I know that simple code does not always turn into fast or efficient execution, but I am just curious as to what could I have missed with this simple code. Other members of our team are arguing that:
it is not flexible enough (no control of the string representation)
may not be memory efficient
may not be fast
Any expert to chime in?
Regards,
Eric.
Regarding the first issue, you could add another 'formatter' parameter to control the conversion of each item into a string:
public static string Join<T>(this IEnumerable<T> items, string separator)
{
return items.Join(separator, i => i.ToString());
}
public static string Join<T>(this IEnumerable<T> items, string separator, Func<T, string> formatter)
{
return String.Join(separator, items.Select(i => formatter(i)).ToArray());
}
Regarding the second two issues, I wouldn't worry about it unless you later run into performance issues and find it to be a problem. It's unlikely to much of a bottleneck however...
For some reason, I thought that String.Join is implemented in terms of a StringBuilder class. But if it isn't, then the following is likely to perform better for large inputs since it doesn't recreate a String object for each join in the iteration.
public static string Join<T>(this IEnumerable<T> items, string separator)
{
// TODO: check for null arguments.
StringBuilder builder = new StringBuilder();
foreach(T t in items)
{
builder.Append(t.ToString()).Append(separator);
}
builder.Length -= separator.Length;
return builder.ToString();
}
EDIT: Here is an analysis of when it is appropriate to use StringBuilder and String.Join.
Why don't you use StringBuilder, and iterate through the collection yourself, appending.
Otherwise you are creating an array of strings (var strings) and then doing the Join.
You are missing null checks for the sequence and the items of the sequence. And yes, it is not the fastest and most memory efficient way. One would probably just enumerate the sequence and render the string representations of the items into a StringBuilder. But does this really matter? Are you experiencing performance problems? Do you need to optimize?
this would work also:
public static string Test(IEnumerable<T> items, string separator)
{
var builder = new StringBuilder();
bool appendSeperator = false;
if(null != items)
{
foreach(var item in items)
{
if(appendSeperator)
{
builder.Append(separator)
}
builder.Append(item.ToString());
appendSeperator = true;
}
}
return builder.ToString();
}
I have a list that needs ordering say:
R1-1
R1-11
R2-2
R1-2
this needs to be ordered:
R1-1
R1-2
R1-11
R2-2
Currently I am using the C# Regex.Replace method and adding a 0 before the occurance of single numbers at the end of a string with something like:
Regex.Replace(inString,#"([1-9]$)", #"0$2")
I'm sure there is a nicer way to do this which I just can't figure out.
Does anyone have a nice way of sorting letter and number strings with regex?
I have used Greg's method below to complete this and just thought I should add the code I am using for completeness:
public static List<Rack> GetRacks(Guid aisleGUID)
{
log.Debug("Getting Racks with aisleId " + aisleGUID);
List<Rack> result = dataContext.Racks.Where(
r => r.aisleGUID == aisleGUID).ToList();
return result.OrderBy(r => r.rackName, new NaturalStringComparer()).ToList();
}
I think what you're after is natural sort order, like Windows Explorer does? If so then I wrote a blog entry a while back showing how you can achieve this in a few lines of C#.
Note: I just checked and using the NaturalStringComparer in the linked entry does return the order you are looking for with the example strings.
You can write your own comparator and use regular expressions to compare the number between "R" and "-" first, followed by the number after "-", if the first numbers are equal.
Sketch:
public int Compare(string x, string y)
{
int releaseX = ...;
int releaseY = ...;
int revisionX = ...;
int revisionY = ...;
if (releaseX == releaseY)
{
return revisionX - revisionY;
}
else
{
return releaseX - releaseY;
}
}