Develop Reference

Fastest way to split a large list of integers into List of strings in C#

My goal is to split a list of 24043 integers into strings like:
"?ids=" + "1,2,3...198,199,200"
Can you think of a better solution than mine in terms of performance?
public List<string> ZwrocListeZapytan(List<int> listaId)
{
var listaZapytan = new List<string>();
if (listaId.Count == 0) return listaZapytan;
var zapytanie = "?ids=";
var licznik = 1;
for (var i = 0; i < listaId.Count; i++)
{
if (licznik == 200 || i == listaId.Count - 1)
{
listaZapytan.Add(zapytanie + listaId[i]);
zapytanie = "?ids=";
licznik = 1;
}
else
{
zapytanie += listaId[i] + ",";
licznik++;
}
}
return listaZapytan;
}

Using Linq:
Assuming listaId is the list of integers that has to be converted:
var result = listaId.GroupBy(x => x / 200)
.Select(y => "?ids=" + string.Join(",", y)).ToList();
.GroupBy() helps take 200 at a time
.Select() is used to combine them together in the format like the OP suggested i.e ?ids=1,2,... using string.Join()

Can you think of a better solution than mine in terms of performance?
It terms of performance the only thing that comes to my mind as an enhancement for your code is to use a StringBuilder when you concatenate the string:
public List<string> ZwrocListeZapytan(List<int> listaId)
{
var listaZapytan = new List<string>();
if (listaId.Count == 0) return listaZapytan;
StringBuilder sb = new StringBuilder();
sb.Append("?ids=");
var licznik = 1;
for (var i = 0; i < listaId.Count; i++)
{
if (licznik == 200 || i == listaId.Count - 1)
{
listaZapytan.Add(sb.ToString() +listaId[i]);
sb.Clear();
sb.Append("?ids=");
licznik = 1;
}
else
{
sb.Append(listaId[i] + ",");
licznik++;
} return listaZapytan;
}
Otherwise you could make the for-loop run in steps of the 200. At each step take the numbers from the given range and use String.Join to create the string:
// TEST DATA
List<int> listaId = Enumerable.Range(1, 420).ToList();
List<string> listaZapytan = new List<string>();
int stepsize = 200;
for (int i = 0; i < listaId.Count; i +=stepsize)
{
listaZapytan.Add("?ids=" + String.Join(",", listaId.Skip(i).Take(stepsize)));
}

Could you please make a try with this and let me know whether this approach helps to solve your issue?
List<int> listaId = Enumerable.Range(0, 24043).ToList();
var items = String.Join("", Enumerable.Range(0, 24043)
.Select((x,i)=>i%200==0?
"\n?ids=" + x.ToString():
"," + x.ToString()));
Running Example
Here we are using Enumerable.Range to generate 24043 continuous numbers starting from 0. Then we can use the Select method to split them into a list of 200 and form the required string. If you want to get the output as a List, Remove the String.Join and add .ToList() at the end of the query. Current query produces output with 0-199 in the first list if you want 200 in that list means change the condition to i%201.

Remove from String Array what is in List

I have a String Array x and a List y and I want to remove all data from Y from the List X, how to do that in the fastest way?
e.g.:
X:
1) "aaa.bbb.ccc"
2) "ddd.eee.fff"
3) "ggg.hhh.jjj"
Y:
1) "bbb"
2) "fff"
Result should be a new List in Which only 3) exist because X.1 gets deleted by Y.1 and X.2 gets deleted by Y.2
How to do that?
I know I could do a foreach on the List X and check with everything in List Y, bit is that the fastest way?

The most convenient would be
var Z = X.Where(x => !x.Split('.').Intersect(Y).Any()).ToList();
That is not the same as "fastest". Probably the fastest (runtime) way to do that is to use a token search, like:
public static bool ContainsToken(string value, string token, char delimiter = '.')
{
if (string.IsNullOrEmpty(token)) return false;
if (string.IsNullOrEmpty(value)) return false;
int lastIndex = -1, idx, endIndex = value.Length - token.Length, tokenLength = token.Length;
while ((idx = value.IndexOf(token, lastIndex + 1)) > lastIndex)
{
lastIndex = idx;
if ((idx == 0 || (value[idx - 1] == delimiter))
&& (idx == endIndex || (value[idx + tokenLength] == delimiter)))
{
return true;
}
}
return false;
}
then something like:
var list = new List<string>(X.Length);
foreach(var x in X)
{
bool found = false;
foreach(var y in Y)
{
if(ContainsToken(x, y, '.'))
{
found = true;
break;
}
}
if (!found) list.Add(x);
}
This:
doesn't allocate arrays (for the output of Split, of for the params char[] of Split)
doesn't create any new string instances (for the output of Split)
doesn't use delegate abstraction
doesn't have captured scopes
uses the struct custom iterator of List<T> rather than the class iterator of IEnumerable<T>
starts the new List<T> with the appropriate worst-case size to avoid reallocations

Iterating over X and Y would indeed be the fastest option because you have this Contains constraint. I really don't see any other way.
It should not be a foreach over X though, because you cannot modify the collection you iterate over with foreach.
So an option would be:
for (int counterX = 0; counterX < X.Length; counterX++)
{
for(int counterY = 0; counterY < Y.Length; counterY++)
{
if (X[counterX].Contains(Y[counterY]))
{
X.RemoveAt(counterX--);
counterY = Y.Length;
}
}
}
This should do it (mind you, this code is not tested).

I think that a fairly fast approach would be to use List's built-in RemoveAll() method:
List<string> x = new List<string>
{
"aaa.bbb.ccc",
"ddd.eee.fff",
"ggg.hhh.jjj"
};
List<string> y = new List<string>
{
"bbb",
"fff"
};
x.RemoveAll(s => y.Any(s.Contains));
(Note that I am assuming that you have two lists, x and y. Your OP mentions a string array but then goes on to talk about "List X" and "List Y", so I'm ignoring the string array bit.)

Try this, using Aggregate function
var xArr = new string[] { "aaa.bbb.ccc", "ddd.eee.fff", "ggg.hhh.jjj" };
var yList = new List<string> { "bbb", "fff" };
var result = xArr.Aggregate(new List<string> { }, (acc, next) =>
{
var elems = next.Split('.');
foreach (var y in yList)
if (elems.Contains(y))
return acc;
acc.Add(next);
return acc;
});

If you've got a relatively small list the performance ramifications wouldn't really be a big deal. This is the simplest foreach solution I could come up with.
List<string> ListZ = ListX.ToList();
foreach (string x in ListX)
{
foreach (string y in ListY)
{
if (x.Contains(y))
ListZ.Remove(x);
}
}

How to select 2 items at a time?

I was writing a PascalCaseParser using Regex.Split and I came to the desire to select two items from an collection at a time.
This example code demonstrates.
void Main()
{
string pascalCasedString = "JustLikeYouAndMe";
var words = WordsFromPascalCasedString(pascalCasedString);
words.Dump();
}
IEnumerable<string> WordsFromPascalCasedString(string pascalCasedString)
{
var rx = new Regex("([A-Z])");
return rx.Split(pascalCasedString)
.Where(c => !string.IsNullOrEmpty(c))
// how to select 2 elements at a time?
;
}
The result of above code is:
IEnumerable<String> (10 items)
J
ust
L
ike
Y
ou
A
nd
M
e
Every two elements of the collection make one result that I want the function WordsFromPascalCasedString to yield.
My questions is: How would you, in general, deal with a requirement to return two items at a time. I'm curious if there are any interesting non-brute-force approaches.

Personally, I'd go with Simon Belanger's answer in this particular case. But in general, to select consecutive pairs, from an IEnumerable, you'd use this:
IEnumerable<Tuple<string, string>> WordsFromPascalCasedString(string pascalCasedString)
{
var rx = new Regex("([A-Z])");
var array = rx.Split(pascalCasedString)
.Where(c => !string.IsNullOrEmpty(c))
.ToArray();
var items = Enumerable.Range(0, array.Length / 2)
.Select(i => Tuple.Create(array[i * 2], array[i * 2 + 1]);
}
Or this, which takes more effort, but it's reusable and more efficient:
IEnumerable<Tuple<T, T>> Pairs<T>(IEnumerable<T> input)
{
var array = new T[2];
int i = 0;
foreach(var x in input)
{
array[i] = x;
i = (i + 1) % 2;
if (i == 0)
{
yield return Tuple.Create(array[0], array[1]);
}
}
}
IEnumerable<Tuple<string, string>> WordsFromPascalCasedString(string pascalCasedString)
{
var rx = new Regex("([A-Z])");
var output = rx.Split(pascalCasedString)
.Where(c => !string.IsNullOrEmpty(c));
var items = Pairs(output);
}
It can easily be extended to groups of n:
IEnumerable<IEnumerable<T>> Batches<T>(IEnumerable<T> input, int n)
{
var array = new T[n];
int i = 0;
foreach(var x in input)
{
array[i] = x;
i = (i + 1) % n;
if (i == 0)
{
yield return array.ToArray();
}
}
if (i != 0)
{
yield return array.Take(i);
}
}
A similar method exists in MoreLINQ.

The regex should be ([A-Z][a-z]*). Adjust the last portion if you want to include numbers too. Use + instead of * if you want at least one lowercase element after the uppercase delimiter.
Edit As for the actual question, you will need to materialize and iterate in a for loop for better performance (passing the list once). In your specific problem, you can just use Regex.Matches
var result = Regex.Matches("([A-Z][a-z]*)([A-Z][a-z]*)?", "AbCdEfGhIj")
.OfType<Match>()
.Where(m => m.Success)
.Select(m => Tuple.Create(m.Groups[1].Value, m.Groups[2].Value));

This is just to share, I'm throwing the solution I came up with after getting inspired by the other answers. It is not better than the others...
void Main()
{
string pascalCasedString = "JustLikeYouAndMe";
var words = WordsFromPascalCasedString(pascalCasedString);
words.Dump();
}
IEnumerable<string> WordsFromPascalCasedString(string pascalCasedString)
{
var rx = new Regex("([A-Z])");
return rx.Split(pascalCasedString)
.Where(c => !string.IsNullOrEmpty(c))
.InPieces(2)
.Select(c => c.ElementAt(0) + c.ElementAt(1));
}
static class Ext
{
public static IEnumerable<IEnumerable<T>> InPieces<T>(this IEnumerable<T> seq, int len)
{
if(!seq.Any())
yield break;
yield return seq.Take(len);
foreach (var element in InPieces(seq.Skip(len), len))
yield return element;
}
}

Easiest is to write function that simply returns pairs.
Something like:
IEnumerable<Tuple<T,T>> Pairs<T>(IEnumerable<T> items)
{
T first = default(T);
bool hasFirst = false;
foreach(T item in items)
{
if (hasFirst)
yield return Tuple.Create(first, item);
else
first = item;
hasFirst = !hasFirst;
}
}
Aggregate is likely only one one-line approach. This is purely entertainment code due to amount of garbage created on a way, but there is no mutable objects used.
IEnumerable<Tuple<T,T>> Pairs<T>(IEnumerable<T> collection)
{
return collection
.Aggregate(
Tuple.Create(false, default(T), Enumerable.Empty<Tuple<T,T>>()),
(accumulate, item)=> !accumulate.Item1 ?
Tuple.Create(true, item, accumulate.Item3) :
Tuple.Create(false, default(T),
accumulate.Item3.Concat(
Enumerable.Repeat(Tuple.Create(accumulate.Item2, item), 1))),
accumulate => accumulate.Item3);
}
Zip of odd and even elements (index %2 ==/!= 0) is 2 line approach. Note that iterates source collection twice.
IEnumerable<Tuple<T,T>> Pairs<T>(IEnumerable<T> collection)
{
return collection
.Where((item, index)=>index %2 == 0)
.Zip(collection.Where((item, index)=>index %2 != 0),
(first,second)=> Tuple.Create(first,second));
}

Split and join multiple logical "branches" of string data

I know there's a couple similarly worded questions on SO about permutation listing, but they don't seem to be quite addressing really what I'm looking for. I know there's a way to do this but I'm drawing a blank. I have a flat file that resembles this format:
Col1|Col2|Col3|Col4|Col5|Col6
a|b,c,d|e|f|g,h|i
. . .
Now here's the trick: I want to create a list of all possible permutations of these rows, where a comma-separated list in the row represents possible values. For example, I should be able to take an IEnumerable<string> representing the above to rows as such:
IEnumerable<string> row = new string[] { "a", "b,c,d", "e", "f", "g,h", "i" };
IEnumerable<string> permutations = GetPermutations(row, delimiter: "/");
This should generate the following collection of string data:
a/b/e/f/g/i
a/b/e/f/h/i
a/c/e/f/g/i
a/c/e/f/h/i
a/d/e/f/g/i
a/d/e/f/h/i
This to me seems like it would elegantly fit into a recursive method, but apparently I have a bad case of the Mondays and I can't quite wrap my brain around how to approach it. Some help would be greatly appreciated. What should GetPermutations(IEnumerable<string>, string) look like?

You had me at "recursive". Here's another suggestion:
private IEnumerable<string> GetPermutations(string[] row, string delimiter,
int colIndex = 0, string[] currentPerm = null)
{
//First-time initialization:
if (currentPerm == null) { currentPerm = new string[row.Length]; }
var values = row[colIndex].Split(',');
foreach (var val in values)
{
//Update the current permutation with this column's next possible value..
currentPerm[colIndex] = val;
//..and find values for the remaining columns..
if (colIndex < (row.Length - 1))
{
foreach (var perm in GetPermutations(row, delimiter, colIndex + 1, currentPerm))
{
yield return perm;
}
}
//..unless we've reached the last column, in which case we create a complete string:
else
{
yield return string.Join(delimiter, currentPerm);
}
}
}

I'm not sure whether this is the most elegant approach, but it might get you started.
private static IEnumerable<string> GetPermutations(IEnumerable<string> row,
string delimiter = "|")
{
var separator = new[] { ',' };
var permutations = new List<string>();
foreach (var cell in row)
{
var parts = cell.Split(separator);
var perms = permutations.ToArray();
permutations.Clear();
foreach (var part in parts)
{
if (perms.Length == 0)
{
permutations.Add(part);
continue;
}
foreach (var perm in perms)
{
permutations.Add(string.Concat(perm, delimiter, part));
}
}
}
return permutations;
}
Of course, if the order of the permutations is important, you can add an .OrderBy() at the end.
Edit: added an alernative
You could also build a list of string arrays, by calculating some numbers before determining the permutations.
private static IEnumerable<string> GetPermutations(IEnumerable<string> row,
string delimiter = "|")
{
var permutationGroups = row.Select(o => o.Split(new[] { ',' })).ToArray();
var numberOfGroups = permutationGroups.Length;
var numberOfPermutations =
permutationGroups.Aggregate(1, (current, pg) => current * pg.Length);
var permutations = new List<string[]>(numberOfPermutations);
for (var n = 0; n < numberOfPermutations; n++)
{
permutations.Add(new string[numberOfGroups]);
}
for (var position = 0; position < numberOfGroups; position++)
{
var permutationGroup = permutationGroups[position];
var numberOfCharacters = permutationGroup.Length;
var numberOfIterations = numberOfPermutations / numberOfCharacters;
for (var c = 0; c < numberOfCharacters; c++)
{
var character = permutationGroup[c];
for (var i = 0; i < numberOfIterations; i++)
{
var index = c + (i * numberOfCharacters);
permutations[index][position] = character;
}
}
}
return permutations.Select(p => string.Join(delimiter, p));
}

One algorithm you can use is basically like counting:
Start with the 0th item in each list (00000)
Increment the last value (00001, 00002 etc.)
When you can't increas one value, reset it and increment the next (00009, 00010, 00011 etc.)
When you can't increase any value, you're done.
Function:
static IEnumerable<string> Permutations(
string input,
char separator1, char separator2,
string delimiter)
{
var enumerators = input.Split(separator1)
.Select(s => s.Split(separator2).GetEnumerator()).ToArray();
if (!enumerators.All(e => e.MoveNext())) yield break;
while (true)
{
yield return String.Join(delimiter, enumerators.Select(e => e.Current));
if (enumerators.Reverse().All(e => {
bool finished = !e.MoveNext();
if (finished)
{
e.Reset();
e.MoveNext();
}
return finished;
}))
yield break;
}
}
Usage:
foreach (var perm in Permutations("a|b,c,d|e|f|g,h|i", '|', ',', "/"))
{
Console.WriteLine(perm);
}

I really thought this would be a great recursive function, but I ended up not writing it that way. Ultimately, this is the code I created:
public IEnumerable<string> GetPermutations(IEnumerable<string> possibleCombos, string delimiter)
{
var permutations = new Dictionary<int, List<string>>();
var comboArray = possibleCombos.ToArray();
var splitCharArr = new char[] { ',' };
permutations[0] = new List<string>();
permutations[0].AddRange(
possibleCombos
.First()
.Split(splitCharArr)
.Where(x => !string.IsNullOrEmpty(x.Trim()))
.Select(x => x.Trim()));
for (int i = 1; i < comboArray.Length; i++)
{
permutations[i] = new List<string>();
foreach (var permutation in permutations[i - 1])
{
permutations[i].AddRange(
comboArray[i].Split(splitCharArr)
.Where(x => !string.IsNullOrEmpty(x.Trim()))
.Select(x => string.Format("{0}{1}{2}", permutation, delimiter, x.Trim()))
);
}
}
return permutations[permutations.Keys.Max()];
}
... my test conditions provided me with exactly the output I expected:
IEnumerable<string> row = new string[] { "a", "b,c,d", "e", "f", "g,h", "i" };
IEnumerable<string> permutations = GetPermutations(row, delimiter: "/");
foreach(var permutation in permutations)
{
Debug.Print(permutation);
}
This produced the following output:
a/b/e/f/g/i
a/b/e/f/h/i
a/c/e/f/g/i
a/c/e/f/h/i
a/d/e/f/g/i
a/d/e/f/h/i
Thanks to everyone's suggestions, they really were helpful in sorting out what needed to be done in my mind. I've upvoted all your answers.

Using LINQ to select item in List<T> and add integer to populate new List<T>

Lets say I have List<string> = new List<string>() {"20","26","32"}
I want to create a new List based on the first number in the previous list and it should have the same number of elements in it. I will be adding a certain number to that first number and so on and so on. As an example, using 6 as the number to add I would get 20,26,32. The resulting list will be List. The number 6 is a class wide property.
The issue comes if I have a list of "N","N","32"
I need to produce the same list of 20,26,32 but I have to use the last number to work out the others.
If I had "N","26","N" I would have to use the middle number to work out the others.
The N represents no data in the input list and it will always be this character
In summary, I need to produce a new list with the same number of elements as the input list and it must take the first or next numerical element to produce the resulting list using a specified number to add/subtract values to.
I wondered if LINQ's aggregate function might be able to handle it but got a bit lost using it.
Examples:
"20","26","32" = 20,26,32
"N","26","32" = 20,26,32
"N","N","32" = 20,26,32
"20","26","N" = 20,26,32

What about something like this:
var n = 6;
List<string> strList = new List<string>() {"20","26","32"};
// list can also be {null, "26", null} , {null, "N", "32"} ,
// {"N", "26", null } etc...
var list = strList.Select(s =>
{
int v;
if(string.IsNullOrEmpty(s) || !int.TryParse(s,out v))
return (int?)null;
return v;
});
var firstValidVal = list.Select((Num, Index) => new { Num, Index })
.FirstOrDefault(x => x.Num.HasValue);
if(firstValidVal == null)
throw new Exception("No valid number found");
var bases = Enumerable.Range(0, strList.Count).Select(i => i * n);
int startVal = firstValidVal.Num.Value - bases.ElementAt(firstValidVal.Index);
var completeSequence = bases.Select(x => x + startVal);

It sounds like you want a function which will
Take a List<int> as input
Make the first element of the original list the first element of the new list
New list has same number of elements as original
Remaining numbers are the first element + a value * position
If so then try the following
static bool TryGetFirstNumber(List<string> list, out number, out index) {
for (var i = 0; i < list.Count; i++) {
var cur = list[0];
if (!String.IsNullOrEmpty(cur) && Int32.TryParse(cur, out number)) {
index = i;
return true;
}
}
number = 0;
index = 0;
return false;
}
static List<T> TheFunction(List<string> list, int increment) {
var newList = new List<int>();
int first;
int index;
if (TryGetFirstNumber(list, out first, out index)) {
first -= index * increment;
} else {
first = 0;
}
newList.Add(first);
for (var i = 1; i < list.Length; i++) {
newList.Add(first + increment);
increment += increment;
}
return newList;
}

For LINQ purposes, I sometimes resort to writing a parse method that returns an int?as the result so that I can return null when it fails to parse. Here's a complete LINQPad implementation that illustrates this and the positional select (taking an approach otherwise similar to digEmAll's):
void Main()
{
var n = 6;
var items = new List<string>
// {"20","N", "N"};
// {"N", "26", "N"};
{"N", "N", "32"};
var first = items
.Select((v,index) => new { val = Parse(v), index })
.First(x => x.val.HasValue);
int start = first.val.Value - n * first.index;
List<string> values = items
.Select((x,i) => (i * n + start).ToString())
.ToList();
}
int? Parse(string strVal)
{
int ret;
if (int.TryParse(strVal, out ret))
{
return ret;
}
return null;
}

Seems like a lot of work to do something kinda simple. Here is a non linq approach.
private List<int> getVals(List<string> input, int modifier)
{
if (input == null) return null; if (input.Count < 1) return null;
foreach (var s in input)
{
int i;
try{i = Convert.ToInt32(s);}
catch{continue;}
var returnList = new List<int>(input.Count);
for (int n = 0; n < input.Count;n++ )returnList[n] = ((n - input.IndexOf(s)) * modifier) + i;
return returnList;
}
return null;
}

DevGeezer's answer, but without the cruft.
But I still learned alot!
static List<String> genlist2(List<String> list, int interval)
{
if (list == null) return null;
var vali = list
.Select((x, i) => x != "N" ? new {val = Convert.ToInt32(x), i } : null)
.First(x => x != null);
if (vali == null) return list.ToList();
return Enumerable.Range(0, list.Count)
.Select(x => (vali.val - (vali.i - x) * interval).ToString())
.ToList();
}