Develop Reference

Quick (sub)string search in large set of data

Given a city:
public class City
{
public int Id { get; set; }
public string Name { get; set; }
public string Country { get; set; }
public LatLong Location { get; set; }
}
I have a list of close to 3,000,000 cities (and towns and villages etc.) in a file. This file is read into memory; I have been playing with arrays, lists, dictionaries (key = Id) etc.
I want to find, as quick as possible, all cities matching a substring (case insensitive). So when I search for 'yor' I want to get all matches (1000+) ASAP (matching 'York Town', 'Villa Mayor', 'New York', ...).
Functionally you could write this as:
cities.Values.Where(c => c.Name.IndexOf("yor", StringComparison.OrdinalIgnoreCase) >= 0)
I don't mind doing some pre-processing when reading the file; as a matter of fact: that's what I'm mostly looking for. Read the file, "chew" on the data creating some sort of index or... and then be ready to answer queries like "yor".
I want this to be standalone, self-contained. I do not want to add dependencies like an RDBMS, ElasticSearch or whatever. I don't mind having (parts of) the list in memory more than once. I don't mind spending some memory on a datastructure to help me find my results quickly. I don't want libraries or packages. I want an algorithm I can implement myself.
Basically I want the above LINQ statement, but optimized for my case; currently plowing through almost 3,000,000 records takes about +/- 2 seconds. I want this sub 0.1 second so I could use the search and it's results as 'autocomplete'.
Creating an "index"(-alike) structure is probably what I need. As I'm writing I remember something about a "bloom filter" but I'm not sure if that would help or even supports substring search. Will look into that now.
Any tips, pointers, help very much appreciated.

I created a bit of a hybrid based on a suffix array / dictionary. Thanks to saibot for suggesting it first and all other people helping and suggesting.
This is what I came up with:
public class CitiesCollection
{
private Dictionary<int, City> _cities;
private SuffixDict<int> _suffixdict;
public CitiesCollection(IEnumerable<City> cities, int minLen)
{
_cities = cities.ToDictionary(c => c.Id);
_suffixdict = new SuffixDict<int>(minLen, _cities.Values.Count);
foreach (var c in _cities.Values)
_suffixdict.Add(c.Name, c.Id);
}
public IEnumerable<City> Find(string find)
{
var normalizedFind = _suffixdict.NormalizeString(find);
foreach (var id in _suffixdict.Get(normalizedFind).Where(v => _cities[v].Name.IndexOf(normalizedFind, StringComparison.OrdinalIgnoreCase) >= 0))
yield return _cities[id];
}
}
public class SuffixDict<T>
{
private readonly int _suffixsize;
private ConcurrentDictionary<string, IList<T>> _dict;
public SuffixDict(int suffixSize, int capacity)
{
_suffixsize = suffixSize;
_dict = new ConcurrentDictionary<string, IList<T>>(Environment.ProcessorCount, capacity);
}
public void Add(string suffix, T value)
{
foreach (var s in GetSuffixes(suffix))
AddDict(s, value);
}
public IEnumerable<T> Get(string suffix)
{
return Find(suffix).Distinct();
}
private IEnumerable<T> Find(string suffix)
{
foreach (var s in GetSuffixes(suffix))
{
if (_dict.TryGetValue(s, out var result))
foreach (var i in result)
yield return i;
}
}
public string NormalizeString(string value)
{
return value.Normalize().ToLowerInvariant();
}
private void AddDict(string suffix, T value)
{
_dict.AddOrUpdate(suffix, (s) => new List<T>() { value }, (k, v) => { v.Add(value); return v; });
}
private IEnumerable<string> GetSuffixes(string value)
{
var nv = NormalizeString(value);
for (var i = 0; i <= nv.Length - _suffixsize ; i++)
yield return nv.Substring(i, _suffixsize);
}
}
Usage (where I assume mycities to be an IEnumerable<City> with the given City object from the question):
var cc = new CitiesCollection(mycities, 3);
var results = cc.Find("york");
Some results:
Find: sterda elapsed: 00:00:00.0220522 results: 32
Find: york elapsed: 00:00:00.0006212 results: 155
Find: dorf elapsed: 00:00:00.0086439 results: 6095
Memory usage is very, very acceptable. Only 650MB total having the entire collection of 3,000,000 cities in memory.
In the above I'm storing Id's in the "SuffixDict" and I have a level of indirection (dictionary lookups to find id=>city). This can be further simplified to:
public class CitiesCollection
{
private SuffixDict<City> _suffixdict;
public CitiesCollection(IEnumerable<City> cities, int minLen, int capacity = 1000)
{
_suffixdict = new SuffixDict<City>(minLen, capacity);
foreach (var c in cities)
_suffixdict.Add(c.Name, c);
}
public IEnumerable<City> Find(string find, StringComparison stringComparison = StringComparison.OrdinalIgnoreCase)
{
var normalizedFind = SuffixDict<City>.NormalizeString(find);
var x = _suffixdict.Find(normalizedFind).ToArray();
foreach (var city in _suffixdict.Find(normalizedFind).Where(v => v.Name.IndexOf(normalizedFind, stringComparison) >= 0))
yield return city;
}
}
public class SuffixDict<T>
{
private readonly int _suffixsize;
private ConcurrentDictionary<string, IList<T>> _dict;
public SuffixDict(int suffixSize, int capacity = 1000)
{
_suffixsize = suffixSize;
_dict = new ConcurrentDictionary<string, IList<T>>(Environment.ProcessorCount, capacity);
}
public void Add(string suffix, T value)
{
foreach (var s in GetSuffixes(suffix, _suffixsize))
AddDict(s, value);
}
public IEnumerable<T> Find(string suffix)
{
var normalizedfind = NormalizeString(suffix);
var find = normalizedfind.Substring(0, Math.Min(normalizedfind.Length, _suffixsize));
if (_dict.TryGetValue(find, out var result))
foreach (var i in result)
yield return i;
}
private void AddDict(string suffix, T value)
{
_dict.AddOrUpdate(suffix, (s) => new List<T>() { value }, (k, v) => { v.Add(value); return v; });
}
public static string NormalizeString(string value)
{
return value.Normalize().ToLowerInvariant();
}
private static IEnumerable<string> GetSuffixes(string value, int suffixSize)
{
var nv = NormalizeString(value);
if (value.Length < suffixSize)
{
yield return nv;
}
else
{
for (var i = 0; i <= nv.Length - suffixSize; i++)
yield return nv.Substring(i, suffixSize);
}
}
}
This bumps the load time up from 00:00:16.3899085 to 00:00:25.6113214, memory usage goes down from 650MB to 486MB. Lookups/searches perform a bit better since we have one less level of indirection.
Find: sterda elapsed: 00:00:00.0168616 results: 32
Find: york elapsed: 00:00:00.0003945 results: 155
Find: dorf elapsed: 00:00:00.0062015 results: 6095
I'm happy with the results so far. I'll do a little polishing and refactoring and call it a day! Thanks everybody for the help!
And this is how it performs with 2,972,036 cities:
This has evolved into a case-insensitive, accent-insensitive search by modifying the code to this:
public static class ExtensionMethods
{
public static T FirstOrDefault<T>(this IEnumerable<T> src, Func<T, bool> testFn, T defval)
{
return src.Where(aT => testFn(aT)).DefaultIfEmpty(defval).First();
}
public static int IndexOf(this string source, string match, IEqualityComparer<string> sc)
{
return Enumerable.Range(0, source.Length) // for each position in the string
.FirstOrDefault(i => // find the first position where either
// match is Equals at this position for length of match (or to end of string) or
sc.Equals(source.Substring(i, Math.Min(match.Length, source.Length - i)), match) ||
// match is Equals to on of the substrings beginning at this position
Enumerable.Range(1, source.Length - i - 1).Any(ml => sc.Equals(source.Substring(i, ml), match)),
-1 // else return -1 if no position matches
);
}
}
public class CaseAccentInsensitiveEqualityComparer : IEqualityComparer<string>
{
private static readonly CompareOptions _compareoptions = CompareOptions.IgnoreCase | CompareOptions.IgnoreNonSpace | CompareOptions.IgnoreKanaType | CompareOptions.IgnoreWidth | CompareOptions.IgnoreSymbols;
private static readonly CultureInfo _cultureinfo = CultureInfo.InvariantCulture;
public bool Equals(string x, string y)
{
return string.Compare(x, y, _cultureinfo, _compareoptions) == 0;
}
public int GetHashCode(string obj)
{
return obj != null ? RemoveDiacritics(obj).ToUpperInvariant().GetHashCode() : 0;
}
private string RemoveDiacritics(string text)
{
return string.Concat(
text.Normalize(NormalizationForm.FormD)
.Where(ch => CharUnicodeInfo.GetUnicodeCategory(ch) != UnicodeCategory.NonSpacingMark)
).Normalize(NormalizationForm.FormC);
}
}
public class CitiesCollection
{
private SuffixDict<City> _suffixdict;
private HashSet<string> _countries;
private Dictionary<int, City> _cities;
private readonly IEqualityComparer<string> _comparer = new CaseAccentInsensitiveEqualityComparer();
public CitiesCollection(IEnumerable<City> cities, int minLen, int capacity = 1000)
{
_suffixdict = new SuffixDict<City>(minLen, _comparer, capacity);
_countries = new HashSet<string>();
_cities = new Dictionary<int, City>(capacity);
foreach (var c in cities)
{
_suffixdict.Add(c.Name, c);
_countries.Add(c.Country);
_cities.Add(c.Id, c);
}
}
public City this[int index] => _cities[index];
public IEnumerable<string> Countries => _countries;
public IEnumerable<City> Find(string find, StringComparison stringComparison = StringComparison.OrdinalIgnoreCase)
{
foreach (var city in _suffixdict.Find(find).Where(v => v.Name.IndexOf(find, _comparer) >= 0))
yield return city;
}
}
public class SuffixDict<T>
{
private readonly int _suffixsize;
private ConcurrentDictionary<string, IList<T>> _dict;
public SuffixDict(int suffixSize, IEqualityComparer<string> stringComparer, int capacity = 1000)
{
_suffixsize = suffixSize;
_dict = new ConcurrentDictionary<string, IList<T>>(Environment.ProcessorCount, capacity, stringComparer);
}
public void Add(string suffix, T value)
{
foreach (var s in GetSuffixes(suffix, _suffixsize))
AddDict(s, value);
}
public IEnumerable<T> Find(string suffix)
{
var find = suffix.Substring(0, Math.Min(suffix.Length, _suffixsize));
if (_dict.TryGetValue(find, out var result))
{
foreach (var i in result)
yield return i;
}
}
private void AddDict(string suffix, T value)
{
_dict.AddOrUpdate(suffix, (s) => new List<T>() { value }, (k, v) => { v.Add(value); return v; });
}
private static IEnumerable<string> GetSuffixes(string value, int suffixSize)
{
if (value.Length < 2)
{
yield return value;
}
else
{
for (var i = 0; i <= value.Length - suffixSize; i++)
yield return value.Substring(i, suffixSize);
}
}
}
With credit also to Netmage and Mitsugui. There are still some issues / edge-cases but it's continually improving!

You could use a suffix tree: https://en.wikipedia.org/wiki/Suffix_tree
It requires enough space to store about 20 times your list of words in memory
Suffix array is a space efficient alternative: https://en.wikipedia.org/wiki/Suffix_array

in query benchmark contains very faster then indexOf >0
cities.Values.Where(c => c.Name.Contans("yor"))

How to build a sequence using a fluent interface?

I'm trying to using a fluent interface to build a collection, similar to this (simplified) example:
var a = StartWith(1).Add(2).Add(3).Add(4).ToArray();
/* a = int[] {1,2,3,4}; */
The best solution I can come up with add Add() as:
IEnumerable<T> Add<T>(this IEnumerable<T> coll, T item)
{
foreach(var t in coll) yield return t;
yield return item;
}
Which seems to add a lot of overhead that going to be repeated in each call.
IS there a better way?
UPDATE:
in my rush, I over-simplified the example, and left out an important requirement. The last item in the existing coll influences the next item. So, a slightly less simplified example:
var a = StartWith(1).Times10Plus(2).Times10Plus(3).Times10Plus(4).ToArray();
/* a = int[] {1,12,123,1234}; */
public static IEnumerable<T> StartWith<T>(T x)
{
yield return x;
}
static public IEnumerable<int> Times10Plus(this IEnumerable<int> coll, int item)
{
int last = 0;
foreach (var t in coll)
{
last = t;
yield return t;
}
yield return last * 10 + item;
}

A bit late to this party, but here are a couple ideas.
First, consider solving the more general problem:
public static IEnumerable<A> AggregateSequence<S, A>(
this IEnumerable<S> items,
A initial,
Func<A, R, A> f)
{
A accumulator = initial;
yield return accumulator;
foreach(S item in items)
{
accumulator = f(accumulator, item);
yield return accumulator;
}
}
And now your program is just new[]{2, 3, 4}.AggregateSequence(1,
(a, s) => a * 10 + s).ToArray()
However that lacks the "fluency" you want and it assumes that the same operation is applied to every element in the sequence.
You are right to note that deeply nested iterator blocks are problematic; they have quadratic performance in time and linear consumption of stack, both of which are bad.
Here's an entertaining way to implement your solution efficiently.
The problem is that you need both cheap access to the "right" end of the sequence, in order to do an operation on the most recently added element, but you also need cheap access to the left end of the sequence to enumerate it. Normal queues and stacks only have cheap access to one end.
Therefore: start by implementing an efficient immutable double-ended queue. This is a fascinating datatype; I have an implementation here using finger trees:
https://blogs.msdn.microsoft.com/ericlippert/2008/01/22/immutability-in-c-part-10-a-double-ended-queue/
https://blogs.msdn.microsoft.com/ericlippert/2008/02/12/immutability-in-c-part-eleven-a-working-double-ended-queue/
Once you have that, your operations are one-liners:
static IDeque<T> StartWith<T>(T t) => Deque<T>.Empty.EnqueueRight(t);
static IDeque<T> Op<T>(this IDeque<T> d, Func<T, T> f) => d.EnqueueRight(f(d.PeekRight()));
static IDeque<int> Times10Plus(this IDeque<int> d, int j) => d.Op(i => i * 10 + j);
Modify IDeque<T> and Deque<T> to implement IEnumerable<T> in the obvious way and you then get ToArray for free. Or do it as an extension method:
static IEnumerable<T> EnumerateFromLeft(this IDeque<T> d)
{
var c = d;
while (!c.IsEmpty)
{
yield return c.PeekLeft();
c = c.DequeueLeft();
}
}

You could do the following:
public static class MySequenceExtensions
{
public static IReadOnlyList<int> Times10Plus(
this IReadOnlyList<int> sequence,
int value) => Add(sequence,
value,
v => sequence[sequence.Count - 1] * 10 + v);
public static IReadOnlyList<T> Starts<T>(this T first)
=> new MySequence<T>(first);
public static IReadOnlyList<T> Add<T>(
this IReadOnlyList<T> sequence,
T item,
Func<T, T> func)
{
var mySequence = sequence as MySequence<T> ??
new MySequence<T>(sequence);
return mySequence.AddItem(item, func);
}
private class MySequence<T>: IReadOnlyList<T>
{
private readonly List<T> innerList;
public MySequence(T item)
{
innerList = new List<T>();
innerList.Add(item);
}
public MySequence(IEnumerable<T> items)
{
innerList = new List<T>(items);
}
public T this[int index] => innerList[index];
public int Count => innerList.Count;
public MySequence<T> AddItem(T item, Func<T, T> func)
{
Debug.Assert(innerList.Count > 0);
innerList.Add(func(item));
return this;
}
public IEnumerator<T> GetEnumerator() => innerList.GetEnumerator();
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}
}
Note that I'm using IReadOnlyList to make it possible to index into the list in a performant way and be able to get the last element if needed. If you need a lazy enumeration then I think you are stuck with your original idea.
And sure enough, the following:
var a = 1.Starts().Times10Plus(2).Times10Plus(3).Times10Plus(4).ToArray();
Produces the expected result ({1, 12, 123, 1234}) with, what I think is, reasonable performance.

You can do like this:
public interface ISequence
{
ISequenceOp StartWith(int i);
}
public interface ISequenceOp
{
ISequenceOp Times10Plus(int i);
int[] ToArray();
}
public class Sequence : ISequence
{
public ISequenceOp StartWith(int i)
{
return new SequenceOp(i);
}
}
public class SequenceOp : ISequenceOp
{
public List<int> Sequence { get; set; }
public SequenceOp(int startValue)
{
Sequence = new List<int> { startValue };
}
public ISequenceOp Times10Plus(int i)
{
Sequence.Add(Sequence.Last() * 10 + i);
return this;
}
public int[] ToArray()
{
return Sequence.ToArray();
}
}
An then just:
var x = new Sequence();
var a = x.StartWith(1).Times10Plus(2).Times10Plus(3).Times10Plus(4).ToArray();

Can not deal with IEnumerable<int?> in generic

I have a method to concatenate strings provided by int?.
public string ConcatenateNumber(IEnumerable<int?> myList)
{
return myList
.Distinct()
.Aggregate(
new StringBuilder(),
(current, next) => current.Append("'").Append(next))
.ToString()
.Substring(1);
}
Now I want to do unit test.
[TestMethod]
public void InputIntegers_Should_Be_Concatenated_When_Consider_Distinct()
{
var myList = CreateEnumerable(1, 2, 2, 3);
var logic = new MainLogic();
var result = logic.ConcatenateNumber(myList);
Assert.AreEqual("1'2'3", result);
}
public IEnumerable<T> CreateEnumerable<T>(params T[] items)
{
if (items == null)
yield break;
foreach (T mitem in items)
yield return mitem;
}
However I have the compile error.
C#: Unknown method ConcatenateNumber(System.Collections.Generic.IEnumerable) of ....
I think it is caused by nullable integer int?. But I am not sure how to fix it.

Explicitly pass the type as a nullable int.
var myList = CreateEnumerable<int?>(1, 2, 2, 3);
For example:
using System;
using System.Linq;
using System.Collections.Generic;
public class Program
{
public static void Main()
{
var p = new Program();
var list = p.CreateEnumerable<int?>(1, 2, 3, 4);
p.DoWork(list);
}
public void DoWork(IEnumerable<int?> enumerable)
{
enumerable.ToList().ForEach(x =>
{
Console.WriteLine(x);
});
}
public IEnumerable<T> CreateEnumerable<T>(params T[] items)
{
if (items == null)
yield break;
foreach (T mitem in items)
yield return mitem;
}
}

public void InputIntegers_Should_Be_Concatenated_When_Consider_Distinct()
{
var myList = CreateEnumerable(1, 2, 2, 3);
var logic = new MainLogic();
var result = logic.ConcatenateNumber(myList);
}
public IEnumerable<T> CreateEnumerable<T>(params T[] items)
{
return items ?? Enumerable.Empty<T>();
}
public class MainLogic
{
public string ConcatenateNumber<T>(IEnumerable<T> myList)
{
// do this if you want to remove nulls
return string.Join("'", myList.Where(i => i != null).Distinct());
//return string.Join("'", myList.Distinct()); // otherwise do this
}
}

C#, Updating a List using a delegate function as a filter

I want to update a list of objects(mycapsule, has many other members) when a condition (pred) holds for one of its class members(value). Whenever I change something another thing fails, I'm newbie in C# and really confused.
Can somebody fix my code:
In the best condition I only get get this error but I think many things are missing in my code
The type or namespace name `T' could not be found. Are you missing a using directive or an assembly reference?
here is my code
using System;
using System.Collections.Generic ;
namespace test
{
class MainClass
{
public static List< mycapsule<int,double> > sample= new List< mycapsule<int,double>>();
public static void Main (string[] args)
{
sample.Add(new mycapsule<int,double> {id=1 , value= 1.2});
update(pred, 12.3);
}
public static bool pred(double x)
{
if (x==2.5) return true;
return false;
}
public class mycapsule<KT, T>
{
public KT id {get ; set; }
public T value { get ; set; }
public int p; // and more
}
public bool update(Func<T, bool> predicate, T i)
{
foreach (var x in sample.FindAll(item => predicate(JustValue(item))) )
{
x.value = i ;
}
return true ;
}
public T JustValue(mycapsule<int,T> i)
{
return i.value;
}
}
}

Look at your update method:
public bool update(Func<T, bool> predicate, T i)
{
foreach (var x in KeyRecord.FindAll(item => predicate(JustValue(item))) )
{
x.value = i ;
}
return true ;
}
What do you expect T to be here? The method is not generic (it's not written as update<T>) and it's not declared in a generic class.
It's possible that you just want:
public bool update<T>(Func<T, bool> predicate, T i)
... but it's hard to say without knowing what KeyRecord.FindAll looks like. Oh, and you've got the same problem with JustValue.
As a side issue, the method names update and pred don't follow .NET naming conventions, and JustValue is a poor method name in terms of descriptiveness. mycapsule also doesn't follow .NET naming conventions. These things really matter in terms of readability.

I'm afraid this won't work due to type safety reasons. I corrected the code as much as possible and got this:
public static List<mycapsule<int, double>> sample = new List<mycapsule<int, double>>();
public static void Main(string[] args)
{
sample.Add(new mycapsule<int, double> { id = 1, value = 1.2 });
update(pred, 12.3);
}
public static bool pred(double x)
{
if (x == 2.5) return true;
return false;
}
public class mycapsule<KT, T>
{
public KT id { get; set; }
public T value { get; set; }
public int p; // and more
}
public static bool update<T>(Func<T, bool> predicate, T i)
{
List<mycapsule<int, double>> list = sample.FindAll(item => predicate(JustValue(item)));
foreach (var x in list)
{
x.value = i;
}
return true;
}
public static T JustValue<T>(mycapsule<int, T> i)
{
return i.value;
}
The error is:
predicate(JustValue(item)) => Argument 1: cannot implicitly convert from double to T
This comes from the fact that you are trying to forcefully call a method that you've specified as taking a generic type T (Func<T, bool>) with a value that is known to be double. Although we know that T will be double from the call to update(pred, 12.3);, nothing prevents me from passing in a predicate that takes an incompatible type e.g.:
public static bool pred(string x)
{
return false;
}
and
update(pred, "asdf");
Which is clearly inconsistent. The compiler is simply trying to prevent you from shooting yourself in the foot accidentally.
To solve this, you could explicitly pass the collection to the update method, thus ensuring that the types are consistent:
public static List<mycapsule<int, double>> sample = new List<mycapsule<int, double>>();
public static void Main(string[] args)
{
sample.Add(new mycapsule<int, double> { id = 1, value = 1.2 });
update(pred, 12.5, sample);
}
public static bool pred(double x)
{
if (x == 2.5) return true;
return false;
}
public class mycapsule<KT, T>
{
public KT id { get; set; }
public T value { get; set; }
public int p; // and more
}
public static bool update<T>(Func<T, bool> predicate, T i, List<mycapsule<int, T>> list)
{
foreach (var x in list.FindAll(item => predicate(JustValue(item))))
{
x.value = i;
}
return true;
}
public static T JustValue<T>(mycapsule<int, T> i)
{
return i.value;
}

Try this, And for generic method the method definition you have used is not correct.
It should be
//MethodName<T,U>(T para1, U para2)
I have change the code to include reflections, this should work.
Please try and give feedback.
using System;
using System.Collections.Generic;
using System.Reflection;
namespace test
{
class MainClass
{
public static List<Mycapsule<int, double>> sample = new List<Mycapsule<int, double>>();
public static void Main(string[] args)
{
sample.Add(new Mycapsule<int, double> { id = 1, value = 1.2 });
update(pred, 12.3);
}
public static bool pred(double x)
{
if (x == 2.5) return true;
return false;
}
public static bool update<T>(Func<T, bool> predicate, T i)
{
var myCollection = sample.FindAll(p => pred(JustValue<double>(p)));
MainClass mainClass = new MainClass();
foreach (var x in myCollection)
{
MethodInfo changeTypeMethod = typeof(MainClass).GetMethod("GetValue");
object value = changeTypeMethod.Invoke(mainClass, new object[] { i, typeof(T) });
PropertyInfo valueProperty = x.GetType().GetProperty("value");
valueProperty.SetValue(x, value);
}
return true;
}
public T GetValue<T>(T i)
{
return (T)Convert.ChangeType(i, typeof(T));
}
public static T JustValue<T>(Mycapsule<int, T> i)
{
return i.value;
}
}
//Outside the MainClass inside the same namespace
public class Mycapsule<KT, T>
{
public KT id { get; set; }
public T value { get; set; }
public int p; // and more
}
}

How to flatten a type (not an enumerable) through Linq?

All of the examples for SelectMany I see are flattening arrays of arrays and so on. I have a different angle on this question.
I have an array of a type, and I want to extract that type's contents into a stream. Here's my example code:
public class MyClass
{
class Foo
{
public int X, Y;
}
static IEnumerable<int> Flatten(Foo foo)
{
yield return foo.X;
yield return foo.Y;
}
public static void RunSnippet()
{
var foos = new List<Foo>()
{
new Foo() { X = 1, Y = 2 },
new Foo() { X = 2, Y = 4 },
new Foo() { X = 3, Y = 6 },
};
var query = foos.SelectMany(x => Flatten(x));
foreach (var x in query)
{
Console.WriteLine(x);
}
}
}
This outputs what I'd like: 1, 2, 2, 4, 3, 6.
Can I eliminate the yields? I know that the plumbing to support that is nontrivial, and probably has a significant cost. Is it possible to do it all in linq?
I feel like I'm very close to the answer and am just missing the magic keyword to search on. :)
UPDATE:
As mentioned in the answer below, it works to use something like this:
foos.SelectMany(x => new[] { x.X, x.Y });
However, I was hoping to find a way to do this without generating n/2 temporary arrays. I'm running this against a large selection set.

If you're worried about the cost of the compiler trickery involved with yield and/or the cost of SelectMany, you could try to minimize the impact of those by not calling Flatten on each Foo but instead Flatten the foos directly:
public class MyClass
{
class Foo
{
public int X, Y;
}
static IEnumerable<int> Flatten(IEnumerable<Foo> foos)
{
foreach (var foo in foos)
{
yield return foo.X;
yield return foo.Y;
}
}
public static void RunSnippet()
{
var foos = new List<Foo>()
{
new Foo() { X = 1, Y = 2 },
new Foo() { X = 2, Y = 4 },
new Foo() { X = 3, Y = 6 },
};
var query = Flatten(foos);
foreach (var x in query)
{
Console.WriteLine(x);
}
}
}
I've run a small test app for this and I have seen that there are some performance benefits with the second implementation. On my machine, flattening 100,000 Foos with both algorithms took 36ms and 13ms, respectively. As always YMMV.

Well, if you want to avoid temporary array creation and yet you want short and nice code using LINQ, you can go with -
var query = foos.Aggregate(
new List<int>(),
(acc, x) => { acc.Add(x.X); acc.Add(x.Y); return acc; }
);

You could do this:
var query = foos.SelectMany(x => new[] { x.X, x.Y });

This kind of reverses IEnumerable<T>, and is more comparable to what we did with PushLINQ - but it is a lot simpler than implementing an iterator block on the fly (through IL), while retaining blinding performance thanks to dynamic-method; the use of object is in case your data is non-orthogonal and you need multiple types through the same API:
using System;
using System.Reflection;
using System.Reflection.Emit;
// the type we want to iterate efficiently without hard code
class Foo
{
public int X, Y;
}
// what we want to do with each item of data
class DemoPusher : IPusher<int>
{
public void Push(int value)
{
Console.WriteLine(value);
}
}
// interface for the above implementation
interface IPusher<T>
{
void Push(T value);
}
static class Program
{
// see it working
static void Main()
{
Foo foo = new Foo { X = 1, Y = 2 };
var target = new DemoPusher();
var pushMethod = CreatePusher<int>(typeof(Foo));
pushMethod(foo, target);
}
// here be dragons
static Action<object, IPusher<T>> CreatePusher<T>(Type source)
{
DynamicMethod method = new DynamicMethod("pusher",
typeof(void), new[] { typeof(object), typeof(IPusher<T>) }, source);
var il = method.GetILGenerator();
var loc = il.DeclareLocal(source);
il.Emit(OpCodes.Ldarg_0);
il.Emit(OpCodes.Castclass, source);
il.Emit(OpCodes.Stloc, loc);
MethodInfo push = typeof(IPusher<T>).GetMethod("Push");
foreach (var field in source.GetFields(BindingFlags.Instance
| BindingFlags.Public | BindingFlags.NonPublic))
{
if (field.FieldType != typeof(T)) continue;
il.Emit(OpCodes.Ldarg_1);
il.Emit(OpCodes.Ldloc, loc);
il.Emit(OpCodes.Ldfld, field);
il.EmitCall(OpCodes.Callvirt, push, null);
}
il.Emit(OpCodes.Ret);
return (Action<object, IPusher<T>>)
method.CreateDelegate(typeof(Action<object, IPusher<T>>));
}
}