Develop Reference

Why does it take more time when you run a LINQ OrderBy before Select?

While writing a solution for a coding problem I discovered an interesting behavior of my LINQ statements. I had two scenarios:
First:
arr.Select(x => x + 5).OrderBy(x => x)
Second:
arr.OrderBy(x => x).Select(x => x + 5)
After a little bit of testing with System.Diagnostics.Stopwatch I got the following results for an integer array of length 100_000.
For the first approach:
00:00:00.0000152
For the second:
00:00:00.0073650
Now I'm interested in why it takes more time if I do the ordering first. I wasn't able to find something on google so I thought about it by myself.
I ended up with 2 Ideas:
1. The second scenario has to convert to IOrderedEnumerable and then back to IEnumerable while the first scenario only has to convert to IOrderedEnumerable and not back.
2. You end up having 2 loops. The first for sorting and the second for the selecting while approach 1 does everything in 1 loop.
So my question is why does it take much more time to do the ordering before select?

Let's have a look at the sequences:
private static void UnderTestOrderBySelect(int[] arr) {
var query = arr.OrderBy(x => x).Select(x => x + 5);
foreach (var item in query)
;
}
private static void UnderTestSelectOrderBy(int[] arr) {
var query = arr.Select(x => x + 5).OrderBy(x => x);
foreach (var item in query)
;
}
// See Marc Gravell's comment; let's compare Linq and inplace Array.Sort
private static void UnderTestInPlaceSort(int[] arr) {
var tmp = arr;
var x = new int[tmp.Length];
for (int i = 0; i < tmp.Length; i++)
x[i] = tmp[i] + 5;
Array.Sort(x);
}
In order to perform benchmark, let's run 10 times and average 6 middle results:
private static string Benchmark(Action<int[]> methodUnderTest) {
List<long> results = new List<long>();
int n = 10;
for (int i = 0; i < n; ++i) {
Random random = new Random(1);
int[] arr = Enumerable
.Range(0, 10000000)
.Select(x => random.Next(1000000000))
.ToArray();
Stopwatch sw = new Stopwatch();
sw.Start();
methodUnderTest(arr);
sw.Stop();
results.Add(sw.ElapsedMilliseconds);
}
var valid = results
.OrderBy(x => x)
.Skip(2) // get rid of top 2 runs
.Take(results.Count - 4) // get rid of bottom 2 runs
.ToArray();
return $"{string.Join(", ", valid)} average : {(long) (valid.Average() + 0.5)}";
}
Time to run and have a look at the results:
string report = string.Join(Environment.NewLine,
$"OrderBy + Select: {Benchmark(UnderTestOrderBySelect)}",
$"Select + OrderBy: {Benchmark(UnderSelectOrderBy)}",
$"Inplace Sort: {Benchmark(UnderTestInPlaceSort)}");
Console.WriteLine(report);
Outcome: (Core i7 3.8GHz, .Net 4.8 IA64)
OrderBy + Select: 4869, 4870, 4872, 4874, 4878, 4895 average : 4876
Select + OrderBy: 4763, 4763, 4793, 4802, 4827, 4849 average : 4800
Inplace Sort: 888, 889, 890, 893, 896, 904 average : 893
I don't see any significant difference, Select + OrderBy seems to be slightly more efficient (about 2% gain) than OrderBy + Select. Inplace Sort, however, has far better performance (5 times) than any of Linq.

Depending on which Linq-provider you have, there may happen some optimization on the query. E.g. if you´d use some kind of database, chances are high your provider would create the exact same query for both statements similar to this one:
select myColumn from myTable order by myColumn;
Thus performamce should be identical, no matter if you order first in Linq or select first.
As this does not seem to happen here, you probably use Linq2Objects, which has no optimization at all. So the order of your statements may have an efffect, in particular if you´d have some kind of filter using Where which would filter many objects out so that later statements won´t operate on the entire collection.
To keep long things short: the difference most probably comes from some internal initialzation-logic. As a dataset of 100000 numbers is not really big - at least not big enough - even some fast initialization has a big impact.

Rotate List<T> based on a condition

Say I have a List<Point> { {5,2}, {7,2}, {3,9} }
I want my output to always start with the point with the smallest X value
So something like:
var output = input.Rotate((x, y) => x.X < y.X));
output = `List<Point> { {3,9}, {5,2}, {7,2} }`
It's not hard to do manually:
Find the index of the object meeting the condition
Create a new List
Add everything from index to end to that list
Add everything from 0 to index-1
I'm just wondering if there is a LINQ way of doing this?

First find the minimum X value:
var minX = input.Min(p => p.X);
Next get the first occurrence where X is that minimum value:
var index = input.FindIndex(p => p.X == minX);
Now create a new list by splitting the original into two sections:
var newInput = input.Skip(index).Concat(input.Take(index));

You can't do it in one step, you will need at least two iterations through the array. But you could do this (kinda hacky) approach:
var range = Enumerable.Range(0, list.Count);
var index = range.Aggregate((p,c) => list[p].X> list[c].X? c : p);
var rotated = range.Select(i => list[(i + index) % list.Count]).ToList();
Here's an example (using Tuple rather than Point, but it's the same principle)
The first step is to find the index of the lowest value in your array. The second step them builds your new array starting at that index and looping around.
If you wanted to encapsulate it in an extension method, you could do something like this:
public static IEnumerable<T> Rotate<T>(this List<T> list, Func<T,T, bool> comparer)
{
var range = Enumerable.Range(0, list.Count);
var index = range.Aggregate((p,c) => predicate(list[p],list[c]) ? p : c);
return range.Select(i => list[(i + index) % list.Count]);
}
Which you'd call like this:
var output = input.Rotate((x, y) => x.X < y.X));
You pass in a function which, if it evaluates to true, will select x instead of y

Quality of an estimation function

I am trying to find a way to determine the quality of an estimation function.
I have a dictionary that contains int values.
The total "sum" of this dictionary is Dictionary Key * Value.
public int RealValue
{
get
{
return Items.Sum(x => x.Key * x.Value);
}
}
The estimated sum of the Dictionary is calculated by using windows and weights.
public int EstimatedValue
{
get
{
return Items.Where(x => x.Key < window1).Sum(x => weight1 * x.Value) +
(Items.Where(x => x.Key >= window1 && x.Key < window2).Sum(x => weight2 * x.Value)) +
Items.Where(x => x.Key >= window2 && x.Key < window3).Sum(x => weight3 * x.Value);
}
}
Now I want to assing a rating to this Estimation Function, i.e. to the quality of the choosen windows and weights.
The estimation function is good, if it can successfully determine which of two dictionaries contain the greater value. It does not matter how close the estimation is to the real count. Of course the Estimation Function is supposed to work with any random pair of dictionaries that are candidates for testing.
What would be a good approach to solve the above problem?

It seems that in the end you are looking at the ranking problem, as you assign each object a speciffic order as compares to the remaining ones ("which one is bigger?") and do not care about actual value, thus there is a whole research regarding ranking evaluation and in general learning to rank. There are plenty of reliable metrics here, from the simpluest ones - testing number of correct pairwise comparisons (how many times your model gives good ordering of two objects as compared to all compared) to more statistical measures like Kendall tank correlation coefficient.

Take N values from observable on each interval

I have an observable which streams a value for each ms. , this is done every 250 ms. ( meaning 250 values in 250 ms (give or take) ).
Mock sample code :
IObservable<IEnumerable<int>> input = from _ in Observable.Interval(TimeSpan.FromMilliseconds(250))
select CreateSamples(250);
input.Subscribe(values =>
{
for (int i = 0; i < values.Count(); i++)
{
Console.WriteLine("Value : {0}", i);
}
});
Console.ReadKey();
private static IEnumerable<int> CreateSamples(int count)
{
for (int i = 0; i < 250; i++)
{
yield return i;
}
}
What i need is to create some form of process observable which process the input observable in a rate of 8 values every 33 ms
Something along the line of this :
IObservable<IEnumerable<int>> process = from _ in Observable.Interval(TimeSpan.FromMilliseconds(33))
select stream.Take(8);
I was wondering 2 things :
1) How can i write the first sample with the built in operators that reactive extensions provides ?
2) How can i create that process stream which takes values from the input stream
which with the behavior iv'e described ?
I tried using Window as a suggestion from comments below .
input.Window(TimeSpan.FromMilliseconds(33)).Take(8).Subscribe(winObservable => Debug.WriteLine(" !! "));
It seems as though i get 8 and only 8 observables of an unknown number of values
What i require is a recurrence of 8 values every 33 ms. from input observable.
What the code above did is 8 observables of IEnumrable and then stand idle.
EDIT : Thanks to James World . here's a sample .
var input = Observable.Range(1, int.MaxValue);
var timedInput = Observable.Interval(TimeSpan.FromMilliseconds(33))
.Zip(input.Buffer(8), (_, buffer) => buffer);
timedInput.SelectMany(x => x).Subscribe(Console.WriteLine);
But now it get's trickier i need for the Buffer value to calculated
i need this to be done by the actual MS passed between Intervals
when you write a TimeSpan.FromMilliseconds(33) the Interval event of the timer would actually be raised around 45 ms give or take .
Is there any way to calculate the buffer , something like PSUDO
input.TimeInterval().Buffer( s => s.Interval.Milliseconds / 4)

You won't be able to do this with any kind of accuracy with a reasonable solution because .NET timer resolution is 15ms.
If the timer was fast enough, you would have to flatten and repackage the stream with a pacer, something like:
// flatten stream
var fs = input.SelectMany(x => x);
// buffer 8 values and release every 33 milliseconds
var xs = Observable.Interval(TimeSpan.FromMilliseconds(33))
.Zip(fs.Buffer(8), (_,buffer) => buffer);
Although as I said, this will give very jittery timing. If that kind of timing resolution is important to you, go native!

I agree with James' analysis.
I'm wondering if this query gives you a better result:
IObservable<IList<int>> input =
Observable
.Generate(
0,
x => true,
x => x < 250 ? x + 1 : 0,
x => x,
x => TimeSpan.FromMilliseconds(33.0 / 8.0))
.Buffer(TimeSpan.FromMilliseconds(33.0));

Simplest and possibly most efficient way to check for data in a line in an array

I'm making a Connect 4 Console game in C#. I have all the application set up and working correctly except the hasGameBeenWon() function. The function is called in the Main do...while loop and returns false when the game has not been won.
static bool gameHasEnded(int currentTurn)
{
int currentTurnPiece;
if (currentTurn.Equals(1))
{
currentTurnPiece = BLUE;
}
else if (currentTurn.Equals(2))
{
currentTurnPiece = RED;
}
return false;
}
(obviously, this code is incomplete)
...Is how it current looks, the currentTurn var is int value of the current player's turn (it can be either 1 or 2). BLUE and RED are constants that are assigned the values of 1 and 2 that are converted into Blue and Red colours on the board when the application prints to the console.
Now I'm stuck on how to check if the game has been won efficiently and as simple as possible. The board is stored in an Array called board, I need to check to see if there is a set of 4 same-values going up/down, left/right and also diagonal in all directions. The checking up/down and left/right is not that hard, but the diagonal I thought maybe a little slow in having to check every single time.
Is there anyway I could speed up checking the entire board or possibly skip checking the entire board altogether and possibly searching just the relevant areas?

I'm going to assume that board is just some 2-D array of enum Cell {NONE, BLUE, RED} values with width and height and that you want to "connect N" (where in your example N is 4).
A simple vector class:
class Vec
{
public int i;
public int j;
public Vec(int i, int j) { this.i = i; this.j = j; }
}
The types of patterns/templates that describe a win:
var winTemplates = new List<List<Vec>>
{
Enumerable.Range(0, N).Select(x => new Vec(x, 0)).ToList(),
Enumerable.Range(0, N).Select(x => new Vec(0, x)).ToList(),
Enumerable.Range(0, N).Select(x => new Vec(x, x)).ToList(),
Enumerable.Range(0, N).Select(x => new Vec(N - x - 1, x)).ToList()
};
Is there a (Cell, List<Vec>) tuple describing a win at position p?
Func<Vec, Tuple<Cell, List<Vec>>> getWin = p => winTemplates
.SelectMany(winTemplate => winTemplate
.Select(q => new Vec(p.i + q.i, p.j + q.j))
.GroupBy(v => board[v.i, v.j])
.Where(g => g.Key != Cell.NONE && g.Count() == N)
.Select(g => Tuple.Create(g.Key, g.ToList())))
.FirstOrDefault();
Is there a win anywhere on the board?
Func<Tuple<Cell, List<Vec>>> findWin = () => Enumerable.Range(0, height - N)
.SelectMany(i => Enumerable.Range(0, width - N).Select(j => new Vec(i, j)))
.Select(p => getWin(p))
.FirstOrDefault();
This is just browser code, so it may have typos. If you need more help understanding it, let me know in a comment.