Develop Reference

Comparing 1 million integers in an array without sorting it first

I have a task to find the difference between every integer in an array of random numbers and return the lowest difference. A requirement is that the integers can be between 0 and int.maxvalue and that the array will contain 1 million integers.
I put some code together which works fine for a small amount of integers but it takes a long time (so long most of the time I give up waiting) to do a million. My code is below, but I'm looking for some insight on how I can improve performance.
for(int i = 0; i < _RandomIntegerArray.Count(); i++) {
for(int ii = i + 1; ii < _RandomIntegerArray.Count(); ii++) {
if (_RandomIntegerArray[i] == _RandomIntegerArray[ii]) continue;
int currentDiff = Math.Abs(_RandomIntegerArray[i] - _RandomIntegerArray[ii]);
if (currentDiff < lowestDiff) {
Pairs.Clear();
}
if (currentDiff <= lowestDiff) {
Pairs.Add(new NumberPair(_RandomIntegerArray[i], _RandomIntegerArray[ii]));
lowestDiff = currentDiff;
}
}
}
Apologies to everyone that has pointed out that I don't sort; unfortunately sorting is not allowed.

Imagine that you have already found a pair of integers a and b from your random array such that a > b and a-b is the lowest among all possible pairs of integers in the array.
Does an integer c exist in the array such that a > c > b, i.e. c goes between a and b? Clearly, the answer is "no", because otherwise you'd pick the pair {a, c} or {c, b}.
This gives an answer to your problem: a and b must be next to each other in a sorted array. Sorting can be done in O(N*log N), and the search can be done in O(N) - an improvement over O(N2) algorithm that you have.

As per #JonSkeet try sorting the array first and then only compare consecutive array items, which means that you only need to iterate the array once:
Array.Sort(_RandomIntegerArray);
for (int i = 1; i < _RandomIntegerArray.Count(); i++)
{
int currentDiff = _RandomIntegerArray[i] - _RandomIntegerArray[i-1];
if (currentDiff < lowestDiff)
{
Pairs.Clear();
}
if (currentDiff <= lowestDiff)
{
Pairs.Add(new NumberPair(_RandomIntegerArray[i], _RandomIntegerArray[i-1]));
lowestDiff = currentDiff;
}
}
In my testing this results in < 200 ms elapsed for 1 million items.

You've got a million integers out of a possible 2.15 or 4.3 billion (signed or unsigned). That means the largest possible min distance is either about 2150 or 4300. Let's say that the max possible min distance is D.
Divide the legal integers into groups of length D. Create a hash h keyed on integers with arrays of ints as values. Process your array by taking each element x, and adding it to h[x/D].
The point of doing this is that any valid pair of points is either contained in h(k) for some k, or collectively in h(k) and h(k+1).
Find your pair of points by going through the keys of the hash and checking the points associated with adjacent keys. You can sort if you like, or use a bitvector, or any other method but now you're dealing with small arrays (on average 1 element per array).

As elements of the array are b/w 0 to int.maxvalue, so I suppose maxvalue will be less than 1 million. If it is so you just need to initialise the array[maxvalue] to 0 and then as you read 1 million values increment the value in your array.
Now read this array and find the lowest value as described by others as if all the values were sorted. If at any element is present more than 1 than its value will be >1 so you could easily say that min. difference is 0.
NOTE- This method is efficient only if you do not use sorting and more importantly int.maxvalue<<<<<(less than) 10^6(1 million).

It helps a little if you do not count on each iteration
int countIntegers = _RandomIntegerArray.Count();
for(int i = 0; i < countIntegers; i++) {
//...
for(int ii = i + 1; ii < countIntegers; ii++) {
//...
Given that Count() is only returning the number of Ints in an array on each successful count and not modifying the array or caching output until modifications.

How about splitting up the array into arraysize/number of processors sized chunks and running each chunk in a different thread. (Neil)
Assume three parts A, B and C of size as close as possible.
For each part, find the minimum "in-part" difference and that of pairs with the first component from the current part and the second from the next part (A being the next from C).
With a method taking O(n²) time, n/3 should take one ninth, done 2*3 times, this amounts to two thirds plus change for combining the results.
This calls to be applied recursively - remember Карацу́ба/Karatsuba multiplication?
Wait - maybe use two parts after all, for three fourth of the effort - very close to "Karatsuba". (When not seeing how to use an even number of parts, I was thinking multiprocessing with every processor doing "the same".)

Getting all combinations of K and less elements in List of N elements with big K

I want to have all combination of elements in a list for a result like this:
List: {1,2,3}
1
2
3
1,2
1,3
2,3
My problem is that I have 180 elements, and I want to have all combinations up to 5 elements. With my tests with 4 elements, it took a long time (2 minutes) but all went well. But with 5 elements, I get a run out of memory exception.
My code presently is this:
public IEnumerable<IEnumerable<Rondin>> getPossibilites(List<Rondin> rondins)
{
var combin5 = rondins.Combinations(5);
var combin4 = rondins.Combinations(4);
var combin3 = rondins.Combinations(3);
var combin2 = rondins.Combinations(2);
var combin1 = rondins.Combinations(1);
return combin5.Concat(combin4).Concat(combin3).Concat(combin2).Concat(combin1).ToList();
}
With the fonction: (taken from this question: Algorithm to return all combinations of k elements from n)
public static IEnumerable<IEnumerable<T>> Combinations<T>(this IEnumerable<T> elements, int k)
{
return k == 0 ? new[] { new T[0] } :
elements.SelectMany((e, i) =>
elements.Skip(i + 1).Combinations(k - 1).Select(c => (new[] { e }).Concat(c)));
}
I need to search in the list for a combination where each element added up is near (with a certain precision) to a value, this for each element in an other list. There is all my code for this part:
var possibilites = getPossibilites(opt.rondins);
possibilites = possibilites.Where(p => p.Sum(r => r.longueur + traitScie) < 144);
foreach(BilleOptimisee b in opt.billesOptimisees)
{
var proches = possibilites.Where(p => p.Sum(r => (r.longueur + traitScie)) < b.chute && Math.Abs(b.chute - p.Sum(r => r.longueur)) - (p.Count() * 0.22) < 0.01).OrderByDescending(p => p.Sum(r => r.longueur)).ElementAt(0);
if(proches != null)
{
foreach (Rondin r in proches)
{
opt.rondins.Remove(r);
b.rondins.Add(r);
possibilites = possibilites.Where(p => !p.Contains(r));
}
}
}
With the code I have, how can I limit the memory taken by my list ? Or is there a better solution to search in a very big set of combinations ?
Please, if my question is not good, tell me why and I will do my best to learn and ask better questions next time ;)

Your output list for combinations of 5 elements will have ~1.5*10^9 (that's billion with b) sublists of size 5. If you use 32bit integers, even neglecting lists overhead and assuming you have a perfect list with 0b overhead - that will be ~200GB!
You should reconsider if you actually need to generate the list like you do, some alternative might be: streaming the list of elements - i.e. generating them on the fly.
That can be done by creating a function, which gets the last combination as an argument - and outputs the next. (to think how it is done, think about increasing by one a number. you go from last to first, remembering a "carry over" until you are done)
A streaming example for choosing 2 out of 4:
start: {4,3}
curr = start {4, 3}
curr = next(curr) {4, 2} // reduce last by one
curr = next(curr) {4, 1} // reduce last by one
curr = next(curr) {3, 2} // cannot reduce more, reduce the first by one, and set the follower to maximal possible value
curr = next(curr) {3, 1} // reduce last by one
curr = next(curr) {2, 1} // similar to {3,2}
done.
Now, you need to figure how to do it for lists of size 2, then generalize it for arbitrary size - and program your streaming combination generator.
Good Luck!

Let your precision be defined in the imaginary spectrum.
Use a real index to access the leaf and then traverse the leaf with the required precision.
See PrecisLise # http://net7mma.codeplex.com/SourceControl/latest#Common/Collections/Generic/PrecicseList.cs
While the implementation is not 100% complete as linked you can find where I used a similar concept here:
http://net7mma.codeplex.com/SourceControl/latest#RtspServer/MediaTypes/RFC6184Media.cs
Using this concept I was able to re-order h.264 Access Units and their underlying Network Access Layer Components in what I consider a very interesting way... outside of interesting it also has the potential to be more efficient using close the same amount of memory.
et al, e.g, 0 can be proceeded by 0.1 or 0.01 or 0.001, depending on the type of the key in the list (double, float, Vector, inter alia) you may have the added benefit of using the FPU or even possibly Intrinsics if supported by your processor, thus making sorting and indexing much faster than would be possible on normal sets regardless of the underlying storage mechanism.
Using this concept allows for very interesting ordering... especially if you provide a mechanism to filter the precision.
I was also able to find several bugs in the bit-stream parser of quite a few well known media libraries using this methodology...

I found my solution, I'm writing it here so that other people that has a similar problem than me can have something to work with...
I made a recursive fonction that check for a fixed amount of possibilities that fit the conditions. When the amount of possibilities is found, I return the list of possibilities, do some calculations with the results, and I can restart the process. I added a timer to stop the research when it takes too long. Since my condition is based on the sum of the elements, I do every possibilities with distinct values, and search for a small amount of possibilities each time (like 1).
So the fonction return a possibility with a very high precision, I do what I need to do with this possibility, I remove the elements of the original list, and recall the fontion with the same precision, until there is nothing returned, so I can continue with an other precision. When many precisions are done, there is only about 30 elements in my list, so I can call for all the possibilities (that still fits the maximum sum), and this part is much easier than the beginning.
There is my code:
public List<IEnumerable<Rondin>> getPossibilites(IEnumerable<Rondin> rondins, int nbElements, double minimum, double maximum, int instance = 0, double longueur = 0)
{
if(instance == 0)
timer = DateTime.Now;
List<IEnumerable<Rondin>> liste = new List<IEnumerable<Rondin>>();
//Get all distinct rondins that can fit into the maximal length
foreach (Rondin r in rondins.Where(r => r.longueur < (maximum - longueur)).DistinctBy(r => r.longueur).OrderBy(r => r.longueur))
{
//Check the current length
double longueur2 = longueur + r.longueur + traitScie;
//If the current length is under the maximal length
if (longueur2 < maximum)
{
//Get all the possibilities with all rondins except the current one, and add them to the list
foreach (IEnumerable<Rondin> poss in getPossibilites(rondins.Where(rondin => rondin.id != r.id), nbElements - liste.Count, minimum, maximum, instance + 1, longueur2).Select(possibilite => possibilite.Concat(new Rondin[] { r })))
{
liste.Add(poss);
if (liste.Count >= nbElements && nbElements > 0)
break;
}
//If this the current length in higher than the minimum, add it to the list
if (longueur2 >= minimum)
liste.Add(new Rondin[] { r });
}
//If we have enough possibilities, we stop the research
if (liste.Count >= nbElements && nbElements > 0)
break;
//If the research is taking too long, stop the research and return the list;
if (DateTime.Now.Subtract(timer).TotalSeconds > 30)
break;
}
return liste;
}

Understanding this C# code, I mean only the code, not the theory of Entropy [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 6 years ago.
Improve this question
I finally found a code possibly working Shannon Entropy calculation, but as I don't fully understand C# at all, could someone help me fully grasp it please? I mean purely understand the code, NOT what it is doing. I do understand Delphi, if you ask.
public static double ShannonEntropy(string s)
{
var map = new Dictionary<char, int>();
foreach (char c in s)
{
if (!map.ContainsKey(c))
map.Add(c, 1);
else
map[c] += 1;
}
double result = 0.0;
int len = s.Length;
foreach (var item in map)
{
var frequency = (double)item.Value / len;
result -= frequency * (Math.Log(frequency) / Math.Log(2));
}
return result;
}

You have a function called ShannonEntropy which takes a string s and returns something of type double (basically a floating point number).
You instantiate a map (dictionary) of characters -> integers and call it map.
Loop over every character in the string s. During that loop check to see if the character is in our map (map). If it is not, map that character to 1. If it is already in the map, increment the value. At the end we'll have a map of each character to a count of how often it appears in s.
Declare a double name result and set it 0.
Declare an integer called len which is the length of the input string.
Loop over every character in our map. During the loop we're going to compute the frequency of how often it appears.
We declare a variable called frequency and set it to the count (the value we stored in the map) divided by the length of the string - so the frequency gets set to the percentage of each character that appears in s every time we loop.
Now we take the frequency (the percentage the given character appeared) and multiply it by the log (base 2) of the frequency, take the product and subtract it from result (which was initially 0).
Once we've finished looping we return the result.

Var is initializing variable map , new dictionary has added a dictionary to the var . dictionary gives the string or a char value an integer Value, let's say that red = 1 , blue = 2 , green = 3 etc . I think foreach runs the loop for as long as there there are char in string .
You must know if( ) it compiles the code only if the statement inside is true or false as in this case.math.log is a method . then at end we return result . I am not very familiar with c# I am more familiar with c and Java .

Fibonacci Series recursive c# method

I have hard time writing a method which returns a fibonacci series using recursion.
My iterative method is as follows:
public static string Fibonacci(int n)
{
if (n < 2)
return "1";
int[] numbers = new int[n];
numbers[0]=0;
numbers[1]=1;
for (int i = 2; i < n; i++)
{
numbers[i] = numbers[i - 1] + numbers[i - 2];
}
return string.Join(" ", numbers);
}
I want above method to be changed to recursive call but with same signature ie return type is string and it returns Fibonacci series e.g 0,1,1,2,3,5,8,13,21,34,55
I googled but everywhere I see its done using Console.WriteLine() i.e. values are printed to screen but not returned as string from function.
Thanks

There are, of course, many examples of Fibonacci algorithms on the web, recursive and otherwise. Naturally, the key to a recursive implementation is to take the results from the recursive call, combine that with the current result, and then return that to the caller.
So let's start with the basic Fibonacci idea, with a recursive method that just writes out the numbers as they are generated:
void Fibonacci(int iMinus2, int iMinus1, int count)
{
if (count == 0) return;
int current = iMinus2 + iMinus1;
Console.WriteLine(current);
Fibonacci(iMinus1, current, count - 1);
}
So then the question is, how do we adjust the above so that it returns a string instead of just writing out the numbers one at a time. Again, remember that the key is to always have a way to combine the current result with the combined result of the previous calls. In this case, that means we want to convert the current number to a string, and prepend that string to the string returned by the recursive call (which is all the numbers after the current number:
string Fibonacci(int iMinus2, int iMinus1, int count)
{
if (count == 0) return null;
int current = iMinus2 + iMinus1;
string nextNumbers = Fibonacci(iMinus1, current, count - 1);
return nextNumbers != null ?
current.ToString() + ", " + nextNumbers : current.ToString();
}
Note: the above is a bit more complicated than I'd suggested, because it handles avoiding the addition of a comma when there are no more next numbers. There are other ways to implement this, but I prefer implementing recursive methods such that the terminating condition is handled by the callee rather than the caller.
I leave it as an exercise for the reader to deal with calling the above, and with how to deal with short sequences (i.e. the degenerate case where one is looking at only the first or first two numbers in the sequence). :)
Note: your question sounds a lot like a homework question. If so: these are appropriate on Stack Overflow, but I can't emphasize enough that Stack Overflow should not be considered a substitute for your teacher. We are happy to provide advice and help on homework questions, but it's important that you maintain a relationship with your teacher and seek advice from them so that they better understand where you are having trouble.

Fibonacci program does not complete execution

Problem Description
By considering the terms in the Fibonacci sequence whose values do not exceed four million, find the sum of the even-valued terms
Program
using System;
class fibonacci {
// function to return Nth value of fibonacci
public static long fibo_n(long N) {
long fibon=0;
switch (N) {
case 1: fibon=1; break;
case 2: fibon=2; break;
}
while(N>2) {
fibon=fibo_n(N-1)+fibo_n(N-2);
}
return fibon;
}
}
class fibo_tester {
static void Main() {
int N=2;
long sum=0;
while(fibonacci.fibo_n(N) <= 13) {
sum = sum + fibonacci.fibo_n(N);
N=N+3;
}
Console.WriteLine("Sum is {0}: ", sum);
}
}
I reduced the number to 13 for testing instead of 4 million, but it still hangs. Could someone please advise?
EDIT 2
switch (N) {
case 1: fibon=1; break;
case 2: fibon=2; break;
default: fibon=fibo_n(N-1)+fibo_n(N-2); break;
}

while(N>2) {
fibon=fibo_n(N-1)+fibo_n(N-2);
}
Is going to loop indefinitely, N is never updated in the loop. You probably need to remove the While() clause and change it all to return fibo_n(N-1)+fibo_n(N-2); I'm not sure what you are doing with the switch statements, etc. But that should be a start.
I would actually replace it with this (if you want to use the switch):
class fibonacci
{
// function to return Nth value of fibonacci
public static long fibo_n(long N)
{
switch (N)
{
case 0:
return 0;
case 1:
return 1;
default:
return fibo_n(N - 1) + fibo_n(N - 2);
}
}
}
You might want to consider storing the values for each value of N in a dictionary or some sort of set so that you can look them up later. Since in your main program it seems like you are going to be looping over the values (successively larger N's that may have already been calculated). I'm not sure what the N+3 is about in your main loop, but you probably are going to miss something there (false assumption of the N-1, N-2 in the recursion perhaps?)
Also, if summing and dependent on your platform and how large of a value you test for (sicne your testing the sum of the first X Fibonacci numbers) you may have to use a ulong or find some other datatype that can handle larger numbers. If I don't change everything from long to ulong on my system, the values wrap around. Fibonacci number can't be negative anyways, so why not use a ulong, or uint64 or something with more bits.

You are using BOTH a loop AND recursion - you'll need to pick one or the other. The problem spot in your code is here:
while (N>2) {
fibon = fibo_n(N-1)+fibo_n(N-2);
}
In this spot the value of N never actually changes - that happens in the recursive step. One example (not good) way to write this is:
public static long fibo_n(long N) {
if (N <= 0) return 0;
if (N == 1) return 1;
if (N <= 4) return N - 1;
return fibo_n(N-1) + fibo_n(N-2);
}
Best of luck!
Performance Note:
The reason this isn't a good approach is because function calls use memory in C# and they also take time. Looping is always faster than recursion and in this case there isn't a good reason to solve this with recursion (as opposed to a simple loop). I'll leave the code as is so you can compare to what you have, but I think you can find much better better code options elsewhere (although not all of these will be in C#).

Let's say you call your method with N equal to 3.
The switch doesn't do anything.
N is greater than 2, so we go into the while loop. The while loop does something; it computes a value, assigns it to fibon. Then N, since it hasn't changed, is still greater than 2, so we compute it again. It'll still be greater than two, so we compute it again. We never stop computing it.
Every time you assign a value to fibon you should instead be returning that value, because you're done; you have nothing left to compute. There's also no need for the while loop. You never need to execute that code more than once.
Your implementation is also super inefficient. To compute fib(5) you compute fib(4) and fib(3). To Then when computing fib(4) you compute fib(3) [again] and fib(2), when computing fib(3) both times they each compute fib(2) (so that's three times there), and if we do the math we end up seeing that you perform on the order of 2^n computations when computing fib(n). On top of that, you're calling fib n times as you're counting up from zero in your main loop. That's tons and tons and tons of needless work re-computing the same values. If you just have a simple loop adding the numbers together as you go you can compute the result with on the order of N computations.