Develop Reference

Translate Excel Spreadsheets with SOLVER in C#

After several searches and mistakes on my part, I finally managed to get out of it and get a result from the MSF solver.
However, it's not perfect, because I still have a difference against me in my C# code.
In the Excel workbook I have 6 solvers, relatively identical.
Only one solver per tab, but I have a lot of calculations.
In order to best stick to the Excel workbook, I created one method per cell containing a formula.
My code works, in the sense that if I give it the same data as Excel I have the same results, but with the solver I have a little difference.
Here's what I did, and I'd like you to tell me if there's anything I can improve by trying to keep my methods (representing my Excel Cells)
Each representation of the cells is created twice.
I need to have the value of my cell to do other calculations and it seems that I can't put methods returning a double, in the solver.
Classic method:
private double Cell_I5()
{
double res = 0;
res = (Math.Exp(-Var.Calc.Var4 * Var.Calc.De * M23) - 1) / (-Var.Calc.Var4 * Var.Calc.De * M23);
return res;
}
Method for the solver:
private Term Solv_I5()
{
Term res = 0;
res = (Model.Exp(-Var.Calc.Var4 * Var.Calc.De * Solver_M23) - 1) / (-Var.Calc.Var4 * Var.Calc.De * Solver_M23);
return res;
}
'M23' is a double
'Solver_M23' is a Decision
'Var4' is a double as well as 'De'.
So I use the return value with "Term" and I change all the Math functions to 'Model', except Math.Pi which is a constant.
You can imagine that there are close to 60 to 70 methods involved like that.
My method for the solver:
public void StartSolver()
{
var solver = SolverContext.GetContext();
solver.ClearModel();
var model = solver.CreateModel();
//Instanciation des variables du Solver en format Real(double) Non Negative
Solver_M22 = new Decision(Domain.RealNonnegative, "M22");
Solver_M23 = new Decision(Domain.RealNonnegative, "M23");
Solver_M24 = new Decision(Domain.RealNonnegative, "M24");
Solver_M25 = new Decision(Domain.RealNonnegative, "M25");
Solver_M26 = new Decision(Domain.RealNonnegative, "M26");
model.AddDecision(Solver_M22);
model.AddDecision(Solver_M23);
model.AddDecision(Solver_M24);
model.AddDecision(Solver_M25);
model.AddDecision(Solver_M26);
model.AddConstraint("M22a", Solver_M22 <= 4);
model.AddConstraint("M22b", Solver_M22 >= 0);
model.AddConstraint("M23a", Solver_M23 <= 2);
model.AddConstraint("M23b", Solver_M23 >= 0.001);
model.AddConstraint("M24a", Solver_M24 <= 2);
model.AddConstraint("M24b", Solver_M24 >= 0);
model.AddConstraint("M25a", Solver_M25 <= 2);
model.AddConstraint("M25b", Solver_M25 >= 0);
model.AddConstraint("M26a", Solver_M26 <= 2);
model.AddConstraint("M26b", Solver_M26 >= 0.001);
//Test with classical calculation methods
double test = Cell_H33() + Cell_H23();
//Adding Solver Methods
model.AddGoal("SommeDesCarresDesEquartsGlobal", GoalKind.Minimize, Solv_H33() + Solv_H23());
// Solve our problem
var solution = solver.Solve();
// Get our decisions
M22 = Solver_M22.ToDouble();
M23 = Solver_M23.ToDouble();
M24 = Solver_M24.ToDouble();
M25 = Solver_M25.ToDouble();
M26 = Solver_M26.ToDouble();
string s = solution.Quality.ToString();
//For test
double testSortie = Cell_H33() + Cell_H23();
}
Questions:
1)
At no time do I indicate whether it is a linear calculation or not. How to indicate if necessary?
In Excel it is declared nonlinear
I saw that the solver was looking for the best method on its own.
2)
Is there something I'm not doing right, because I don't have the same value (with Excel)? I checked several times all the methods by one, with the amount that I missed, maybe, something, I will recheck tomorrow.
3)
Apart from doing the calculation with the classic methods, I have not found a way to find my result. From the 'solution' object
How to extract it from the result if possible?
4)
Here is the result of the 5 variables I find MSF C#:
0.06014756519010750
0.07283670953453890
0.07479568348101340
0.02864805010533950
0.00100000002842722
And what I find the Excel solver:
0.0000
0.0010
0.0141
0.0000
0.0010
Is there a way to restrict the number of decimal places directly in the calculations?
Because when I reduce manually (after calculation) that changes my result quite a bit?
Thank you.
[EDIT] Forgot to post this message it was still pending.
This morning I ran the C# solver calculation again and the result is really different with a huge difference in the result.
I remind you that I want to minimize the result.
Excel = 3.92
C#=8122.34
Result not acceptable at all.
[EDIT 2]
I may have a clue:
When I doing a simple calculation, such as:
private Term Solv_I5()
{
Term res = 0;
res = Model.Exp(-Var.Calc.Var4 * Var.Calc.Den * Solver_M25);
return res;
}
the result is:
{Exp(Times(-4176002161226263/70368744177664, M25))}
Why "Times"
All formulas with multiplication contain Times.
For divisions there is 'Quotient', additions 'Plus', but multiplications 'Times !!!
Question 4)
Am I doing the multiplications wrong in a 'Term'.?
Do you have an idea?
[EDIT 3]
I just saw that "times" was not a stupid term, another misunderstanding on my part of the English language, sorry.
So that doesn't solve my problem.
Can you help me please.

or-tools - Compute the stdev from a SumArray()

I need to generate plannings for employees using Google's Optimization Tools.
One of the constraints would be that every employee has approximately the same amount of working hours.
Thus, I want to aggregate in a list how many hours each employee is working, and then minimize the standard deviation of this list.
var workingTimes = new List<SumArray>();
foreach (var employee in employees) {
// Gather the duration of each task the employee is
// assigned to in a list
// o.IsAssign is an IntVar and task.Duration is an int
var allDurations = shifts.Where(o => o.Employee == employee.Name)
.Select(o => o.IsAssigned * task[o.Task].Duration);
// Total time the employee is working
var workTime = new SumArray(allDurations);
workingTimes.Add(workTime);
}
Now I want to minimize the stdev of workingTimes. I tried the following:
IntegerExpression workingTimesMean = new SumArray(workingTimes) * (1/workingTimes.Count);
var gaps = workingTimes.Select(o => (o - workingTimesMean)*(o - workingTimesMean));
var stdev = new SumArray(gaps) * (1/gaps.Count());
model.Minimize(stdev);
But the LINQ query at the 2nd line of the last code snippet is throwing me an error:
Can't apply operator * to IntegerExpression and IntegerExpression
How can I compute the standard deviation of a Google.OrTools.Sat.SumArray?

The 'natural' API only supports linear expressions.
You need to use the AddProductEquality() API.
Please note that 1 / Gaps.Count() will always return 0 (we are in integer arithmetic).
So you need to scale everything up.
Personally, I would just minimize the unscaled sum of abs(val - average). No need to divide by the number of elements.
Just check that the computation of the average has the right precision (once again, we are in integer arithmetic).
You could also consider just minimize the max(abs(val - average)). This is simpler and may be good enough.

C# - Taking top 10 values and multiplying them

I am currently working on a bayesian spam filter, made a filter using an algorithm, but it wil not work for long emails, there are just too much values to multiply and it excedes the range of double. I thought about making it so that I only take 10 or 20 most important (highest values for both spam and ham) and multiply only them. I thought about making another Dictionary inside and then multiply values out of it.
This is how it looks right now:
if (countsWordOccurenceSpam.ContainsKey(word.Key) && (!countsWordOccurenceOk.ContainsKey(word.Key)))
{
int spamValue = 0;
countsWordOccurenceSpam.TryGetValue(word.Key, out spamValue);
totals = spamValue ;
fprob_spam = ((double)spamValue) / ile_spam;
sum_spam = (((weight * probability) + (totals * fprob_spam)) / (totals + weight));
sum_ok = ((weight * probability) / (totals + weight));
sum_spam = Math.Pow(sum_spam, word.Value);
sum_ok = Math.Pow(sum_ok, word.Value);
wp_spam_1 = wp_spam_1 * sum_spam;
last_o_1 = last_o_1 * sum_ok;
}
This is one part of algorithm, now I am thinking about putting all the values from sum_spam to one Dictionary, and all the values from sum_ok to another and take using .Take(10) to select 10 highest values and multiply all of them.
Does it seem right? I am really thinking it would be very inefficient, Is there any way to do it?

FSK Demodulation - Parsing Japanese EWS Data

【This is not a duplicate. Similar questions are about scenarios where people have control over the source data. I do not.】
In Japan there's something called the "Emergency Warning Broadcasting System." It looks like this when activated: http://www.youtube.com/watch?v=9hjlYvp9Pxs
In the above video, at around 2:37, a FSK-modulated signal is sent. I want to parse this signal; i.e. given a WAV file that contains the signal, I want to end up with a StringBuilder that contains 0s and 1s to process them later. I have the spec for the binary data and all, but the problem is that I know nothing about audio programming. :(
This is just for a hobby project, but I became hooked. TV and radio makers can pick up this signal and have their appliances do stuff in reaction to it, so it can't be that hard, right? :(
Facts about the signal:
The mark tone is 1024Hz, and the stop tone is 640Hz
Each tone is 15.625ms long
2 second pause before signal begins and after it ends (probably for detection purposes)
What I did so far:
Write a simple RIFF parser that accepts 8bit mono WAV files and allows me to get samples from them. I've tested it and it works.
A loop that takes 15.625ms of samples and:
Uses RMS to look for two seconds of silence
Uses the Goertzel algorithm to decide if the signal is 1024Hz or 640Hz
The problems I have:
0s and 1s are swallowed during the loop depending on the test data.
Given the clarity of the signal (YouTube-to-MP3 rip), that shouldn't happen.
If I generate a repeating 01 sequence in Audacity 30 times, my program will pick up around 10 of the 01 pairs, instead of 30
Sometimes 0s and 1s are swapped (side effect of the above?)
If I tweak the code so it works with one test sound file, other test sound files stop working
My questions:
Can anyone give me a high level overview on how FSK decoding would be done properly in software?
Do I need to apply some sort of filter that limits the signal to 640Hz+1024Hz and mutes everything else?
What is the best approach to keep the timing right? Maybe I'm doing it wrong?
Any links to beginner's literature on this kind of audio processing? I'd really like to learn and get this working.
The code that reads samples is (simplified):
StringBuilder ews_bits = new StringBuilder();
double[] samples = new double[(int)(samplesPerMs * 16.625D)];
int index = 0, readTo = /* current offset + RIFF subChunk2Size */;
BinaryReader br = /* at start of PCM data */;
while (br.BaseStream.Position < readTo)
{
switch (bitsPerSample / 8)
{
case 1: // 8bit
samples[index++] = ((double)br.ReadByte() - 127.5D) / 256D;
break;
case 2: // 16bit
samples[index++] = (double)br.ReadInt16() / 32768D;
break;
}
if (index != samples.Length)
continue;
/****** The sample buffer is full and we must process it. ******/
if (AudioProcessor.IsSilence(ref samples))
{
silence_count++;
if (state == ParserState.Decoding && silence_count > 150)
{
// End of EWS broadcast reached.
EwsSignalParser.Parse(ews_bits.ToString());
/* ... reset state; go back looking for silence... */
}
goto Done;
}
/****** The signal was not silence. ******/
if (silence_count > 120 && state == ParserState.SearchingSilence)
state = ParserState.Decoding;
if (state == ParserState.Decoding)
{
AudioProcessor.Decode(ref samples, sampleRate, ref ews_bits);
bool continue_decoding = /* check first 20 bits for signature */;
if (continue_decoding) goto Done;
// If we get here, we were decoding a junk signal.
state = ParserState.SearchingSilence;
}
/* Not enough silence yet */
silence_count = 0;
Done:
index = 0;
}
The audio processor is just a class with:
public static void Decode(ref double[] samples, int sampleRate, ref StringBuilder bitHolder)
{
double freq_640 = GoertzelMagnitude(ref samples, 640, sampleRate);
double freq_1024 = GoertzelMagnitude(ref samples, 1024, sampleRate);
if (freq_640 > freq_1024)
bitHolder.Append("0");
else
bitHolder.Append("1");
}
public static bool IsSilence(ref double[] samples)
{
// power_RMS = sqrt(sum(x^2) / N)
double sum = 0;
for (int i = 0; i < samples.Length; i++)
sum += samples[i] * samples[i];
double power_RMS = Math.Sqrt(sum / samples.Length);
return power_RMS < 0.01;
}
/// <remarks>http://www.embedded.com/design/embedded/4024443/The-Goertzel-Algorithm</remarks>
private static double GoertzelMagnitude(ref double[] samples, double targetFrequency, int sampleRate)
{
double n = samples.Length;
int k = (int)(0.5D + ((double)n * targetFrequency) / (double)sampleRate);
double w = (2.0D * Math.PI / n) * k;
double cosine = Math.Cos(w);
double sine = Math.Sin(w);
double coeff = 2.0D * cosine;
double q0 = 0, q1 = 0, q2 = 0;
for (int i = 0; i < samples.Length; i++)
{
double sample = samples[i];
q0 = coeff * q1 - q2 + sample;
q2 = q1;
q1 = q0;
}
double magnitude = Math.Sqrt(q1 * q1 + q2 * q2 - q1 * q2 * coeff);
return magnitude;
}
Thanks for reading. I hope you can help me.

This is how I would do it (high level description)
Run your signal through a FFT
look for steady peaks at about 640Hz+1024Hz (I would say at least +/- 10Hz)
if the signal is steady for about 10 ms (with steady I mean about 95% of the samples are in the same range 640Hz+/-10Hz (or 1024Hz+/-10Hz) take it as a detection of the tone. Use this detection also to synchronize your timer that tells you when to expect the next tone.

I got it about 90% working now after rewriting the sample parsing loop and silence detection parts. There were two main problems in my implementation. The first was that the silence detector was overeager, so I changed it from processing every millisecond of samples to every half-millisecond of samples. That brought me exactly to the start of FSK data.
The next problem was that I then thought I could naively let the demodulator look at 15.625ms of samples as it works itself through the WAV file. It turns out that while this works great for the first 90 bits or so, eventually tones become a little longer or shorter than expected and the demodulator goes out of sync. The current code finds and corrects 13 bits with such a timing mismatch. Particularly vulnerable to this are spots where the signal changes from mark to space and vice versa.
Guess there's a reason the word "analog" contains "anal". It is. I really wish I knew more about signal theory and digital signal processing. :(
How I discovered all of this: I imported the MP3 and trimmed it down to the FSK part using Audacity. Then I had Audacity generate labels for every bit. After that I went through highlighting bits according toe the labels.

Numerically solving an equation

Algorithm to be coded in C#:
fn = f(xn)
f′n = df(xn)/dx
∆xn = -fn / f′n
Update: xn+1 = xn + ∆xn
Repeat the process until ∆xn ≤ e
I must use the Newton-Raphson method to solve but I do not know how to do a loop that puts in the next answer each time. How do I compute this?
This is my broken code
double a = 1, Lspan = 30, Lcable = 33, fn, fdn, dfn, j;
fn = (2 * a * (Math.Sinh(Lspan / 2 * a))) - Lcable;
fdn = (2 * (Math.Sinh(Lspan / 2 * a)) - ((Lspan / 2 * a) * Math.Cosh(Lspan / 2 * a)));
dfn = -fn / fdn;
do
j = a + dfn;
while (dfn > 0.00000000001);
Console.WriteLine( " {0} ",j) ;
Console.ReadKey();

Your loop performs the same calculation each time, because neither a or dfn change between iterations. I'm sure I've actually implemented a Newton-Raphson method myself years ago, but I don't remember enough about it to check that your arithmetic is correct without looking it up.
I expect that you intended fdn and dfn to be updated on each iteration - although your pseudocode statement of the method is ambiguous since it implies that only the whole solution is updated on each iteration, whereas actually each term needs to be updated or you'll just keep adding the starting value of ∆xn forever. I think the solution is to move the second, third and fourth lines inside the loop.
Does this make sense?
(It looks as though you were expecting C# to work with symbolic mathematics, which isn't the case. C# is basically procedural within the body of a method, so making an assignment statement fn = some terms; happens once, when the program hits that line. There is no knowledge built into that variable of how it was calculated, it's just a box with a number in it.)