I've been reading a lot about floating-point determinism in .NET, i.e. ensuring that the same code with the same inputs will give the same results across different machines. Since .NET lacks options like Java's fpstrict and MSVC's fp:strict, the consensus seems to be that there is no way around this issue using pure managed code. The C# game AI Wars has settled on using Fixed-point math instead, but this is a cumbersome solution.
The main issue appears to be that the CLR allows intermediate results to live in FPU registers that have higher precision than the type's native precision, leading to impredictably higher precision results. An MSDN article by CLR engineer David Notario explains the following:
Note that with current spec, it’s still a language choice to give
‘predictability’. The language may insert conv.r4 or conv.r8
instructions after every FP operation to get a ‘predictable’ behavior.
Obviously, this is really expensive, and different languages have
different compromises. C#, for example, does nothing, if you want
narrowing, you will have to insert (float) and (double) casts by hand.
This suggests that one may achieve floating-point determinism simply by inserting explicit casts for every expression and sub-expression that evaluates to float. One might write a wrapper type around float to automate this task. This would be a simple and ideal solution!
Other comments however suggest that it isn't so simple. Eric Lippert recently stated (emphasis mine):
in some version of the runtime, casting to float explicitly gives a
different result than not doing so. When you explicitly cast to float,
the C# compiler gives a hint to the runtime to say "take this thing
out of extra high precision mode if you happen to be using this
optimization".
Just what is this "hint" to the runtime? Does the C# spec stipulate that an explicit cast to float causes the insertion of a conv.r4 in the IL? Does the CLR spec stipulate that a conv.r4 instruction causes a value to be narrowed down to its native size? Only if both of these are true can we rely on explicit casts to provide floating point "predictability" as explained by David Notario.
Finally, even if we can indeed coerce all intermediate results to the type's native size, is this enough to guarantee reproducibility across machines, or are there other factors like FPU/SSE run-time settings?
Just what is this "hint" to the runtime?
As you conjecture, the compiler tracks whether a conversion to double or float was actually present in the source code, and if it was, it always inserts the appropriate conv opcode.
Does the C# spec stipulate that an explicit cast to float causes the insertion of a conv.r4 in the IL?
No, but I assure you that there are unit tests in the compiler test cases that ensure that it does. Though the specification does not demand it, you can rely on this behaviour.
The specification's only comment is that any floating point operation may be done in a higher precision than required at the whim of the runtime, and that this can make your results unexpectedly more accurate. See section 4.1.6.
Does the CLR spec stipulate that a conv.r4 instruction causes a value to be narrowed down to its native size?
Yes, in Partition I, section 12.1.3, which I note you could have looked up yourself rather than asking the internet to do it for you. These specifications are free on the web.
A question you didn't ask but probably should have:
Is there any operation other than casting that truncates floats out of high precision mode?
Yes. Assigning to a static field, instance field or element of a double[] or float[] array truncates.
Is consistent truncation enough to guarantee reproducibility across machines?
No. I encourage you to read section 12.1.3, which has much interesting to say on the subject of denormals and NaNs.
And finally, another question you did not ask but probably should have:
How can I guarantee reproducible arithmetic?
Use integers.
The 8087 Floating Point Unit chip design was Intel's billion dollar mistake. The idea looks good on paper, give it an 8 register stack that stores values in extended precision, 80 bits. So that you can write calculations whose intermediate values are less likely to lose significant digits.
The beast is however impossible to optimize for. Storing a value from the FPU stack back to memory is expensive. So keeping them inside the FPU is a strong optimization goal. Inevitable, having only 8 registers is going to require a write-back if the calculation is deep enough. It is also implemented as a stack, not freely addressable registers so that requires gymnastics as well that may produce a write-back. Inevitably a write back will truncate the value from 80-bits back to 64-bits, losing precision.
So consequences are that non-optimized code does not produce the same result as optimized code. And small changes to the calculation can have big effects on the result when an intermediate value ends up needing to be written back. The /fp:strict option is a hack around that, it forces the code generator to emit a write-back to keep the values consistent, but with the inevitable and considerable loss of perf.
This is a complete rock and a hard place. For the x86 jitter they just didn't try to address the problem.
Intel didn't make the same mistake when they designed the SSE instruction set. The XMM registers are freely addressable and don't store extra bits. If you want consistent results then compiling with the AnyCPU target, and a 64-bit operating system, is the quick solution. The x64 jitter uses SSE instead of FPU instructions for floating point math. Albeit that this added a third way that a calculation can produce a different result. If the calculation is wrong because it loses too many significant digits then it will be consistently wrong. Which is a bit of a bromide, really, but typically only as far as a programmer looks.
Related
No, this is not another "Why is (1/3.0)*3 != 1" question.
I've been reading about floating-points a lot lately; specifically, how the same calculation might give different results on different architectures or optimization settings.
This is a problem for video games which store replays, or are peer-to-peer networked (as opposed to server-client), which rely on all clients generating exactly the same results every time they run the program - a small discrepancy in one floating-point calculation can lead to a drastically different game-state on different machines (or even on the same machine!)
This happens even amongst processors that "follow" IEEE-754, primarily because some processors (namely x86) use double extended precision. That is, they use 80-bit registers to do all the calculations, then truncate to 64- or 32-bits, leading to different rounding results than machines which use 64- or 32- bits for the calculations.
I've seen several solutions to this problem online, but all for C++, not C#:
Disable double extended-precision mode (so that all double calculations use IEEE-754 64-bits) using _controlfp_s (Windows), _FPU_SETCW (Linux?), or fpsetprec (BSD).
Always run the same compiler with the same optimization settings, and require all users to have the same CPU architecture (no cross-platform play). Because my "compiler" is actually the JIT, which may optimize differently every time the program is run, I don't think this is possible.
Use fixed-point arithmetic, and avoid float and double altogether. decimal would work for this purpose, but would be much slower, and none of the System.Math library functions support it.
So, is this even a problem in C#? What if I only intend to support Windows (not Mono)?
If it is, is there any way to force my program to run at normal double-precision?
If not, are there any libraries that would help keep floating-point calculations consistent?
I know of no way to way to make normal floating points deterministic in .net. The JITter is allowed to create code that behaves differently on different platforms(or between different versions of .net). So using normal floats in deterministic .net code is not possible.
The workarounds I considered:
Implement FixedPoint32 in C#. While this is not too hard(I have a half finished implementation) the very small range of values makes it annoying to use. You have to be careful at all times so you neither overflow, nor lose too much precision. In the end I found this not easier than using integers directly.
Implement FixedPoint64 in C#. I found this rather hard to do. For some operations intermediate integers of 128bit would be useful. But .net doesn't offer such a type.
Implement a custom 32 bit floatingpoint. The lack of a BitScanReverse intrinsic causes a few annoyances when implementing this. But currently I think this is the most promising path.
Use native code for the math operations. Incurs the overhead of a delegate call on every math operation.
I've just started a software implementation of 32 bit floating point math. It can do about 70million additions/multiplications per second on my 2.66GHz i3.
https://github.com/CodesInChaos/SoftFloat . Obviously it's still very incomplete and buggy.
The C# specification (§4.1.6 Floating point types) specifically allows floating point computations to be done using precision higher than that of the result. So, no, I don't think you can make those calculations deterministic directly in .Net. Others suggested various workarounds, so you could try them.
The following page may be useful in the case where you need absolute portability of such operations. It discusses software for testing implementations of the IEEE 754 standard, including software for emulating floating point operations. Most information is probably specific to C or C++, however.
http://www.math.utah.edu/~beebe/software/ieee/
A note on fixed point
Binary fixed point numbers can also work well as a substitute for floating point, as is evident from the four basic arithmetic operations:
Addition and subtraction are trivial. They work the same way as integers. Just add or subtract!
To multiply two fixed point numbers, multiply the two numbers then shift right the defined number of fractional bits.
To divide two fixed point numbers, shift the dividend left the defined number of fractional bits, then divide by the divisor.
Chapter four of Hattangady (2007) has additional guidance on implementing binary fixed point numbers (S.K. Hattangady, "Development of a Block Floating Point Interval ALU for DSP and Control Applications", Master's thesis, North Carolina State University, 2007).
Binary fixed point numbers can be implemented on any integer data type such as int, long, and BigInteger, and the non-CLS-compliant types uint and ulong.
As suggested in another answer, you can use lookup tables, where each element in the table is a binary fixed point number, to help implement complex functions such as sine, cosine, square root, and so on. If the lookup table is less granular than the fixed point number, it is suggested to round the input by adding one half of the granularity of the lookup table to the input:
// Assume each number has a 12 bit fractional part. (1/4096)
// Each entry in the lookup table corresponds to a fixed point number
// with an 8-bit fractional part (1/256)
input+=(1<<3); // Add 2^3 for rounding purposes
input>>=4; // Shift right by 4 (to get 8-bit fractional part)
// --- clamp or restrict input here --
// Look up value.
return lookupTable[input];
Is this a problem for C#?
Yes. Different architectures are the least of your worries, different framerates etc. can lead to deviations due to inaccuracies in float representations - even if they are the same inaccuracies (e.g. same architecture, except a slower GPU on one machine).
Can I use System.Decimal?
There is no reason you can't, however it's dog slow.
Is there a way to force my program to run in double precision?
Yes. Host the CLR runtime yourself; and compile in all the nessecary calls/flags (that change the behaviour of floating point arithmetic) into the C++ application before calling CorBindToRuntimeEx.
Are there any libraries that would help keep floating point calculations consistent?
Not that I know of.
Is there another way to solve this?
I have tackled this problem before, the idea is to use QNumbers. They are a form of reals that are fixed-point; but not fixed point in base-10 (decimal) - rather base-2 (binary); because of this the mathematical primitives on them (add, sub, mul, div) are much faster than the naive base-10 fixed points; especially if n is the same for both values (which in your case it would be). Furthermore because they are integral they have well-defined results on every platform.
Keep in mind that framerate can still affect these, but it is not as bad and is easily rectified using syncronisation points.
Can I use more mathematical functions with QNumbers?
Yes, round-trip a decimal to do this. Furthermore, you should really be using lookup tables for the trig (sin, cos) functions; as those can really give different results on different platforms - and if you code them correctly they can use QNumbers directly.
According to this slightly old MSDN blog entry the JIT will not use SSE/SSE2 for floating point, it's all x87. Because of that, as you mentioned you have to worry about modes and flags, and in C# that's not possible to control. So using normal floating point operations will not guarantee the exact same result on every machine for your program.
To get precise reproducibility of double precision you are going to have to do software floating point (or fixed point) emulation. I don't know of C# libraries to do this.
Depending on the operations you need, you might be able to get away with single precision. Here's the idea:
store all values you care about in single precision
to perform an operation:
expand inputs to double precision
do operation in double precision
convert result back to single precision
The big issue with x87 is that calculations might be done in 53-bit or 64-bit accuracy depending on the precision flag and whether the register spilled to memory. But for many operations, performing the operation in high precision and rounding back to lower precision will guarantee the correct answer, which implies that the answer will be guaranteed to be the same on all systems. Whether you get the extra precision won't matter, since you have enough precision to guarantee the right answer in either case.
Operations that should work in this scheme: addition, subtraction, multiplication, division, sqrt. Things like sin, exp, etc. won't work (results will usually match but there is no guarantee). "When is double rounding innocuous?" ACM Reference (paid reg. req.)
Hope this helps!
As already stated by other answers:
Yes, this is a problem in C# - even when staying pure Windows.
As for a solution:
You can reduce (and with some effort/performance hit) avoid the problem completely if you use built-in BigInteger class and scaling all calculations to a defined precision by using a common denominator for any calculation/storage of such numbers.
As requested by OP - regarding performance:
System.Decimal represents number with 1 bit for a sign and 96 bit Integer and a "scale" (representing where the decimal point is). For all calculations you make it must operate on this data structure and can't use any floating point instructions built into the CPU.
The BigInteger "solution" does something similar - only that you can define how much digits you need/want... perhaps you want only 80 bits or 240 bits of precision.
The slowness comes always from having to simulate all operations on these number via integer-only instructions without using the CPU/FPU-built-in instructions which in turn leads to much more instructions per mathematical operation.
To lessen the performance hit there are several strategies - like QNumbers (see answer from Jonathan Dickinson - Is floating-point math consistent in C#? Can it be?) and/or caching (for example trig calculations...) etc.
Well, here would be my first attempt on how to do this:
Create an ATL.dll project that has a simple object in it to be used for your critical floating point operations. make sure to compile it with flags that disable using any non xx87 hardware to do floating point.
Create functions that call floating point operations and return the results; start simple and then if it's working for you, you can always increase the complexity to meet your performance needs later if necessary.
Put the control_fp calls around the actual math to ensure that it's done the same way on all machines.
Reference your new library and test to make sure it works as expected.
(I believe you can just compile to a 32-bit .dll and then use it with either x86 or AnyCpu [or likely only targeting x86 on a 64-bit system; see comment below].)
Then, assuming it works, should you want to use Mono I imagine you should be able to replicate the library on other x86 platforms in a similar manner (not COM of course; although, perhaps, with wine? a little out of my area once we go there though...).
Assuming you can make it work, you should be able to set up custom functions that can do multiple operations at once to fix any performance issues, and you'll have floating point math that allows you to have consistent results across platforms with a minimal amount of code written in C++, and leaving the rest of your code in C#.
I'm not a game developer, though I do have a lot of experience with computationally difficult problems ... so, I'll do my best.
The strategy I would adopt is essentially this:
Use a slower (if necessary; if there's a faster way, great!), but predictable method to get reproducible results
Use double for everything else (eg, rendering)
The short'n long of this is: you need to find a balance. If you're spending 30ms rendering (~33fps) and only 1ms doing collision detection (or insert some other highly sensitive operation) -- even if you triple the time it takes to do the critical arithmetic, the impact it has on your framerate is you drop from 33.3fps to 30.3fps.
I suggest you profile everything, account for how much time is spent doing each of the noticeably expensive calculations, then repeat the measurements with 1 or more methods of resolving this problem and see what the impact is.
Checking the links in the other answers make it clear you'll never have a guarantee of whether floating point is "correctly" implemented or whether you'll always receive a certain precision for a given calculation, but perhaps you could make a best effort by (1) truncating all calculations to a common minimum (eg, if different implementations will give you 32 to 80 bits of precision, always truncating every operation to 30 or 31 bits), (2) have a table of a few test cases at startup (borderline cases of add, subtract, multiply, divide, sqrt, cosine, etc.) and if the implementation calculates values matching the table then not bother making any adjustments.
Your question in quite difficult and technical stuff O_o. However I may have an idea.
You sure know that the CPU makes some adjustment after any floating operations.
And CPU offer several different instructions which make different rounding operation.
So for an expression, your compiler will choose a set of instructions which lead you to a result. But any other instruction workflow, even if they intend to compute the same expression, can provide another result.
The 'mistakes' made by a rounding adjustment will grow at each further instructions.
As an exemple we can say that at an assembly level: a * b * c is not equivalent to a * c * b.
I'm not entirely sure of that, you will need to ask for someone who know CPU architecture a lot more than me : p
However to answer your question: in C or C++ you can solve your problem because you have some control on the machine code generate by your compiler, however in .NET you don't have any. So as long as your machine code can be different, you'll never be sure about the exact result.
I'm curious in which way this can be a problem because variation seems very minimal, but if you need really accurate operation the only solution I can think about will be to increase the size of your floating registers. Use double precision or even long double if you can (not sure that's possible using CLI).
I hope I've been clear enough, I'm not perfect in English (...at all : s)
I've been reading a lot about floating-point determinism in .NET, i.e. ensuring that the same code with the same inputs will give the same results across different machines. Since .NET lacks options like Java's fpstrict and MSVC's fp:strict, the consensus seems to be that there is no way around this issue using pure managed code. The C# game AI Wars has settled on using Fixed-point math instead, but this is a cumbersome solution.
The main issue appears to be that the CLR allows intermediate results to live in FPU registers that have higher precision than the type's native precision, leading to impredictably higher precision results. An MSDN article by CLR engineer David Notario explains the following:
Note that with current spec, it’s still a language choice to give
‘predictability’. The language may insert conv.r4 or conv.r8
instructions after every FP operation to get a ‘predictable’ behavior.
Obviously, this is really expensive, and different languages have
different compromises. C#, for example, does nothing, if you want
narrowing, you will have to insert (float) and (double) casts by hand.
This suggests that one may achieve floating-point determinism simply by inserting explicit casts for every expression and sub-expression that evaluates to float. One might write a wrapper type around float to automate this task. This would be a simple and ideal solution!
Other comments however suggest that it isn't so simple. Eric Lippert recently stated (emphasis mine):
in some version of the runtime, casting to float explicitly gives a
different result than not doing so. When you explicitly cast to float,
the C# compiler gives a hint to the runtime to say "take this thing
out of extra high precision mode if you happen to be using this
optimization".
Just what is this "hint" to the runtime? Does the C# spec stipulate that an explicit cast to float causes the insertion of a conv.r4 in the IL? Does the CLR spec stipulate that a conv.r4 instruction causes a value to be narrowed down to its native size? Only if both of these are true can we rely on explicit casts to provide floating point "predictability" as explained by David Notario.
Finally, even if we can indeed coerce all intermediate results to the type's native size, is this enough to guarantee reproducibility across machines, or are there other factors like FPU/SSE run-time settings?
Just what is this "hint" to the runtime?
As you conjecture, the compiler tracks whether a conversion to double or float was actually present in the source code, and if it was, it always inserts the appropriate conv opcode.
Does the C# spec stipulate that an explicit cast to float causes the insertion of a conv.r4 in the IL?
No, but I assure you that there are unit tests in the compiler test cases that ensure that it does. Though the specification does not demand it, you can rely on this behaviour.
The specification's only comment is that any floating point operation may be done in a higher precision than required at the whim of the runtime, and that this can make your results unexpectedly more accurate. See section 4.1.6.
Does the CLR spec stipulate that a conv.r4 instruction causes a value to be narrowed down to its native size?
Yes, in Partition I, section 12.1.3, which I note you could have looked up yourself rather than asking the internet to do it for you. These specifications are free on the web.
A question you didn't ask but probably should have:
Is there any operation other than casting that truncates floats out of high precision mode?
Yes. Assigning to a static field, instance field or element of a double[] or float[] array truncates.
Is consistent truncation enough to guarantee reproducibility across machines?
No. I encourage you to read section 12.1.3, which has much interesting to say on the subject of denormals and NaNs.
And finally, another question you did not ask but probably should have:
How can I guarantee reproducible arithmetic?
Use integers.
The 8087 Floating Point Unit chip design was Intel's billion dollar mistake. The idea looks good on paper, give it an 8 register stack that stores values in extended precision, 80 bits. So that you can write calculations whose intermediate values are less likely to lose significant digits.
The beast is however impossible to optimize for. Storing a value from the FPU stack back to memory is expensive. So keeping them inside the FPU is a strong optimization goal. Inevitable, having only 8 registers is going to require a write-back if the calculation is deep enough. It is also implemented as a stack, not freely addressable registers so that requires gymnastics as well that may produce a write-back. Inevitably a write back will truncate the value from 80-bits back to 64-bits, losing precision.
So consequences are that non-optimized code does not produce the same result as optimized code. And small changes to the calculation can have big effects on the result when an intermediate value ends up needing to be written back. The /fp:strict option is a hack around that, it forces the code generator to emit a write-back to keep the values consistent, but with the inevitable and considerable loss of perf.
This is a complete rock and a hard place. For the x86 jitter they just didn't try to address the problem.
Intel didn't make the same mistake when they designed the SSE instruction set. The XMM registers are freely addressable and don't store extra bits. If you want consistent results then compiling with the AnyCPU target, and a 64-bit operating system, is the quick solution. The x64 jitter uses SSE instead of FPU instructions for floating point math. Albeit that this added a third way that a calculation can produce a different result. If the calculation is wrong because it loses too many significant digits then it will be consistently wrong. Which is a bit of a bromide, really, but typically only as far as a programmer looks.
I have a function in C# that returns the following:
...
float amount = smallestPercentage * (float)quantity;
return (int)amount;
Now I know I am suppose to use Convert.Int32(amount) rather than type cast an int, and that has fixed the problem. But my problem was really this...
When developing my program at home (Windows Vista) I would get a return value of 1, but when a deployed the program to another environment (Windows XP), I would get a return value of 0.
I was wondering if this has to do with the Windows version, the .NET version or even the CPU processor?
Thanks.
David
In fact, you can get different results:
on different machines
depending on whether you compiled with debug or retail build settings
depending on whether you did the math in compile-time constants or runtime values
how local variables and other temporary values are used in your method
and so on.
The C# specification calls out that floating point arithmetic may be done in higher precision than you expect. It's never done in lower precision than you'd expect, but we reserve the right to use higher-precision algorithms on certain hardware and with certain optimizations available, any time the jitter thinks that it can get away with it. That means that in operations that are highly sensitive to small changes in precision -- like rounding -- can give very different results seemingly without explanation.
You are not the first Stack Overflow user to discover this fact: Problem converting from int to float
When dealing with floating point numbers its really advisable that you use some sort of rounding routine. In C#, I believe the best approach is the Math.Round method.
As to why its happening, different processors have different routines for computing floats and doubles. On your target machine you're likely getting a value slightly below 1 (say, .999987) that when casted gets turned into 0. Floats have been around since the CLR was created, so this is most likely a processor specific thing. OSes very rarely interfere with direct application code.
No, this is not another "Why is (1/3.0)*3 != 1" question.
I've been reading about floating-points a lot lately; specifically, how the same calculation might give different results on different architectures or optimization settings.
This is a problem for video games which store replays, or are peer-to-peer networked (as opposed to server-client), which rely on all clients generating exactly the same results every time they run the program - a small discrepancy in one floating-point calculation can lead to a drastically different game-state on different machines (or even on the same machine!)
This happens even amongst processors that "follow" IEEE-754, primarily because some processors (namely x86) use double extended precision. That is, they use 80-bit registers to do all the calculations, then truncate to 64- or 32-bits, leading to different rounding results than machines which use 64- or 32- bits for the calculations.
I've seen several solutions to this problem online, but all for C++, not C#:
Disable double extended-precision mode (so that all double calculations use IEEE-754 64-bits) using _controlfp_s (Windows), _FPU_SETCW (Linux?), or fpsetprec (BSD).
Always run the same compiler with the same optimization settings, and require all users to have the same CPU architecture (no cross-platform play). Because my "compiler" is actually the JIT, which may optimize differently every time the program is run, I don't think this is possible.
Use fixed-point arithmetic, and avoid float and double altogether. decimal would work for this purpose, but would be much slower, and none of the System.Math library functions support it.
So, is this even a problem in C#? What if I only intend to support Windows (not Mono)?
If it is, is there any way to force my program to run at normal double-precision?
If not, are there any libraries that would help keep floating-point calculations consistent?
I know of no way to way to make normal floating points deterministic in .net. The JITter is allowed to create code that behaves differently on different platforms(or between different versions of .net). So using normal floats in deterministic .net code is not possible.
The workarounds I considered:
Implement FixedPoint32 in C#. While this is not too hard(I have a half finished implementation) the very small range of values makes it annoying to use. You have to be careful at all times so you neither overflow, nor lose too much precision. In the end I found this not easier than using integers directly.
Implement FixedPoint64 in C#. I found this rather hard to do. For some operations intermediate integers of 128bit would be useful. But .net doesn't offer such a type.
Implement a custom 32 bit floatingpoint. The lack of a BitScanReverse intrinsic causes a few annoyances when implementing this. But currently I think this is the most promising path.
Use native code for the math operations. Incurs the overhead of a delegate call on every math operation.
I've just started a software implementation of 32 bit floating point math. It can do about 70million additions/multiplications per second on my 2.66GHz i3.
https://github.com/CodesInChaos/SoftFloat . Obviously it's still very incomplete and buggy.
The C# specification (§4.1.6 Floating point types) specifically allows floating point computations to be done using precision higher than that of the result. So, no, I don't think you can make those calculations deterministic directly in .Net. Others suggested various workarounds, so you could try them.
The following page may be useful in the case where you need absolute portability of such operations. It discusses software for testing implementations of the IEEE 754 standard, including software for emulating floating point operations. Most information is probably specific to C or C++, however.
http://www.math.utah.edu/~beebe/software/ieee/
A note on fixed point
Binary fixed point numbers can also work well as a substitute for floating point, as is evident from the four basic arithmetic operations:
Addition and subtraction are trivial. They work the same way as integers. Just add or subtract!
To multiply two fixed point numbers, multiply the two numbers then shift right the defined number of fractional bits.
To divide two fixed point numbers, shift the dividend left the defined number of fractional bits, then divide by the divisor.
Chapter four of Hattangady (2007) has additional guidance on implementing binary fixed point numbers (S.K. Hattangady, "Development of a Block Floating Point Interval ALU for DSP and Control Applications", Master's thesis, North Carolina State University, 2007).
Binary fixed point numbers can be implemented on any integer data type such as int, long, and BigInteger, and the non-CLS-compliant types uint and ulong.
As suggested in another answer, you can use lookup tables, where each element in the table is a binary fixed point number, to help implement complex functions such as sine, cosine, square root, and so on. If the lookup table is less granular than the fixed point number, it is suggested to round the input by adding one half of the granularity of the lookup table to the input:
// Assume each number has a 12 bit fractional part. (1/4096)
// Each entry in the lookup table corresponds to a fixed point number
// with an 8-bit fractional part (1/256)
input+=(1<<3); // Add 2^3 for rounding purposes
input>>=4; // Shift right by 4 (to get 8-bit fractional part)
// --- clamp or restrict input here --
// Look up value.
return lookupTable[input];
Is this a problem for C#?
Yes. Different architectures are the least of your worries, different framerates etc. can lead to deviations due to inaccuracies in float representations - even if they are the same inaccuracies (e.g. same architecture, except a slower GPU on one machine).
Can I use System.Decimal?
There is no reason you can't, however it's dog slow.
Is there a way to force my program to run in double precision?
Yes. Host the CLR runtime yourself; and compile in all the nessecary calls/flags (that change the behaviour of floating point arithmetic) into the C++ application before calling CorBindToRuntimeEx.
Are there any libraries that would help keep floating point calculations consistent?
Not that I know of.
Is there another way to solve this?
I have tackled this problem before, the idea is to use QNumbers. They are a form of reals that are fixed-point; but not fixed point in base-10 (decimal) - rather base-2 (binary); because of this the mathematical primitives on them (add, sub, mul, div) are much faster than the naive base-10 fixed points; especially if n is the same for both values (which in your case it would be). Furthermore because they are integral they have well-defined results on every platform.
Keep in mind that framerate can still affect these, but it is not as bad and is easily rectified using syncronisation points.
Can I use more mathematical functions with QNumbers?
Yes, round-trip a decimal to do this. Furthermore, you should really be using lookup tables for the trig (sin, cos) functions; as those can really give different results on different platforms - and if you code them correctly they can use QNumbers directly.
According to this slightly old MSDN blog entry the JIT will not use SSE/SSE2 for floating point, it's all x87. Because of that, as you mentioned you have to worry about modes and flags, and in C# that's not possible to control. So using normal floating point operations will not guarantee the exact same result on every machine for your program.
To get precise reproducibility of double precision you are going to have to do software floating point (or fixed point) emulation. I don't know of C# libraries to do this.
Depending on the operations you need, you might be able to get away with single precision. Here's the idea:
store all values you care about in single precision
to perform an operation:
expand inputs to double precision
do operation in double precision
convert result back to single precision
The big issue with x87 is that calculations might be done in 53-bit or 64-bit accuracy depending on the precision flag and whether the register spilled to memory. But for many operations, performing the operation in high precision and rounding back to lower precision will guarantee the correct answer, which implies that the answer will be guaranteed to be the same on all systems. Whether you get the extra precision won't matter, since you have enough precision to guarantee the right answer in either case.
Operations that should work in this scheme: addition, subtraction, multiplication, division, sqrt. Things like sin, exp, etc. won't work (results will usually match but there is no guarantee). "When is double rounding innocuous?" ACM Reference (paid reg. req.)
Hope this helps!
As already stated by other answers:
Yes, this is a problem in C# - even when staying pure Windows.
As for a solution:
You can reduce (and with some effort/performance hit) avoid the problem completely if you use built-in BigInteger class and scaling all calculations to a defined precision by using a common denominator for any calculation/storage of such numbers.
As requested by OP - regarding performance:
System.Decimal represents number with 1 bit for a sign and 96 bit Integer and a "scale" (representing where the decimal point is). For all calculations you make it must operate on this data structure and can't use any floating point instructions built into the CPU.
The BigInteger "solution" does something similar - only that you can define how much digits you need/want... perhaps you want only 80 bits or 240 bits of precision.
The slowness comes always from having to simulate all operations on these number via integer-only instructions without using the CPU/FPU-built-in instructions which in turn leads to much more instructions per mathematical operation.
To lessen the performance hit there are several strategies - like QNumbers (see answer from Jonathan Dickinson - Is floating-point math consistent in C#? Can it be?) and/or caching (for example trig calculations...) etc.
Well, here would be my first attempt on how to do this:
Create an ATL.dll project that has a simple object in it to be used for your critical floating point operations. make sure to compile it with flags that disable using any non xx87 hardware to do floating point.
Create functions that call floating point operations and return the results; start simple and then if it's working for you, you can always increase the complexity to meet your performance needs later if necessary.
Put the control_fp calls around the actual math to ensure that it's done the same way on all machines.
Reference your new library and test to make sure it works as expected.
(I believe you can just compile to a 32-bit .dll and then use it with either x86 or AnyCpu [or likely only targeting x86 on a 64-bit system; see comment below].)
Then, assuming it works, should you want to use Mono I imagine you should be able to replicate the library on other x86 platforms in a similar manner (not COM of course; although, perhaps, with wine? a little out of my area once we go there though...).
Assuming you can make it work, you should be able to set up custom functions that can do multiple operations at once to fix any performance issues, and you'll have floating point math that allows you to have consistent results across platforms with a minimal amount of code written in C++, and leaving the rest of your code in C#.
I'm not a game developer, though I do have a lot of experience with computationally difficult problems ... so, I'll do my best.
The strategy I would adopt is essentially this:
Use a slower (if necessary; if there's a faster way, great!), but predictable method to get reproducible results
Use double for everything else (eg, rendering)
The short'n long of this is: you need to find a balance. If you're spending 30ms rendering (~33fps) and only 1ms doing collision detection (or insert some other highly sensitive operation) -- even if you triple the time it takes to do the critical arithmetic, the impact it has on your framerate is you drop from 33.3fps to 30.3fps.
I suggest you profile everything, account for how much time is spent doing each of the noticeably expensive calculations, then repeat the measurements with 1 or more methods of resolving this problem and see what the impact is.
Checking the links in the other answers make it clear you'll never have a guarantee of whether floating point is "correctly" implemented or whether you'll always receive a certain precision for a given calculation, but perhaps you could make a best effort by (1) truncating all calculations to a common minimum (eg, if different implementations will give you 32 to 80 bits of precision, always truncating every operation to 30 or 31 bits), (2) have a table of a few test cases at startup (borderline cases of add, subtract, multiply, divide, sqrt, cosine, etc.) and if the implementation calculates values matching the table then not bother making any adjustments.
Your question in quite difficult and technical stuff O_o. However I may have an idea.
You sure know that the CPU makes some adjustment after any floating operations.
And CPU offer several different instructions which make different rounding operation.
So for an expression, your compiler will choose a set of instructions which lead you to a result. But any other instruction workflow, even if they intend to compute the same expression, can provide another result.
The 'mistakes' made by a rounding adjustment will grow at each further instructions.
As an exemple we can say that at an assembly level: a * b * c is not equivalent to a * c * b.
I'm not entirely sure of that, you will need to ask for someone who know CPU architecture a lot more than me : p
However to answer your question: in C or C++ you can solve your problem because you have some control on the machine code generate by your compiler, however in .NET you don't have any. So as long as your machine code can be different, you'll never be sure about the exact result.
I'm curious in which way this can be a problem because variation seems very minimal, but if you need really accurate operation the only solution I can think about will be to increase the size of your floating registers. Use double precision or even long double if you can (not sure that's possible using CLI).
I hope I've been clear enough, I'm not perfect in English (...at all : s)
I have a function in C# that returns the following:
...
float amount = smallestPercentage * (float)quantity;
return (int)amount;
Now I know I am suppose to use Convert.Int32(amount) rather than type cast an int, and that has fixed the problem. But my problem was really this...
When developing my program at home (Windows Vista) I would get a return value of 1, but when a deployed the program to another environment (Windows XP), I would get a return value of 0.
I was wondering if this has to do with the Windows version, the .NET version or even the CPU processor?
Thanks.
David
In fact, you can get different results:
on different machines
depending on whether you compiled with debug or retail build settings
depending on whether you did the math in compile-time constants or runtime values
how local variables and other temporary values are used in your method
and so on.
The C# specification calls out that floating point arithmetic may be done in higher precision than you expect. It's never done in lower precision than you'd expect, but we reserve the right to use higher-precision algorithms on certain hardware and with certain optimizations available, any time the jitter thinks that it can get away with it. That means that in operations that are highly sensitive to small changes in precision -- like rounding -- can give very different results seemingly without explanation.
You are not the first Stack Overflow user to discover this fact: Problem converting from int to float
When dealing with floating point numbers its really advisable that you use some sort of rounding routine. In C#, I believe the best approach is the Math.Round method.
As to why its happening, different processors have different routines for computing floats and doubles. On your target machine you're likely getting a value slightly below 1 (say, .999987) that when casted gets turned into 0. Floats have been around since the CLR was created, so this is most likely a processor specific thing. OSes very rarely interfere with direct application code.