Develop Reference

c# Denormalized Floating Point: is "zero literal 0.0f" slow?

I just read about Denormalized floating point numbers, should i replace all zero literals with almost-zero literal to get better performance.
I am afraid that the evil zero constants in my could pollute my performance.
Example:
Program 1:
float a = 0.0f;
Console.WriteLine(a);
Program 2:
float b = 1.401298E-45f;
Console.WriteLine(b);
Shouldn't program 2 be 1.000.000 times faster than program 1 since b can be represented by ieee floating point representation in cannonized form ? whereas program 1 has to act with "zero" which is not directly representable.
If so the whole software development industry is flawed. A simple field declaration:
float c;
Would automatically initialize it to zero, Which would cause the dreaded performance hit.
Avoid the hustle mentioning "Premature Optimization is the..., blablabla".
Delayed Knowledge of Compilers Optimization Workings could result in the explosion of a nuclear factory. So i would like to know ahead what i am paying, so that i am safe to ignore optimizing it.
Ps. I don't care if float becomes denormalized by the result of a mathematical operation, i have no control in that, so i don't care.
Proof: x + 0.1f is 10 times faster than x + 0
Why does changing 0.1f to 0 slow down performance by 10x?
Question Synopsis: is 0.0f evil ? So all who used it as a constant are also evil?

There's nothing special about denormals that makes them inherently slower than normalized floating point numbers. In fact, a FP system which only supported denormals would be plenty fast, because it would essentially only be doing integer operations.
The slowness comes from the relative difficulty of certain operations when performed on a mix of normals and denormals. Adding a normal to a denormal is much trickier than adding a normal to a normal, or adding a denormal to a denormal. The machinery of computation is simply more involved, requires more steps. Because most of the time you're only operating on normals, it makes sense to optimize for that common case, and drop into the slower and more generalized normal/denormal implementation only when that doesn't work.
The exception to denormals being unusual, of course, is 0.0, which is a denormal with a zero mantissa. Because 0 is the sort of thing one often finds and does operations on, and because an operation involving a 0 is trivial, those are handled as part of the fast common case.
I think you've misunderstood what's going on in the answer to the question you linked. The 0 isn't by itself making things slow: despite being technically a denormal, operations on it are fast. The denormals in question are the ones stored in the y array after a sufficient number of loop iterations. The advantage of the 0.1 over the 0 is that, in that particular code snippet, it prevents numbers from becoming nonzero denormals, not that it's faster to add 0.1 than 0.0 (it isn't).

Why .Net supports 'infinity' value in floats? [duplicate]

I'm just curious, why in IEEE-754 any non zero float number divided by zero results in infinite value? It's a nonsense from the mathematical perspective. So I think that correct result for this operation is NaN.
Function f(x) = 1/x is not defined when x=0, if x is a real number. For example, function sqrt is not defined for any negative number and sqrt(-1.0f) if IEEE-754 produces a NaN value. But 1.0f/0 is Inf.
But for some reason this is not the case in IEEE-754. There must be a reason for this, maybe some optimization or compatibility reasons.
So what's the point?

It's a nonsense from the mathematical perspective.
Yes. No. Sort of.
The thing is: Floating-point numbers are approximations. You want to use a wide range of exponents and a limited number of digits and get results which are not completely wrong. :)
The idea behind IEEE-754 is that every operation could trigger "traps" which indicate possible problems. They are
Illegal (senseless operation like sqrt of negative number)
Overflow (too big)
Underflow (too small)
Division by zero (The thing you do not like)
Inexact (This operation may give you wrong results because you are losing precision)
Now many people like scientists and engineers do not want to be bothered with writing trap routines. So Kahan, the inventor of IEEE-754, decided that every operation should also return a sensible default value if no trap routines exist.
They are
NaN for illegal values
signed infinities for Overflow
signed zeroes for Underflow
NaN for indeterminate results (0/0) and infinities for (x/0 x != 0)
normal operation result for Inexact
The thing is that in 99% of all cases zeroes are caused by underflow and therefore in 99%
of all times Infinity is "correct" even if wrong from a mathematical perspective.

I'm not sure why you would believe this to be nonsense.
The simplistic definition of a / b, at least for non-zero b, is the unique number of bs that has to be subtracted from a before you get to zero.
Expanding that to the case where b can be zero, the number that has to be subtracted from any non-zero number to get to zero is indeed infinite, because you'll never get to zero.
Another way to look at it is to talk in terms of limits. As a positive number n approaches zero, the expression 1 / n approaches "infinity". You'll notice I've quoted that word because I'm a firm believer in not propagating the delusion that infinity is actually a concrete number :-)
NaN is reserved for situations where the number cannot be represented (even approximately) by any other value (including the infinities), it is considered distinct from all those other values.
For example, 0 / 0 (using our simplistic definition above) can have any amount of bs subtracted from a to reach 0. Hence the result is indeterminate - it could be 1, 7, 42, 3.14159 or any other value.
Similarly things like the square root of a negative number, which has no value in the real plane used by IEEE754 (you have to go to the complex plane for that), cannot be represented.

In mathematics, division by zero is undefined because zero has no sign, therefore two results are equally possible, and exclusive: negative infinity or positive infinity (but not both).
In (most) computing, 0.0 has a sign. Therefore we know what direction we are approaching from, and what sign infinity would have. This is especially true when 0.0 represents a non-zero value too small to be expressed by the system, as it frequently the case.
The only time NaN would be appropriate is if the system knows with certainty that the denominator is truly, exactly zero. And it can't unless there is a special way to designate that, which would add overhead.

NOTE:
I re-wrote this following a valuable comment from #Cubic.
I think the correct answer to this has to come from calculus and the notion of limits. Consider the limit of f(x)/g(x) as x->0 under the assumption that g(0) == 0. There are two broad cases that are interesting here:
If f(0) != 0, then the limit as x->0 is either plus or minus infinity, or it's undefined. If g(x) takes both signs in the neighborhood of x==0, then the limit is undefined (left and right limits don't agree). If g(x) has only one sign near 0, however, the limit will be defined and be either positive or negative infinity. More on this later.
If f(0) == 0 as well, then the limit can be anything, including positive infinity, negative infinity, a finite number, or undefined.
In the second case, generally speaking, you cannot say anything at all. Arguably, in the second case NaN is the only viable answer.
Now in the first case, why choose one particular sign when either is possible or it might be undefined? As a practical matter, it gives you more flexibility in cases where you do know something about the sign of the denominator, at relatively little cost in the cases where you don't. You may have a formula, for example, where you know analytically that g(x) >= 0 for all x, say, for example, g(x) = x*x. In that case the limit is defined and it's infinity with sign equal to the sign of f(0). You might want to take advantage of that as a convenience in your code. In other cases, where you don't know anything about the sign of g, you cannot generally take advantage of it, but the cost here is just that you need to trap for a few extra cases - positive and negative infinity - in addition to NaN if you want to fully error check your code. There is some price there, but it's not large compared to the flexibility gained in other cases.
Why worry about general functions when the question was about "simple division"? One common reason is that if you're computing your numerator and denominator through other arithmetic operations, you accumulate round-off errors. The presence of those errors can be abstracted into the general formula format shown above. For example f(x) = x + e, where x is the analytically correct, exact answer, e represents the error from round-off, and f(x) is the floating point number that you actually have on the machine at execution.

How do I detect total loss of precision with Doubles?

When I run the following code, I get 0 printed on both lines:
Double a = 9.88131291682493E-324;
Double b = a*0.1D;
Console.WriteLine(b);
Console.WriteLine(BitConverter.DoubleToInt64Bits(b));
I would expect to get Double.NaN if an operation result gets out of range. Instead I get 0. It looks that to be able to detect when this happens I have to check:
Before the operation check if any of the operands is zero
After the operation, if neither of operands were zero, check if the result is zero. If not let it run. If it is zero, assign Double.NaN to it instead to indicate that it's not really a zero, it's just a result that can't be represented within this variable.
That's rather unwieldy. Is there a better way? What Double.NaN is designed for? I'm assuming some operations must have return it, surely designers did not put it there just in case? Is it possible that this is a bug in BCL? (I know unlikely, but, that's why I'd like to understand how that Double.NaN is supposed to work)
Update
By the way, this problem is not specific for double. decimal exposes it all the same:
Decimal a = 0.0000000000000000000000000001m;
Decimal b = a* 0.1m;
Console.WriteLine(b);
That also gives zero.
In my case I need double, because I need the range they provide (I'm working on probabilistic calculations) and I'm not that worried about precision.
What I need though is to be able to detect when my results stop mean anything, that is when calculations drop the value so low, that it can no longer be presented by double.
Is there a practical way of detecting this?

Double works exactly according to the floating point numbers specification, IEEE 754. So no, it's not an error in BCL - it's just the way IEEE 754 floating points work.
The reason, of course, is that it's not what floats are designed for at all. Instead, you might want to use decimal, which is a precise decimal number, unlike float/double.
There's a few special values in floating point numbers, with different meanings:
Infinity - e.g. 1f / 0f.
-Infinity - e.g. -1f / 0f.
NaN - e.g. 0f / 0f or Math.Sqrt(-1)
However, as the commenters below noted, while decimal does in fact check for overflows, coming too close to zero is not considered an overflow, just like with floating point numbers. So if you really need to check for this, you will have to make your own * and / methods. With decimal numbers, you shouldn't really care, though.
If you need this kind of precision for multiplication and division (that is, you want your divisions to be reversible by multiplication), you should probably use rational numbers instead - two integers (big integers if necessary). And use a checked context - that will produce an exception on overflow.
IEEE 754 in fact does handle underflow. There's two problems:
The return value is 0 (or -1 for negative undreflow). The exception flag for underflow is set, but there's no way to get that in .NET.
This only occurs for the loss of precision when you get too close to zero. But you lost most of your precision way long before that. Whatever "precise" number you had is long gone - the operations are not reversible, and they are not precise.
So if you really do care about reversibility etc., stick to rational numbers. Neither decimal nor double will work, C# or not. If you're not that precise, you shouldn't care about underflows anyway - just pick the lowest reasonable number, and declare anything under that as "invalid"; may sure you're far away from the actual maximum precision - double.Epsilon will not help, obviously.

All you need is epsilon.
This is a "small number" which is small enough so you're no longer interested in.
You could use:
double epsilon = 1E-50;
and whenever one of your factors gets smaller than epislon you take action (for example treat it like 0.0)

margin of error trying to check if a point lays on line

Working with floating point values it is as easy as breathing to run on approximation errors by comparing quantities which should be the same. I want to know if there is a way built in some MSDN (or even external) library for c# to ignore the problem.
An example could be: more than comparing 2 float values like this
if(myVector3.X == anotherVector3.X)
I would appreciate something like this
if(myVector3.X.isInTheNeighbourhood(anotherVector3.X))
This is not well-written, I know. That is just to simplify the explaination. What I am exactly doing is checking if a point (Vector3) lays on line segment. So basically the calculations I make are nothing more than
(x - x1)/(x2 - x1) = (y - y1)/(y2 - y1) = (z - z1)/(z2 - z1)
But these values won't be always the same, so I need to write down some code which includes a sort of tolerance a sort of Neighbourhood mathematical concept to accept values close enought to the line.
Hope that I made myself clear.
Has anyone a solution for this?

I'm proposing the use of an exact predicate. This is an alternative approach to what you actually asked, but it might be worth considering.
Suppose your three points really are at locations indicated by their respective double precision coordinates. A simple double precision computation like the one suggested in your question will likely still return a wrong result. However, there are techniques to obtain exact results to detect these cases. One technique is by turning all numbers into arbitrary precision floating point (or integer) numbers and do most of the computation using integers under the hood. Another, which should work faster on modern hardware, expresses intermediate results as sums of doubles. E.g. when computing a+b, you obtain two resulting numbers, the first is the sum as you'd usually compute it, the other is a correction term to note down the error. In many cases, the bigger terms are sufficient to make a choice, which leads to the concept of adaptive precision.
All of this, including the application to geometric predicates, has been outlined nicely by Jonathan Richard Shewchuk in his page Adaptive Precision Floating-Point Arithmetic and Fast Robust Predicates for Computational Geometry. He has a paper on it, and some C code which should be possible to adapt to C#. Or perhaps to compile in C and link to C#, thus forming a mixed language project. Note however that it makes some assumptions on how the compiler treats floating point computations. In particular, you have to be careful that certain intermediate results don't have excess precision, like the 80-bit numbers stored in a 80387-style floating point unit. I've looked into this myself recently, and the easiest solution might be asking the compiler to use SSE instructions instead of x87 ones.
In your case, you are asking whether a point p lies on the segment connecting p1 and p2. One predicate which would be very much in the spirit of that paper would be the position of a fourth point in relation to a plane spanned by three others: it is either above, below or within that plane, and the orient3d predicate can tell you which. So one possible approach would be taking four arbitrary points q1 through q4 in general position, i.e. not all in a single plane, and no three on a single line. Then you could check whether p,p1,p2,qi are coplanar (sign is zero) for all i ∈ {1,2,3,4}. If they are, then p does at least lie on the line spanned by p1 and p2. Next you could check the orientations of p,p1,qi,qj for different i,j until you find a non-zero sign, then you could see whether p,p2,qi,qj has a different sign. If it does, then p is indeed between p1 and p2, hence on the same line. If you find no i,j such that p,p1,qi,qj is non-zero, then p=p1. Likewise if you have found one non-zero sign, but the corresponding sign of p,p2,qi,qj is zero, then p=p2. It is up to you whether you want to include the endpoints of your line segment. This paragraph might not be the most elegant way to do this, but it leverages the existing orient3d implementation, so it might be easier to use than writing a new predicate from scratch.
Note that in most cases, a point will not exactly lie on a line segment, due to rounding of the point coordinates themselves. The above will only help you to reliably detect those rare cases when it does, so you can deal with them. It may also allow you to make consistent choices for other predicates as well. In the world of CGAL, this approach would be termed “exact predicates, inexact constructions”, since the predicates can be computed reliably, but the geometric objects on which they operate are still subject to approximation. If you really need points that reliably lie on the line segments, and not just close by, then an exact construction approach using some exact number type would be preferable. Or you go with the approaches the other answers suggest.

You need to calculate the distance of the point to the line. That's simple mathematics. Then you decide how far of a distance "close enough" is in your case.

This problem is described as floating-point tolerance in this article and it points out the importance of measuring relative tolerance rather than absolute when comparing large values: http://realtimecollisiondetection.net/blog/?p=89
I've never had a situation where large floating point values are possible so I've always hard-coded a magic value into my comparisons. Eg:
if (Math.Abs(value1, value2) < 0.05f)
{
...
}
Which is "bad" but it works so long as value1 and value2 can't get too big.
In your case you really want to calculate the distance of the point from the line and then check that distance is small enough, rather than testing that the point is exactly on the line. I am rubbish at math myself so don't understand this but let me copy someone else's answer for calculating the shortest distance between a 3D point and a 3D line:
Vector3 u = new Vector3(x2 - x1, y2 - y1, z2 - z1);
Vector3 pq = new Vector3(x3 - x1, y3 - y1, z3 - z1);
float distance = Vector3.Cross(pq, u).Length / u.Length;
3D Perpendicular Point on Line From 3D point

Do floating point numbers lose precision if no operations are done on them

I have a class like such:
public class Test
{
public void CheckValue(float val)
{
if (val == 0f)
{
//Do something
}
}
}
//Somewhere else..
void SomeMethod
{
Test t = new Test();
t.CheckValue(0f);
}
Is it guaranteed that checking if (val == 0f) will return true? Or will floating point numbers only lose precision when you perform operations with them?

They don't lose precision. There are inaccuracies in describing some floating point values in decimal. When doing operations on them, those inaccuracies can add up or cancel each other out.
In your case, 0 can be described completely accurately both in decimal and as a binary floating point, so it should work. Other values might not be and you should (as others point out) not rely on == for floating point values.
See more information here: http://floating-point-gui.de/

What it comes down to is that you can't perfectly represent every floating point number (in all their infinite variance) in the 32 bits worth of memory that holds a float. A number that can't be represented perfectly will only have a very small error value, but it will be there.
For example, 1/3 is (roughly) 0.333333343267 in the nearest float representation. You've got 7 decimal digits of accuracy, then it hits a bit of a snag. So let's say you divide 1 by 3, then later multiply by 3. The result isn't 1, it's 1.0000000298 and change. So (1/3)*3 =/= 1.
These representational errors are why directly comparing floating point values is considered poor practice.
In situations where you need to minimize the impact of these little errors you should do whatever you can to stop them from accumulating. Rotating a set of coordinates a little at a time for instance will give you imperfect results, since the errors introduced by each rotation operation (using sin and cos values that are unlikely to have perfect float representations) will accumulate over time. Better to track the total angular change and rotate the original vector each time to get smaller error values.
Another place this can bite you is in a for loop. If you want to do 10,000 steps at 1/30000 per step, use an int for the loop variable and calculate the current position as a division. Slower, yes, but far more accurate. Using floats your error accumulates until you have only about 3 reliable decimal digits of significance, instead of over 7 if you divide each time.
Speed is nice, but precision is nicer in some circumstances.
And yes, it seems silly, but these errors can really mount up. Not so important in some places, rampant error factories in others.

You should never compare floating-point-numbers with "==". Instead you can use a border. For example instead of comparing == 0 you can do something like
float border = 0.00000001;
if( (val - border) <= 0 && (val + border) >= 0)