AS3: Tips for Floating Point Hell December 4, 2007

Here's some tips for working with floating point round off errors in applications.

use Math.round, or roundToPrecision late as possible, and only when you have to.
when doing number comparisons, use the following notClose or fuzzyEquals instead of '=='
truly think about the degree of precision you need and be consistent.

1) Keep things in MVC format, meaning changes percolate down as they are, and views get notified and can reformat on the way up as they need to for themselves without touching the actual model. This is important as floating point math is largely a blackbox like your x-girlfriend. Well behaved at times, and then damn quirky in the next second on something seeming innocuous, making it impossible to predict (and unit test)
Imagine that we are driving a car along the windy highway, and staying in the lanes is the goal to save us from the cliffs of insanity off to either side. Problem is our steering wheel only goes 100% left or 100% right., so for some particular curves it works well, others...lots of thrashing back and forth to stay on the road, in this case the thrashing is the computer trying to fit oddball things like 1/3 into a number system that really only has left or right angle turns.Â The first tendency is to try and wrangle say 4.0000001 into 4...as they are close right!? well what happens is by puttting on the brakes (e..g rounding) you introduce another set of steering wheels on the car, hoping that it will cancel it out, when in reality you've double the things that can go wrong.

The computer's number system is in binary sacrifices precision for speed, it represents numbers (variable and the results of calculationgs) with 16, 32...and 64 bit precision, which is often good enough for most things.Â What happens in many calculations is that while a particular number say 1/3 may not display perfectly on screen, that 1/3 * 3Â gets us back to a whole number which the computer can represent. So by rounding early, you disrupt this balance, and the computer can never recover the perfect 10 again.
//BAD


model.num1 = round(someCal(var1, var2));
if(model.num1 == 0)...do something
//Better
model.num1 = someCal(var1, var2);
if(Math.round(model.num1) == 0)...do something

2) When working with a series of these black boxes chained together ( this includes assigning and retrieving fromÂ Â displayObject.x, y), it's like playing the telephone game, where with each person playing, the number of people in the chain passing messages, the more problems that can happen grow. Using the car metaphor say we have 2 steering wheels for the front Left wheel and one for the front Right wheel. Both can go either all the way left or all the way right. We have 4 possibilities, 3 outcomes:

L+L=> Left turn
R+R =>Â Right turn
L+R or R+L => Straight ahead, but hella wobbly.

Something similar happens at the end of your calculations. You might get left (1) or right (0) or somewhere in the center (0.0001) . The problem with rounding here is that since the output is wobbly it might be 0.0001 or 0.056, or -0.0005 making it impossible to roundToPrecision well. Thus when operating on the output of a calculation, you have to compare against a range of precision, like center is somewhere between both wheels steering L and steering Right. You can think of this also as a Number holding an umbrella, and we are curious if it's under the umbrella or outside of it.
Here is a variant for outside.

public function notClose(number1:Number, number2:Number, precision:int = 0):Boolean
{
var difference:Number = number1 - number2;
var range:Number = Math.pow(10, -precision);
trace(-range + " " + difference + " " + range);
var res:Boolean = ( difference < -range ) || (range < difference) ;
trace("notClose" + res);
return res;
}

This has two patterns I like. While var res:Object isn't strictly necessary, it's possible to return the calc directly,Â putting it in this pattern makes it easy to trace ..even if the code is working perfectly fine, when you might be combing through a bunch of calculations looking for how big of an umbrella to make.Â The second is
aligning the '<' in the left being lower and right being greater, as they it becomes easier to visualize spatially (for me), like it's on a number axis.
3) For some things like working with displayObjects, in the end it has to be rendered to a fixed number of pixels, and especially with text it's clearer when lines don't lie on half pixels, creating fuzziness. Â Â So while you can calculate in your function down to 0.00000000 etc. in the end it can only be displayed as 0.0, and similarly when expecting 0.0000000 out of your displayObject.x, width etc. you'll find that it's far more limited (lookup twips if your interested)

Comments are closed.