Fun with floating point

Log In or Register

View Resource

Source

blogs.msdn.com (Presentation Source), posted 2010-05-07

Added By

BillHenn on 2010-05-07

Vitals

0score

4871 total views

Summary

A detailed discussion about how floating point math works in .NET and an implementation of a custom floating point type that works around issues with some floating point math.

Excerpt

"WPF uses double-precision floating point numbers (double in C#) in much of its public API and it uses single-precision floating point for much of its internal rendering. So floating point math is something we deal with constantly. Oddly enough, I actually knew very little about the gory details until I recently tried to write a container that had to use doubles as a key, which required working around the problems with precision in floating point math. I found the whole exercise to be very interesting, and so I’ll present what I learned here. Consider this code: if (0.8 - 0.7 == 0.1) { Console.WriteLine("Math makes sense."); } Believe it or not, the comparison will fail."