User Discussions > Other Discussion

Rounding errors part 2: the curious case of 1995hoo's mileage on usai in MD

(1/1)

yakra:
A sequel to Rounding errors & point-to-point great circle distance: the curious case of Oscar's mileage on ME103


--- Quote from: yakra on April 29, 2019, 05:33:19 pm ---The concept at play here is that of cumulative rounding errors when working with finite-precision floating-point numbers.

--- End quote ---
Another thing affecting the result in the least significant digits is the order in which numbers are added together.

A simplified example -- if we can only ever have 3 sig figs each step of the way:
1.67 + 1.67 + 30 = 33.3
30 + 1.67 + 1.67 = 33.4

The values for the clinchedSystemMileageByRegion DB table are affected by this.
They're added together in whatever order they come out of a hash table.

• In Python, this may or may not be deterministic depending on implementation. IIRC it's not under Ubuntu, but is under FreeBSD -- indeed if I look in the archived .sql files on noreaster, 1995hoo has the same mileage on usai in MD from the beginning of 2022 up till the end of February when 95/495 got their DC/MD points adjusted. That number? 488.79137199999917 mi! :D

• In C++, the order is not deterministic.
grep "('usai','MD','1995hoo','488.79137" {$v1,$v2}/TravelMapping.sql
_t2/TravelMapping.sql:,('usai','MD','1995hoo','488.791371999999')
././TravelMapping.sql:,('usai','MD','1995hoo','488.791372')

In practice, not a big deal at all. Just something a bit amusing.
For me, it's a mild PitA if I want to make a tool to compare different sql files for development purposes. Can't just use sed and truncate decimals. Gotta use printf.

Navigation

[0] Message Index