### Author Topic: Rounding errors part 2: the curious case of 1995hoo's mileage on usai in MD  (Read 151 times)

0 Members and 1 Guest are viewing this topic.

#### yakra

• TM Collaborator
• Hero Member
• Posts: 3816
• Last Login:Today at 10:07:59 am
##### Rounding errors part 2: the curious case of 1995hoo's mileage on usai in MD
« on: March 26, 2022, 07:07:04 pm »
A sequel to Rounding errors & point-to-point great circle distance: the curious case of Oscar's mileage on ME103

The concept at play here is that of cumulative rounding errors when working with finite-precision floating-point numbers.
Another thing affecting the result in the least significant digits is the order in which numbers are added together.

A simplified example -- if we can only ever have 3 sig figs each step of the way:
1.67 + 1.67 + 30 = 33.3
30 + 1.67 + 1.67 = 33.4

The values for the clinchedSystemMileageByRegion DB table are affected by this.
They're added together in whatever order they come out of a hash table.

• In Python, this may or may not be deterministic depending on implementation. IIRC it's not under Ubuntu, but is under FreeBSD -- indeed if I look in the archived .sql files on noreaster, 1995hoo has the same mileage on usai in MD from the beginning of 2022 up till the end of February when 95/495 got their DC/MD points adjusted. That number? 488.79137199999917 mi!

• In C++, the order is not deterministic.
grep "('usai','MD','1995hoo','488.79137" {\$v1,\$v2}/TravelMapping.sql
_t2/TravelMapping.sql:,('usai','MD','1995hoo','488.791371999999')
././TravelMapping.sql:,('usai','MD','1995hoo','488.791372')

In practice, not a big deal at all. Just something a bit amusing.
For me, it's a mild PitA if I want to make a tool to compare different sql files for development purposes. Can't just use sed and truncate decimals. Gotta use printf.