Author Topic: Rounding errors part 2: the curious case of 1995hoo's mileage on usai in MD  (Read 223 times)

0 Members and 1 Guest are viewing this topic.

Offline yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 3884
  • Last Login:Yesterday at 08:28:54 pm
A sequel to Rounding errors & point-to-point great circle distance: the curious case of Oscar's mileage on ME103

The concept at play here is that of cumulative rounding errors when working with finite-precision floating-point numbers.
Another thing affecting the result in the least significant digits is the order in which numbers are added together.

A simplified example -- if we can only ever have 3 sig figs each step of the way:
1.67 + 1.67 + 30 = 33.3
30 + 1.67 + 1.67 = 33.4

The values for the clinchedSystemMileageByRegion DB table are affected by this.
They're added together in whatever order they come out of a hash table.

• In Python, this may or may not be deterministic depending on implementation. IIRC it's not under Ubuntu, but is under FreeBSD -- indeed if I look in the archived .sql files on noreaster, 1995hoo has the same mileage on usai in MD from the beginning of 2022 up till the end of February when 95/495 got their DC/MD points adjusted. That number? 488.79137199999917 mi! :D

• In C++, the order is not deterministic.
grep "('usai','MD','1995hoo','488.79137" {$v1,$v2}/TravelMapping.sql
_t2/TravelMapping.sql:,('usai','MD','1995hoo','488.791371999999')
././TravelMapping.sql:,('usai','MD','1995hoo','488.791372')


In practice, not a big deal at all. Just something a bit amusing.
For me, it's a mild PitA if I want to make a tool to compare different sql files for development purposes. Can't just use sed and truncate decimals. Gotta use printf.
Sri Syadasti Syadavaktavya Syadasti Syannasti Syadasti Cavaktavyasca Syadasti Syannasti Syadavatavyasca Syadasti Syannasti Syadavaktavyasca