siteupdate performance: CentOS 7 vs Ubuntu 18.04 LTS vs FreeBSD 13.0

[yakra]

A comparison of siteupdate's multi-threaded performance under Centos, Ubuntu and FreeBSD on the same machine.
Tests were run on a Dell PowerEdge R420 with Dual Xeon E5-2420 CPUs & 12x4GB DDR3-1333 RAM (I can try to find out the timings if anyone wants).
Hard disks: The Linux (CentOS & Ubuntu) installs are on identical Savvio 15K.3s, whereas FreeBSD is on a Kingston HyperX SH103S3/120G. However, all stats CSVs, logs other than siteupdate, nmp_merged files, and graphs were written to /dev/null.

Info from uname -a and clang --version:

• CentOS 7
  Linux lab3 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  clang version 3.4.2 (tags/RELEASE_34/dot2-final)
  Target: x86_64-redhat-linux-gnu
• Ubuntu 18.04.5 LTS x86_64
  Linux lab4 5.4.0-135-generic #152~18.04.2-Ubuntu SMP Tue Nov 29 08:23:49 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
  clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
  Target: x86_64-pc-linux-gnu
• FreeBSD 13.0
  FreeBSD bsdlab 13.0-RELEASE FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Fri Apr  9 04:24:09 UTC 2021     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64
  FreeBSD clang version 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3fe)
  Target: x86_64-unknown-freebsd13.0

Commit info:
HighwayData:    972df9bb65bf0a270b4fc00dcae30b1823ea484b
UserData:       d721ce609a34bd274bc48940f876f51564a89a94
DataProcessing: fbed09107b80b89cc1d2ecf9bb2b9c5a608c1062

Method:
The siteupdate executable was renamed siteupdate_fbed0.
check10.sh --null -v -tp 3 _fbed0 was used to automate 240 runs of siteupdate on each OS, 10 for each number of threads, saving siteupdate.log each time.
For each task, table.sh was used to iterate through each number of threads, find the before & after timestamps in siteupdate.log, calculate the time spent on the task in each of the 10 passes, then average all 10 passes together.

[yakra]

ReadWptThread: Reading waypoints for all routes.

[yakra]

WaypointQuadtree::sortnodes: Sorting waypoints in Quadtree.

[yakra]

NmpSearchThread: Searching for near-miss points.

[yakra]

NmpMergedThread: Writing near-miss point merged wpt files.

[yakra]

RteIntThread: Creating label hashes and checking route integrity.

[yakra]

ReadListThread: Processing traveler list files

[yakra]

ConcAugThread: Augmenting travelers for detected concurrent segments.

[yakra]

CompStatsRThread: Performing per-route data checks and computing stats.

[yakra]

CompStatsTThread: Computing stats per traveler.

[yakra]

UserLogThread: Creating per-traveler stats logs and augmenting data structure.

Edit: Storing & referencing a system_region_mileages iterator didn't really pan out, with the caveat that I've yet to test it out on my laptop. That has only 512K L2 cache, so it may help there.
That aside, I have some other tricks up my sleeve that may improve efficiency and help FreeBSD scale a little better.

[yakra]

HighwayGraph::simplify: Creating unique names and vertices

[yakra]

SubgraphThread: (creating & writing all subgraphs)
MasterTmgThread & SubgraphThread: (writing all .tmg files)

[yakra]

Subtotal: of all the above threaded tasks combined

[yakra]

Total run time:

Edit #2: A couple charts are now fixed. Edit #1 below was written before fixing them.
Edit #1: Eagle-eyed observers may notice some oddities with FreeBSD in the last two charts.
The combined subtotal of all the threaded tasks takes longer at 2 threads than 1 (thanks in no small part to graph generation's poor scaling).
Yet the total run time is faster at 2 threads than 1. How is this possible?

I picked the wrong timestamp to measure graph generation speed; I shouldn't be measuring just the subgraphs.
At 2+ threads, the master graph files are written alongside subgraphs -- numthreads-1 SubgraphThreads are spawned along with one MasterTmgThread, which spawns another SubgraphThread when it finishes. Meaning, when numthreads = 1, no SubgraphThreads are spawned until MasterTmgThread finishes.
At 2+ threads, the master graphs are started at the same time as the continent graphs, so my measurement will include time spent writing the master graphs too.
At 1 thread, the master graphs are finished before the continent graphs start. This makes our 1-thread figures look faster than they should compared to 2+ threads, as we're processing a smaller amount of data.
Writing the master graph files has to happen somewhere -- where is it? It's after the "Setting up subgraphs" message is printed. Indeed if we use table.sh to measure the time spent on this "single-threaded" task, it's 15s @ 1 thread, and hovers around 0.02s from 2 to 24 threads. I think it's acceptable to include this time in our total graph generation figures.

So yeah, I'll run table.sh again with new timestamps, and fix the charts. Changes I expect to see:

• All 3 OSes will be shown taking longer writing graph files at 1 thread.
• The 2 Linux flavors will scale better between 1 & 2 threads.
• FreeBSD will scale... less badly. There will still be a time increase between 1 & 2 threads, just not as pronounced as currently shown.
• This bump will be enough for the 1->2-thread time savings in other tasks to offset the increase, and we should see an initial decrease in the threaded task subtotal on FreeBSD, in line with the the Total Run Time chart shows.