I just made a change that speeds things up quite a bit on maps without too many segments visible. It finds visible waypoints first, then gets segments based on that list. Unfortunately, it's slower than the previous mechanism for large numbers of waypoints, so when over 500 (just a best guess) waypoints are found, it falls back to the original query to find segments.
In any case, for fairly close-in maps, this is a big improvement. Most of the delay we see now in those cases is from the debouncing to avoid rapid reloads during panning/zooming.