TM Server possible hardware upgrades

[Jim]

I am in the process of evaluating some options for upgrades to the server that houses TM and METAL.  Options being considered include an the addition of either a single SSD or multiple SSDs under RAID control for frequently used data (like the directory where I run site updates, the web server files, and the database), the addition of several TB of spinning disk to increase archival storage capabilities (save and make available more data over time), and possibly more memory to allow some "tmpfs" memory filesystem work.

A preliminary test to see how an all "in-memory" Python site update would work just completed.  I had the repositories and working directory for the site update program on a tmpfs partition, and siteupdate.py reports:

Code: [Select]

Total run time: [453.6]
compared to last night's update from the current SATA disks:

Code: [Select]

Total run time: [568.8]
I think a big improvement might be possible by moving the database to SSDs, both in the time is takes to load the DB instances from the .sql file at the end of the site update, and access times when the DB is accessed by the web front end.

The bigger improvement would probably come from switching to the C++ site update program which can make use of the processing cores.  Combine that with faster I/O both in latency and transfer times, and we might be on our way to running automated updates.

I hope to have a decision on any purchases soon.

[yakra]

Hm. Maybe disk access affects the Python execution time a little bit, but I think it may be less than it appears...
Last night (with current repos) I got Total run time: [482.0] single-threaded, off the normal disk.
For comparison, 2020-08-24@21:53:51/logs/siteupdate.log:Total run time: [480.7]. Noreaster runs multithreaded, which will make it a bit slower. Repos were however much smaller in August, which will make it a bit faster.
siteupdate.py appears mostly CPU-bound even when reading .wpts -- lab2 stays mostly 65-85% CPU use when reading.

Threaded C++ OTOH, when reading .wpts that aren't in the disk's cache, CPU use is close to zero. So yes, SSDs will definitely help there.

RE .sql ingestion, based on my experiences....
The way lab2 is set up right now, CPU use is pretty near 100%, so it looks CPU-bound.
It appears the bottleneck isn't disk access unless logging is set up to make it so. If that's what your logging need are, then SSDs can help.

I'll certainly agree that SSDs would help with regular DB access though.

The bigger improvement would probably come from switching to the C++ site update program

which can make use of the processing cores.

It'd be interesting to see what noreaster can do, and where its sweet spot would be.
Few tasks can scale all the way up to 24 threads, though with noreaster's presumably greater memory bandwidth (& greater L3 cache) that could change.

What's noreaster have for RAM? top says 13G free, so... 16? If just 3 people log in and run datacheck.sh, that could break the bank...

Combine that with faster I/O both in latency and transfer times, and we might be on our way to running automated updates.

I'll refrain from getting into the whats & hows for now, and say... neato!

[Jim]

RAM on noreaster is at 32GB, likely will upgrade to 64GB as part of this.  20 physical cores, each of which presents to the OS as 2 logical cores.

I'm hoping for a nice improvement on DB reads, especially the large queries when you're in mapview and need to load several thousand points and connections.  There are still network costs, of course, and then a lot depends on the client browser's capabilities to execute all of the JS to parse the data and then display it in Leaflet.

[yakra]

20 physical cores, each of which presents to the OS as 2 logical cores.

I'd be surprised to see all tasks scale all the way up to 40 threads though. Most hit a sweet spot then slowly get less efficient.

I'm hoping for a nice improvement on DB reads, especially the large queries when you're in mapview and need to load several thousand points and connections.

I can see this helping for the big queries, especially if people several are making them concurrently. (Dangit, don't I have an SSD squirreled away in one of the lab servers?)

There are still network costs, of course,

I see this as the bigger slowdown for end users.
Hey, here's a crazy idea. The data are all transmitted as text, right? JavaScript? Depending on zoom level, truncate decimal places from lat./long. coords.
Probably won't be useful -- by the time we've zoomed out enough to truncate a meaningful amount, the sheer number of elements to load will more than make up for it.

and then a lot depends on the client browser's capabilities to execute all of the JS to parse the data and then display it in Leaflet.

And this is where the slowdown really hits...