Travel Mapping

User Discussions => Welcome & Notices => Topic started by: Jim on January 19, 2021, 11:47:37 am

Title: January 2021 Hardware Upgrades
Post by: Jim on January 19, 2021, 11:47:37 am
The hardware has arrived for memory and disk upgrades to the server that houses TM and METAL.  I hope to install them this week, possibly as soon as tomorrow (Wed 1/20).  There will be multiple complete outages of TM and METAL, including this forum.  Watch this thread, the motd on the main TM page, and the project status on GitHub (https://travelmapping.github.io/) for updates.
Title: Re: January 2021 Hardware Upgrades
Post by: yakra on January 19, 2021, 01:15:17 pm
memory
Been doing some work on C++ subgraph generation this past week where memory bandwidth plays an interesting role; it's made me wonder what noreaster's setup is.

So what's it got?
DDR4?
What mHz?
Dual-channel? Tri-channel? Other?
Title: Re: January 2021 Hardware Upgrades
Post by: Jim on January 19, 2021, 02:11:13 pm
Been doing some work on C++ subgraph generation this past week where memory bandwidth plays an interesting role; it's made me wonder what noreaster's setup is.

So what's it got?
DDR4?
What mHz?
Dual-channel? Tri-channel? Other?

Current:

32GB (8x4GB) 2400MHz DDR4 RDIMM ECC

Adding 8 more of the same to bump to 64GB total.
Title: Re: January 2021 Hardware Upgrades
Post by: Jim on January 19, 2021, 08:30:08 pm
I am planning to do the upgrades starting around 9:15 AM EST on Wednesday, January 20.  Expect TM to be unavailable at times throughout the day.
Title: Re: January 2021 Hardware Upgrades
Post by: yakra on January 19, 2021, 10:51:33 pm
Looks like the DDR4 modules are rated for 2400MHz, but the processor only supports 2133...

On noreaster we have this:

Dual Intel Xeon Processor E5-2630 v4 (10C, 2.2GHz, 3.1GHz Turbo, 2133MHz, 25MB, 85W)
The 2133 here appears to refer to the memory clock:
https://ark.intel.com/content/www/us/en/ark/compare.html?productIds=92981
...which answers my other question: it's 4-channel.

Not bad. That's nominally 178% the bandwidth of lab2, 267% the bandwidth of lab3.
noreaster should be able to churn out subgraphs wicked quick.
Title: Re: January 2021 Hardware Upgrades
Post by: Jim on January 20, 2021, 10:18:40 am
The disk is in, but not yet partitioned or in use.  Turns out there was a misunderstanding on the memory, and it has not yet arrived.  We're up at the moment, but I will be doing a lot of reconfiguring so expect ongoing outages this morning.
Title: Re: January 2021 Hardware Upgrades
Post by: Jim on January 20, 2021, 03:09:11 pm
The database is now on the new SSD.  This post is in part to make sure the forum still works, but also to say TM service disruptions will continue today.
Title: Re: January 2021 Hardware Upgrades
Post by: Jim on January 20, 2021, 03:32:50 pm
Things still look good after a reboot with the DB and some other things moved to the SSD.  The site should be stable for the next hour or two, then I'll be making more changes.
Title: Re: January 2021 Hardware Upgrades
Post by: Jim on January 20, 2021, 07:47:20 pm
The main TM site and the forum are now living on the new SSD.  Hopefully this post will work!
Title: Re: January 2021 Hardware Upgrades
Post by: Jim on January 20, 2021, 07:54:30 pm
I'm still doing some rearranging and configuring, but I don't expect any of it to cause more than very brief interruptions to the operations of the main TM site or the forum.  Please let me know if you notice any new problems that might be a result of today's disk upgrade.

When the new memory chips come in, we'll have another outage, but that should be a lot less disruptive.
Title: Re: January 2021 Hardware Upgrades
Post by: rickmastfan67 on January 21, 2021, 12:14:45 am
Noticed there was a 'theme' link issue since you moved the forum to a different directory.  I've just fixed that issue.
Title: Re: January 2021 Hardware Upgrades
Post by: michih on January 21, 2021, 08:59:31 am
wpt editor does not work anymore (FF, Edge, IE): https://travelmapping.net/wptedit/
Leaflet issue?
Title: Re: January 2021 Hardware Upgrades
Post by: michih on January 21, 2021, 09:03:43 am
no, a general issue - on our end? When I load data, the table does not apear

Not sure whether there is a connection to the HW update. It worked yesterday but I had wpt editor opened during the upgrade.

Of course, no cache issue etc. (automatically deleted on closing browser)
Title: FAVICON --> wpt editor + mapview + showroute failures
Post by: michih on January 21, 2021, 09:08:12 am
[Exception... "Favicon at "https://travelmapping.net/favicon.ico" failed to load: Not Found."  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource:///modules/FaviconLoader.jsm :: onStopRequest :: line 253"  data: no] 2 FaviconLoader.jsm:253:22

wpt editor + mapview + showroute do not work - because of favicon failure!
Title: Re: January 2021 Hardware Upgrades
Post by: dfilpus on January 21, 2021, 11:24:36 am
Map display not working in regions or individual route pages.
Title: Re: January 2021 Hardware Upgrades
Post by: Markkos1992 on January 21, 2021, 01:45:39 pm
wpt editor does not work anymore (FF, Edge, IE): https://travelmapping.net/wptedit/
Leaflet issue?


It works on Chrome but the points on the map are not shown correctly.
Title: Re: January 2021 Hardware Upgrades
Post by: tckma on January 21, 2021, 04:36:36 pm
HB doesn't seem to be working right.  See for example https://travelmapping.net/hb/showroute.php?units=miles&u=tckma&r=de.de001

From chrome://version --


Google Chrome   87.0.4280.141 (Official Build) (64-bit) (cohort: Stable)
Revision   9f05d1d9ee7483a73e9fe91ddcb8274ebcec9d7f-refs/branch-heads/4280@{#2007}
OS   Windows 10 OS Version 1909 (Build 18363.1256)
JavaScript   V8 8.7.220.31
Flash   32.0.0.465 C:\Users\<removed my employee number>\AppData\Local\Google\Chrome\User Data\PepperFlash\32.0.0.465\pepflashplayer.dll
Title: Re: January 2021 Hardware Upgrades
Post by: cl94 on January 21, 2021, 05:36:23 pm
I can confirm that the waypoint browser isn't working on Chrome, Firefox, or (ew) Edge on my computer. "URL Placeholder" is shown in red text and the route shield is shown, but no table or map is generated.
Title: Re: January 2021 Hardware Upgrades
Post by: dfilpus on January 21, 2021, 06:09:14 pm
I can confirm that the waypoint browser isn't working on Chrome, Firefox, or (ew) Edge on my computer. "URL Placeholder" is shown in red text and the route shield is shown, but no table or map is generated.
Ditto on Brave, Safari and Firefox on my iMac.
Title: Re: January 2021 Hardware Upgrades
Post by: oscar on January 21, 2021, 06:51:54 pm
I'm not having any problems with the HB in my Chrome browser. I haven't cleared the cache, wishing to keep whatever functionality I have.

The HB doesn't work in Microsloth Edge on my computer, through.
Title: Re: January 2021 Hardware Upgrades
Post by: Jim on January 21, 2021, 07:40:17 pm
Leaflet must have been cached in my browser for yesterday's testing.  I've put leaflet where it needs to be, and so far so good on my end.  Let me know if I've fixed it for all of you as well.

I'm sure there will be other problems, hopefully not as significant as this one!
Title: Re: January 2021 Hardware Upgrades
Post by: rickmastfan67 on January 21, 2021, 07:42:56 pm
HB working again here on my end Jim.  8)
Title: Re: January 2021 Hardware Upgrades
Post by: oscar on January 21, 2021, 07:55:41 pm
Now works for me in Edge, too.
Title: Re: January 2021 Hardware Upgrades
Post by: cl94 on January 21, 2021, 08:07:47 pm
Yup, that fixed it. Thanks.
Title: Re: January 2021 Hardware Upgrades
Post by: Jim on January 21, 2021, 08:14:27 pm
I suppose next time I move the files on the web server, I shouldn't spend the next day away from the computer.  But the skiing was fantastic!
Title: Re: January 2021 Hardware Upgrades
Post by: Markkos1992 on January 21, 2021, 08:26:50 pm
wpt editor does not work anymore (FF, Edge, IE): https://travelmapping.net/wptedit/
Leaflet issue?


It works on Chrome but the points on the map are not shown correctly.

The WP Editor on Chrome is back to normal.
Title: Re: January 2021 Hardware Upgrades
Post by: tckma on January 21, 2021, 08:30:09 pm
Yep, working again.
Title: Re: January 2021 Hardware Upgrades
Post by: rickmastfan67 on January 21, 2021, 08:46:19 pm
Leaflet must have been cached in my browser for yesterday's testing.

Always test in a 'private' window, as that doesn't use the cache. ;)
Title: Re: January 2021 Hardware Upgrades
Post by: michih on January 22, 2021, 01:42:14 am
I think everything is back to normal - wish we could say that for our lifes too :)
Title: Re: January 2021 Hardware Upgrades
Post by: yakra on January 22, 2021, 02:32:36 am
Turns out there was a misunderstanding on the memory, and it has not yet arrived.
When the new memory arrives and noreaster has to go offline, I don't suppose you could humor me (https://github.com/TravelMapping/DataProcessing/issues/375#issuecomment-764962926) and do a round of memtest86 with the existing DIMMs? :D
Title: Re: January 2021 Hardware Upgrades
Post by: Jim on January 22, 2021, 07:32:11 am
Turns out there was a misunderstanding on the memory, and it has not yet arrived.
When the new memory arrives and noreaster has to go offline, I don't suppose you could humor me (https://github.com/TravelMapping/DataProcessing/issues/375#issuecomment-764962926) and do a round of memtest86 with the existing DIMMs? :D

Can probably make that happen.
Title: Re: January 2021 Hardware Upgrades
Post by: Jim on March 15, 2021, 10:45:25 pm
I'm finally planning to complete the hardware upgrades by doubling the server's RAM on the afternoon of Tuesday, March 16.  Down time shouldn't last long, but TM, the forum and METAL will all be down during the upgrade.
Title: Re: January 2021 Hardware Upgrades
Post by: Jim on March 16, 2021, 12:45:20 pm
Heading to the machine room shortly for the memory upgrade.  Expect several minutes to maybe a half hour of down time between 1:00 and 2:00 PM EDT today.
Title: Re: January 2021 Hardware Upgrades
Post by: Jim on March 16, 2021, 03:10:28 pm
What was supposed to be a few minutes of downtime to pop in the new memory chips turned into about an hour tracking down the fact that 2 of the 8 memory chips were a slightly different part and not compatible with the other memory in the system.  The good news is that the system at least went from 32 GB of RAM to 56 GB, though not the 64 GB it was intended to be.  Everything should now be back to normal.  There will be a hopefully shorter outage whenever the correct chips finally arrive.
Title: Re: January 2021 Hardware Upgrades
Post by: yakra on March 16, 2021, 04:57:46 pm
Oh fun. Sounds like you're not getting the full bandwidth benefit of n channels then?
Title: Re: January 2021 Hardware Upgrades
Post by: Jim on March 16, 2021, 08:33:45 pm
Oh fun. Sounds like you're not getting the full bandwidth benefit of n channels then?

Probably not.  Hopefully the two misfits will be replaced fairly quickly.
Title: Re: January 2021 Hardware Upgrades
Post by: yakra on June 24, 2021, 01:08:25 pm
When the new memory arrives and noreaster has to go offline, I don't suppose you could humor me (https://github.com/TravelMapping/DataProcessing/issues/375#issuecomment-764962926) and do a round of memtest86 with the existing DIMMs? :D
Can probably make that happen.
Did this happen?
The mysterious C++ Pointer Value Loses 0x900000000 bug is back (https://github.com/TravelMapping/DataProcessing/issues/442).
It's weird enough that I can't write off bad memory as a potential cause.
Title: Re: January 2021 Hardware Upgrades
Post by: Jim on June 24, 2021, 02:30:07 pm
When the new memory arrives and noreaster has to go offline, I don't suppose you could humor me (https://github.com/TravelMapping/DataProcessing/issues/375#issuecomment-764962926) and do a round of memtest86 with the existing DIMMs? :D
Can probably make that happen.
Did this happen?
The mysterious C++ Pointer Value Loses 0x900000000 bug is back (https://github.com/TravelMapping/DataProcessing/issues/442).
It's weird enough that I can't write off bad memory as a potential cause.

We didn't do any additional testing.  I still haven't acquired the 2 additional ones to get up to the full 64 GB.
Title: Re: January 2021 Hardware Upgrades
Post by: yakra on June 25, 2021, 01:11:04 pm
The Bad RAM Hypothesis is ruled out.
Title: Re: January 2021 Hardware Upgrades
Post by: Jim on June 25, 2021, 03:49:32 pm
I have probably mentioned this before, but I spent a ridiculous amount of time in grad school tracing a bug that wound up being a bug in IBM's compiler, not my code...
Title: Re: January 2021 Hardware Upgrades
Post by: yakra on June 25, 2021, 10:23:03 pm
Haha, I've run into one compiler bug (https://github.com/TravelMapping/DataProcessing/issues/327) already. At least that time I had something to Google; this time not so much.
Tracking down whatever this is ain't gonna be pretty.
I guess one relatively low-effort thing I can try is to install FreeBSD 13.0 and see if there's a new version of clang & things behave differently...
Title: Re: January 2021 Hardware Upgrades
Post by: yakra on January 06, 2023, 12:22:04 am
Still waiting on those two DIMMs?
Title: Re: January 2021 Hardware Upgrades
Post by: Jim on January 06, 2023, 09:12:18 am
Still waiting on those two DIMMs?

They were unavailable for a while and I never pursued it further.  Maybe I still should.