Site update failed 4/18

[Jim]

I launched a routine site update last night before going to bed and it looks like it caused the DB on the server to become unresponsive.  TM and this forum were both unavailable until I noticed the problem this morning, when I restarted the DB on the server and re-launched the DB update portion of the process.  This was happening occasionally a while back but then we went a long time without a problem.  Now it's happened twice in a few days.  Unfortunately, it's not reproducible so it's hard to track down what's going on.

[yakra]

www.catb.org/jargon/html/H/heisenbug.html ?

[dave1693]

I **hate** HeisenBugs.

[Jim]

I'm thinking it's at least not a bug in my update code.  The mysql server hangs right near the start of the ingestion of the new .sql file in a part that hasn't changed in a long time (something like line 4).  My guess is that there's a deadlock in mysql server that gets triggered by some combination of the site update sql starting at precisely the wrong time relative to something else the server is doing (it handles more than just TM stuff on this server).