Author Topic: Site update failed 4/18  (Read 1040 times)

0 Members and 1 Guest are viewing this topic.

Online Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 1532
  • Last Login:Today at 10:02:41 pm
Site update failed 4/18
« on: April 19, 2018, 07:06:57 am »
I launched a routine site update last night before going to bed and it looks like it caused the DB on the server to become unresponsive.  TM and this forum were both unavailable until I noticed the problem this morning, when I restarted the DB on the server and re-launched the DB update portion of the process.  This was happening occasionally a while back but then we went a long time without a problem.  Now it's happened twice in a few days.  Unfortunately, it's not reproducible so it's hard to track down what's going on.

Online yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2150
  • Last Login:Today at 10:12:27 pm
Re: Site update failed 4/18
« Reply #1 on: April 19, 2018, 11:47:44 am »

Offline dave1693

  • Jr. Member
  • **
  • Posts: 86
  • Last Login:September 17, 2019, 12:54:12 pm
Re: Site update failed 4/18
« Reply #2 on: April 20, 2018, 06:52:22 pm »
I **hate** HeisenBugs.

Online Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 1532
  • Last Login:Today at 10:02:41 pm
Re: Site update failed 4/18
« Reply #3 on: April 21, 2018, 12:54:35 am »
I'm thinking it's at least not a bug in my update code.  The mysql server hangs right near the start of the ingestion of the new .sql file in a part that hasn't changed in a long time (something like line 4).  My guess is that there's a deadlock in mysql server that gets triggered by some combination of the site update sql starting at precisely the wrong time relative to something else the server is doing (it handles more than just TM stuff on this server).