Author Topic: DB hang 2/28/20  (Read 625 times)

0 Members and 1 Guest are viewing this topic.

Offline Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 1903
  • Last Login:Today at 03:19:27 pm
DB hang 2/28/20
« on: February 28, 2020, 10:04:36 pm »
I wasn't paying attention to the start of the site update when it reports "User sleep" DB processes and by letting the update go without fixing those, the DB got into a bad state and had to be restarted.  This caused all of TM and the Forum to fail for a short time.  Things look fine now, and tonight's site update is being re-run.

Offline yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2847
  • Last Login:Yesterday at 02:31:39 pm
Re: DB hang 2/28/20
« Reply #1 on: March 26, 2020, 05:06:27 pm »
What do the results of
echo "show processlist;" | mysql --defaults-group-suffix=travmap -u travmap
look like?

Are the results something you have to kill manually while siteupdate.py works away?

Offline Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 1903
  • Last Login:Today at 03:19:27 pm
Re: DB hang 2/28/20
« Reply #2 on: March 26, 2020, 09:41:48 pm »
Yes, if there are any processes shown in "User Sleep" it means they're hanging, and the DB ingestion will grind things to a halt.  I kill them manually by pid while the update proceeds.  They seem to come in bunches.  A few days in a row there will be those kinds of processes and then a month or two with none.

Offline yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2847
  • Last Login:Yesterday at 02:31:39 pm
Re: DB hang 2/28/20
« Reply #3 on: March 28, 2020, 11:06:04 am »
Thinking about automated updates. Is it possible to isolate the info we need using grep, cut, sed, etc., and kill them automatically? Are there risks involved?

Offline Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 1903
  • Last Login:Today at 03:19:27 pm
Re: DB hang 2/28/20
« Reply #4 on: March 28, 2020, 11:18:02 am »
Probably could.  I'd rather figure out why the processes hang in the first place.  Some earlier discussion:  https://github.com/TravelMapping/DataProcessing/issues/111