Author Topic: DB hang 2/28/20  (Read 1473 times)

0 Members and 1 Guest are viewing this topic.

Online Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2451
  • Last Login:Today at 09:57:41 am
DB hang 2/28/20
« on: February 28, 2020, 10:04:36 pm »
I wasn't paying attention to the start of the site update when it reports "User sleep" DB processes and by letting the update go without fixing those, the DB got into a bad state and had to be restarted.  This caused all of TM and the Forum to fail for a short time.  Things look fine now, and tonight's site update is being re-run.

Online yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 3933
  • Last Login:Today at 09:53:36 am
Re: DB hang 2/28/20
« Reply #1 on: March 26, 2020, 05:06:27 pm »
What do the results of
echo "show processlist;" | mysql --defaults-group-suffix=travmap -u travmap
look like?

Are the results something you have to kill manually while siteupdate.py works away?
Sri Syadasti Syadavaktavya Syadasti Syannasti Syadasti Cavaktavyasca Syadasti Syannasti Syadavatavyasca Syadasti Syannasti Syadavaktavyasca

Online Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2451
  • Last Login:Today at 09:57:41 am
Re: DB hang 2/28/20
« Reply #2 on: March 26, 2020, 09:41:48 pm »
Yes, if there are any processes shown in "User Sleep" it means they're hanging, and the DB ingestion will grind things to a halt.  I kill them manually by pid while the update proceeds.  They seem to come in bunches.  A few days in a row there will be those kinds of processes and then a month or two with none.

Online yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 3933
  • Last Login:Today at 09:53:36 am
Re: DB hang 2/28/20
« Reply #3 on: March 28, 2020, 11:06:04 am »
Thinking about automated updates. Is it possible to isolate the info we need using grep, cut, sed, etc., and kill them automatically? Are there risks involved?
Sri Syadasti Syadavaktavya Syadasti Syannasti Syadasti Cavaktavyasca Syadasti Syannasti Syadavatavyasca Syadasti Syannasti Syadavaktavyasca

Online Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2451
  • Last Login:Today at 09:57:41 am
Re: DB hang 2/28/20
« Reply #4 on: March 28, 2020, 11:18:02 am »
Probably could.  I'd rather figure out why the processes hang in the first place.  Some earlier discussion:  https://github.com/TravelMapping/DataProcessing/issues/111