Author Topic: DB hang 2/28/20  (Read 4011 times)

0 Members and 1 Guest are viewing this topic.

Offline Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2732
  • Last Login:Today at 10:27:44 am
DB hang 2/28/20
« on: February 28, 2020, 10:04:36 pm »
I wasn't paying attention to the start of the site update when it reports "User sleep" DB processes and by letting the update go without fixing those, the DB got into a bad state and had to be restarted.  This caused all of TM and the Forum to fail for a short time.  Things look fine now, and tonight's site update is being re-run.

Offline yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 4234
  • Last Login:February 13, 2024, 07:19:36 pm
  • I like C++
Re: DB hang 2/28/20
« Reply #1 on: March 26, 2020, 05:06:27 pm »
What do the results of
echo "show processlist;" | mysql --defaults-group-suffix=travmap -u travmap
look like?

Are the results something you have to kill manually while siteupdate.py works away?
Sri Syadasti Syadavaktavya Syadasti Syannasti Syadasti Cavaktavyasca Syadasti Syannasti Syadavatavyasca Syadasti Syannasti Syadavaktavyasca

Offline Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2732
  • Last Login:Today at 10:27:44 am
Re: DB hang 2/28/20
« Reply #2 on: March 26, 2020, 09:41:48 pm »
Yes, if there are any processes shown in "User Sleep" it means they're hanging, and the DB ingestion will grind things to a halt.  I kill them manually by pid while the update proceeds.  They seem to come in bunches.  A few days in a row there will be those kinds of processes and then a month or two with none.

Offline yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 4234
  • Last Login:February 13, 2024, 07:19:36 pm
  • I like C++
Re: DB hang 2/28/20
« Reply #3 on: March 28, 2020, 11:06:04 am »
Thinking about automated updates. Is it possible to isolate the info we need using grep, cut, sed, etc., and kill them automatically? Are there risks involved?
Sri Syadasti Syadavaktavya Syadasti Syannasti Syadasti Cavaktavyasca Syadasti Syannasti Syadavatavyasca Syadasti Syannasti Syadavaktavyasca

Offline Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2732
  • Last Login:Today at 10:27:44 am
Re: DB hang 2/28/20
« Reply #4 on: March 28, 2020, 11:18:02 am »
Probably could.  I'd rather figure out why the processes hang in the first place.  Some earlier discussion:  https://github.com/TravelMapping/DataProcessing/issues/111