Author Topic: TM data, logs, stats, etc. archives  (Read 6016 times)

0 Members and 1 Guest are viewing this topic.

Offline Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2732
  • Last Login:Today at 02:09:10 pm
TM data, logs, stats, etc. archives
« on: May 27, 2019, 03:43:54 pm »
For about the last year, the site update process has archived the previous state of TM as part of each update.  I had pretty much forgotten it was keeping things for so long.  It takes a lot of storage (about 357 GB at the moment) but the server has no shortage of disk space at this point, so I plan to keep saving things.  So if you ever want to have access to an old log file, old stats about the contents of a highway system, etc., you can send along a request by email to me or in a PM here on the forum.   Saved for each site update are the .sql file used to populate the DB, and the contents of the graphdata, lots, nmp_merged, and stats directories.

Also from these archives, tools like "grep" can easily generate fun output like one attached which shows my overall stats in active+preview systems over the last year.  Nice to see I've had a productive year, adding 3000 miles between new travels and previous travels on systems that have reached preview or active status in TM.

Offline yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 4234
  • Last Login:February 13, 2024, 07:19:36 pm
  • I like C++
Re: TM data, logs, stats, etc. archives
« Reply #1 on: May 27, 2019, 05:05:27 pm »
I remember digging into these files a bit when troubleshooting... something, during C++ development.

Out of curiosity, what was the grep command you used? I see just a partial line, of the relevant stats sans the "Overall in active+preview systems:" bit.
Sri Syadasti Syadavaktavya Syadasti Syannasti Syadasti Cavaktavyasca Syadasti Syannasti Syadavatavyasca Syadasti Syannasti Syadavaktavyasca

Offline Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2732
  • Last Login:Today at 02:09:10 pm
Re: TM data, logs, stats, etc. archives
« Reply #2 on: May 27, 2019, 05:12:51 pm »
I cheated.  I used

Code: [Select]
grep "Overall in active+preview" */logs/users/terescoj.log
but then edited out the extra redundant bits in the file.

Edit: this one would give the output I posted:

Code: [Select]
grep "Overall in active+preview" */logs/users/terescoj.log | sed "s=/logs/users/terescoj.log:Overall in active+preview systems==g"
« Last Edit: May 27, 2019, 05:15:07 pm by Jim »

Offline michih

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 4555
  • Last Login:Today at 04:44:01 pm
Re: TM data, logs, stats, etc. archives
« Reply #3 on: May 28, 2019, 10:26:43 am »
It takes a lot of storage (about 357 GB at the moment) but the server has no shortage of disk space at this point, so I plan to keep saving things.

I'd love seeing historical mileage progress on the site one day :)
I think that it would be enough to store data of the first* update each year or each month (or week?) if there would be any shortage one day...

*Which would represent the last update of the previous year/month/week.

Offline yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 4234
  • Last Login:February 13, 2024, 07:19:36 pm
  • I like C++
Re: TM data, logs, stats, etc. archives
« Reply #4 on: April 04, 2020, 04:42:00 am »
Saved for each site update are the .sql file used to populate the DB, and the contents of the graphdata, lots, nmp_merged, and stats directories.
This just came in handy, as it turns out. I just grabbed one of the .sql files from when this issue was live, and installed it on lab2. It seems MySQL 5.7 & MySQL 8.0 differ in some pretty significant ways on how they handle multi-byte characters and over-length fields.

Sri Syadasti Syadavaktavya Syadasti Syannasti Syadasti Cavaktavyasca Syadasti Syannasti Syadavatavyasca Syadasti Syannasti Syadavaktavyasca