Author Topic: Project Statistics  (Read 7027 times)

0 Members and 1 Guest are viewing this topic.

Offline Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 1188
  • Last Login:Today at 04:36:14 pm
Project Statistics
« on: February 14, 2016, 12:31:46 am »
I haven't posted anything like this since we moved to this forum, so I thought it was time.

Our highway and user data now results in a database that gets constructed from 3,727,950 lines of SQL.  Over 20,000 .wpt files now get processed, along with 113 .list files.

A site update now takes about a half hour from start to finish.  When I get a chance over the summer to unify the regular database and the one that includes preview routes for mapping and stats, I expect we'll cut that nearly in half.  If I run it on my FreeBSD desktop instead of my Mac Book Pro, it also seems to be 25% faster, at least.  I'm sure other efficiency improvements will also present themselves the next time I can get my hands deep into the site update code.

Load on the server to serve maps and stats pages (including running the cgi scripts and DB queries) seems nearly negligible.

Offline michih

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 1014
  • Gender: Male
  • Last Login:Today at 04:09:21 pm
Re: Project Statistics
« Reply #1 on: February 14, 2016, 03:23:29 am »
How many users do mapping their travels / how many list files of real travelers do we have now?
I think CHM had about 350.

Offline Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 1188
  • Last Login:Today at 04:36:14 pm
Re: Project Statistics
« Reply #2 on: February 14, 2016, 08:46:52 am »
How many users do mapping their travels / how many list files of real travelers do we have now?
I think CHM had about 350.

It's right around 100.  We have a bunch of dummy files that are used to help find errors, etc. The next time I work on getting more statistics computation and reporting set up, I'd like to do something to flag those so that aren't listed in rankings, included in averages, etc.

Offline yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 1406
  • Last Login:Today at 10:56:34 am
Re: Project Statistics
« Reply #3 on: February 14, 2016, 12:16:34 pm »
It's right around 100.  We have a bunch of dummy files that are used to help find errors, etc. The next time I work on getting more statistics computation and reporting set up, I'd like to do something to flag those so that aren't listed in rankings, included in averages, etc.
On a related note, there are still some vestigial log files (yaxtest-ma138.log, yaxtest-ma79.log, yaxtest-us6.log) for dummy lists that I deleted. They're in the /logs directory but not /devlogs. (Maybe they predate the creation of devlogs...) Might be a good idea to clean up that directory / those directories, and/or automate a cleanup / rebuild-from-scratch as part of the site update process.

Offline michih

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 1014
  • Gender: Male
  • Last Login:Today at 04:09:21 pm
Re: Project Statistics
« Reply #4 on: February 14, 2016, 12:27:12 pm »
^^ These log files must be deleted manually (by Jim). There are also michih2.log files ;).

Offline Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 1188
  • Last Login:Today at 04:36:14 pm
Re: Project Statistics
« Reply #5 on: February 14, 2016, 03:28:11 pm »
It's right around 100.  We have a bunch of dummy files that are used to help find errors, etc. The next time I work on getting more statistics computation and reporting set up, I'd like to do something to flag those so that aren't listed in rankings, included in averages, etc.
On a related note, there are still some vestigial log files (yaxtest-ma138.log, yaxtest-ma79.log, yaxtest-us6.log) for dummy lists that I deleted. They're in the /logs directory but not /devlogs. (Maybe they predate the creation of devlogs...) Might be a good idea to clean up that directory / those directories, and/or automate a cleanup / rebuild-from-scratch as part of the site update process.

For now, I've done a quick cleanup.

Offline Thing342

  • Computer Guy
  • TM Collaborator
  • Newbie
  • *****
  • Posts: 13
  • Gender: Male
  • Last Login:June 08, 2017, 10:58:40 pm
  • V.T.C.S.
    • Personal Page
Re: Project Statistics
« Reply #6 on: February 15, 2016, 11:08:14 am »
Furthermore, the website now contains 40956 lines of code, with 2,488 being standard HTML/PHP stuff. Most of the other stuff is split among the SVG shield definitions and styling files.
« Last Edit: February 15, 2016, 01:49:33 pm by Thing342 »

Offline yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 1406
  • Last Login:Today at 10:56:34 am
Re: Project Statistics
« Reply #7 on: February 15, 2016, 01:06:07 pm »
Furthermore, the website now contains 40956 lines of code, with 2,488 being standard HTML/PHP stuff. Most of it is split among the SVG shield definitions and styling files.
What are the other 38,468?

Offline Thing342

  • Computer Guy
  • TM Collaborator
  • Newbie
  • *****
  • Posts: 13
  • Gender: Male
  • Last Login:June 08, 2017, 10:58:40 pm
  • V.T.C.S.
    • Personal Page
Re: Project Statistics
« Reply #8 on: February 15, 2016, 01:51:58 pm »
Furthermore, the website now contains 40956 lines of code, with 2,488 being standard HTML/PHP stuff. Most of it is split among the SVG shield definitions and styling files.
What are the other 38,468?
Mistyped my post. Most of the rest of the code is in SVG shield definitions, CSS files, and the WPT editor (which is written in JS)

Offline Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 1188
  • Last Login:Today at 04:36:14 pm
Re: Project Statistics
« Reply #9 on: February 20, 2016, 06:15:22 pm »
There are currently 404 .wpt files that do not have entries in .csv files.  Nothing wrong with that, as its perfectly reasonable to commit partial work on wpt files before they're ready for a csv file that will put them into the system in devel mode. 

But the fact that "404" were "not found" grabbed my attention.

Offline mapcat

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 616
  • Last Login:Yesterday at 11:20:12 pm
Re: Project Statistics
« Reply #10 on: February 20, 2016, 07:27:00 pm »
There are currently 404 .wpt files that do not have entries in .csv files.  Nothing wrong with that, as its perfectly reasonable to commit partial work on wpt files before they're ready for a csv file that will put them into the system in devel mode.

Where can we see a list of those? I'll check to see if I forgot any.

Quote
But the fact that "404" were "not found" grabbed my attention.
;D
Clinched:

Offline Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 1188
  • Last Login:Today at 04:36:14 pm
Re: Project Statistics
« Reply #11 on: February 20, 2016, 11:09:20 pm »
There are currently 404 .wpt files that do not have entries in .csv files.  Nothing wrong with that, as its perfectly reasonable to commit partial work on wpt files before they're ready for a csv file that will put them into the system in devel mode.

Where can we see a list of those? I'll check to see if I forgot any.

http://tm.teresco.org/logs/unprocessedwpts.log

Some of these belong, as they are part of not-yet-in-devel systems or (like some usava) are copies for routes that will later be moved in from other systems.  But there are quite a few that don't need to stay.  I'd consider it a low priority cleanup, but one worth doing when time permits.

Offline froggie

  • TM Collaborator
  • Sr. Member
  • *****
  • Posts: 304
  • Last Login:Today at 02:10:35 pm
Re: Project Statistics
« Reply #12 on: February 21, 2016, 07:21:12 am »
The following WPTs can be deleted.  They have been "turned back" (MnDOT-speak for decommissioned):

hwy_data/MN/usamn/mn.mn290.wpt
hwy_data/MN/usamn/mn.mn293.wpt
hwy_data/MN/usamn/mn.mn294.wpt
hwy_data/MN/usamn/mn.mn295.wpt
hwy_data/MN/usamn/mn.mn297.wpt
hwy_data/MN/usamn/mn.mn333.wpt
hwy_data/MN/usamn/mn.mn361.wpt
hwy_data/MN/usamn/mn.mn801.wpt
hwy_data/MN/usamn/mn.mn860.wpt


I'll have to look back and see why Tim deactivated MN 309.  Will also have to see why the leg of MN 23 through Wisconsin is on the "unprocessed list".

Offline si404

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 686
  • Last Login:Today at 02:21:57 pm
Re: Project Statistics
« Reply #13 on: February 21, 2016, 08:34:55 am »
Will also have to see why the leg of MN 23 through Wisconsin is on the "unprocessed list".
It's in the wrong folder (there's one in the right folder that is processed) - MN/usamn rather than WI/usamn.

I've made a pull request deleting the ones you listed, some in my regions that are there for no reason, and a handful of wrong folder files (checking that there were right folder equivalents)

Offline Bickendan

  • TM Collaborator
  • Full Member
  • *****
  • Posts: 205
  • Last Login:Today at 01:13:25 am
Re: Project Statistics
« Reply #14 on: February 21, 2016, 03:57:05 pm »
The yt.bc097s need to be checked -- I think they're appearing correctly in the BC directory, so the ones in the YT directory might be superfluous.
That said, I know that bc.bc097con isn't showing on the map.