Difference in Statistics -- December 2015 to December 2016

[Rothman]

I e-mailed Jim about this tonight and he told me to post it in the forum in case there isn't as simple an explanation as I thought there would have been (if you can follow that):

I was digging into a little history.  Last December, on the interstate highway system, according to the site, I had clinched 29,483 miles out of 49,827 (59%).  Added some mileage and lost I-164 mileage (since it doesn't exist anymore, I don't count it) since then -- checked all the .list files I submitted between then and now and definitely added mileage -- no log errors.  The site now says (nearly a year to the day) that I clinched 29,293 out of 48,570 (just over 60%).   So, somehow lost 190 miles and the interstate system lost over 1,200 miles, but I still have clinched more of it.

Anyone have any idea what happened here?  I was hoping it was just because kinks were working their way out of the calculating engine a year ago and things are more accurate now.

[mapcat]

Check where on the site you're getting your stats; sometimes the totals are different. I think at some point the explanation was something about concurrencies not being taken into account.

Example: my interstate mileage on this page http://tm.teresco.org/user/?u=mapcat:

Code: [Select]

USA   usai   United States Interstate Highways   Tier 1   active   46028.20   49960.11   92.13%   Map   HB
differs from my interstate mileage on this one http://tm.teresco.org/user/system.php?u=mapcat&sys=usai

Code: [Select]

Miles Driven   44639.72 of 48570.06 mi (91.91%) Rank: 2


[yakra]

The first thing that comes to my mind is: Multiplexes.

If I-50 and I-60 are multiplexed together, all the point coordinates along their concurrency must match up perfectly, or else the concurrency will not be detected.
For every segment where the points aren't an exact match, the mileage will be counted twice.

I and others have been looking thru the data we maintain, making sure that things that *should* be concurrencies *are* counted as concurrencies, and making edits as needed. There's also been an effort among a few of us to clean up "near miss points", which should also fix broken concurrencies in the process.

This should reduce the mileage for the interstates a little bit, but not 1257 miles worth, especially when you take into account Future Interstates being upgraded to full Interstates & other new additions to the system.
On that note, mapcat's post hits the nail on the head. The difference between the mileage quoted on the two pages is 1390.05 mi, which puts us in just about the right ballpark. Add in some mileage for new additions to the system and subtract some mileage for fixing broken multiplexes, and I can easily see that putting us near 1257.

[bejacob]

Check where on the site you're getting your stats; sometimes the totals are different. I think at some point the explanation was something about concurrencies not being taken into account.

It's one thing to be talking about differences between last year and this. It's another to be noting that different parts of the site show different numbers. The latter begs the questions, "which is the right number?"

My Traveler Stats page http://tm.teresco.org/user/?u=bejacob has Interstate mileage of 32,137.42 in the Stats by System section. When I click on that section to go to the usai system http://tm.teresco.org/user/system.php?u=bejacob&sys=usai, the number is 30,912.19. That is the same as the number showing in my log file.

It should also be noted that the total mileage from the usai system drops from 49.960.11 to 48,750.06 between these two pages.

Given that the system specific stats and the log file match, it appears the overall Traveler Stats page is somehow pulling the wrong info. Cleaning up the data and double checking multiplexes aside, at any point in time, the site should be reporting the same data regardless of which part of the site one is viewing.

I admit, I tend to look more at the map pages and at the percentages, rather than the total mileage, so this is not one of my major concerns. It would be nice, however, to know that at any snapshot in time, each page will be displaying the same statistics.

[mapcat]

It's one thing to be talking about differences between last year and this. It's another to be noting that different parts of the site show different numbers. The latter begs the questions, "which is the right number?"

I just meant that he may have collected his stats from one page last year and a different page this year.

[Rothman]

Could have sworn that I snagged the number both times from the first line of the "System Stats" table on my Traveler Stats page while using the usai parameter in the URL, but I suppose there is the possibility I added up all the routes together, which would have captured the concurrencies as has been mentioned.

We'll see where we are next year.

Thanks, everyone.

ETA:  Hm. I do have ye olde .list file from back then.  Probably could check it out somehow, even though it'd be against the current dataset.

[yakra]

Could have sworn that I snagged the number both times from the first line of the "System Stats" table on my Traveler Stats page while using the usai parameter in the URL, but I suppose there is the possibility I added up all the routes together, which would have captured the concurrencies as has been mentioned.

It could also be that maybe, back then, the data in the "System Stats" table on the Traveler Stats page were not yet calculated to take multiplexes into account. Dunno; I'm rusty on when exactly that change first happened, and I can't be arsed to try to look it up right now.

ETA:  Hm. I do have ye olde .list file from back then.  Probably could check it out somehow, even though it'd be against the current dataset.

As a useful aside: You can view your .list file at any point in its history on GitHub: when viewing your .list file in the UserData repo, click on the "History" button in the gray header bar above the .list code itself. That'll take you to a page listing all the commits to the file since it was first uploaded.
ETA: You would also be able to view the whole HighwayData repo from the same point in history, with a little work. Dunno about how to go about downloading it easily. Even so, setting up your own means to calculate stats from it (with/without multiplexes) would be the real PitA. I suppose you could download the DataProcessing code (and Web code too, probably?) and set up a version of the site on your own machine -- probably with the code from that same point in time if you wanted to be sure you were creating the same info. Also potentially a PitA (it would be in my case) depending on your level  of technical expertise. It'd be your call on whether or not this would all be worth the trouble of course.

[Jim]

I'm also thinking concurrencies are the issue here.  I will go back through the computations and make sure they're consistent, and mean what we think they mean.  It was one of the hardest things to do right when I was setting up the TM data processing infrastructure and it would not surprise me at all if there are some bugs still lurking.

[Rothman]

(Boy, I'm out of it...issue already mentioned)

[bejacob]

My Traveler Stats page http://tm.teresco.org/user/?u=bejacob has Interstate mileage of 32,137.42 in the Stats by System section. When I click on that section to go to the usai system http://tm.teresco.org/user/system.php?u=bejacob&sys=usai, the number is 30,912.19. That is the same as the number showing in my log file.

Something went wrong in the latest update. Looking at the Stats by System section http://tm.teresco.org/user/?u=bejacob, my usai now shows 1,607,433.95 miles driven out of 2,428,502.75 miles 66.19%. Much as I'd love to be over 1M miles  , I know the correct number should be 30,912.19 which shows fine here http://tm.teresco.org/user/system.php?u=bejacob&sys=usai. Several other systems seem to have the same problem (usaif, usaib, usaus, usansf, usasf, usausb, eure, gbmn). I'm sure there are others, but these are the ones I noticed on my travels.

 

[Jim]

My Traveler Stats page http://tm.teresco.org/user/?u=bejacob has Interstate mileage of 32,137.42 in the Stats by System section. When I click on that section to go to the usai system http://tm.teresco.org/user/system.php?u=bejacob&sys=usai, the number is 30,912.19. That is the same as the number showing in my log file.

Something went wrong in the latest update. Looking at the Stats by System section http://tm.teresco.org/user/?u=bejacob, my usai now shows 1,607,433.95 miles driven out of 2,428,502.75 miles 66.19%. Much as I'd love to be over 1M miles  , I know the correct number should be 30,912.19 which shows fine here http://tm.teresco.org/user/system.php?u=bejacob&sys=usai. Several other systems seem to have the same problem (usaif, usaib, usaus, usansf, usasf, usausb, eure, gbmn). I'm sure there are others, but these are the ones I noticed on my travels.

I'm sure the DB is still good, since the stats are OK in other places.  Looks like my new SQL code to fix the concurrency problem still has some issues...

[Jim]

I think it's good now.  Let me know if you still see errors.

[bejacob]

I think it's good now.  Let me know if you still see errors.

It appears to be okay now. I'll dig a bit deeper to see if anything else seems out of whack. Thanks for fixing this so quickly.