Travel Mapping

Web Design Discussion => General Web Design Discussion => Topic started by: Markkos1992 on September 21, 2020, 06:35:01 pm

Title: Connected Route Question
Post by: Markkos1992 on September 21, 2020, 06:35:01 pm
This is something that has confused me since we made it doable to see entire Connected Routes.

The Connected Route page for DE/PA/NJ I-295 (https://travelmapping.net/hb/showroute.php?units=miles&u=markkos1992&r=nj.i295&cr)  currently has 139 drivers with 129 having clinched it.

However, when you look at the specific state pages, you get this:

DE: 110 drivers, 109 clinched (noting that this is one of if not the most traveled sections of route in TM overall)
NJ:  110 drivers, 57 clinched
PA:  65 drivers, 55 clinched

So theoretically, there should be at most 55 people (including myself) that have clinched the entire route.  I wonder what is in the program causing these odd discrepancies.
Title: Re: Connected Route Question
Post by: Jim on September 21, 2020, 07:56:44 pm
Doesn't sound right for sure.

I don't have time to check it out immediately, so I've created a GitHub Issue to make sure it does get investigated.

https://github.com/TravelMapping/Web/issues/542
Title: Re: Connected Route Question
Post by: yakra on September 24, 2020, 09:42:49 pm
Quote from: siteupdate.log
HighwayData @ 4001f70ba7d298b2a019114696a0fce72455f76b
UserData @ bafd52681fb7ac64062a1aa7e563d351f543261a
DataProcessing @ 11be967a819fcd28f05582cb1050a721494c3596
Start: 2020-09-23 21:49:20.690292

The Connected Route page for DE/PA/NJ I-295 (https://travelmapping.net/hb/showroute.php?units=miles&u=markkos1992&r=nj.i295&cr)  currently has 139 drivers with 129 having clinched it.
However, when you look at the specific state pages, you get this:
DE: 110 drivers, 109 clinched
NJ:  110 drivers, 57 clinched
PA:  65 drivers, 55 clinched
The HB still reports this as of this writing, on both noreaster & lab2.

user logs:
ConnectedRoute has 160 drivers; 42 clinched
Code: [Select]
[yakra@noreaster /home/www/tm/logs/users]$ grep 'Delaware - New Jersey - Pennsylvania' *.log | wc -l
     160
[yakra@noreaster /home/www/tm/logs/users]$ grep 'Delaware - New Jersey - Pennsylvania.*(100\.00%)$' *.log | wc -l
      42

Choped routes:
Code: [Select]
[yakra@noreaster /home/www/tm/logs/users]$ grep 'DE I-295:' *.log | wc -l
     144
[yakra@noreaster /home/www/tm/logs/users]$ grep 'DE I-295:.*(100\.00%)$' *.log | wc -l
     138
[yakra@noreaster /home/www/tm/logs/users]$ grep 'NJ I-295:' *.log | wc -l
     152
[yakra@noreaster /home/www/tm/logs/users]$ grep 'NJ I-295:.*(100\.00%)$' *.log | wc -l
      57
[yakra@noreaster /home/www/tm/logs/users]$ grep 'PA I-295:' *.log | wc -l
      65
[yakra@noreaster /home/www/tm/logs/users]$ grep 'PA I-295:.*(100\.00%)$' *.log | wc -l
      55
Same results on lab2.

db queries on lab2:
Code: [Select]
mysql> select * from clinchedConnectedRoutes where route = 'de.i295';
160 rows in set (0.01 sec)
mysql> select * from clinchedConnectedRoutes where route = 'de.i295' and clinched = '1';
42 rows in set (0.00 sec)
mysql> select * from clinchedRoutes where route = 'de.i295';
144 rows in set (0.00 sec)
mysql> select * from clinchedRoutes where route = 'de.i295' and clinched = '1';
138 rows in set (0.00 sec)
mysql> select * from clinchedRoutes where route = 'nj.i295';
152 rows in set (0.00 sec)
mysql> select * from clinchedRoutes where route = 'nj.i295' and clinched = '1';
57 rows in set (0.00 sec)
mysql> select * from clinchedRoutes where route = 'pa.i295';
65 rows in set (0.00 sec)
mysql> select * from clinchedRoutes where route = 'pa.i295' and clinched = '1';
55 rows in set (0.00 sec)
...whether a C++ or Python-flavored siteupdate.

It appears something goes wrong on the Web (PHP?) side while/after getting results from the DB.
Title: Re: Connected Route Question
Post by: yakra on September 24, 2020, 11:11:24 pm
If I had some way to conveniently copy the tooltip text for Total Drivers & Total Clinched, maybe that could provide an avenue to explore...
Title: Re: Connected Route Question
Post by: Jim on September 25, 2020, 08:24:57 am
Offending query seems likely to be here:

https://github.com/TravelMapping/Web/blob/02861e7723d7bc64f937fd59277359677c4e94c5/lib/getRouteStats.php#L51
Title: Re: Connected Route Question
Post by: yakra on September 25, 2020, 10:51:52 am
I just ran that query on lab2 for de.i295, nj.i295 and pa.i295.
Results were 144, 138; 152, 57; 65, 55.
Are you getting different results on noreaster?

How about moving this topic to the public side of the forum, to get more sets of eyes on it? There was that time mikandkristie figured out some SQL issues...
Title: Re: Connected Route Question
Post by: yakra on December 03, 2020, 08:16:07 pm
Quote from: siteupdate.log on both noreaster and lab2
HighwayData @ 9c66c8418df9657c20b1f1a99f82bb30cebcb4d6
UserData @ 09c99fc211a14ab8fae2dcf4e35521428b3c00d0
DataProcessing @ a905d2a / 3ac929e, same thing.
noreaster (https://travelmapping.net/hb/showroute.php?units=miles&u=harvey&r=ga.us023&lat=32.844512&lon=-83.628402&zoom=14&cr) (MySQL 5.7?) shows 185 Total Drivers on US23; lab2 (MySQL 8.0) shows 186.
Title: Re: Connected Route Question
Post by: Markkos1992 on December 29, 2020, 11:56:22 am
Looking at MD I-95 (https://travelmapping.net/hb/showroute.php?units=miles&u=markkos1992&r=md.i095) this morning, I saw the following:

Total drivers: 109
Total clinched: 107 (98.17%)
Average traveled: 86.20 miles (79.15%)

Expanding that to all of I-95 (https://travelmapping.net/hb/showroute.php?units=miles&u=markkos1992&r=md.i095&cr) has 202 out of 208 drivers having clinched.  I wonder if there is more going on than I originally suspected.  While I would expect many travelers to clinch I-95 in MD, I do not see how only two drivers would not have.


Title: Re: Connected Route Question
Post by: mapmikey on December 29, 2020, 02:13:04 pm
I know nothing about the programming that goes into TM but I can offer this which might be helpful in diagnosing:

If you pull up each individual state for I-5, you can hover the cursor at the lower part of "total clinched" and see the actual usernames that have fully clinched.  I did this for CA, OR and WA and wrote down the usernames.  Then I looked at the connected I-5 screen and it shows 51 clinching all of I-5.  When you hover over the "total clinched" on that screen, you get the clinched names from CA, followed by the clinchers of OR who were not on the CA list, followed by the WA clinchers who were not on either CA or OR.  In other words, the number of people who have clinched at least one full I-5 segment are given full clinch credit in the count.

Each individual I-5 page does appear to have the number of clinchers correct in that state.

Whatever causes that is likely doing a similar thing with distances.

Title: Re: Connected Route Question
Post by: Markkos1992 on December 29, 2020, 04:57:50 pm
Looking at MD I-95 (https://travelmapping.net/hb/showroute.php?units=miles&u=markkos1992&r=md.i095) this morning, I saw the following:

Total drivers: 109
Total clinched: 107 (98.17%)
Average traveled: 86.20 miles (79.15%)

Expanding that to all of I-95 (https://travelmapping.net/hb/showroute.php?units=miles&u=markkos1992&r=md.i095&cr) has 202 out of 208 drivers having clinched.  I wonder if there is more going on than I originally suspected.  While I would expect many travelers to clinch I-95 in MD, I do not see how only two drivers would not have.




I created a spreadsheet that I put in as a Google Doc to expand on this. (https://docs.google.com/spreadsheets/d/1Pkp_jPBWr34OYsf1w5EJmnakQrnAj9q_3hJ6mQtRcYo/edit?usp=sharing)

Searching "MD I-95" in Github got me 186 travelers in MD on I-95 with 105 having clinched it.  I know that this may not be exact due to some files using the multi-region lines now, but it should give a good idea of how much the numbers are off.
Title: Re: Connected Route Question
Post by: yakra on December 29, 2020, 08:11:37 pm
Searching "MD I-95" in Github got me 186 travelers in MD on I-95 with 105 having clinched it.  I know that this may not be exact due to some files using the multi-region lines now, but it should give a good idea of how much the numbers are off.

Code: [Select]
[yakra@noreaster /home/www/tm/logs/users]$ grep '  MD I-95: .* mi (.*)' * | wc -l
     189
[yakra@noreaster /home/www/tm/logs/users]$ grep '  MD I-95: .* mi (100.00%)' * | wc -l
     107
Title: Re: Connected Route Question
Post by: Markkos1992 on December 29, 2020, 08:59:02 pm
yakra, is there a way to find out which two list files are being counted in the "109" number? I think it is clear that the 107 number having clinched it is correct.
Title: Re: Connected Route Question
Post by: Jim on December 29, 2020, 09:35:46 pm
Thanks for the reminders on this.  I'm taking a look now.  I'm currently not suspecting the SQL queries, but how the JS code uses the results to populate those stats.
Title: Re: Connected Route Question
Post by: Jim on December 29, 2020, 09:53:39 pm
The MD I-95 case is fixed with a mysqld parameter, as the results of the query to get the lists of drivers/clinchers were being truncated to 1024 characters.  No TM code changes for that.
Title: Re: Connected Route Question
Post by: Jim on December 29, 2020, 09:55:48 pm
I know nothing about the programming that goes into TM but I can offer this which might be helpful in diagnosing:

If you pull up each individual state for I-5, you can hover the cursor at the lower part of "total clinched" and see the actual usernames that have fully clinched.  I did this for CA, OR and WA and wrote down the usernames.  Then I looked at the connected I-5 screen and it shows 51 clinching all of I-5.  When you hover over the "total clinched" on that screen, you get the clinched names from CA, followed by the clinchers of OR who were not on the CA list, followed by the WA clinchers who were not on either CA or OR.  In other words, the number of people who have clinched at least one full I-5 segment are given full clinch credit in the count.

Each individual I-5 page does appear to have the number of clinchers correct in that state.

Whatever causes that is likely doing a similar thing with distances.

Good catch - investigating and should be fixed soon.
Title: Re: Connected Route Question
Post by: yakra on December 29, 2020, 10:26:48 pm
Code: [Select]
From https://github.com/TravelMapping/Web
   96e1a70..f72a08d  master     -> origin/master
Updating 96e1a70..f72a08d
Fast-forward
 README.md | 2 ++
 1 file changed, 2 insertions(+)
yakra@BiggaTomato:~/TravelMapping/Web$ git log -n 1 README.md
commit f72a08d7a1de1a821211cccb3f1efafb0538e2b5 (HEAD -> master, origin/master, origin/HEAD)
Author: Jim Teresco <terescoj@gmail.com>
Date:   Tue Dec 29 22:01:24 2020 -0500

    Note the change to group_concat_max_len DB server parameter

The MD I-95 case is fixed with a mysqld parameter, as the results of the query to get the lists of drivers/clinchers were being truncated to 1024 characters.  No TM code changes for that.

I'll have to check this out -- noreaster correctly shows 189 Total Drivers; lab2 shows 109.
Title: Re: Connected Route Question
Post by: yakra on December 29, 2020, 10:35:29 pm
Jim is this in the my.cnf file? Do I want to add in a
Code: [Select]
group_concat_max_len = 10000?

Edit: Yes.
Title: Re: Connected Route Question
Post by: Jim on December 29, 2020, 10:56:58 pm
I believe I've got the fix up on the main site.  Looking at various usai and usaus routes, I am seeing much more reasonable results.  Thanks again to mapmikey for the observation that pointed out the problem with my new showroute code from last summer.
Title: Re: Connected Route Question
Post by: yakra on December 29, 2020, 11:43:04 pm
Thanks again to mapmikey for the observation
Yes. Very well done.
Title: Re: Connected Route Question
Post by: Markkos1992 on December 30, 2020, 08:08:47 am
Thanks everyone.  I agree that the numbers look to be much more realistic now.
Title: Re: Connected Route Question
Post by: mapmikey on December 30, 2020, 08:57:50 am
Thanks again to mapmikey for the observation
Yes. Very well done.

Happy to have been helpful...