Site didn't work (fixed)

[michih]

It seems that something failed with the last update.

Dunno why no one made a notification here. Is it a temorary problem only? That why I wann describe how the site works:

The start page is displayed currently from the top but the "Travel Mapping Highway Data" field ends with

Active system encompass 17,085 routes for
Query failed: SELECT SUM(activeMileage) AS s FROM overallMileageByRegion;

The rest of the page is blank. No info about last update, no footer.

Traveler Stats

Traveler Statistics
Query failed: SELECT SUM(activeMileage) AS s FROM overallMileageByRegion;

Highway Browser

works. You can select systems and routes but then there's an error message like:

Query failed: SELECT COUNT(*) as numDrivers, SUM(cr.clinched) as numClinched, @numUsers := ( SELECT COUNT(DISTINCT traveler) FROM clinchedOverallMileageByRegion ) as numUsers, ROUND(COUNT(*) / @numUsers * 100, 2) as drivenPct, ROUND(SUM(cr.clinched) / @numUsers * 100, 2) as clinchedPct, ROUND(SUM(cr.clinched) / COUNT(*) * 100, 2) as drivenClinchedPct, GROUP_CONCAT(cr.traveler SEPARATOR ', ') as drivers, GROUP_CONCAT(IF(cr.clinched = 1, cr.traveler, null) separator ', ') as clinchers, ROUND(AVG(cr.mileage), 2) as avgMileage, ROUND(r.mileage, 2) as totalMileage, ROUND(avg(cr.mileage) / r.mileage * 100, 2) as mileagePct FROM clinchedRoutes as cr JOIN routes as r ON cr.route = r.root WHERE cr.route = 'alb.a001'

Updates

Highway System Status Changes
Query failed: select * from systemUpdates;

[Jim]

Looks like there were some errors in the site update that scrolled by without me noticing that caused the DB update to fail.  There are some routes in the bihm system that have 0 or 1 points, which makes them invalid.  I'm commenting out bihm and starting another site update.

Reminder to all highway data contributors: please don't include any routes in csv files until they're substantially complete.

[michih]

Looks like there were some errors in the site update that scrolled by without me noticing that caused the DB update to fail.

Don't you see that the update is completed incorrectly? If not offline, maybe a quick check online? Just reload the start page when the upload is completed...

There are some routes in the bihm system that have 0 or 1 points, which makes them invalid.

M017, M017Dob and M0174.

I'm commenting out bihm and starting another site update.

Thanks!

[rickmastfan67]

Looks like there were some errors in the site update that scrolled by without me noticing that caused the DB update to fail.

Don't you see that the update is completed incorrectly? If not offline, maybe a quick check online? Just reload the start page when the upload is completed...

Jim most of the time starts the update right before he goes to sleep.  So, it's not always possible for him to do a double check.

[Jim]

Looks like there were some errors in the site update that scrolled by without me noticing that caused the DB update to fail.

Don't you see that the update is completed incorrectly? If not offline, maybe a quick check online? Just reload the start page when the upload is completed...

Yes, I normally watch for the most common errors (missing/misnamed .wpt, invalid .csv entries) that get reported at the start of the update process but the errors from last night are less common and get reported later.  I can see that the process completes or not, and I also usually load up the start page to make sure it all went well, but this one got launched late and the process now takes about 40 minutes, so I didn't stay up to check it out.

In any case, this is more evidence that I need to provide better testing capabilities that anyone who helps maintain highway data would be required to run before putting in a pull request.  It remains high on my list for the next time I get into the code in a significant way to provide this.  I'm also open to ideas about how to integrate some of this into GitHub itself, such as a check that every csv file line has the right number of fields, each wpt line is a valid format, each wpt has at least 2 entries, etc.   Eliminating those errors would save me a lot of time and frustration when I pull in changes and attempt to run a site update.

All that said, the quality of work all of you do is really great, and the vast majority of changes that get submitted require no manual intervention on my part beyond merging, pulling, and launching a site update.

[michih]

Thanks. The site is working again

I don't know if or how a data check could be executed on GitHub but contributers should have all highway data offline. If there would be a "data check application" on GitHub - e.g. in the HighwayData repository (that means one would "automatically" get it offline) - one could execute it before making a pull request.

Next technical question: How can I mark this thread "solved"? Why is the option not available on this board?

[Jim]

Jim most of the time starts the update right before he goes to sleep.  So, it's not always possible for him to do a double check.

I could run them earlier, but it seems the majority of .list updates come in between 8 and 11 PM Eastern each day, so I normally hold off as long as I can.

Fortunately, most errors cause the site update program to fail, not the DB update, so no one knows they happened.  The site continues with its old data until I fix the problems and re-run.

An easy fix on my end that I hope to put in today is to add a flag that makes sure if any error with the potential to cause a DB update failure is detected, the process will stop before generating the big .sql file and trying to use it to repopulate the DB.

[michih]

Last update failed. Site is currently not working.

Message from start page:

Query failed: SELECT COUNT(*) AS c FROM connectedRoutes LEFT JOIN systems ON connectedRoutes.systemName = systems.systemName WHERE systems.level = 'active';

[Jim]

Duplicate entries in the new Iceland work slipped through and I went to bed before waiting to see if the update finished. I've fixed the error and just launched a new update.  Should be good by around 9:30 EDT.

[michih]

Thanks. Seems to be fine now