yakra's collaborator thread

[yakra]

grep-o-roonie
inna USA/CAN stylee

1. Potential malformed city suffixes, including directional
grep '^[A-Z][A-Z][0-9]\{1,3\}[A-Z][a-z][a-z]_[NEWS]' */*/*.wpt | grep -v '[A-Z][A-Z][0-9]\{1,3\}Alt_[NEWS]\|[A-Z][A-Z][0-9]\{1,3\}Bus_[NEWS]\|[A-Z][A-Z][0-9]\{1,3\}Byp_[NEWS]\|[A-Z][A-Z][0-9]\{1,3\}Trk_[NEWS]\|[A-Z][A-Z][0-9]\{1,3\}His_[NEWS]\|[A-Z][A-Z][0-9]\{1,3\}Spr_[NEWS]\|[A-Z][A-Z][0-9]\{1,3\}Con_[NEWS]'

2. Potential malformed city suffixes, more complete list
grep '^[A-Z][A-Z][0-9]\{1,3\}[A-Z][a-z][a-z]' */*/*.wpt | grep -v '[A-Z][A-Z][0-9]\{1,3\}Alt\|[A-Z][A-Z][0-9]\{1,3\}Bus\|[A-Z][A-Z][0-9]\{1,3\}Byp\|[A-Z][A-Z][0-9]\{1,3\}Trk\|[A-Z][A-Z][0-9]\{1,3\}His\|[A-Z][A-Z][0-9]\{1,3\}Spr\|[A-Z][A-Z][0-9]\{1,3\}Con'

3. "Good Alabama"
grep _ AL/*/*.wpt | grep -v '_[NEWS] \|_[A-Za-z]\{4\}'
A few false positives.

4. "Bad Maine"
grep 'US[0-9]\{1,3\}Alt.\+ ' ME/*/*.wpt | grep -v 'US[0-9]\{1,3\}Alt_[NEWS] '

5. 4-character LONG_UNDERSCORE datacheck
grep -i '_[A-Z]\{3\}[abcdfghijklmopqrtuvxyz] ' */*/*.wpt

Ubuntu:	grep '_...[A-DF-MO-RTUVXYZa-z] ' */*/*.wpt
Error results:	for rg in `ls`; do grep -s '_...[A-DF-MO-RTUVXYZa-z] ' $rg/*/*.wpt; done
Error counts:	for rg in `ls`; do echo -en "$rg\t"; grep -s '_...[A-DF-MO-RTUVXYZa-z] ' $rg/*/*.wpt | wc -l; done
Good results:	for rg in `ls`; do grep -s '_...[NEWS] ' $rg/*/*.wpt; done
Good counts:	for rg in `ls`; do echo -en "$rg\t"; grep -s '_...[NEWS] ' $rg/*/*.wpt | wc -l; done

6. Old highway designation labels that may have a current visible name
grep '^Old..[0-9]' */*/*.wpt
This one will have a lot of false positives.

[yakra]

Operation Good Labels
MyRegions='CT MA ME NH RI KS NE OK NJ NY TX AB MB NB NS PE NL'
NewEngland='CT MA ME NH RI'
CentralStates='KS NE OK'
Canada='AB MB NB NS PE NL'
MiscRegions='NJ NY TX'

Operation 4-Char Suffix (with NEWS)
CT(14) MA(6) ME(33) NH(12) RI(12)
for rg in $NewEngland; do grep '_.... ' $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep '_....$'; done

Are all city suffixes appropriate?
Excluding 4-char cases with directional "subfix", as those are covered above
CT(7) MA(13) ME(69) NH(24) RI(1)
NE(7) OK(17)
NJ(3) NY(51) TX(3)
AB(8) NB(4) NS(17)
grep -v '^+' $rg/*/*.wpt | cut -f1 -d' ' | egrep ':[A-Z]{2}[0-9]+.*_...$'
grep -v '^+' $rg/*/*.wpt | cut -f1 -d' ' | egrep ':[A-Z]{2}[0-9]+.*_...$' | sed -r 's~.*/(.*).wpt.*~\1~' | uniq | sed 's~.*~http://192.168.1.20/hb/showroute.php?r=&~' | xargs firefox

Operation 2-Char Suffix
Excludes 4-char cases with directional "subfix"; those are covered above
CT(1) KS(3) MA(1) ME(11) NB(2)
for rg in $MyRegions; do grep '_.. ' $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep -v ':+' | grep '_..$'; done

Operation Too Many Words
CT(5) MA(1) ME(4) NE(1) NH(5) NY(6) NB(16) NL(29)
for rg in $MyRegions; do grep -s '[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]*' $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep '[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]*'; done
Not sufficient to match all cases. Refine regex & redo.

FooBarBRd
CT(1) MA(2) ME(2) NH(12) (17)
KS(3) NE(7) OK(10) (20)
NJ(5) NY(83) TX(9) (97)
AB(3) NB(25) NS(6) PE(1) NL(3) (38)
for rg in $MyRegions; do grep '[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]\?[A-Z]\+[a-z]*' $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep -v '[NPS]P$' | grep '[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]\?[A-Z]\+[a-z]*'; done
for rg in $MyRegions; do grep -s '[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]\?[A-Z]\+[a-z]*' $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep -v '[NPS]P$' | grep '[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]\?[A-Z]\+[a-z]\+'; done

FooBBazRd
CT(1) MA(1) ME(7) NH(1) (10)
NJ(2) NY(8) TX(1) (11)
OK(1) (1)
NB(6) NL(4) (10)
for rg in $MyRegions; do grep -s '[A-Z]\+[a-z]\+[A-Z]\+[a-z]\?[A-Z]\+[a-z]\+[A-Z]\+[a-z]*' $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep -v ':Mc[A-Z][A-Z]' | grep '[A-Z]\+[a-z]\+[A-Z]\+[a-z]\?[A-Z]\+[a-z]\+[A-Z]\+[a-z]*'; done

FBarBazRd
CT(3) MA(1) ME(2) NH(3) RI(1) (10)
NY(28) TX(2) (30)
NB(16) NS(2) PE(1) NL(1) (20)
NE(1) (unprocessed WPT deleted) (1)
for rg in $MyRegions; do grep -s '[A-Z]\+[a-z]\?[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]*' $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep '[A-Z]\+[a-z]\?[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]*'; done

"All of the Above" mammoth regex
for rg in `ls`; do grep -s . $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep -v '[NPS]P$\|:[A-Z]*[a-z]*Mc[A-Z][A-Z][a-z]*$\|:Mc[A-Z][A-Z]\+[a-z]*[A-Z][a-z]*$\|:[A-Z]*Mc[A-Z]\+[a-z]\?$' | grep '[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]*\|[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]\?[A-Z]\+[a-z]\+\|[A-Z]\+[a-z]\+[A-Z]\+[a-z]\?[A-Z]\+[a-z]\+[A-Z]\+[a-z]*\|[A-Z]\+[a-z]\?[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]*'; done

Operation _End
MA(2) ME(3) NE(1) NH(2) OK(1) RI(2) AB(1)
for rg in $MyRegions; do grep _End $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep _End; done

Operation Predir
This got its own post

Operation Postdir
CT(6) ME(7) NH(5) (18)
KS(16) NE(1) (17)
NJ(3) NY(10) TX(6) (19)
MB(2) NB(9) NS(3) NL(5) (19)
for rg in $MyRegions; do grep '[NEWS]' $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep '[NEWS]$' | grep -v ':+\|_\|[0-9]\{1,3\}[NEWS]$\|:[A-Z][A-Z]/[A-Z][A-Z]$\|:CR[A-Z]$\|:Rd[A-Z]$\|:Ave[A-Z]$\|:I\-[0-9]\{1,3\}BS$'; done

Operation Long Words
ME(1) NL(1)
for rg in $MyRegions; do grep . $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep -v ':+' | grep ':.*[a-z]\{4\}'; done

Operation McPotrzebie
MA(1) ME(2)
AB(2)
NJ(6) NY(6) TX(4)
for rg in $MyRegions; do grep Mc[A-Z][a-z] $rg/*/*.wpt; done
for rg in $Canada; do grep Mac $rg/*/*.wpt; done

Tpke -> Tpk
CT(7) MA(1) NH(10) RI(1)
OK(2)
NJ(4) NY(19) TX(1)
for rg in $MyRegions; do grep -s Tpke $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep 'Tpke'; done

[yakra]

Saving a few browser tabs...

Window 1
https://github.com/TravelMapping/HighwayData/pull/3073

Window 2
https://github.com/TravelMapping/HighwayData/pulls/
https://github.com/TravelMapping/HighwayData/pull/3093

Window 3
https://github.com/TravelMapping/HighwayData/pull/3115
https://github.com/TravelMapping/HighwayData/pull/3115/files
GMSV
GMSV -> GMSV
GMSV
wptedit:
AniPer http://www.openstreetmap.org/?lat=19.483130&lon=-99.029721
+X169273 http://www.openstreetmap.org/?lat=19.486867&lon=-99.027940
CTM14 http://www.openstreetmap.org/?lat=19.488021&lon=-99.017533
EMEX5D_Per http://www.openstreetmap.org/?lat=19.487826&lon=-99.015865

Window 4
http://forum.travelmapping.net/index.php?topic=3147
http://forum.travelmapping.net/index.php?topic=3165
http://forum.travelmapping.net/index.php?topic=3167
http://forum.travelmapping.net/index.php?topic=3175
http://forum.travelmapping.net/index.php?topic=3176.msg15305#msg15305

[yakra]

Ideas to reject. Or not.

abbrev_as_suffix
These can show up in graph labels. 31 examples in:
egrep -i '([A-Z]{3})_\1' waypointsimplification.log
egrep -i '([A-Z]{3})_\1' tm-master-simple.tmg
I may have some data to fix; there's stuff in my regions
30 cases of 3+ intersection + one Exit/Intersection out of 430031 collapsed vertices. May well be fixable on the waypoint simplification side.

datacheck-style pseudocode

Code: [Select]

if (underscore != string::npos)
  for (Waypoint *p : *(w->colocated))
  { if (p == w) continue;
if ( p->route->abbrev.size()
     && w->label.substr(underscore+1, p->route->abbrev.size()) == p->route->abbrev
     && w->label.substr(0, underscore) == p->route->name_no_abbrev()
   ) { datacheckerrors->add(&r, w->label, "", "", "ABBREV_AS_SUFFIX", p->route->abbrev);
break;
     }
  }
/*TODO: else for (Waypoint *p : *(w->colocated))
     { if (p == w) continue;
if (regex("[0-9]+[A-Za-z]{3}") in label)
if ( p->route->route ends with same [0-9]+ \
and not p.route.banner.startswith(same [A-Za-z]{3}) \
and not p.route.abbrev.startswith(same [A-Za-z]{3})
   ) { datacheckerrors->add(&r, w->label, "", "", "FALSE_ABBREV", "Lorem Ipsum");
break;
     }
     }//*/
      }
Not really worth doing as a datacheck. A child route can intersect or overlap its parent midway, resulting in a city_suffix: see MA122@MA122A_Wor. These cases alone + desire to be consistent with otherwise-same-style labels + enough of a gray area in the manual = abbrev_as_suffix should be accepted as normal, albeit awkward for graph labels.
false_abbrev or unexpected_abbrev, OTOH...


LABEL_SELFREF
yakra@BiggaTomato:~/TravelMapping/yakra/DataProcessing/siteupdate/cplusplus/logs$ diff waypointsimplification.old waypointsimplification.i5 | egrep '^< Straightforward concurrency'
< Straightforward concurrency: L3238Her@L3238&L3238Mar@L3238 -> L3238Her/L3238Mar@L3238
< Straightforward concurrency: B279@B279&B279Hol@B279 -> B279/B279Hol@B279
< Straightforward concurrency: A329@A329_E&A329Rea@A329_E -> A329/A329Rea@A329_E
yakra@BiggaTomato:~/TravelMapping/yakra/DataProcessing/siteupdate/cplusplus/logs$ egrep 'L3238Her@L3238&L3238Mar@L3238|B279@B279&B279Hol@B279|A329@A329_E&A329Rea@A329_E' waypointsimplification.i5
Straightforward intersection: L3238Her@L3238&L3238Mar@L3238 -> L3238/L3238
Straightforward intersection: B279@B279&B279Hol@B279 -> B279/B279
Straightforward intersection: A329@A329_E&A329Rea@A329_E -> A329_E/A329_E
[yakra@noreaster /home/terescoj/travelmapping/HighwayData]$ egrep -n 'L3238|B279|A329' datacheckfps.csv
4030:deuby.b279;B279;;;LABEL_SELFREF;
4031:deuby.b279hol;B279;;;LABEL_SELFREF;
4109:deuhe.l3238her;L3238;;;LABEL_SELFREF;
4110:deuhe.l3238mar;L3238;;;LABEL_SELFREF;
4425:eng.a0329;A329_E;;;LABEL_SELFREF;
4426:eng.a0329;A329_W;;;LABEL_SELFREF;
4427:eng.a0329rea;A329_E;;;LABEL_SELFREF;
4428:eng.a0329rea;A329_W;;;LABEL_SELFREF;
Should arguably have the abbrev included...


INTERSTATE_NO_HYPHEN (9)
for rg in AK AL AR AS AZ CA CO CT DC DE FL GA GU HI IA ID IL IN KS KY LA MA MD ME MI MN MO MP MS MT NC ND NE NH NJ NM NV NY OH OK OR PA PR RI SC SD TN TX UT VA VI VT WA WI WV WY; do egrep '^I[0-9]|ToI[0-9]' $rg/*/*.wpt; done

UNDERSCORE_NUMBER (40)
numeral immediately follows underscore
for rg in `ls | grep -v _`; do grep -sv '^+' $rg/*/*.wpt | cut -f1 -d' ' | grep ':.*_[0-9]'; done

CLOSED_OPEN_COLOC (25)
Not always a true error.
Check with the other datachecks while WPTs are read, and thus while colocation lists are being populated. Iterate thru earlier points in list, breaking when current point is reached.

$rg/Can, Can/$rg, Mex/$rg, $rg/Mex border labels in USA (10)
for rg in AK AL AR AS AZ CA CO CT DC DE FL GA GU HI IA ID IL IN KS KY LA MA MD ME MI MN MO MP MS MT NC ND NE NH NJ NM NV NY OH OK OR PA PR RI SC SD TN TX UT VA VI VT WA WI WV WY; do egrep -i "^$rg/Can|^Can/$rg|^Mex/$rg|^$rg/Mex" $rg/*/*.wpt; done

banner_before_number (41)
for rg in `ls | grep -v _`; do for banner in Bus Trk His Alt Byp BS Fut AltTrk Sce BusTrk AltBus AltByp Wye; do grep -sv "^+" $rg/*/*.wpt | cut -f1 -d' ' | egrep -i ":.*$banner[A-Z]{,2}-?[0-9]+"; done | sort; done
http://forum.travelmapping.net/index.php?topic=2601.msg15607#msg15607

Old routes with false abbrevs that should be after the underscore (44)
Be sure to eliminate all potential banners.
Work on excluding FPs, e.g. Old3RRd Old9FootRd Old41Hwy Old4Hwy_N OldNo9Hwy
Tie into LACKS_GENERIC routines?
for rg in `ls | grep -v _`; do grep -sv '^+' $rg/*/*.wpt | cut -f1 -d' ' | egrep -v 'Alt$|Alt_[NEWS]$|Bus$|Bus_[NEWS]$|Byp$|Byp_[NEWS]$|Spr$|Spr_[NEWS]$' | egrep -i ':.*Old.*[0-9]+[A-Z]{3}'; done

NAMED_SLASH (141)
https://forum.travelmapping.net/index.php?topic=3245.msg21076#msg21076

CR-42, FI-42, ME-42, US-42, etc. (203)
for rg in `ls | grep -v _`; do egrep -s '^[A-Z]{2}-' $rg/*/*.wpt; done

SLASH_LONG_PREFIX {stash} (245)
for rg in `ls | grep -v _`; do grep -sv '^+' $rg/*/*.wpt | cut -f1 -d' ' | egrep -v ':\*?[A-Za-z]+/[A-Za-z]+$' | egrep ':.*/[A-Za-z]{2,}[0-9]'; done

slashed designation, unless: (number, optionally following single letter) (338)
for rg in `ls | grep -v _`; do grep -sv '^+' $rg/*/*.wpt | cut -f1 -d' ' | egrep -iv ':.*/[A-Z]?[0-9]|:\*?[A-Z]+/[A-Z]+$' | egrep ':.*+/'; done
overlap with graph-based unexpected_slash?

Take a closer look at the differences between these two

UNIQUE_SLASH_SUFFIX (445)
http://travelmapping.net/devel/manual/wayptlabels.php#avoidsuffix
http://travelmapping.net/devel/manual/wayptlabels.php#identicalmultiplex
https://travelmapping.net/devel/manual/wayptlabels.php#plexnosuffix

PARENS_LONG_PREFIX (611)
for rg in `ls | grep -v _`; do grep -sv '^+' $rg/*/*.wpt | cut -f1 -d' ' | grep -v '([A-Z]\?[0-9]' | grep '(..[0-9]'; done

only_1_intersecting
Points with a '/' in label colocated with only 1 (or 0?) other
Lots of FP potential. Borders! Don't flag when...
• points are in different regions?
• point is at beginning or end of route?

extraneous_suffix
either parenthetical or underscored, when only one jct with route
this could be a little tricky

I-designation + unexpected letter, excluding common banners
egrep -sv '^\+|^\*?I\-[0-9]+B[LS]|^\*?I\-[0-9]+Fut|^\*?I\-[0-9]+Spr|^\*?I\-[0-9]+Trk|^\*?I\-35[EW]|^\*?I-69[CEW]' $rg/*/*.wpt | cut -f1 -d' ' | egrep 'I\-[0-9]+[A-Za-z]'; done
A couple "Con" examples, incl 2 in NJ

KS US50 I-35(222) vice I-35/435
further examples from this thread, or the grep thread

route names in CSVs that break label naming rules
E.G. too many words
E.G. mexsf.csv: AutUrbSurCue

[yakra]

mysql> select distinct banner from routes left join systems on systems.systemName = routes.systemName where countryCode = 'USA';
+--------+
| banner |
+--------+
|        |
| Bus    |
| Trk    |
| His    |
| Alt    |
| Con    |
| Lp     |
| Byp    |
| Spr    |
| BL     |
| BS     |
| Fut    |
| AltTrk |
| Sce    |
| BusTrk |
| AltBus |
| AltByp |
| Wye    |
+--------+
18 rows in set (0.07 sec)

[yakra]

TX/usaib/tx.i040bsgle.wpt:OldUS66
TX/usatxf2/tx.fm0524.wpt:OldFM425
TX/usatxf2/tx.fm0546.wpt:OldFM546_W
TX/usatxf3/tx.fm1021.wpt:OldFM2366
TX/usatxf/tx.fm0078.wpt:OldTXLp539_E
TX/usatxf/tx.fm0078.wpt:OldTXLp539_W
TX/usatxf/tx.fm0105.wpt:OldHwy90
TX/usatxf/tx.fm0413.wpt:OldTX6_N
TX/usatxf/tx.fm0413.wpt:OldTX6_S
TX/usatxf/tx.fm0418.wpt:OldFM418
TX/usatxf/tx.fm0418.wpt:OldHwy418
TX/usatxf/tx.fm0482.wpt:OldUS81
TX/usatxl/tx.lp0121.wpt:OldFM594
TX/usatxl/tx.lp0223.wpt:OldHwy20_W
TX/usatxl/tx.lp0223.wpt:OldHwy20_E
TX/usatxr/tx.rm0473.wpt:OldNo9Hwy
TX/usatxr/tx.rm1431.wpt:OldRM1431_W
TX/usatxr/tx.rm1431.wpt:OldRM1431_E
TX/usatxs/tx.sp0186.wpt:OldHwy20
TX/usatxs/tx.sp0419.wpt:OldFM524_S
TX/usatxs/tx.sp0419.wpt:OldFM524_N
TX/usatx/tx.tx025.wpt:OldUS82
TX/usatx/tx.tx034.wpt:OldTX34
TX/usatx/tx.tx035.wpt:OldTXLp419
TX/usatx/tx.tx036.wpt:OldHwy36
TX/usatx/tx.tx066.wpt:OldFM3211
TX/usatx/tx.tx137.wpt:OldTX137
TX/usatx/tx.tx151.wpt:OldHwy90
TX/usatx/tx.tx154.wpt:OldTX154
TX/usatx/tx.tx163.wpt:OldTX163
TX/usatx/tx.tx290.wpt:OldUS290
TX/usatx/tx.tx359.wpt:OldTX359
TX/usausb/tx.us059busspl.wpt:OldHwy59
TX/usaush/tx.us066hisgle.wpt:OldUS66
TX/usaus/tx.us067.wpt:OldSpr310
TX/usaus/tx.us079.wpt:OldFM1460
TX/usaus/tx.us090.wpt:OldUS90
TX/usaus/tx.us175.wpt:OldUS175_S
TX/usaus/tx.us175.wpt:OldUS175_N
TX/usaus/tx.us181.wpt:OldUS181_B
TX/usaus/tx.us181.wpt:OldUS181_C
TX/usaib/tx.i035blalv.wpt:*OldUS67Bus
TX/usai/tx.i030.wpt:*OldI-30_W
TX/usai/tx.i030.wpt:*OldI-30_E
TX/usatxf2/tx.fm0546.wpt:*OldFM546_E
TX/usatxf5/tx.fm2087.wpt:*OldTXLp281
TX/usatxf/tx.fm0121.wpt:*OldFM121_A
TX/usatxf/tx.fm0121.wpt:*OldFM121_B
TX/usatxf/tx.fm0121.wpt:*OldFM121_C
TX/usatxf/tx.fm0121.wpt:*OldFM121_D
TX/usatxf/tx.fm0481.wpt:*OldFM481_W
TX/usatxf/tx.fm0481.wpt:*OldFM481_E
TX/usatxl/tx.lp0281.wpt:*OldTXLp281
TX/usatxs/tx.sp0334.wpt:*OldUS183
TX/usatx/tx.tx005.wpt:*OldTX5
TX/usatx/tx.tx043.wpt:*OldFM1794
TX/usatx/tx.tx114.wpt:*OldUS82
TX/usatx/tx.tx121.wpt:*OldTX5
TX/usatx/tx.tx188.wpt:*OldTX188_W
TX/usatx/tx.tx188.wpt:*OldTX188_E
TX/usatx/tx.tx195.wpt:*OldTX195
TX/usausb/tx.us067busalv.wpt:*OldUS67Bus_S
TX/usausb/tx.us067busalv.wpt:*OldUS67Bus_N
TX/usaus/tx.us082.wpt:*OldUS82
TX/usaus/tx.us082.wpt:*OldUS82_W
TX/usaus/tx.us082.wpt:*OldUS82_E
TX/usaus/tx.us181.wpt:*OldUS181_A
TX/usaus/tx.us181.wpt:*OldTX359
TX/usaus/tx.us277.wpt:*OldUS277_S
TX/usaus/tx.us277.wpt:*OldUS277_N
TX/usaus/tx.us377.wpt:*OldI-30_W
TX/usaus/tx.us377.wpt:*OldI-30_E

[yakra]

EXTRANEOUS_EXIT (853)
Find unique parenthetical suffixes combined with unique prefixes. There will be false positives.
count=0
for rg in $(ls | grep -v _); do
  for dummy in $(ls $rg/*/*.wpt | tr ' ' '%'); do
    file=$(echo $dummy | tr '%' ' ')
    Ls=$(grep -v '^+' "$file" | cut -f1 -d' ' | grep '(.*)' | cut -f1 -d'(' | sed 's~^\*~~' | sort | uniq)
    Rs=$(grep -v '^+' "$file" | cut -f1 -d' ' | grep '(.*)' | cut -f2 -d'(' | sed 's~)$~~'  | sort | uniq)
    for l in $Ls; do
      if [ $(grep -m 2 "^\*\?$l(" "$file" | wc -l) = '1' ]; then
        for r in $Rs; do
          if [ $(cut -f1 -d' ' "$file" | grep -m 2 "($r)" | wc -l) = '1' ]; then
            result=$(grep -H "^$l($r)" "$file")
            if [ $(echo $result | wc -w) -gt '0' ]; then
              count=$(echo $count+1 | bc)
              echo -en "$count\t"
              echo $result
            fi
          fi
        done
      fi
    done
  done
done | tee extraneous_suffix.log

Send Firefox to all routes in each region, centered at the waypoint in question
for rg in $(cat ~/tmregions); do echo -n $rg; cut -f2 extraneous_suffix.log | grep "^$rg/" | sed "s~^$rg/[a-z0-9]\+/\(.*\)\.wpt:.*\(lat=[-.0-9]*&lon=[-.0-9]*\)~https://travelmapping.net/hb/showroute.php?r=\1\&\2\&zoom=14~" | xargs firefox; read go; done

Exclude some false positives, but produce some false negatives too (661)
count=0
for rg in $(ls | grep -v _); do
   for dummy in $(ls $rg/*/*.wpt | tr ' ' '%'); do
     file=$(echo $dummy | tr '%' ' ')
     Ls=$(grep -v '^+' "$file" | cut -f1 -d' ' | grep '(.*)' | cut -f1 -d'(' | sed 's~^\*~~' | sort | uniq)
     Rs=$(grep -v '^+' "$file" | cut -f1 -d' ' | grep '(.*)' | cut -f2 -d'(' | sed 's~)$~~'  | sort | uniq)
     for l in $Ls; do
       if [ $(grep -c -m 2 "^\*\?$l[( _]" "$file") = '1' ]; then
         for r in $Rs; do
           if [ $(cut -f1 -d' ' "$file" | grep -c -m 2 "($r)") = '1' ]; then
             result=$(grep -H -C 1 "^$l($r)" "$file")
             if [ $(echo "$result" | sed "s~^$file.~~" | cut -f1 -d' ' | grep -c '(') = 1 ]; then
               count=$(echo $count+1 | bc)
               echo -en "$count\t"
               echo "$result" | grep "^$file:"
             fi
           fi
         done
       fi
     done
   done
 done | tee EXTRANEOUS_EXIT.log

Don't require unique parenthetical part; avoid EE(RR) results & some false negatives (560)
count=0
for rg in $(ls | grep -v _); do
  for dummy in $(ls $rg/*/*.wpt | tr ' ' '%'); do
    file=$(echo $dummy | tr '%' ' ')
    for l in $(egrep -v '^\+|^\*?[0-9]' "$file" | cut -f1 -d' ' | grep '(.*)' | cut -f1 -d'(' | sed 's~^\*~~' | sort | uniq); do
      if [ $(grep -c -m 2 "^\*\?$l[( _]" "$file") = '1' ]; then
        result=$(grep -H -C 1 "^\*\?$l[( _]" "$file")
        if [ $(echo "$result" | sed "s~^$file.~~" | cut -f1 -d' ' | grep -c '(') = 1 ]; then
          count=$(echo $count+1 | bc)
          echo -en "$count\t"
          echo "$result" | grep "^$file:"
        fi
      fi
    done
  done
done | tee EXTRANEOUS_EXIT.log

[yakra]

UNIQUE_SLASH_SUFFIX (445)
http://travelmapping.net/devel/manual/wayptlabels.php#avoidsuffix
http://travelmapping.net/devel/manual/wayptlabels.php#identicalmultiplex
https://travelmapping.net/devel/manual/wayptlabels.php#plexnosuffix
count=0
for rg in $(ls | grep -v _); do
  for dummy in $(ls $rg/*/*.wpt | tr ' ' '%'); do
    file=$(echo $dummy | tr '%' ' ')
    Ls=$(grep -v '^+' "$file" | cut -f1 -d' ' | grep '/.*_' | cut -f1 -d_ | sed 's~^\*~~' | sort | uniq)
    for l in $Ls; do
      if [ $(grep -m 2 "^\*\?$l" "$file" | wc -l) = '1' ]; then
        count=$(echo $count+1 | bc)
        echo -en "$count\t"
        grep -H "^\*\?$l" "$file"
      fi
    done
  done
done | tee UNIQUE_SLASH_SUFFIX.log

Send Firefox to all routes in each region, centered at the waypoint in question
for rg in $(cat ~/tmregions); do echo -n $rg; cut -f2 UNIQUE_SLASH_SUFFIX.log | grep "^$rg/" | sed "s~^$rg/[a-z0-9]\+/\(.*\)\.wpt:.*\(lat=[-.0-9]*&lon=[-.0-9]*\)~https://travelmapping.net/hb/showroute.php?r=\1\&\2\&zoom=14~" | xargs firefox; read go; done

[yakra]

Operation Spur

C++: Compile root pairs

Code: [Select]

std::unordered_map<std::string, std::list<Route*>> RteBan;

for (auto h : highway_systems)
  for (auto &r : h->route_list)
    RteBan[r.region->code+r.route+r.banner].push_back(&r);

for (auto &RteList : RteBan)
  if (RteList.second.size() > 1)
    for (Route* r : RteList.second)
      for (Waypoint *w : r->point_list)
if (w->colocated)
  for (Waypoint *p : *w->colocated)
    if (w < p && w->route != p->route && \
w->route->region->code + w->route->route + w->route->banner == \
p->route->region->code + p->route->route + p->route->banner)
      std::cout << "RESULT:\t" << w->route->root << '\t' << p->route->root << std::endl;
return 0;
label search
   std::unordered_map<std::string, std::list<Route*>> RteBan;

   for (auto h : highway_systems)
     for (auto &r : h->route_list)
       RteBan[r.region->code+r.route+r.banner].push_back(&r);

   for (auto &RteList : RteBan)
     if (RteList.second.size() > 1)
       for (Route* r : RteList.second)
         for (Waypoint *w : r->point_list)
      if (w->colocated)
      { std::string wrb = w->route->route + w->route->banner;
        for (Waypoint *p : *w->colocated)
        { std::string prb = p->route->route + p->route->banner;
          if (w != p && w->route != p->route && w->route->region->code + wrb == p->route->region->code + prb)
          {   if (w->route->abbrev.size() && w->label.substr(0,prb.size()) != prb)
           std::cout << "RESULT:" << w->route->region->code << '/' << w->route->system->systemname << '/' << w->route->root << ".wpt\t" << w->label << std::endl;
          }
        }
      }
   return 0;

Sh: Unique root list
t=`echo -en '\t'`
results=`grep '^RESULT' Spr/logs/siteupdate.log | sed "s~^RESULT:$t~~" | tr '\t' '\n' | sort | uniq`
echo $results | wc -w
1940

Sh: Look up in system CSVs
cd ~/TravelMapping/HighwayData/
systems=`tail -n +2 systems.csv | grep -v '^#' | cut -f1 -d';' | sed 's~.*~&.csv~'`
cd hwy_data/_systems
for r in $results; do grep ";$r;" $systems; done | wc -l
1940

cd ~/TravelMapping/HighwayData/hwy_data/_systems
n_abbrev=`for r in $results; do egrep ".*;.*;.*;.*;;.*;$r;.*" $systems | cut -f7 -d';'; done`
710
y_abbrev=`for r in $results; do egrep ".*;.*;.*;.*;.+;.*;$r;.*" $systems | cut -f7 -d';'; done`
1230
bad=`for r in $results; do egrep ".*;.*;.*;.*;.+;.*;$r;.*" $systems; done | grep '(spur)\|(loop)\|(branch)' | cut -f7 -d';'`
204
good=`for r in $results; do egrep ".*;.*;.*;.*;.+;.*;$r;.*" $systems; done | grep -v '(spur)\|(loop)\|(branch)' | cut -f7 -d';'`
1026

Sh: How many regions?
echo $good | tr ' ' '\n' | cut -f1 -d. | uniq | wc -l
112 (9.2 rtes/rg)
echo $bad | tr ' ' '\n' | cut -f1 -d. | uniq | wc -l
41 (5 rtes/rg)

for rg in `echo $good | tr ' ' '\n' | cut -f1 -d. | uniq`; do echo -n "$rg "; echo $good |tr ' ' '\n' | grep "^$rg" | sed 's~.*~https://travelmapping.net/hb/showroute.php?r=&~' | xargs firefox; read go; done; echo

To be continued...

[yakra]

Toward reducing the number of LABEL_SELFREF false positives


LABEL_SELFREF 1 (match after /)
az.az095;I-10BL/95;;;LABEL_SELFREF1;
de.de202;I-95/202;;;LABEL_SELFREF1;
deuhe.l3308;St2305/3308;;;LABEL_SELFREF1;
fin.e8;Vt4/8;;;LABEL_SELFREF1;
tx.tx070;US62/70;;;LABEL_SELFREF1;
FP because: Digits after slash are a different designation type. Route included in the HB.

al.al106;CR1/106;;;LABEL_SELFREF1;
fl.fl276;CR167/276;;;LABEL_SELFREF1;
...and West Virginia LOL
FP because: Digits after slash are a different designation type: not AL106 but CR106 which is not and will not be in HB. Must be manually marked FP. Consider this the canonical FP example. :)

al.us072altdec;US43/72;;;LABEL_SELFREF1;
eng.a1077m;A18/A1077;;;LABEL_SELFREF1;
nj.us009trkjer;US1/9;;;LABEL_SELFREF1;
etc. ...
FP because: Testing a bannered route. Require banner match at unbannered parent route.


LABEL_SELFREF 2 (match between / and _)
{ Folded into LABEL_SELFREF 1 }
ga.ga023;US1/23_N;;;LABEL_SELFREF2;
ga.ga023;US1/23_S;;;LABEL_SELFREF2;
FP because: Digits after slash are a different designation type: US23 not GA23.

sd.us385trkhil;US16/385_N;;;LABEL_SELFREF2;
sd.us385trkhil;US16/385_S;;;LABEL_SELFREF2;
va.us019trkbri;US11E/19_S;;;LABEL_SELFREF2;
FP because: Testing a bannered route. Require banner match at unbannered parent route.


LABEL_SELFREF 3 (match entire string)
ita.a001sas;A1;;;LABEL_SELFREF3;
ita.a013pad;A13;;;LABEL_SELFREF3;
Flagged because: Graph connection not present.
The rest all look like genuine errors.
Don't separate out slash matches via a switch -- even with banner_after_slash, coloc_same_designation is still necessary.


LABEL_SELFREF 3 (match before /)
ga.ga011;GA11/135Byp_S;;;LABEL_SELFREF3;
ga.us001;US1/23Bus_N;;;LABEL_SELFREF3;
etc. ...
FP because: Banner of intersecting route is included after slash in label.

nj.us001;US1/9Trk;;;LABEL_SELFREF1;
FP because: No graph connection with bannered route. Must be manually marked FP. Consider this the canonical "/Ban" FP?

tn.us079;US79/70AltByp;;;LABEL_SELFREF3;
Watch out for what happens with this one. Intersects US79BypHum & US70AltBypHum.
Wait, this makes sense as (US79/70Alt)(Byp). It's FP, and not flagged. Good to go.

indwb.nh027;NH27/517;;;LABEL_SELFREF3;
indwb.nh114;NH114/WB14_E;;;LABEL_SELFREF3;
indwb.nh114;NH114/WB14_W;;;LABEL_SELFREF3;
indwb.nh312;NH312/WB3;;;LABEL_SELFREF3;
jpn.n011;N11/N193;;;LABEL_SELFREF3;
True positives.


LABEL_SELFREF 4 (match before _)
srb.e75bel;E75_N;;;LABEL_SELFREF4;
mexgro.mex200vie;MEX200_Pan;;;LABEL_SELFREF4;
jpn.n193kag;N193_S;;;LABEL_SELFREF4;
Flagged because: Graph connection not present.
The rest all look like genuine errors.
Nothing to do here.


Inspect stuff in the HB

lines=$(diff $v1/logs/datacheck.log $v2/logs/datacheck.log | grep '^>' | cut -f2 -d' ' | tail -n +2) #new errors
lines=$(diff $v1/logs/unmatchedfps.log $v2/logs/unmatchedfps.log | grep '^>' | cut -f2 -d' ' | tail -n +2)  #removed FPs
lines=$(grep -vF -f <(grep LABEL_SELFREF _obj/logs/unmatchedfps.log) ~/tm/HighwayData/datacheckfps.csv | grep LABEL_SELFREF) #remaining FPs

for line in $lines; do
  r=`echo $line | cut -f1 -d';'`
  l=`echo $line | cut -f2 -d';'`
  grep "^$l " ~/tm/HighwayData/hwy_data/*/*/$r.wpt \
  | sed -r "s~.+(lat=.+)~https://travelmapping.net/hb/showroute.php?r=$r\&\1\&zoom=17~"
done | xargs firefox

An unrelated exercise for the reader ;)
grep -hF -f <(tail -n +2 $(ls | grep -v _) | egrep '.*;.*;.*;[^\-].*;;.*;.*;.*' | sed -r 's~(.*;.*;.*;[^\-].*;);.*;.*;.*~\1~') $(ls | grep -v _) | sed -r "s~(.*;.*;.*;.*;)(.*)(;.*;.*;.*)~\1$e[1;31m\2$e[0m\3~" | less -r

strd.o
Postslash exception for "Old"?
• *OldUS93/95
• OldSD20/63_N
• OldSD20/63_S
• *OldWI35/64

[yakra]

speed micro-comparison: what if breaks were returns? Faster / margin of error
case '_': break if FP; mark it once. This will facilitate further experimentation. Speed? Marginally faster.
banner_after_slash FIXME: call strdstr with slash, not slash+1 <--It'll be grand. There's needs to be something (e.g. numerals) between slash & banner anyway.
banner_after_slash speed: FRA dept banners?
banner_after_slash: whitespace @ right parens
Meta-exceptions: put some cases back in

Code: [Select]

switch(*c)
{   case 0: if (route->abbrev.empty() || !coloc_same_designation(rte))
  Datacheck::add(route, label, "", "", "LABEL_SELFREF", "");
break;
    case '_': { if (*++c == 'U')
{ while (isdigit(*++c));
if (!*c) break;
}
Waypoint* w = coloc_same_designation(rte);
if (!w || (route->banner.empty() || route->banner[0] == '-') && w->route->abbrev.size())
Datacheck::add(route, label, "", "", "LABEL_SELFREF", "");
      } break;
_ua: if (!w || w->route->abbrev.size())

[yakra]

e1c106c4cb26c6579df2e2b607fd09382454112d
categorize everything again; get wicked specific

Code: [Select]

#define XINFO(w)\
     std::string(route->banner.empty() ? ":" : (route->banner[0]=='-' ? "-" : "B")) + \
(route->abbrev.empty() ? ":" : "A") + \
(w ? std::string(w->route->banner.empty() ? ":" : (route->banner[0]=='-' ? "-" : "B")) + \
(w->route->abbrev.empty() ? ":" : "A") : "!")
Qty  Subtype                | TODO description/comments
6    LABEL_SELFREF/1::::    | TOSS same number different prefix
1    LABEL_SELFREF/1:::A    | TOSS same number different prefix
7    LABEL_SELFREF/1::!     | KEEP 6 FP CR; 1 true error
1    LABEL_SELFREF/1:A!     | KEEP FP CR
1    LABEL_SELFREF/D-A-:    | TOSS "bannered routes and the suffixed equivalents" vs #parentonly
1    LABEL_SELFREF/D::::    | TOSS self-intersection
1    LABEL_SELFREF/D:::A    | TOSS https://travelmapping.net/devel/manual/wayptlabels.php#slash
3    LABEL_SELFREF/D::!     | KEEP true errors
8    LABEL_SELFREF/D:A::    | TOSS "bannered routes and the suffixed equivalents" (6 #parentonly, 2 #parentplussuffixes)
2    LABEL_SELFREF00-:-:    | TOSS same designation different system, comparable to NH16 in ME intersecting ME16
41   LABEL_SELFREF00-:!     | KEEP true errors (new FRA)
25   LABEL_SELFREF00-A-:    | TOSS child intersecting parent
2    LABEL_SELFREF00-A!     | KEEP true errors (new FRA)
10   LABEL_SELFREF00::::    | KEEP self-intersection (>1 intersection = should have underscore or slash)
1    LABEL_SELFREF00:::A    | KEEP #abbrev
18   LABEL_SELFREF00::!     | KEEP 17 true errors, 1 FP (RUS)
718  LABEL_SELFREF00:A::    | TOSS child intersecting parent
74   LABEL_SELFREF00:A:A    | KEEP #abbrev
3    LABEL_SELFREF00:A!     | KEEP 1 true, 1 FP, 1 could be fixed by syncing coords
1    LABEL_SELFREF00BA!     | KEEP true error (MS315SceSar @ MS315Sce)
1    LABEL_SELFREF00BAB:    | TOSS child intersecting parent (eng.a0627msla)
2    LABEL_SELFREF00BABA    | KEEP #abbrev
22   LABEL_SELFREF__-:-:    | TOSS self-intersection
21   LABEL_SELFREF__-:!     | KEEP true errors (new FRA)
2    LABEL_SELFREF__-A-:    | TOSS child intersecting parent
1    LABEL_SELFREF__-A!     | KEEP true error (new FRA)
89   LABEL_SELFREF__::::    | TOSS self-intersection
7    LABEL_SELFREF__:::A    | KEEP #abbrev
31   LABEL_SELFREF__::!     | KEEP 9 "_U[0-9]+[NEWS]", 4 local rds, 18 true errors (2 AL, 4 AUS, 7 IND, 2 ISR, 2 MS, 1 RUS)
294  LABEL_SELFREF__:A::    | TOSS child intersecting parent
34   LABEL_SELFREF__:A:A    | KEEP #abbrev 11 piecemeal (2 DEU, 6 MEX, 3 TUR), 2 self-branch (NOR), 21 branches (6 IRL, 13 MEX, 2 OR)
2    LABEL_SELFREF__:A!     | KEEP 1 FP, 1 could be fixed by syncing coords
29   LABEL_SELFREF__BABA    | KEEP #abbrev 28 US66His cases + PA420AltTrkRid @ PA420AltTrk_Pro
1    LABEL_SELFREF//-:::    | TOSS same designation different system, comparable to DE202 vs US202
16   LABEL_SELFREF//-:!     | KEEP true errors (new FRA)
2    LABEL_SELFREF//::::    | TOSS self-intersection
1    LABEL_SELFREF//:::A    | TOSS https://travelmapping.net/devel/manual/wayptlabels.php#slash
10   LABEL_SELFREF//::!     | KEEP 1 BAS, 4 local rds, 4 true errors (1 missing banner)
48   LABEL_SELFREF//::!S    | TOSS banner_after_slash
10   LABEL_SELFREF//:A::    | TOSS "bannered routes and the suffixed equivalents" (9 #parentonly, 1 that connects 2x, but only @1 end)
1    LABEL_SELFREF//:A:A    | TOSS "bannered routes and the suffixed equivalents" vs #parentplussuffixes
2    LABEL_SELFREF++:A:A:A! | TOSS self-intersection
3    LABEL_SELFREF++:A!:A!  | KEEP true errors
1    LABEL_SELFREF++:A!:ABA | TOSS coloc_banner_matches_abbrev()
2    LABEL_SELFREF++BA!BA!  | KEEP true errors
1556 Total

Is every banger a keeper? No, just the obvious ones -- banner_after_slash is an exclusion. In ++ cases, ! represents 2 different variables; FP possible with just 1.
Is every keeper a banger? Yes, so far, those explicitly listed at least. No.
Double-check tossers.
Review self-intersections. 00 not tossers, right? Any others?
Should self-intersection detection be combined into coloc_same_designation? Nah. Any occurrences other than those explicitly listed above? Just one.
coloc_same_designation should always return a vanilla route if there is one: '@' sorts before letters, and no routes have abbrevs starting with numerals. If any exist in the future, this should be rare enough to not bother with.

Potential underscore_abbrev(const char*) candidates
dnk.mr;MR_Gly;;;LABEL_SELFREF__:::A;
or.or223;OR223_Dal;;;LABEL_SELFREF__:::A;
or.or223dal;OR223_Ric;;;LABEL_SELFREF__:A:A;
or.or223ric;OR223_Dal;;;LABEL_SELFREF__:A:A;
pa.pa420alttrkrid;PA420AltTrk_Pro;;;LABEL_SELFREF__BABA;

[yakra]

.:.:   (is(00) && returned_route == route) ? KEEP : TOSS
.:.A   KEEP except slashes
.A.:   TOSS
.A.A   KEEP except slashes


1556 total
- 3 "++"-flavored = 1553
- 63 "//"-flavored = 1490
- 18 "/1" & "/D" flavored = 1472
- 407 "__"-flavored = 1065
- 746 "00"-flavored = 319


_sa0 -> _uu:
+ 70 errors = 7 :::A + 34 :A:A + 29 BABA

_uu -> _00:
+ 87 errors = 10 :::: + 1 :::A + 74 :A:A + 2 BABA

[yakra]

Q: Why the extra {}s in the switch block again?
A: Otherwise, cannot jump from switch statement to later case labels bypassing variable initialization

Types of new helper functions:
   Route* coloc_same_number(const char*);
   Route* coloc_same_designation(const std::string&);
   Route* self_intersection();
   bool banner_after_slash(const char*);
   Route* coloc_banner_matches_abbrev();

To modify datacheckfps.csv

Code: [Select]

./s1.sh -v -e -y
echo "$(sed $(grep LABEL_SELFREF logs/nearmatchfps.log | tr '\n' '~' | sed -e 's/FP_ENTRY: /\n-e s~/g' -e 's~CHANGETO: ~~g') ~/tm/HighwayData/datacheckfps.csv)" > ~/ytm/HighwayData/datacheckfps.csv
./s1.sh -v -e -y
echo "$(grep -vF -f <(grep LABEL_SELFREF logs/unmatchedfps.log) ~/ytm/HighwayData/datacheckfps.csv)" > ~/ytm/HighwayData/datacheckfps.csv

In case I ever wanna refer to this again

Code: [Select]

#define XINFO(r)     std::string(route->banner.empty() ? ":" : (route->banner[0]=='-' ? "-" : "B")) + \
(route->abbrev.empty() ? ":" : "A") + \
(r ? std::string(r->banner.empty() ? ":" : (route->banner[0]=='-' ? "-" : "B")) + \
(r->abbrev.empty() ? ":" : "A") : "!")

[yakra]

Code: [Select]

15: splitregion: unknown error for michih in line ITA SS685 SS3 SP64
16: splitregion: unknown error for michih in line ITA SS685 SP129 SS4
17: splitregion: unknown error for michih in line ITA SS685 SP129 SS4
18: splitregion: DISCONNECTED_ROUTE in italig.e74 itapie.e74 for michih in line ITA E74 A21 FRA/ITA
Unknown? I want to know!

Edit: ITA route goes east-to west.
Split-region ConRoute goes west-to-east.
All the more reason to not try to automatically generate new .list lines (the way it's been done till now at least) and let hidden systems work their magic instead.