Author Topic: yakra's collaborator thread  (Read 12363 times)

0 Members and 1 Guest are viewing this topic.

Offline yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2938
  • Last Login:Yesterday at 09:04:27 pm
Re: yakra's collaborator thread
« Reply #15 on: September 22, 2019, 11:30:36 pm »
grep-o-roonie
inna USA/CAN stylee

1. Potential malformed city suffixes, including directional
grep '^[A-Z][A-Z][0-9]\{1,3\}[A-Z][a-z][a-z]_[NEWS]' */*/*.wpt | grep -v '[A-Z][A-Z][0-9]\{1,3\}Alt_[NEWS]\|[A-Z][A-Z][0-9]\{1,3\}Bus_[NEWS]\|[A-Z][A-Z][0-9]\{1,3\}Byp_[NEWS]\|[A-Z][A-Z][0-9]\{1,3\}Trk_[NEWS]\|[A-Z][A-Z][0-9]\{1,3\}His_[NEWS]\|[A-Z][A-Z][0-9]\{1,3\}Spr_[NEWS]\|[A-Z][A-Z][0-9]\{1,3\}Con_[NEWS]'

2. Potential malformed city suffixes, more complete list
grep '^[A-Z][A-Z][0-9]\{1,3\}[A-Z][a-z][a-z]' */*/*.wpt | grep -v '[A-Z][A-Z][0-9]\{1,3\}Alt\|[A-Z][A-Z][0-9]\{1,3\}Bus\|[A-Z][A-Z][0-9]\{1,3\}Byp\|[A-Z][A-Z][0-9]\{1,3\}Trk\|[A-Z][A-Z][0-9]\{1,3\}His\|[A-Z][A-Z][0-9]\{1,3\}Spr\|[A-Z][A-Z][0-9]\{1,3\}Con'

3. "Good Alabama"
grep _ AL/*/*.wpt | grep -v '_[NEWS] \|_[A-Za-z]\{4\}'
A few false positives.

4. "Bad Maine"
grep 'US[0-9]\{1,3\}Alt.\+ ' ME/*/*.wpt | grep -v 'US[0-9]\{1,3\}Alt_[NEWS] '

5. 4-character LONG_UNDERSCORE datacheck
grep -i '_[A-Z]\{3\}[abcdfghijklmopqrtuvxyz] ' */*/*.wpt
Ubuntu:grep '_...[A-DF-MO-RTUVXYZa-z] ' */*/*.wpt
Error results:for rg in `ls`; do grep -s '_...[A-DF-MO-RTUVXYZa-z] ' $rg/*/*.wpt; done
Error counts:for rg in `ls`; do echo -en "$rg\t"; grep -s '_...[A-DF-MO-RTUVXYZa-z] ' $rg/*/*.wpt | wc -l; done
Good results:for rg in `ls`; do grep -s '_...[NEWS] ' $rg/*/*.wpt; done
Good counts:for rg in `ls`; do echo -en "$rg\t"; grep -s '_...[NEWS] ' $rg/*/*.wpt | wc -l; done

6. Old highway designation labels that may have a current visible name
grep '^Old..[0-9]' */*/*.wpt
This one will have a lot of false positives.
« Last Edit: September 26, 2019, 11:03:06 pm by yakra »

Offline yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2938
  • Last Login:Yesterday at 09:04:27 pm
Re: yakra's collaborator thread
« Reply #16 on: September 27, 2019, 03:21:53 am »
Operation Good Labels
MyRegions='CT MA ME NH RI KS NE OK NJ NY TX AB MB NB NS PE NL'
NewEngland='CT MA ME NH RI'
CentralStates='KS NE OK'
Canada='AB MB NB NS PE NL'
MiscRegions='NJ NY TX'


Operation 4-Char Suffix (with NEWS)
CT(14) MA(6) ME(33) NH(12) RI(12)
for rg in $NewEngland; do grep '_.... ' $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep '_....$'; done

Are all city suffixes appropriate?
Excluding 4-char cases with directional "subfix", as those are covered above
CT(16) MA(41) ME(181) NH(93) RI(7) (338)
KS(123) NE(103) OK(68) (294)
NJ(28) NY(81) TX(14) (123)
AB(28) MB(5) NB(18) NS(51) NL(6) (108)
for rg in $MyRegions; do grep '_... ' $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep -v ':+' | grep '_...$'; done

Operation 2-Char Suffix
Excludes 4-char cases with directional "subfix"; those are covered above
CT(1) KS(3) MA(1) ME(11) NB(2)
for rg in $MyRegions; do grep '_.. ' $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep -v ':+' | grep '_..$'; done

Operation Too Many Words
CT(5) MA(1) ME(4) NE(1) NH(5) NY(6) NB(16) NL(29)
for rg in $MyRegions; do grep -s '[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]*' $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep '[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]*'; done
Not sufficient to match all cases. Refine regex & redo.

FooBarBRd
CT(1) MA(2) ME(2) NH(12) (17)
KS(3) NE(7) OK(10) (20)
NJ(5) NY(83) TX(9) (97)
AB(3) NB(25) NS(6) PE(1) NL(3) (38)
for rg in $MyRegions; do grep '[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]\?[A-Z]\+[a-z]*' $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep -v '[NPS]P$' | grep '[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]\?[A-Z]\+[a-z]*'; done
for rg in $MyRegions; do grep -s '[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]\?[A-Z]\+[a-z]*' $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep -v '[NPS]P$' | grep '[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]\?[A-Z]\+[a-z]\+'; done

FooBBazRd
CT(1) MA(1) ME(7) NH(1) (10)
NJ(2) NY(8) TX(1) (11)
OK(1) (1)
NB(6) NL(4) (10)
for rg in $MyRegions; do grep -s '[A-Z]\+[a-z]\+[A-Z]\+[a-z]\?[A-Z]\+[a-z]\+[A-Z]\+[a-z]*' $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep -v ':Mc[A-Z][A-Z]' | grep '[A-Z]\+[a-z]\+[A-Z]\+[a-z]\?[A-Z]\+[a-z]\+[A-Z]\+[a-z]*'; done

FBarBazRd
CT(3) MA(1) ME(2) NH(3) RI(1) (10)
NY(28) TX(2) (30)
NB(16) NS(2) PE(1) NL(1) (20)
NE(1) (unprocessed WPT deleted) (1)
for rg in $MyRegions; do grep -s '[A-Z]\+[a-z]\?[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]*' $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep '[A-Z]\+[a-z]\?[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]*'; done

"All of the Above" mammoth regex
for rg in `ls`; do grep -s . $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep -v '[NPS]P$\|:[A-Z]*[a-z]*Mc[A-Z][A-Z][a-z]*$\|:Mc[A-Z][A-Z]\+[a-z]*[A-Z][a-z]*$\|:[A-Z]*Mc[A-Z]\+[a-z]\?$' | grep '[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]*\|[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]\?[A-Z]\+[a-z]\+\|[A-Z]\+[a-z]\+[A-Z]\+[a-z]\?[A-Z]\+[a-z]\+[A-Z]\+[a-z]*\|[A-Z]\+[a-z]\?[A-Z]\+[a-z]\+[A-Z]\+[a-z]\+[A-Z]\+[a-z]*'; done

Operation _End
MA(2) ME(3) NE(1) NH(2) OK(1) RI(2) AB(1)
for rg in $MyRegions; do grep _End $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep _End; done

Operation Predir
This got its own post

Operation Postdir
CT(6) ME(7) NH(5) (18)
KS(16) NE(1) (17)
NJ(3) NY(10) TX(6) (19)
MB(2) NB(9) NS(3) NL(5) (19)
for rg in $MyRegions; do grep '[NEWS]' $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep '[NEWS]$' | grep -v ':+\|_\|[0-9]\{1,3\}[NEWS]$\|:[A-Z][A-Z]/[A-Z][A-Z]$\|:CR[A-Z]$\|:Rd[A-Z]$\|:Ave[A-Z]$\|:I\-[0-9]\{1,3\}BS$'; done

Operation Long Words
ME(1) NL(1)
for rg in $MyRegions; do grep . $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep -v ':+' | grep ':.*[a-z]\{4\}'; done

Operation McPotrzebie
MA(1) ME(2)
AB(2)
NJ(6) NY(6) TX(4)
for rg in $MyRegions; do grep Mc[A-Z][a-z] $rg/*/*.wpt; done
for rg in $Canada; do grep Mac $rg/*/*.wpt; done


Tpke -> Tpk
CT(7) MA(1) NH(10) RI(1)
OK(2)
NJ(4) NY(19) TX(1)

for rg in $MyRegions; do grep -s Tpke $rg/*/*.wpt | sed -e 's/ /%/' -e 's/\(.\+\)%.\+/\1/' | grep 'Tpke'; done
« Last Edit: December 26, 2019, 01:32:51 pm by yakra »


Offline yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2938
  • Last Login:Yesterday at 09:04:27 pm
Re: yakra's collaborator thread
« Reply #18 on: November 20, 2019, 02:10:33 am »
Ideas to reject. Or not.

abbrev_as_suffix
These can show up in graph labels. 31 examples in:
egrep -i '([A-Z]{3})_\1' waypointsimplification.log
egrep -i '([A-Z]{3})_\1' tm-master-simple.tmg

I may have some data to fix; there's stuff in my regions
30 cases of 3+ intersection + one Exit/Intersection out of 430031 collapsed vertices. May well be fixable on the waypoint simplification side.

datacheck-style pseudocode
Code: [Select]
if (underscore != string::npos)
  for (Waypoint *p : *(w->colocated))
  { if (p == w) continue;
if ( p->route->abbrev.size()
     && w->label.substr(underscore+1, p->route->abbrev.size()) == p->route->abbrev
     && w->label.substr(0, underscore) == p->route->name_no_abbrev()
   ) { datacheckerrors->add(&r, w->label, "", "", "ABBREV_AS_SUFFIX", p->route->abbrev);
break;
     }
  }
/*TODO: else for (Waypoint *p : *(w->colocated))
     { if (p == w) continue;
if (regex("[0-9]+[A-Za-z]{3}") in label)
if ( p->route->route ends with same [0-9]+ \
and not p.route.banner.startswith(same [A-Za-z]{3}) \
and not p.route.abbrev.startswith(same [A-Za-z]{3})
   ) { datacheckerrors->add(&r, w->label, "", "", "FALSE_ABBREV", "Lorem Ipsum");
break;
     }
     }//*/
      }
Not really worth doing as a datacheck. A child route can intersect or overlap its parent midway, resulting in a city_suffix: see MA122@MA122A_Wor. These cases alone + desire to be consistent with otherwise-same-style labels + enough of a gray area in the manual = abbrev_as_suffix should be accepted as normal, albeit awkward for graph labels.
false_abbrev or unexpected_abbrev, OTOH... ;)



LABEL_SELFREF
yakra@BiggaTomato:~/TravelMapping/yakra/DataProcessing/siteupdate/cplusplus/logs$ diff waypointsimplification.old waypointsimplification.i5 | egrep '^< Straightforward concurrency'
< Straightforward concurrency: L3238Her@L3238&L3238Mar@L3238 -> L3238Her/L3238Mar@L3238
< Straightforward concurrency: B279@B279&B279Hol@B279 -> B279/B279Hol@B279
< Straightforward concurrency: A329@A329_E&A329Rea@A329_E -> A329/A329Rea@A329_E
yakra@BiggaTomato:~/TravelMapping/yakra/DataProcessing/siteupdate/cplusplus/logs$ egrep 'L3238Her@L3238&L3238Mar@L3238|B279@B279&B279Hol@B279|A329@A329_E&A329Rea@A329_E' waypointsimplification.i5
Straightforward intersection: L3238Her@L3238&L3238Mar@L3238 -> L3238/L3238
Straightforward intersection: B279@B279&B279Hol@B279 -> B279/B279
Straightforward intersection: A329@A329_E&A329Rea@A329_E -> A329_E/A329_E
[yakra@noreaster /home/terescoj/travelmapping/HighwayData]$ egrep -n 'L3238|B279|A329' datacheckfps.csv
4030:deuby.b279;B279;;;LABEL_SELFREF;
4031:deuby.b279hol;B279;;;LABEL_SELFREF;
4109:deuhe.l3238her;L3238;;;LABEL_SELFREF;
4110:deuhe.l3238mar;L3238;;;LABEL_SELFREF;
4425:eng.a0329;A329_E;;;LABEL_SELFREF;
4426:eng.a0329;A329_W;;;LABEL_SELFREF;
4427:eng.a0329rea;A329_E;;;LABEL_SELFREF;
4428:eng.a0329rea;A329_W;;;LABEL_SELFREF;

Should arguably have the abbrev included...



INTERSTATE_NO_HYPHEN (9)
for rg in AK AL AR AS AZ CA CO CT DC DE FL GA GU HI IA ID IL IN KS KY LA MA MD ME MI MN MO MP MS MT NC ND NE NH NJ NM NV NY OH OK OR PA PR RI SC SD TN TX UT VA VI VT WA WI WV WY; do egrep '^I[0-9]|ToI[0-9]' $rg/*/*.wpt; done


UNDERSCORE_NUMBER (40)
numeral immediately follows underscore
for rg in `ls | grep -v _`; do grep -sv '^+' $rg/*/*.wpt | cut -f1 -d' ' | grep ':.*_[0-9]'; done


CLOSED_OPEN_COLOC (25)
Not always a true error.
Check with the other datachecks while WPTs are read, and thus while colocation lists are being populated. Iterate thru earlier points in list, breaking when current point is reached.


$rg/Can, Can/$rg, Mex/$rg, $rg/Mex border labels in USA (10)
for rg in AK AL AR AS AZ CA CO CT DC DE FL GA GU HI IA ID IL IN KS KY LA MA MD ME MI MN MO MP MS MT NC ND NE NH NJ NM NV NY OH OK OR PA PR RI SC SD TN TX UT VA VI VT WA WI WV WY; do egrep -i "^$rg/Can|^Can/$rg|^Mex/$rg|^$rg/Mex" $rg/*/*.wpt; done

banner_before_number (41)
for rg in `ls | grep -v _`; do for banner in Bus Trk His Alt Byp BS Fut AltTrk Sce BusTrk AltBus AltByp Wye; do grep -sv "^+" $rg/*/*.wpt | cut -f1 -d' ' | egrep -i ":.*$banner[A-Z]{,2}-?[0-9]+"; done | sort; done
http://forum.travelmapping.net/index.php?topic=2601.msg15607#msg15607

Old routes with false abbrevs that should be after the underscore (44)
Be sure to eliminate all potential banners.
Work on excluding FPs, e.g. Old3RRd Old9FootRd Old41Hwy Old4Hwy_N OldNo9Hwy
Tie into LACKS_GENERIC routines?
for rg in `ls | grep -v _`; do grep -sv '^+' $rg/*/*.wpt | cut -f1 -d' ' | egrep -v 'Alt$|Alt_[NEWS]$|Bus$|Bus_[NEWS]$|Byp$|Byp_[NEWS]$|Spr$|Spr_[NEWS]$' | egrep -i ':.*Old.*[0-9]+[A-Z]{3}'; done

CR-42, FI-42, ME-42, US-42, etc. (203)
for rg in `ls | grep -v _`; do egrep -s '^[A-Z]{2}-' $rg/*/*.wpt; done

Quote from: Quotey McQuoteface
SLASH_LONG_PREFIX {stash} (245)
for rg in `ls | grep -v _`; do grep -sv '^+' $rg/*/*.wpt | cut -f1 -d' ' | egrep -v ':\*?[A-Za-z]+/[A-Za-z]+$' | egrep ':.*/[A-Za-z]{2,}[0-9]'; done

slashed designation, unless: (number, optionally following single letter) (338)
for rg in `ls | grep -v _`; do grep -sv '^+' $rg/*/*.wpt | cut -f1 -d' ' | egrep -iv ':.*/[A-Z]?[0-9]|:\*?[A-Z]+/[A-Z]+$' | egrep ':.*+/'; done
overlap with graph-based unexpected_slash?
Take a closer look at the differences between these two

UNIQUE_SLASH_SUFFIX (445)
http://travelmapping.net/devel/manual/wayptlabels.php#avoidsuffix
http://travelmapping.net/devel/manual/wayptlabels.php#identicalmultiplex
https://travelmapping.net/devel/manual/wayptlabels.php#plexnosuffix

PARENS_LONG_PREFIX (611)
for rg in `ls | grep -v _`; do grep -sv '^+' $rg/*/*.wpt | cut -f1 -d' ' | grep -v '([A-Z]\?[0-9]' | grep '(..[0-9]'; done

only_1_intersecting
Points with a '/' in label colocated with only 1 (or 0?) other
Lots of FP potential. Borders! Don't flag when...
• points are in different regions?
• point is at beginning or end of route?

extraneous_suffix
either parenthetical or underscored, when only one jct with route
this could be a little tricky

I-designation + unexpected letter, excluding common banners
egrep -sv '^\+|^\*?I\-[0-9]+B[LS]|^\*?I\-[0-9]+Fut|^\*?I\-[0-9]+Spr|^\*?I\-[0-9]+Trk|^\*?I\-35[EW]|^\*?I-69[CEW]' $rg/*/*.wpt | cut -f1 -d' ' | egrep 'I\-[0-9]+[A-Za-z]'; done
A couple "Con" examples, incl 2 in NJ

further examples from this thread, or the grep thread

route names in CSVs that break label naming rules
E.G. too many words
E.G. mexsf.csv: AutUrbSurCue
« Last Edit: November 11, 2020, 12:17:07 pm by yakra »

Offline yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2938
  • Last Login:Yesterday at 09:04:27 pm
Re: yakra's collaborator thread
« Reply #19 on: November 28, 2019, 02:56:10 pm »
mysql> select distinct banner from routes left join systems on systems.systemName = routes.systemName where countryCode = 'USA';
+--------+
| banner |
+--------+
|        |
| Bus    |
| Trk    |
| His    |
| Alt    |
| Con    |
| Lp     |
| Byp    |
| Spr    |
| BL     |
| BS     |
| Fut    |
| AltTrk |
| Sce    |
| BusTrk |
| AltBus |
| AltByp |
| Wye    |
+--------+
18 rows in set (0.07 sec)

Offline yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2938
  • Last Login:Yesterday at 09:04:27 pm
Re: yakra's collaborator thread
« Reply #20 on: March 10, 2020, 04:27:29 am »
TX/usaib/tx.i040bsgle.wpt:OldUS66
TX/usatxf2/tx.fm0524.wpt:OldFM425
TX/usatxf2/tx.fm0546.wpt:OldFM546_W
TX/usatxf3/tx.fm1021.wpt:OldFM2366
TX/usatxf/tx.fm0078.wpt:OldTXLp539_E
TX/usatxf/tx.fm0078.wpt:OldTXLp539_W
TX/usatxf/tx.fm0105.wpt:OldHwy90
TX/usatxf/tx.fm0413.wpt:OldTX6_N
TX/usatxf/tx.fm0413.wpt:OldTX6_S
TX/usatxf/tx.fm0418.wpt:OldFM418
TX/usatxf/tx.fm0418.wpt:OldHwy418
TX/usatxf/tx.fm0482.wpt:OldUS81
TX/usatxl/tx.lp0121.wpt:OldFM594
TX/usatxl/tx.lp0223.wpt:OldHwy20_W
TX/usatxl/tx.lp0223.wpt:OldHwy20_E
TX/usatxr/tx.rm0473.wpt:OldNo9Hwy
TX/usatxr/tx.rm1431.wpt:OldRM1431_W
TX/usatxr/tx.rm1431.wpt:OldRM1431_E
TX/usatxs/tx.sp0186.wpt:OldHwy20
TX/usatxs/tx.sp0419.wpt:OldFM524_S
TX/usatxs/tx.sp0419.wpt:OldFM524_N
TX/usatx/tx.tx025.wpt:OldUS82
TX/usatx/tx.tx034.wpt:OldTX34
TX/usatx/tx.tx035.wpt:OldTXLp419
TX/usatx/tx.tx036.wpt:OldHwy36
TX/usatx/tx.tx066.wpt:OldFM3211
TX/usatx/tx.tx137.wpt:OldTX137
TX/usatx/tx.tx151.wpt:OldHwy90
TX/usatx/tx.tx154.wpt:OldTX154
TX/usatx/tx.tx163.wpt:OldTX163
TX/usatx/tx.tx290.wpt:OldUS290
TX/usatx/tx.tx359.wpt:OldTX359
TX/usausb/tx.us059busspl.wpt:OldHwy59
TX/usaush/tx.us066hisgle.wpt:OldUS66
TX/usaus/tx.us067.wpt:OldSpr310
TX/usaus/tx.us079.wpt:OldFM1460
TX/usaus/tx.us090.wpt:OldUS90
TX/usaus/tx.us175.wpt:OldUS175_S
TX/usaus/tx.us175.wpt:OldUS175_N
TX/usaus/tx.us181.wpt:OldUS181_B
TX/usaus/tx.us181.wpt:OldUS181_C
TX/usaib/tx.i035blalv.wpt:*OldUS67Bus
TX/usai/tx.i030.wpt:*OldI-30_W
TX/usai/tx.i030.wpt:*OldI-30_E
TX/usatxf2/tx.fm0546.wpt:*OldFM546_E
TX/usatxf5/tx.fm2087.wpt:*OldTXLp281
TX/usatxf/tx.fm0121.wpt:*OldFM121_A
TX/usatxf/tx.fm0121.wpt:*OldFM121_B
TX/usatxf/tx.fm0121.wpt:*OldFM121_C
TX/usatxf/tx.fm0121.wpt:*OldFM121_D
TX/usatxf/tx.fm0481.wpt:*OldFM481_W
TX/usatxf/tx.fm0481.wpt:*OldFM481_E
TX/usatxl/tx.lp0281.wpt:*OldTXLp281
TX/usatxs/tx.sp0334.wpt:*OldUS183
TX/usatx/tx.tx005.wpt:*OldTX5
TX/usatx/tx.tx043.wpt:*OldFM1794
TX/usatx/tx.tx114.wpt:*OldUS82
TX/usatx/tx.tx121.wpt:*OldTX5
TX/usatx/tx.tx188.wpt:*OldTX188_W
TX/usatx/tx.tx188.wpt:*OldTX188_E
TX/usatx/tx.tx195.wpt:*OldTX195
TX/usausb/tx.us067busalv.wpt:*OldUS67Bus_S
TX/usausb/tx.us067busalv.wpt:*OldUS67Bus_N
TX/usaus/tx.us082.wpt:*OldUS82
TX/usaus/tx.us082.wpt:*OldUS82_W
TX/usaus/tx.us082.wpt:*OldUS82_E
TX/usaus/tx.us181.wpt:*OldUS181_A
TX/usaus/tx.us181.wpt:*OldTX359
TX/usaus/tx.us277.wpt:*OldUS277_S
TX/usaus/tx.us277.wpt:*OldUS277_N
TX/usaus/tx.us377.wpt:*OldI-30_W
TX/usaus/tx.us377.wpt:*OldI-30_E

Offline yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2938
  • Last Login:Yesterday at 09:04:27 pm
Re: yakra's collaborator thread
« Reply #21 on: November 02, 2020, 10:35:37 am »
EXTRANEOUS_EXIT (845)
Find unique parenthetical suffixes combined with unique prefixes. There will be false positives.
count=0
for rg in $(ls | grep -v _); do
  for dummy in $(ls $rg/*/*.wpt | tr ' ' '%'); do
    file=$(echo $dummy | tr '%' ' ')
    Ls=$(grep -v '^+' "$file" | cut -f1 -d' ' | grep '(.*)' | cut -f1 -d'(' | sed 's~^\*~~' | sort | uniq)
    Rs=$(grep -v '^+' "$file" | cut -f1 -d' ' | grep '(.*)' | cut -f2 -d'(' | sed 's~)$~~'  | sort | uniq)
    for l in $Ls; do
      if [ $(grep -m 2 "^\*\?$l(" "$file" | wc -l) = '1' ]; then
        for r in $Rs; do
          if [ $(cut -f1 -d' ' $file | grep -m 2 "($r)" | wc -l) = '1' ]; then
            result=$(grep -H "^$l($r)" "$file")
            if [ $(echo $result | wc -w) -gt '0' ]; then
              count=$(echo $count+1 | bc)
              echo -en "$count\t"
              echo $result
            fi
          fi
        done
      fi
    done
  done
done | tee extraneous_suffix.log


Send Firefox to all routes in each region, centered at the waypoint in question
for rg in $(cat ~/tmregions); do echo -n $rg; cut -f2 extraneous_suffix.log | grep "^$rg/" | sed "s~^$rg/[a-z0-9]\+/\(.*\)\.wpt:.*\(lat=[-.0-9]*&lon=[-.0-9]*\)~https://travelmapping.net/hb/showroute.php?r=\1\&\2\&zoom=14~" | xargs firefox; read go; done
« Last Edit: November 20, 2020, 12:58:25 am by yakra »

Offline yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2938
  • Last Login:Yesterday at 09:04:27 pm
Re: yakra's collaborator thread
« Reply #22 on: November 02, 2020, 03:55:01 pm »
UNIQUE_SLASH_SUFFIX (445)
http://travelmapping.net/devel/manual/wayptlabels.php#avoidsuffix
http://travelmapping.net/devel/manual/wayptlabels.php#identicalmultiplex
https://travelmapping.net/devel/manual/wayptlabels.php#plexnosuffix
count=0
for rg in $(ls | grep -v _); do
  for dummy in $(ls $rg/*/*.wpt | tr ' ' '%'); do
    file=$(echo $dummy | tr '%' ' ')
    Ls=$(grep -v '^+' "$file" | cut -f1 -d' ' | grep '/.*_' | cut -f1 -d_ | sed 's~^\*~~' | sort | uniq)
    for l in $Ls; do
      if [ $(grep -m 2 "^\*\?$l" "$file" | wc -l) = '1' ]; then
        count=$(echo $count+1 | bc)
        echo -en "$count\t"
        grep -H "^\*\?$l" "$file"
      fi
    done
  done
done | tee UNIQUE_SLASH_SUFFIX.log


Send Firefox to all routes in each region, centered at the waypoint in question
for rg in $(cat ~/tmregions); do echo -n $rg; cut -f2 UNIQUE_SLASH_SUFFIX.log | grep "^$rg/" | sed "s~^$rg/[a-z0-9]\+/\(.*\)\.wpt:.*\(lat=[-.0-9]*&lon=[-.0-9]*\)~https://travelmapping.net/hb/showroute.php?r=\1\&\2\&zoom=14~" | xargs firefox; read go; done

Offline yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2938
  • Last Login:Yesterday at 09:04:27 pm
Re: yakra's collaborator thread
« Reply #23 on: November 22, 2020, 01:46:51 am »
Operation Spur

C++: Compile root pairs
Code: [Select]
std::unordered_map<std::string, std::list<Route*>> RteBan;

for (auto h : highway_systems)
  for (auto &r : h->route_list)
    RteBan[r.region->code+r.route+r.banner].push_back(&r);

for (auto &RteList : RteBan)
  if (RteList.second.size() > 1)
    for (Route* r : RteList.second)
      for (Waypoint *w : r->point_list)
if (w->colocated)
  for (Waypoint *p : *w->colocated)
    if (w < p && w->route != p->route && \
w->route->region->code + w->route->route + w->route->banner == \
p->route->region->code + p->route->route + p->route->banner)
      std::cout << "RESULT:\t" << w->route->root << '\t' << p->route->root << std::endl;
return 0;

label search
   std::unordered_map<std::string, std::list<Route*>> RteBan;

   for (auto h : highway_systems)
     for (auto &r : h->route_list)
       RteBan[r.region->code+r.route+r.banner].push_back(&r);

   for (auto &RteList : RteBan)
     if (RteList.second.size() > 1)
       for (Route* r : RteList.second)
         for (Waypoint *w : r->point_list)
      if (w->colocated)
      { std::string wrb = w->route->route + w->route->banner;
        for (Waypoint *p : *w->colocated)
        { std::string prb = p->route->route + p->route->banner;
          if (w != p && w->route != p->route && w->route->region->code + wrb == p->route->region->code + prb)
          {   if (w->route->abbrev.size() && w->label.substr(0,prb.size()) != prb)
           std::cout << "RESULT:" << w->route->region->code << '/' << w->route->system->systemname << '/' << w->route->root << ".wpt\t" << w->label << std::endl;
          }
        }
      }
   return 0;


Sh: Unique root list
t=`echo -en '\t'`
results=`grep '^RESULT' Spr/logs/siteupdate.log | sed "s~^RESULT:$t~~" | tr '\t' '\n' | sort | uniq`
echo $results | wc -w

1940

Sh: Look up in system CSVs
cd ~/TravelMapping/HighwayData/
systems=`tail -n +2 systems.csv | grep -v '^#' | cut -f1 -d';' | sed 's~.*~&.csv~'`
cd hwy_data/_systems
for r in $results; do grep ";$r;" $systems; done | wc -l

1940

cd ~/TravelMapping/HighwayData/hwy_data/_systems
n_abbrev=`for r in $results; do egrep ".*;.*;.*;.*;;.*;$r;.*" $systems | cut -f7 -d';'; done`
710
y_abbrev=`for r in $results; do egrep ".*;.*;.*;.*;.+;.*;$r;.*" $systems | cut -f7 -d';'; done`
1230
bad=`for r in $results; do egrep ".*;.*;.*;.*;.+;.*;$r;.*" $systems; done | grep '(spur)\|(loop)\|(branch)' | cut -f7 -d';'`
204
good=`for r in $results; do egrep ".*;.*;.*;.*;.+;.*;$r;.*" $systems; done | grep -v '(spur)\|(loop)\|(branch)' | cut -f7 -d';'`
1026


Sh: How many regions?
echo $good | tr ' ' '\n' | cut -f1 -d. | uniq | wc -l
112
(9.2 rtes/rg)
echo $bad | tr ' ' '\n' | cut -f1 -d. | uniq | wc -l
41
(5 rtes/rg)

for rg in `echo $good | tr ' ' '\n' | cut -f1 -d. | uniq`; do echo -n "$rg "; echo $good |tr ' ' '\n' | grep "^$rg" | sed 's~.*~https://travelmapping.net/hb/showroute.php?r=&~' | xargs firefox; read go; done; echo


To be continued...
« Last Edit: November 23, 2020, 02:03:38 am by yakra »