User uploads

[pderocco]

If Travel Mapping had simple passworded accounts, I would think it would be pretty easy to allow a user named xyz to upload a file called xyz.list to a processing directory, from whence the server could automatically process it and make it visible. It would be really nice if we could see the results of our additions within a minute or so. And it would eliminate the need for Jim or anyone to do this manually.

[michih]

https://github.com/TravelMapping/DataProcessing/issues/22

[Jim]

Two thoughts related to this:

- As long as the TM server resides on my research server hosted by my employer, I am strictly limiting who has write access to any part of it.
- We can get much of this functionality with automated merges and pulls from GitHub in what I believe would be a much more secure process.  It wouldn't give results in a minute, since a site update process takes several minutes, mostly to reload the databases.  I am thinking more along the lines of hourly updates.  I am still very hesitant on the automated merges of GH pull requests, so this might be accomplished by having several people with ability to merge PRs, and those changes would go into the next automated update.

[michih]

having several people with ability to merge PRs, and those changes would go into the next automated update.

Sounds great!

[pderocco]

I'm curious: why are everyone's .list files stored in GitHub? I understand why the software is; that's what GitHub is for. But .list files? Why is it necessary to maintain a history of updates? Isn't just storing the latest version of each file all that's necessary? That would eliminate the whole process of pull requests.

[Jim]

I'm curious: why are everyone's .list files stored in GitHub? I understand why the software is; that's what GitHub is for. But .list files? Why is it necessary to maintain a history of updates? Isn't just storing the latest version of each file all that's necessary? That would eliminate the whole process of pull requests.

The history isn't essential, though many users, including me, put in meaningful commit messages about changes to our files.

It was chosen because we were already going to use it for the site update software, the web front end code, and the highway data.  I saw no reason to use a different solution for user data.

I am open to suggestions about other ways to store them that would work better.

[dave1693]

I personally was using CVS to maintain a history of my .list file dating back to whenever it was I started with CHM. So I myself am quite thankful for the choice to use GitHub for storing .list fines on TM. YMMV.

[yakra]

^ One thing I love about git is the ability to go back thru the history & blame and see when and in what context a given line was added or edited.

It's great for source code, though I bet most people will not make heavy use of it for .list files. Still, it's a tool at our disposal for answering "What else was I doing on that road trip; how did I get there?" type questions.

[pderocco]

As long as GitHub is used to record old versions of .list files, wouldn't it be useful to have a repo for old versions of .log files too? That would let us look up our past stats.

[michih]

Here it is: https://travelmapping.net/updates/

[pderocco]

I was talking about user update .log files, which are full of personal mileages and percentages.

[michih]

Yours from 2024-10-01: https://travelmapping.net/updates/2024-10-01@21:33:39/logs/users/pderocco.log

[pderocco]

I see it now, but it's not in a very useful form, because I have no memory of which days I submitted updates. My point is that, since GitHub has my .list file in the repo, and I can use the normal GitHub or git tools to brows through the past versions of it, my update .log file should also be in there, so I can browse it in the same way.

[michih]

Good point! However, changes to your stats can happen with every update. Due to hwy data changes. you could manually look your list file updates up on Github, and then open the right file in our log archive.
Technically, we do not yet automatically upload data to Github.
In general, we lack interested contributors willing to invest time for coding.

[Jim]

I'm definitely interested in a way to track user histories, and I am sure the log files will be helpful to build such a thing.  As it stands, we can do Unix diffs, the equivalent of Git change logs, on the text files.  But this is not available to regular users directly, and as michih points out, changes in the logs occur not only for your new travels but as highway data is updated and expanded.