Failure to restore CSV backup in latest TRACKS appliance (14.2-jessie)

It does seem like the directory not existing (and permissions of it) are clearly issues - well worth resolving. But it sounds like there may be something more to it than that?!

I'm pretty tied up with development for our upcoming v15.0 release, so can't look into this further right now. But please don't take my lack of further testing etc to mean lack of interest. Please post back once you've had another look at it and we may well open a new issue against the appliance if you can confirm it's an issue (especially if you can document how to reliably reproduce the issue).

Having said that, it does sound as if it may actually be an upstream issue (i.e. an issue with the actual software itself, as opposed to our specific install/implementation). FWIW I did have a browse of the upstream issue tracker and couldn't find anyone else reporting your exact problem. So perhaps it is somewhat unique to our install?

Also another thing I noticed was that whilst the version we install (v2.3.0) is still the latest stable release, that was released nearly 3 years ago! The master branch has had an additional ~360 commits since then! From what I gather, their master branch is their development branch, so I'm unclear on whether trying to install that (or upgrade to that) would be a worthy exercise or not, but perhaps worth keeping in mind.

If you wish to try that, I'm open to assisting as best I can, but can only offer "best effort". And as I hinted earlier, Ruby is not something I'm at all familiar with.

Hello, you're right. I forgot to check user permissions. Your second command changed ownership and solved the issue. Sort of. I mean, the upload problem is gone: now I can send CSV files and tracks grab and munch all of them. However, the restore process is actually... not working.

I mean, it does not complain about upload issues anymore. But it does not restore my projects, either.
To reproduce, create a single project with a couple of test tasks. Export. Delete the project and tasks. When trying to import the CSV file, it does not work as expected.

Wait, something weird just happened. My projects are gone, but a loooong list of todo's (hundreds) are showing. Wow. Time to stop. Tomorrow, after resting, I'll be able to test this properly. Will return the snapshot, clear the database status and see what happens.

Talk to you tomorrow.
Thank you!
Paulo

My guess is that you're right that it's a permissions thing. FWIW I'm not very familiar with Ruby myself, but this appears to be pretty clear. So please try this:

mkdir -p /var/www/tracks/public/uploads/csv
chown -R www-data:www-data /var/www/tracks/public/uploads/csv

Although as I noted in the issue I've opened on our tracker, perhaps giving www-data (the webserver user) ownership of the whole uploads tree would be best (in case there is some other stuff that can be uploaded that would also fail)?

Your feedback on whether this actually solves the issue would be warmly welcomed. :)

FWIW some time ago (literally years ago) I also used Tracks, with Android Shuffle. But I stopped using Shuffle because it became incompatible with one release of Tracks (I forget the exact issue, but may perhaps be similar to what you report?).

I did continue to use tracks, but just through the web interface for a while, but for some reason I stopped using it (again I forget the details sorry).

Regardless, great work and anything you can add, I'm more than pleased to hear about it. I intend to implement the 2 fixes you've already provided us with, into the v15.0 appliance. And if you can work out something for this one more specifically, then we can add that too.

As for the Tracks upstream, I too have noticed the somewhat re-invigoration of the project. However, I recall reading somewhere recently (on the Tracks issue tracker on GH I think?) that the main dev has set himself a timeframe of "this August"(?) to get it back on track (excuse the pun). If so, it will continue to move forward. Otherwise, he says that he will abandon it completely and that will be the end of it (unless someone else forks it and maintains it).

Hello Jeremy,

sorry for another long post, but I need to record (and share) my last tests, or I will quickly forget them in the next days.

My first tests show that the restore process from CSV is less than perfect. It's quite fragile, indeed. I ran several test scenarios, and here's what I got.

1) Just to record: Recreating the folders and adding permissions really solve the upload issue. That's quite important, because it's possible to do a partial restore this way (see next comment). And yes, there's more than that.

2) About my tests with CSV files created by this appliance. I created a few files: one was an empty export, another added tasks/exported and so on. Restoring from these files can partially recover tasks, but all of them are separated from their respective, original projects. But that certainly helps a lot. By the way, this test was with a reduced set of only a few tasks, all of them in plain english. That was to avoid issues with punctuation, foreign characters and the like. My mother tongue is Portuguese: I'm from Recife, Brasil, by the way. I'm tired to stumble upon problems relater to language questions.

3) I also tested with real CSV files, created by me using other version of tracks. I use it to track my GTD actions and had a quite large set of projects, with hundreds of tasks recorded (a few MB). Both files share exactly the same structure: same number of columns, in the same order, with the same names. That's important to make sure that an older version was not breaking my tests, due to a changed DB structure. These are large, and all in Portuguese, with á, ç, ã and things like that. It imports with the same behavior: tasks are restored, but without an assigned project. Oh, the project information is present inside the CSV files: this was checked also.

4) Another test was to restore data through the sync api. Being a fan of open-source projects, I try to use them in a daily basis. I love Android Shuffle, for synchronization with tracks (see next links). It has some rough edges, but for me it works quite well. At least, as long as you keep in mind that it seems to consider the tracks server as the "main source of data".

https://play.google.com/store/apps/details?id=org.dodgybits.android.shuf...
https://github.com/andybryant/shuffle-android
From Andy Briant

Well, I had my database backed up also in my cell phone (inside shuffle and with a backup file). Unfortunately, syncing from an Android Shuffle instance (full of data) agains an empty Tracks server was a complete disaster. The server is considered to be newer and more up-to-date, which in turn rendered the Shuffle database completely empty. So, I could not use it (yet) to restore the Tracks server.

5) Another test was to sync Shuffle agains the server, which had both in sync and cleared shuffle. Then I restored my Shuffle backup, which recreated all lost projects and tasks. Then tried to sync again. I was expecting that maybe, with a lot of luck, perhaps some hidden timestamp could change. That, in turn, could make Tracks think that the newly restored projects were actually new stuff, created after syncing inside my phone. Unfortunately, the end result was the same: an empty database in both sides (server and cell/client).

6) Yet another test was to transfer the shuffle backup file into my linux machine, in order to take a look and see what was inside of it. Was expecting to find a sqlite database, binary file, CSV, XML or something readable. But it looks like a binary, specific format (at least the "file" command was unable to identify it). It was not easy to inspect, even with "strings" and a binary editor. Even tried to take a quick look into Shuffle code, but this is not a trivial task. It uses a Google API to sync, with more than one ID per record, in a way I could not grasp quickly.  So, I quit this approach for now.

7) However, the other way around works reasonably well. Adding data through the Tracks web interface and then syncing it into Shuffe works almost like a charm. Syncing back (updating shuffle and syncing it to tracks)  is still not perfect. Sometimes you may end up with duplicated entries (among other bad things). But for the average user that add tasks through the web interface and use the cell phone only to read and mark as "done", it's good enough (at least for me).

8) The next step was moving away from Shuffle and into GitHub. Tracks is a project with varying support along the time. It has several contributors, but spread along the years. From 2008 until the end of 2015, it went through a strong, steady sequence of very frequent commits, until the first months of 2016. However, things changed. During 2016, 2017 and the first months of 2018, fixes and commits were much less frequent. But in the last two months, it seems to be gaining some traction again. Maybe there's a new version coming, in the next months: who knows?

https://github.com/TracksApp/tracks/graphs/contributors?from=2016-04-12&...

9) Considering my tests and the little I know about Tracks (and the http://www.getontracks.org/ site), I don't thinks this is something unique to TurneyLinux install. Actually I could not test in another scenario (local server), so I cannot state that a clean install would behave in a better way. But this server seems to behave mostly like what I'm already used (backup restoring not tested before).

Uff. That stated, here are a couple of important results/next actions.

10) One (very manual) workaround to restore projects seems to be this. Restore your CSV file, which will make tasks appear. Then recreate manually all projects. Then, reassign all of them to their respective projects, one by one. I have more than 900 tasks still not done, so this will not work for me (unless there's no other way).

11) I will continue testing and see what I can learn about this problem. Maybe updating tracks could be interesting. Next week will be hard, so I may write back only next weekend. Maybe before, let's see.

Jeremy, about being busy, that's ok. Take your time. I feel I need more time to dig deeper into this problem. Not sure yet if it's worth updating into an unstable branch for tracks, but perhaps it will be.

Will post back as soon as something new arises.

Regards,
Paulo