Problem with TKLBAM backup (+simulation)

Hello,
I setup the TKLBAM but I'm getting an error. Everytime the missing file is different and most of the time related to WP Total Cache files.
Why is this?

Traceback (most recent call last):
  File "/usr/bin/tklbam-backup", line 510, in <module>
    main()
  File "/usr/bin/tklbam-backup", line 443, in main
    opt_resume, True, dump_path if dump_path else "/")
  File "/usr/lib/tklbam/backup.py", line 237, in __init__
    self._create_extras(extras_paths, profile_paths, backup_conf)
  File "/usr/lib/tklbam/backup.py", line 175, in _create_extras
    conf.overrides.fs)
  File "/usr/lib/tklbam/backup.py", line 111, in _write_whatchanged
    changes = whatchanged(dirindex, paths)

  File "/usr/lib/tklbam/changes.py", line 265, in whatchanged
    changes += [ Change.Overwrite(path) for path in new + edited ]
  File "/usr/lib/tklbam/changes.py", line 92, in __init__
    self.uid = self.stat.st_uid
  File "/usr/lib/tklbam/changes.py", line 67, in stat
    self._stat = os.lstat(self.path)
  OSError: [Errno 2] No such file or directory: '/var/www/--------/wp-content/cache/page_enhanced/----/----/_index_ssl.html_old'

Replies (3)

Hi there, I suggest an override

Reply 1 2020-02-04

Firstly, I hope you don't mind but I reformatted your post and added a couple of tags that I thought might be useful... I also removed part of the TKLBAM output to make the stacktrace stand out a little more. FWIW, if you're unsure on what/how much to post of an error message, you are better off posting more rather than less; so you did the right thing, I just wanted to clarify the important part in this instance (the stacktrace itself).

Essentially, it appears what is happening is that there is a race condition when TKLBAM is trying to create a backup. The first thing it does is scans for files to backup, then it backs them up. But in your case, at least one of the files that it's trying to backup does not exist when it tries to back it up...

However, seeing as it appears to be a cache file in this instance (and judging from your earlier comment, is always) then that's easily fixed. Besides, you probably don't want cached files included in your backup anyway, as they're just going to be wasted space. By the time you come to restore, they'll almost certainly be stale anyway...

So the easiest (and best) way to work around that issue would be to exclude the cache location from being part of the backup. TBH, I'm not 100% sure whether you are best off excluding the complete '/var/www/--------/wp-content/cache' directory, or whether you shoudl try to drill down a bit more to exclude specific sub-directories. I suggest that you do some testing, but of backups and restores (perhaps even to a separate VM to ensure that not only is it ok to exclude those dirs, but also that it all works as expected in general).

To add an exclusion, add the full path to the directory (or file) as a separate line in the /etc/tklbam/overrides file. E.g. If you wished to exclude /var/www/wordpress/cache:

echo "/var/www/wordpress/cache" >> /etc/tklbam/overrides

I hope that helps. Please post back and let us know how you go! :)

Allowing it to succeed with missing files sounds dangerous!?

Reply 2 2020-02-06

As I mentioned, it's a race condition. So we're talking about fractions of a second between when TKLBAM collects the names of the files it will backup and then archiving them ready for upload. We may be able to make some improvements there next time we do some major work on TKLBAM, but in the meantime, IMO allowing a backup to succeed with missing files sounds potentially dangerous!?

Allowing the default WordPress cache location to be included in the backup is an oversight and should be changed in the default TKLBAM profile. I've opened a bug report to ensure that that is changed in the next release.

Beyond the cache location, any reputable production ready application should not be storing volatile files randomly within it's file hierarchy. Cached files are clearly an exception, but should only be being stored within a specific pre-allocated cache location.

TBH, I'd probably go as far as to suggest that the filesystem is not the right place to be storing short lived cache files like that anyway (disk I/O is slow at the best of times...). A far superior path is to use a "proper" purpose designed caching/optimisation application, e.g. Memcached, Varnish or Redis. These (and similar) will cache to RAM giving a far superior performance improvement and will not put additional load on other action which do require writing to disk. When I get a chacne, perhaps I can write up some tutorials on how to install and configure caching applications such as those?!

FWIW generally third party web app plugins that provide on disk caching (which it sounds like you are using) are more aimed at users within a "shared hosting" scenario. Shared hosting is where many users/sites share a server with many others. To minimise the impact of one user on another, things such as installing additional purpose designed software is rarely an option. On a TurnKey server where you have complete control, using one of the above mentioned apps (or another similar product) is generally going to be a much better option; and provide a much better experience.

Regardless of the above, any temporarily cached files aren't required (or desired) to be included in a backup, so should be excluded from the backup anyway.

So beyond cached files, there shouldn't be any files that disappear within a timeframe that will matter to TKLBAM. Also, if you have the system email SMTP relay set up (as is advisable) you should be notified via system email if any TKLBAM error such as this occurs. Once the correct paths are excluded from the backup, this should generally not occur. It's not the first time that users have reported issues such as this, but excluding the cache locations is the best fix. Considering how many thousands of servers that are running and using TKLBAM without issue, assuming correct configuration (which it's clearly not in this case), it's pretty much unheard of.

So to circle back to your suggestion, I would not like to see TKLBAM default to continuing a backup if there are missing files. The right fix IMO is to ensure that the problematic file locations are excluded as required (Ideally via the TKLBAM profile itself so it's invisible to end users).

Although OTOH, I guess that we can't always ensure that other software developers follow best practice, so to have the option to allow users to explicitly decide to allow missing files to only create a warning (rather than an error) is quite a good idea IMO. So I've also created a feature request for that. Once that option is available, perhaps we can reconsider whether we default to that behaviour or not...

As I say though, if you exclude the top level cache location, then everything should be fine. Please let me know if that's not the case. As I note, we'll have a closer look at this soon ourselves anyway, but any specific info you can provide will certainly assist.

Thanks a lot for the

Reply 3 2020-02-06

Thanks a lot for the suggestions Jeremy!

Is there a way to override this error with missing files in general because I don't find the backup reliable if the whole process of backup gets interrupted by a missing file?

There are many websites which cache I need to exclude and it can happen a file to be missing anytime not only within cache.

Thanks!