You are here
OnePressTech - Sun, 2020/10/25 - 08:51
Can't run TKLBAM on my LAMP 16.1 server...it kills MySQL (well MariaSQL). LAMP server with WordPress installed (backs up the WordPress MySQL fine...it fails on the MySQL db Backup)
I added a swapfile and ran top while TKLBAM was running. Plenty of CPU, plenty of disk, plenty of RAM free when it fails. Not a resource issue. It literally kills MySQL while it is taking a snapshot. Here is as far as it got:
I added a swapfile and ran top while TKLBAM was running. Plenty of CPU, plenty of disk, plenty of RAM free when it fails. Not a resource issue. It literally kills MySQL while it is taking a snapshot. Here is as far as it got:
database: mysql table: mysql/column_stats table: mysql/columns_priv table: mysql/db table: mysql/event table: mysql/func table: mysql/gtid_slave_pos table: mysql/help_category table: mysql/help_keyword table: mysql/help_relation table: mysql/help_topic table: mysql/host table: mysql/index_stats table: mysql/innodb_index_stats table: mysql/innodb_table_stats table: mysql/plugin table: mysql/proc table: mysql/procs_priv table: mysql/proxies_priv table: mysql/roles_mapping table: mysql/servers table: mysql/table_stats table: mysql/tables_priv table: mysql/time_zone table: mysql/time_zone_leap_second table: mysql/time_zone_name table: mysql/time_zone_transition table: mysql/time_zone_transition_type table: mysql/user table: mysql/general_log table: mysql/slow_log table: mysql/transaction_registry Traceback (most recent call last): File "/bin/tklbam-backup", line 443, in main opt_resume, True, dump_path if dump_path else "/") File "/usr/lib/tklbam/backup.py", line 237, in __init__ self._create_extras(extras_paths, profile_paths, backup_conf) File "/usr/lib/tklbam/backup.py", line 183, in _create_extras limits=conf.overrides.mydb, callback=mysql.cb_print()) if self.verbose else None File "/usr/lib/tklbam/mysql.py", line 570, in backup mna.stop() File "/usr/lib/tklbam/mysql.py", line 729, in stop self.command.wait() File "/usr/lib/python2.7/dist-packages/command.py", line 299, in wait self._child.wait() File "/usr/lib/python2.7/dist-packages/popen4.py", line 204, in wait pid, sts = os.waitpid(self.pid, 0) KeyboardInterrupt
Forum:
The stacktrace suggests that it was waiting for MySQL?!
Thanks for reporting Tim and thanks for checking RAM (that would have been my first guess).
Firstly, do you mean v15.1 or v16.0 (there is no LAMP v16.1 yet)?
Also, out of interest, does this DB come from a previous TurnKey server (i.e. migrated via TKLBAM)?
We've had a somewhat similar issue reported previously (which AFAIK was never properly resolved). The user experiencing it had been using TKLBAM for years with this DB (and migrating between many TurnKey versions) and it turned out that there were a load of old, no longer supported config that only appeared to be causing issues under certain scenarios. However, even after tidying that up, there were still issues. He shared a sanitised copy of the DB dump (IIRC only user emails and passwords removed) and I restored that to a local VM (I didn't even give it a lot of and couldn't reproduce the issue. So in the end, we just disabled TKLBAM backing up the DB and created a TKLBAM hook to dump the DB to a SQL file pre backup (then just backup the SQL as a SQL dump file).
It's not ideal and I'd really love to understand what the actual issue is/was but without access to the data/config that can reproduce it locally, it's near impossible.
I don't know for a fact (and actually this is total speculation) but my suspicion is that it's something to do with character encoding. I suspect that there is some encoding mismatch (between MySQL and TKLBAM) somewhere that is causing a character that can't be properly interpreted and/or converted between different encodings. MySQL chokes and the whole thing falls over...
So if you're open to sharing your data/database/config/etc, then I'm more than happy to see if I can reproduce it. And if I can, I'm confident that we can develop a fix... Please feel free to send it through to me (you've got my email).
Fresh VM, 16.0, No TKLBAM migration involved
This is a fresh V16 LAMP ( sorry about the 16.1 slip...late night). I added a blank WP database and used a WordPress duplicator to migrate WordPress from a pre-existing older TKXL VM. Manual WordPress migration...no TKLBAM involved.
So greenfield LAMP, WordPress installed (multisite which is why I did not use the TKLX WordPress image), T2 Medium. Plenty of disk, RAM, CPU, etc. I didn't do anything else.
I'll take a look at the MySQL database. It seems to always hang on the transaction table.
I'm not sure why others are not experiencing this type of issue...I am not really doing anything fancy here...LAMP stack with a migrated WordPress installation. If TKLBam crapped out in the WordPress database or the start of the MySQL database I would probably have something to go on...but TKLBAM processes the WordPress tables fine then starts to process the MySQL tables fine...then hangs part way through. Nothing I did would have touched the MySQL table...it should be an untouched out-of-the-box TKLX LAMP MySQL db table...weird!
Cheers,
Tim (Managing Director - OnePressTech)
Hmm yes, very weird...
From your report it certainly does seem like the issue should be occurring on a default LAMP/MySQL (MariaDB) instance. However, I explicitly tested LAMP way back when I was pre-release testing TKLBAM (before we relases v16.0).
I might give it another test soon though as I know that there have been updates to MariaDB since we released LAMP v16.0.
FWIW my suspicion of encoding is due to the move to "proper" UTF-8 encoding support in MySQL/MariaDB and the fact that TKLBAM is still in Python2 (which doesn't handling encoding particularly well). Although as I say, it seems like it's actually MariaDB that chokes (and TKLBAM waits for it - at least that's what the stacktrace you posted suggests...).
Please share any further info you come across and if you want to try the workaround that I mentioned, I'm more than happy to be more explicit on testing and setting that up. Also if I find anything else on my end, I'll post back.
Adding to the weirdness, I definately can't reproduce... :(
So I've tried to reproduce this. I started with a fresh LAMP v16.0 (installed to a local KVM VM). In an effort to try every possibility that I could think of, I started by not installing the security updates. Then:
That went fine, so I installed the sec updates:
I note that there were a number of MariaDB related security updates:
That too succeeded, so I then did all the remaining (i.e. non security) updates. There were none that appeared to be relevant to MariaDB nor TKLBAM. I rebooted again to be sure, then forced a 3rd full backup:
That one too went ok. So I installed WordPress. It's been a while since I manually installed WordPress, so I just followed a random online tutorial I found. FWIW, these are the steps that I ran:
I then edited wp-config.php to add the DB details and finished the install via the web browser. I checked everything was ok and and updated the Akismet Anti-Spam plugin.
For good measure, I rebooted again and then forced another full TKLBAM backup:
However, this one worked fine too?! FWIW here's the matching bit to where your's crashes:
It doesn't even hesitate where yours seems to get stuck... My VM only has 1GB RAM and 2 vCPU cores.
Looking back over everything, I note that the WordPress docs don't note explicit use of "DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci" - so I'm not sure if that has any impact?
So I dumped the existing DB and recreated it:
I then reinstalled WordPress (the web browser bit) and again forced a full TKLBAM backup:
Same deal, everything "just works"...?!? I'm not really sure where we can go from here?
Add new comment