TKLBAM kills MySQL

Thanks for reporting Tim and thanks for checking RAM (that would have been my first guess).

Firstly, do you mean v15.1 or v16.0 (there is no LAMP v16.1 yet)?

Also, out of interest, does this DB come from a previous TurnKey server (i.e. migrated via TKLBAM)?

We've had a somewhat similar issue reported previously (which AFAIK was never properly resolved). The user experiencing it had been using TKLBAM for years with this DB (and migrating between many TurnKey versions) and it turned out that there were a load of old, no longer supported config that only appeared to be causing issues under certain scenarios. However, even after tidying that up, there were still issues. He shared a sanitised copy of the DB dump (IIRC only user emails and passwords removed) and I restored that to a local VM (I didn't even give it a lot of and couldn't reproduce the issue. So in the end, we just disabled TKLBAM backing up the DB and created a TKLBAM hook to dump the DB to a SQL file pre backup (then just backup the SQL as a SQL dump file).

It's not ideal and I'd really love to understand what the actual issue is/was but without access to the data/config that can reproduce it locally, it's near impossible.

I don't know for a fact (and actually this is total speculation) but my suspicion is that it's something to do with character encoding. I suspect that there is some encoding mismatch (between MySQL and TKLBAM) somewhere that is causing a character that can't be properly interpreted and/or converted between different encodings. MySQL chokes and the whole thing falls over...

So if you're open to sharing your data/database/config/etc, then I'm more than happy to see if I can reproduce it. And if I can, I'm confident that we can develop a fix... Please feel free to send it through to me (you've got my email).

From your report it certainly does seem like the issue should be occurring on a default LAMP/MySQL (MariaDB) instance. However, I explicitly tested LAMP way back when I was pre-release testing TKLBAM (before we relases v16.0).

I might give it another test soon though as I know that there have been updates to MariaDB since we released LAMP v16.0.

FWIW my suspicion of encoding is due to the move to "proper" UTF-8 encoding support in MySQL/MariaDB and the fact that TKLBAM is still in Python2 (which doesn't handling encoding particularly well). Although as I say, it seems like it's actually MariaDB that chokes (and TKLBAM waits for it - at least that's what the stacktrace you posted suggests...).

Please share any further info you come across and if you want to try the workaround that I mentioned, I'm more than happy to be more explicit on testing and setting that up. Also if I find anything else on my end, I'll post back.

So I've tried to reproduce this. I started with a fresh LAMP v16.0 (installed to a local KVM VM). In an effort to try every possibility that I could think of, I started by not installing the security updates. Then:

tklbam-init MY_HUB_API_KEY
tklbam-backup

That went fine, so I installed the sec updates:

turnkey-install-security-updates

I note that there were a number of MariaDB related security updates:

• mariadb-client-10.3
• mariadb-client-core-10.3
• mariadb-server-10.3
• mariadb-server-core-10.3
• mariadb-common

tklbam-backup --full-backup now

That too succeeded, so I then did all the remaining (i.e. non security) updates. There were none that appeared to be relevant to MariaDB nor TKLBAM. I rebooted again to be sure, then forced a 3rd full backup:

tklbam-backup --full-backup now

That one too went ok. So I installed WordPress. It's been a while since I manually installed WordPress, so I just followed a random online tutorial I found. FWIW, these are the steps that I ran:

mysql --execute "CREATE DATABASE wordpress DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci;"
mysql --execute "GRANT ALL ON wordpress.* TO 'wordpress'@'localhost' IDENTIFIED BY 'my_password';"
apt install php-curl php-gd php-mbstring php-xml php-xmlrpc php-soap php-intl php-zip

cat > /etc/apache2/sites-available/wordpress.conf <<EOF
<VirtualHost *:80>
        DocumentRoot /var/www/wordpress/
</VirtualHost>

<VirtualHost *:443>
        SSLEngine on
        DocumentRoot /var/www/wordpress/
</VirtualHost>

<Directory /var/www/wordpress/>
        AllowOverride All
</Directory>
EOF

a2dissite 000-default.conf
a2ensite wordpress.conf
a2enmod rewrite

systemctl restart apache2

cd /var/www
curl -O https://wordpress.org/latest.tar.gz
tar xzf latest.tar.gz
touch wordpress/.htaccess
cp wordpress/wp-config-sample.php wordpress/wp-config.php
chown -R www-data:www-data /var/www/wordpress

I then edited wp-config.php to add the DB details and finished the install via the web browser. I checked everything was ok and and updated the Akismet Anti-Spam plugin.

For good measure, I rebooted again and then forced another full TKLBAM backup:

tklbam-backup --full-backup now

However, this one worked fine too?! FWIW here's the matching bit to where your's crashes:

[...]
Serializing MySQL database to /TKLBAM/myfs
------------------------------------------

database: mysql
table: mysql/column_stats
table: mysql/columns_priv
table: mysql/db
table: mysql/event
table: mysql/func
table: mysql/gtid_slave_pos
table: mysql/help_category
table: mysql/help_keyword
table: mysql/help_relation
table: mysql/help_topic
table: mysql/host
table: mysql/index_stats
table: mysql/innodb_index_stats
table: mysql/innodb_table_stats
table: mysql/plugin
table: mysql/proc
table: mysql/procs_priv
table: mysql/proxies_priv
table: mysql/roles_mapping
table: mysql/servers
table: mysql/table_stats
table: mysql/tables_priv
table: mysql/time_zone
table: mysql/time_zone_leap_second
table: mysql/time_zone_name
table: mysql/time_zone_transition
table: mysql/time_zone_transition_type
table: mysql/user
table: mysql/general_log
table: mysql/slow_log
table: mysql/transaction_registry
database: wordpress
table: wordpress/wp_commentmeta
table: wordpress/wp_comments
table: wordpress/wp_links
table: wordpress/wp_options
table: wordpress/wp_postmeta
table: wordpress/wp_posts
table: wordpress/wp_term_relationships
table: wordpress/wp_term_taxonomy
table: wordpress/wp_termmeta
table: wordpress/wp_terms
table: wordpress/wp_usermeta
table: wordpress/wp_users

UNCOMPRESSED BACKUP SIZE: 60.80 MB in 2229 files
[...]

It doesn't even hesitate where yours seems to get stuck... My VM only has 1GB RAM and 2 vCPU cores.

Looking back over everything, I note that the WordPress docs don't note explicit use of "DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci" - so I'm not sure if that has any impact?

So I dumped the existing DB and recreated it:

mysql --execute "DROP DATABASE wordpress;"
mysql --execute "CREATE DATABASE wordpress;"
mysql --execute "GRANT ALL ON wordpress.* TO 'wordpress'@'localhost' IDENTIFIED BY 'my_password';"

I then reinstalled WordPress (the web browser bit) and again forced a full TKLBAM backup:

tklbam-backup --full-backup now

Same deal, everything "just works"...?!? I'm not really sure where we can go from here?