Debian security update breaks v15.x LAMP based servers!

UPDATE: v15.1 update & bugfix release is now available. New versions of all affected appliances can now be downloaded.

As reported earlier, an automatically installed Debian MariaDB (drop-in replacement for MySQL) security update broke all appliances that rely on MariaDB database engine. It will likely hit all relevant users as their servers install auto security updates. That means that all appliances which use MariaDB ("MySQL"), including (but not limited to) LAMP based appliances, such as WordPress, Drupal, Joomla, etc. So about 70% of the v15.x library will break and not have a functioning MySQL-compatible DB engine running!

A full list of all affected appliances has been noted at the top of the relevant issue on our GitHub issue tracker. If your server is v15.x, is having database connection issues and is on that list, the fix below almost certainly should fix it!

Fix: Reinstall 'default-mysql-server'

To resolve the issue simply reinstall the 'default-mysql-server' package. This meta-package depends on the latest version of 'mariadb-server-10.1' and will reinstall MariaDB. Assuming that you have not made other changes in the meantime, your existing config should remain untouched and everything should return to normal. I.e.:

apt update
apt install default-mysql-server

If you have taken other steps in the meantime and this does not fix your server, please feel free to open a new thread in the forums and I'll be along to assist ASAP (usually within a few days, often faster). Please note that starting new threads requires you to signed in as a registered user (website registration is free). Paid up TurnKey Hub subscribers can also access our assistance via the Hub's "in app" Support portal (expect a response within one work day, usually much faster).

What happened?

We're still not completely clear on the exact cause, but we're working to understand it. At this point, it seems likely that a new dependency requirement was added which was not supplied as part of the security update. Because the security updates will only install packages from the Debian (and/or TurnKey) repository additional dependencies will not be installed. In this case, that forced the removal of important MariaDB packages. This stopped the database running and thus broke any applications that depend on the database backend.

What happens next?

We have lodged a bug report upstream with Debian. We are aiming to work with them to not only understand the cause of this issue better, but also see how we can reduce the chances of this sort of issue re-occurring in the future.

It seems highly likely that we'll need to rebuild all our appliances that include MariaDB (~70% of the library) to ensure that the images are working as our users expect. We intend to get that done ASAP.

In the meantime, please apply the fix as required and we'll keep you up to date on any further developments. Where relevant I'll edit this post, but I will also aim to note what I've changed as a comment.

Ensure you get notified!

If you didn't get an email notification regarding this issue (and a link to this blog post), then please ensure that you are registered as a TurnKey website user. Our "News and Security" email notifications are very low traffic (maximum of one "news" post per month, as many "security" posts as required, although rarely more than one or 2 in a single month).

Comments

Very Siberian's picture

You guys are setting the bar very high with your rapid response and fix! Thanks so much. Of course, this event was very unfortunate but you jumped on it immediately. :)

Jeremy Davis's picture

Thanks for your kind words of encouragement. We certainly do our best! :)

Dan Kelley's picture

Thank you So much for your rapid resolution to this. Really saved my bacon today! After digging online I found the post in the forums.... then when I finally got it going I finally checked my email.... and wouldn't you know it.... the solution was in there from earlier this morning. I need emails from you guys to send me text alert or something!  It would have saved a few gray hairs! Thanks again!  
Jeremy Davis's picture

Hmm, SMS text alerts. I really like that idea. Perhaps we can look into that a little further?! FWIW, I've added it to our issue tracker so it doesn't get forgotten. Although I can't promise when it might get implemented. As per usual, we have a lot of competing priorities.

Dan Kelley's picture

I actually made a rule in Outlook to forward all emails like that to my cell phone. AT&T like many cellular providers will let you send TXT to your phone via your phone number and a special email address.  So mine with AT&T is just MYNUMBER@txt.att.net  It’s working since I got the text when you replied to my comment! :)   
Jeremy Davis's picture

The additional info includes a link to the corresponding GH issue which now also includes a full list of affected appliances.

I also added a link to the bug on the Debian bug tracker, plus clarified some info on the cause of the problem (seems likely that a new dependency was added, which wasn't included in the security repo)..

Mariusz's picture

Jeremy - thank You very much! It was first founded link with solution and Your solution works perfectly. If You will be in Poland - feel invited for big vodka :)
Jeremy Davis's picture

Thanks for your kind offer. Although I'm not sure when I might make it to Poland (I live in Australia). Still, I appreciate the offer and you never know...! :)

Good luck with it all.

Pages

Add new comment