Max's picture

Hey everybody,

I use the Turnkey Wordpress appliance on a Server running Proxmox and everything has been running perfectly, but since yesterday, it stopped working and I am not able to fix it.

I did not install any updates or new plugins or didnt post content. Wordpress gives the "Error establishing a database connection" message, so I thought it might be a problem with mysql. When I tried to log in into the VW, I noticed it is running really slow, it is using over 900mb of the assigned 1000mb RAM. When I quit the mysql service with "service mysql stop", RAM usage drops to 200mb again. If I start it again, the system gets really slow again and high RAM usage is back.

All packages are up to date with apt update and apt upgrade.

I checked the error.log in /var/log/mysql, but I am not very familiar with mysql. I attached the error log as a text file.

I already thought about it being an error in the database, so I restored to a snapshot in Proxmox, where I know everything was working, but its still the same problem.

Any help would be higly appreciated, as I really don't know how to fix it anymore.

 

Thank you so much!

Forum: 
Jeremy Davis's picture

Hi Max, I can't see your log so it seems attaching it didn't work?!

Anyway, there are a few likely culprits which are worth excluding before we dig in too deep. All my instructions will assume that you have logged in as root at the commandline. Personally I prefer (and recommend) just using an SSH session, but the specifics of how you get a shell are more-or-less irrelevant. Seeing as your running on Proxmox, I'll assume that you'll be using root, but just in case someone else reads this (who has root disabled), you can either enable the root account and log in there, or get a root shell like this:

sudo su -

The first thing I'd try is checking it's statust:

systemctl status mysql

Look for where it says "Active:" and if it's running it should be "active (running)". If it's "inactive (dead)" then it's not running and IIRC for MySQL "active (exited)" is also considered a failed state. If you're not sure, please feel free to post the output of that command and I can assist interpret.

If it's not running, then try starting it:

systemctl start mysql

Or if it is running, try restarting it:

systemctl restart mysql

If WordPress is still failing to connect, then recheck the status and the most recent log entries which are also displayed. Hopefully this should give some hint on what has gone wrong. Again please share it if you want a hand interpreting.


If it refuses to start and you're not quite sure what the logs messages mean, the first thing to check is see if you've run out of disk space. This is a very common cause of MySQL (technically MariaDB in v15.x - but for ease, I'll just call it MySQL from here on) crashing and refusing to start. You can check that like this:

df -h

And to assist you to decode that, this is the output from a TKLDev (KVM) server I have running locally:

Filesystem                Size  Used Avail Use% Mounted on
udev                      991M     0  991M   0% /dev
tmpfs                     201M  4.9M  196M   3% /run
/dev/mapper/turnkey-root   16G  3.9G   12G  26% /
tmpfs                    1003M     0 1003M   0% /dev/shm
tmpfs                     5.0M     0  5.0M   0% /run/lock
tmpfs                    1003M     0 1003M   0% /sys/fs/cgroup
overlay                    16G  3.9G   12G  26% /turnkey/fab/products/core/build/bootstrap

And in an effort to give you something closer to what you're likely to see, here's the output from a Proxmox LXC container that I have running locally:

Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/pve-vm--101--disk--0  7.9G  2.0G  5.5G  27% /
none                              492K     0  492K   0% /dev
tmpfs                              16G     0   16G   0% /dev/shm
tmpfs                              16G  8.2M   16G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
tmpfs                              16G     0   16G   0% /sys/fs/cgroup

The line that notes the root (i.e. '/') mount is the one you'll want to check (unless you've done something fancy with your mount points). So in my first example (KVM VM) I have /dev/mapper/turnkey-root mounted as '/' and it's 16GB total, 3.9GB used (26%) with 12GB free.

In my second (LXC) I have /dev/mapper/pve-vm--101--disk--0 mounted as '/' and it's 7.9GB total, 2.0GB (27%) used and 5.5GB free. Both my examples have plenty enough room. if you have less than 1GB, it's probably time to evaluate your options and either have a big cleanup, or extend your root partition. Although unless you have 1% or less free, that's unlikely to be the cause of MySQL/MariaDB issues.

It's also worth noting that on Proxmox, because by default it uses LVM-thin as a storage volume, it's possible to over allocate disk space. TBH, I'm not 100% sure, but I suspect that if the Proxmox storage your container volume resides on is full, within your container it would still report that there is free space (unless you've used it's allocation). So it'd also be worth double checking the host storage mount where your container's volume is stored.


The next thing to check is RAM. If MySQL starts again ok, but then continues to crash (seemingly) randomly, then running out of RAM might be the cause? Unfortunately this is a bit harder to test as with MySQL crashed, you likely have tons of free RAM. And if the issue is caused by a memory leak somewhere (e.g. a poorly written WordPress module), even with it running it may not be apparent. Regardless, here's how to check for free RAM:

free -m

On my VM:

              total        used        free      shared  buff/cache   available
Mem:           2004          63         822           2        1118        1754
Swap:          2043           2        2041

On my container:

              total        used        free      shared  buff/cache   available
Mem:           1024         339         277           8         406         684
Swap:          2048           0        2048

Because Linux uses free RAM to buffer disk writes, you need to consider the "available" value (not the "free" value). If the system needs the cached space back, it will flush it straight to disk. So I have 1754MB free in my VM and 684MB free in my container. Considering that's well over 50% of the RAM that each has (and they're both running fine) it's clearly not an issue on my machines. If MySQL is running and isn't crashing regularly, and you have a similar amount of headroom, then RAM is likely not the issue. Although as I say, if it is crashing every few days, then a memory leak may be the issue.

Again it's worth noting that you can also over allocate RAM on Proxmox. So if you have a ton of other servers running and the host system is reporting high RAM usage, then perhaps RAM is an issue?


Having said all that, re-reading your post, it seems that MySQL is not starting at all and as such RAM seems an unlikely culprit. Whilst lack of free space may still be a possibility, as you noted that you tried restoring an old backup and that didn't help. That makes me think that free disk space is less likely (or at least on the guest, host running out of space may still be a possibility) as IIRC Proxmox usually warns about lack of free space.

The only other likely possibility that occurs to me is data corruption. But again, that seems unlikely as you've restored to a "known good point" and it's still broken. So TBH, I reckone we'll need some more info.

FWIW you can get the last few lines of the log via "tail /path/to/logfile". If you'd like to post the output of that here (perhaps also with the output of the status command I gave earlier) and we'll see if there is anything that sticks out...

Max's picture

Hey Jeremy,

 

thank you so much for your detailed answer and trying to help me out!

I always use the root account via SSH, and already checked the storage before I asked in the forum, and the output was the following:

root@xxx ~# df -h

Filesystem                Size  Used Avail Use% Mounted on

udev                      487M     0  487M   0% /dev

tmpfs                     100M   12M   89M  12% /run

/dev/mapper/turnkey-root   17G  2.2G   14G  14% /

tmpfs                     499M     0  499M   0% /dev/shm

tmpfs                     5.0M     0  5.0M   0% /run/lock

tmpfs                     499M     0  499M   0% /sys/fs/cgroup

 

Anyway, I just wanted to let you, and everyone else who might run into this error tell you how I fixed the problem in the meantime: 

I found an article explaining how to repair the mysql database automatically on https://www.thegeekstuff.com/2011/12/mysqlcheck/

I ran the commands shown there and now everything works perfectly again, however I am still surprised, why the error existed in the snapshot as well, which ran perfectly at the time I took the snapshot. Maybe the database was already corrupted at the time I took the snapshot. 

Many thanks again!

Jeremy Davis's picture

Ah ok. So it was a corrupted DB after all. I'm glad to hear that you managed to resolve it.

Add new comment