Ugh. MySQL stopped, have no idea what to do

Hey guys,

Hoping someone here has some ideas, as I've searched and tried several things and have come up short.

Someone told me a website I run was down and went to check it out. "Error establishing database connection".

Logged into webmin to discover this little gem of a message:

"MySQL is not running on your system - database list could not be retrieved.

Click this button to start the MySQL database server on your system with the command /etc/init.d/mysql start >/dev/null 2>&1 &. This Webmin module cannot administer the database until it is started."

Next to it was a button that says "Start MySQL Server" which does nothing but re-load the page when clicked. I also tried starting it through commands on webshell, only to get an error message that says "Job failed to start."

Also, when I type "mysql" in the shell, I get this:

"ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)"

Finally, I checked my error logs out. A few new clues that lead nowhere:

121105 22:05:18 [Note] Plugin 'FEDERATED' is disabled.

121105 22:05:18  InnoDB: Initializing buffer pool, size = 8.0M

121105 22:05:18  InnoDB: Completed initialization of buffer pool

121105 22:05:18  InnoDB: Started; log sequence number 0 44253

121105 22:05:18  InnoDB: Starting shutdown...

121105 22:05:24  InnoDB: Shutdown completed; log sequence number 0 44253

121105 22:05:24 [Note] Plugin 'FEDERATED' is disabled.

121105 22:05:24  InnoDB: Initializing buffer pool, size = 8.0M

121105 22:05:24  InnoDB: Completed initialization of buffer pool

121105 22:05:24  InnoDB: Started; log sequence number 0 44253

ERROR: 1050  Table 'plugin' already exists

121105 22:05:24 [ERROR] Aborting


121105 22:05:24  InnoDB: Starting shutdown...

121105 22:05:34  InnoDB: Shutdown completed; log sequence number 0 44253

121105 22:05:34 [Note] /usr/sbin/mysqld: Shutdown complete


121105 22:05:35 [Note] Plugin 'FEDERATED' is disabled.

121105 22:05:36  InnoDB: Initializing buffer pool, size = 8.0M

121105 22:05:36  InnoDB: Completed initialization of buffer pool

121105 22:05:39  InnoDB: Started; log sequence number 0 44253

121105 22:05:48 [Note] Event Scheduler: Loaded 0 events

121105 22:05:48 [Note] /usr/sbin/mysqld: ready for connections.

Version: '5.1.66-0ubuntu0.10.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)

121127  0:07:10 [ERROR] /usr/sbin/mysqld: Disk is full writing './wordpress/wp_comments.MYD' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)

121127  0:07:10 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  0:17:10 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  0:27:10 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  0:37:10 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  0:47:10 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  0:57:10 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  1:07:10 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  1:17:10 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  1:27:10 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  1:37:10 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  1:47:11 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  1:57:11 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  2:07:11 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  2:17:11 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  2:27:11 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  2:37:11 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  2:47:11 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  2:57:11 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  3:07:11 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  3:17:11 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  3:27:12 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  3:37:12 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  3:47:12 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  3:57:12 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  4:07:12 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  4:17:12 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

121127  4:27:12 [ERROR] /usr/sbin/mysqld: Retry in 60

I wiped out the /wordpress/wp_comments.MYD file clear, nothing happened.

If all else fails, I have an old backup...but I really don't want to use that since this is an e-commerce store and we would lose the last four months worth of sales data, plus blog posts, web copy, etc.

I'm not too familiar with this stuff, so if you have an explanation or steps towards a fix to offer up, please do so in dumbed-down words...I'm not as smart as you Turnkey guys! :)

Thanks,

Tom

Replies (9)

Disk full?

Reply 1 2012-12-11

It looks like your disk may be full, preventing mysqld from starting. Might want to check that...

121127  0:07:10 [ERROR] /usr/sbin/mysqld: Disk is full writing './wordpress/wp_comments.MYD' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)

You can run 'df -k' to see the disk utiliziation for your filesystems.

Thanks, but...

Reply 2 2012-12-11

Thanks Eric, but I tried typing "df-k" into my webshell and it returned "command not found".

Not sure what I would even do if it turns out my disk is in fact full.

It looks like you tried

Reply 3 2012-12-11

It looks like you tried "df-k" (without a space) , but there's a space between the f and the dash... so "df<space>-k" should work.

If it turns out that you're out of space, there may be some options... Is this a turnkey appliance? if so, which one?

Wordpress

Reply 4 2012-12-11

Ah, I didn't mention that I'm running Wordpress.

Here's a screenshot of my usage. I see a "100%" in there...

Yup, it's full

Reply 5 2012-12-11

That's the cause of your problem, or at least the first thing you need to address.

If it were me, I would:

It looks like your backup job just completed but if you could not start mysql, I am not sure if the backup is complete.
Clean up anything you don't need. look for big files in /var/log or /tmp that are suspicious or old and remove them.
Once you get enough space to start mysql, run a backup again.
Depending on how much space you can clean up, you may want to increase the filesystem size. If you created this applaince with a logical volume, you can add a disk and extend the volume giving you more space. If you did not use a logical volume it's trickier but you can either extend the partition and filesystem (if you have physical disk space unused) or deploy a new wordpress instance and use tklbam to restore your data.

OK, I'll try and delete some of these

Reply 6 2012-12-12

Are there any files I absolutely DON'T want to delete, i.e. anything that might currupt the whole system if it's missing? I'm worndering if I can just wipe out my entire /var/log folder and see if that frees up enough space?

I would hate to give you a

Reply 7 2012-12-12

I would hate to give you a list, or just say 'delete everything'.

Generally speaking, if the file in /var/log is over a week old... I'd be tempted to delete it.

That said, I'd clean up /tmp first... applying the same rule... delete old & big files first.

Sounds like it might be too late?

Reply 8 2012-12-12

But /var/cache would have been worth a look IMO too. Also you could have tried a SQL dump of your existing DB prior to overwriting...

As for finding big files the ncdu program is very useful (you'll need to install it first - 'apt-get update && apt-get install ncdu') for findling where all your free space is being chewed up...

One other thing is that depending on how it was installed/where your server is running, it may be (have been?) possible to extend your available space with either mount binding another drive/partition to your FS and/or extending the volume with LVM.

A final thought is that a fresh instance (if you are running a VM or via the Hub) may be the best option (especially if you are restoring a backup) to give you a clean start.

Thanks, dude. Nothing was in

Reply 9 2012-12-12

Thanks, dude. Nothing was in the /tmp folder, and I deleted everything in the /var/log folder to no avail. Went ahead and re-launched a new server with my more recent backup and will just have to bite the bullet on the lost data I guess.

Not sure what caused this, but HOPEFULLY it won't happen again! Thanks again for your help!