Site Down - TKLBAM restore not functioning?

Hi Reis, Sounds like a major pain...! :(

I would recommend that you check the error logs (/var/log) - particularly MySQL logs (IIRC mysqld.log & mysqld.error) and daemon.log (i.e. services starting/stopping). The main log (messages) probably also contains info that may be relevant...

Just manually starting MySQL (i.e. 'service mysql start') may even give you some clue.

TBH my guess (without anywhere near enough info...) is a corrupt DB although if your DB is using innodb then even just corrupt log files can stop it from starting...

If you still can't work out what's going on I suggest that you post (any MySQL relevant) logs; along with some more info about anything else you've tried or tested.

Hi Reis,

I'm glad you sorted out your reboot issue. During my 35 year career, I found that log files filling up disk space is the number one cause of servers failing to operate correctly. The second likely cause of problems is dumb-as-a-stump watchdog software that does things it shouldn't. You have experienced both in one go unfortunately.

Regarding reboots...you will experience fewer reboot situations with a small appliance but they will still happen unless you adjust the apache default settings so that Apache can't exhaust your RAM. See the following blog article I contributed sample settings to: /forum/support/20150117/replacement-turnkey .

Regarding swapping IPs it depends how you have set things up. I am not familiar with the TKLX Server Hub so not sure how to do it via the hub but if you log into your AWS account and assuming you are using an Elastic IP address, it's one click to dis-associate the public elastic IP address from the old micro instance and one additional click to associate the Elastic IP address with the new small instance. See the following link for a screenshot of the elastic IP associate / dis-associate feature http://media.amazonwebservices.com/articles/nat_monitor_files/ec2_associated_eips.jpg

Jeremy would be able to tell you how to do it via the Hub.

With a Micro instance you will have a small amount of RAM which is augmented with swap space on your disk. There is a process monitor on linux systems called the OOM Killer. When you run low on memory it kills the biggest RAM user which is usually the database. The RAM hog though is usually apache. If you get a lot of robots accessing your site Apache default settings allow too much memory to be held by Apache...and the OOM killer kills your database. When you reset the appliance the apache RAM allocation is reset to a minimal amount so your server runs again until the robots hit your site again.

If you have logging enabled your logs can get really big. I have noticed a lot of heavy apache traffic logging over the past few months and make a point of going in and deleting old achive logfiles which appear as zipfiles on your disk. If you are low on disk you won't be able to create a sufficient swap space for your MySQL to run.

That's my best bet for a likely culprit based on the limited information you have provided.
 

You obviously have a much deeper understanding of all this than me! :)

This was a freshly launched instance so I am guessing that the Hub needs a bit of an update in that regard... I'll let Alon know...

Thanks again! :)

WIth a non-default VPC you can associate the Elastic IP to the instance or you can associate it with the VPC elastic network interface (ENI) and then attach / detach instances from the ENI. It is possible that you are experiencing a collision of old and new worlds.

See http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/vpc-migrate.html

and

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html#VPC_EIPConcepts

and

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html

There are a number of possible ways that the TKLX hub has been designed to configure things but you should be able to reverse engineer it by looking via your AWS console. Do you have an ENI?

NOTE: You will see different behaviour if this is a new instance or an old instance due to the switch from EC2-Classic to EC2 VPC. A new instance only has EC2-VPCs and and old instance has both.

Fun and games :-)

Yeah VPC! :)

Funny thing was though: The elastic IP (that I created via the Hub) couldn't be associated with the Micro instance (that was also launched via the Hub) - hence the error. However in AWS Console I could associate the same IP to the same instance... The Hub also showed that it was successful...

Just checking...was that a typo on the error message...did you mean VPN or VPC. I could see the latter more than the former. Managing ip addresses in a non-default VPC is different then managing it in a default VPC (EC2 Classic replacement a the default). It is possible that the hub is configured as a TKLX VPC or set of VPCs specific to each client. That should be apparant via your AWS console.

Just curious :-)

I've been crazy busy...

How is your current DNS setup? Are you using HubDNS and/or an elastic IP?

If you have an elastic IP attached to your current server then you should be able to simply detach it and reattach it to your new small server. However I've just been testing it in the Hub and for some reason it was giving me errors (about my server being in a VPN - correction: VPC!!). But I just used the AWS Management Console to do it. And I'll raise the issue with Alon.