TurnKey Linux Virtual Appliance Library

Has anyone else experienced "soft lockups" when running under Xen?

 I deployed a Turnkey LAMP appliance to VPS.net's cloud (which uses Xen virtualisation). From time to time the server (which is fully up to date) experiences what is reported on the console as "soft lockups", like below. I find this baffling. VPS.net support say this is a kernel bug with Ubuntu 32bit. Is this correct? Is there anything I can do to solve the problem?

The machine becomes unresponsive and an endless stream of these messages are spewed on the console:

[52846.563059] BUG: soft lockup - CPU#0 stuck for 11s! [apache2:4744]
[52873.811124] BUG: soft lockup - CPU#1 stuck for 11s! [kswapd0:106]
[52858.304554] BUG: soft lockup - CPU#0 stuck for 11s! [apache2:4744]
[52885.550344] BUG: soft lockup - CPU#1 stuck for 11s! [kswapd0:106]

Here is what VPS.net support said to me:

"This error seems to be from a buggy kernel version that you have. This is a 32 bit Ubuntu - Xen issue rather than a problem with vps.net itself, so you may try updating your kernel and check if it helps. Please let us know if we could be of any assistance."

Liraz Siri's picture

GigaTux and GPLHost also support TurnKey appliances

We're aware of the problem and we're very frustrated about it. It's been reported before. Unfortunately, we haven't really managed to make progress with VPS.NET on this. It seems to be a bug that's triggered by an interaction between that version of the Ubuntu kernel and VPS.NET's Xen implementation. When Ubuntu 10.04 LTS comes out in a few months we'll release new versions for all our appliances which will include a new kernel and will hopefully solve this issue.

In the meantime other VPS provides such as GigaTux and GPLHost also support TurnKey, and none of their users have reported this issue so you may want to check them out...

This is shocking

Dear Liraz. Thanks for the feedback, it is good to know it was not a configuration problem.

I actually find the state of affairs a bit shocking. VPS.net is listed as a partner on your page. And they have turnkey appliance "templates" on their system. If there is an incompatibility they should not support Turnkey -- or fix the issue. Anyway, I realise I am not a paid subscriber to Turnkey, but at least from VPS.net I would expect better as a paying customer.

In the end I went with the bog-standard Ubuntu 8.04 x64 installation, but I had to do a lot of work to get the configuration right -- stuff that Turnkey supplies out of the box. So this has been an unpleasant experience for me, but I realise you are probably doing your best to get it sorted.

I hope to try Turnkey again after the Lucid release. Good luck with the work.

Liraz Siri's picture

You're right of course

We weren't aware of the issue when we partnered up with VPS.NET and ever since it was discovered we were hoping the issue would be identified and resolved quickly but that didn't happen.

Unfortunately, we're out of the loop here. We don't have access to VPS.NET's infrastructure so we can't work on this directly. Since other Xen based VPS providers have not reported this issue perhaps more could have been done to at least find a workaround but that's just speculation on our part. Their engineering team doesn't report to us so we don't know what they've tried. Heck, we don't even know if the technical suggestions we've made have been attempted, such as forcing a different kernel to be used (e.g., Redhat kernels or Linode kernel) or using a 64-bit kernel like GigaTux does.

FWIW, if it was our decision we would remove the images entirely until a fix or workaround could be found. VPS.NET aren't to blame for a bad interaction between an Ubuntu kernel and Xen, but leaving a pitfall like that around is not (in my opinion) the right thing to do. That's up to them though.

For now, the best we can do is add a warning about this issue to the original partnership announcement.

I'm very sorry for the inconvenience. Do you think there is anything more we can do?

A warning is a good idea

 A warning would be good. I do hope VPS.net addresses this issue, since your appliances are a really compelling tool. Good luck!

64 bit kernel works as well

We use the CentOS 5.4 64 bit to host TKL and it works as well.

http://UnmeteredVPS.net/tkl
Try our TurnKey Linux appliance hosting FREE!

Also try my company's hosting

We are also supporting TKL appliances now.  We am offering a free week for people to try it out and see how well it works.  We are still setting up our web site, but you can contact me if you would like to try our service.

http://UnmeteredVPS.net/tkl
Try our TurnKey Linux appliance hosting FREE!

Has this been resolved?

I am interested in using Turnkey Linux with VPS.NET ... Seems like this issue has been going on for quite a while now.  Has it been resolved?

Thanks,

Richard

VPS.NET problem still exists

The VPS.NET problem still exists.

If you would like to try an appliance on my company's servers for free, please check out http://UnmeteredVPS.net/tkl
 

http://UnmeteredVPS.net/tkl
Try our TurnKey Linux appliance hosting FREE!

Liraz Siri's picture

Only happens when ram usage is nearly full?

I just exchanged emails with Carlos Rego from VPS.NET who says:

"It also only happens when the user's ram usage is near fully used, on many tests we ourselves did, we where unable to reproduce the error on most instances."

To solve this I urged VPS.NET to try and use a CentOS kernel as GigaTux have reported success with that.

VPS.NET possible fix

http://www.vps.net/forum/public-forums/technical-discussions/1677-turnke...

This should fix the compability issues.

:)
Ditlev

I've tried it but it didn't

I've tried it but it didn't work. I'm trying a couple of more thigns though. I'll post them if they work.

 

Thanks.

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)