TurnKey Linux Virtual Appliance Library

Reducing the load on a small VPS by 80% in 5 minutes

By Liraz Siri - 16 comments | Latest by baruz

A few days ago I noticed the average load on a newly setup VPS was too high relative to its workload. The server was noticeably sluggish. Load was high and a page the wiki it was running would occasionally take 10 seconds to load.

I didn't have too much time to kill but I decided I would at least try picking off the lowest hanging fruit before upgrading to a beefier, more expensive plan which would cost a few hundred dollars extra a year.

When I noticed swap usage was relatively high, I suspected the server was constantly moving memory in and out of swap.

One of the first things you should try when you're tuning the performance of a server is to reduce memory usage so that it doesn't use as much swap and can use more of your memory for disk buffering. Accessing buffered blocks from memory is a gazillion times faster than accessing them from disk.

So how do you reduce memory usage without reducing functionality? By tuning the trade off between memory and CPU time.

One example of this trade off in practice is pre-forking, which is a common technique used by many systems that can improve user-perceived performance. The principle is that by starting the process in advance of when it's needed then that bit of overhead is not experienced by the user who would otherwise need to wait for the process to start up before it can serve him.

The catch is that keeping spare processes around requires memory.

When you have a limited amount of real memory, then any performance gains from pre-forking can quickly evaporate due to reduced IO performance.

On this server, we didn't really have that much activity for things like mail or web, but many subsystems were configured (by default) to pre-fork at least 5 worker processes in advance.

This is a reasonable configuration for a dedicated server with lots of memory and hundreds of users, but not ideal for that cheap VPS you just setup for your workgroup.

I reduced memory consumption by over 200MB by simply decreasing the number of preforked worker processes for apache (20-25MBs per process), spamassassin (30MBs per process), and saslauthd. Average load dropped immediately by about 80%.

Total time spent picking this low hanging fruit: 5 minutes.

Enjoyed this post? Get future posts delivered by email or get the RSS feed.
Find us on Facebook and Twitter.
Last update: Thu, December 15 - 19:00

Comments
Guest's picture

I substantially reduced the

MikeC - June 08, Tue 14:35.

I substantially reduced the memory usage on my own VPS a few weeks back by moving from the Apache Prefork mpm to the Worker mpm, and by disabling FastCGI so I didn't have idle php processes sat around using up memory.

liraz's picture

Some libraries not thread-safe? Also, try lighttpd, nginx...

Liraz Siri - June 08, Tue 15:06.
If I recall that's the main reason prefork is so widely used. Worker is known to be more memory efficient but not everything is thread-safe. In particular I remember this created a problem with PHP, so you had to run it outside of Apache in its own little pre-forking application server (e.g., with FastCGI).

Personally I prefer lighttpd over Apache. There's a bit of a learning curve involved but it's not only super-fast and efficient, it also has a very powerful configuration system which is in my opinion superior to Apache. These days the only reason I would use Apache is backwards compatibility on a server where memory resources are not an issue.

I've also played with nginx, and it's nice too. Slightly faster than lighttpd in many cases. But on the other hand, lighttpd is far more flexible and configurable.

Guest's picture

I'd expect the php issue to

MikeC - June 08, Tue 16:25.

I'd expect the php issue to only exist if you're using mod_php, but I'm running php as a CGI for security reasons. I did have to remove some mod_perl handlers when I moved away from prefork though.

I'd like to move to lighttpd to save resources, but I'm running quite a few sites, and some of them use very complicated rewrite rules so for the moment, I'm just going to stick with what I know.

liraz's picture

You can rewrite any Apache rewrite rule into Lua

Liraz Siri - June 12, Sat 00:27.
We had similar issues with porting rewrite rules to lighttpd from Apache and found the solution in lighttpd's mod_magnet, which allows you to script these things in Lua. Sure there's a learning curve involved but you can pretty much get anything done. I like solving these sorts of challenges so for me it was fun.

Also, lighttpd rewrite rules are getting much more powerful so in the future I bet you'll only need dive into Lua for the really complicated stuff.

Guest's picture

Due to copy-on-write and the

Miguel - June 08, Tue 16:52.

Due to copy-on-write and the way virtual memory works, that does come as a surprise to me. Pre-forking shouldn't significantly increase the consumption of RAM and thrash your server that much, at least not for as little as 5 idle-just-forked processes.

I'd say it was good that it seems to have fixed your problem, but I don't believe pre-forking to be the real issue here.

liraz's picture

In theory, theory and practice are the same, but in practice...

Liraz Siri - June 12, Sat 00:22.
So with copy-on-write semantics pre-forking shouldn't necessarily be an issue but I think that depends a lot on where you fork and what the child process does afterwards. Also, I'm guessing with dynamic languages like Perl in which spamassassin is implemented the performance hit from running multiple instances of the Perl runtime could be significantly greater.
Guest's picture

Pre-Forking Sped Us Up

Jason B. - February 18, Fri 19:46.

I used your tip about tuning the trade off between memory and CPU time and it helped speed up our server.  Pre-forking wored wonders.  We have lots of workers that all run automotive crm software and we all noticed the difference after I altered the server.

liraz's picture

Glad to hear it Jason!

Liraz Siri - February 18, Fri 20:06.
Thanks for the positive feedback. It's delightful to hear that people are still benefiting from the posts on our little tips and tricks.
Guest's picture

Trade-off did not work for me

Muhammad Sohail - May 25, Wed 04:02.

Liked your article dude. I tried the trade off technique using prefork, reduced Apache conf to bare minimum and removed all unwanted modules but did not get the expected results out of it. Had to move to worker model. Although both prefork and worker caused the load average to shoot up to 20, the worker model was able to withstand 250 concurrent users against 15 concurrent users from prefork.

It would be extremely helpful if you can share tips on how to reduce the load average which is caused by Apache coz at this stage I would like to stick with Apache instead of switching over to Lighttpd or nginx although both are better than Apache any time.

Guest's picture

managed vps

managed vps - September 22, Thu 20:00.

Nice Post!!!!!!!!

Thaks for sharing.....

Guest's picture

Virtual private servers

Virtual private servers - September 26, Mon 14:12.

Exceptionally great post!!!!!!

Looking forward for more post like these!!!!!!!!!!

Thanks for sharing!!!!!!

ajblane's picture

Could someone explain how to do this?

Andrew - October 05, Wed 12:04.

I'm using Amazon EC2 so I'd like to give this a try. Could you give me a clue where to start please?

ajblane's picture

Here's a clue

Andrew - October 14, Fri 06:07.

For anyone wondering where to look to tweak Apache Prefork conditions, you'll find the settings in /etc/apache2/apache2.conf as follows:

<IfModule mpm_prefork_module>

    StartServers          3
    MinSpareServers       3
    MaxSpareServers      3
    MaxClients          50
    MaxRequestsPerChild   1000
</IfModule>

These are my settings for now (using Amazon EC2 Micro Instance). Once you've made your changes, you'll need to restart Apache.

Guest's picture

Load reducing for 24 GB Ram installed server

baruz - December 15, Thu 16:05.

Hi Liraz Siri, could you please write an easy Apache tuning and optimization guide for 24GB ram installed server?

my entertainment site runing on 6 x AMD Opteron(tm) Processor 4174 HE, Speed 2300.000 MHz, Cache 512 KB

and 24GB Ram installed but my server getting heavy load, hope you write an easy server optimization guide. 

Thanks advance

liraz's picture

Wow, that's one beefy

Liraz Siri - December 15, Thu 16:27.

Wow, that's one beefy machine. Use lighttpd or nginx. You may be surprised how well that works, once you set it up. :)

Guest's picture

Thanks

baruz - December 15, Thu 19:00.

Thank you so much, i'm gonna try and update here how is working.

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)