TurnKey Linux Virtual Appliance Library

Reducing the load on a small VPS by 80% in 5 minutes

A few days ago I noticed the average load on a newly setup VPS was too high relative to its workload. The server was noticeably sluggish. Load was high and a page the wiki it was running would occasionally take 10 seconds to load.

I didn't have too much time to kill but I decided I would at least try picking off the lowest hanging fruit before upgrading to a beefier, more expensive plan which would cost a few hundred dollars extra a year.

When I noticed swap usage was relatively high, I suspected the server was constantly moving memory in and out of swap.

One of the first things you should try when you're tuning the performance of a server is to reduce memory usage so that it doesn't use as much swap and can use more of your memory for disk buffering. Accessing buffered blocks from memory is a gazillion times faster than accessing them from disk.

So how do you reduce memory usage without reducing functionality? By tuning the trade off between memory and CPU time.

One example of this trade off in practice is pre-forking, which is a common technique used by many systems that can improve user-perceived performance. The principle is that by starting the process in advance of when it's needed then that bit of overhead is not experienced by the user who would otherwise need to wait for the process to start up before it can serve him.

The catch is that keeping spare processes around requires memory.

When you have a limited amount of real memory, then any performance gains from pre-forking can quickly evaporate due to reduced IO performance.

On this server, we didn't really have that much activity for things like mail or web, but many subsystems were configured (by default) to pre-fork at least 5 worker processes in advance.

This is a reasonable configuration for a dedicated server with lots of memory and hundreds of users, but not ideal for that cheap VPS you just setup for your workgroup.

I reduced memory consumption by over 200MB by simply decreasing the number of preforked worker processes for apache (20-25MBs per process), spamassassin (30MBs per process), and saslauthd. Average load dropped immediately by about 80%.

Total time spent picking this low hanging fruit: 5 minutes.

You can get future posts delivered by email or good old-fashioned RSS.
TurnKey also has a presence on Google+, Twitter and Facebook.

Comments

I substantially reduced the

I substantially reduced the memory usage on my own VPS a few weeks back by moving from the Apache Prefork mpm to the Worker mpm, and by disabling FastCGI so I didn't have idle php processes sat around using up memory.

Liraz Siri's picture

Some libraries not thread-safe? Also, try lighttpd, nginx...

If I recall that's the main reason prefork is so widely used. Worker is known to be more memory efficient but not everything is thread-safe. In particular I remember this created a problem with PHP, so you had to run it outside of Apache in its own little pre-forking application server (e.g., with FastCGI).

Personally I prefer lighttpd over Apache. There's a bit of a learning curve involved but it's not only super-fast and efficient, it also has a very powerful configuration system which is in my opinion superior to Apache. These days the only reason I would use Apache is backwards compatibility on a server where memory resources are not an issue.

I've also played with nginx, and it's nice too. Slightly faster than lighttpd in many cases. But on the other hand, lighttpd is far more flexible and configurable.

I'd expect the php issue to

I'd expect the php issue to only exist if you're using mod_php, but I'm running php as a CGI for security reasons. I did have to remove some mod_perl handlers when I moved away from prefork though.

I'd like to move to lighttpd to save resources, but I'm running quite a few sites, and some of them use very complicated rewrite rules so for the moment, I'm just going to stick with what I know.

Liraz Siri's picture

You can rewrite any Apache rewrite rule into Lua

We had similar issues with porting rewrite rules to lighttpd from Apache and found the solution in lighttpd's mod_magnet, which allows you to script these things in Lua. Sure there's a learning curve involved but you can pretty much get anything done. I like solving these sorts of challenges so for me it was fun.

Also, lighttpd rewrite rules are getting much more powerful so in the future I bet you'll only need dive into Lua for the really complicated stuff.

Due to copy-on-write and the

Due to copy-on-write and the way virtual memory works, that does come as a surprise to me. Pre-forking shouldn't significantly increase the consumption of RAM and thrash your server that much, at least not for as little as 5 idle-just-forked processes.

I'd say it was good that it seems to have fixed your problem, but I don't believe pre-forking to be the real issue here.

Liraz Siri's picture

In theory, theory and practice are the same, but in practice...

So with copy-on-write semantics pre-forking shouldn't necessarily be an issue but I think that depends a lot on where you fork and what the child process does afterwards. Also, I'm guessing with dynamic languages like Perl in which spamassassin is implemented the performance hit from running multiple instances of the Perl runtime could be significantly greater.

Pre-Forking Sped Us Up

I used your tip about tuning the trade off between memory and CPU time and it helped speed up our server.  Pre-forking wored wonders.  We have lots of workers that all run automotive crm software and we all noticed the difference after I altered the server.

Liraz Siri's picture

Glad to hear it Jason!

Thanks for the positive feedback. It's delightful to hear that people are still benefiting from the posts on our little tips and tricks.

Trade-off did not work for me

Liked your article dude. I tried the trade off technique using prefork, reduced Apache conf to bare minimum and removed all unwanted modules but did not get the expected results out of it. Had to move to worker model. Although both prefork and worker caused the load average to shoot up to 20, the worker model was able to withstand 250 concurrent users against 15 concurrent users from prefork.

It would be extremely helpful if you can share tips on how to reduce the load average which is caused by Apache coz at this stage I would like to stick with Apache instead of switching over to Lighttpd or nginx although both are better than Apache any time.

managed vps

Nice Post!!!!!!!!

Thaks for sharing.....

Virtual private servers

Exceptionally great post!!!!!!

Looking forward for more post like these!!!!!!!!!!

Thanks for sharing!!!!!!

Could someone explain how to do this?

I'm using Amazon EC2 so I'd like to give this a try. Could you give me a clue where to start please?

Here's a clue

For anyone wondering where to look to tweak Apache Prefork conditions, you'll find the settings in /etc/apache2/apache2.conf as follows:

<IfModule mpm_prefork_module>

    StartServers          3
    MinSpareServers       3
    MaxSpareServers      3
    MaxClients          50
    MaxRequestsPerChild   1000
</IfModule>

These are my settings for now (using Amazon EC2 Micro Instance). Once you've made your changes, you'll need to restart Apache.

Load reducing for 24 GB Ram installed server

Hi Liraz Siri, could you please write an easy Apache tuning and optimization guide for 24GB ram installed server?

my entertainment site runing on 6 x AMD Opteron(tm) Processor 4174 HE, Speed 2300.000 MHz, Cache 512 KB

and 24GB Ram installed but my server getting heavy load, hope you write an easy server optimization guide. 

Thanks advance

Liraz Siri's picture

Wow, that's one beefy

Wow, that's one beefy machine. Use lighttpd or nginx. You may be surprised how well that works, once you set it up. :)

Thanks

Thank you so much, i'm gonna try and update here how is working.

How can I do it?

Hi Liraz,

since I'm a REAL LAMP's rookie, can you provide me the optimal settings you tried (as Andrew did)? I've a "resoruce limited LAMP" with 384Mb RAM who runs our company's website... :P

Thank you for your past (and future, I hope) help!

Liraz Siri's picture

Wow, 384MB is very little for

Wow, 384MB is very little for a LAMP server. I'd reduce the number of children as much as possible to conserve memory but you're not going to get very far with that kind of firepower.

If resources are constrained I suggest using an asynchronous web server such as Lighty or Nginx. Good luck!

I'll second that

I'm using Nginx now and it's lightweight and super-fast. Tons better than Apache

I Want to Konw server load depend on...

I have a VPS with 2 server. I want to konw about server load 

when 150 visitors  online on my site then server load going up to 2.5 it is increased my vps limits and when 200 visitors online the server load at 1.6, does the server depand upon the visitors or any other activities of website 

 

Please tell me about this Thanks

 

VPS Server Using Too Much Memory, I Think?

Every few days my VPS server with my host provider runs out of memory and I cannot access anything. I have to try to reboot it from my end. If this does not work I have to contact the host team, and then sometimes trying to access the server causes a Firewall to Block me out from my machine. Can I ask a couple of Questions regarding VPS Servers

My VPS Server along with CPanel Licences costs me about €36 per month.

I have 5 websites and 5 Cpanels set up for those sites.

My VPS server has 1GB of memory I am told. But every 3-4 days I have to reboot it to keep it visible.

Would you have any suggestions why this happens. My hosting company keep saying I need to buy more memory, if that is the case everytime I ad a new account to my hosting plan I will need to buy new memory and if you are hosting many websites then your bill to a hosting company will be astronomical?

All Help Greatly Appreciated.

This is the largest website I have on the server. http://www.kitchens4u.ie - OH and BTW this will not be accessible until Monday 14th October as they have suspended me over the weekend.

Thanks!

Thanks for the post.  My FreeBSD VPS (512MB RAM, dual-core) was swapping like crazy (almost all of the 512M on 1st virtual HD and some of the 2G swap on 2nd virtual HD).

Mysqld was running in its default state (w/o a config file), probably due to a recent fiasco with ports after updating from 8.2 > 9.2.  Copied /usr/local/share/mysql/my-small.cnf to /usr/local/etc/my.cnf.  Copied the "mpm_prefork_module" section of /usr/local/etc/apache22/extra/httpd-mpm.conf into the httpd.conf file.  Set "$max_servers = 2;" in /usr/local/etc/amavisd.conf and rebooted.

*NO* swapping!

Now, don't criticize ... sloppy management, yes.  But I just keep this box in memory of my former business ;-)

Your post was helpful in getting me fixed up without spending *hours* trying stuff :-)

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)