TurnKey Linux Virtual Appliance Library

Out of Memory - Serious Apache problems!

Rune Skretteberg's picture

Running Joomla - turnkey linux.distro, on vmware esx.

Components:
Akeeba Backup
Allvideos
eXtplorer
JCE Administation
Joomla LMS
Project Log

Templates from Artisteer

TunkeyLinux .iso out of box installed to  with webmin 1.50, Curl libraries (SSL) + AKA TurnKey Linux Backup and Migration

Problems started after some updates from WEBmin 

dmesg:
		[24274.131693] Free swap  = 336852kB
[24274.131693] Total swap = 706820kB
[24274.131694] Free swap:       336852kB
[24274.132443] 131072 pages of RAM
[24274.132444] 0 pages of HIGHMEM
[24274.132445] 2243 reserved pages
[24274.132445] 67 pages shared
[24274.132446] 350 pages swap cached
[24274.132447] 0 pages dirty
[24274.132447] 352 pages writeback
[24274.132448] 5 pages mapped
[24274.132448] 1587 pages slab
[24274.132449] 592 pages pagetables
[24274.132450] Out of memory: kill process 9714 (mysqld) score 31822 or a child

[24274.132595] Killed process 9714 (mysqld)
 
Stituation..
site.domain/administrator  - is working..  Pretty strange..
site.domain - is not working and this error goes to console..
 
[24274.132450] Out of memory: kill process 9714 (mysqld) score 31822 or a child
[24274.132595] Killed process 9714 (mysqld)
 

Database Error: "Unable to connect to the database:Could not connect to MySQL"

Liraz Siri's picture

Are you sure you installed from ISO?

You may by running the ISO in LiveCD / demo mode which is non-persistent. Changes to the filesystem go to RAM. That would explain why you ran out of memory after updates. You can install the appliance to HD through the confconsole. Either that or choose install to HD from the bootloader menu. FYI, in the next release the install to HD option will be the default.

If it's not that you probably just need to allocate more memory to your appliance. If memory serves some of the extensions you are using are memory hungry.

Rune Skretteberg's picture

ITS installed to HDD

Could it be some memory tuning? - apache2.conf?

http://httpd.apache.org/docs/2.0/mod/mpm_common.html#maxmemfree

There must be some tweek here?

Liraz Siri's picture

Use the ESX4 images instead of the ISO

I think I know why you're using the ISO instead of the VM build. You're using ESX4 aren't you and the VM build which is for ESX3.5 broke. What you don't know because we've done a terrible job of promoting it is that there are new ESX4 optimized images available on sourceforge. I'll update the documentation "virtualization notes" in the documentation to reflect this until we update the appliance templates.
Rune Skretteberg's picture

turnkey-joomla-2009.10-2-hardy-x86.ovf

Yes, your ovf files should be for "ESX 4.0" but, installing from ISO were fine until I met "out of memory".

The virtualization notes could be a help for solving the problem with "out of memory"

Again I guess its a configuration issue..

Liraz Siri's picture

What processes are using the most memory?

If you log into webmin -> system -> running processes, you can see which processes are eating all of your memory. It's probably Apache but it never hurts to find out for sure.

Try rebooting the system and monitoring memory usage every now and then. If there is a process that is constantly eating more and more memory there's a memory leak somewhere. If the process is Apache you'll need to experiment to find out which web application is causing it.

Hope this helps!

Rune Skretteberg's picture

Out of Memory - Serious Apache problems! VMWARE ESX4

Solution:

I just downloaded a better distirbution from http://sourceforge.net/projects/turnkeylinux/files/

I am now up and running with ESX4 using: http://sourceforge.net/projects/turnkeylinux/files/turnkey-joomla/2009.10-2-hardy-x86/turnkey-joomla-2009.10-2-hardy-x86-esx4.zip/download

Site moved with: Xcloner from http://www.joomlaplug.com/aff/idevaffiliate.php?id=1305

That saved my day!

John Carver's picture

Out of Memory errors on Drupal 11.0 appliance

I encountered these "Out of memory" errors while trying to debug performance issues on the TurnKey Drupal appliance.

Background:
I'm running version 11.0 on a VirtualBox 3.2.10 client.  The host is a quad-core cpu with 4 GB of ram.  The Drupal appliance was first installed from the version 11.0-RC on a virtual machine with 512 MB allocated.  Soon after getting the site operational, I noticed performance issues and a good bit of swapping, so I increased the client ram first to 1024 MB and later to 1536 MB (without increasing the size of the swap partition).  

# free -ml
             total       used       free     shared    buffers     cached
Mem:          1507       1451         56          0          9         44
Low:           859        804         55
High:          647        646          1
-/+ buffers/cache:       1397        110
Swap:         1229       1081        148

Hoping that the problem would be fixed in the final release, I applied the updates when they were released, but the problem continues.  After searching the logs, I came across the "Out of memory" errors and a Google search led me to this thread.

Eventually I decided to install munin server on the host machine and munin-node on the TurnKey client to collect more data about memory usage.  The graphs showed what I think is a classic case of memory leakage.  

Munin graph showing memory leakage

Munin was installed and data collection started yesterday (Monday) about 3 pm local time.  A couple of hours later, I restarted apache while attempting to get Munin to log apache activity.  You can see a couple of restarts before I found the problem and installed the missing libwww-perl.  Following that, I did little but watch as the app memory slowly increased.  The server has essentially no load other than a daily googlebot scan.

The Munin swap in/out graph for the same time looks like this:

Munin Swap in/out graph

Notice the swap outs greatly out number the swap ins.

On a hunch, I decided to take a closer look at apache.
# date
Tue Jan 25 17:33:21 UTC 2011  (11:33:21 local time on the graphs above)
# ps aux | grep '^www-data'
www-data 20586  0.0  5.3 235056 82280 ?        S    Jan24   0:35 /usr/sbin/apache2 -k start
www-data 20587  0.0  4.8 225816 74512 ?        S    Jan24   0:33 /usr/sbin/apache2 -k start
www-data 20588  0.0  4.8 214564 74840 ?        S    Jan24   0:30 /usr/sbin/apache2 -k start
www-data 20589  0.0  4.6 229364 71876 ?        S    Jan24   0:35 /usr/sbin/apache2 -k start
www-data 20615  0.0  5.3 252468 82548 ?        S    Jan24   0:40 /usr/sbin/apache2 -k start
www-data 21285  0.0  5.3 221772 82228 ?        S    Jan24   0:31 /usr/sbin/apache2 -k start
www-data 21287  0.0  6.0 238768 93312 ?        S    Jan24   0:31 /usr/sbin/apache2 -k start
www-data 21288  0.0  6.7 247280 104864 ?       S    Jan24   0:35 /usr/sbin/apache2 -k start
www-data 21289  0.0  4.5 194984 69556 ?        S    Jan24   0:26 /usr/sbin/apache2 -k start
www-data 23931  0.0  3.7 209656 57612 ?        S    00:11   0:28 /usr/sbin/apache2 -k start

If I'm understanding these numbers, we have over 200 MB allocated to each of ten processes which seems way more than they should have.

pmap shows one particular entry has allocated the bulk of the memory.
# pmap -x 20586 20587 20588 20589 20615 21285 21287 21288 21289 23931 | grep 21caa000
21caa000  184980       -       -       - rw---    [ anon ]
21caa000  175740       -       -       - rw---    [ anon ]
21caa000  164488       -       -       - rw---    [ anon ]
21caa000  179288       -       -       - rw---    [ anon ]
21caa000  202324       -       -       - rw---    [ anon ]
21caa000  176288       -       -       - rw---    [ anon ]
21caa000  188692       -       -       - rw---    [ anon ]
21caa000  197204       -       -       - rw---    [ anon ]
21caa000  144840       -       -       - rw---    [ anon ]
21caa000  159604       -       -       - rw---    [ anon ]

At 18:10 (12:10 local) we finally ran out of swap space and had to kill a process.  You can see the downward tic on the memory graph above as the apache2 process is killed.

Jan 25 18:10:24 ns02 kernel: [1459876.232143] Killed process 20588 (apache2)

# date
Tue Jan 25 18:36:52 UTC 2011 (12:36 local time)

# ps aux | grep '^www-data'
www-data 20586  0.0  5.8 268236 90760 ?        S    Jan24   0:41 /usr/sbin/apache2 -k start
www-data 20587  0.0  5.1 250012 79668 ?        S    Jan24   0:38 /usr/sbin/apache2 -k start
www-data 20589  0.0  6.2 272972 96520 ?        S    Jan24   0:46 /usr/sbin/apache2 -k start
www-data 20615  0.0  5.3 274156 82532 ?        S    Jan24   0:44 /usr/sbin/apache2 -k start
www-data 21285  0.0  6.0 242880 92776 ?        S    Jan24   0:36 /usr/sbin/apache2 -k start
www-data 21287  0.0  4.9 248132 76272 ?        S    Jan24   0:35 /usr/sbin/apache2 -k start
www-data 21288  0.0  6.0 260972 93080 ?        S    Jan24   0:38 /usr/sbin/apache2 -k start
www-data 21289  0.0  6.3 237904 97824 ?        S    Jan24   0:33 /usr/sbin/apache2 -k start
www-data 23931  0.0  4.2 226764 66212 ?        S    00:11   0:32 /usr/sbin/apache2 -k start

Notice that we have one fewer apache2 process running, but the remainder continue to use more memory.

# pmap -x 20586 20587 20589 20615 21285 21287 21288 21289 23931 | grep 21caa000
21caa000  218160       -       -       - rw---    [ anon ]
21caa000  199936       -       -       - rw---    [ anon ]
21caa000  222896       -       -       - rw---    [ anon ]
21caa000  224012       -       -       - rw---    [ anon ]
21caa000  192736       -       -       - rw---    [ anon ]
21caa000  199956       -       -       - rw---    [ anon ]
21caa000  210896       -       -       - rw---    [ anon ]
21caa000  187760       -       -       - rw---    [ anon ]
21caa000  176712       -       -       - rw---    [ anon ]

apache2 -M shows the following modules are currently loaded.

# APACHE_RUN_USER=www-data APACHE_RUN_GROUP=www-data apache2 -M
Loaded Modules:
 core_module (static)
 log_config_module (static)
 logio_module (static)
 mpm_prefork_module (static)
 http_module (static)
 so_module (static)
 alias_module (shared)
 auth_basic_module (shared)
 authn_file_module (shared)
 authz_default_module (shared)
 authz_groupfile_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 cgi_module (shared)
 deflate_module (shared)
 dir_module (shared)
 env_module (shared)
 mime_module (shared)
 negotiation_module (shared)
 php5_module (shared)
 reqtimeout_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 ssl_module (shared)
 status_module (shared)
 substitute_module (shared)
Syntax OK

At this point I'm stumped as what to do next.  I'm pretty sure one of the apache modules has a memory leak, but my level of knowledge doesn't help me figure out which one.  At first I thought that it was something I had done wrong, but in reading this thread and another one indicating others are experiencing problems with Mediawiki and Joomla, I decided to post here in the hope that sharing the information would help get the problem resolved.  If anyone has additional suggestions about what to try next, I'd be happy to learn.

Information is free, knowledge is acquired, but wisdom is earned.

John Carver's picture

memory leak in mod_substitute

I never found a solution to the suspected memory leak documented above.  I did make some changes to the apache config which seemed to help slow down the problem.  This morning I ran across a thread on Old Nabble that describes a memory leak in mod_substitute about the same time we were seeing the memory problems with the TurnKey appliances.  I'm not sure which of the appliances use mod_substitute other than the Drupal6 appliance.  According to the Apache bug report 50559, it was fixed in trunk in r1176019 by Stefan Fritsch on 2011-09-26 20:11:17 UT.

I now have

# apache2 -v       
Server version: Apache/2.2.14 (Ubuntu)
Server built:   Nov  3 2011 03:29:16

so I think I have the fixed version.  I'll have to try switching apache settings back to their defaults and see how the server memory behaves.

Information is free, knowledge is acquired, but wisdom is earned.

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)