Out of memory: kill process 4217 (apache2) score 185030 or a child
Hello I am using the latest Turnkey witth Joomla 1.5.11, as I am a Joomla fan and wanted to have a test server. All is running in VMware server having 512Mb in the VM. I downloaded; Download 169MB: turnkey-joomla-2009.06-hardy-x86.iso (sig) (manifest) and installed. All very well and nice. But after a couple of days an error (see title) and the server could not be used any more. So I installed again and after 1 day same error, and server can nolonger be used, Webadmin still works but Apache and MySQL are down and so Joomla is down.
Is there maybe a relation with apt-get upgrade, as I did upgrade both VM's?
Is there a solution a fix for this?
I looked in the log files and found in the kernel messages (dmsesg) the following lines;
Mem-info: [ 933.358326] DMA per-cpu: [ 933.358336] CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 [ 933.358359] CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 [ 933.358399] Normal per-cpu: [ 933.358409] CPU 0: Hot: hi: 186, btch: 31 usd: 109 Cold: hi: 62, btch: 15 usd: 39 [ 933.358432] CPU 1: Hot: hi: 186, btch: 31 usd: 28 Cold: hi: 62, btch: 15 usd: 9 [ 933.358456] Active:103840 inactive:20863 dirty:0 writeback:0 unstable:0 [ 933.358470] free:1209 slab:1295 mapped:0 pagetables:318 bounce:0 [ 933.358491] DMA free:2044kB min:88kB low:108kB high:132kB active:1532kB inactive:8668kB present:16256kB pages_scanned:16330 all_unreclaimable? yes [ 933.358523] lowmem_reserve[]: 0 492 492 492 [ 933.358545] Normal free:2792kB min:2792kB low:3488kB high:4188kB active:413828kB inactive:74784kB present:503936kB pages_scanned:738147 all_unreclaimable? yes [ 933.358580] lowmem_reserve[]: 0 0 0 0 [ 933.358601] DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2044kB [ 933.358646] Normal: 1*4kB 2*8kB 1*16kB 4*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2724kB [ 933.358691] Swap cache: add 191672, delete 191643, find 25097/25767, race 0+0 [ 933.358710] Free swap = 0kB [ 933.358720] Total swap = 240932kB [ 933.358730] Free swap: 0kB [ 933.360013] 131072 pages of RAM [ 933.360025] 0 pages of HIGHMEM [ 933.360035] 2207 reserved pages [ 933.360044] 56 pages shared [ 933.360053] 29 pages swap cached [ 933.360064] 0 pages dirty [ 933.360072] 0 pages writeback [ 933.360082] 0 pages mapped [ 933.360091] 1295 pages slab [ 933.360100] 318 pages pagetables [ 933.360113] Out of memory: kill process 4217 (apache2) score 185030 or a child [ 933.360358] Killed process 4217 (apache2) [ 933.441754] apache2 invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 [ 933.441782] Pid: 4204, comm: apache2 Not tainted 2.6.24-24-generic #1 [ 933.441807] [<c0170bba>] oom_kill_process+0x10a/0x120 [ 933.441810] [<c0171007>] out_of_memory+0x167/0x1a0 [ 933.441863] [<c017347c>] __alloc_pages+0x36c/0x3a0 [ 933.441882] [<e08c9660>] mpt_get_msg_frame+0xc0/0x100 [mptbase] [ 933.441911] [<c01755ed>] __do_page_cache_readahead+0x11d/0x250 [ 933.441935] [<c0175b0c>] do_page_cache_readahead+0x4c/0x70 [ 933.441956] [<c01700f4>] filemap_fault+0x2f4/0x420 [ 933.441977] [<c017b4f1>] __do_fault+0x61/0x420 [ 933.441996] [<e088fa50>] scsi_next_command+0x30/0x50 [scsi_mod] [ 933.442033] [<c012109d>] kunmap_atomic+0x3d/0xb0 [ 933.442052] [<c017da78>] handle_mm_fault+0x118/0x730 [ 933.442071] [<c012f483>] do_wait+0x413/0xba0 [ 933.442091] [<c031e9ef>] do_page_fault+0x13f/0x730 [ 933.442111] [<c019fb84>] sys_select+0x194/0x1c0 [ 933.442131] [<c031e8b0>] do_page_fault+0x0/0x730 [ 933.442149] [<c031d082>] error_code+0x72/0x80 [ 933.442168] ======================= [ 933.442179] Mem-info: [ 933.442187] DMA per-cpu: [ 933.442197] CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 [ 933.442314] CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 [ 933.442338] Normal per-cpu: [ 933.442348] CPU 0: Hot: hi: 186, btch: 31 usd: 109 Cold: hi: 62, btch: 15 usd: 39 [ 933.442371] CPU 1: Hot: hi: 186, btch: 31 usd: 37 Cold: hi: 62, btch: 15 usd: 9 [ 933.442395] Active:103016 inactive:21684 dirty:0 writeback:57 unstable:0 [ 933.442410] free:1209 slab:1295 mapped:0 pagetables:318 bounce:0 [ 933.442430] DMA free:2044kB min:88kB low:108kB high:132kB active:1516kB inactive:8668kB present:16256kB pages_scanned:16394 all_unreclaimable? yes [ 933.442463] lowmem_reserve[]: 0 492 492 492 [ 933.442485] Normal free:2792kB min:2792kB low:3488kB high:4188kB active:410548kB inactive:78068kB present:503936kB pages_scanned:739675 all_unreclaimable? yes [ 933.442520] lowmem_reserve[]: 0 0 0 0 [ 933.442538] DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2044kB [ 933.442583] Normal: 1*4kB 2*8kB 1*16kB 4*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2724kB [ 933.442628] Swap cache: add 191736, delete 191661, find 25097/25768, race 0+0 [ 933.442647] Free swap = 206672kB [ 933.442658] Total swap = 240932kB [ 933.442668] Free swap: 206672kB [ 933.443944] 131072 pages of RAM [ 933.443956] 0 pages of HIGHMEM [ 933.443966] 2207 reserved pages [ 933.443976] 56 pages shared [ 933.443985] 75 pages swap cached [ 933.443995] 0 pages dirty [ 933.444004] 57 pages writeback [ 933.444014] 0 pages mapped [ 933.444023] 1295 pages slab [ 933.444032] 318 pages pagetables [ 933.444045] Out of memory: kill process 4286 (mysqld) score 22921 or a child [ 933.444227] Killed process 4286 (mysqld)
Are you sure you're not running the appliance in demo mode?
demo mode
can you describe exactly how you installed TurnKey Joomla
It would help if you could describe EXACTLY how you installed TurnKey Joomla.
Demo or no demo mode
apache maxclients too high?
The weird thing is that no one has reported similar issues, so I am not quiet sure what the problem could be.
Have you installed any memory intensive joomla extensions?
apache maxclients?
no one has reported this b4
The old auto-update bug is long irrelevant
How about you try a fresh reinstall of TurnKey Joomla? See if you can reproduce the bug in there?
maxclients too high?
max clients to high
Don't dispair!
Don't be discouraged!
Hi
Many reasons to run out of memory
In general, when you run out of memory like this, there is probably some kind of memory leak going on, where a process keeps allocating more and more memory without freeing it back to the system. If this happens it doesn't really matter how much memory you have because it will all eventually run out.
A couple of suggestions:
1) If you could diagnose which process is leaking memory (e.g., run "top" in the command line and then press "M" to sort by memory usage) that would help.
2) How is TurnKey Joomla configured? Did you install any Joomla extensions? Can you reproduce the problem with a clean install? Let's try to isolate this!
memory usage on 112
top - 19:24:34 up 8 min, 1 user, load average: 0.00, 0.02, 0.00 Tasks: 62 total, 1 running, 61 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4045 mysql 20 0 92432 13m 4292 S 0 1.3 0:00.48 mysqld 4241 root 20 0 24356 7572 4392 S 0 0.7 0:00.48 apache2 4257 root 20 0 10328 6328 1396 S 0 0.6 0:00.04 miniserv.pl 4283 root 20 0 6360 4076 2116 S 0 0.4 0:00.22 confconsole 4252 www-data 20 0 24356 3824 632 S 0 0.4 0:00.00 apache2 4253 www-data 20 0 24356 3820 628 S 0 0.4 0:00.00 apache2 4254 www-data 20 0 24356 3820 628 S 0 0.4 0:00.00 apache2 4255 www-data 20 0 24356 3820 628 S 0 0.4 0:00.00 apache2 4256 www-data 20 0 24356 3820 628 S 0 0.4 0:00.00 apache2 4314 root 20 0 8108 2508 2024 S 0 0.2 0:00.04 sshd 3928 klog 20 0 2964 1844 408 S 0 0.2 0:00.38 klogd 4165 root 20 0 5292 1720 1392 S 0 0.2 0:00.06 master 4168 postfix 20 0 5340 1700 1392 S 0 0.2 0:00.06 qmgr 1 root 20 0 2756 1672 524 S 0 0.2 0:01.58 init 4167 postfix 20 0 5300 1668 1368 S 0 0.2 0:00.02 pickup 4207 proftpd 20 0 9900 1628 584 S 0 0.2 0:00.02 proftpd 4316 root 20 0 2744 1532 1220 S 0 0.1 0:00.00 bashBut that doesn't help me. You? I think I need to copy and paste somewhere else, help?I will work on 2) now. The answer on 2) is; No it's not a clean install, I installed a new template, a visitor counter and Joomexplorer. Your' right isolation is the job, I'll try to install #3 asap and let you know.
best regards from a Joomla user.
3th server is running, 113
to Liraz, did a clean install, number 113, I did
$top; top - 22:08:22 up 2 min, 1 user, load average: 0.16, 0.29, 0.13 Tasks: 61 total, 1 running, 60 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.2%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 515524k total, 86856k used, 428668k free, 3240k buffers Swap: 240932k total, 0k used, 240932k free, 42872k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4026 mysql 20 0 92424 13m 4292 S 0 2.7 0:00.29 mysqld 4202 root 20 0 24292 7516 4380 S 0 1.5 0:00.42 apache2 4218 root 20 0 10308 6320 1396 S 0 1.2 0:00.02 miniserv.pl 4243 root 20 0 6360 4060 2104 S 0 0.8 0:00.14 confconsole 4213 www-data 20 0 24292 3776 628 S 0 0.7 0:00.00 apache2 4214 www-data 20 0 24292 3772 624 S 0 0.7 0:00.00 apache2 4215 www-data 20 0 24292 3772 624 S 0 0.7 0:00.00 apache2 4216 www-data 20 0 24292 3772 624 S 0 0.7 0:00.00 apache2 4217 www-data 20 0 24292 3772 624 S 0 0.7 0:00.00 apache2 4272 root 20 0 7956 2492 2020 S 0 0.5 0:00.06 sshd 3909 klog 20 0 2964 1840 408 S 0 0.4 0:00.20 klogd 4146 root 20 0 5292 1716 1392 S 0 0.3 0:00.04 master 4150 postfix 20 0 5340 1704 1392 S 0 0.3 0:00.02 qmgr 1 root 20 0 2756 1672 524 S 0 0.3 0:01.58 init 4149 postfix 20 0 5300 1672 1368 S 0 0.3 0:00.00 pickup 4271 root 20 0 2800 1568 740 S 0 0.3 0:00.04 dialog 4274 root 20 0 2744 1532 1220 S 0 0.3 0:00.02 bashNot so much difference on the first line, the first mysql line. Keep in touch.
best regards from a Joomla user.
server #4 is running
In my experience...
I think this probably applies even more-so to mission critical systems such as servers.
Perhaps that is the common theme between you and other users experiencing this issue whilst many others aren't??
why do we upgrade?
Automatic security updates
security upgrade
next morning 114 is down
Memory limit for php scripts
PHP.ini
Joomla 2009.07 released
We just announced the release of Joomla 2009.07, could you give it a try to see if it solves your problems.
Update on Joomla appliance
Hello alon, up to now my Joomla 2009.6 is running for more then a week without going down and without manual upgrade.
At the moment I am downloading 2009.7 release and I will put that in test as well and inform you via this forum.
Added 2009/7/10, I think we can close this topic moderator, 2009.6 is running fine now for 12 days.
I love Turnkey and the support, many thanks, I'll 'keep you informed of my progress.
sorry alon
I did 3 installations on 2 different servers, just to avoid typing errors, but all 3 gave; Database Error: Unable to connect to the database:Could not connect to MySQL.
I wait.
Very serious bug
My apologies for not looking into this immediately when you reported the problem. It turns out you discovered a very serious bug in the latest version of Joomla which we released a couple of days ago. This should never have happened. Thank you so much for reporting this.
Same problem with Mediawiki
Hi,
I encountered the same problem with the latest version of Turnkey Mediawiki.
I installed it into a VMware 5.5 virtual machine (256MB RAM, 10GB virtual disc, static IP address) and it ran well first. When I retried it a few days later I got a timeout message in the web browsers (tried IE on WinXP, Firefox and Konqueror on SuSE 10.0) and the "Out of Memory" errors in the VM.
I set it up a second time (new VM and new installation) with the same effect: first ok, after some while broken.
Running "top" with "M" key pressed I saw apache2 consuming up to about 87% of the memory and then going away (thus being killed).
Unfortunately I have absolutely no experience with that web stuff (that is the reason why I didn't install mediawiki directly in my SuSE machine but searched for a solution "out of the box"), so I really hope someone can help me.
confirmed: out of memory error on TKMW mediawiki
Guys,
I can confirm that I also have 2 TKMW VMs that show the exact same error after a recent upgrade. I used an older version of the wiki, not the most recent. I think it was 2009.6 but I recently deleted the old ISO and saved the new 2009.10 one.
It first happened on one VM that I upgraded using "apt-get upgrade", nothing special. I noted move to apache 2.2.9 but don't remember if I saw an upgrade of PHP.
The first VM started to bail with "out of memory" errors of apache2 and/or mysqld. I compared settings/config files on the two VMs that run TKMW but found no major differences apart from email addresses, etc. The first one was upgraded, the second was not. So I decided to upgrade that one as well and this morning my team lead called my in shock that the wiki was down. :)
So something does get messed up with the upgrade... If I can do anything to help, plz let me know.
Chris.
problem maybe related to mysql/mysqld_safe?
On my VMs running TKMW 2009.6, I get issues with either apache2 or mysqld reporting "out of memory" errors". I can stop apache fine, but I can't stop mysql regulary. /etc/init.d/mysql stop will leave mysqld_safe alone and the process keeps hogging 100% CPU usage.
Could there be a similar issue with some secret key for MW and MySQL, much like the Joomla issue?
Chris.
Joomla issue not related
As I replied on the joomla issue thread, the issues are not related.
I'm sure you've read the comments in this thread, but you might want to take another look - It might spark some other ideas.
Take a look at the mysql log files to see if they include any errors that might help zero in on the issue. Regarding mysqld_safe, if you need to, just kill it: killall mysqld_safe
Sorry I can't be much help on this issue, its really difficult to debug remotely.
Can I suggest starting a new thread when you post your findings on the mysql issue. This one is getting really long and its not clear yet whether the issues are related.
Side note: Have you given the 2009.10-2 release a try?
Thanks for the update
Thanks for the update, Alon.
I don't think it is mysql related. Maybe it's PHP related, I vaguely recall a PHP update as well. It started to happen after an apt-get upgrade, just like janadmin reported. I've shutdown mysqld_safe, thanks. That was weird, as I could shutdown the mysql process, but mysqld_safe would keep running.
No I haven't looked at 2009.10 yet, there was no reason to until last week, when the first VM started to use 100% CPU and 100% RAM in either apache2/mysqld. It is my last resort as I have a hunch that it is very similar issue to the Joomla problem.
I'll start a new thread, thanks. :)There is a new thread for the out of memory issue with mediawiki.
Same problem with Oct2009 version of Drupal (6.14)
I do experience the same problem with Turnkey Drupal version 6.14 package. Previous package version (6.10) was working fine but when I upgrade to 6.14, I started to get "Out of memory: kill process xxxx (apache2) score xxxxxx or a child" type of error after one day running...
I download a brand new copy of the appliance (.vmx file) of "Turnkey key Drupal " and install it as is without adding any new module or content and run it". It stop working after one day as well.
Does anyone else experienced that problem with Turnkey Drupal 6.14 package? Any known solution to this?
Thanks
Same problem with Oct2009 version of Drupal (6.14)
I do experience the same problem with Turnkey Drupal version 6.14 package. Previous package version (6.10) was working fine but when I upgrade to 6.14, I started to get "Out of memory: kill process xxxx (apache2) score xxxxxx or a child" type of error after one day running...
I download a brand new copy of the appliance (.vmx file) of "Turnkey key Drupal " and install it as is without adding any new module or content and run it". It stop working after one day as well.
Does anyone else experienced that problem with Turnkey Drupal 6.14 package?
Do you have swap? How much memory are you giving the appliance?
regenerate your secrets
See if running /usr/lib/live-installer.d/40regen-drupal-secrets fixes your problems. A similar problem in Joomla was solved this way.
It seems the upgrade using apt-get replaces some files but fails to regen the secret keys used in communication (don't know the specifics). I am trying to solve my Mediawiki error using the work-around mentioned.
Post new comment