Forum archive
Out of memory: kill process 4217 (apache2) score 185030 or a child
Hello I am using the latest Turnkey witth Joomla 1.5.11, as I am a Joomla fan and wanted to have a test server. All is running in VMware server having 512Mb in the VM. I downloaded; Download 169MB: turnkey-joomla-2009.06-hardy-x86.iso (sig) (manifest) and installed. All very well and nice. But after a couple of days an error (see title) and the server could not be used any more. So I installed again and after 1 day same error, and server can nolonger be used, Webadmin still works but Apache and MySQL are down and so Joomla is down.
Is there maybe a relation with apt-get upgrade, as I did upgrade both VM's?
Is there a solution a fix for this?
I looked in the log files and found in the kernel messages (dmsesg) the following lines;
Mem-info: [ 933.358326] DMA per-cpu: [ 933.358336] CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 [ 933.358359] CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 [ 933.358399] Normal per-cpu: [ 933.358409] CPU 0: Hot: hi: 186, btch: 31 usd: 109 Cold: hi: 62, btch: 15 usd: 39 [ 933.358432] CPU 1: Hot: hi: 186, btch: 31 usd: 28 Cold: hi: 62, btch: 15 usd: 9 [ 933.358456] Active:103840 inactive:20863 dirty:0 writeback:0 unstable:0 [ 933.358470] free:1209 slab:1295 mapped:0 pagetables:318 bounce:0 [ 933.358491] DMA free:2044kB min:88kB low:108kB high:132kB active:1532kB inactive:8668kB present:16256kB pages_scanned:16330 all_unreclaimable? yes [ 933.358523] lowmem_reserve[]: 0 492 492 492 [ 933.358545] Normal free:2792kB min:2792kB low:3488kB high:4188kB active:413828kB inactive:74784kB present:503936kB pages_scanned:738147 all_unreclaimable? yes [ 933.358580] lowmem_reserve[]: 0 0 0 0 [ 933.358601] DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2044kB [ 933.358646] Normal: 1*4kB 2*8kB 1*16kB 4*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2724kB [ 933.358691] Swap cache: add 191672, delete 191643, find 25097/25767, race 0+0 [ 933.358710] Free swap = 0kB [ 933.358720] Total swap = 240932kB [ 933.358730] Free swap: 0kB [ 933.360013] 131072 pages of RAM [ 933.360025] 0 pages of HIGHMEM [ 933.360035] 2207 reserved pages [ 933.360044] 56 pages shared [ 933.360053] 29 pages swap cached [ 933.360064] 0 pages dirty [ 933.360072] 0 pages writeback [ 933.360082] 0 pages mapped [ 933.360091] 1295 pages slab [ 933.360100] 318 pages pagetables [ 933.360113] Out of memory: kill process 4217 (apache2) score 185030 or a child [ 933.360358] Killed process 4217 (apache2) [ 933.441754] apache2 invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 [ 933.441782] Pid: 4204, comm: apache2 Not tainted 2.6.24-24-generic #1 [ 933.441807] [<c0170bba>] oom_kill_process+0x10a/0x120 [ 933.441810] [<c0171007>] out_of_memory+0x167/0x1a0 [ 933.441863] [<c017347c>] __alloc_pages+0x36c/0x3a0 [ 933.441882] [<e08c9660>] mpt_get_msg_frame+0xc0/0x100 [mptbase] [ 933.441911] [<c01755ed>] __do_page_cache_readahead+0x11d/0x250 [ 933.441935] [<c0175b0c>] do_page_cache_readahead+0x4c/0x70 [ 933.441956] [<c01700f4>] filemap_fault+0x2f4/0x420 [ 933.441977] [<c017b4f1>] __do_fault+0x61/0x420 [ 933.441996] [<e088fa50>] scsi_next_command+0x30/0x50 [scsi_mod] [ 933.442033] [<c012109d>] kunmap_atomic+0x3d/0xb0 [ 933.442052] [<c017da78>] handle_mm_fault+0x118/0x730 [ 933.442071] [<c012f483>] do_wait+0x413/0xba0 [ 933.442091] [<c031e9ef>] do_page_fault+0x13f/0x730 [ 933.442111] [<c019fb84>] sys_select+0x194/0x1c0 [ 933.442131] [<c031e8b0>] do_page_fault+0x0/0x730 [ 933.442149] [<c031d082>] error_code+0x72/0x80 [ 933.442168] ======================= [ 933.442179] Mem-info: [ 933.442187] DMA per-cpu: [ 933.442197] CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 [ 933.442314] CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 [ 933.442338] Normal per-cpu: [ 933.442348] CPU 0: Hot: hi: 186, btch: 31 usd: 109 Cold: hi: 62, btch: 15 usd: 39 [ 933.442371] CPU 1: Hot: hi: 186, btch: 31 usd: 37 Cold: hi: 62, btch: 15 usd: 9 [ 933.442395] Active:103016 inactive:21684 dirty:0 writeback:57 unstable:0 [ 933.442410] free:1209 slab:1295 mapped:0 pagetables:318 bounce:0 [ 933.442430] DMA free:2044kB min:88kB low:108kB high:132kB active:1516kB inactive:8668kB present:16256kB pages_scanned:16394 all_unreclaimable? yes [ 933.442463] lowmem_reserve[]: 0 492 492 492 [ 933.442485] Normal free:2792kB min:2792kB low:3488kB high:4188kB active:410548kB inactive:78068kB present:503936kB pages_scanned:739675 all_unreclaimable? yes [ 933.442520] lowmem_reserve[]: 0 0 0 0 [ 933.442538] DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2044kB [ 933.442583] Normal: 1*4kB 2*8kB 1*16kB 4*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2724kB [ 933.442628] Swap cache: add 191736, delete 191661, find 25097/25768, race 0+0 [ 933.442647] Free swap = 206672kB [ 933.442658] Total swap = 240932kB [ 933.442668] Free swap: 206672kB [ 933.443944] 131072 pages of RAM [ 933.443956] 0 pages of HIGHMEM [ 933.443966] 2207 reserved pages [ 933.443976] 56 pages shared [ 933.443985] 75 pages swap cached [ 933.443995] 0 pages dirty [ 933.444004] 57 pages writeback [ 933.444014] 0 pages mapped [ 933.444023] 1295 pages slab [ 933.444032] 318 pages pagetables [ 933.444045] Out of memory: kill process 4286 (mysqld) score 22921 or a child [ 933.444227] Killed process 4286 (mysqld)
Replies (21)
Demo or no demo mode
Reply 1Apart from all details mentioned above, I selected "install to harddisk" from the menu. I know that running demo mode cannot save changes you made.
I hope this will make it clear.
Thx again for your reply.
best regards for a Joomla user.
can you describe exactly how you installed TurnKey Joomla
Reply 2Again, I believe you are running in "live" demo mode rather than actually installing the appliance to disk. Did you select the "Install to hard drive" option from the bootloader menu (second option) or the "Live system" option (first option)?
It would help if you could describe EXACTLY how you installed TurnKey Joomla.
apache maxclients?
Reply 3Hello alon, thx for the threads, they are not so easy for me, will take some time to understand all of the info. I'll come back on this. The first I'll try is the maxclients. But can you tell me a bit more about slicehost, is that some kind of virtual machines like vmware?
best regards for a Joomla user.
The old auto-update bug is long irrelevant
Reply 4The old auto-update bug you mention was fixed in the March 2009 release and didn't effect the memory utilization of appliances at all. It just broke the auto-update feature so you would have to update manually.
How about you try a fresh reinstall of TurnKey Joomla? See if you can reproduce the bug in there?
no one has reported this b4
Reply 5Your right, i did some google again and had 5 hits on the subject of this topic, 3 of them where pionting to my post. But one of them is explaining the mechanism behind the kill, http://www.held.org.il/blog/?p=18 , it's not giving an answer but maybe helps.
And on Turnkey I found this one: >Fixed auto updates
We discovered a nasty bug that broke auto-updates whenever a package update tried to upgrade a configuration file that has been locally modified. This happened because the package management system checks for local modifications before upgrading configuration files and if the configuration file has been modified since installation the user is asked what to do - use the new package version, the local modified version or try merging the two?<
kit.
best regards for a Joomla user.
maxclients too high?
Reply 6to alon, slicehost seem to be the place. This is the topic I'll go for;
http://forum.slicehost.com/comments.php?DiscussionID=3629
kyi.
best regards for a Joomla user.
memory usage on 112
Reply 7to Liraz, 112 is ip# as I have two servers i need to know which one i use, you as well i gues.
The answer on 1)is;
top - 19:24:34 up 8 min, 1 user, load average: 0.00, 0.02, 0.00
Tasks: 62 total, 1 running, 61 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4045 mysql 20 0 92432 13m 4292 S 0 1.3 0:00.48 mysqld
4241 root 20 0 24356 7572 4392 S 0 0.7 0:00.48 apache2
4257 root 20 0 10328 6328 1396 S 0 0.6 0:00.04 miniserv.pl
4283 root 20 0 6360 4076 2116 S 0 0.4 0:00.22 confconsole
4252 www-data 20 0 24356 3824 632 S 0 0.4 0:00.00 apache2
4253 www-data 20 0 24356 3820 628 S 0 0.4 0:00.00 apache2
4254 www-data 20 0 24356 3820 628 S 0 0.4 0:00.00 apache2
4255 www-data 20 0 24356 3820 628 S 0 0.4 0:00.00 apache2
4256 www-data 20 0 24356 3820 628 S 0 0.4 0:00.00 apache2
4314 root 20 0 8108 2508 2024 S 0 0.2 0:00.04 sshd
3928 klog 20 0 2964 1844 408 S 0 0.2 0:00.38 klogd
4165 root 20 0 5292 1720 1392 S 0 0.2 0:00.06 master
4168 postfix 20 0 5340 1700 1392 S 0 0.2 0:00.06 qmgr
1 root 20 0 2756 1672 524 S 0 0.2 0:01.58 init
4167 postfix 20 0 5300 1668 1368 S 0 0.2 0:00.02 pickup
4207 proftpd 20 0 9900 1628 584 S 0 0.2 0:00.02 proftpd
4316 root 20 0 2744 1532 1220 S 0 0.1 0:00.00 bash
But that doesn't help me. You?
I think I need to copy and paste somewhere else, help?I will work on 2) now. The answer on 2) is; No it's not a clean install, I installed a new template, a visitor counter and Joomexplorer. Your' right isolation is the job, I'll try to install #3 asap and let you know.
best regards from a Joomla user.
3th server is running, 113
Reply 8to Liraz, did a clean install, number 113, I did
$top;
top - 22:08:22 up 2 min, 1 user, load average: 0.16, 0.29, 0.13
Tasks: 61 total, 1 running, 60 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.2%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 515524k total, 86856k used, 428668k free, 3240k buffers
Swap: 240932k total, 0k used, 240932k free, 42872k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4026 mysql 20 0 92424 13m 4292 S 0 2.7 0:00.29 mysqld
4202 root 20 0 24292 7516 4380 S 0 1.5 0:00.42 apache2
4218 root 20 0 10308 6320 1396 S 0 1.2 0:00.02 miniserv.pl
4243 root 20 0 6360 4060 2104 S 0 0.8 0:00.14 confconsole
4213 www-data 20 0 24292 3776 628 S 0 0.7 0:00.00 apache2
4214 www-data 20 0 24292 3772 624 S 0 0.7 0:00.00 apache2
4215 www-data 20 0 24292 3772 624 S 0 0.7 0:00.00 apache2
4216 www-data 20 0 24292 3772 624 S 0 0.7 0:00.00 apache2
4217 www-data 20 0 24292 3772 624 S 0 0.7 0:00.00 apache2
4272 root 20 0 7956 2492 2020 S 0 0.5 0:00.06 sshd
3909 klog 20 0 2964 1840 408 S 0 0.4 0:00.20 klogd
4146 root 20 0 5292 1716 1392 S 0 0.3 0:00.04 master
4150 postfix 20 0 5340 1704 1392 S 0 0.3 0:00.02 qmgr
1 root 20 0 2756 1672 524 S 0 0.3 0:01.58 init
4149 postfix 20 0 5300 1672 1368 S 0 0.3 0:00.00 pickup
4271 root 20 0 2800 1568 740 S 0 0.3 0:00.04 dialog
4274 root 20 0 2744 1532 1220 S 0 0.3 0:00.02 bash
Not so much difference on the first line, the first mysql line. Keep in touch.
best regards from a Joomla user.
security upgrade
Reply 9Thx alon for your reply, every reply brings me one step further. But what do you think about apt-get upgrade and apache2 (and mysql are) is going down?
best regards from a Joomla user.
Automatic security updates
Reply 10Just an FYI - TurnKey Linux appliances are configured to automatically apply security updates only, which are configured in security.sources.list
why do we upgrade?
Reply 11to JedMeister, looks like your right. I will stop upgrading "untill futher notice". But how do you upgrade a security risk without apt-get upgrade? My experiance in Linux is also limited.
best regards from a Joomla user.
In my experience...
Reply 12In my (limited) Linux experience, apt-get upgrade has a nasty habit of breaking things. After borking a number of systems (based on a couple of different distros - but all desktop ones) with it I now avoid it like the plauge! I find a much more reliable way is to not upgrade anything that is working unless there is a bug fix, security risk, or if there is added functionality which I want/need.
I think this probably applies even more-so to mission critical systems such as servers.
Perhaps that is the common theme between you and other users experiencing this issue whilst many others aren't??
I think this probably applies even more-so to mission critical systems such as servers.
Perhaps that is the common theme between you and other users experiencing this issue whilst many others aren't??
next morning 114 is down
Reply 13This morning from the internet I cannot not reach my server, "cannot make a connection with mysql" or just blank browser screen. Half a day up and running (or less) and down after apt-get upgrade. That is the only thing I did.
Back to 113.
best regards from a Joomla user.
server #4 is running
Reply 14Hi Liraz, I just cloned server 113 into 114, did all the settings and both are running now. After that I did apt-get update and apt-get upgrade from the console. This is also what I did before on one of the servers, #112.
Lets see what happens, I will route it to the outside.
best regards from a Joomla user.
Many reasons to run out of memory
Reply 15I agree this shouldn't be happening but there are many reasons a Linux system could potentially run out of memory. Other people may be running out of memory for different reasons so the forum posts may or may not apply.
In general, when you run out of memory like this, there is probably some kind of memory leak going on, where a process keeps allocating more and more memory without freeing it back to the system. If this happens it doesn't really matter how much memory you have because it will all eventually run out.
A couple of suggestions:
1) If you could diagnose which process is leaking memory (e.g., run "top" in the command line and then press "M" to sort by memory usage) that would help.
2) How is TurnKey Joomla configured? Did you install any Joomla extensions? Can you reproduce the problem with a clean install? Let's try to isolate this!
Hi
Reply 16to Liraz, thx for your quick response. I am maybe the first to report on this with a Turnkey appliance, I am not the first with this problem in Debian/Ubuntu and/or LAMP. Even on Slicehost you can find posts with the same problem and no solution. I don't give up easy, I did many Joomla installations and on many different servers, even on a iPod shuffle of 512Mb. In a VMware machine of 1Gig it's running out of memory, something is not right here. Whether it's apache or mysql that's to complex for me to find out. But I will follow the post and hope that one of the appliance builders come with an answer. I like Ubuntu very much and Turnkey witch is "optimized for ease of use in server-type usage scenarios and can be deployed in just a few minutes on bare metal" is ideal for me. For now a few minutes are four days/evenings of my spare time. Sorry, but I'll will still follow the post and hope for an answer. Thx again.
p.s. I'll support you with everything I can, just ask.
best regards from a Joomla user.
Don't dispair!
Reply 17Since you are the first to report this problem out of many thousands of users I am going to venture and guess there is something unique about your setup that is causing this problem. If we just find what that something is then TurnKey Joomla should work for you just like it seems to work for everyone else...
Don't be discouraged!
max clients to high
Reply 18to alon, in followed the link i posted and did what chris jonston wrote on his site; http://chrisjohnston.org/2008/configuring-a-lightweight-apache-mysql-ins... . My posted link and chris are covering the same issue. With folling these instructions first joomla could nolonger connect to mysql (had to restore the copy of mysql.conf and had it running again. Second, fine tuning apache2 and apache2 is down. For more details on how it's done, see chris his site, see link above.
I found out that many people have problems with apache and memory usage, some even recommend to use LEMP and not LAMP.
Sorry I spend 6 hours for setting up 2 sites and 6 hours on thoubleshooting, it's too much, it's not gonna work.
best regards from a Joomla user.
apache maxclients too high?
Reply 19Assuming that you have installed to disk and not running in demo mode, I did a quick search and found these threads [1,2] which might help.
The weird thing is that no one has reported similar issues, so I am not quiet sure what the problem could be.
Have you installed any memory intensive joomla extensions?
The weird thing is that no one has reported similar issues, so I am not quiet sure what the problem could be.
Have you installed any memory intensive joomla extensions?
demo mode
Reply 20Hello Iraz thx for your reply The appliance is installed in VMware having 512Mb of RAM and 3Gb of HD. Memory, ram or hd, cannot be a problem as I am using VMware and Joomla for a long time with succes.
The VM (virtual machine) is also not the problem as the Turnkey developers also test in a VM (virtualbox).
Personally I think it has something to do with the upgrade, which changes some files, maybe something like configuration, passwords or things like that. I found an MySQL issue on the forum which started after an upgrade and seem to be a know Ubuntu bug.
The two installation of the appliance have upgrading in common (after the appliance it self of course). The first install worked ok for a couple of days, until I upgraded. Within minutes it showed the error. With the second install I upgrade directly after install and the error showed within minutes.
I also looked around on other forums before I posted.
best regards froma Joomla user.
Are you sure you're not running the appliance in demo mode?
Reply 21I suspect you may be running the Joomla appliance in demo mode rather than actually installing it. In demo mode any updates to the filesystem are stored in ram disk, which would explain why you're running out. Check to make sure.