janadmin's picture
Submitted by janadmin on Sat, 2009/06/27 - 13:31

Hello I am using the latest Turnkey witth Joomla 1.5.11, as I am a Joomla fan and wanted to have a test server. All is running in VMware server having 512Mb in the VM. I downloaded; Download 169MB: turnkey-joomla-2009.06-hardy-x86.iso (sig) (manifest) and installed. All very well and nice. But after a couple of days an error (see title) and the server could not be used any more. So I installed again and after 1 day same error, and server can nolonger be used, Webadmin still works but Apache and MySQL are down and so Joomla is down.

Is there maybe a relation with apt-get upgrade, as I did upgrade both VM's?

Is there a solution a fix for this?

I looked in the log files and found in the kernel messages (dmsesg) the following lines;

Mem-info:
[  933.358326] DMA per-cpu:
[  933.358336] CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
[  933.358359] CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
[  933.358399] Normal per-cpu:
[  933.358409] CPU    0: Hot: hi:  186, btch:  31 usd: 109   Cold: hi:   62, btch:  15 usd:  39
[  933.358432] CPU    1: Hot: hi:  186, btch:  31 usd:  28   Cold: hi:   62, btch:  15 usd:   9
[  933.358456] Active:103840 inactive:20863 dirty:0 writeback:0 unstable:0
[  933.358470]  free:1209 slab:1295 mapped:0 pagetables:318 bounce:0
[  933.358491] DMA free:2044kB min:88kB low:108kB high:132kB active:1532kB inactive:8668kB present:16256kB pages_scanned:16330 all_unreclaimable? yes
[  933.358523] lowmem_reserve[]: 0 492 492 492
[  933.358545] Normal free:2792kB min:2792kB low:3488kB high:4188kB active:413828kB inactive:74784kB present:503936kB pages_scanned:738147 all_unreclaimable? yes
[  933.358580] lowmem_reserve[]: 0 0 0 0
[  933.358601] DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2044kB
[  933.358646] Normal: 1*4kB 2*8kB 1*16kB 4*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2724kB
[  933.358691] Swap cache: add 191672, delete 191643, find 25097/25767, race 0+0
[  933.358710] Free swap  = 0kB
[  933.358720] Total swap = 240932kB
[  933.358730] Free swap:            0kB
[  933.360013] 131072 pages of RAM
[  933.360025] 0 pages of HIGHMEM
[  933.360035] 2207 reserved pages
[  933.360044] 56 pages shared
[  933.360053] 29 pages swap cached
[  933.360064] 0 pages dirty
[  933.360072] 0 pages writeback
[  933.360082] 0 pages mapped
[  933.360091] 1295 pages slab
[  933.360100] 318 pages pagetables
[  933.360113] Out of memory: kill process 4217 (apache2) score 185030 or a child
[  933.360358] Killed process 4217 (apache2)
[  933.441754] apache2 invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
[  933.441782] Pid: 4204, comm: apache2 Not tainted 2.6.24-24-generic #1
[  933.441807]  [<c0170bba>] oom_kill_process+0x10a/0x120
[  933.441810]  [<c0171007>] out_of_memory+0x167/0x1a0
[  933.441863]  [<c017347c>] __alloc_pages+0x36c/0x3a0
[  933.441882]  [<e08c9660>] mpt_get_msg_frame+0xc0/0x100 [mptbase]
[  933.441911]  [<c01755ed>] __do_page_cache_readahead+0x11d/0x250
[  933.441935]  [<c0175b0c>] do_page_cache_readahead+0x4c/0x70
[  933.441956]  [<c01700f4>] filemap_fault+0x2f4/0x420
[  933.441977]  [<c017b4f1>] __do_fault+0x61/0x420
[  933.441996]  [<e088fa50>] scsi_next_command+0x30/0x50 [scsi_mod]
[  933.442033]  [<c012109d>] kunmap_atomic+0x3d/0xb0
[  933.442052]  [<c017da78>] handle_mm_fault+0x118/0x730
[  933.442071]  [<c012f483>] do_wait+0x413/0xba0
[  933.442091]  [<c031e9ef>] do_page_fault+0x13f/0x730
[  933.442111]  [<c019fb84>] sys_select+0x194/0x1c0
[  933.442131]  [<c031e8b0>] do_page_fault+0x0/0x730
[  933.442149]  [<c031d082>] error_code+0x72/0x80
[  933.442168]  =======================
[  933.442179] Mem-info:
[  933.442187] DMA per-cpu:
[  933.442197] CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
[  933.442314] CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
[  933.442338] Normal per-cpu:
[  933.442348] CPU    0: Hot: hi:  186, btch:  31 usd: 109   Cold: hi:   62, btch:  15 usd:  39
[  933.442371] CPU    1: Hot: hi:  186, btch:  31 usd:  37   Cold: hi:   62, btch:  15 usd:   9
[  933.442395] Active:103016 inactive:21684 dirty:0 writeback:57 unstable:0
[  933.442410]  free:1209 slab:1295 mapped:0 pagetables:318 bounce:0
[  933.442430] DMA free:2044kB min:88kB low:108kB high:132kB active:1516kB inactive:8668kB present:16256kB pages_scanned:16394 all_unreclaimable? yes
[  933.442463] lowmem_reserve[]: 0 492 492 492
[  933.442485] Normal free:2792kB min:2792kB low:3488kB high:4188kB active:410548kB inactive:78068kB present:503936kB pages_scanned:739675 all_unreclaimable? yes
[  933.442520] lowmem_reserve[]: 0 0 0 0
[  933.442538] DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2044kB
[  933.442583] Normal: 1*4kB 2*8kB 1*16kB 4*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2724kB
[  933.442628] Swap cache: add 191736, delete 191661, find 25097/25768, race 0+0
[  933.442647] Free swap  = 206672kB
[  933.442658] Total swap = 240932kB
[  933.442668] Free swap:       206672kB
[  933.443944] 131072 pages of RAM
[  933.443956] 0 pages of HIGHMEM
[  933.443966] 2207 reserved pages
[  933.443976] 56 pages shared
[  933.443985] 75 pages swap cached
[  933.443995] 0 pages dirty
[  933.444004] 57 pages writeback
[  933.444014] 0 pages mapped
[  933.444023] 1295 pages slab
[  933.444032] 318 pages pagetables
[  933.444045] Out of memory: kill process 4286 (mysqld) score 22921 or a child
[  933.444227] Killed process 4286 (mysqld)
Forum: 
Support
Liraz Siri's picture

Liraz Siri - Sun, 2009/06/28 - 03:09
I suspect you may be running the Joomla appliance in demo mode rather than actually installing it. In demo mode any updates to the filesystem are stored in ram disk, which would explain why you're running out. Check to make sure.
janadmin's picture

- Mon, 2009/06/29 - 00:07
Hello Iraz thx for your reply The appliance is installed in VMware having 512Mb of RAM and 3Gb of HD. Memory, ram or hd, cannot be a problem as I am using VMware and Joomla for a long time with succes. The VM (virtual machine) is also not the problem as the Turnkey developers also test in a VM (virtualbox). Personally I think it has something to do with the upgrade, which changes some files, maybe something like configuration, passwords or things like that. I found an MySQL issue on the forum which started after an upgrade and seem to be a know Ubuntu bug. The two installation of the appliance have upgrading in common (after the appliance it self of course). The first install worked ok for a couple of days, until I upgraded. Within minutes it showed the error. With the second install I upgrade directly after install and the error showed within minutes. I also looked around on other forums before I posted. best regards froma Joomla user.
best regards from a Joomla user.
Liraz Siri's picture

Liraz Siri - Mon, 2009/06/29 - 11:20
Again, I believe you are running in "live" demo mode rather than actually installing the appliance to disk. Did you select the "Install to hard drive" option from the bootloader menu (second option) or the "Live system" option (first option)?

It would help if you could describe EXACTLY how you installed TurnKey Joomla.

janadmin's picture

- Mon, 2009/06/29 - 12:15
Apart from all details mentioned above, I selected "install to harddisk" from the menu. I know that running demo mode cannot save changes you made. I hope this will make it clear. Thx again for your reply. best regards for a Joomla user.
best regards from a Joomla user.
Alon Swartz's picture

Alon Swartz - Mon, 2009/06/29 - 12:02
Assuming that you have installed to disk and not running in demo mode, I did a quick search and found these threads [1,2] which might help.

The weird thing is that no one has reported similar issues, so I am not quiet sure what the problem could be.

Have you installed any memory intensive joomla extensions?
janadmin's picture

- Mon, 2009/06/29 - 12:28
Hello alon, thx for the threads, they are not so easy for me, will take some time to understand all of the info. I'll come back on this. The first I'll try is the maxclients. But can you tell me a bit more about slicehost, is that some kind of virtual machines like vmware? best regards for a Joomla user.
best regards from a Joomla user.
janadmin's picture

- Mon, 2009/06/29 - 12:53
Your right, i did some google again and had 5 hits on the subject of this topic, 3 of them where pionting to my post. But one of them is explaining the mechanism behind the kill, http://www.held.org.il/blog/?p=18 , it's not giving an answer but maybe helps. And on Turnkey I found this one: >Fixed auto updates We discovered a nasty bug that broke auto-updates whenever a package update tried to upgrade a configuration file that has been locally modified. This happened because the package management system checks for local modifications before upgrading configuration files and if the configuration file has been modified since installation the user is asked what to do - use the new package version, the local modified version or try merging the two?< kit. best regards for a Joomla user.
best regards from a Joomla user.
Liraz Siri's picture

Liraz Siri - Mon, 2009/06/29 - 21:10
The old auto-update bug you mention was fixed in the March 2009 release and didn't effect the memory utilization of appliances at all. It just broke the auto-update feature so you would have to update manually.

How about you try a fresh reinstall of TurnKey Joomla? See if you can reproduce the bug in there?

janadmin's picture

- Mon, 2009/06/29 - 14:09
to alon, slicehost seem to be the place. This is the topic I'll go for; http://forum.slicehost.com/comments.php?DiscussionID=3629 kyi. best regards for a Joomla user.
best regards from a Joomla user.
janadmin's picture

- Mon, 2009/06/29 - 20:57
to alon, in followed the link i posted and did what chris jonston wrote on his site; http://chrisjohnston.org/2008/configuring-a-lightweight-apache-mysql-ins... . My posted link and chris are covering the same issue. With folling these instructions first joomla could nolonger connect to mysql (had to restore the copy of mysql.conf and had it running again. Second, fine tuning apache2 and apache2 is down. For more details on how it's done, see chris his site, see link above. I found out that many people have problems with apache and memory usage, some even recommend to use LEMP and not LAMP. Sorry I spend 6 hours for setting up 2 sites and 6 hours on thoubleshooting, it's too much, it's not gonna work. best regards from a Joomla user.
best regards from a Joomla user.
Liraz Siri's picture

Liraz Siri - Mon, 2009/06/29 - 21:07
Since you are the first to report this problem out of many thousands of users I am going to venture and guess there is something unique about your setup that is causing this problem. If we just find what that something is then TurnKey Joomla should work for you just like it seems to work for everyone else...

Don't be discouraged!

janadmin's picture

- Mon, 2009/06/29 - 21:45
to Liraz, thx for your quick response. I am maybe the first to report on this with a Turnkey appliance, I am not the first with this problem in Debian/Ubuntu and/or LAMP. Even on Slicehost you can find posts with the same problem and no solution. I don't give up easy, I did many Joomla installations and on many different servers, even on a iPod shuffle of 512Mb. In a VMware machine of 1Gig it's running out of memory, something is not right here. Whether it's apache or mysql that's to complex for me to find out. But I will follow the post and hope that one of the appliance builders come with an answer. I like Ubuntu very much and Turnkey witch is "optimized for ease of use in server-type usage scenarios and can be deployed in just a few minutes on bare metal" is ideal for me. For now a few minutes are four days/evenings of my spare time. Sorry, but I'll will still follow the post and hope for an answer. Thx again. p.s. I'll support you with everything I can, just ask. best regards from a Joomla user.
best regards from a Joomla user.
Liraz Siri's picture

Liraz Siri - Mon, 2009/06/29 - 22:18
I agree this shouldn't be happening but there are many reasons a Linux system could potentially run out of memory. Other people may be running out of memory for different reasons so the forum posts may or may not apply.

In general, when you run out of memory like this, there is probably some kind of memory leak going on, where a process keeps allocating more and more memory without freeing it back to the system. If this happens it doesn't really matter how much memory you have because it will all eventually run out.

A couple of suggestions:

1) If you could diagnose which process is leaking memory (e.g., run "top" in the command line and then press "M" to sort by memory usage) that would help.

2) How is TurnKey Joomla configured? Did you install any Joomla extensions? Can you reproduce the problem with a clean install? Let's try to isolate this!

janadmin's picture

- Mon, 2009/06/29 - 22:33
to Liraz, 112 is ip# as I have two servers i need to know which one i use, you as well i gues. The answer on 1)is; 
top - 19:24:34 up 8 min,  1 user,  load average: 0.00, 0.02, 0.00
Tasks:  62 total,   1 running,  61 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4045 mysql     20   0 92432  13m 4292 S    0  1.3   0:00.48 mysqld
 4241 root      20   0 24356 7572 4392 S    0  0.7   0:00.48 apache2
 4257 root      20   0 10328 6328 1396 S    0  0.6   0:00.04 miniserv.pl
 4283 root      20   0  6360 4076 2116 S    0  0.4   0:00.22 confconsole
 4252 www-data  20   0 24356 3824  632 S    0  0.4   0:00.00 apache2
 4253 www-data  20   0 24356 3820  628 S    0  0.4   0:00.00 apache2
 4254 www-data  20   0 24356 3820  628 S    0  0.4   0:00.00 apache2
 4255 www-data  20   0 24356 3820  628 S    0  0.4   0:00.00 apache2
 4256 www-data  20   0 24356 3820  628 S    0  0.4   0:00.00 apache2
 4314 root      20   0  8108 2508 2024 S    0  0.2   0:00.04 sshd
 3928 klog      20   0  2964 1844  408 S    0  0.2   0:00.38 klogd
 4165 root      20   0  5292 1720 1392 S    0  0.2   0:00.06 master
 4168 postfix   20   0  5340 1700 1392 S    0  0.2   0:00.06 qmgr
    1 root      20   0  2756 1672  524 S    0  0.2   0:01.58 init
 4167 postfix   20   0  5300 1668 1368 S    0  0.2   0:00.02 pickup
 4207 proftpd   20   0  9900 1628  584 S    0  0.2   0:00.02 proftpd
 4316 root      20   0  2744 1532 1220 S    0  0.1   0:00.00 bash
But that doesn't help me. You? I think I need to copy and paste somewhere else, help?

I will work on 2) now. The answer on 2) is; No it's not a clean install, I installed a new template, a visitor counter and Joomexplorer. Your' right isolation is the job, I'll try to install #3 asap and let you know.

best regards from a Joomla user.

best regards from a Joomla user.
janadmin's picture

- Mon, 2009/06/29 - 23:29

to Liraz, did a clean install, number 113, I did

$top;
top - 22:08:22 up 2 min,  1 user,  load average: 0.16, 0.29, 0.13
Tasks:  61 total,   1 running,  60 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.2%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    515524k total,    86856k used,   428668k free,     3240k buffers
Swap:   240932k total,        0k used,   240932k free,    42872k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4026 mysql     20   0 92424  13m 4292 S    0  2.7   0:00.29 mysqld
 4202 root      20   0 24292 7516 4380 S    0  1.5   0:00.42 apache2
 4218 root      20   0 10308 6320 1396 S    0  1.2   0:00.02 miniserv.pl
 4243 root      20   0  6360 4060 2104 S    0  0.8   0:00.14 confconsole
 4213 www-data  20   0 24292 3776  628 S    0  0.7   0:00.00 apache2
 4214 www-data  20   0 24292 3772  624 S    0  0.7   0:00.00 apache2
 4215 www-data  20   0 24292 3772  624 S    0  0.7   0:00.00 apache2
 4216 www-data  20   0 24292 3772  624 S    0  0.7   0:00.00 apache2
 4217 www-data  20   0 24292 3772  624 S    0  0.7   0:00.00 apache2
 4272 root      20   0  7956 2492 2020 S    0  0.5   0:00.06 sshd
 3909 klog      20   0  2964 1840  408 S    0  0.4   0:00.20 klogd
 4146 root      20   0  5292 1716 1392 S    0  0.3   0:00.04 master
 4150 postfix   20   0  5340 1704 1392 S    0  0.3   0:00.02 qmgr
    1 root      20   0  2756 1672  524 S    0  0.3   0:01.58 init
 4149 postfix   20   0  5300 1672 1368 S    0  0.3   0:00.00 pickup
 4271 root      20   0  2800 1568  740 S    0  0.3   0:00.04 dialog
 4274 root      20   0  2744 1532 1220 S    0  0.3   0:00.02 bash

Not so much difference on the first line, the first mysql line. Keep in touch.

best regards from a Joomla user.

best regards from a Joomla user.
janadmin's picture

- Tue, 2009/06/30 - 22:59
Hi Liraz, I just cloned server 113 into 114, did all the settings and both are running now. After that I did apt-get update and apt-get upgrade from the console. This is also what I did before on one of the servers, #112. Lets see what happens, I will route it to the outside. best regards from a Joomla user.
best regards from a Joomla user.
Jeremy Davis's picture

Jeremy Davis - Wed, 2009/07/01 - 03:26
In my (limited) Linux experience, apt-get upgrade has a nasty habit of breaking things. After borking a number of systems (based on a couple of different distros - but all desktop ones) with it I now avoid it like the plauge! I find a much more reliable way is to not upgrade anything that is working unless there is a bug fix, security risk, or if there is added functionality which I want/need.

I think this probably applies even more-so to mission critical systems such as servers.

Perhaps that is the common theme between you and other users experiencing this issue whilst many others aren't??
janadmin's picture

- Wed, 2009/07/01 - 12:43
to JedMeister, looks like your right. I will stop upgrading "untill futher notice". But how do you upgrade a security risk without apt-get upgrade? My experiance in Linux is also limited. best regards from a Joomla user.
best regards from a Joomla user.
Alon Swartz's picture

Alon Swartz - Wed, 2009/07/01 - 13:06
Just an FYI - TurnKey Linux appliances are configured to automatically apply security updates only, which are configured in security.sources.list
janadmin's picture

- Wed, 2009/07/01 - 13:26
Thx alon for your reply, every reply brings me one step further. But what do you think about apt-get upgrade and apache2 (and mysql are) is going down? best regards from a Joomla user.
best regards from a Joomla user.
janadmin's picture

- Wed, 2009/07/01 - 12:37
This morning from the internet I cannot not reach my server, "cannot make a connection with mysql" or just blank browser screen. Half a day up and running (or less) and down after apt-get upgrade. That is the only thing I did. Back to 113. best regards from a Joomla user.
best regards from a Joomla user.

Add new comment