mike's picture

It's my first day on the job and I have been asked to solve the following issue:

mediawiki installed on a VM with 1GB mem and 16g disk.

Anicdotal history:

Server ran fine for months then performance fell off a bit.  After and apt-get for aptitude, stated problem started.

Logs indicate a number of failures by cron-apt to get hardy-security fixes AND that appache2 is killing procs due to lack of memory:

Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.620330] apache2 invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.620427] Pid: 4217, comm: apache2 Not tainted 2.6.24-28-generic #1
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.620595]  [oom_kill_process+0x10a/0x120] oom_kill_process+0x10a/0x120
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.620820]  [out_of_memory+0x167/0x1a0] out_of_memory+0x167/0x1a0
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.620829]  [agpgart:__alloc_pages+0x36c/0x3a0] __alloc_pages+0x36c/0x3a0
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.620839]  [__do_page_cache_readahead+0x11d/0x250] __do_page_cache_readahead+0x11d/0x250
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.620844]  [sync_page+0x0/0x40] sync_page+0x0/0x40
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.620855]  [do_page_cache_readahead+0x4c/0x70] do_page_cache_readahead+0x4c/0x70
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.620861]  [filemap_fault+0x2f4/0x420] filemap_fault+0x2f4/0x420
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.620865]  [__wake_up_common+0x4b/0x80] __wake_up_common+0x4b/0x80
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.620879]  [__do_fault+0x61/0x420] __do_fault+0x61/0x420
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.620888]  [<f888f223>] scsi_run_queue+0xd3/0x1a0 [scsi_mod]
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621070]  [libata:kunmap_atomic+0x3d/0x2f40] kunmap_atomic+0x3d/0xb0
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621076]  [handle_mm_fault+0x435/0x730] handle_mm_fault+0x435/0x730
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621081]  [do_wait+0x413/0xba0] do_wait+0x413/0xba0
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621088]  [do_page_fault+0x13f/0x730] do_page_fault+0x13f/0x730
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621096]  [<c0126010>] default_wake_function+0x0/0x10
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621103]  [sys_waitpid+0x27/0x30] sys_waitpid+0x27/0x30
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621107]  [do_page_fault+0x0/0x730] do_page_fault+0x0/0x730
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621112]  [error_code+0x72/0x80] error_code+0x72/0x80
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621156]  =======================
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621186] Mem-info:
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621214] DMA per-cpu:
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621246] CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621271] Normal per-cpu:
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621274] CPU    0: Hot: hi:  186, btch:  31 usd: 165   Cold: hi:   62, btch:  15 usd:  54
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621276] HighMem per-cpu:
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621278] CPU    0: Hot: hi:   42, btch:   7 usd:  12   Cold: hi:   14, btch:   3 usd:   2
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621320] Active:126001 inactive:126229 dirty:0 writeback:0 unstable:0
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621322]  free:2993 slab:1246 mapped:1 pagetables:621 bounce:0
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621370] DMA free:4068kB min:68kB low:84kB high:100kB active:4204kB inactive:3996kB present:16256kB p
ages_scanned:12806 all_unreclaimable? yes
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621379] lowmem_reserve[]: 0 873 1000 1000
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621426] Normal free:7776kB min:3744kB low:4680kB high:5616kB active:436704kB inactive:436560kB prese
nt:894080kB pages_scanned:1514315 all_unreclaimable? yes
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621429] lowmem_reserve[]: 0 0 1016 1016
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621433] HighMem free:128kB min:128kB low:264kB high:400kB active:63096kB inactive:64360kB present:13
0048kB pages_scanned:335539 all_unreclaimable? yes
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621440] lowmem_reserve[]: 0 0 0 0
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621467] DMA: 1*4kB 0*8kB 2*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4
068kB
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621553] Normal: 88*4kB 38*8kB 15*16kB 11*32kB 6*64kB 6*128kB 1*256kB 2*512kB 0*1024kB 0*2048kB 1*409
6kB = 7776kB
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621560] HighMem: 2*4kB 3*8kB 0*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB
 = 128kB
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621595] Swap cache: add 229574, delete 229574, find 92/161, race 0+0
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621605] Free swap  = 0kB
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621616] Total swap = 915664kB
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.621634] Free swap:            0kB
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.626476] 262144 pages of RAM
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.626585] 32768 pages of HIGHMEM
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.626596] 3367 reserved pages
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.626607] 83 pages shared
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.626615] 0 pages swap cached
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.626627] 0 pages dirty
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.626637] 0 pages writeback
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.626647] 1 pages mapped
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.626661] 1246 pages slab
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.626670] 621 pages pagetables
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.667742] apache2 invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.667750] Pid: 4217, comm: apache2 Not tainted 2.6.24-28-generic #1
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.667762]  [oom_kill_process+0x10a/0x120] oom_kill_process+0x10a/0x120
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.667774]  [out_of_memory+0x167/0x1a0] out_of_memory+0x167/0x1a0
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.667782]  [agpgart:__alloc_pages+0x36c/0x3a0] __alloc_pages+0x36c/0x3a0
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.667791]  [__do_page_cache_readahead+0x11d/0x250] __do_page_cache_readahead+0x11d/0x250
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.667796]  [sync_page+0x0/0x40] sync_page+0x0/0x40
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.667805]  [do_page_cache_readahead+0x4c/0x70] do_page_cache_readahead+0x4c/0x70
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.667810]  [filemap_fault+0x2f4/0x420] filemap_fault+0x2f4/0x420
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.667815]  [__wake_up_common+0x4b/0x80] __wake_up_common+0x4b/0x80
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.667822]  [__do_fault+0x61/0x420] __do_fault+0x61/0x420
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.667827]  [<f888f223>] scsi_run_queue+0xd3/0x1a0 [scsi_mod]
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.667851]  [libata:kunmap_atomic+0x3d/0x2f40] kunmap_atomic+0x3d/0xb0
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.667857]  [handle_mm_fault+0x435/0x730] handle_mm_fault+0x435/0x730
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.667861]  [do_wait+0x413/0xba0] do_wait+0x413/0xba0
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.667868]  [do_page_fault+0x13f/0x730] do_page_fault+0x13f/0x730
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.667874]  [<c0126010>] default_wake_function+0x0/0x10
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.667882]  [sys_waitpid+0x27/0x30] sys_waitpid+0x27/0x30
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.667886]  [do_page_fault+0x0/0x730] do_page_fault+0x0/0x730
Jul 19 14:58:47 fahqsna09vapp45 kernel: [  340.667890]  [error_code+0x72/0x80] error_code+0x72/0x80
etc, etc

after which the system is brain dead and must be restarted from the VM consol.

Thanks for any help.

At this point I'm willing to save current data and just reinstall.   Can I use the "Importing a previous wiki into TKMW" cookbook to accomplish this.  I really can't loose any of my boss' boss's data.

-- Mike

Forum: 
mike's picture

looks like the libresolv.so.2 was updated on 10 june.

can't rebuild the resolver file with resolvconf() because several s links are missing/empty

Add new comment