fpoff's picture

I may have found the problem with the issue below, but I could still use some help from your big tkl brains to fix it.

I realized my backups were failing after I added a separate development server. Here's my set up:

 

Production Server: 
A micro EC2 instance with a domain name, let's call it mytklserver.com

Dev Server: 
A small EC2 instance which I pointed DNS to using a subdomain, like dev.mytklserver.com 

The reason I think there is an issue with TKLBAM using the same domain name on two servers is this:

When I do a "tklbam list" on both servers I get the exact same backups listed. I think there might be some type of error by having both servers on the same domain, even though the Dev box is on the dev subdomain.

I tried deleting the latest backup and trying a tklbam backup --disable-resume 

But that resulted in the same error as below.   

I think if I could re-initialize or reinstall tklbam to start fresh it might work. So my question(s) is: 

How should I un-install and re-install tklbam on my production server? 

OR

How else could I "reset" tklbam to see if the "two servers on one domain" issue is the problem? 

Thanks again for any input. 

PREVIOUS ISSUE POST BELOW:
___________________________________________

 

I'm getting the following error when trying to run tklbam backup from the command line.

The instance is a m1.small with kernal id of aki-88aa75e1

It's a tkl lamp wordpress box with multiple wordpress installs running on it. The instance has a 500 gb volume attached, which has 35 gb being used. 

Thanks in advance for any insight you can provide. Here's the error: 

Traceback (most recent call last):

  File "/usr/bin/tklbam", line 46, in <module>

    CliWrapper.main()

  File "/usr/lib/tklbam/cliwrapper.py", line 88, in main

    commands[command].main()

  File "/usr/lib/tklbam/cmd_backup.py", line 446, in main

    opt_resume, True, dump_path if dump_path else "/")

  File "/usr/lib/tklbam/backup.py", line 237, in __init__

    self._create_extras(extras_paths, profile_paths, backup_conf)

  File "/usr/lib/tklbam/backup.py", line 180, in _create_extras

    if mysql.MysqlService.is_running():

  File "/usr/lib/tklbam/mysql.py", line 629, in is_running

    executil.getoutput('mysqladmin -s ping')

  File "/usr/lib/python2.6/dist-packages/executil.py", line 63, in getoutput

    error, output = commands.getstatusoutput(command)

  File "/usr/lib/python2.6/commands.py", line 55, in getstatusoutput

    pipe = os.popen('{ ' + cmd + '; } 2>&1', 'r')

OSError: [Errno 12] Cannot allocate memory

Forum: 
Steve Goldsmith's picture

Hi,

 

I was wondering if you ever got this resolved...I am having exactly the same issue. (I tried posting the error message but it thought it was spam :()

Thanks

Steve

 

Jeremy Davis's picture

My first guess is that it's run out of memory. Can you tell us a little bit more about the environment where it's running and the resources and load it has?
Steve Goldsmith's picture

It was orginially running in AWS on an M1.small with 1.7GB so I thought memory might be the issue as well so I bumped it to an M1.medium with 3.75GB and I get the same error.

Here is the results of free -m

 

             total       used       free     shared    buffers     cached            
Mem:          3836        369       3467          0        131        124            
-/+ buffers/cache:        112       3723                                             
Swap:            0          0          0        

This is a TKL Lamp box (12.0) 32-bit

It is a copy of a production box running in a different VPC and I am the only user so the load is minimal.

Thanks for your response                                    

Steve Goldsmith's picture

Hey Jeremy,

I think I got to the bottom of it.

I was watching the memory usage as tklbam ran and it only ever used about 50%

Howeve since I didnt have a Swap partition on my instance I thought it woulnd hurt to give it a go.

As soon as I added a Swap, it started working, even though it didnt use any of it.

It is still going, so I will let you know if it suceeds

 

Steve

Jeremy Davis's picture

If you look at the error it actually says "OSError: [Errno 12] Cannot allocate memory".

So my guess is that at some point it tries to allocate a big chunk of memory for something it's about to do (copy a big file? perhaps a database?) and if you don't have enough free (to allocate) then it fails (so you won't see a spike in memory usage).

I'm no kernel expert but AFAIK the way that the Linux kernel works, is that when a new thread starts it tries to allocate as much RAM as the parent process already has allocated (usually what it's actually using plus a bit in buffers/cache). Once it does that, the new thread then starts but it will only use as much RAM as it actually needs (not as much as it was allocated). Again, why the swap is needed but not actually used...

Thanks for posting back. I'm sure that this will be helpful for other users.

Add new comment