TurnKey Linux Virtual Appliance Library

TKLBAM private hub

Hans Harder's picture

The TKLBAM is a nice solution for an integrated backup/restore, only I am not allowed to use your hub (company policies).

So I want to backup to a different local Turnkey Appliance which can act as a Hub for TKLBAM with local storage only.

Are there any intensions of making the hub available ?

Jeremy's picture

I can't speak for Core Devs but has been mentioned

But AFAIK it is not even a definate that they will ever end up doing that, let alone when.

Personally I agree that it would be a great thing, but we'll have to wait and see.

In the meantime I would personally be arguing the point further up the chain of command at your work. Surely they undestand the importance of backups? (There's probably a policy about that?!) And AWS storage is pretty cheap and far more reliable than a local hard drive, tape or DVD will ever be! I think if you did a bit of homework around what your works privacy, security and data protection policies are, and some research on how they are satisfied by the Hub and its interface with AWS (eg data encryption, private keys, AWS reliability, etc) I think you could make a near bullet-proof case for allowing use of TKL Hub for backups in your corporation.

Hans Harder's picture

Yes I know that, but again it

Yes I know that, but again it is not allowed.... Not to mention the burdan it is to arrange such things.....administration/approval/auditors....

Storage is not a problem, got enough nonlocal storage

I now do simple rsync's to one TKLcore which acts as a backup storage in a different location, but it would be much nicer to use the integrated TKLBAM....

 

 

 

QUOTE:  ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol

Jeremy's picture

Fair call Hans

I work part time for Govt & part time for small NGO. Its much much easier to get things of this nature done through the NGO thats for sure, so I can understand your frustration there.

Liraz Siri's picture

We would need to change the design to support a private Hub

Hans, I'm pretty sure you realize this, but just in case - you don't have to store your backups or keys on the Hub/S3, though you still have to register for the Hub because that's where TKLBAM auto-updates the backup profiles from. This is not really that different from how you get security updates from Ubuntu though.

So could you focus in more specifically on what part of the Hub your organization would object to you using? Maybe we can work around that somehow.

Regarding supporting private instances of the Hub, that's tricky. The current design doesn't support that. It might not be obvious, but TKLBAM owes a big part of the streamlined usability to the tight integration between the appliances, the Hub and Amazon Web Services.

The Hub was designed as infrastructure, not as standalone software that you could run private instances of.

OTOH, it actually does make sense that you would want to do this in certain circumstances, but I imagine this would mostly be a requirement of large bureaucratic organizations, which TurnKey so far has not been targeted at. It's a different world which we don't really know much about.

It helps when people such as yourself share their perspectives from behind "he iron enterprise curtain". Eventually, we'd like to make TurnKey as universally appealing as possible, but it will take time. On the flip side, there could be opportunities in serving large organizations. What they sometimes lack in good sense they make up for with access to resources which the open source community could find a good use for.

Hans Harder's picture

-

OK, Liraz maybe I misunderstood some parts....

I am only interested in the backup and migration part...without being dependent on the hub while I doing a backup or a restore...

If I need to connect to the hub once every xx time for getting the correct backup profile or a profile for a migration, I don't mind. that is just as the security updates... 

So at first registering at the hub and getting the correct backup profile for the current appliance sounds good. But what is needed for a restore if something went wrong ? Do I need to connect to the hub first or is that information stored with the backup.

The only objection about the hub depends how critical it is in a backup/restore situation. If it is not required at that moment, then there is no problem.  If it is required then I have to guarantee it will be there when needed. But what I understand from the demo is that I need the hub for seeing the backuplist and I need to use it for restoring..

 

Yes, large bureaucratic organizations is a challenging world...  Sometimes it is easier to try to work around the rules, buy something yourself and write some extra hours instead of trying to do that officially, which probably won't work or take months...... purchase orders, approvals, preferred suppliers..... For technical people it can be very very frustrating... :)

 

 


Liraz Siri's picture

Why don't you experiment a little?

If you store the backups in a manual storage target with --address then you don't need the Hub to restore at all because restoring doesn't require a backup profile. Just the backup itself. I propose you experiment a little with the manual --address option until you figure it all out. Also, read the FAQ in the documentation carefully, it covers this.
Hans Harder's picture

ok, my experiment so far...

  • I have now a TKL hub account....
  • I added the apikey to the tklbam lamp distro
  • Create a local directory for a backup as /usr/backup (just to keep it simple)
  • excluded the /usr/backup directory in /etc/tklbam/overrides
  • tried : tklbam-backup --accept=/usr/backup -s

Didn't work.... because backups seems not to be enabled in the TKLhub

Seems it requires me to get an Amazon S3 account first, which I am reluctant to do.

 

QUOTE:  ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol

Hans Harder's picture

output from a simulate backup from webmin

> tklbam-backup --simulate
Traceback (most recent call last):
File "/usr/bin/tklbam-backup", line 267, in <module>
main()
File "/usr/bin/tklbam-backup", line 196, in main
conf.profile = get_profile(hb)
File "/usr/bin/tklbam-backup", line 123, in get_profile
new_profile = hb.get_new_profile(turnkey_version, profile_timestamp)
File "/usr/lib/tklbam/hub.py", line 207, in get_new_profile
response = self._api('GET', 'archive/timestamp/', attrs)
File "/usr/lib/tklbam/hub.py", line 185, in _api
return API.request(method, self.API_URL + uri, attrs, headers)
File "/usr/lib/tklbam/hub.py", line 127, in request
raise NotSubscribedError()
hub.NotSubscribedError: Backups are not yet enabled for your TurnKey Hub account. Log
into the Hub and go to the "Backups" section for instructions.

Is there another way to get the hub account active without creating an amazon account, so I can do local backups ?

 

 

QUOTE:  ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol

Jeremy's picture

AFAIK you must have Amazon account - even if you don't use it

I'm not sure of the reasoning but it seems you do need an Amazon account. If you don't use though, there will be no charge.

Hans Harder's picture

-

I can't imagine that this was the intension.... seems to me, it is something they overlooked in the design of the first version of the hub (no problem, that happens...)

I am not creating an account and giving my creditcard number to a 3rd party for something I am not going to use. 

Seems now tklbam only fails on the first connection where it tries to retrieve the tklbam profile for the current appliance, all other actions, (local) backup/restore, should work without requiring a connection to the hub or having an amazon account.

QUOTE:  ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol

Liraz Siri's picture

TKLBAM designed primarily for cloud backups

TKLBAM's design is optimized for storing into Amazon S3 and that is intentional as it streamlines usability and enables usage scenarios that require a globally accessible backup store, which aren't practical when you store backups locally (e.g., migrating between clouds). True, TKLBAM also allows you to store your backups to any target supported by Duplicity (e.g., FTP, rsync, file, SFTP), but we mainly envisioned this as being used for local caching of backups rather than the primary mechanism. Perhaps this is failure of imagination on our part, but we wanted to make TKLBAM not only powerful but also super simple to use and we couldn't figure out how to do that with ad-hoc local storage.

Bottom line, you shouldn't have any issues with trusting Amazon with your credit card data. If you insist you don't want to use them for storage they just won't charge you for it. If you change your mind about that the ability to use that feature will always be at your fingertips and future versions of TKLBAM will add additional features that build on that assumption.

Hans Harder's picture

-

Yes, I understand all that, It would be more open if you could use TKLbam without the Amazon restriction. The only thing needed is the TKLBAM initialisation for the appliance backup profile to work always, independently of a selected provider (currently being only Amazon)

Because now the initialisation does not work, I can't adapt the python sources in /usr/lib/tklbam to make it work for local/san backups.

Another problem is:

  • I make an Amazon account and put my creditcard in it
  • I put up 20 TKL appliances in our company (using in company systems/storage)
  • Someone discovers that nice webmin TKLBAM interface and uses it
  • I get the bill on my private credit card.

So I hope in the future version of TKLBAM, you can make the initialisation more flexible.

Thanks anyway for always answering that nicely... TKL still is one of my preferred distro's

QUOTE:  ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol

Liraz Siri's picture

We'll look into it...

You raise a few points we hadn't really considered. At first I thought it would be strange that you would have to put in your own private credit card for this, but then I remembered what it was like when I worked for a huge corporation a decade ago. All that red tape really sucked. Glad I'm out.

OTOH, even if you put in your private credit card, and someone "accidentally" uses TKLBAM to store to Amazon S3, costs are unlikely to spiral out of control quickly. Since TKLBAM only saves changes to the base installation, backups are usually just a few GBs. At $0.15/GB you'd have to store a lot for it to rack up to a meaningful expense. And you'd probably notice the load on the network too. Uploading hundreds of GBs to the Internet from most corporate LANs is still uncommon.

Still the same error

Today I started using the TKLBAM with local storage. But this issue is stille there. Will there be any change to be expected in short term? I understand the S3 discussion completly, but in my current situation it is no option to enforce an Amazon S3 account.

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)