You are here
Hi TKL users and devs,
This post is intended to be a discussion on how TKL and TKLBAM have benefitted me and some features I would like to see.
If this is served better in another place then please let me know, otherwise I will start by saying I am very impressed so far and look forward to continuing development.
I write software for windows and Linux that is 90% of the time a database frontend and consequently I provide onsite and offsite servers that include file servers, db servers and printing servers (so people can manage printing costs).
I have used TKLBAM for several MySQL servers in a test scenario to ensure it can do what I need. I plan to use it for my first full scale backup in the comming week. Twice it has saved me hours of work recreating tables when a VM went KAPUT!
My testing is not estensive and there may be features I have missed but for what I intend to do I have tested thoroughly.
The following features I would like to see;
-
Backup statistics in hub or updated in the server CLI itself.
My motivation behind this is to determine how long a large backup might take and the ease of mind provided by "numbers" - Ability to cancel and delete a "backup in progress" in the HUB
- Ability to reuse numbers from deleted backups in the HUB
- Ability to group backups into "Folders" or other management methods.
- Postgres to work with TKLBAM
- Micro instances to become available in cloud deployments
- A light version of TKLBAM that backs up configuration scripts, settings etc
- Several new Turnkey appliances - I have posted a few in the blueprints section
- More as I think of them
I think the best way to help this occur is to evangelise the product, which I am doing with my network of colleagues. If there are practical exmples of ways to provide help I would be limited (due to the need to earn money!) but will do what I can where I can.
Its funny that I found TKL when I was researching some apps I was going to use to create my own image templates - a welcomed surprise!
Comments
The general forum is the perfect place for this type of feedback.
Backup statistics: Not sure how TKLBAM can estimate how long duplicity will take to perform the upload. This might be something that can be hacked into duplicity itself. I recall chatting to Liraz about this, but I don't recall his answer unfortunately...
Delete backup-in-progress: It's on my todo list.
Reuse numbers from deleted backups: Can you elaborate. If backups are deleted they are deleted, along with all associated data.
Grouping backups: An idea that has been tossed around in the past is to support tagging in backups / servers, and provide a filter listing option. Would that solve your issue?
TKLBAM postgres support: High on Liraz's todo list.
Micro / EBS backed instances: Work in progress, and will probably be included with the upcoming TKL 11.2 maintenance release.
TKLBAM light: Can you please elaborate? Keep in mind that you can override the backup profile to include/exclude databases and paths.
More TKL appliances: TKL 11 part 2 is long overdue, but very high on our todo list as well. If you're interested in pushing certain appliances forward, having a TKLPatch available will most certainly help.
Elaborating on original posts
Backup statistics: (CLI)
Reuse numbers from deleted backups:
Here is a screen dump to demonstrate what I am talking about. I must admit I do not understand the inner workings so this might not even be possible.
as you can see there are many holes where I deleted a backup. Is it possible that when the next new backup is generated it fills the first available spot rather than the next incremental spot? Also I understand this may cause issues with backups but can it also be possible to move (in this example) say #16 to #2?
Grouping backups:
Filter listings can be good and possibly for you easier to implement but I prefer to categorise clients and their backups in a hierachial type system much like windows explorer or the standard "tree" control in a GUI might work. Either way as I use this more and more I will have backups flowing for pages at a time. A choice for peoples preferences would be good too.
TKLBAM light:
I understand the override feature for including/exluding but there was something specific I was after with this which I just cannot remember. I will have to revisit the test I was performing earlier and report back.
[edit] I must be on drugs when I wrote this! what I was actually after is a message to a syslog server or an email confirming success. While the HUB interface is good I dont really want to hunt through a huge list making sure everything backed up. I want to use my own logging server that throws alerts when it isn't updated with correct and timely responses.
Chris Musty
Director
Specialised Technologies
Thoughts
Re-using backup ids: TKLBAM intentionally doesn't re-use old backup ids because it's not safe. Otherwise an old VM you forgot about (or bring back to life) may overwrite a "new" backup.
However, the order the backups show up as in the Hub could conceivably by controlled separately though from the backup id. It's a UI issue.
Grouping: we've always understood that as people use TKLBAM more heavily we may have to introduce better management interfaces than a flat list. At the same time we would still like to be very careful about not complicating the interface for new users. Whatever solution we eventually go with will have to balance the needs of new users with more advanced users.
TKLBAM reporting: I get it. I've put this down in my todo list and will think about it some more later.
My 2c
A couple of the things that Chris mentioned have got me thinking...
The idea of optional TKLBAM confirmation emails, and/or some sort of logging (that could log to an unyet designed TKL log server) would be cool I think.
Please correct me if I'm wrong but what I take from Chris' suggestion of TKLBAM-light is as basically to allow common site-wide config. eg at home I like to use my ISP mirror for Ubuntu repos so rather than setting each image individually I could just run my generic TKLBAM-site-setup. This could be also used to configure other site settings and standard software that you want on all machines.
Another little thing: It'd be great if I could set my Hub default eg for me here in Australia I'd like to set SE Asia as my default location.
And of course I'm loving TKL too, but you guys know that already! :)
Thought about making a syslog appliance
It has long been on my to do list as every syslog server I use is built from scratch and hobbled together.
I have been considering making this with a custom web interface (as I find free syslog gui projects very limited and rare unless you pay - does anyone have a suggestion on a free gui?) that runs from the server itself. This will obviously not be a high traffic server but should sustain mabey 10,20,30 possibly 50-100 inputs per second?
I certainly use syslog servers allot so I assume others would too.
Would be interesting making my first TKL appliance.
Guess I need to read up on how its all done properly!
Chris Musty
Director
Specialised Technologies
@jeremy, could you expand on
@jeremy, could you expand on the idea of TKLBAM confirmation emails.
@chris, @jeremy - A TKL logging server is interesting, if someone could come up with the TKLPatch that would be awesome.
I'd like to get Liraz's thoughts on TKLBAM-light, it's an interesting idea...
Lastly, the Hub does (or at least should) set the default Amazon region for both your backups and when launching appliances, based on your IP. If it's not setting it, thats a bug and needs to be fixed.
Just a simple confirmation email
Having said that I'm more than happy to help out Chris if there is anything I can help you with let me know. TKLPatch is pretty well documented and with a basic understanding of Linux is pretty easy to use. In essence it's an install script(s) with a file overlay (files to be overlayed over the root filesystem) that can then be applied to an ISO (idealy) or a running system (mixed results).
more comments...
TKLBAM email: nice ideas, but needs some thought regarding user-experience and implementation. Added to my todo list.
TKL logging server: Can't wait to see what Chris comes up with...
Hub auto-location: For reference, this is the first blog post on the issue. Looking at the auto-generated map, AU should be associated with AP-SOUTHEAST. Could you take a look at your backups location? Also, could you try running the following and let me know the results (don't worry, it won't change anything, just output the closest archives per your IP address).
Could you also drop me an email with your ipaddress, so I can perform some tests on the geo-location service.
When I find some time
I would like to do this but am restrained by the need to earn money. Probably in about 4-6 weeks I should be able to start something. I really can't find a good interface (free) so I will probably write my own, which unfortunately will add additional time to the project. Might start another thread for ideas and requests.
Chris Musty
Director
Specialised Technologies
Process and standards
I can read up on TKL patch and build on top of things like lamp but is there anything in particular that is not used with TKL? eg for the web interface is there a preference over php or cgi etc
Anything else anyone wants to add?
Chris Musty
Director
Specialised Technologies
I can't speak for the core devs
But my take on it is that TKL appliances need to be as user-friendly as possible but also somewhat generic in their initial config (ie no specific hardware optimisation etc). Having said that if you were to say, tune Apache to run better with a particular web app then that would be cool.
I know the TKL guys are big Python fans, but Webmin uses Perl so if you wished to build on Webmin (a custom Webmin module?) then you would need to go the Perl route. Bottom line is that if you make something yourself, then do what works for you although don't be scared to ask for feedback early in your development, I'd especially recommend getting some from Alon and Liraz (seeing as they're the core devs).
As for software, ideally if it comes from the Ubuntu Lucid repos is preferred. If what you want isn't there, of the version is considered to be missing features, or is buggy then you can get it elsewhere. Official PPAs or alternative (Lucid) repos (eg one provided by the software maintainer) are quite good options. Otherwise installing from source is fine, although try to stick with stable software if you can. As you'd know installing from version controlled repos (eg SVN, git, etc) under heavy development can have unpredictable and unreliable results.
Look forward to see what you come up with! :)
Backup Notifications
Hi all,
As mentioned above it would be good to have some idea of what a backup is doing. I currently have an incremental backup underway but the only way I can tell is to look at the routers traffic logs. Is there a CLI command or similar I can use to tell what the server is doing? TKLHUB does not tell me an increment is in progress, the "screen" in proxmox does not tell me and I have no real idea of whats happening.
Chris Musty
Director
Specialised Technologies
Interesting ideas...
Anyhow, if you've enabled the daily backup cron job, the only way you can currently tell that it is in progress is to use the "ps" command to give you a process listing.
Incremental backup
Still testing the lasrge file system backups and something strange happened.
In my log file I get the following;
Chris Musty
Director
Specialised Technologies
Is this still a problem?
Problem Solved
Alon was right onto it and the problem seems to be gone now - the hub displays what has been backed up. For lengthy backups it would be nice to get some indication of where it is at though.
Chris Musty
Director
Specialised Technologies
Hub and TKLBAM could use the following...
I Love TKLBAM! Unfortunately I use it enough that I would love to see a few tweaks. No criticism intended (isn't there a spell checker here in the edit box - I had criticizm just a minute ago?)
Understandably many of these points are interrelated.
Supporting notes for current backup issue #1:
1: I happen to have a Domain Controller that I made the mistake of backing up when I was using it to Backup a big batch of Archive Photos. Now I have abould 30gb in storage and it looks like about 3 cycles of full backups and incremental backups took place on this server -- It is costing me 2/3 of my monthly Amazon Storage Cost.
I will likely just Delete the whole backup on Amazon but I would like to retain the Domain Controller setup. Probably safer to delete the photos, run one more Backup, then run a Restore and move on. I don't want to risk moving 30gb over the network though (as that potentially has its own cost).
I would rather remove then install tklbam.. Not sure if it will pickup where it left off or start a new backup. Simply deleting the Big Backup makes me nervous in case I also break the ability to backup the ap itself. (Caution is often unwarranted, though hard to measure if it is a waste of time)
Good feedback Arnold!
Many thanks for taking the time to flesh out your thoughts on how we can improve TKLBAM.
Items 1 and 2 are easy to implement technically, we just have to think a bit more about the implications and how to get the user experience right.
Regarding backup deletion, I'm assuming you mean you want the ability to delete a particular backup session?
If you accidentally backed up 30GB, why not just delete the backup record and start over from scratch? Do you mean that you want the ability to undo the last backup so that you don't have to start over?
In other words, if you noticed you accidentally backed up too many files, you could add them to the overrides list and then delete the bad backup sessions on the Hub. Is that it?
Item 3 I don't fully understand. I think you can turn the cron job off and on in the Webmin module. Is that what you mean?
Item 4 is technically difficult to implement and would take significant computing resources on our end, so that will probably have to wait until we get other higher priority items out of the way.
Item 5 is already supported if I understand your meaning correctly. If you backup an appliance then delete the backup record on the Hub the next backup will create a new backup record. That seems to be equivalent to clearing the backup?
Finally, regarding item 6 what would you envision happening once you toggle a backup to "archive" mode and what problem would this solve?
Add new comment