TurnKey Linux Virtual Appliance Library

TKL web configurator

Hi everyone, I'd like to post here some ideas for the TKLs and the community. So this will be the first of various post. First some suggestions for the website:

- A link in the front page to the forums. I know there's a widget there but I really miss a button in the primary nav. (left to Blog link)

- Are you guys going to relaunch the theme with a bit of purple and less brown? ;)

About TKLs:

- Web configurator: Working on the openldap appliance (which needs you to ssh to the server and run a config script), I was thinking if you ever thought of a web page configurator instead of the current confconsole. It's just an idea. Maybe this kind of appliances can run and by default have a /setup/ page where the user is presented a way to configure his appliance, without the need to ssh to the server. It can serve as a way to stablish initial passwords, set hostname, etc in an easy manner. I think maybe webmin was put into the TKLs to that end, or to what end? (I've havent used webmin never using TKL, so I would like to know how others use this interface). But for a task like changing the hostname, or setting a fqdn, it would be nice to just nav to setup, login and change your settings. Maybe a "micro framework" can be used by developers to add actions and configuration options there. 

- Have you think about a default monitoring job running to get info about the services in the appliance? Just thinking for the future of a service like Hub (which I haven't tested yet) to have monitoring tool. Or using something like puppet or chef, to serve as backend configurator and monitoring tool. 

Well these are justs ideas, I would like the community to step in and comment, I think we all wan't the same: easy deployment of infraestructure. 

Other setting

Other setting for the web configurator could be the timezone for the appliance. So that times/dates of the apps reflect the timezone of the users of the appliance. Again, maybe this is easily done with webmin, I'll take a look at it when I get the time. I'm testing replication right now ;)

Liraz Siri's picture

It should be possible to extend Webmin

Web configuration: You should give Webmin a try. It's under active development and it is fairly easy to extend with additional modules. It should be possible for us to extend Webmin to serve the usage scenario you describe: a single easy to access web wizard that allows you to set up the important stuff. The only thing that sucks about Webmin is that it's written in Perl which puts me off these days. 

Ideally we should look into figuring out a way to automatically "Webminize" the di-live.d hooks so that the same underlying logic would control console configuration screens and the Web configuration wizard.

Console configuration in Lucid: in the Hardy releases, you would only get a configuration wizard (for stuff like passwords) if you installed through the ISO. For the Lucid releases we're going to be separating the configuration stuff so that it will run instead on first boot. That way it will be available in the VM builds as well.

Appliance configuration scripts: if possible a di-live hook would be a better way to configure your application (e.g., ldap) than a script in /usr/local/bin. Di-live hooks go into /usr/lib/di-live.d. We implemented appliance-specific hooks in the domain-controller, ejabberd, projectpier and torrentserver appliances. You might want to look at that for reference.

Distributed configuration:  Puppet and Chef have significant learning curves, but it can be worth it if you're a sysadmin in charge of a large number of machines. I've played with Puppet though I wouldn't call myself an expert. If there's a significant demand for it I'm not opposed to integrating one of them into TurnKey. I believe the easy things should be easy and the hard things possible.

Monitoring: the trouble with providing out of the box monitoring is that it's resource intensive. Imagine if many thousands of TurnKey installation all over the world are all updating the Hub every few minutes regarding the status of their services. To process, store and do something useful with all of that data you need significant computing resources. If it was free like the rest of TurnKey scaling and maintaining that infrastructure could get expensive for us quickly. So it's something we need to look into carefully. I'm not ruling it out, it just needs more thought.

The only thing that sucks

The only thing that sucks about Webmin is that it's written in Perl which puts me off these days. 

No ponys in the perl world? ;) I haven't take a look at perl but I know a friend who is very perlish. 

Ideally we should look into figuring out a way to automatically "Webminize" the di-live.d hooks so that the same underlying logic would control console configuration screens and the Web configuration wizard.

Could you elaborate a little more? So that I can understand fully the requirement?

For the Lucid releases we're going to be separating the configuration stuff so that it will run instead on first boot.

That makes sense. Did you try in virtualbox the issue of not having keyboard input on boot? 

 

Appliance configuration scripts: I'll take a look at one of those appliances. I still don't get the di-live hooks subject. I think you mentioned it in the training. I'll find some spare time to study a bit. I'm really enjoying learning new things while working for this project!

 

Distributed configuration: One of the advantages thinking in the future is that when TKL Master appliance is out, it can use a system like those as a base. We could create custom recipes for configuration, and the Master controller can easily send commands to config things etc etc. Of course this is one use for this, there could be other uses, and OTOH, maybe you would never use it like this and you have resources wasted in your VM. I myself find it usefull to build applications like TKLMaster on top of it instead of reinventing the wheel. It can be more easy to develop new recipes for future appliances. I'm not an expert on puppet or chef niether, but I had to manage an increasing number of servers and I know that if I don't learn something like that I would die trying to keep all things under control. Of course these are just ideas I'm writing without much thinking, I don't know where you guys are heading the project to (I'd like to!), I think I asked the question in the interviews post. 

 

Monitoring: I wasn't thinking of the appliances updating status to a central Hub. Just that maybe there could be a Monitoring appliance (I think you have Nagios created) that it was easy to configure your own instalations to update the monitoring appliance. Something like ubuntu's landscape but in a local environment. Maybe I'm just going crazy, hehe. 

 

I think the most important issue here is: What kind of help do you guys need to achieve the Milestone of releasing TKL 10.04? If you post your ideas, community can step in and help in those places where it's most needed.

Other suggestion: Have you think of a ticket system for TKL community? I think the forums are nice but not the more efficient way of handling support requests. I also suggest that the General forum could be split as it is very General. You could add Off-topics and TKLPatch releases for example.

Please remember these are just suggestions, and I recognize the amazing work you both have done in this project. So please don't take this as demands or complains, you're free to say "NO, YOU WON'T HAVE A PONY". 

Liraz Siri's picture

TurnKey configuration, monitoring, upcoming release, tickets...

Perl: You got me. The real reason I've been put off Perl is their mascot - an ugly camel. Not much of a contest in the looks department with a pony. Seriously though, I used to be a huge fan of Perl a few years back. But gradually I grew more and more dissatisfied with how ugly the language was. Quite often I literally had to painstakingly decipher code I had written just a few months back. It was frustrating. You can write bad code in any language but with Perl it was much too easy. Eventually I started looking for alternatives. I learned  Python in a couple of days and was shocked to discover that I was immediately more productive in it than I was in Perl despite years of experience. So after that I said goodbye to Perl with a final sick joke: a free Perl obfuscation web service.

Configuration: Let me explain how we got to where we are. In the beginning TurnKey didn't have VM builds. There was only the ISO build. It had an installer (di-live) which could ask you questions after installation to configure your system (e.g., root password, locale, hostname, etc.). In the last major release batch we added the VM builds in the last minute. Since we skipped the installer, we also skipped the configuration screens. We'd like to change this in the next release so that the configuration screens run at firstboot. This way they work for both the ISO and the VM builds.

But now that TurnKey has evolved we have a new problem - cloud deployment. In a cloud deployment type usage scenario you provision your appliance on a remote machine and you can't rely on having console access. So it doesn't matter that we ask you a bunch of questions on firstboot.

What I meant by "webminizing" the configuration screens was to figure out how to let you access them from Webmin. This is just an idea though. We haven't yet taken a look at what would be required to translate the console-based configuration screens to something Webmin can work with...

Distributed configuration: Puppet/chef and friends are most useful if you're managing a large number of semi-identical machines (e.g., an array of application servers, a mirror network, a file array). If each machine is different, distributed configuration tools will only help you with common tasks (e.g., changing a password on all machines, or adding an SSH key). It's not a magic genie that does what you want. It's an amplifier. Testing is even more essential because if you're messing around with the configuration of a large batch of machines you better know what's going to happen in advance.

I'm not that enthusiastic about the recipe approach for appliance development because it seems to me that just complicates the development/testing cycle by adding another layer of indirection.

But for some common tasks (e.g., password management, package management, backups, etc.) a distributed configuration tool could be very useful in the future. Basically anything you would want to do at the Core level of an appliance.

I suspect most of TurnKey's user base is still running just a handful of appliances right now. Support for large scale configuration tools is more of a  future need then something users are asking for right now. I'd rather wait a while longer and get a really good handle on the common problems we need to solve than design a solution in anticipation of a future need. In the future we will know more about what needs to be done, so let the future take care of itself.

Monitoring: a monitoring appliance would be a pretty neat idea. I've setup Nagios and Cacti before so I know what that's like. What I don't like about the current monitoring solutions is that they're complicated to setup. We can make installation TurnKey but the real difficulty is in configuring everything afterwards to work with your monitoring infrastructure. It's not really turn-key at all but highly customized. Perhaps in the future we can figure out how to make this stupidly easy (like we did with backups for TKLBAM). That would be neat.

Upcoming release: mostly we just needed TKLBAM out the door so we could refocus our efforts on the next release batch. There are some things only we can do right now because our internal development infrastructure, while powerful is also a complicated and messy patchwork of custom repositories, build machines, and ad-hoc glue scripts. For it to be useful to the community we would need to clean it up and document how things work. Otherwise it won't work for anyone and I'll be spending way too much of my time in open source support hell. I suspect it would probably be easier to just redesign a new system from scratch now that we have more experience. TKLPatch was our first attempt at designing something new that the community could use. A good patch saves most of the work of adding a new appliance to the library.

Release schedule: Right now we have a very clear, aggressive plan for pushing out betas of the next release within a few weeks. We'll probably split this up into two parts. The first part will be the existing appliances. The second part, the new community-developed appliances.

Support tickets and web site improvements: Launchpad bug reports are the closest thing we have to a ticket tracking system. You can subscribe to bugs, get updates on their status, etc. Both you and JedMeister have raised some good points and interesting ideas. Reconsidering the community toolbox to accomodate the expansion of the project is something I have on my personal todo list. But that's another thing we'll only get to after the release.

As the unofficial webmaster for TurnKey my philosophy is to try to keep things as simple as possible in terms of usability - but no simpler. There is also an implementation cost that needs to be considered. Behinds the scenes the site is already complex. We're already using over 70 add-on modules. I kid you not! It's gotten a bit hairy since I took over web site maintenance.

Alon actually designed the first version of the site two years ago based on Drupal 5. Then about a year later I stepped in and converted the site to Drupal 6 while redesigning a bunch of stuff. You might remember that seeing as how you've been around for a while...

Jeremy's picture

Can't help myself...

Its almost off topic (sorry Adrian) but it was nice reading about the TKL site on drupal.org.

One thing that jumped out at me though (after following a link you posted there) was how damn UGLY the new 'Ubuntu' community theme looks. I guess in fairness it may just be a poor screenshot of poor examples but compared to the old one (which TKL uses currently) it doesn't come close IMO.

While I must admit the current one could loose the brown (now Ubuntu have moved away from the brown it dates the site a little) I think that otherwise its very nice and clean and professional looking (all the things you want in a web theme). OTOH the new one looks like something that I bashed out in about 2 minutes using some minor html hacks on a generic template I found randomly on google (actually it doesn't look quite that bad but I reckon you get that I don't like it much).

Liraz Siri's picture

Random idea: new TKL web site design contest?

I'm not that much of a web designer myself so when I get things right it's usually by brute force (e.g., trying all possible combinations). I'm also color blind which further complicates my design explorations. Usually I have to resort to measuring RGB values within GIMP! That's how I colored the logo on the top left to match the web site, which erhm... I hope it does but come to mention it I'm not really sure.

So I'd rather someone with real design skills work on any significant changes to the web site's design. Not sure how we would find that special someone but maybe at some point we might try a TKL web site redesign contest. Who knows, it might even work! Maybe it will be best to let donations pile up again so we have a little bonus to offer (in addition to the fame and glory).

Worst case scenario we stay with this design. It ain't so bad. Gulp (I hope).

Jeremy's picture

Now you mention it...

The TKL logo (top left) does have a little more red/orange hue to it than the brown background. But it can't be too bad as I have never noticed it for the last year and a half (until you mentioned it just then). Doing design stuff like that while being colour blind must be a challenge!

As for the sites current look: I really like it and I don't think its to bad at all, and as I'm not much of a web designer either I certainly won't complain. I have art/creativitiy deficit! OTOH though whilst I'm not good at creative design ideas, or even when I do, turning them into reality doesn't really work - regardless I do know what I like and what I don't! If I were to have any complaint about the current site appearance it's only the brownness of it. Like I said above, now that Ubuntu has moved away from the brown it sort of dates the overall look a little. Still if its good enough for Ubuntu forums, then surely its good enough for TKL for now!

I think the idea of a TKL Web design comp down the track at some point is a really good one!

I'm no designer either

But I can see some flaws on the site. It's a bit messy. The brown, as Jed suggest, makes the site a bit dated. Here are some comments without giving a deep examination of the site:

The main page opens with a big banner. Then you have the benefits/about box. Then you have a big testimonial. On the right you see the latests blogs post (I like it) and the semi-hidden forum tab which I feel a bit not useful (It's a very small and vertical space with latest post but you can look for what you're really interested there (more on the forums ahead). Main navigation shows CLOUD DEPLOYMENT, SCREENSHOTS, HELP, FAQ (I'd remove this one and place it inside help) and BLOG. So by default, if I don't scroll down (and I'm currently using a 4/3 monitor at my job, not a WS laptop monitor) I can't see anything related to the appliances themselves! The first main nav link takes me to another site! (The turnkey hub site!) So I'd vote for a re-arrangement of the site navigation and widgets. Main navigation should have an Appliances (or Library) link. A forums link also will get you immediately were the action is. The wiki is also a bit hidden. Maybe because the wiki is some drupal plugin? The dev wiki is only accessible by a link in the Development page inside the article! 

Appliances pages: here we have the main appliance info. And a news feed (nice). But besides that, nothing more about his appliance. The new design could be more appliances-centric. Each page should have the actual info, but also blog posts related to the appliance (maybe filtered by tags), links to the docs on how to install/secure your appliance, and specific documentation for the appliance. A link to the wiki page of the appliance. Maybe screencasts. I think you, as a guest browsing the site, should have everything at hand. The less clicks I have to do to get the info I'm looking for, the happier I'll be browsing the site. For example, click on any appliance icon. Now, try to go to the forums. There's no way! you have to go to the main screen, hit the tab, and click on more to get there (3 clicks). Or you can always hit help first (maybe you're not looking for help, but just want to see what's going on). Then click Forums. (2 clicks away in a not so intuitive path). Got my ideas?

Forums: The forums deserve some care. I think here is where the action really is. You can KISS them. But a General forum-for-all is starting to become an over-simplistic approach. Also, you can think on using a real Forums software (There are some good TKL appliances you can try ;)) It's like I recommended earlier with the support. Maybe a ticket system with knowledge database can be a better way to later delegate administration of these areas. 

Well, these are my first impressions. As I'm no designer, maybe there are things that escape my eye, and maybe my comments suggests wrong solutions. But these are my 2 cents! 

As redesigning a site can be a daunting task, you'll surely need help. I can lay you a hand with things not related to graphic design, colors, and the such. Add some purple is all I can say hahaha. But If you organize a brainstorming with community I can throw more ideas there and help with some mockups. 

Sorry if I've been hard at the site! We can always stay with this design that its.... ahhhh... not so bad :D

Growing interest in Puppet

I just want to add that since this post was started that there's been a lot of development in the puppet community.

There's a lot of material about this for Drupal as there are for other applications:

http://projects.puppetlabs.com/projects/1/wiki/Drupal_Patterns

http://evolvingweb.ca/story/controlling-your-cloud-puppet

http://groups.drupal.org/devops

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)