TurnKey Linux Virtual Appliance Library

Roadmap and Keeping Appliances Up-To-Date

Hi guys, I think it can benefit the project to keep an open roadmap on features being developed, just so that the community knows what comes ahead and what to wait of TKL. Also, predictable TKL updates would be a nice feature to have. 

Adding the version of the appliances and maybe defining a predictable date of update for them (maybe twice a year or every 4 months) will also be appreciated and give more value to the project. I think keeping appliances with latest releases of the software is important. It keeps support for upgrading the appliance low, and it shouldn't be much effort to modify the scripts and test the new versions. Of course I understand not all appliances are created using the TKLPatch utility. But at least I hope to keep up to date the ones I'm contributing. 

Have you ever think of this? :)

Liraz Siri's picture

Thoughts of appliance updates and the development roadmap

Regular appliance updates: this is more relevant now than it was before as an increasing number of appliances are created from upstream tarballs and we can update the version and not have to worry about breaking security updates (because you don't get them any way).

OTOH, for appliances where the primary components are installed from package management this is less of an issue because by default, we configure TurnKey to install security updates on first boot and after that auto-update on a daily basis. But even in this case it's still a good idea to refresh the images every once in a while so you don't have to download a hundred MBs worth of updates right after installation.

We can do package management refreshes more or less automatically (as we did with the April maintenance release). The part that is more difficult is refreshing the non-package management stuff. Or more precisely, making sure everything still works out of the box after you refresh them. Remember these are new versions, not backported security fixes so testing is required and we can expect things to break.

Ideally we would do that every quarter or so, but unlike Canonical we don't yet have a large staff of full-time developers so that sort of rigid schedule may not be feasible. For now we'll have to go with the Debian model of it's done when it's done. Again the main bottleneck is testing, so this is an area where the community could step in and help shoulder the burden in a big way.

Alon and I hope to accelerate that by making substantial improvements to the community development infrastructure. Imagine for example doing an upgrade manually (e.g., in the cloud or in a VM) and then being able to export the result automatically to a rough TKLPatch. We can leverage TKLBAM to do most of the heavy lifting (e.g., identifying the delta).

Bottom line, we want TurnKey to work smarter, not just harder. Instead of solving problems by brute force we can come up with creative, innovative solutions that gradually eliminate the friction that is currently preventing most of our users from contributing back to the project. This should allow the project to develop in a more decentralized fashion that empowers and leverages the community to do much more. Our goal is to support a community with hundreds of active contributors.

Development roadmap: we have a good idea of what we want to accomplish in the mid-long term and we openly share these goals with the community (e.g., client side support, full Debian support, 64-bit support images, support for more VPS and clouds). The tricky part is getting a handle on the schedule. There are many unanticipated delays and distractions. Often we're doing something for the first time. We don't know what we don't know and our estimates for how much effort is involved is off, etc.

So instead of making grandiose development roadmaps with schedules based mostly on wishful thinking we just focus our energies on the next highest priority goal and when that is out of the way move on to the next one.

In the future when the project grows enough we'll probably start exploring the pros and cons of various business models (e.g., support, partnering with hosting providers) that can financially sustain a larger Core development team. Maybe then I'll buy a buy a whip and crack it at the team whenever we fall behind schedule. :)

More on this topic

"The part that is more difficult is refreshing the non-package management stuff. Or more precisely, making sure everything still works out of the box after you refresh them" : 

At any moment, you will need to setup an automatic testing software, or at least some programmed testing with basic functional tests. This will never replace a human test, but it can help a lot, at least for the most obvious tests (verify if we get the main page of the app at the desired url, check if x port is listening, etc). I haven't found anything nice in the opensource world for testing, but I haven't looked for software like this in a long time. I did saw some commercial offerings. Maybe they can simply be programmed tests in python. You already must know what I'm talking about (BTW, I'm taking my first steps in the python/django world! looks nice)

"Imagine for example doing an upgrade manually (e.g., in the cloud or in a VM) and then being able to export the result automatically to a rough TKLPatch. We can leverage TKLBAM to do most of the heavy lifting (e.g., identifying the delta)."

I see, this can be a nice feature. Surely working on the base of the development structure will help us help you. And help everybody. HELP!

"This should allow the project to develop in a more decentralized fashion that empowers and leverages the community to do much more."

Yes, I'm amazed at how an army of two had been able to handle all that have been done in the TKL world. MMMmmm and you even have a TKLmobile... do you guys use capes and fight crime at night to get free of stress? I'll have to ask this in the corresponding post! Anyway, you got it right empowering community participation. Just be prepared to manage this participation. There are several things to put in place so that everything runs smoothly. 

"we have a good idea of what we want to accomplish in the mid-long term and we openly share these goals with the community"

Yes but the ideas get lost in the forums. If I'm a newcomer to the project, I usually look for whats the current status and where is heading. That's me of course, this is my opinion. I think its ok to go with the Debian model of it's done when it's done. But at least the roadmap could give you a clue of what's the current feature being worked. And the order in which they are coming. I know right now because I've been actively searching the forums. If you see proxmox roadmap (currently offline :S) they just have a list of features. They don't specify release dates. But you know there's a new gui coming someday! and an api! The things that they have finished working, are up. 

"Often we're doing something for the first time. We don't know what we don't know and our estimates for how much effort is involved is off, etc."

I can perfectly understand that! I've been in a constant research trying things for the first time. One does not know what's ahead. 

"In the future when the project grows enough we'll probably start exploring the pros and cons of various business models (e.g., support, partnering with hosting providers) that can financially sustain a larger Core development team. Maybe then I'll buy a buy a whip and crack it at the team whenever we fall behind schedule. :)"

Yeah, Liraz Jobs, get that appliances ready or you're fired! :) hahaha. I hope this never happens. The whip part of course! Now expanding the core team is a matter of finding the right people with the same vision. As my personal mentor Jim Rohn says, "If you're able to see the promise, it easiest to pay the price" Or something like that :P 

So ok, no schedules right now, just more organized info. And a forum link in the front page please :D!!!

Liraz Siri's picture

Quick response

Test automation: I agree that an automated testing suite will be a good investment. Even if it only checks the basics (e.g., availability of services). This needs more thought on what would be the right approach. There doesn't seem to be anything we can leverage out of the box so we'll probably have to develop a custom system that plugs into a nice virtualization system on the back-end. We'll take a look at what we can come up with that after the next release. Added to the todo list.

Managing community participation: Once community participation scales beyond a certain point we'll probably need help with this as well. It's a bit early to discuss the details but in principle it might make sense to empower our most experienced community members to help direct the community. That already happens to some extent without any formal structure. Naturally open source projects tend to be flatter and more loosely structured than their proprietary counterparts but that doesn't mean they have to be entirely flat. Also we could improve the documentation to help the unintiated.

Development wiki for development roadmap: What's your saying makes a lot of sense. The forum is a good place to have a discussion but it's not the ideal resource for newcomers who want an overview of what's going on. Perhaps the development wiki would be better suited for this.

Forum link on front page: already exists (click on the forum tab and then more) but your comments make me think that it's too subtle. I've put this on my todo list for after the release. Perhaps we could make HELP link in the toplinks a pop-up linking directly to sub resources for quick access. Would something like that satisfy the usage scenario?

Team building: finding the right people is always hard. One of the big advantages of developing things out in the open in the framework of an open source project is that you attract good people that share your vision and passion. You get to know them, they get to know you. That way when you have the resources to expand you already have a head start!

Fighting crime: I can neither confirm nor deny that we have mutant super-powers or use said unconfirmed powers to fight crime and bring justice for all. To the TKL Mobile!

Jeremy's picture

My thoughts...

Managing community participation: So far I think things trundle along pretty nicely, but I agree that some of the documentation could be better. But hey I won't say too much cause I haven't really put a lot of effort into making it better!

Development wiki for development roadmap: +1 for a Proxmox style 'Roadmap' on the Dev wiki. Good plan, good place for it. No more need be said IMO. Another (afterthought) perhaps some sort of 'news' page/link/tab that will only show strictly TKL news blog posts. Perhaps this could be done somehow using the tags that are on the blog posts?

Forum link on front page: The forum tab is nice and should perhaps stay. But I really like the idea of the HELP link (in the toplinks) being a drop down (that was what you were suggesting Liraz?)

Fighting crime: Now we just need a TKLPOW to go with the TKLBAM :)

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)