TurnKey Linux Virtual Appliance Library

For great justice: all your criticism are belong to us!

I started writing this blog post as a comment to forum discussion that really got me thinking titled "State of TurnKey from a users viewpoint". Many thanks to Eric, Jeremy and Carl for their insightful and thought provoking comments. I've decided to post my response on the blog to draw attention to the discussion. I believe it reflects what many of you in the community must be thinking and I'm hoping to hear more people voice their opinions on the matter.

Our critics are our best friends (and vice versa)

I'm humbled when people care enough about what we're doing to take the time to share their thoughts in detail. I usually don't have a lot of free time on my hands. I know most of you are like that as well so I really appreciate it when somebody goes out of there way to give us a "piece of their mind". Even when it's critical and not nearly as flattering as I might like.

Many thanks to all of you for the honest constructive criticism, as well as the help and encouragement. I believe TurnKey ultimately benefits from that.

What really pains me that I don't have the time to respond to each and every point and idea raised by everyone in the community individually. That doesn't mean a discussion, your input, feedback and ideas aren't extremely useful. You strengthen our resolve, heighten the sense of urgency and dig deeper into important paths of thought even when you aren't breaking new ground, even in cases when we have already mentally explored some of the same paths you have suggested.

Brainstorming our way to greater glory

Behind the scenes, Alon and I have these regular brainstorming sessions that often last hours. We talk about the community's feedback, throw ideas out there, discuss the pros and cons of this priority vs that and ultimately decide what we need to focus on to get TurnKey to the next level. Then we stick with it. At least until the next brainstorming session.

Our last brainstorming was about a month ago. We have a lot on our plate just keeping the metaphorical trains going at TurnKey but with regards to new developments we resolved to focus on 64-bit support, productizing a prototype of the TurnKey Desktop system, RCs for the next Debian release wheezy, and creating a TurnKey factory "meta-appliance" that would be capable of building TurnKey products from public source code (e.g., on GitHub).

Red Queen's race

"Well, in our country," said Alice, still panting a little, "you'd generally get to somewhere else — if you run very fast for a long time, as we've been doing."

"A slow sort of country!" said the Queen. "Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!"

—Through the Looking-Glass, Lewis Carroll

We can't make progress on all the items and issues raised by the community as quickly as we'd like to. We're doing our best, and in a way we've been both successful and victims of our success. So far we've managed to keep TurnKey on its feet while continuing to make progress with a ridiculously small amount of resources, but there's a price to pay...

What I don't think we fully appreciated when we began is how much time and energy it would take to keep the project in roughly the same place. Communicating with our users and partners, maintaining the project's infrastructure, tracking API changes, bugfixing, and maintaining the existing library of solutions. All of that could easily take over 100% of our time without leaving time and energy to push the project forward.

It always comes back to the resources we have to work with. What we have is never enough.

Great things come from small beginnings

A few years ago when the project was just starting out it was several orders of magnitude less complex. It was a small, fun little side project. Just a couple of products. A handful of very undemanding users. Very simple infrastructure. Not too many opportunities. A lot of ideas on where this could go, but no pressure. Nearly every user that commented on the forum got a personal reply from me. It was wonderful to receive even a trickle of interest from the outside world. People all over the outside world in fact!

TurnKey was small, not very demanding and it usually didn't feel like we were neglecting anything important.

Choose your battles

A few years later TurnKey had ballooned in size and complexity. There's now much more important stuff to get done than we could ever hope to achieve without increasing our manpower and resources several times over.

So it's become a very unfortunate fact of life that some important things are not going to get done, or not going to get done as quickly as we'd like. This means many great opportunities are lost and many great ideas and even great people are not going to get the attention they deserve from us.

We'd like to do everything, talk to everyone, go in all directions at once - then reality intervenes and forces us to pick our battles.

There's no magic way to change this, but we do believe there is a real, pragmatic path to meaningful change that doesn't involve us "selling out" and turning TurnKey into a full-fledged business backed by outside investors, with everything that comes with that (good and bad).

If you love somebody, set them free

In a nutshell, our plan is to tough it out and do whatever it takes to keep TurnKey alive and kicking while radically open up the project's development to the community. Then gradually getting various stakeholders interested in helping out and multiplying the resources the project has to work with.

In other words turning TurnKey into something more resembling a truly free open source project with a living kicking screaming community behind it bearing most of the burden involved in churning out more turnk-key open source solutions of ever higher quality.

It's a tall order, but we're in this for the long haul. Even if it takes another couple years to get there, we're fine with that.

More important than money?!

"Making money isn't hard in itself... What's hard is to earn it doing something worth devoting one's life to."

—Carlos Ruiz Zafón

We've come to understand (the hard way) how important it is to become financially sustainable and have sources of revenue. Thank goodness we can pay the rent now while working on TurnKey full-time. But this isn't about the money for us. We want TurnKey to be more of an open source project that has a little side business going than a business that has a little open source project going. As far as we're concerned the business stuff should just another piece of infrastructure that helps keep TurnKey ticking. Like our servers. It shouldn't be the focal point and it should serve the open source project rather than other way around.

We do need to grow the team and bring more full-time people on board and that requires expanding our sources of revenue. But ultimately what we want our team to be doing is supporting a rich, vibrant community that celebrates open source. Not "maximizing shareholder value". We think there's a way to do that that isn't based on wishful thinking and open source pixie dust, but only time will tell if we're right.

In the meantime if we want to hold on to our somewhat ideal, naive notions we have to stay in control. That means going against the flow, ignoring the siren call of investors and doing stuff with our own time, on our own dime. We pay for that by going slower, but in our minds that's a small price to pay for staying true to ourselves and to the open source spirit that led us to found TurnKey in the first place.

I agree wholeheartedly there's a tremendous amount of room for improvement. By all means voice your opinions, ideas and criticism. We read everything, even when we don't have time to respond.

Bear with us: from where I'm looking there are some very exciting times ahead as we lay the critical pieces in place required to unlock TurnKey's true potential. We're working on that right now. We still have some hidden aces to play that we hope will delight our users and the open source community. I'd share more details except that we've made enough public promises that I'm worried of breaking so I won't make the situation worse by making even more of those. For now let's just say we're making good progress on a lot of stuff behind the "core" event horizon and we are feeling very optimistic about TurnKey's prospects for the future. Stay tuned!

You can get future posts delivered by email or good old-fashioned RSS.
TurnKey also has a presence on Google+, Twitter and Facebook.

Comments

Came across this article by

Came across this article by chance and must admit I liked it...:-)

The trials & tribulations of good men in a mercenary world :-)

Liraz,

Let me start by saying that I have rarely met as balanced a group of people as I have on TKLX including you, Alon, and your core contributors such as Jeremy. It makes me proud to be a member.

Tssgery has raised a timely query...what is the TKLX business model (www.turnkeylinux.org/forum/general/20121016/state-turnkey-users-viewpoint) as distinct from a commercial model. We all want you and Alon to be able to earn a living at this. You deserve it and it's critical for the future of TKLX.

I don't believe you and Alon have quite nailed it down in your own minds yet and I understand the dilemma. If you fully open TKLX then it can get hijacked, if you don't open it fully the initiative may not expand as a community. You've created something special and you don't want to spoil it by involving people or paths that are purely mercenary in nature. Having said that, you realise that TKLX would benefit by investment in volunteer labour or cash invested in contracted labour. There are only 220 days in a balanced work year so the time goes quickly. A support-based business model (e.g. Redhat) is tough because it would take you away from enhancing the product as you get bogged down in contract management.

TKLX is at a critical juncture. TKLX was ahead of the curve when created and is still a leader but, without expansion over 2012-2013, it may be left behind by other cloud initiatives. The part of TKLX not opened is the build side. Other initiatives such as Chef, Puppet, and CFEngine are filling that gap so how long before TKLX members are building their own releases and life-cycle managing them via GIT / Chef like combo! That negates the need for pre-packaged appliances completely...you just press a button and build one.

TKLX currently has only one main alternative...Bitnami. I ruled them out for my business due to their locked-in nature. I'm a managed service provider and can't put my clients on an environment that could be sold out to a large S.I. firm and priced out of my target market. I have not moved yet to a GIT/Cfengine build infrastructure because I want to see where TKLX goes...but I will need to if TKLX continues on it's current path.

I sense from the TKLX blog that you and Alon are attempting to find that elusive sustainable opensource model and I and others would be more than happy to assist with some ideas. We do want you to earn a living. It's in all our best interests.

But you need to do this soon...the cloud world has moved from formation phase to expansion phase and current TKLX model may be obsolete rather quickly. That would be a shame because you have incubated what I believe could be one of the great open source initiatives.

Have a chat with Alon. I think it may be time for you to let your core community assist you in defining TKLX2013. This is bigger than a "65bit vs fewer stronger appliances".

Just one man's opinion :-)

PS: By early 2013 my startup will be in a position to contribute back to TKLX in a tangible fashion and am looking for TKLX to transition to a firm business model to ensure my investment helps to grow TKLX productively in a competitive marketplace.

I meant 64bit not 65bit...damn hp keyboard :-)

Just correcting a typo in my previous post...I meant 64bit not 65bit...damn hp keyboard :-)

Cheers,

Tim (Managing Director - OnePressTech)

Liraz Siri's picture

TurnKey's place in the ecosystem

Tim,

Thanks for your thoughts and support. I agree with you that moving more quickly will benefit TurnKey, especially on the cloud front where an important market expansion is happening, with all the opportunities that implies.

Regarding technology moving forward or the current TurnKey appliance model becoming obsolete, at least on the cloud - that is probably true. Eventually. But that's true for nearly any technical solution. Technology moves forward and a new better way often comes along of solving a given problem.

On the other hand, if a better alternative comes along we'll be happy to change our problem solving technique and provide solutions to our users with that. So as far as we're concerned we're not really competing with initiatives based on other open source approaches. Those are simply different branches of experimentation to us.

I believe TurnKey will have place in the ecosystem as long as there is a gap between what open source can deliver in potential and actually delivers in practice, as long as there is value in experts shaving down some of the rough edges for a subset of users that don't want to bleed from the very cutting edge of open source innovation, as long as there is value in a slightly more curated user experience...

But what is the 2013 TKLX place in the ecosystem?

Hi Liraz,

Curated user experience or coddled user experience?

Is the TKLX model to cater to the home amateur or the commercial enterprise...or both?

A home user / SOHO may be able to use a standard TKLX appliance for the years that elapse between TKLX releases but a commercial enterprise is unlikely to be able to do the same.

I refer to the sentiments of a more seasoned TKLX contributor Adrian Moya who says "The outdated version of applications in the appliances have slowly moved me out of using pure TKLs in my home network, where I have moved to using a plain ubuntu and installing from upstream" in his post http://www.turnkeylinux.org/blog/for-great-justice#comment-14780.

Progressive releases to multiple commercial clients necessitate a life-cycle management process over time that re-builds against three operating system releases (legacy, mature, latest) and three application releases (legacy, mature, latest) across all major hypervisors. Clients are then migrated on an as-needed basis with new clients put on the latest release. Unmodified TKLX appliances are challenged to meet commercial requirements since only one OS/App release version is maintained at any time. In my case I need 27 variants of Wordpress appliances to meet my ongoing client needs (54 if you include single-site / multi-site variations).

It is far more likely that the TKLX community can share the build-source for the Virtual Appliances than the Virtual Appliances themselves.

Let's look at a simple example for my company.

I download a TKLX Wordpress appliance. As it is currently built it is not optimally supportive of multi-site (you can do it but it's messy...Wordpress prefers multi-site to be built from root rather than a sub-folder). So lets say I download the LAMP stack and install Wordpress myself. The LAMP stack has default tuning so I then need to optimally tune the stack (OOM-killer control, Time-Wait sockets, etc). I am already starting to deviate from the TKLX plug-&-play model without even intending to. Ok...I can make a TKLX Patch to capture these variations and TKLBAM will be helpful in managing the delta. But how do I future-proof? Migrating my clients to a new TKLX release is not going to be a simple TKLBAM backup and restore exercise...APIs will have changed, features will have been phased out. And I would have 27 variations of O/S & application versions in play built against multiple hypervisors.

If the TKLX community can share the build-source for the Virtual Appliances this would be feasible. As it is, though, TKLX build source is closed forcing TKLX community members to migrate to a Chef/Puppet/AppEngine model to handle their commercial client requirements.

Any reason why the TKLX build process is not open sourced?

PS: I hope you appreciate that my post is in the spirit of positive suggestion. I think what you and the TKLX community has done to date is top notch. I just don't know what you have in mind for TKLX in 2013. If it's more of the same I expect I will be forced to move away from TKLX by then and I would rather feed back now than have you wonder in 2013 why my company moved on rather than stay and help to make TKLX grow (which is what I would like to do).

Liraz Siri's picture

TurnKey's current and future audience

Up until now, TurnKey's audience has been individual users, enthusiasts, consultants, and small business.

Large businesses do use TurnKey, but usually it's brought in through the backdoor by individual developers or small teams within the business.

I don't think we've given high-end business users enough of a reason to use TurnKey yet. That might change in the future as TurnKey gradually improves.

In any case, TurnKey is never going to be the best solution for everyone all the time. There's a very very long tail of needs and it's impossible to serve all of them. TurnKey is just another tool in the toolbox. When using another distribution, tool (e.g., Chef) or starting from scratch makes more sense (e.g., Adrian's example), users should just do that. We're not vying for global domination here.

What we do aim for is to gradually improve what TurnKey does best - which is to provide users with a good starting point for a particular use case and make common tasks easier (e.g., backup and restore) when it's technically feasible. The community can definitely help us with that so we're going in that direction.

As for why the build process isn't out there yet, I've answered that many times before but I'll answer it again. The old build process was too much of a complex and messy patchwork to be fit for public use. Reproducing it was not trivial. It would have required extensive documentation and community support and we didn't want to invest that into a dead-end. So we redesigned the build process and we're going to open source and document that instead. The last release was produced using the new build process so we're pretty close to releasing a first version. Stay tuned.

Thanks for clarifying...much appreciated :-)

Thanks Liraz. I understand now. Thanks for clarifying. I look forward to continuing the journey with TKLX and seeing where it's going and how we can help :-)

There's nothing more to add

After reading the whole forum post and this post and comments, this is just another sign of the love and hate relationship with the TKL project. I've been there as Jeremy comment in the forums.

I think the core devs understood they must open the project to collaboration, in the past we have discussed a lot in the forums and email, and also have discussed a lot of ideas about how they can capitalize a bit on their efforts.

I myself felt happy that finally some of my contribution on the gitlab and icescrum patchs got into the main distribution. I did my effort on promoting the patches development environment, I really think appliances should be created by the community and rated by community, following guides from the core devs. And I'm pretty sure there heading that way, just that in a slowly fashion.

The outdated version of applications in the appliances have slowly moved me out of using pure TKLs in my home network, where I have moved to using a plain ubuntu and installing from upstream. I have even played with chef so I think Tim has some strong points in his comments. In this matter I would say that maybe the problem is that the project is based on a custom build method. My two cents on this right now is you could build on top of puppet/chef, and this could be beneficial in two ways: builders work on chef recipes (in the case of chef) and can contribute them to the chef community, and the appliances are built using those recipes. We could have a set of base recipes that make the core appliance and then extend the core with extra recipes. Add some TKL sauce and you got your appliance. It's an idea, I haven't though a lot on it.

If you focus on the main objectives of TKL, which are a wide library, easy to use, secure, and with X Y Z features, the way they get built doesn't matter to end users. So the project could benefit on leveraging an existing and mature "patching/customization" system. Maybe this could solve the 64 bit appliances/open-sourciness of the project? And people get more interested as they will learn something not just turnkey related but a sysop skill that will help with it other nontkl work?

I think this could be easier/more extensible than learning and packing stuff. But I haven't think about upgrading, so I'm not sure this could be a solution. What I know is that there is already an ecosystem to develop such patches so no tklpatch package / build tools / closed tkl factory mantainance burden will exists. But yes, there will be a lot of work converting patches, so maybe the feeling of loosing the actual work could be a stopper for this idea.

I'll be watching the thread...

Liraz Siri's picture

How to get the community involved

Hi Adrian,

As I mentioned in the blog post we're working on packaging the TurnKey build system into a meta appliance that can be used to build TurnKey appliances from source.

We may consider using Chef in the future if the advantages are significant enough, but we're not likely to start from scratch but rather extend the current system as needed. FWIW, the redesigned build system already supports 64-bit.

Besides providing a working, self-contained TurnKey build environment, we've also been kicking around the idea of a public TKLBAM repository and maybe also a TKLBAM conversion tool to lower the bar on community involvement.

That way instead of having to learn how a potentially complex build process works, and going through a development cycle you just install an appliance, change it in some fashion, and then use TKLBAM to save the changes.

If it's a public TKLBAM backup other users can use TKLBAM to reproduce your changes. There's a potential trust problem, because a TKLBAM backup isn't necessarily very transparent, but that's something we can solve. Also a conversion tool could translate a TKLBAM backup into source code or TKLPatch.

Allowing users to rate community appliances and developers is a separate problem which we'll need to solve. Just throwing out the tools without the infrastructure will probably do a lot less good than introducing the tools within the context of a well designed community development process.

What can community users do to help?

Is there a link or somethig available that community members can view to see what opportunities there are to contribute to the project?  I've only been exposed to the turnkey software/solutions ecosystem (of what appears to be remarkably well established and polished) solutions.  I am still very new and learning the product so forgive me if this question as been answered and posted...I am still only a few hours into learning more about the great products, the community and the tremendous potential ahead.

 

I look forward to interfacing with others in this community and want to take this time to say thanks to everyone involved that's gotten the project this far. 

OpenSource strikes a blow for freedom!!!!   <grin>

 

 

Jeremy's picture

AFAIK there isn't a page that spells it out

But there is plenty to do! :)

OTTOMH (although obviously will depend on your skill level):

  • Help people out in the forums - many of the issues that people come across are generic Linux/Debian issues rather than specifically TKL ones. Even when they are specific ones, they often relate to the upstream software rather than TKL.
  • Support updates - many of the appliances incorporate upstream software - often users want to update to newer versions so if you have the know how documenting how to upgrade can be very handy for people.
  • Transfer documentation to the wiki - there is a lot of good info here in the forums but it often gets burried. The search and the tags help, but if you come across anything that should ideally be in the wiki, please feel free to copy/paste (the wiki is publicly editable). There may also be good documentation elsewhere that relates to appliances that may be worth having in the TKL wiki, or at least links to it.
  • Create new documentation - Anything that you think may be useful info for users can be documented and written up in the wiki.
  • Develop new appliances - Any software that you come across that you think should be included please feel free to document it's installation, or better still create a TKL patch.
  • Just using TKL and sharing the experience can be useful for testing and feedback.

Tiny typo

goes out of there way to give -> their

Turnkey Linux - great when it works

I question how you can even claim to support OpenVZ given that next to nothing works out of the box.

Turnkey Linux Etherpad, one of the very few counter-examples, is great on OpenVZ.

Why Zurmo, Django, otrs, vtiger, etc can't simply work properly out of the box is entirely on the TKL folks: the interactive startup scripts don't run properly on a headless system. It'd be better to give default usernames and passwords with a simple script (in root's ${HOME} perhaps) to change it. You might look at this as a step back in security, but your model doesn't fit everyone's model.

In my experience, TKL is almost impossible to deploy, except when it's not.

Note: Proxmox VEs, VMs, etc work great. You've built a good system for most solutions... but I suggest completely dropping OpenVZ appliances or fixing your core up correctly. Pick one.

Jeremy's picture

Not sure what your problem is but they work great for me! :)

Although perhaps you haven't read the documentation (posted in various places including the docs and multiple posts in the forums)?

As you correctly note (and I have written many times) as OpenVZ does not have a true console and TKL appliances require initialisation, the firstboot init scripts can't be autorun on firstboot. Thus you need to launch them manually. It pretty simple:

turnkey-init

And off you go...

In fairness to you on, on occasion (which I am yet to work out why - seems completely intermittent but the bug has been noted with the devs) the firstboot scripts still don't run correctly first time. The workaround for that though is simple - run them a second time (straight after) and all works as expected.

Ideally it'd be great if the values could be preseeded from within the PVE UI prior to setup (like the Hub does - which would eliminate the need to manually run the init scripts - they will run fine non-interactively). And perhaps you are right, that default values could be preseeded anyway.

Regardless they work well for me, so just because they don't work the way you want (or work at all for you) please don't assume that they don't work for anyone else.

Given the last "feedback"...

I felt compelled to once again say thank you to you and all those involved with making such products available to the public.  I am a bit new to Linux and many of your platforms and at times struggle with some basic tasks, however it's pretty AMAZING all of the great work you folks have done and continue to do!

 

So again thank you...because of your hard work people have more "options" than they otherwise would.

Idea

Here is an idea for you.

You need to gain momentum ? Media Press ? Buzz ?

Samba 4 was just release some days back with a great news : it is ActiveDirectory compatible !!!

"All you need to do" is to update http://www.turnkeylinux.org/domain-controller so that it will use Samba 4 and provide :

A full working Microsoft Active Directory replacement !

If you do then a PR saying : "Free drop-in replacement for Active Directory", be sure lots of buzz will start in the enteprise world ;-)

 

Rgs,
TM

Samba4...AD compatible solution!

Thanks for posting... this is great news!  I agree with the poster!  GET THE WORD OUT!!!!

Jeremy's picture

Security concerns

Just wanted to post a link to a thread where some TKL security risks were discussed. The discussion was initially a little heated but we got there in the end (I hope!)

I post this here more for the benefit of Alon and Liraz (so they can easily find it if they are evalutating feedback at some point) than anything else...

<3 tkl ----> Add a 'Donate' button to the front page!

... seriously, that way tkl can get some extra resources with few strings attached...

 

I've got $10 I'd donate through paypal or some such...

 

Whats to lose?

 

(also, thanks so much for your work, and your ethic)

 

cheers

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)