From the project
For great justice: all your criticism are belong to us!
I started writing this blog post as a comment to forum discussion that really got me thinking titled "State of TurnKey from a users viewpoint". Many thanks to Eric, Jeremy and Carl for their insightful and thought provoking comments. I've decided to post my response on the blog to draw attention to the discussion. I believe it reflects what many of you in the community must be thinking and I'm hoping to hear more people voice their opinions on the matter.
Our critics are our best friends (and vice versa)
I'm humbled when people care enough about what we're doing to take the time to share their thoughts in detail. I usually don't have a lot of free time on my hands. I know most of you are like that as well so I really appreciate it when somebody goes out of there way to give us a "piece of their mind". Even when it's critical and not nearly as flattering as I might like.
Many thanks to all of you for the honest constructive criticism, as well as the help and encouragement. I believe TurnKey ultimately benefits from that.
What really pains me that I don't have the time to respond to each and every point and idea raised by everyone in the community individually. That doesn't mean a discussion, your input, feedback and ideas aren't extremely useful. You strengthen our resolve, heighten the sense of urgency and dig deeper into important paths of thought even when you aren't breaking new ground, even in cases when we have already mentally explored some of the same paths you have suggested.
Brainstorming our way to greater glory
Behind the scenes, Alon and I have these regular brainstorming sessions that often last hours. We talk about the community's feedback, throw ideas out there, discuss the pros and cons of this priority vs that and ultimately decide what we need to focus on to get TurnKey to the next level. Then we stick with it. At least until the next brainstorming session.
Our last brainstorming was about a month ago. We have a lot on our plate just keeping the metaphorical trains going at TurnKey but with regards to new developments we resolved to focus on 64-bit support, productizing a prototype of the TurnKey Desktop system, RCs for the next Debian release wheezy, and creating a TurnKey factory "meta-appliance" that would be capable of building TurnKey products from public source code (e.g., on GitHub).
Red Queen's race
"Well, in our country," said Alice, still panting a little, "you'd generally get to somewhere else — if you run very fast for a long time, as we've been doing."
"A slow sort of country!" said the Queen. "Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!"
—Through the Looking-Glass, Lewis Carroll
We can't make progress on all the items and issues raised by the community as quickly as we'd like to. We're doing our best, and in a way we've been both successful and victims of our success. So far we've managed to keep TurnKey on its feet while continuing to make progress with a ridiculously small amount of resources, but there's a price to pay...
What I don't think we fully appreciated when we began is how much time and energy it would take to keep the project in roughly the same place. Communicating with our users and partners, maintaining the project's infrastructure, tracking API changes, bugfixing, and maintaining the existing library of solutions. All of that could easily take over 100% of our time without leaving time and energy to push the project forward.
It always comes back to the resources we have to work with. What we have is never enough.
Great things come from small beginnings
A few years ago when the project was just starting out it was several orders of magnitude less complex. It was a small, fun little side project. Just a couple of products. A handful of very undemanding users. Very simple infrastructure. Not too many opportunities. A lot of ideas on where this could go, but no pressure. Nearly every user that commented on the forum got a personal reply from me. It was wonderful to receive even a trickle of interest from the outside world. People all over the outside world in fact!
TurnKey was small, not very demanding and it usually didn't feel like we were neglecting anything important.
Choose your battles
A few years later TurnKey had ballooned in size and complexity. There's now much more important stuff to get done than we could ever hope to achieve without increasing our manpower and resources several times over.
So it's become a very unfortunate fact of life that some important things are not going to get done, or not going to get done as quickly as we'd like. This means many great opportunities are lost and many great ideas and even great people are not going to get the attention they deserve from us.
We'd like to do everything, talk to everyone, go in all directions at once - then reality intervenes and forces us to pick our battles.
There's no magic way to change this, but we do believe there is a real, pragmatic path to meaningful change that doesn't involve us "selling out" and turning TurnKey into a full-fledged business backed by outside investors, with everything that comes with that (good and bad).
If you love somebody, set them free
In a nutshell, our plan is to tough it out and do whatever it takes to keep TurnKey alive and kicking while radically open up the project's development to the community. Then gradually getting various stakeholders interested in helping out and multiplying the resources the project has to work with.
In other words turning TurnKey into something more resembling a truly free open source project with a living kicking screaming community behind it bearing most of the burden involved in churning out more turnk-key open source solutions of ever higher quality.
It's a tall order, but we're in this for the long haul. Even if it takes another couple years to get there, we're fine with that.
More important than money?!
"Making money isn't hard in itself... What's hard is to earn it doing something worth devoting one's life to."
—Carlos Ruiz Zafón
We've come to understand (the hard way) how important it is to become financially sustainable and have sources of revenue. Thank goodness we can pay the rent now while working on TurnKey full-time. But this isn't about the money for us. We want TurnKey to be more of an open source project that has a little side business going than a business that has a little open source project going. As far as we're concerned the business stuff should just another piece of infrastructure that helps keep TurnKey ticking. Like our servers. It shouldn't be the focal point and it should serve the open source project rather than other way around.
We do need to grow the team and bring more full-time people on board and that requires expanding our sources of revenue. But ultimately what we want our team to be doing is supporting a rich, vibrant community that celebrates open source. Not "maximizing shareholder value". We think there's a way to do that that isn't based on wishful thinking and open source pixie dust, but only time will tell if we're right.
In the meantime if we want to hold on to our somewhat ideal, naive notions we have to stay in control. That means going against the flow, ignoring the siren call of investors and doing stuff with our own time, on our own dime. We pay for that by going slower, but in our minds that's a small price to pay for staying true to ourselves and to the open source spirit that led us to found TurnKey in the first place.
I agree wholeheartedly there's a tremendous amount of room for improvement. By all means voice your opinions, ideas and criticism. We read everything, even when we don't have time to respond.
Bear with us: from where I'm looking there are some very exciting times ahead as we lay the critical pieces in place required to unlock TurnKey's true potential. We're working on that right now. We still have some hidden aces to play that we hope will delight our users and the open source community. I'd share more details except that we've made enough public promises that I'm worried of breaking so I won't make the situation worse by making even more of those. For now let's just say we're making good progress on a lot of stuff behind the "core" event horizon and we are feeling very optimistic about TurnKey's prospects for the future. Stay tuned!
Comments (8)
I meant 64bit not 65bit...damn hp keyboard :-)
Reply 1TurnKey's current and future audience
Reply 2Up until now, TurnKey's audience has been individual users, enthusiasts, consultants, and small business.
Large businesses do use TurnKey, but usually it's brought in through the backdoor by individual developers or small teams within the business.
I don't think we've given high-end business users enough of a reason to use TurnKey yet. That might change in the future as TurnKey gradually improves.
In any case, TurnKey is never going to be the best solution for everyone all the time. There's a very very long tail of needs and it's impossible to serve all of them. TurnKey is just another tool in the toolbox. When using another distribution, tool (e.g., Chef) or starting from scratch makes more sense (e.g., Adrian's example), users should just do that. We're not vying for global domination here.
What we do aim for is to gradually improve what TurnKey does best - which is to provide users with a good starting point for a particular use case and make common tasks easier (e.g., backup and restore) when it's technically feasible. The community can definitely help us with that so we're going in that direction.
As for why the build process isn't out there yet, I've answered that many times before but I'll answer it again. The old build process was too much of a complex and messy patchwork to be fit for public use. Reproducing it was not trivial. It would have required extensive documentation and community support and we didn't want to invest that into a dead-end. So we redesigned the build process and we're going to open source and document that instead. The last release was produced using the new build process so we're pretty close to releasing a first version. Stay tuned.
TurnKey's place in the ecosystem
Reply 3Tim,
Thanks for your thoughts and support. I agree with you that moving more quickly will benefit TurnKey, especially on the cloud front where an important market expansion is happening, with all the opportunities that implies.
Regarding technology moving forward or the current TurnKey appliance model becoming obsolete, at least on the cloud - that is probably true. Eventually. But that's true for nearly any technical solution. Technology moves forward and a new better way often comes along of solving a given problem.
On the other hand, if a better alternative comes along we'll be happy to change our problem solving technique and provide solutions to our users with that. So as far as we're concerned we're not really competing with initiatives based on other open source approaches. Those are simply different branches of experimentation to us.
I believe TurnKey will have place in the ecosystem as long as there is a gap between what open source can deliver in potential and actually delivers in practice, as long as there is value in experts shaving down some of the rough edges for a subset of users that don't want to bleed from the very cutting edge of open source innovation, as long as there is value in a slightly more curated user experience...
How to get the community involved
Reply 4Hi Adrian,
As I mentioned in the blog post we're working on packaging the TurnKey build system into a meta appliance that can be used to build TurnKey appliances from source.
We may consider using Chef in the future if the advantages are significant enough, but we're not likely to start from scratch but rather extend the current system as needed. FWIW, the redesigned build system already supports 64-bit.
Besides providing a working, self-contained TurnKey build environment, we've also been kicking around the idea of a public TKLBAM repository and maybe also a TKLBAM conversion tool to lower the bar on community involvement.
That way instead of having to learn how a potentially complex build process works, and going through a development cycle you just install an appliance, change it in some fashion, and then use TKLBAM to save the changes.
If it's a public TKLBAM backup other users can use TKLBAM to reproduce your changes. There's a potential trust problem, because a TKLBAM backup isn't necessarily very transparent, but that's something we can solve. Also a conversion tool could translate a TKLBAM backup into source code or TKLPatch.
Allowing users to rate community appliances and developers is a separate problem which we'll need to solve. Just throwing out the tools without the infrastructure will probably do a lot less good than introducing the tools within the context of a well designed community development process.
There's nothing more to add
Reply 5After reading the whole forum post and this post and comments, this is just another sign of the love and hate relationship with the TKL project. I've been there as Jeremy comment in the forums.
I think the core devs understood they must open the project to collaboration, in the past we have discussed a lot in the forums and email, and also have discussed a lot of ideas about how they can capitalize a bit on their efforts.
I myself felt happy that finally some of my contribution on the gitlab and icescrum patchs got into the main distribution. I did my effort on promoting the patches development environment, I really think appliances should be created by the community and rated by community, following guides from the core devs. And I'm pretty sure there heading that way, just that in a slowly fashion.
The outdated version of applications in the appliances have slowly moved me out of using pure TKLs in my home network, where I have moved to using a plain ubuntu and installing from upstream. I have even played with chef so I think Tim has some strong points in his comments. In this matter I would say that maybe the problem is that the project is based on a custom build method. My two cents on this right now is you could build on top of puppet/chef, and this could be beneficial in two ways: builders work on chef recipes (in the case of chef) and can contribute them to the chef community, and the appliances are built using those recipes. We could have a set of base recipes that make the core appliance and then extend the core with extra recipes. Add some TKL sauce and you got your appliance. It's an idea, I haven't though a lot on it.
If you focus on the main objectives of TKL, which are a wide library, easy to use, secure, and with X Y Z features, the way they get built doesn't matter to end users. So the project could benefit on leveraging an existing and mature "patching/customization" system. Maybe this could solve the 64 bit appliances/open-sourciness of the project? And people get more interested as they will learn something not just turnkey related but a sysop skill that will help with it other nontkl work?
I think this could be easier/more extensible than learning and packing stuff. But I haven't think about upgrading, so I'm not sure this could be a solution. What I know is that there is already an ecosystem to develop such patches so no tklpatch package / build tools / closed tkl factory mantainance burden will exists. But yes, there will be a lot of work converting patches, so maybe the feeling of loosing the actual work could be a stopper for this idea.
I'll be watching the thread...
AFAIK there isn't a page that spells it out
Reply 6But there is plenty to do! :)
OTTOMH (although obviously will depend on your skill level):
- Help people out in the forums - many of the issues that people come across are generic Linux/Debian issues rather than specifically TKL ones. Even when they are specific ones, they often relate to the upstream software rather than TKL.
- Support updates - many of the appliances incorporate upstream software - often users want to update to newer versions so if you have the know how documenting how to upgrade can be very handy for people.
- Transfer documentation to the wiki - there is a lot of good info here in the forums but it often gets burried. The search and the tags help, but if you come across anything that should ideally be in the wiki, please feel free to copy/paste (the wiki is publicly editable). There may also be good documentation elsewhere that relates to appliances that may be worth having in the TKL wiki, or at least links to it.
- Create new documentation - Anything that you think may be useful info for users can be documented and written up in the wiki.
- Develop new appliances - Any software that you come across that you think should be included please feel free to document it's installation, or better still create a TKL patch.
- Just using TKL and sharing the experience can be useful for testing and feedback.
Not sure what your problem is but they work great for me! :)
Reply 7Although perhaps you haven't read the documentation (posted in various places including the docs and multiple posts in the forums)?
As you correctly note (and I have written many times) as OpenVZ does not have a true console and TKL appliances require initialisation, the firstboot init scripts can't be autorun on firstboot. Thus you need to launch them manually. It pretty simple:
turnkey-init
And off you go...
In fairness to you on, on occasion (which I am yet to work out why - seems completely intermittent but the bug has been noted with the devs) the firstboot scripts still don't run correctly first time. The workaround for that though is simple - run them a second time (straight after) and all works as expected.
Ideally it'd be great if the values could be preseeded from within the PVE UI prior to setup (like the Hub does - which would eliminate the need to manually run the init scripts - they will run fine non-interactively). And perhaps you are right, that default values could be preseeded anyway.
Regardless they work well for me, so just because they don't work the way you want (or work at all for you) please don't assume that they don't work for anyone else.
Security concerns
Reply 8Just wanted to post a link to a thread where some TKL security risks were discussed. The discussion was initially a little heated but we got there in the end (I hope!)
I post this here more for the benefit of Alon and Liraz (so they can easily find it if they are evalutating feedback at some point) than anything else...
Just correcting a typo in my previous post...I meant 64bit not 65bit...damn hp keyboard :-)