What if all Debian/Ubuntu based dists used TKLDev?

Imagine if every Debian distribution in the world in the world was using TurnKey's build chain and collaborating with us on its development instead of limping along with various inefficient ad-hoc tools?

TurnKey currently doesn't get as many contributions back from the free software community as we'd like (our fault). I think there are two main reasons for that:

  1. We too started out with an ad-hoc development process and it took us too long to clean up our act, rework everything and fully open up the development process with TKLDev.
  2. We haven't done enough targeted outreach

I'd love for that to change and I think promoting more people to use TKLDev is key to that. It would be really good for TurnKey, good for the Debian community, and good for free software innovation.

So far we've been thinking mainly of convincing people who would never otherwise develop a Linux distribution to use TKLDev to do that, but isn't that a much harder sell than say ... the people who are already developing Linux distributions?

We wouldn't need to convince them to do something they haven't done before but rather improve the way they're doing something that they are already motivated/committed to doing.

So I think it would be a really good, high-leverage idea to compile a list of contacts for all Debian/Ubuntu based dists and get a dialogue going with the intention of educating them to the benefits of using TurnKey's 100% free software build system and encouraging them to collaborate with TurnKey.

I mean, what better way to bolster TurnKey's standing in the community than by helping out the many free software projects who are already developing Debian based distributions and share our values?

It will be a win-win. TurnKey gets exposed to more developers, increases its prestige in the community and increasing the likelihood of getting contributions back.

Developers accelerate development of their own projects and maybe join forces on some level and contribute improvements back to TurnKey. I bet most of those we convince to try it will realize it is better than whatever they are currently using, especially once we add in a few more specialized features such as deterministic builds.

What do you guys think?

Comments

Adam powell's picture

I support it! I think you have done an amazing job, but yes there are some limitations that prevent the community from engaging source contibutions . I think opening that up would solve that, also some basic documentation for understanding how add on instances are installed. For example john draper and his problems with the django turnkey. He doesn't know where to seek help for starting to use the appliance. While I think the appliance is perfect from the perspective of a systems administrator, it might not be as ready to dive into from the perspective of a developer. I think that if a simple user forum link was built into the web administrative page for the appliance it could help solve some of these problems without users seeking comment posts relative to answers for info related to the appliance. Let me know if I can help out, I really love turnkey and have been posting comments and lurking for awhile. Thanks!!!

 

facebook.com/adamkpowell

Liraz Siri's picture

Thanks for the feedback Adam. I appreciate it more than you know. My last post didn't get quite the response I was hoping for. Guess that's my fault for not posting regularly.

One thing I disagree with is that TurnKey solutions are just about perfect from a sysadmin standpoint. That is far from true, but we have some ideas for improving some of the major issues in the next version. That is another area we'd like to get feedback on.

Anyhow, Alon helped solve the issue Cap'N Crunch was having with shellinabox yesterday so it all worked out. Pretty wild that a legend like that is using TurnKey. I grew up reading stories about John and even met up with him about 14 years ago when  he came to Israel for the Y2Hack conference. That was a formative experience. If you've ever met John you'll know what I mean.

 

Anyhow, we're open to ideas on how to better engage with developers. Do you really think people are having trouble finding the forum? Because in that case maybe we should go a step further and redesign the help page or the site navigation.

What more do you think we could do to get people more engaged with TurnKey?

 

John Carver's picture

It was the 1971 Esquire article about John Draper aka Cap'N Crunch that inspired me to change careers.  At the time, I was a nuclear reactor operator aboard the USS George Washington SSBN 598.  Running a reactor on an FBM on patrol gives you a lot of time to think.  The dangers of nuclear power were just starting to get attention back then and we had to make changes, like stop dumping our radioactive waste at sea.  Anyway the article about Cap'N Crunch made me realize that I knew nothing about how the telephone system worked but that it was very interesting.  

When my enlistment came to an end, I returned to the University of Colorado and got my BSEE with emphasis on communication theory and computer architecture.  My first job after college was working for Collins Radio (now Rockwell Collins) who was in the process of building one of the first digital telephone switches.  My first assignment was to design TouchTone (DTMF) transmitters and receivers for the digital switch.  Believe it or not, when I arrived they were still using the computer to count dial pulses.  My next project was to design the multifrequency (MF) transmitters and receivers so our switch would be compatible with Ma Bell's equipment. MF is the system Cap'N Crunch exploited with his famous BlueBox.

You could say that Cap'N Crunch changed my life.

Information is free, knowledge is acquired, but wisdom is earned.

Landis Arnold's picture

Somewhat like a turtle I have been progressing towards using TKLDEV.  I have a machine installed.  I recall that I got through all the dependencies (but need to check).  I have built at least one GitHub Branch of a new App that I want to try to build.

Right now I am asking myself, "now where are all those checklists"?  Yesterday I even loaded a Ap Image of MediaWiki so that I can start building some of those lists...  (sidetracked by the question of whether I should use Dokuwiki, MoinMoin, Twiki, Mediawiki or something else....)

My process next will be to Look again at the TKLDEV documentation.  Look at the Various Blog posts which have "hands on tips".  Look at any readme's in GitHub.

I used to be a ski jumper and really where I am is I have my skis at the top of the jump and I need to put them on, settle down, get a clear and confident image of what I am about to do, and get in the track.

The good thing about Turnkeylinux and Ski Jumping is that you usually get to try again (as long as you don't hurt yourself - or make your build irreprable).  The bad thing about Turnkeylinux and Ski Jumping is that sometimes you run out of time.

I would like to work on a Documentation/NotePad Integration system.  Everyone will have slightly different checklists.  Perhaps they should be public.  For sure they should be reusable.  Maybe today there won't be too much wind and I can pull my list together.

Liraz Siri's picture

Thanks for the feedback Arnold! Alon and I have tried hard to look at things from an outsider's perspective and work that into the documentation. But much like expert Skiers, we've forgotten what it's like to now know. It turns out to be really hard to ignore what you know. This is a such a well known problem it even has its own Wikipedia entry:

Curse of knowledge 

I think the best remedy would be more user feedback, which will help us better understand what our intuition is blinding us from seeing. Speaking of intuition, if you hang in there and immerse yourself in TKLDev you'll eventually develop a feel for it, much like skiing. You won't have to think things through carefully with a checklist because it will become automatic.

In the meantime, we can make lemons into lemonade by taking advantage of your confusion to improve the instruction process for future fellow travelers. But that will only happen if you (and others in a similar position) provide feedback we need to make this better.

Landis Arnold's picture

Documented in the forums.  It wasn't perfect but I was able to massage what it built anyway into a usable system.

Jeremy has been very helpful at explaining some of the components and what they do.

I am drafting now a OpenERP appliance.  I have not typed "make" yet but I am changing lots of files (from the original PostgreSQL).

I do think there would be value in a systematic documentation system.  Starting from the fundamentals of what every "folder" is generally for in a TK system (to what it is for in a specific system if it varies).  Then minimally a App specific set of docs or links to other communities so that the App itself can be navigated.

In working on the OpenERP appliance I keep having the thought that it would be nice to put some controls for the OpenERP Server (akin to Apache) into Webmin.  I have no idea if that is a huge project or a very simple one.  I know there is a Webmin world out there but Google searches basically make you a fly on the wall and you don't even know which wall you just bumped into.

Anyway, thanks for all the great work and help.

 

OnePressTech's picture

As the gurus of the build, is there a comparison table that you could whip up to help people like me and others who are technically literate but not build-domain experts to have that "aha" moment that helps us to know that investing time and energy into TKLDev would be a better investment of time and energy than in Vagrant?

I just mention this in the context of this post's hypothesis which appears to be a proposal of TKLDev as the centre-piece of the TKLX community contribution world moving forward.

For example...my primary use of TKLX is Wordpress. There are now two emerging Vagrant / Worpdress appliance open source individuals / teams forming:

http://vagrantpress.org/

https://github.com/Varying-Vagrant-Vagrants/VVV

This allows me to build my boxes in a developer-friendly / deployment-friendly fashion with Puppet or Chef for the life-cycle management.

Don't get me wrong...I'm not looking for reasons to leave the TKLX community. Just looking to participate in the TKLX journey. Would TKLX be better off embracing Vagrant and phasing out TKLDev?

I'm not proposing this...just asking the question with the expectation that any answer will be educational as always :-)

 

Cheers,

Tim (Managing Director - OnePressTech)

Jeremy Davis's picture

Hi Tim, TBH I'm not really up with Vagrant, although your comment has prompted me to do a little research... But please don't take this is any sort of authoritative answer! :)

Following my (somewhat brief) research I don't think that Vagrant is aiming to acheive the same ends as TKLDev. My understanding is that what Liraz is talking about is OS developers (that are already building an OS based on Debian) use TKLDev to build their OS. Whereas Vagrant is more aimed at allowing developers to create disposable development environments from an existing OS (for software level development rather than OS level development).

From my understanding and perspective a 'Vagrant box' is an OS with a specific set of packages installed and configurations set. I guess in that sense Vagrant and TKLDev are somewhat similar. Although I think in practice the resulting OS has a different intention/market. Vagrant is more aimed at developers wanting a consistent VM starting point, whereas TKLDev is more aimed at producing an end user OS.

Having said that, perhaps there is room for TurnKey to develop 'Vagrant box' builds of the current appliances too? I guess a TKL build more aimed at software developers rather than end users...!?

OnePressTech's picture

Thanks Jeremy. I admit I haven't waded into the Vagrant pool just yet. That's why the query. I figured Liraz / Alon may be best placed to whip up a comparison table to enlighten the followers like me.

You are correct that Vagrant started out Dev / Virtualbox centric...but they have been broadening out and now target VMWare and AWS.

In the end a build is a build is a build. It doesn't matter why a build environment is developed but what its capabilities are. And Vagrant appears pretty comprehensive and active. A look at ohloh (my favorite activity checker) shows a mature, active and growing community (http://www.ohloh.net/p/vagrantup).

TKLX is brilliant at making the complex simple. Maybe Liraz & Alon and TKLX community can simplify the use of Vagrant in TKLX.

Just a thought...

Cheers,

Tim (Managing Director - OnePressTech)

Jeremy Davis's picture

In essence I agree with you ("a build is a build is a build"). Although because of the intention of Vagrant (i.e. the problem it is trying to solve) it doesn't have some (important IMO) features baked in (like backup, etc). Although, using Vagrant with TKL might be a winner for people using TKL products day to day (developing/managing servers/websites for customers, etc.). For example if you have specific plugins that you use everytime, rather than building them into the TKL appliance, adding them with a Vagrant script might be more useful (and quicker and easier - especially considering that ATM TKLDev only outputs ISOs).

Something else that Vagrant doesn't do is produce real machines. From what I've read it's great at provisioning virtual machines and as you said, now even includes support for AWS. But what about other hosting platforms? What about people that want to run TKL on bare metal hardware? Going down an exclusive Vagrant path would remove the TKL relevance for those users... And a solution that loses TKL users (if ISO builds and alternate cloud provider support are lost) or increases maintenance overheads (by needing to create a Vagrant build and an ISO/build for alternate hosting providers) is probably not a great idea...

Thinking about this more though... Now TKL supports LXC (as a host and as a container) and there is a Vagrant plugin for LXC perhaps that would be a development option?

Also in some respects Docker is trying to solve similar issues to Vagrant, although in quite a different way. With Docker support perhaps that is another direction for TKL to take? In my travels I found quite an interesting article on Docker vs Vagrant. At the end it talks about integrating the usage of Docker and Vagrant... So perhaps there is a place for Vagrant, even if TKL goes the Docker route? FWIW there is also a Vagrant Docker provider!

[off topic] Also you prompted me to update TurnKey's Ohloh profile. I added a heap more repos so it looks a bit healthier now! :)

OnePressTech's picture

Good point J.

Perhaps if we look at TKLDev as an ISO builder and add Vagrant to the TKLX core then the ISO could be self-updating based on a set of vagrant files maintained in the TKLX Git hub. That way appliances would not go out of date as quickly. Periodic major TKLX core releases could trigger a rebuild of all the appliances to the level of their latest TKLX Vagrant files to keep appliance installation time small. Team leaders could be assigned, nominated or volunteered to maintain the Vagrant files. I'm happy to lead / participate in Wordpress, LAMP, OTRS at present.

Think of this architecturally as a 2-stage build process: a static build stage (TKLDEV) followed by a dynamic build stage (Vagrant).

Perhaps TKLX could establish itself as the defacto standard for Vagrant Debian box at http://www.vagrantbox.es/ to broaden its audience (something Liraz has been discussing in a number of recent threads). TKLX is a trustworthy brand.

Perhaps Liraz and Alon could chat with the Vagrant leads and discuss a more visual partnership to grow both communities. They seem complementary.

 

Cheers,

Tim (Managing Director - OnePressTech)

Jeremy Davis's picture

Also see my post below (perhaps I should have kept all my out-loud thinking in one post...!?).

I think that TKL devs probably need to have a discussion about how/whether using Vagrant could be a great mechanism for keeping appliances up to date!

Along the lines of what you said and what I mentioned below; maybe we could reuse the TKLDev appliance code somehow so only one code base needs to be used. TKL releases (in all their build types) could built from TKLDev build code and for relevant appliances (i.e. those running in VMs and on AWS) could be updated (using the same code) with Vagrant...?

That could be a real winner!

Jeremy Davis's picture

Doing some more research on these ideas, pulled up a couple of stackoverflow posts that I thought were worth passing on:

"Should I use Vagrant or Docker.io for creating an isolated environment?". I specificly suggest reading the interesting answer by the author of Vagrant.

"How Docker is better than Vagrant+LXC+Chef?".

OnePressTech's picture

Good find J. I liked the discussion from one of the Vagrant authors.

Ok, so if I were to summarise in short form, and correct me if I am wrong:

1) TKLDev is for static Application-centric VM / Appliance builds

2) Docker / LXC is for static Application component builds

3) Vagrant is for dynamic VM build and deploy

4) Puppet / Chef / Ansible / Salt / CFEngine are for dynamic O/S & application configuration & deployment

Interesting overlap!

Cheers,

Tim (Managing Director - OnePressTech)

Jeremy Davis's picture

But perhaps someone will come along and correct us! :)

Landis Arnold's picture

It often takes a lot of work to customize a platform.  I am sure that Wordpress is similar to my experience in Joomla.  Being able to bring a set of templates, plugins components and connectors all take time and prebuilding such would be helpful.

Built and customized Apps (like the Vagrant Box described above) is a good approach.  This can be done in TKL by usin TKLBAM and making a generic box w/ those components which are brought in in a first restore.  The problem with this is TKLBAM is not always easy.  A bigger problem is that many of the core settings don't move forward and need to be manually retooled.

By this I mean, php.ini  mysql.ini and other related settings.  I am looking at spending a good part of the day retooling these for my Magento Migration.  I have needed to do this just about every time I have brought Magento forward.  Perhaps the same with Joomla.  (I have Joomla and Magento working together but because I don't have enough resources open by default, makin that link is dogging my Magento platform.  Iv'e had to turn off the link until I can get it retooled.

Having a Magento-Joomla box (or a few o them actually) would be ideal.  In fact from a speed point of view, one where they were both on the same box and not having to talk through my firewall and dns world would be ideal.

Another project.  Might be a better use of my time today and tommorrow.

OnePressTech's picture

I couldn't agree more L.A. My build experience is simliar to yours and I suspect most if not all others in the TKLX community.

Being able to drop in a pre-built appliance provides a great sense of instant satisfaction and incredible value for which we are very grateful for TKLX. But the appliances becomes obsolete quickly and we find ourselves in a lifecycle management mode involving continuous upgrade and testing. This takes us into the land of: packaging, configuration management, logging, etc.

At the moment TKLX has not been positioned in the emerging DevOps (http://theagileadmin.com/what-is-devops/) landscape and if I work backwards from Wordpress looking at where the DevOps action is it leads to Vagrant.

So I'm interested in working out how TKLDev and Vagrant relate to each other. Do I need only one or both! I'll publish my discovery here as I sort it out unless someone beats me to the punch.

Cheers,

Tim (Managing Director - OnePressTech)

Jeremy Davis's picture

I think this is fantastic feedback from both you guys.

It gives real clues to how things could really move forward and 'amp up'...! I think that the logical progression for TKL is containerised builds (both as a host and as a container - think Docker or LXC as per my post above). I think that may well solve some of the issues you speak about (although perhaps not all automatically...). And as I mentioned above; perhaps Vagrant might be part of that solution? I don't know really...?

Perhaps it would provide an alternate way to build TKL containers/VMs? It might even be worth exploring how Vagrant could be used to leverage the existing TKLDev appliance build code (i.e. rather than producing the build with TKLDev and outputting an ISO, Vagrant could reuse the code to build a Vagrant box ontop of Core?).

I don't know enough about Vagrant, but perhaps a cool build process could be start with TKL LXC/Docker appliance, then use Vagrant to provision a Core container, add the desired specific TKL appliance code (to the LXC/Docker container) using Vagrant and add on top your own customised code (e.g. specific plugins, specific settings, etc), again using Vagrant...?

Definitely food for thought...!

OnePressTech's picture

Thanks J. Lot's of interesting options.

Well done updating Ohloh. I think it's important that Ohloh be checked periodically so anyone who looks sees an active and vibrant community. Two points of note you might look into...1) You seem to appear as "your fullname" and 2) the summary page shows 5 months since last commit. It might be good to have a more regular commit model both from a release perspective and an OhLoh marketing perspective.

While I'm thinking of it...regarding Vagrant...there is a GUI for it which might make it friendlier for our Webmin crowd...http://getprotobox.com/.

Cheers,

Tim (Managing Director - OnePressTech)

Jeremy Davis's picture

Thanks for the link Tim, Protobox looks awesome! Tethered to the Vagrant-LXC plugin, that could be a fantastic web frontend for TKL-LXC (with Vagrant as well...); at least for much of TKL LXC... If I have time I might even work on a prototype - I've been meaning to play with LXC for a while now and this could be an awesome excuse! (Not that I really need one...!)

Also regarding Ohloh; that person wasn't me... It was someone who commented on some code with a suggestion that was later introduced by TKL. Their GitHub profile doesn't have a 'real name' so it shows as "your fullname". Most of my contributions have been via the 'Issue Tracker' rather than specific code additions (although I have provided a few now but they haven't yet been merged). And I'm not sure how Ohloh works and/or whether adding the issue tracker would really be a valid way to go (as far as TKL stats on Ohloh are concerned).

I will add all the appliance code as well (from the GitHub turnkey-apps account) and that will make it look even more healthy! :)

Landis Arnold's picture

It seems if Vagrant is very robust and very diverse perhaps there could be a way to do a one two:  TKL then Vagrant over.

I like being able to do backups, generally migrate, have fast installs etc w/ TKL.

It is complicated moving some TKL systems forward (as noted).

It is especially complicated having "customize builds' move forward.  TKLDEV probably can solve this.  Perhaps TKLDEV could bring Vagrants in and everyones goals could be served.

I will study Vagrant.  It seems possible w/ TKL (without Vagrant) but as noted above, there are issues moving forward w/ TKL that could be easier.  Maybe Vagrant would be a way to make those subjects easier.  It could also be as easy as a simple "config" file applied to TKL too.  Really it seems that we need to be able to allocate resources in TKL based on what is available instead of a 256k assumption.

Thanks for the great comments Tim and Jeremy.

Pages

Add new comment