Forum archive
Ansible "Configs" for Turnkey
Hey guys,
I've been doing some work on how we might address providing Ansible roles that are tailored to the Turnkey Linux configuration of each appliance. I had suggested this a while back and Jeremy opened issue 553 on the tracker to document it. Here are the approaches that I've tried and my comments/suggestions along with requests for feedback.
The first approach that was in my original suggestion is to provide separate roles that would handle different custom configuration tasks in a similar manner to how I was using tklpatch to customize appliances. Some examples can be found at https://github.com/jstruebel/ansible-roles. The downside that I've found is that it doesn't fit well into the typical Ansible architecture. Because roles had to preserve existing settings in files the roles are kind of "hacky" in using line-in-file and shell commands. It was especially difficult, and most likely fragile with an existing config, for the Samba configurations. While I was working on that I came across the DebOps project which lead to my next two approaches.
The second approach was to find a way to leverage the DebOps roles yet customize the configurations to the default Turnkey Linux settings. My primary goal was to avoid customizing the roles directly so as to minimize the maintenance required, i.e. don't fork them. To that end I decided to create new playbooks that included a vars file that redefined the defaults for the DebOps variables so that they matched the default Turnkey config. Examples of this approach can be found in https://github.com/jstruebel/tkl-debops-plays. Some downsides to this approach are that it requires creating a playbook and vars file for every service that we want to support, it changes the operation of DebOps so that the user has to fork the site.yml in order to use other DebOps roles that haven't been included, redefining the variables creates problems for some conditionals in the roles, and it creates a entirely new set of variables in order to allow the user to customize the Turnkey defaults. The advantage to this approach is that no customizations to the roles are required and all of the Turnkey custom "stuff" can easily be provided in a single git repo for the user to clone. It also allows easily adding custom TurnKey roles, e.g. to configure TKLBAM.
The third approach grew out of the second approach, and this is to provide customized group_vars for each type of TurnKey appliance and a mapping of Turnkey appliance groups to DebOps roles. An example of this approach can be found at https://github.com/jstruebel/tkl-debops-inv. I feel that generally this approach is cleaner and more "Ansible". It also allows to minimize the maintenance since only the redefined defaults need to be included in the group_vars. Since it uses all of the DebOps playbooks, the user is free to mix in any other roles they may want by configuring their hosts into the appropriate group(s). Some downsides to this approach are that it mixes the Turnkey config with the user's own inventory, and it is harder to add custom Turnkey roles and have them executed along with the DebOps roles.
Right now, my preference would be the third approach since I think it fits better with how Ansible was designed to work. The downsides that I've come up with can be mitigated without much extra hassle for a competent user. For instance the inventory can be handled by the user keeping their site specific customizations in a branch. (it also has the advantage of pushing the user to keeping their inventory under configuration control) I haven't quite figured out how to address the custom Turnkey roles, the only ideas look a lot like the second and third approaches mixed which seems like it's adding complexity without much benefit. I welcome any comments and feedback on what I've done as well as suggestions for what approach would work best for Turnkey Linux (or possibly a new direction that I haven't thought of).
Replies (7)
Thanks for the Links
Reply 1Sounds great Jonathan!
Reply 2I had a quick chat with Alon about it (just before I went away - I had last week off) and initially he was very hesitant; that is until I made it clear that you are spearheading it as a community project! :)
Then he was very supportive. He suggested that we incorporate it as a "community" project sooner rather than later. I'm not sure about the best way to go on that front, but perhaps we could just wait until you have something closer to a "product" and then clone it into the TurnKey GitHub organisation (and give you write access to that)?
Or alternatively, we could start a new GitHub organisation (perhaps "TurnKey Community"?!) and again give you write access there. If we went that way, then as one of our trusted community champions, we could give you full admin rights. IMO that's probably the better option!?
Obviously that side of things is somewhat irrelevant to your immediate plans but worthy of consideration for the longer term.
Apologies I got sidetracked and forgot to ping John. I have just sent him an email. I would have cc'd you in, but as a rule I don't share people's private contact details with others without their explicit permission. For reference, if you want to personally get in touch with him, you could try his contact form accessible from his TurnKey user page. Another option would be to mention him somewhere on GitHub.
I look forward to hearing how you go with this and having a bit of a play myself once you're a little further along (and I have some pare time...).
I know all about those rabbit holes.
Reply 3I know all about those rabbit holes. Sometime it seems like the internet is just one big rabbit hole...
I think the key to using DebOps with Turnkey is to have a project "layer" that configures the DebOps roles to the Turnkey configurations. It would then provide a base for customizing the appliances in a way that promotes best practices from DevOps such as configuration management. This is the basic approach that the projects Maciej linked to use and where my option 3 is headed. This way we leverage the strength of Turnkey as an easy-to-deploy Linux Server and the strength of DebOps as a comprehensive suite of Ansible roles to configure and manage Linux Servers.
While you're correct that there's overlap between the two projects in that they both seek to simplify the task of deploying a Linux server, I think they also complement each other in that Turnkey Linux is ready-to-run and guides the user through the necessary configuration while DebOps provides the tools to efficiently manage changes to the base configuration. I see Turnkey as a great way to get up-and-running quickly with an appliance/application without having to dive into the nitty-gritty of configuring the services. It also doesn't require much in the way of external infrastructure, just a server or VM to run it on. Whereas DebOps requires the additional "infrastructure" of Ansible and it's host configuration, it allows for efficiently managing more complex scenarios with Turnkey, especially multiple appliances or instances.
I think we are in agreement on using other projects to avoid "re-inventing the wheel", I just wanted to clarify how I see DebOps fitting in. My focus is on DebOps because in the limited searches that I did, it seemed to be the most comprehensive set of Ansible roles and is well maintained. If there is another project that is similar and fits better with Turnkey Linux I wouldn't hesitate to use it.
Regarding the new TKLX project, when I first read the blog post on it I was very excited since containers are a technology that intrigues me, but I don't have much time to dive into the details of getting them setup. I imagine that it would allow scenarios like merging Wordpress and Mediawiki into a single VM to use for a personal site. Or as you mention, scaling up by splitting the services between different hosts. I could see using Ansible/DebOps/TKLX-DebOps (name made up on-the-spot) to manage deployment and user configuration of them.
For now I'm planning on migrating what I've done so far with Ansible and DebOps toward a project like the ones Maciej linked. As I need additional services they'll be added and I welcome any other contributions from the community. Whenever the Turnkey staff is able to review it and/or make it an official project, I'll gladly help as needed to transition it over.
What's the best way to ping John. I've seen his activity on here with the Ansible appliance and I'd value his input.
This looks really awesome Jonathan!
Reply 4Bottom line is that I don't have enough experience or knowledge of Ansible to provide any really meaningful comments about the explicit implementation. I get the concept and Ansible seems like probably the best of those sort of "orchestration" DevOps tools for use with TurnKey. As an aside I have heard on the grapevine that Red Hat (the newish owners of Ansible) have plans to open source Ansible Tower which will be really cool!
I have also come across DebOps in my travels (some time ago). [Hi Maciej - BTW thanks tons for dropping in!] I thought it looked really cool but IMO at the time it seemed like it had too much overlap with TurnKey to be that useful for us. But it's quite possible I was missing something.
I think that there is quite a bit of overlap with TurnKey but I really think that some sort of optimisation/customisation tool (e.g. for your use case using OpenLDAP auth) is required.
The tools we have work well for us as TurnKey developers/maintainers. And I think it's really cool that our build chain is public and accessible. Its awesome that TurnKey users can add patches, or tweak build code using the same tools that we use behind the scenes. But having a better tool for end users; that doesn't require rebuilding ISOs is highly desirable IMO!
Also the advantage of using more poplar tools is that many new users can have a head start. And it may well draw in users that are already familiar with those tools? Working with other open source projects and communities is also really cool I reckon.
Personally I'd really like to work towards something that could be an "official" TurnKey tool. Ideally integrating with a third party project as you are suggesting. Unfortunately we don't have a lot of spare resources to put towards that though. I'm more than happy to throw in my 2c, but that's about the best I can do, at least until after we release 14.2. Ideally I'd like that be been out before the end of the year, but I can't guarantee that will happen.
One other thing very worthy of mention is that IMO we should be aiming to be using a tool that will be (TurnKey) "future proof". v15.x will be a radical change to how we currently do things. It will be container based model.
The idea is that the containers will be compatible with docker and rkt (and probably nspawn too) so TurnKey won't necessarily need a TurnKey OS base anymore. We still anticipate producing a Core-like appliance but it will be a host for the TKLX containers. The idea will be to default to having the containers all on one host so current users will still pretty much get what they expect on face value (e.g. a WordPress appliance that they can launch from an ISO or OVA etc). But under the hood it will use containers and support easily splitting the workload between different hosts.
Check out https://tklx.org (actually https://github.com/tklx) for a sneak peek at what Alon and Anton have been playing with. BTW I anticipate that there will be some crossover between the old model and the new. It's still way too early to say much more than what I have though.
PS I might ping John Carver. He's been a long time Ansible advocate and enthusiast. He might have some valuable ideas to add to the mix?
I got hold of John
Reply 5Thanks for the plug Disha, but this is TurnKey website!?!
Reply 6Thanks for the great Links
Reply 7TurnKey Ansible helps save you time and money by providing a ready-to-run Ansible solution that is secure, supported and easy to maintain. The system auto-updates itself with security fixes and is built in a transparent 100% open source process free of hidden backdoors.Ansible is a simple, agentless IT automation engine that automates cloud provisioning, configuration management, application deployment and intra-service ochestration. It can configure systems, deploy software, and streamline advanced IT tasks such as continuous deployments or zero downtime rolling updates. For more information about check this link
I hadn't thought of making it a whole project like the ones you linked, but now I see that that would fit pretty well with the DebOps methods. It would also make including extra playbooks or roles easy.
I had been trying to come up with a way to keep the user's custom config separate from the Turnkey defaults to make it simple for them to upgrade when needed. However, if it's all just one big git repo, then they can create their own branch and merge in updates or just keep everything on master and merge or rebase. The only gotcha would be if they wanted to contribute any fixes/improvements back, we would need to be careful not to get their personal config mixed in. However, it adds just a little more work in that situation and overall I think it works out cleaner.
Thanks again for the other examples of projects that use DebOps.