Announcing TurnKey Docker optimized builds

Please note: This blog post is quite dated, for the latest updated info regarding usage of TurnKey Docker builds, please see the doc page.

docker logoAs we've mentioned before, making TurnKey easy to deploy no matter your platform of choice is an important goal for the project. TurnKey already supports a mirade of build types including ISO, VMDK, OVF, Amazon EC2, OpenStack, OpenVZ, OpenNode, Xen, and recently added support for LXC.

I'm pleased to announce that today we are updating the above list, with support for Docker.

 

Docker is an open-source project to easily create lightweight, portable, self-sufficient containers from any application. The same container that a developer builds and tests on a laptop can run at scale, in production, on VMs, bare metal, OpenStack clusters, public clouds and more.

Deploying TurnKey on Docker

All TurnKey appliances are available on the public docker index (generously provided by Docker, Inc), which streamlines deployment. For example:

docker pull turnkeylinux/core-13.0
docker run -i -t -d turnkeylinux/core-13.0

Docker containers can be run in the foreground or the background, so we've tried our best to support all use cases with regards to initialization (aka. inithooks) - secret regeneration, setting of passwords, application configuration, etc. 

Depending on your use case, we recommend two options:

Option 1: Initialization via ssh (interactive)

On first login, you will be prompted to initialize the appliance.

CID=$(docker run -i -t -d turnkeylinux/core-13.0)
CIP=$(docker inspect -format='{{.NetworkSettings.IPAddress}}' $CID)
docker logs $CID | grep "Random initial root password"
ssh root@$CIP

Option 2: Create new image with preseeded values (non-interactive)

The appliance will initialize itself with the provided configuration.  Once initialized, the configuration will be deleted. For more information see inithooks.

mkdir /root/wordpress

cat > /root/wordpress/inithooks.conf <<EOF
export ROOT_PASS=secretrootpass
export DB_PASS=secretmysqlpass
export APP_PASS=secretadminwppass
export APP_EMAIL=admin@example.com
export APP_DOMAIN=www.example.com
export HUB_APIKEY=SKIP
export SEC_UPDATES=FORCE
EOF

cat > /root/wordpress/Dockerfile <<EOF
FROM turnkeylinux/wordpress-13.0
ADD inithooks.conf /etc/inithooks.conf
EOF

docker build -t wordpress-13.0 /root/wordpress
docker run -i -t -d wordpress-13.0

Notes

Pre-configured run command

Docker is designed for "application or process" containers - for example, running mysql, and only mysql. Docker short-circuits /sbin/init so you can't really "boot" a container like in vanilla LXC.

To work around this, we've included /usr/sbin/start.sh (default run command) which will start all services and drop to a shell. When the shell is exited, the services will be stopped. For this reason, SSH is recommended for regular console usage.

STDIN and TTY options required

The -i and -t options are required to attach STDIN and allocate a TTY to the container. Unfortunately this cannot be pre-configured as it is not yet supported.

Pre-configured to expose ports

All TurnKey Docker appliances are configured to expose their custom services. This means that the host can access the services, but they are not exposed to the network.

Exposing ports to the network needs to be done at runtime (docs), for example:

# bind port 80 on the host to the container's port 80
docker run -i -t -d -p 80:80 turnkeylinux/lamp-13.0

Skipping security updates on first boot

During development I added support to start.sh to override the default SEC_UPDATES value to speed up my testing. I was going to remove this support or leave it undocumented, but decided others might find it useful when testing (and only in testing).

# THIS IS NOT RECOMMENDED, USE AT YOUR OWN RISK!
docker run -i -t -d -e SEC_UPDATES=SKIP turnkeylinux/openldap-13.0


Keep in mind, this is the initial release of TurnKey Docker support, so let us know what you think. If you have ideas for improvement, or if you come across any issues (ie. we haven't tested all appliances as of yet), drop us a line.

You can get future posts delivered by email or good old-fashioned RSS.
TurnKey also has a presence on Google+, Twitter and Facebook.

Comments

Eric (tssgery)'s picture

I'll try this out next week!


Adam Kegg Powell's picture

I was waiting for this release! Please shoot me an email! I have a few ideas about turnkeys and have been coming to your website since your first release! I love them and thank you for the dedication and effort that you put into making them!

Adrian Moya's picture

This sounds really great, specially with the possibility to combine with Solum on openstack! Too much to play for 2014!

Congrats once again on this release, I haven't even got time to play with LXC, now this! TurnkeySanta has been generous this december...

I'm going to need some help on this one.

I get TKLX appliances. An appliance is a self-contained computer. TKLX value includes: pre-build scripts, reduced footprint O/S, pre-configured management tools, support Apps (e.g. DB), backup / restore, SSL, security patch cron, etc.

I get TKLX / LXC. Multi-TKLX appliances per VM with a reduced footprint (O/S core shared across LXC containers).

I would get TKLX / LXC / Docker if that is what TKLX / Docker was... where Docker is used as an application container running on top of LXC within a TKLX VM. This would provide app portability across TKLX VMs and would improve the TKLX upgrading process (the current TKLBAM backup on O/S V1 TKLX VM and restore on O/S V2 TKLX VM is not always a point-and-click upgrade mechanism).

But that is not what this is. This is not TKLX / Docker this is Docker / TKLX. I don't get Docker / TKLX!

I do understand that TKLX / Docker would be a major change to the current TKLX appliances since Docker expects to containerise a single app (so, for example, the current TKLX Wordpress / LAMP appliance would translate into a multi-docker-container appliance...wordpress, apache, MySQL, PHPmyAdmin, Webmin, etc, each in separate containers in the appliance). But I see great value in TKLX / Docker. I don't see the value in Docker / TKLX.

Perhaps a use-case example might help my Christmas turkey befuddled brain :-)

So here's a Christmas question...if I go to Docker library and search for Worpress I get 3 pages of Docker / Wordpress containers one of which is a Docker / TKLX-Wordpress container. What does a Docker/TKLX-Wordpress container "contain" that makes it special relative to the other Docker / Wordpress containers?

Thanks in advance and apologies to those in the TKLX audience that get Docker / TKLX. I'm just not one who gets it..yet :-)

 

Cheers,

Tim (Managing Director - OnePressTech)

Alon Swartz's picture

True, Docker's design is application-centric, and optimized for the deployment of applications, as opposed to machines. This is reflected in its API, user interface, design philosophy and documentation.

That said, it is just a tool, and tools can be used for other things that the original designer did not intend or imagine. For example, nucular power, Linux, or even LXC itself (it wasn't designed for application-centric containers).

Using Docker for system containers still provides benefits, such as integration with AUFS and other copy-on-write technologies, handling network configuration transparently, portability, as a distribution channel, and a growing community and ecosystem.

Additionally, there are benefits to using system containers vs. application containers, but it all depends on your use case. I like to think of this using the analogy of multi-tier vs. single tier. In a multi-tier deployment (eg. frontend servers, backend servers) you are approaching the idea of application-centric-containers, where as each server does one thing, and relies on other servers to fulfil related needs. Each server needs to be configured to talk to the other servers. This is great approach, but is much more complex to setup and maintain than a single unit - where everything is self-contained in one logical unit, easier to understand, easier to bugix and easier to maintain. Both approaches have their own pros and cons ofcourse, but as I said, it all depends on the use case.

Coming back to your example, using Docker as intended to setup a Wordpress deployment can become quiet complex when seperating out each function and linking them together. OTOH, you can leverage the power and (most of the) benefits of docker when using a pre-integrated self-contained TurnKey Wordpress image.

I am not ruling out creating TurnKey application-centric Docker images and some magic pixie dust to configure and link the containers together in the future, but at present, supporting TurnKey on Docker does make sense for some use cases.

I had more to say but your spam filter rejected it for no apparent reason. I have submitted my follow-on comment via email to your webmaster.

Edit by Alon: Tim's comment via email

Ah...I get you now. So using my non-techie speak a conventional Docker /
Wordpress container is non-functional without other Docker containers such
as MySQL container and Apache container while the Docker / TKLX / Wordpress
container is operational from the get-go (App vs. Appliance). Plus a little
extra cross-VM portability. Clever albeit non-conventional.

I am glad you are still looking at TKLX / Docker though. You see your
comment above related to the complexity of multi-server vs. single server
deployment disappears with TKLX / Docker. If a TKLX user can install an
appliance containing a Wordpress container + MySQL container + Apache
container + Tools container all pre-configured to work together we would
have a seriously corporate grade appliance. Need to expand the Wordpress
database to scale.... add the multi-db plugin to Wordpress, add another
MySQL DB container to the appliance and upgrade the VM to a larger AWS
instance type....done. Easily automated as well. Additionally separation of
concerns helps in the QA dept w.r.t. upgrades.

Commercially TKLX is a containerisation project as is Docker. They compete
for mindshare.

My concern is that, although there is some technical merit as you described
above in Docker / TKLX, by placing TKLX subservient to Docker project TKLX
risks diffusing its market position. I would respectfully suggest that the
TKLX project position Docker as a subservient component and be cautious
about positioning TKLX commercially subservient to Docker.

It is inevitable that others will build pre-configured Docker packages
(multi-container versions of TKLX appliances) pre-built with products like
SaltStack. These would be inferior to TKLX / Docker appliances in my opinion
(no TKLBAM, TKLDev, reduced footprint O/S, etc) but viable TKLX competitors.
If the Docker community embraces these Docker Packages...and they would,
TKLX would be fighting for mindshare. It would not be the first time that a
high volume inferior product beats a lower volume superior product (VHS vs.
Beta, etc).

Redhat and Docker are working together

RHEL7 will contain Docker

Redhat has made containerisation a top priority for 2014

Docker is still stabilizing so there is no immediate rush to embrace it as a
TKLX component, but I think it would be a mistake for our TKLX community not
to seriously start to work to create a couple of Beta pre-configured TKLX /
Docker appliances. I would need your help on the Docker side but I am happy
to take on the Wordpress side of a TKLX / Docker / Wordpress appliance. This
is core to my business so I am serious about committing time and resources
to this.

Just one man's two cents worth 

Cheers,

Tim (Managing Director - OnePressTech)

Hans D's picture

Will the Dockerfiles, and any scripts used for entrypoint etc, used to generate the containers be available?

Alon Swartz's picture

The docker builds weren't created via a Dockerfiles per say, but rather by importing a slightly patched rootfs as a base image, then using a Dockerfile to set the run command and ports to expose.

The patched rootfs had the kernel removed, some packages purged (e.g., resolvconf, ntpd), added start.sh for the run command, and some other misc tweaks.

An example Dockerfile used for LAMP looked something like this:

FROM turnkeylinux/lamp-13.0
CMD ["/usr/sbin/start.sh"]
EXPOSE 22 80 443 12320 12321 12322

 

 

Anonymous's picture

Hi

Fist of all, Thank you all for your great works.

Can you explain how to reproduce the "slightly patched rootfs"?

Do you, or would you plan to add in TKLDev the feature to genarate docker images in docker's local registry?

Regards

Liraz Siri's picture

If I'm not mistaken Alon used a bit of a hack to create the Docker builds. He tweaked the scripts we used to convert ISO to various formats which usually lives in buildtasks, which is used by cloudtasks, our cloud batch system to convert the images.

One of the things high on our todo list is moving the conversion code from buildtasks to TKLDev. In the meantime, if anyone wants to take a look at a specific conversion script I can upload the (unmodified) scripts so they can understand how it works, but you'll still need to tweak it a bit to get it to actually do anything useful otherwise we would have already just rolled it into TKLDev and be done with it.

Jean-Claude Wippler's picture

Great stuff - I tried tkl/redmine, and it works nicely. I did add "-p 80:80 -p 2222:22" to the "docker run", since this was in a cloud VM, and port 22 was already in use to reach that VM.

This might be of interest: http://docs.docker.io/en/latest/use/working_with_links_names/

There could be a MySQL container, with Redmine linked to it, for example.

The property which makes it so interesting IMO, is the TKLBAM side of things: being able to set up containers with state (records in MySQL and files/attachments uploaded to Redmine for example), and still have daily incremental backups of the different sub-systems.

The other benefit I see, is running multiple appliances in one VM: I current have 3 VMs running nginx, wordpress, and redmine, respectively. This eats resources for no good reason (and there are two MySQL instances running when done this way). With containers, the different pieces are still distinct and can be relocated / updated individually as needed.

One side effect is that there's also plenty of rope to hang myself with all the options this opens up...

Thanks for making these containers available.

-jcw

Jean-Claude Wippler's picture

Reading back this thread more carefully, I see that a lot of my comments are duplicated from what's already been said. Sorry about that. Great stuff nonetheless.

Brad's picture

I am runing this on AWS and have opend up EC2 security groups ports 2222 + 22 

I keep getting connection refused. I would love to hear how Jean-Claude Wippler go this to work. 

 

ubuntu@ip-10-110-95-162:~$ CID=$(docker run -i -t -p 2222:22 -d turnkeylinux/core-13.0)
ubuntu@ip-10-110-95-162:~$ CIP=$(docker inspect -format='{{.NetworkSettings.IPAddress}}' $CID)
ubuntu@ip-10-110-95-162:~$ docker logs $CID | grep "Random initial root password"
                        Random initial root password: 58da35a3
ubuntu@ip-10-110-95-162:~$ ssh root@$CIP
ssh: connect to host 172.17.0.8 port 22: Connection refused
ubuntu@ip-10-110-95-162:~$ ssh root@localhost -p 2222
ssh_exchange_identification: Connection closed by remote host
ubuntu@ip-10-110-95-162:~$ ssh root@172.17.0.8 -p 2222
ssh: connect to host 172.17.0.8 port 2222: Connection refused

I tried logging in by attching to the container as root /usr/sbin/start/.sh and it errors, has to be by remote ssh. 

 

Any ideas????

Jean-Claude Wippler's picture

My last note on this is to use "ssh root@$CIP" - no port 2222 stuff on the ssh command after all. I don't quite understand why, since the "-p 2222:22" on the docker command would indicate that the port was renamed.

One last thing to try: use "-p 22:22" on the docker run, and none on the ssh. In hindsight, that makes the most sense: port 22 is on the $CIP port, not localhost, so there should be no conflict.

-jcw, a bit confused

Alon Swartz's picture

You only need to specify port redirection (ie. -p HOST_PORT:CONTAINER_PORT) if you want to expose a containers port to the network. By default, turnkey containers are preconfigured to expose certain ports (such as SSH) to the host, so in theory once a container is up and running you can SSH into it from the host:

ssh root@$CIP

My guess as to why this didn't work for you was that the ssh daemon was not running yet, most likely due to the security updates being installed. 

@brad, what error did you get when attaching to the container?

Jean-Claude Wippler's picture

I think I have an explanation for what's going on.

You have to wait (quite) a bit for the docker container to start up ssh. It does all sorts of things on first startup. To see what's going on anyway, use "docker attach $CID". To detach, use ctrl-p ctrl-q (don't use ctrl-d to logout, as this will stop the redmine stuff).

Brad's picture

That was it, I had to wait for the image to spin up. 

Jean-Claude Wippler's picture

(I've never sailed so close to the edge of a server meltdown as today... luckily, it's back up, sort of...)

This may be a very odd scenario, but unfortunately I have little choice:

* a month or so ago, I installed the TKL Docker build of Redmine into a VM running Docker

* due to my own stupidity, I managed to break the docker setup, and found out the hard way, after a reboot many days later

* the good news: restoring the backup to an EC2 instance worked, after minor tweaks - yeay, thank you TKLBAM developers!

* alas, the EC2 instance turned out to be too small, so I had to migrate again within hours

* I've set up a fresh *normal* Redmine VM, from a fresh TKL image, on local hardware

* re-inited tklbam, did a restore, and lo and behold - it's working directly on the VM again (thx again!)

* minor tweaks, but no sweat, everything is back to normal

One last check: reboot. Whoops - the Docker build of Redmine got restored over the fresh VM version of Redmine, and... it drops straight into grub command line. Never mind - didn't take long to find the magic incantation to boot up manually. Running "update-grub" after that was not enough to fix the reboot, alas.

All is well, except for one "subtle" detail: this server now requires entering grub commands manually on booting. Not a long term solution, clearly. I'm delighted to have brought everything back up, and the sit eis back online, but this ticking timebomb is still there.

I have a virgin Redmine VM to compare against, but perhaps someone here could give some tips on what to copy over. Vaguely recall seeing a mention of a special init.sh in the Docker. That, and the grub stuff is probably what needs to be recovered? The console shows a login prompt, not the usual blue-and-gray admin console.

Sorry for the long post, but I do hope there is a practical solution at the end of this tunnel I steered myself into.

Cheers,

-jcw

-jcw

pee's picture

Hi, 

The docker images are terrific! I hope you will continue to provide them, and make installation a bit easier ;)

But I encountered some problems

1) inithooks.conf:

it seems that the method above does not change /etc/inithooks.conf (file is empty) in the container, because I can not access it with those values

I had to 'docker attach' the container and run turnkey-init to set my root (and other) passwords.

Then I could login but then I got again the turnkey-init screens ....

 

2) ssh:

#ssh root@8cdcf723a4d7
#ssh: Could not resolve hostname 8cdcf723a4d7: Name or service not known

But as you can see below the docker cid is correct

What can be wrong?

 

root@ubu14:~/prestashop# docker ps
CONTAINER ID        IMAGE                           COMMAND                CREATED             STATUS              PORTS                                                      NAMES
8cdcf723a4d7        prestashop-13.0:latest          /usr/sbin/start.sh     30 minutes ago      Up 21 minutes       12320/tcp, 12321/tcp, 12322/tcp, 22/tcp, 443/tcp, 80/tcp   lonely_lovelace
924e336fb67a        crosbymichael/dockerui:latest   ./dockerui -e /docke   16 hours ago        Up 16 hours         0.0.0.0:9000->9000/tcp                                     dreamy_archimedes

root@ubu14:~/prestashop# ssh root@8cdcf723a4d7
ssh: Could not resolve hostname 8cdcf723a4d7: Name or service not known

 

I am using the docker.io v0.9 provided with ubuntu 14.04 as host

 

3) docker redmine appliance

same problems as above

additionally I could not get the redmine webgui working, I got errors:


An error occurred while starting the web application. It sent an unknown response type "".

Application root
/var/www/redmine
Environment (value of RAILS_ENV, RACK_ENV, WSGI_ENV and PASSENGER_ENV)
production
Ruby interpreter command
/usr/bin/ruby
User and groups
Unknown
Environment variables
Unknown
Ulimits
Unknown

/var/lib/gems/1.9.1/gems/activesupport-3.2.12/lib/active_support/dependencies.rb:251:in `block in require': iconv will be deprecated in the future, use String#encode instead.

 

I tried to upgrade redmine but that didn't help either

Any help would be appreciated

Thanks

Jeremy Davis's picture

Sorry I haven't yet taken the Docker appliances for a test drive yet so can't help you at all... :(

Hopefully someone else that has will give you a hand...

pee's picture

thanks for your quick reaction jeremy.

docker is now insanely popular with a huge comunity and ubuntu and redhat supporting it officially.

It was a wise decision to provide docker builds, so I hope turnkeylinux will stay devoted to provide the docker appliances. ;)

and if anyone has a clue what went wrong, please advise ;)

tx

Jeremy Davis's picture

And TurnKey intend to support it for the long term. So Docker builds won't be going anywhere! :)

Hope someone comes to help you soon though!

pee's picture

I reinstalled redmine docker again on a fresh ubuntu 14.04:

the problem is that mysql can not be started:

root@464bf65b5383 ~# service mysql restart
[ ok ] Stopping MySQL database server: mysqld.
df: Warning: cannot read table of mounted file systems: No such file or directory
[FAIL] Starting MySQL database server: mysqld . . . . . . . . . . . . . . failed!
root@464bf65b5383 ~#
 

Web application could not be started

Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) (Mysql2::Error)
  /var/lib/gems/1.9.1/gems/mysql2-0.3.13/lib/mysql2/client.rb:58:in `connect'
  /var/lib/gems/1.9.1/gems/mysql2-0.3.13/lib/mysql2/client.rb:58:in `initialize'
  /var/lib/gems/1.9.1/gems/activerecord-3.2.12/lib/active_record/connection_adapters/mysql2_adapter.rb:16:in `new'

 

someone an idea how to solve this?

pee's picture

Hi,

I did some more testing and I think the error is in all appliances with mysql (at least it is in lamp and in prestashop and in redmine)

(I have error logs, but could not post them here because of your spamfilter)

Any clue what is wrong?

Thanks

 

Jeremy Davis's picture

If you start a new thread, then you should be able to attach your relevant log files (attachments can only go on the first post in a thread).

And/or post an issue in the Issue Tracker (hopefully you should be able to include logs on GitHub).

pee's picture

Hi,

 

I found the cause of the troubles above ...

apparantly mysql has stopped due to low memory (on a vm with 512mb), giving the sockets error.

thanks for your help guys!

Jeremy Davis's picture

Good work. Thanks for posting back.
rastaman's picture

Hi

I've troubles setting up admin console.

I tried using ssh port forward trick but it is not working for me.

web2py listens on  8000 on the guest

its translated to 8001 on the host

I did :

 ssh -L 2999:127.0.0.1:8001 root@localhost

in my browser pointing to http://127.0.0.1:2999 works fine but going to /admin yelds 

Admin acces in deactivated beacuse insecue channel (although using 127.0.0.1)

Could you help ?

Tia

Stéphane

Sven Thomas's picture

Hey all...

So I've managed to get TKL:Drupal7-13.0 container running in CentOS minimal.  Thing is... while Drupal and everything comes up and SSHing into the container [from the host] works great, I can't seem to access TKL's Webmin or any of the other web based admin consoles.  Exposing the ports using -p 12321:12321 or even 12321:8080 also doesn't work.

I don't plan on exposing anything other port 80 to the outside world but it would be nice if I could have access to the admin consoles from within the house.

- Sven

pee's picture

sven,

check if your firewall (on the host) does not block these ports

Sven Thomas's picture

I thought of that too, but running:

service iptables stop

returns

Fails to issue method call: Unit iptables.service not loaded.

CentOS minimal install doesn't run iptables by default.  Aside from Docker and package updates, this is a vanilla minimum install.

Update:

So after some reading I've learned that CentOS 7 doesn't rely on iptable.service, but uses RHEL's firewalld instead.  Stopping the firewalld service didn't fix the issue.  I'll keep playing with it.

Update 2:

So here's some good news.  I disabled the firewalld service, rebooted and suddenly everything was working.   If anyone is wondering how to make exceptions to CentOS 7's firewall, you can follow these handy instructions in conjunction with the page I linked in my first update.

Now this is all well and good, but I'd like to map the ports to something other than 123xx as I intend to run more than one TKL container on this server and don't want to have multiple containers competing for host ports.  How do I go about telling webmin, et al. to listen on other ports?  Is this an Apache configuration or do I have to modify the docker image?

pee's picture

hi sven,

you can use something like this to change all the ports to something in the 10xxx range for example:

docker run -i -t -d -p 10080:80 -p 10022:22 -p 10443:443 -p 10320:12320 -p 10321:12321 -p 10322:12322 turnkeylinux/drupal7-13.0

Brian's picture

I'm accustomed to running re-startable docker containers with some kind of volume specified in the docker run command, where they can store their state.  In the absence of TLKBAM to back up running dockerized TKL appliances, is there a standard directory that could be used with "docker run -v" for this purpose?

For example, I wanted to try out the TKL OpenVPN appliance in a docker container.  Won't it lose all the keys I create, unless I give it a proper place outside the container to save its data?  What directory would I bind on the inside of the container, and would it be different for different appliances?

Thanks!

Toord Mayne's picture

Have you figured anything out in regards to this? I've been playing with the OrangeHRM docker container, and it seems like it's the /var directory that needs to be bound, but I always end up with errors (permissions errors I think) when I try it.

Brian's picture

It turned out that I was always in the habit of doing 'docker kill' when I was done with an application, but when dockerizing a whole system, it makes more sense to use 'docker stop' and 'docker start' instead; then it doesn't delete the container's files.  You could also docker-bind an unused directory and use tklbam to back up into it instead: http://www.turnkeylinux.org/faq/backup-and-migration-tklbam#t601n2384

Pages

Post new comment