You are here
Comparing Debian vs Alpine for container & Docker apps
Background: For TurnKey 15 (codenamed TKLX) we're evaluating a change of architecture from the current generation of monolithic systems to systems as collections of container based micro-services. Essentially the service container replaces the package as the highest level system abstraction.
There are several layers to the new architecture, but the first step is to figure out the best way to create the service containers. Alon has been quietly working on this for the last couple of months and managed to slim down Debian to 12MB compressed for the base image:
https://hub.docker.com/r/tklx/base/
With Anton's help we added PoC tklx containers for Mongodb, Nginx, Postgres, Apache, Django and others:
https://hub.docker.com/u/tklx/
So far the most thought provoking question we've received is: why are we using Debian for this instead of Alpine Linux, the trendy minimalist upstart blessed by the powers at Docker?
That is a very good question, and it deserves a good answer.
Alpine Linux has its roots in LEAF, an embedded router project, which was in turn forked from the Linux Router on a Floppy project.
As far as I can tell Alpine would have stayed on the lonely far fringe of Linux distributions if not for Docker. I suspect a big part of Dockers motivation for adopting Alpine was the petabytes of bandwidth they could save if people using Docker didn't default to using a fat Ubuntu base image.
Debian is superior compared to Alpine Linux with regards to:
- quantity and quality of supported software
- the size and maturity of its development community
- amount of testing everything gets
- quality and quantity of documentation
- present and future security of its build infrastructure
- the size and maturity of its user community, number of people who know its ins and outs
- compatibility with existing software (libc vs musl)
Alpine Linux's advantages on the other hand:
- it has a smaller filesystem footprint than stock Debian.
- its slightly more memory efficient thanks to BusyBox and musl library
Alpine touts security as an advantage but aside from defaulting to a grsecurity kernel (which isn't advantage for containers) they don't really offer anything special. If anything the small size and relative immaturity of the Alpine dev community makes it much more likely that their infrastructure and build systems are compromised. Debian is also at risk but there are more eyes on the prize, and they're working to mitigate this with reproducible/deterministic builds, which isn't on Alpine's roadmap and may be beyond their resources.
Though Alpine advertises a range of benefits the thing its dev community seems to obsess about the most is size. As small as possible.
Regarding the footprint, Alon showed you can slim down Debian so the footprint advantage is small. If that isn't enough we can take it one step further and use Debian Embedded to slim things down further by using BusyBox, and smaller libc versions, just like Alpine.
Choosing Alpine over Debian for this use case trades off people-oriented advantages that increase with value over time (skilled dev labour, bug hunters, mindshare, network effects) for machine-oriented advantages (storage and memory) that devalue rapidly thanks to Moore's Law.
I can see Alpine's advantages actually mattering in the embedded space circa 2000, but these days Debian runs fine on the $5 Raspberry Pi Zero computer, while the use case Alpine is actually being promoted for are servers with huge amounts of disk space and memory by comparison.
Maybe I'm missing something but doesn't that seem awfully short sighted?
OTOH, I can see how from Docker's POV, assuming bandwidth isn't getting as cheap as fast as storage or memory, and they're subsidizing petabytes of it, swinging from the fattest image to the slimmest image could help cut down costs. I bet Docker also like that they can have much more influence over Alpine after hiring its founder than they could ever hope to have over a big established distribution like Debian.
Summary of Debian pros:
- vastly larger dev & user community
- more packages
- more testing
- more derived distributions
- more likely to still be in robust health in 10 years
- working towards reproducible builds
- better documentation
- libc more compatible than musl, less likely to trigger bugs
- more trustworthy infrastructure
Summary of Alpine pros:
- lighter: community obsessed with footprint
- musl: more efficient libc alternative
- simpler init system: OpenRC instead of systemd
- lead dev & founder is a Docker employee
- trendy
Comments
I have both Turnkey and
I have both Turnkey and Alpine lxc images running at the moment.
I use Alpine when I need single services running in a container which are available on Alpine, but sometimes you run into problems with alpine, because of the musl library used. sometimes buidling from source does not work and on Debian it is no problem.
Its great to see that you trimmed done the Debian container to 12 MB, I am gonna try that out. If it is similar in size, my preference goes to Debian.... although I like the simplicity of Alpine
Alpine rcinit system I find much better for an lxc setup... simple and straightforward.
QUOTE: ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
any idea how to get this base
any idea how to get this base working in a vanilla lxc enviroment.
I am missing a /sbin/init usually a softlink to systemd, but i cannot find a systemd or systemctl
QUOTE: ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
That's a good question Hans
Can you please comment if or
Can you please comment if or when this tiny debian base would be available for LXC?
No it's for Docker
Please also note, that at the time this post was published, we were hoping to transition to this TKLX model for v15.0. But we've had to change tack and now v15.0 will be using the current monolithic build model. As soon as that is out, we hope to reboot the TKLX development. We'll probably continue to develop TKLX in parallel to the current appliance build method. Thus v15.x will remain monolithic builds, with TKLX development using proper semantic versioning v0.1. At this point the plan is to have TKLX v1.0 ready by the time Debian Buster (aka Debian 10) and so our TurnKey v16.0 release will be built using TKLX containers. Whether it works out like that remains to be seen, but fingers crossed...
TurnKey v16 using TKLX containers for Docker?
Hi Amaury
At this late stage (Buster has already been out for well over a month) we've had to reconsider and are currently pushing ahead with monolithic builds for v16.0.
We certainly aren't taking containerisation off the table, but we've decided that having a Buster base would be the required first step either way. Leaving our current users completely behind is not really an option... So even if we were to containerise everything, we still need a base for the users who run standalone servers, bare metal and local VMs, etc. Ideally I'd also like to be able to provide some upgrade path for existing users who would be open to containerising.
So the plan at this point goes something like this: Buster based Core ASAP (as per previous releases, probably an RC initially; which I'd like to think I'm pretty close to, although having said that, I've been saying that internally for a few weeks now...). Then update the rest of the library to Buster base. Then start to actively work on a container based release in parallel (not yet sure whether we'll just call it v17.0; or something else). Initially we'll likely focus on a few specific applications, but the grand plan would be to containerise all the applications, although we may well still continue to offer some monolithic appliances.
no problem... Its a
no problem...
Its a difficult decision, debian or alpine, the debian base is about 55 MB uncompressed, alpine about 6 Mb. So if you want containers for single services... almost nothing can beat Alpine... For libc dependencies I added the libc apks from https://github.com/sgerrand/alpine-pkg-glibc These seems to work ok with somelinux binaries I needed to run...
QUOTE: ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Stick to Debian. I noticed
Stick to Debian. I noticed that I am getting into problems when I needed perl modules which are not in the default distribution of Alpine. Then I have to use alternate ways in getting those modules, having all kind of problems with dependencies/different interface and update synchronisations.
I find that almost all used modules are existing in Debian, that probably is also the case for php or python or ...
The 40MB extra storagesize in that case is not a problem.
Also with Docker you can do a base image of TKL debian and then on top have the images of the services... so really with multiple services the extra space is no reason to switch. Personally i don't like Docker, I rather have containers running using a stock LXC which I can tailor to my needs... so keep on supplying those base images please...
QUOTE: ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
If you don't like Docker, RKT is also an option
Anyhow, we do plan on sticking with Debian and if necessary extending it to fit the use cases we're trying to solve rather than seeking a new starting point. Debian has the largest community of any Linux distribution and the network effects are powerful enough that I don't see how a temporary technical advantage by another distribution could possibly offset that.
The only downside of Debian
The only downside of Debian (for me) in a container at the moment is it uses systemd as init system.
If you don't want that, perhaps a look to Devuan is in order, which uses the same base, but give you the choice which init system you require.
QUOTE: ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Containers that run on bare metal, not VMs...
Hello! Did you think about building TKLX over SmartOS / Triton?
https://github.com/joyent/triton
Containers that run on bare metal, not VMs... promising re/ security & performances, see https://www.joyent.com/triton
I have a dream that one day every appliance would just deploy, run and backup on such a container-native infrastructure :)
OpenSolaris is dead, long live Illumos !
Had Sun moved to open source Solaris a decade earlier, I'd probably by writing this on an OpenSolaris base system. Unfortunately, once they realized Linux was disrupting their business it was too late to catch up. The train had left.
Once Oracle bought up Sun the future stewardship of Sun's open source efforts became even more uncertain. Oracle, true to their culture, didn't take too long to squash the fragile open source communities around Sun's "too late, too little" open source efforts.
Triton and sun's technology
Obviously you know nothing about triton or smartos, you do however know about illumos
"Once Oracle bought up Sun the future stewardship of Sun's open source efforts became even more uncertain. Oracle, true to their culture, didn't take too long to squash the fragile open source communities around Sun's "too late, too little" open source efforts. "
while it is true that oracle did squash opensolaris they did not however squash the opensource community as evidenced by illumos and it's child smartos. SmartOs is a hypervisor that yes is built out of the illumos kernal and leverages the illumos gate and pkgsrc repositories. There are several very much alive projects that spun out of the forked opensolaris codebase and many improvements many made by joyent.
Also, triton is a node.js and c based cloud system that uses SmartOS as it's hypervisor which provides three methods of getting things done.
1. Zones = similar to jails or even LXC or simply put containers
2. Docker = ported to opensolaris technology by joyent another container
3. KVM = ported to opensolaris technology by joyent for non unix/linux support
I don't think the other user actually was asking you if you would make TKLX using a opensolaris or illumos base but more rather developing TKLX to leverage the triple play hypervisor capabilities of SmartOS and the triton cloud itself. Which in my opinion blows away most other cloud systems as evidenced by the 1.8 billion dollar purchase of joyent's triton and smartos by samsung last month.
1.8 billion, i would hardly call that dead squahed by oracle little to late on sun's part. This move by samsung proves that this technology is alive and viable. You have the turnkey hub which uses amazon however triton is a complete opensource cloud system complete with it's solaris based hypervisor which you could use to break away from amazon and stand up your own turnkey cloud that out the box supports bare metal lx brand zones/containers, bare metal docker containers, and qemu-kvm vps.
I could be wrong but i do belive the other users was actually asking you have you considered using triton the fact it is open source and can be deployed easier then any other cloud, is used in the enterprise and it's triple play capabilities to define a new modal for TKLX in that you would create a base debian image as you already have make it run on smartos zones, smartos docker and smartos kvm. you already have docker covered which is compatible with triton, rtk is covered, iso, ec2 images so all you would need its a zone or lx brand image and a kvm and turnkey could be deployed to any of smartos's capabilities and be orchestrated by the triton cloud.
point being that it's a cloud based on opensolaris technology however you would not stop what your doing now and throw away debian in favor of solaris or illumos as it is the same concept as any other cloud and will run your debian based product on triton and smartos so really you would only convert for example your vmdk to kvm images and create a zone image your docker work will run out the box on the smartos hypervisor
just saying ... Oh and by the way not bashing i love what trunkey does and congrats on creating a really slim debian base i love it.
TKLX on SmartOS / Triton
Patrick has made my point. Thank you!
TKLX on SmartOS / Triton, that's just a dream... I didn't know it was worth a billion dollars :)
sorry actually 1.2 billion
sorry actually 1.2 billion not 1.8 my bad lol
You're right, I'm not familiar with SmartOS / Triton
The very strong endorsement from two community members indicates it may be wise to take a deeper look on whether this is something we could leverage for TKLX. I don't think I've encountered such enthusiastic support for any other hypervisor.
We'll still need to consider other factors besides the technology, such as the vitality of the open source community relative to other options, but that's mainly an issue if we're cornered into picking a winning horse. As a counter example of when that's not the case, the base images for TKLX support both Docker and RKT.
TKLX and triton would be nice
TKLX and triton would be nice and can be done easily.
TKLX micro-services architected using the container autopilot pattern using ContainerPilot for orchestration with Consul for service discovery and TKLX would be a nice platform that indeed would still run anywhere not just on triton.
That's gold open source isn't it?
Definitely worth having a look at :)
Triton DataCenter is an open-source cloud management platform
https://github.com/joyent/triton
ContainerPilot is an application-centric micro-orchestrator
https://github.com/joyent/containerpilot
SmartOS / Container hypervisor
https://www.joyent.com/smartos
We will!
Here is why I am enthused
Turnkey is a wonderful project and provides a solid platform for getting things done. Triton and SmartOS provide TKLX direction and architecture guidelines.
1. SmartOS as a hypervisor provides more deployment options then any other hypervisor on the market. Sure Xen, ESXi and the others work but they focus on one technology mainly Virtual Machines. Proxmox and some of the others including Xen and VMware are integrating docker or openvz containers but no other hypervisor provides bare metal access to Containers, Docker and VM's in one stack like SmartOS does.
2. Triton as a management platform and datacenter is based on node.js and C and it does not get any better or easier then this. Triton is well thought out and very opinionated and stands up gracefully without all the headaches of say deploying an openstack system. Triton is more then just a cloud with smartos as it's hypervisor it is a complete datacenter.
3. Using the container autopilot pattern http://autopilotpattern.io/ provides a clear architecture for developing TKLX micro services and even standard VM's using this in conjunction with containerpilot https://github.com/joyent/containerpilot which automates the process of discovery and configuration that's needed to connect application components in separate containers and Consul https://www.consul.io/ which makes it simple for services to register themselves and to discover other services via a DNS or HTTP interface. Using a scheduler like Docker Compose, Mesosphere Marathon or even Google Kubernetes, TKLX becomes more then just a bunch of container apps they become self aware and capable of scaling in the datacenter with fault tolerance and high availability.
4. architecting for triton does not make TKLX locked as the micro services will still run anyware containers, docker and vm's can be run on any cloud in any data center.
5. Because it's so easy to deploy triton the turnkey project can leverage triton to deploy it's own datacenters breaking away from amazon ec2. This cuts out any middle man and allows all monies collected to go to the project. Using server wholesale ISP's like OVH.com which rents cloud grade servers for super cheap in multiple datacenters worldwide it would be actually affordable to stand up a triton datacenter in a geographical distributed nature fast.
6. Building TKLX and further orchestrating Turnkey to leverage this technology coupled with a graphical drag and drop interface to triton would allow enterprises customers to visually build complete self aware networks with all the bells, for example building TKLX micro services for:
A. An Input Firewall
B. HA Proxy Load Balancer
C. NAS Storage
D. MySQL
E. Ruby on Rails
F. An output rules filter
in this example all of the above are TKLX containers architected using the technologies i mentioned. They are all self aware and created to work with a drag and drop interface and represent classes. these classes can be visually drag and dropped on a grid that represents what the user wants so say we drag a input firewall then wire that to a HA proxy that wires to three ruby on rails classes that share a NAS service and wire to a Mysql cluster that then connects to an output filter for internet access and the we press run and it uses one of the schedulers to stand up a complete firewalled, load balanced ruby on rails cluster with database clustering and NAS storage.
thats what triton and a little bit of love and coding can bring to TKLX. I have deployed a 3 node 12 core 24 thread 48 gig ram triton cluster in 1 hour flawless as my test lab.
only supports the v1 Docker Registry API...
Unfortunately, imgadm's support for pulling Docker images still only supports the v1 Docker Registry API. That is quite old at this point and many images in Docker Hub (i.e. docker.io) no longer support v1. Basic support for the v2 registry API is there in the 'node-docker-registry-client' that imgadm is using, but it would still take some coding work on imgadm to get it doing v2 pulls. See #644 for that work
SmartOS as a Virtualisation Platform
Hello! An interesting blog post from /proc/mind
31/01/2016
SmartOS as a Virtualisation Platform
Virtualisation platforms and technologies represent a big focal point of the technology scene these days.
Recently I’ve watched a dockercon 2015 presentation by Bryan Cantrill, CTO of Joyent, an OS kernel developer for 20 years and father of Dtrace as he calls himself [1], about how to debug docker containers gone bad in production. [2]
I recommend to anyone working or thinking of working with linux containers and docker especially to watch this presentation !
I have to say that this is one of the best presentations I’ve seen when it comes to showing the full picture of the docker tooling, lifecycle and ecosystem.
In his presentation, Bryan brings the operational point of view of running applications, in docker, in a production environment and how to deal with and debug failure when applications inside docker containers go wrong.
It is very rare to see a good presentation on the failure modes of docker since most presentation and talks focus on why docker is amazing and it will solve all your problems.
Irrespective of using docker or not, applications have bugs, they go wrong, it is very important to have adequate tooling and discipline to debug and improve them.
Towards the end of his talk Bryan shows some amazing tools and services that the Joyent team has built since 2004 when they’ve started their journey as a “cloud platform” company.
All these tools are built upon their platform hypervisor called SmartOS. [3]
The presentation plus the details I’ve read about SmartOS intrigued me, and I gave SmartOS a spin in a KVM virtual machine to see what it can do.
What is SmartOS
Disclaimer: I’m no authority on SmartOS, I’m relaying to you what I’ve found out about it until now.
Go and search for yourself to find out more.
Historically speaking SmartOS derives from the Solaris OS. [4]
A fork of Solaris, called OpenSolaris, was created in 2004.
After the Oracle aquisition of SUN Microsystems in 2010, a group of Solaris engineers created the illumos kernel [5] which was used subsequently to power OpenIndiana from which SmartOS sprang.
The Solaris kernel developers have started working on OS virtualisation since 2005, it looks like they are 10 years or so ahead of the Linux containers and it shows. [6]
SmartOS is not a general purpose OS, it appears to be designed from the ground up to run virtual workloads.
It is effectively a Read-Only (almost full RO) platform hypervisor running in RAM and managing different kinds of virtual workloads.
SmartOS can run these virtual workloads at the same time using the same tooling:
Because SmartOS is built on the powerfull legacy of Solaris zones, it has a very useful and powerfull feature compared to Linux containers: complete zone isolation !
From a security point of view SmartOS zones ( read containers ) are fully isolated, an attacker that has been able to gain root privileges in the zone cannot gain root access on the host hypervisor. [11]
I’ve heard that this is why the Joyent cloud runs containers on bare-metal, while other cloud providers like AWS or Google run containers in VMs.
Ramdisk is where SmartOS feels at home
General purpose OSes have to be installed on disk to function.
SmartOS on the other hand boots of and ISO or USB stick or PXE booted and it runs entirely in RAM. It has no installation to disk option.
Here are some arguments about why booting from RAM is a feature in SmartOS. [7]
The SmartOS hypervisor/OS, or what is called the global zone, [10] is mostly Read-Only.
I’ve seen recently in the Linux world this kind of approach by the people behind CoreOS. Surely they can draw more inspiration from the SmartOS/OpenSolaris developers.
How can anyone test it ?
I’ve tested it by using the SmartOS iso and booting it in a KVM VM.
I could have achieved the same thing by booting of the SmartOS USB drive.
If you have a type of virtualisation on your laptop/desktop ( KVM, Virtualbox, VMware …) than you can give it a spin in a VM. [8]
What can a user run on SmartOS ?
A user can run KVM VMs and SmartOS OS virtualisation zones.
Since I’m running SmartOS in KVM, even if I have enabled KVM passthrough on my desktop, I haven’t tried to run KVM VMs because the boot sequence of SmartOS says that KVM is not supported on my VM, therefore I’ve only been able to run zones.
SmartOS hypervisor
After booting from the iso image or the USB image, you’ll follow a few basic questions to setup networking and the ZFS pools in the global zone.
SmartOS global zone
Once that is done you’re all setup to start running virtual workloads.
Another very useful feature of SmartOS is that SmartOS treats all 4 types of virtualisation described above as the same thing:
All 4 types of virtualisation are created, lifecycle managed, and destroyed using the exact same tools:
That is it!
No more running docker ... or rkt .. for a container workload, then qemu-system-x86_64 or interfacing with libvirt for KVM VM, each coming with its own tool for creating, lifecycle managing and destroying the virtual workloads.
Disclaimer: all zones that I’ll show you how to start are started using the “admin” networking, which basically means they’ll all be in bridged network mode and you’ll be able to access them on your internal network as if they were other separate physical hardware.
SmartOS zone
Lets run another instance of SmartOS, as an isolated zone, lets say for SmartOS package building !
find SmartOS datasets
LX branded zone - full Linux userland
This type of virtualisation resembles to OpenVZ or LXC virtualisation in Linux, a full OS operating system running in a “container”
This time we’ll boot a full debian8 userland on the SmartOS kernel using the lx branded zone.
We’ll follow the same steps and use the same tools to boot into the debian8 zone as we did for the SmartOS zone.
LX branded zone - docker container
This is still an LX branded zone ( Linux userland on SmartOS kernel ) but it will boot and run a docker disk container from docker hub. [9]
The interesting part is that docker containers on smartOS appear on the network bridge like any other VMs if you launch them on the “admin” network.
Lets launch a docker container in a SmartOS zone:
Conclusions
SmartOS comes by default equiped with:
From the SmartOS wiki: [12]
I’m impressed by its virtualisation tooling consistency and by the OS feature set as a virtualisation platform !
To me SmartOS looks like the perfect virtualisation platform, one of the most advanced platform hypervisors OSes if not the most advanced platform hypervisor OS these days.
Resources and inspiration
http://www.procmind.com/blog/2016/01/31/smartos-as-a-virtualisation-plat...
Its working
Its working
What the side of the page?
Pages
Add new comment