Forum: 

LXD Image Server?

spammy's picture

Are there any plans to make TKL images available via a LXD compatible image server?

Jeremy Davis's picture

TBH I wasn't aware that LXD had any image servers. But a quick google demonstrates that indeed there are! :)

Do you know much about it? Like what requirements there are to have an app listed? From what I can see the "offical" LXD library images are all built by sripts contained within a specific GitHub repo. If that's the requirement then it's too hard IMO. If there is some other mechanism then it may well be possible...

Another thought is that perhaps if we can understand the requirements of a "LXD library" we could create our own?

spammy's picture

>Another thought is that perhaps if we can understand the requirements of a "LXD library" we could create our own?

Actually... that was exactly what I was asking for :). To be honest I only know what I learned from here:

https://linuxcontainers.org/lxd/try-it/

It appears that anyone can host LXD images; it's part of what it is, similar to Git. I suspect that you should be able to present images even more transparently than you do for eg OpenVz. In fact it'd probably be preferable for you to have your own image server rather than contribute to a public one, although I'm not sure what the resource/security implications are.

Jeremy Davis's picture

TBH I don't know when we'll get chance to look into this further but I've added it to the tracker (https://github.com/turnkeylinux/tracker/issues/670) so we don't forget about it. TBH though unless someone from the community looks into it further I don't imagine that it will happen any time soon...
bmullan's picture

Stephane Graber (stgraber) who is a lead developer for LXD/LXC described basically how to create a public facing LXD "image" server in a thread recently:

https://github.com/lxc/lxd/issues/1581

Although I've not done this (yet) myself I intend to in the future.

As LXD already easily supports both local/remote LXD servers via either the CLI or its REST API it seems to me from reading Stephane's response above that configuring a public LXD image server should probably be a very simple task.

Another user (erickeller) actually later in the thread described how he implemented his own public LXD image server using the steps Stephane provided and it sounds like he found it quick & simple to do.

sidenote -- yes I too think Turnkey could easily create LXD published images of its applications that any LXD user could then launch ...

example:  

lxc launch <some name>:<image identifier> <container name>

so if turnkey called their image server "turnkey"  and a user wanted to install say turnkey "wordpress" into a local container called "mywordpress" the LXD end-user command might be:

lxc launch turnkey:wordpress mywordpress

of if that user were managing local & remote LXD servers they could use the above command but specify which remote LXD server to launch the "wordpress" onto.   You could alsp specify what OS you want the container to be (centos, debian, ubuntu, fedora, alpine etc etc)

Brian

 

 

bmullan's picture

of course end-users would have to register that "turnkey" image server before accessing it the first time to "launch" images from it.

 

spammy's picture

Exactly what I had in mind, but I didn't quite grasp the fuller detail enough to present it. Thank you for the elaboration.

Jeremy Davis's picture

That's great! It does indeed appear to be fairly straight forward.

My only reservation is that it sounds like we will need an additional (LXD) server to host the images. Obviously that in and of itself is not a big deal. But then it's another server that needs to be maintained. The "weird" Apache config (that pretends to be an LXD server - as noted by Stephane) sounds like more what we'd want. Then we could just add an additional vhost to our existing webserver (i.e. for this website) that provides the images direct from our mirror. FWIW I just posted on the GitHub thread.

Regardless it's certainly on our todo list. Unfortunately though our todo list is very long so I have no idea when we might get to it.

Jeremy Davis's picture

And has provided some alternate info that sounds like it should be really easy. We'll still need to have a play with it and configure our webserver to use it, but we should be able to script the relevant info and generate the json files automatically.

I'm still not sure when we'll get to it, but it sounds doable. If one of you guys wants to have a play with it that'd certainly push things forward, although I can't guarantee when we'd be able to implement it. The less we need to do ourselves and the easier it is for us to implement, the sooner we can roll it out...

John Carver's picture

Hi Jeremy, I took a look at Stephane's suggestions and made an attempt to adapt a 14.1 lxc image to create an lxd container running on Ubuntu Xenial. I used the debian jessie files from the links Stephane provided and adapted them for a TurnKey appliance. (see the post below).

In addition to the two json files that Stephane described, you will also need to create a tarred metadata file with accompanying templates. LXD uses .xz compression by default, but gzip will also work. By convention, linuxcontainers.org uses lxd.tar.xz for the tarred metadata and templates, keeping all images in separate folders. I chose to use the same name as the original image except inserting a .lxd so that all files can be stored in the same folder.

.
├── debian-8-turnkey-tkldev_14.1-1_amd64.lxd.tar.gz
├── debian-8-turnkey-tkldev_14.1-1_amd64.tar.gz
├── metadata.yaml
└── templates
    ├── hostname.tpl
    └── hosts.tpl

The metadata file must be named
metadata.yaml

{
    "architecture": "x86_64",
    "creation_date": 1460431832,
    "properties": {
        "architecture": "x86_64",
        "description": "TKLDev - TurnKey Development Toolchain and Build System (20160411)",
        "name": "turnkey-tkldev-14.1-jessie-amd64",
        "os": "turnkey",
        "release": "jessie",
        "variant": "14.1"
    },
    "templates": {
        "/etc/hostname": {
            "template": "hostname.tpl",
            "when": [
                "create",
                "copy"
            ]
        },
        "/etc/hosts": {
            "template": "hosts.tpl",
            "when": [
                "create",
                "copy"
            ]
        }
    }
}

The accompanying template files will probably be the same for all appliances.  The supplied templates simply set the container.name into /etc/hostname and /etc/hosts whenever the container is created or copied. It's possible we could create other templates to perform additional functions.

templates/hostname.tpl

{{ container.name }}

templates/hosts.tpl

127.0.0.1   localhost
127.0.1.1   {{ container.name }}

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Stephane makes mention of two ways to implement the lxd repo, one using a customized apache server and the other using simplestreams.  I couldn't figure out if simplestreams was a requirement for second method or if it was just a convenient way to create and manage the json files.  There doesn't seem to be a package for simplestreams on Debian.

Information is free, knowledge is acquired, but wisdom is earned.

Jeremy Davis's picture

TBH this is really high on our agenda. I think it would be really awesome to provide, but not a priority for us ATM.

Still fantastic info you've shared here! Hopefully we'll get a chance to come back to this and give it some love! :)

bmullan.mail's picture

I know Turnkey builds LXC (lxc1) containers but there is a good external thread recently about "how to" convert original lxc1 to lxc2 (re LXD) containers.

https://lists.linuxcontainers.org/pipermail/lxc-users/2016-July/012000.html

Fajar Nugraha is one of the LXD developers and per his last statement on that thread:

There's a script to convert lxc -> lxd somewhere on this list, but I usually do things manually: 
(1) create a container in lxd. Start it, stop it, then look at its uid mapping (i.e. "which u/gid owns /var/lib/lxd/containers/container_name/rootfs") 
(2) use fuidshift with "-r" to shift your lxc container u/gid back to privileged, using the starting u/gid value in your original lxc config (should be 951968) 
(3) use fuidshift again, but this time without "-r", to shift your lxc container to unprivileged, using the starting u/gid value from (1) 
(4) move your new lxd container's original rootfs somewhere else (or delete it if you want), then replace it with rootfs from (3) 
(5) start your lxd containers 

So if anyone has the interest to migrate an LXC container to LXD so you can then take advantage of LXD to manage it locally or remotely, use CRIU (live migration) and some of the other nice features LXD added to the original LXC (lxc1)

The "script" that Fajar refered to may be this one:

https://github.com/lxc/lxd/blob/master/scripts/lxc-to-lxd

Brian

 

John Carver's picture

I recently started looking at LXD as an alternative to Proxmox for a development environment. I simply can't lug my home development server when I hit the road.  I'm looking for a way to run multiple TurnKey images on my laptop for a self-contained development environment. I tried using LXC for this purpose a few years ago, but never quite got it to work. The good news is that LXD is compatible with the TurnKey images for Proxmox 4.x. 

Following are my user notes from testing on a laptop running Ubuntu 16.04 (Xenial).

LXD Users Guide

LXD is designed to work alongside LXC and provide a simpler user interface for managing LXC containers.

Installation:

$ sudo apt-get install lxd lxd-tools
$ newgrp lxd 

Initialize LXD:

Run the following command, selecting the default settings, except for the creation of an IPv6 network.

$ sudo lxd init
Name of the storage backend to use (dir or zfs) [default=dir]:
Would you like LXD to be available over the network (yes/no) [default=no]?
Do you want to configure the LXD bridge (yes/no) [default=yes]?
Warning: Stopping lxd.service, but it can still be activated by:
  lxd.socket
LXD has been successfully configured.

Working with LXD:

$ lxc image list
+-------+-------------+--------+-------------+------+------+-------------+
| ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCH | SIZE | UPLOAD DATE |
+-------+-------------+--------+-------------+------+------+-------------+
$ lxc image copy lxc-org:/debian/jessie/amd64 local: --alias=jessie-amd64
Image copied successfully!
$ lxc image list
+--------------+--------------+--------+----------------------------------------+--------+---------+-----------------------------+
|    ALIAS     | FINGERPRINT  | PUBLIC |              DESCRIPTION               |  ARCH  |  SIZE   |         UPLOAD DATE         |
+--------------+--------------+--------+----------------------------------------+--------+---------+-----------------------------+
| jessie-amd64 | 06b11f7b270f |   no   | Debian jessie (amd64) (20170104_22:42) | x86_64 | 94.05MB | Jan 5, 2017 at 5:40pm (UTC) |
+--------------+--------------+--------+----------------------------------------+--------+---------+-----------------------------+
$ lxc launch jessie-amd64 jessie-01
Creating jessie-01
Starting jessie-01
$ lxc list
+-----------+---------+---------------------+------+------------+-----------+
|   NAME    |  STATE  |        IPV4         | IPV6 |    TYPE    | SNAPSHOTS |
+-----------+---------+---------------------+------+------------+-----------+
| jessie-01 | RUNNING |                     |      | PERSISTENT | 0         |
+-----------+---------+---------------------+------+------------+-----------+
$ lxc stop jessie-01 --force
$ lxc list
+-----------+---------+---------------------+------+------------+-----------+
|   NAME    |  STATE  |        IPV4         | IPV6 |    TYPE    | SNAPSHOTS |
+-----------+---------+---------------------+------+------------+-----------+
| jessie-01 | STOPPED |                     |      | PERSISTENT | 0         |
+-----------+---------+---------------------+------+------------+-----------+
$ lxc delete jessie-01
$ lxc list
+-----------+---------+---------------------+------+------------+-----------+
|   NAME    |  STATE  |        IPV4         | IPV6 |    TYPE    | SNAPSHOTS |
+-----------+---------+---------------------+------+------------+-----------+

Importing TurnKey Image:

The goal is to be able to import and run TurnKey images under LXD. Fortunately the 14.x images created for Proxmox LXC format are compatible with LXD. Following the examples by Stéphane Graber, we must manually (for now) create some metadata to accompany the TurnKey image. First download the desired image from the proxmox directory on mirror.turnkeylinux.org, for example debian-8-turnkey-tkldev_14.1-1_amd64.tar.gz. Try to use a method such as rsync that preserves the original creation/modification dates.

$ rsync -av -P rsync://mirror.turnkeylinux.org/turnkeylinux/images/proxmox/debian-8-turnkey-tkldev_14.1-1_amd64.tar.gz ./

Determine the creation_date from the last modified date of the image file.

$ stat -c%y debian-8-turnkey-tkldev_14.1-1_amd64.tar.gz 
2016-04-11 22:30:32.000000000 -0500
$ stat -c%Y debian-8-turnkey-tkldev_14.1-1_amd64.tar.gz 
1460431832

20160411 date the image was created. 

1460431832 date and time the image was created in epoch format. 

architecture is x86_64 for an amd64 image. amd64 may also work for the architecture.

Create a subdirectory "templates" and the following three files in yaml format. Change the name, description, etc. corresponding to the appliance image.

metadata.yaml

{
    "architecture": "x86_64",
    "creation_date": 1460431832,
    "properties": {
        "architecture": "x86_64",
        "description": "TKLDev - TurnKey Development Toolchain and Build System (20160411)",
        "name": "turnkey-tkldev-14.1-jessie-amd64",
        "os": "turnkey",
        "release": "jessie",
        "variant": "14.1"
    },
    "templates": {
        "/etc/hostname": {
            "template": "hostname.tpl",
            "when": [
                "create",
                "copy"
            ]
        },
        "/etc/hosts": {
            "template": "hosts.tpl",
            "when": [
                "create",
                "copy"
            ]
        }
    }
}

templates/hostname.tpl

{{ container.name }}

templates/hosts.tpl

127.0.0.1 localhost
127.0.1.1 {{ container.name }}

# The following lines are desirable for IPv6 capable hosts 
::1     ip6-localhost ip6-loopback 
fe00::0 ip6-localnet 
ff00::0 ip6-mcastprefix 
ff02::1 ip6-allnodes 
ff02::2 ip6-allrouters

Create a tar file of the metadata and templates for LXD. The name could be anything, but I chose to name it similar to the original image.

$ tar -czvf debian-8-turnkey-tkldev_14.1-1_amd64.lxd.tar.gz metadata.yaml templates/*

Now import the image into LXD.

$ lxc image import debian-8-turnkey-tkldev_14.1-1_amd64.lxd.tar.gz debian-8-turnkey-tkldev_14.1-1_amd64.tar.gz --alias turnkey-tkldev_14.1_amd64
$ lxc image list
+---------------------------+--------------+--------+--------------------------------------------------------------------+--------+----------+-----------------------------+
|           ALIAS           | FINGERPRINT  | PUBLIC |                            DESCRIPTION                             |  ARCH  |   SIZE   |         UPLOAD DATE         |
+---------------------------+--------------+--------+--------------------------------------------------------------------+--------+----------+-----------------------------+
| turnkey-tkldev_14.1_amd64 | 20c8484e42bf | no     | TKLDev - TurnKey Development Toolchain and Build System (20160411) | x86_64 | 197.68MB | Jan 6, 2017 at 6:48pm (UTC) |
+---------------------------+--------------+--------+--------------------------------------------------------------------+--------+----------+-----------------------------+

Next you can create (launch) a container running the image.

$ lxc launch turnkey-tkldev_14.1_amd64 tkldev
Creating tkldev
Starting tkldev
$ lxc list
+-----------+---------+---------------------+------+------------+-----------+
|   NAME    |  STATE  |     IPV4            | IPV6 |    TYPE    | SNAPSHOTS |
+-----------+---------+---------------------+------+------------+-----------+
| tkldev    | RUNNING | 10.76.85.211 (eth0) |      | PERSISTENT | 0         |
+-----------+---------+---------------------+------+------------+-----------+

Caveats:

Notice that launch both creates and starts the container unlike LXC which had separate commands. There is no method for loading an inithooks.conf file, but the init sequence creates one on the fly with a random password. Since there is no /dev/console, the turnkey-init is not triggered and must be run manually. Connect to the container using the 'exec' command.

$ lxc exec tkldev bash
root@tkldev ~# turnkey-init
...

Information is free, knowledge is acquired, but wisdom is earned.

brian mullan's picture

John Nice writeup. Just fyi but Fajar works with Stephane Graber & Serge Hallyn on lxd. In any case simplifying migration of traditional lxc1 to lxc2 (lxd) is good news given lxd's many advanced capabilities such as image publishing, criu, zfs & btrfs cow support etc Thx Brian
John Carver's picture

Hi Brian. I didn't see your preceding post because I was busy composing my post. :)

I think Fajar's script addresses a different use case, i.e. converting or moving an existing lxc1 container to an lxd container. My effort was in exploring how to use an existing proxmox (lxc1) rootfs image to create the lxd container. Admittedly I'm just learning to use LXD, so I may not be aware of all the possibilities. I haven't tried to use a 13.x (openvz) image with LXD but I suspect it would not work. I think you would first have to create a lxc1 container using the lxc-turnkey template, and then use Fajar's script to convert to lxd format.

Information is free, knowledge is acquired, but wisdom is earned.

brian mullan's picture

John... boy what a coincidence for 2 related posts by 2 diff people on this subject w/in same day. :-) By the way if you use reddit I started both an lxc and lxd "subreddits" a year or so ago. Lots of related technologies posted in each you may find useful and interesting. A recent one in the lxd subreddit described an innovative use of overlayfs with lxd rootfs to greatly simplify container OS updates at scale and to save even more space on disk than lxd already does thru sharing the hosts kernel.
John Carver's picture

Brian, I've run into a problem trying to run TKLdev appliance in a LXD unprivileged container. Inside the container, mount is being blocked by AppArmor even though I'm root. I've been looking for an example of how to create a profile that allows mount to operate within the container but not outside for security. I haven't found anything helpful so far. Do you have any ideas?

mount: permission denied
fatal: non-zero exitcode (32) for command: mount -t aufs -o 'dirs=/turnkey/fab/bootstraps/jessie,udba=reval'  'none'  'build/bootstrap'
/usr/share/fab/product.mk:476: recipe for target 'build/stamps/bootstrap' failed
make: *** [build/stamps/bootstrap] Error 1

dmesg shows a number of these errors.

[360426.345856] audit: type=1400 audit(1483730835.496:72): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-tkldev_</var/lib/lxd>" name="/sys/fs/pstore/" pid=7328 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
[471499.216660] audit: type=1400 audit(1483841915.104:76): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-tkldev_</var/lib/lxd>" name="/" pid=25654 comm="mount" flags="ro, remount, relatime"

Information is free, knowledge is acquired, but wisdom is earned.

bmullan.mail's picture

John..  you are correct re "I think you would first have to create a lxc1 container using the lxc-turnkey template, and then use Fajar's script to convert to lxd format."

 

Junaid Shahid's picture

yeah he is right, I have tried that and it's worked.. !

brian mullan's picture

See here for step by step conversion. Its really pretty quick & simple. https://www.reddit.com/r/LXD/comments/5nk5re/how_to_convert_an_lxc_lxc_v...
Junaid Shahid's picture

Hey thanks, I have check this, that's really cool :)

John Carver's picture

After I completed work on the 14.2 updates for the LXC appliance, I turned  my attention back to LXD.  I soon found that although images created using the process I outlined above initially worked, there were nagging issues like not being able to stop a running container or use mount within a running container.   After far too much experimentation, it finally hit me that some of the issues were caused by not applying the container patches included in the lxc-turnkey template and others such as mounting in an unprivileged container required the additional changes outlined by Brian Mullan.

My first thought was to just install LXD on the 14.2 LXC appliance and script the conversion process. To my surprise, I found that development of LXD for Debian is lagging and it likely won't be ready for the upcoming release of stretch. That means a fully supported LXD appliance would have to wait until TurnKey 16.0 release (Debian 10). Stéphane Graber has outlined an alternate procedure for installing LXD on Debian stretch using snaps.  If Jeremy, Alon, and Liraz are agreeable, this might be a path to develop a 15.0 LXD appliance.

In the meantime, I plan on scripting the download and conversion on my Ubuntu laptop because I really want to have a portable development environment.

Information is free, knowledge is acquired, but wisdom is earned.

Jeremy Davis's picture

Your suggestion of using an LXD snap to provide a v15.x LXD host appliance sounds like a legitimate plan to me. Obviously I'll need to discuss it with the other guys, but on face value I don't see why they wouldn't be open to that.
brian mullan's picture

John

There is a new Discourse site to discuss LXD related technology.

If you run into any obstacles ask a question there & I am sure you will get some assistance.

https://discuss.linuxcontainers.org/
 

Brian

 

bmullan.mail's picture

John

I just ran across this and you might find it interesting in regards to migrating LXC (v1) containers to LXD (lxc v2 containers)...

thread about script to migrate LXC v1 containers to LXD (lxc v2) containers

 

John Carver's picture

Thanks for the link to the lxc-to-lxd script. Actually I had found it earlier in my research and have had some good success in using it to create LXD images.  I've been planning to write a new post about how I created a TKLdev workstation on Ubuntu 16.04 using LXC and LXD.  The process starts by using lxc v1 to download a TurnKey Proxmox image to a v1 container. Some additional tweaks are performed and then the lxc-to-lxd script is used to convert to a v2 container. The normal LXD commands are then used to create a v2 image. I haven't yet written a script for the process, but that's the obvious next step.

I did have an issue with DNS name resolution which I only figured out today. I couldn't figure out why the LXD container names were not being resolved by dnsmasq as they are with the LXC containers. It finally dawned on me that the hostname used within the container defaults to the appliance name, not the container name. When the container is first started, an IP address is assigned to the initial default name.  For example, when I create a new container, linuxgeeks, from a Drupal8 image, the hostname for the container will be drupal8.lxd not the desired linuxgeeks.lxd. I had a dickens of a time trying to change the hostname and get dnsmasq to resolve properly. I think the hostname must be changed in three places, /etc/hosts, /etc/hostname, and /etc/network/interfaces; then networking restarted 'service networking restart'. The dnsmasq service on the host may also need to be reloaded. I think it may be possible to change the hostname before starting the container so the correct name is added at the start.

Information is free, knowledge is acquired, but wisdom is earned.

John Carver's picture

After doing some more testing yesterday, I realized that the DNS hostname resolution problem was caused by my forgetting to include metadata.yaml and associated templates before building the LXD image. I found that I needed to add three templates, one for each of the files where the hostname needs to be changed. With the templates in place, the hostname is changed to the container name as expected and dnsmasq answers DNS queries properly. Copying or renaming the container should also change the hostname.

Now to figure out a better way to handle inithooks.conf.

Information is free, knowledge is acquired, but wisdom is earned.

Jeremy Davis's picture

And there are plans to allow the confconsole plugins to be leveraged via commandline (and therefore could be preseeded via inithooks).

Hopefully that will be ready by the time we release v15.0 but I can't promise anything...

Post new comment