BuildTasks - first run, issues with LAMP applicance in TKLDEV

I must have been replying to your initial post when you posted these updates. I'll respond individually to them.

Yep, if you run:

./bt-vm turnkey-lamp-15.1-stretch-amd64

It will look for an iso named "turnkey-turnkey-lamp-15.1-stretch-amd64". So as you've noticed, the command is instead:

./bt-vm lamp-15.1-stretch-amd64

so, having now tried to run

./bt-iso ..."as in

./bt-vm turnkey-lamp-15.1-stretch-amd64"

I get:"  

root@tkldev /turnkey/buildtasks# ./bt-vm turnkey-lamp-15.1-stretch-amd64
WARNING [bt-vm]: --publish was not specified
--2019-09-27 00:30:57--  http://mirror.turnkeylinux.org/turnkeylinux/images/iso/turnkey-turnkey-l...
Resolving mirror.turnkeylinux.org (mirror.turnkeylinux.org)... 131.188.12.211, 137.226.34.46, 2801:82:80ff:8000::e
Connecting to mirror.turnkeylinux.org (mirror.turnkeylinux.org)|131.188.12.211|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2019-09-27 00:30:57 ERROR 404: Not Found.

/turnkey/buildtasks/bin/iso-download: line 51: aws: command not found
FATAL [iso-download]: turnkey-turnkey-lamp-15.1-stretch-amd64.iso download failed
"

I just removed the "turnkey" from the invoke command and it looks like it is running now.

Wahoo.  On to the next steps  :)

I'm not 100% sure why you hit that issue with bt-vm. It appears that there is something still running inside the chroot where the VM was being built. If you wish to retry, you can remove the existing v15.1 LAMP related stuff from the VM build directory. I.e. like this:

rm -rf /mnt/builds/vm/turnkey-lamp-15.1-stretch-amd64*

Then try again. Hopefully you will have better luck the second time.

At what seemed the end of the process I got the following:

"[ ok ] Stopping LVM2 metadata daemon: lvmetad.
Installing for i386-pc platform.
Installation finished. No error reported.
INFO [vm-bundle]: creating fstab
INFO [vm-bundle]: umounting the jumble...
INFO [vm-bundle]: cleaning up volume groups and mounts
FATAL [vm-bundle]: Process systemd-udevd locking volume group; unsure how to proceed"

..............

several files built in /mnt/builds/vm

I just also ran xen.  Seemed to complete.

The first error appears to be because it's trying to download the hash file for LAMP v15.2 and is failing (because it doesn't exist on the mirror). It then tries to fallback to downloading from the build bucket but the AWS cli tool isn't installed (but even if it was, that would have failed too, because there is no public access to the TurnKey build bucket).

However, my reading of the buildtasks source code (explicitly the iso-verify script in this case) suggests that it should see that the hash file that already exists, and skip downloading it. So I'm not really sure what has gone wrong there...

Is this a fresh TKLDev that you've just installed? If so, then it should have the latest code. If it's an older one you've had laying around for a while, please try updating buildtasks. Do that like this:

cd /turnkey/buildtasks
git checkout master
git pull origin master

To double check that you are on the latest commit, try this:

git log -1

That should return this:

commit 628ce81e5fb6a9a37e3718aac6ea2e2ee97770d4
Author: Jeremy Davis 
Date:   Thu Aug 1 12:22:31 2019 +1000

    tidy up debugging

TBH, I'm not 100% clear on what you are trying to do, but there are a few things worth noting:

If you wish to build alternate builds with buildtasks, you can use the existing TurnKey releases from the mirror (no need to run bt-iso first, it will download them from the mirror).

If you do wish to build your own custom LAMP, then the first thing you'll want to do is add a new changelog entry (which I'm guessing you did when you built the 15.2 one). When built with bt-iso, the ISO will be named:

turnkey-APP_NAME-VERSION-DEBIAN_CODENAME-ARCH.iso

where APP_NAME is the name of the directory where the code is (e.g. if you clone the lamp repo to /turnkey/fab/products/lamp - then the APP_NAME will be 'lamp').

VERSION in the version number at the top of the changelog. E.g. without changes, bt-iso this changelog will build v15.1.

DEBIAN_CODENAME should always be "stretch" (at least for now). And ARCH will always be amd64 (we currently only support 64 bit builds - which despite the name, works fine on Intel processors too).

All the v15.1 files that you have in your /mnt/isos directory look ok and would be what you'd expect for a successful ISO build. The 'turnkey-turnkey-lamp-15.2-stretch-amd64.iso' file doesn't look right?! My guess is that perhaps the directory was/is named "turnkey-lamp", rather than just "lamp"? The 15.2 build looks right, too, but I'm wondering if you just renamed those files, or if they were built with buildtasks? If you just renamed them, then the hash file not working (and buildtasks trying to download it) are expected behaviour. This is because the contents of the hash file will likely be incorrect.

I suggest that you try this:

First double check that you have added a new changelog entry so you are building v15.2:

head -1 /turnkey/fab/products/lamp/changelog

To build a v15.2 ISO, that should return the following:

turnkey-lamp-15.2 (1) turnkey; urgency=low

If it doesn't, then try editing the changelog and add a new entry. You can read all about our changelogs in the TKLDev docs.

Then cd to buildtasks and run bt-iso:

cd /turnkey/buildtasks
./bt-iso lamp

Assuming that runs successfully, then afterwards you should find in /mnt/isos the following files:

turnkey-lamp-15.2-stretch-amd64.changelog
turnkey-lamp-15.2-stretch-amd64.iso
turnkey-lamp-15.2-stretch-amd64.iso.buildenv
turnkey-lamp-15.2-stretch-amd64.iso.hash
turnkey-lamp-15.2-stretch-amd64.manifest
turnkey-lamp-15.2-stretch-amd64.tklbam

The only important files at this point are 'turnkey-lamp-15.2-stretch-amd64.iso' & 'turnkey-lamp-15.2-stretch-amd64.iso.hash'. Then assuming that you wish to then convert that ISO to some other build (let's assume Xen) then run like this:

cd /turnkey/buildtasks
./bt-xen lamp-15.2-stretch-amd64

I hope that helps.

As I hinted in my previous post, you can generally get an idea of the latest release, from the version at the top of the changelog. Although sometime when we're doing updates, the changelog may have been bumped, but the new version is not yet on the mirror (that's not likely to happen anytime soon, at least until we release v16.0RC1).

FWIW, all the other appliances and build types should build in exactly the same way. If you want to change the appliance itself, that is best done within the appliance code repo (or in common as relevant). As you've likely guessed, to change the individual build types, edit the scripts in buildtasks.

Whilst git may be a bit of a learning curve, it is really cool IMO and well worth learning at least the basics. The beauty of it is that you can hack around and make changes and commit them. Then later on, you can revert back to the original code (or just undo one particular change, etc). It's a very cool thing IMO...

Good luck with it, and if you have any more questions, please ask! :)

The 15.2 entered when I thought I had to name the file with the "proper name"...  (including stretch etc).  Turns out, in the Mirrors anyway, that Lamp is only built to version 15.1.

It came together in the end.

This is a brand new Hub TKLDEV Microserver so it should have current build code.  I will study your notes in more detail.

Basically this was all an exercize for me in that I wanted to get BuildTasks running and go through the process on something simple (LAMP).  

At present I  have not done any changes.  A bit nervous about what might happen if I clone a distro and start to change it more..  will there be searches on the Turnkey Mirrors or can the independence of the new build allow a new VM to also be built. (?)

All of these things are like anything.  It is important to practice in the beginning and document as you go.  At least I try to do that.  

Next task is refreshing up on Git processes.  Intuitive I will say not.  :)

Thanks again for your help Jeremy.