ChrisB's picture

I'm trying to revive the old Bitkey.io project and build it using TKLDev.  I understand there are issues with the move from 14.2 -> 15 -> 16, but notwithstanding that, the makefile for it uses the /turnkey/fab/common/mk/turnkey-desktop.mk file, which references things such as turnkey.d/apt, turnkey.d/bashrc and so on, all of which don't actually exist in the common/overlays/turnkey.d directory.

Am I missing something?

The bitkey make process fails with the error:

fab-apply-overlay /turnkey/fab/common/overlays/turnkey.d/apt build/bootstrap;
error: does not exist: /turnkey/fab/common/overlays/turnkey.d/apt

How can I get this to make with the turnkey-desktop configuration?

Thanks,

Chris

 

Forum: 
Tags: 
Jeremy Davis's picture

It'd be awesome to get BitKey working again, but LOTs has changed in both Debian and TurnKey since v14.x (based on Debian 8/Jessie). The reason why a v15.x (based on Debian 9/Stretch) release never occurred was because the guy that started working on it, got bogged down with Debian changes/issues in our minimalist install (I suspect that it was missing packages or new config required, that we weren't including). I imagine that issue is probably still outstanding.

Beyond that, we've also made significant changes in the TurnKey build infrastructure itself. These are most likely going to take a fair bit of work to bring BitKey up to date. Whilst there were some TurnKey changes between v14.x & v15.x, most of the changes were actually in Debian itself. Most of the TurnKey specific changes occurred between v15.x and v16.x and some were quite significant.

The particular issue you're hitting right now is the renaming, refactoring and move of the old apt conf script to bootstrap_apt.

But I anticipate once you fix that, there will be lots more...

Probably what might help get you started is looking for the changes that we've made in common since the 15.x tag. Here you can see a consolidated view of all changes since 15.x. Obviously not all of them will be relevant to you (and actually many of them won't be relevant at all). Regarding the specific turnkey-desktop.mk file, I suggest that you look at what we changed in the turnkey.mk file.

I suspect that one of the biggest changes that will be relevant to BitKey is a rebase of the TurnKey "Live" system on default Debian components. Prior to v16.0 we were building our own busybox-initramfs package and leveraging an old Ubuntu live boot project; casper. Now we use live-boot and live-tools direct from Debian.

It's perhaps also worth noting that we have ported many of our packages to python3, but much of TKLDev (e.g. fab) is still using python2.

Bottom line is that it's going to be a fair bit of work, but I'm happy to assist as best I can. Please keep in mind that I'm still fairly busy tidying up the last few v16.0 appliances (among other things) so I may not always be able to give exhaustive insight...

ChrisB's picture

Hi Jeremy,

Thanks for your reply.  Sounds like this will be a task - what I'd like to do is break it down a bit and see if we can't start with something more basic first, as a starting framework.  How about using tkldev 16 to produce a live linux that has no networking (completely offline), no USB or other peripheral support (don't need to plug in anything else than the live USB to boot), and a simple desktop like BitKey currently is, with a dock containing just a text editor and Chromium (for eventually running HTML/JS apps within), and maybe terminal.  That's it, 3 apps.

Is that achievable?  All the examples seem to be for headless stuff.  Is there a desktop appliance I can take as a framework to begin with and build on?

After that, the plan would be to extend it with HTML/JS apps for crypto wallets, much like <code>cryptojsappslauncher/offline.html</code> now.  Keeping it simple with more modern tools that are available now.  Maybe other dedicated apps for wallets (Electron etc) can be added in due course, but for now, a basic desktop that runs Chromium would be a good starter.

I like very much the simplicity of the old BitKey layout.  A few apps in the dock, no other fluff.  That's what I need to start with.  How do I achieve that?

Thanks for any pointers!

Chris

Jeremy Davis's picture

Breaking a big "problem" down into smaller chunks is always a good idea. Although in this instance, unfortunately we don't have any other "desktop" type appliances for you to look at.

FWIW we did always intend to provide our own minimalist TurnKey "desktop" distro, but in the end the idea stalled because the interest seemed limited. We had also considered a "security focused" minimalist live distro, but Tails appeared and gained a lot of traction pretty quick (confirming it was a good idea, but we got beaten to the execution). There is clearly interest in BitKey, but we're a small team with a mile long "todo" list related to the headless servers we provide and as BitKey is essentially a "side project" it hasn't received the love that it deserves...

It's also worth noting that a community member, Steven did a fair bit of work maintaining and extending the (default) "community" branch of BitKey. As I think I may have hinted previously, he also started work on a Stretch based (i.e. v15.x) version. I wouldn't necessarily recommend that you try to complete that, but I'm sure that it's at least worth checking out. One part that you won't need to worry about is the python update (to v3.7) - IIRC that was for a particular app that required a newer version than Debian Stretch had at the time (v3.5). Debian Buster (basis of v16.x) has v3.8 so that shouldn't be an issue.

Anyway, back to the matter at hand. If you look a bit closer under the hood, the TKLDev build process essentially has 3 main components, "plans", "overlays" and "conf scripts". The make files are just a nice way of consolidating sets of overlays and conf scripts (plans are handled somewhat separately).

The "plan" provides a list of packages to install. Note that only the listed packages and their hard dependencies (i.e. "depends") are installed. If you want recommended dependencies (i.e. "recommends") then the relevant package names need to be explicitly listed in a plan. In the case of the desktop plan you'll need to ensure that the packages required to make it all work are included in a plan.

As the name suggests, "overlays" are files that are overlaid over the filesystem (relative to '/'). E.g. the desktop overlay.

Again, as the name suggests, conf scripts are executable scripts that run at build time to configure specific components. As this occurs after packages have been installed, technically these scripts could be in any language supported by the OS being built. In practice, they are all shell (sh) or bash scripts (IMO bash scripts are preferable).

It might also be worth having a bit of a look through the TKLDev docs. Unfortunately, they're far from complete; some bits out of date and some bits a bit confusing... However, there is still some gold in there if you can be bothered...

So to circle back to a process to get this working...:

I suggest that you start by modifying the desktop make file (i.e. turnkey-desktop.mk to fix the initial issue you noted. I've just had a quick look through and I think that should be all that's required to get that base component "working".

The next thing will be modifying the plan(s). Once the above make file is fixed, then next issues you'll hit IMO will be packages that no longer exist and/or have been renamed.

You could just comment out the package names that cause issues, until it successfully builds. Then worry about what to replace them with (or if they need to be replaced at all) later. Alternatively, you could research each one as you go. Here are some resources which I imagine you will find invaluable (FWIW I managed the v15.x -> v16.0 migration, so have a fair bit of first hand experience).:

FWIW note that the plans are written in (and parsed as) C language format. So full line comments are a double forward slash at the start of the line (i.e. '//') - inline comments are forward slash, asterisk at the start of the comment and asterisk, forward slash after (i.e. '/* COMMENT */').

Once you have the plans sorted out, then you may find some other issues with conf scripts that break. I generally tend to comment out problematic lines initially. In my experience, overlays rarely cause build time issues, although can obviously cause issues later.

Once you have the build process completing, you should get an ISO. I suggest testing that in a VM and see if you can boot into live mode. If it fails, I find that a screencast of the boot process can be an invaluable tool to troubleshoot the issue (often the lines go by too fast to read them, if you record them, you can play them back slowly to read them).

I can't guarantee how much I'll be able to help out, but I'd love to see an updated BitKey. So please don't hesitate to ask and I'll do my best to help out. It may also be worth posting your interest/intent on the BitKey tracker. Who knows, perhaps there might be others keen to get involved too?!

ChrisB's picture

Hi Jeremy,

Thanks again for your detailed reply.

I have made first steps.  I have indeed reviewed the tkldev docs over the last 3 days in an effort to understand this a bit more.  However, I'm a relative novice in Linux (Mac background with some webserver experience on linux) so bear with me...

I modified turnkey-desktop.mk to change references to turnkey.d/apt to bootstrap_apt instead, and commented out all other turnkey.d items.

make then results in errors finding various packages in the plan elements: iproute (changed to iproute2), myspell-en-us (changed to hunspell-en-us), consolekit (removed), libpam-ck-connector (removed), dconf-tools (changed to dconf-cli and dconf-editor), luks (removed), theme-zukitwo (removed), mrxvt (removed) and usbmount (removed).

make clean and make on that finally runs quite a long way, until stumbling here: perhaps you can point me in the correct direction to rectify?  Seems related to core?

Thanks,

Chris

Jeremy Davis's picture

As you suspected (and note below) the issue is that the Changelog is malformed (it needs to be formatted as per Debian specs.

Out of interest, you if you use dch to edit the changelog (contained within the devscripts package - i.e. install devscripts in your TKLDev).

insserv is a init.d (SysvInit - i.e. the legacy initialisation system) ordering script, which is no longer relevant with systemd (the new/current initialisation system). It's not installed in v16.x appliances. So that line shouldn't be running and you can safely remove it. You'll find it in the turnkey-desktop.mk file.

ChrisB's picture

Hi Jeremy,

A further update: I think I was using the 14.2 bitkey version on previous post in error, so I scrapped that and used the stretch branch from Steven.  I think I'm tantalizingly close to getting an ISO built, but it trips as below image shows on the fab-chroot build/root.patched "which insserv >/dev/null && insserv" line.

For info, all I've done (aside from modify tkldev turknkey-desktop.mk as above to use bootstrap_apt) is take out a load of packages from the plan to strip it right back, including commenting out all the python3 stuff.  Would this affect the ability to generate the ISO?  See image below if it helps.

(I think the changelog was causing my issue in previous post - I had to make sure the product name was on the first line and not the author of the changes).  This changelog I've used turnkey-bit-key-16.0.1 as the name, hence you see that referenced in the image below.

I can provide more detail on what I've done if necessary, just need a steer first to check I'm on the right track...

Cheers,

Chris

Jeremy Davis's picture

Removing packages will (obviously) have impact on the software installed in final ISO. Whether or not it will also have an impact on the ISO build process, will depend on whether commands/files from the package are used during build time.

To use the 'insserv' issue as an example: Whilst the inserv package and that command are no longer needed, it doesn't actually do any harm. So instead of removing that line from turnkey-desktop.mk; you could just (re)include the 'insserv' package in one of the plans that are installed. It's because the build process is trying to call a command from a package that is no longer installed, that the build chokes on that line.

I hope that makes sense.

ChrisB's picture

If it helps, changes I made are here:

Buster BitKey commit changes first try

Chris

Jeremy Davis's picture

looks like you are getting close to getting an ISO to build! Hopefully removing the insserv line from the turnkey-desktop.mk file might even be enough to get you over the line...

Probably the next tricky part will be getting it to boot. I may have mentioned already, but to assist troubleshooting the boot of the ISO, I suggest testing in a VM and recording the boot process (e.g. with some sort of screen recording software) so that you can review the boot messages. Or perhaps you'll be lucky and it will boot ok?!

ChrisB's picture

Hi Jeremy,

Thanks for your helpful pointers.  I had assumed insserv was vital to tkldev as it was in the turnkey-desktop.mk file, but commenting that line out indeed builds product.iso!

Mounting that in a VM as you suggest, it offers the splash screen and upon choosing bitkey offline option (or any option), it complains /casper/vmlinuz:file not found so I changed the menu.cfg to reference live instead of casper, and all mounts OK and fires up.

However, after the initial bitkey welcome info box, the desktop is black with something on the right hand side with a strange icon on it - hovering the mouse cursor over this suggests it's clickable but doesn't do anything.

I'm guessing that the wbar setup is not happening, nor any of the splash screen bitmaps symlinked as desired.  Digging around I see there is a  bitkey/overlay/usr/share/initramfs-tools/scripts/casper-bottom/90bitkey file that seems set up to initiate all this stuff from the /usr/lib/bitkey.d directory.  My guess is that, as casper is no longer used, this is not getting called, hence the basic, useless desktop at present time.  (There are no errors that I can see in the bootup process).

I'm afraid I don't know enough to solve this myself - any clues as to how this part of the boot up process can be run properly?  How can I modify things to ensure the /usr/lib/bitkey.d scripts get run at the right time to set up the desktop appropriately?

Thanks for your help.  Screenshot of how it boots currently is attached.

Chris 

 

ChrisB's picture

Tried changing casper-bottom to init-bottom, no change.  Same result. (Following only bit of info I could find on the options from http://manpages.ubuntu.com/manpages/xenial/man8/initramfs-tools.8.html)

Maybe I'm on the wrong path here?

ChrisB's picture

Commenting out . /scripts/casper within 90bitkey of init-bottom fixes things.  Boot up ok!

Thanks for your helpful pointers along the way, Jemery.  I will now attempt to start building a list of software packages to put into the new bitkey.

Chris

Jeremy Davis's picture

It's been a while since I did the transition from casper to live-boot, but IIRC the casper-bottom dir should be renamed live-bottom. That way it should only run when in "live" mode; the init-* dirs should be run on everyboot. Although I could be wrong...

I assume the line you are referring to is this one. In which case it should be changed from:

. /scripts/casper-functions

to:

. /scripts/functions

Regardless, it looks like you are having some solid progress...

ChrisB's picture

Hi Jeremy,

I've managed to get the JS offline apps working, together with a working shutdown icon and chromium installed properly.  My github buster branch refers.

What I'd like to do now is make some changes locally within my BitKey instance by downloading stuff from the internet and installing locally to test things work, before then modifying the build accordingly.

The problem I have is the internet connection doesn't work on my new BitKey whereas on the old v14 BitKey, it does.  Below are 2 screenshots, one from each version.  My new BitKey is giving "Network is unreachable" errors.  Both are VMs running in VirtualBox with the same Network settings.

Any idea what I'm missing as far as v16 config is concerned to ensure internet connectivity via the VM host?  I haven't modified any of the plan/net packages (except iproute -> iproute2).

A secondary issue is that of the machine name: my build is showing user@tkldev whereas the v14 BitKey shows user@bitkey.  How can I change the build to use "bitkey" as machine hostname too?

Thanks for any pointers.

Chris

 

 

 

Jeremy Davis's picture

I've managed to get the JS offline apps working, together with a working shutdown icon and chromium installed properly. My github buster branch refers.

Great work! :)

What I'd like to do now is make some changes locally within my BitKey instance by downloading stuff from the internet and installing locally to test things work, before then modifying the build accordingly.

Yes, very sensible approach! You can then install stuff in your test bitkey and then add them to the plan.

The problem I have is the internet connection doesn't work on my new BitKey whereas on the old v14 BitKey, it does. Below are 2 screenshots, one from each version. My new BitKey is giving "Network is unreachable" errors. Both are VMs running in VirtualBox with the same Network settings.
Any idea what I'm missing as far as v16 config is concerned to ensure internet connectivity via the VM host? I haven't modified any of the plan/net packages (except iproute -> iproute2).

Ok, so looking at your screenshots, I think that there are a few things going on here. But before I start suggesting fixes/workarounds, it's worth being aware that TurnKey (and therefore BitKey) uses a legacy network config. It hasn't been the recommended way to do networking for some time. We still use it for our headless servers because it's easy and continues to work. However, it may not be the recommended way to do things in a desktop environment anymore. I don't know for sure, but I suspect that desktop UI tools may no longer support the old way of doing things?!

So first thing I note is the interface in the new BitKey is enp0s3 (vs eth0 in the old one). This "new" naming scheme is known as "predictable network interface names". We disable that by default via the conf script: common/conf/turnkey.d/grub-iface-naming.

If you wish to try to transition to the newer networking paradigm, then you almost certainly don't want to disable this as we do. If you want to try to just use the legacy networking (which may or may not work with GUI tools such as Wicd, but should work fine via commandline) then it's recommended to use the legacy interface names.

FWIW you can find more info about the Debian networking config in Debian manual - chapter 5. There is also a Debian wiki page on Networking but I can't say how up to date that might be (Debian wiki pages can be a bit hit and miss re how up to date they are...).

So our default config uses the /etc/network/interfaces file and my guess is that it only explicitly notes the loopback device and eth0. That is likely configuring DHCP for the interface.

At some point we're going to need to decide whether we update networking for BitKey or whether we just persist with the legacy setup. I guess to some degree it depends on whether Wicd can still cope with the legacy set up... For now though, to just get your networking working, you could edit the interfaces file and replace 'eth0' with 'enp0s3'. Then try this:

ifdown enp0s3
ifup enp0s3
A secondary issue is that of the machine name: my build is showing user@tkldev whereas the v14 BitKey shows user@bitkey. How can I change the build to use "bitkey" as machine hostname too?

My guess is that for some reason it's picking up the TKLDev hostname at build time. I vaguely recall that we explcitly do that somewhere to workaround an build time issue. To fix that, just add this line to one of your conf.d script like this:

echo bitkey > /etc/hostname
ChrisB's picture

Hi Jeremy,

Thanks for your pointers: got the hostname changed ok.

Regarding the networking: /etc/network/interfaces had only "auto lo" entry (nothing else for eth0 or any other) so I needed to add:

auto enp0s3
iface enp0s3 inet dhcp

and then run sudo /etc/sbin/ifup enp0s3 in order to get internet connection in the VM.

A couple of questions you may be able to assist with on this:

  • Is there any way of automating this entry in the hot-online option so I don't have to edit the file manually each time it live boots?
     
  • Is there a way of entering the correct interface in the file depending on what the distro is running on (my VM is as above but for my MacBook Air with WiFi adaptor, ip address show lists wlp2s0b1 instead of enp0s3)?  I guess some form of parsing "ip address show" in a script to build /etc/network/interfaces and exec the ifup, but I wouldn't know where to start with that.
     
  • Is there a way of force-disabling all networking when booting in cold-offline mode?  It kind of already is like that but malicious software entries could re-enable networking as above automatically if not careful, so force disabling networking is vital for a truly safe cold-offline experience.

Thanks again for your time!

Chris

 

Jeremy Davis's picture

To answer your questions:

Is there any way of automating this entry in the hot-online option so I don't have to edit the file manually each time it live boots?

Yes, but not without modifying (and/or rebuilding) the ISO. Just add it as an overlay or edit the interfaces file from a conf script.

Is there a way of entering the correct interface in the file depending on what the distro is running on (my VM is as above but for my MacBook Air with WiFi adaptor, ip address show lists wlp2s0b1 instead of enp0s3)? I guess some form of parsing "ip address show" in a script to build /etc/network/interfaces and exec the ifup, but I wouldn't know where to start with that.

Possibly, but I'm not exactly sure how you'd do about that. I assume that there is probably a way to add scripts to get run as part of the network config.

The easy way to do that is to follow our lead from the headless servers as disable the "predictable network interface names" (wlp2s0b1 and enp0s3 are these so-called "predictable network interface names"). On our headless servers, that gives you the legacy 'eth0' rather than 'enp0s3', etc. However, have said that, 'wlp2s0b1' looks like a wifi interface and I suspect that 'eth0' won't cut it...

Besides the likely wifi issue, as I already noted, I'm not 100% sure whether that will work with Wicd (or other UI networking tools). So any/all of my suggestions above are likely only temporary hacks that will allow you to focus on other aspects for now.

Bottom line it seems likely to me that the whole networking issue will need to be resolved properly at some point. So perhaps it's worth doing a bit of reading as per the links I provided in my last post and fixing properly now?!

After having a closer look at the links I provided, it appears that the wiki is no help. However, have a look at Debian manual - sec 5.2.1 - GUI network configuration tools. That has notes regarding Wicd which will hopefully be of value?! It may also be necessary to (re)configure the DHCP client too. You can find info about the config file via the dhclient.conf man page (AFAIK BitKey has isc-dhcp-client installed as a DHCP client).

Is there a way of force-disabling all networking when booting in cold-offline mode? It kind of already is like that but malicious software entries could re-enable networking as above automatically if not careful, so force disabling networking is vital for a truly safe cold-offline experience.

TBH, I'm not particularly familiar with BitKey but by my understanding is that originally when booted as "cold-offline" all networking tools were purged (i.e. uninstalled and config removed) and all networking kernel modules (i.e. drivers) were unloaded (perhaps even blacklisted?).

Note that whilst that would stop any user level malicious software from re-enabling networking, if the malicious software could gain root access (or was running as root) then all bets are off. There is not really any way to stop root from undoing anything that you can do.

However, if the disabling was done as root at boot time (as it should be) and sudo was also removed and as a final step root was disabled, then a limited user would not have any mechanism to re-enable/re-install that stuff. So that would work pretty well.

Actually how you would best go about that, I'm not completely sure though. My suspicion is that previously it was done via the 'casper-scripts'. If so, it should be possbile to re-implement via 'live-scripts'.

I'm sure there are other options too. E.g. I suspect that it'd be possible to leverage systemd and set up custom systemd targets (e.g. "cold-offline" etc) to boot to. Systemd targets are essentially sets of services to run at boot and can be specified as kernel commandline options (which can be included in the boot menu options). Although unless you are familiar with systemd, that may be a convoluted method with a lot of trial and error... (I'm speaking from my experience with the early v16.0 development cycle).

ChrisB's picture

Hi Jeremy,

I've managed to get net.ifnames=0 to work, but only by specifying it in the kernal command line options of menu.cfg of isolinux.  I couldn't get it to work by using your `grub-iface-naming` conf script in common/conf/turnkey.d and enabling it within the mk file.  The end ISO would boot up still with the new interface names rather than eth0/wlan0, but the `etc/default/grub` file shows it has been correctly modified by the grub-iface-naming script.  Strange.  So I specify in menu.cfg directly and that seems to work.

Next I wanted to ensure all sudo access when running in cold-offline mode was removed, and wrote a quick script to remove the `%sudo ALL` line from /etc/sudoers.  This works ok.

I'm having trouble now forcing iptables to deny all network traffic.  Again for cold-offline mode, I want to ensure that iptables disallows all inbound and outbound traffic whatsoever (belt and braces) at boot time, the theory being that in conjuction with removing sudo group from /etc/sudoers, the threat of opening up any network connection without users knowledge is reduced.  I tried to do this as part of a script that runs within `overlay/usr/lib/bitkey.d/` (and called by `overlay/usr/share/initramfs-tools/scripts/init-bottom/90bitkey`) by using iptables-legacy -F and iptables-legacy -A INPUT -j REJECT (same for OUTPUT and FORWARD also) but on booting the ISO and screen recording, it seems the kernel is not ready for iptables (or iptables-legacy) at this point.  It prints the error message `iptables v1.8.2 (legacy): can't initialize iptables table 'filter': Table does not exist (do you need to insmod?)`.  I am wondering if you know of a way to implement these iptables restrictions an alternative way at boot time?  The key is to ensure they are made before anything else can run to undo them as sudo.

The script I've written is here: offline-ensure

Any ideas welcome and thanks in advance.

Chris

Jeremy Davis's picture

Sorry it didn't occur to me that the live functionality doesn't boot from grub! That's why the grub.cfg changes didn't work and you needed to make the changes in menu.cfg (the ISO and therefore the live functionality, boots via isolinux/syslinux)...

Now that you have disabled sudo access when running in cold-offline mode, I'd be inclined to put that aside for now (i.e. disable it/comment it out/etc). Not having access to sudo will make troubleshooting the iptables issue (and others) much harder.

Also as something of an aside, you can remove 'sudo' and any other packages that you don't want users using in cold-offline as part of the start up process too. E.g.:

apt-get purge -y sudo ANOTHER_PKG ANOTHER_PKG2 ...

Going back to the iptables issue, the message that you note suggests (as you've possibly already realised) that the required iptables kernel module (essentially a driver) isn't loaded. The first thing I'd check is to see if it is loaded after boot (hence why I suggest ensuring that sudo still works for now). It may be that it's just not loaded at that point in the boot cycle (and you'll need to run your command later in the process) or perhaps it's not being loaded at all? (In which case you'll need to explicitly load it)

Also there may be other ways around it. I'm not sure, but perhaps it's possible to blacklist all networking drivers so networking isn't possible at all? I also thought that there may be a kernel commandline option to completely disable networking, but I couldn't find anything other than disabling ipv6... AppAmor may be another option (IIRC it's enabled by default, but isn't configured that strictly). Also as per above note about removing packages, all the required networking tools could also be removed?! And one final note, some methods of disabling networking may have unintended consequences! That is because some software may rely on loopback networking!

Anyway, sorry that I don't have any really solid advice for you here... :(

PS oops... I just looked at your script and I can see that you HAVE commented out the sudo disable line... :)

Add new comment