Arthur Summers's picture

I have an idea that I think would be great for both the Turnkey Linux project and for the Raspberry Pi Foundation. I just posted on the following thread (my username is summersab):

https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=101859

Without reiterating my post too much, what if TKLDev were used to define and create RPi images? Would it be possible (without too much effort) to port Turnkey images to an ARM architecture? Could these two projects somehow work together to co-host available images? I think that by doing so, it would bring attention to both projects, help bring on a lot of new users, and generate an interest in creating new spins/flavors.

Thoughts?

Forum: 
Jeremy Davis's picture

I like your thinking and we'd love to do it... Actually building RPi images (and/or supporting ARM in general) has been on our todo list for some time. I even started work on it last year but hit a wall as I was not able to get my initial image to boot on a Pi (I kept getting stuck on the "rainbow" screen). I got called away to work on the v14.0 release; which then flowed into the v14.1 release. I've nearly finished with that but there are other priorities which demand my attention before I will get a chance to get back to it. So unfortunately it is mothballed and I have never had time to try to work out what my issue might have been.

So I am willing. In fact I'm more than willing; I'm keen! But I have no idea when I might get some time to get back to this...

If you have an interest and would like to have a play yourself, then all of the components required should be on GitHub. Although it may require building RPi compatible packages for the TurnKey custom software (which could turn into a bit of a chicken and egg type scenario).

I also dumped all my notes and code snippets in a repo in my github. It's a horrible mess (and TBH a little embarrassing) but I didn't want my experience to go completely to waste.

I imagine that without quite a bit of background experience and knowledge of TurnKey the learning curve will be very steep. Some of the software that we use internally is on GitHub but not well documented. There may also be some missing ingredients (due to the curse of knowledge) but I don't think so.

If you (or anyone) wants to try and push this ahead I will provide as much support as I can.

Arthur Summers's picture

I replied on the RPi forums to keep the discussion alive over there and hopefully get some interest from their development team. However, I had some thoughts that I thought would be more of interest to you.

While I understand that the x86 and current Turnkey development is a priority for you and your team, I wanted to encourage you to possibly consider putting some effort into ARM/RPi development. The reason is twofold. First, ARM is becoming a more and more common processor, and having images available opens up an entire world of possibilities for Turnkey. Second (and more importantly), the RPi, embedded systems, and IoT platforms are becoming a HUGE deal. If TKLDev became the de-facto image development and creation platform for the RPi, imagine how much traffic and attention Turnkey would get. I'm regularly surprised to find that my infrastructure colleagues don't know that Turnkey exists. I remember reading this post by Liraz Siri two years ago, and his first point was simply: "[TKL is the] best kept secret of the internet." TKLDev for the RPi may very well be the key to solving this marketing problem.

You mentioned in your post over on the RPi forum that you already have a substantial amount of distributed hosting. I don't know how much of your budget is spent on this, but considering that the RPi Foundation has support and funding from companies like Google and Microsoft, perhaps a relationship with them would be a strategic partnership that would help with your resources and overhead. If you consider my previous point (making TKLDev become the RPi tool of choice), you may be able to work with these corporate sponsors to secure some heavily discounted (maybe free?) hosting and storage. I'm sure that would certainly help free up funds and allow you and your team to work on other backlogged efforts.

That's all I have for now. If you wanted to take this offline as PMs or emails, let me know, and I'd be glad to discuss further.

Jeremy Davis's picture

You make some great points and I agree that an RPi port has a lot of potential for raising awareness of TurnKey and even perhaps engaging more developers. So you can be sure that my lack of attention to this is not a lack of interest or lack of enthusiasm. It is simply a lack of immediate resources (mostly my time and/or access to labour). Also perhaps a little bit that I really want to work on it myself (even though I don't have the time ATM)! :)

Our extensive global mirror network is provided to us gratis. So whilst officially partnering with the RPi Foundation would be awesome, it wouldn't necessarily (at least not initially) provide a direct and immediate "payoff". Don't get me wrong, we're not purely driven by having a "payoff" but we do need to pay the bills and keep the lights on. And whilst we have a fantastic community that do a ton of work to make TurnKey awesome for free; the bulk of the work is done by a very few people (who we don't want to burn out). Obviously being formerly involved with RPi Foundation would help raise awareness and hopefully be a boost to our community if nothing else. But currently we have other things on the go that have some sort of immediate dividend.

So we need to clear the plate a little before we can push this any further (directly) ourselves.

Arthur Summers's picture

Poo. This was supposed to be a reply to your comment, not to the main post. I don't see a delete button. Oh well...

Understood on all fronts. I think that in order to take a serious stab at this project, we really need to have one of the Rasbian or NOOBS developers/contributors involved. There are a good deal of kernel tweaks that have been made to squeeze out maximum performance from the RPi, and while they are all documented, reinventing the wheel is pretty much always a stupid idea. I'm looking at their GitHub repos and trying to figure out who might be the best people to contact. Getting the "leadership" from both projects talking so that this can be a collaborative effort would be much better and more efficient than trying to do things in a silo.

In my opinion, I think the priority should be 1) porting TKLDev to the RPi, 2) creating the TKLDev scripts necessary to build Rasbian, 3) building RPi ports of TKL images, and 4) compiling TKLDev with QEMU. I know that's almost the reverse of what you were working twoard, but I think that taking steps that cater more immediately to the RPi community will help develop interest, and that interest may very well come with resources. That said, I'm not sure what the LOE would be for each of these efforts, and quick wins are always worth considering first.

Any other thoughts from the rest of your dev team, or is it pretty much consistent with your previous message?

Jeremy Davis's picture

There is perhaps a little "chicken and egg" stuff going on here, but your suggested workflow/prioritisation is very interesting and I think you raise some valid points on how we could do things.

The biggest bonus I see of your suggested workflow is that it wouldn't require a massive chunk of our energy upfront. We could do our bits incrementally and the community could play with it in the meantime! Very cool!

We (as in TurnKey) would (probably) still need to do some ground work e.g. we'd need a bootstrap at least and a buildroot would be useful; but they probably wouldn't take a huge amount of work.

TKLDev with QEMU would be required for us to build the whole library but wouldn't be necessary if building on an RPi itself.

I haven't spoken with the other members of our "core" TKL team about this specifically, although I must say that your timing is quite impeccable! :)

Liraz (one of the co-founders) internally circulated an email recently stating the case for an RPi port. Around the same time I have been discussing "what next" priorities for me post v14.1 release. And I was super keen to work on RPi and he suggested that perhaps I could do some work on it... Even if I could put a solid day on it I think that I could probably come up with something worth sharing...

Arthur Summers's picture

While I believe that a TKLDev with QEMU would be the best long-term direction, it's probably also the most complicated. Most RPi users would probably be more interested in using TKLDev on their RPi to create and build images for flashing (at least at first). It would probably be a pretty slow process, but for novice users, it's logically more straightforward to have everything occur on the same platform. (I can hear all of the confusion about "chroot? What's that?")

That being said, perhaps the very first step after discussing more with the TKL team would be reaching out to the RPi developers and people in the Foundation (I'm not seeing that the forums are going to get the necessary attention, unfortunately). While I'm sure you could get a bootstrap in pretty short order, there have been a lot of tweaks specific to the hardware. For instance, Debian doesn't (or at least didn't) support ARM's hardware-assisted floating point unit, and it took a good deal of development to get it working. Also, there's the GPIO mess. While the board uses pretty standard hardware, there have been tons of tweaks to squeeze out performance and keep the kernel small. I'm sure it's documented, but it would be a headache to track it all down and figure it out yourself.

There are two directions. On one hand, you could contact the leadership of the Foundation. They're probably more responsive, but they're also likely a bit less technical and may brush you off. On the other hand, you could try your luck pinging the lead contributors to the GitHub repository. They might be less responsive, and it's harder to tell who is "officially" involved with the project, but they would better understand what you're talking about and needing. Perhaps take both approaches in tandem? Either way, having a good elevator pitch that concisely explains TKL/TKLDev, what it could bring to the RPi, and what you need is a must. Since most people aren't familiar with TKL and don't know how well-polished TKLDev is at building ISOs, you need to get their attention without sounding like a bad infomercial, if you get what I mean.

Let me know how I can help. At the very least, I'm really good at tracking down contact information, and I could pull together a list of people to reach out to.

Jeremy Davis's picture

Apologies on another delayed response...

Regardless of the specific path we take, we'd be looking to use Raspbian as the base OS for TurnKey RPi images. As you may not be aware; Raspbian is a RPi specific fork of Debian so all the RPi specific tweaks etc would be included in any TurnKey RPi images. FWIW the Debian ARM architecture release doesn't work with an RPi anyway (although from what I've read it may work with RPi2 & almost definitely would work with RPi3). So whilst some of the changes we'd need to make to support RPi would be relevant to other ARM boards, mostly they would be unique to RPi.

You are right I think that getting in touch with RPi Foundation peeps &/or devs &/or Raspbian devs etc would be a really good place to start. We may get some support & assistance and really have nothing to lose. Worst case they all say no and we are where we are right now...

Also for the record, I have discussed this whole idea further with Alon (one of the TKL core devs and a co-founder) and he is happy for me to have some involvement; although obviously not as a priority.

If you are willing to take the time to assist with gathering some contact details of good people to speak further with that'd be super cool. Also if you'd like to help draft a letter (email) that doesn't sound too infomercially that'd be great too! :) I generally like to keep things out in the open and public but perhaps for the "letter drafting" stage we could work on that via email if you want? Also sharing contact details via email may be a little less invasive for them. Once we're done though perhaps we could publish an open letter (to RPi hackers everywhere...) on our blog too once we're done? Just throwing ideas around...

Arthur Summers's picture

From your last message, I gather that the path you'd want to take is tweaking Raspbian to create TKL and TKLDev instead of porting TKLDev to the RPi directly and then using it to create Raspbian and RPi versions of TKL images? I imagine that would be much easier from the forefront, but I don't know how complicated and specific TKL/TKLDev code may be. It depends on which system is more complex: Raspbian or TKL. If there are more intricacies to Raspbian, then use it as a base for TKL; if TKL has more intricacies, then port TKL to the RPi and tweak according to the hardware. Does that make sense?

As an analogy, this reminds me of a few months back when I tried to make my own mini desktop OS by starting from the very base Debian and installing packages as necessary. However, just the core of Debian doesn't even include a lot of networking tools I took for granted, and it was hell trying to figure out what I needed. In the end, I took the opposite approach - I took an XFCE image of Debian and started stripping out the fat. However, there's a chance that I didn't catch everything that could be removed to speed up system performance. The first method would have been the most efficient while the second is just quicker. Porting TKL to the RPi is like the first scenario while starting with Raspbian is like the second scenario (at least from what I know, but I don't know how complex TKLDev is).

I'll go looking for pertinent contacts and get them to you via email (agreed with not making it all public at this point). After that, we can work on how best to approach them. I'm a little busy for the rest of the week, so I may take a little time getting back to you.

Jeremy Davis's picture

I was contacted (privately) by a guy a couple of days ago who is working on porting TurnKey to RPi. He doesn't want to make his work public yet as he wants something that at least sort of works first (which I can understand). His primary goal is actually to port BitKey first (it was a side project from a couple of years ago). Even though Bitkey is built with TKLDev it sounds like his approach is slightly different so it probably won't quite get us over the line; but it will be a great start. Hopefully he comes up with something. FWIW he is following the path I originally took and using QEMU to emulate similar ARM hardware to a RPi.

Essentially Raspbian is sort of a fork of Debian - somewhat like Ubuntu is sort of a fork of Debian too. The difference is that Raspbian is aimed at specific hardware (whereas Ubuntu is just doing things differently). Vanilla ARM Debian does not work (properly) on RPis (at least not on RPi1) so Raspbian must be used. But TurnKey used to be Ubuntu based, and Raspbian is also Debian based. The significant difference here is architecture; although even that isn't completely new to TurnKey (we used to be a 32 bit only release). Having said that; 64 bit can handle 32 bit builds no problem; ARM is totally different (hence the need for QEMU). Another difference is that instead of an ISO we need to build an image that can be copied direct to an SD card. But we (almost) have that covered too. Our VM images are very similar to the type of image we'd need to produce to write straight to an SD card and boot... So many of the pieces are there; they just need some tweaking...

My original plan was to use QEMU to virtualise ARM hardware and create the RPi builds in that. But as I mentioned I had issues. I also saw the value in your suggestion to actually just get it running on RPi for starters. I think that's possibly a quicker easier way to go (although it would not be conducive to us mass producing all the existing images).

FWIW your analogy is not bad. The way that TKLDev works is essentially builds the whole OS from the ground up. Mostly it's just Debian but we add a few things extra; and of course the software on top... You could theoretically use TKLDev to build a custom desktop distro of Debian if you wanted. And with some tweaking you could even use Ubuntu instead. So the end goal would be to use TKLDev on amd64 hardware (then we can mass produce the RPi builds when we build all the other appliances).

But the alternative is to just work straight on the RPi. It'll still be a ton of work, but it would mean that one piece of the puzzle could be put aside, at least for now. Anyway, this may all be a moot point as I don't have the time to work on it myself ATM, and it sounds like this other guy is planning to go more along the original route that I started...

No problem. Just get in touch when you can. And thanks tons for getting this ball rolling and helping out! Talk more soon. :)

Arthur Summers's picture

Jeremy,

Sorry I've been so quiet for the last few months. I've been working on getting a startup of mine off the ground, and it has consumed a fair bit of my time. After spending part of my weekend following three different "how-to" guides for getting a RPi print server set up and wishing that there had just been an image available, I thought perhaps I should check in. Has there been much progress with the guy from Bitkey or TKLDev on the Pi in general? Have you heard from anyone at the RPi Foundation? I could always scour around a little to find some people from over there end who might be interested. Let me know!

Jeremy Davis's picture

I think that the TurnKey RPi port has stalled (again) :( . The person I mentioned did get BitKey ported to RPi and he sent me a copy. However he hasn't published how he created it so we can't replicate it. Besides, I don't think that his methods would directly transpose to the other appliances as BitKey is a different use case.

FWIW we have the "issue" noted on GitHub. And we have put up a $200 bounty on it on BountySource.

Arthur Summers's picture

Perhaps reaching out to people at the Foundation would be a way to go, at this point. I noticed this page that at gives two names for the developers of Raspbian and the email address of one of them:

https://www.raspbian.org/RaspbianTeam

Past that, most of the key people are easy to find on LinkedIn, so that may be a good way to approach them if an email address can't be found (I'm not bad at scouring the net to track people down, though). Would you be interested in reaching out? If so, I'd be glad to help wordsmith things, but I imagine it would make a bigger difference if the email came from someone with an @turnkeylinux.org email address.

Jeremy Davis's picture

I have recently spoken with plugwash (Peter Green) regarding a separate matter but perhaps I will float it with him? He's a busy guy though (AFAIK he is the main maintainer for Raspbian). To contract him to do the work will cost way more than $200!

But perhaps we should approach the RaspberryPi Foundation (AFAIK they are not involved with Raspbian at all, other than endorsing it). Perhaps the RaspberryPi Foundation (or one of their affiliates) would be interested in sponsoring development? Then we could either offer a higher bounty or prioritise development ourselves (depending on what sort of sponsorship arrangement we could develop)?

I'd really love to see this move forward but due to competeing priorities I'm not in a position to drive it myself. If you would like to collect the relevant contact details and draft an email then I am happy to send it. I suggest that we do that part offline, rather than publicly. Email me direct if you want: jeremy AT turnkeylinux.org

Arthur Summers's picture

Hey, if you already have a relationship with Peter (at least to some degree), that's certainly a start. I'll gather up some names and contact info for you, but you might float the idea to him and see if he could facilitate an introduction to people he knows at the Foundation.

Yes, I'm sure contracting the work would be more than a $200 task, but if the right people were involved, we demonstrated that the effort would be mutually beneficial, and Turnkey Linux helped host the images, that would certainly make people more willing to work on it. Peter might be interested just for the sheer fact that it could drastically simplify his workload as a maintainer. At any rate, I'll get you an email in the next few days . . .

Arthur Summers's picture

I was going back through my posts and our conversation, and I forgot that someone mentioned this site:

http://piimagehub.com/about-me/

How about you bring up the idea with Peter, explain how it could simplify his work, and see if he knows people at the Foundation, and I'll reach out Ryan from this site? Shoot me an email when you get a response, or I'll email you if I hear back from Ryan first: arthur [dot] summers [at] gmail.

Arthur Summers's picture

Have you ever used Qubes OS or have any experience with it? That might be an interesting avenue for collaboration, too . . .

Jeremy Davis's picture

TBH I find the right-shift to replies a little off putting once a conversation goes on for a while, so i will reply to both your posts here...

Re contacting Peter, yes I'll shoot him an email and let you know how that goes.

Re contacting Ryan, I agree Pi Image Hub looks pretty cool and it would be great exposure (once we have images). He may be able to assist with contacts in the meantime?

Re Qubes, I have seen it and think that it's a very cool concept. I haven't actually used it though. I'm not exactly sure how we might collaborate...

Jeremy Davis's picture

Peter said that he didn't know of anyone that would be interested in being involved or helping out unless we can pay. :( I guess this makes sense as most people that are capable are busy on their own projects, whereas those that are the most keen for TurnKey style systems don't have the knowledge/skills...

His only suggestion was the RPi forums.

Arthur Summers's picture

And not booty in a good way. Does he know people at the Foundation who might help or have contacts? I know that they don't maintain images, but perhaps they would be intersted. TKLDev + RPi really seems like a match made in heaven that could organize and push the maker community forward. I'm surprised something similar hasn't happened yet. I haven't had any response from Ryan from the Pi Image Hub, but I plan on pinging him in a few days just to follow up.

Jeremy Davis's picture

I explicitly asked him for contacts, or to pass on my email to anyone he thought might be of value, and he had no suggestions (although perhaps he handed my email on; I'm not sure).

He did show some interest in being involved if we can pay, but that was it. TBH I somewhat understand, he's a busy guy (doing a PhD and AFAIK the main contributor/maintainer of Raspbian, not to mention all the support he gives on the RPi forums)) so he probably doesn't have a lot of additional energy to give (without a cash incentive). I was grateful that he took the time to respond and really don't want to hound him, but it leaves us at a dead end in that direction...

Unless we can find a sponsor and/or raise enough cash to pay someone, I think the guy that updated BitKey is our best option. He has already expressed interest in the $200 bounty but is really busy and I think has moved onto other things. Maybe if we can raise some more cash (to add to the $200 bounty) we could entice him to take it on?!

Jeremy Davis's picture

Whilst I'd love to see RPi support, I don't have time or energy to pursue this too heavily at the moment. But if you are happy to do the legwork, then I'm happy to send the email.

TBH, I'm a bit sceptical of whether or not anyone would be that interested in doing this for us for free. I thought that if the RPi Foundation was interested in supporting it, then perhaps we might get some traction, but my experience to date suggests that's not something they're interested in pursuing.

As I'm pretty sure I mentioned earlier, I think we're stuck in a place where those that have the skills don't have the interest. Those that have the interest, don't have the skills.

The only way I really see this being able to move forward is if we can come up with some cash to pay someone (who has the skills and knowledge) to do it. We'd be potentially interested in fronting up some more cash (i.e. add to the $200 we've already put up) but considering that no one else has contributed to the pool, we've assumed that it's not something that people are super keen to push forward (and $200 is clearly nowhere near enough to entice someone to take it on).

Jeremy Davis's picture

Yannick (a TurnKey community member) has built "experimental" TurnKey RPi builds. If any of you can, it'd be awesome if you could test these out and see how they go. Please post any feedback in that thread.

Please note that these builds only support RPi 4 (none of the other models, RPi 400 is a RPi 4, built into a keyboard).

Add new comment