Jeremy Davis's picture

This is a thread to discuss the (ongoing) work of developing a new install method for the next (v15.2) release of the TurnKey GitLab appliance. It is a continuation of the discussion (inadvertently) kicked off by Royce by this comment. FWIW, there is also a related open feature request regarding this on our tracker.

Summary of things so far:

  • There are maintenance issues with the current GitLab "source install" method (both for TurnKey and for end users).
  • As such, following discussions (both internal and public) it has been decided the best path forward is to use the GitLab Omnibus install for future releases (v15.2 onwards) of the GitLab appliance.
  • Jeremy (i.e. me) will do the preliminary work to develop a "base" GitLab appliance which has GitLab installed via Omnibus package.
  • Tim and Royce (and anyone else who is interested) will assist with testing and/or further development (via feedback and/or code and/or discussion, etc).
  • Tim and Royce (and others?) will also assist with testing migration from recent versions of TurnKey GitLab (i.e. =
  • Once we have a working product that at least matches the functionality of the current GitLab appliance, plus some base documentation on how to migrate, etc, we'll release it as the next (v15.2) GitLab appliance release.

We'll then look at further improvements to make it (much) better than the current appliance. That will include easy config to:

  • Configure GitLab settings to support AWS as external data-store for large data items (i.e. git-lfs using AWS S3 as backing).
  • Confiure GitLab settings for Mattermost connectivity.
  • Configure SSL Certificates (TKL already provides basic server-wide Let's Encrypt TLS certs - but we'll make sure it fulfills the needs of GitLab, including any sub-domains etc).
  • Configure a RAM-disk swapfile.

Anything I've missed guys?!

Let the discussion continue... :)

Forum: 
OnePressTech's picture

Here is a good VM config wizard sample doc from GitLab:

https://docs.gitlab.com/ce/install/aws/

This article has two interesting reference points:

1) How to create a high-availability GitLab configuration on AWS

2) How to configure the AWS GitLab AMI

These are both useful references for mapping out the TKLX AMI configuration wizard steps.

There is also the Bitnami GitLab documentation:

https://docs.bitnami.com/general/apps/gitlab/

 

NOTE:

For those reading this who wonder why bother creating a TKLX GitLab VM when there is already a GitLab AMI or a Bitnami GitLab VM...

1)  Supporting additional VM formats

2) TKLBAM incremental backup

3) Debian security updates

4) VM tools to manage the VM

5) TKLDev to manage variations

6) The TKLX community

7) Jeremy Davis (what can I say Jed...you are a big reason I am still with TKLX :-)

 

Cheers,

Tim (Managing Director - OnePressTech)

Jeremy Davis's picture

Your encouragement and kind words are warmly welcomed! :) Also those resources look like they'll be handy once we get to that bit.

Unfortunately, I don't have much to report yet (other than I REALLY hate Omnibus now I know more about it! Bloody Chef and Ruby and ... Argh!). I've already spent hours and can't yet get GitLab to install to the build environment (chroot). I keep thinking I'm close, but no cigar yet... :(

The more time I spend with web apps written in Ruby, the more I wonder why the hell anybody in their right mind would do that!? Anyway, that's enough whinging... Hopefully, I'll have a breakthrough soon and have something to share...

Jeremy Davis's picture

Ok, so there's still plenty of work left to do, but the major hurdle of getting the GitLab Omnibus package to install appears to be complete (some tidying up left to do, but it appears to install ok...).

I currently have it building to ISO so I can do some more development. FWIW SystemD doesn't work properly within a chroot, so building to an ISO (then installing to a VM) makes everything a bit more "real world".

I don't plan to do a ton of battle testing yet. I'll do just enough to make sure that everything appears to work. Then I'll test a few ideas I have for regenerating secrets. Plus make sure that we can set the first (admin) user email and password etc.

If you're at all interested, I have pushed the build code that I have so far (definitely not ready for production!) to a new branch on my personal GitHub. If you do wish to build yourself on TKLDev it should "just work". Essentially though, at this point it's just a vanilla GitLab-CE Omnibus install (no GitLab related inithooks, etc).

If you plan to do any development and/or build multiple times, I recommend that you run make root.patched initially, then copy out the deb package from the apt cache. E.g. from scratch in TKLDev:

cd products
git clone  -b omnibus-pkg-install https://github.com/JedMeister/gitlab.git
cd gitlab
make root.patched
APT_CACHE=var/cache/apt/archives
cp build/root.patched/$APT_CACHE/gitlab-ce_*.deb overlay/$APT_CACHE/

Not having to download the ~450MB omnibus package each rebuild will certainly make things a bit quicker! Although please note that it's still quite slow. In part because the install takes a while anyway (which won't change). But in part because I'm currently committing large areas of the filesystem to git repos to see exactly what is going on! That will be removed before the final build.

If you just want to build an ISO (or other builds) then you can just cd to products dir and do the git clone first (as per above), then skip the rest and use buildtasks to build the ISO. Once you have an ISO built, you can build the other builds. I.e.:

cd buildtasks
./bt-iso gitlab
# when done you should have the following file:
# /mnt/isos/turnkey-gitlab-15.2-stretch-amd64.iso
# then to build an alternate build, such as Proxmo/LXC
./bt-container gitlab-15.2-stretch-amd64
# or OVA/VMDK:
./bt-vm gitlab-15.2-stretch-amd64

As soon as I'm a bit further along, I'll upload an ISO and a Proxmox/LXC build for you guys to test. This is mostly just so you can see some progress and have a bit of a play if you're super keen.

OnePressTech's picture

Nice work Jed. Disappointingly on my end I will need to wait 2 weeks while our intrepid national Telcos try to get me on NBN. After a week of Optus stuffing around unsuccessfullly I am running on expensive mobile data in the interim so big downloads are out for now. I'm in the process of trying to switch to Telstra. Fingers crossed that they can actually get me a working service any time soon. Telstra just announced more technical jobs being shipped to India. Apparently there are no available qualified Telecom people in Australia. Huh...I'm a ex-Telstra qualified Telco engineer...no one called me!

So I am expecting another two weeks of delay before I can test the new VM.

Cheers,

Tim (Managing Director - OnePressTech)

Jeremy Davis's picture

For fear of getting a bit too political here, seriously the NBN (explicitly MTM/FTTN) has been the most massive Government fail! I'm one of the lucky ones who got "proper" NBN (FTTH/FTTP) and it's been pretty good for me since I got connected (a couple of years ago now) but I've heard plenty of horror stories.

Bit of a pity you won't get a chance to test much within the next couple of weeks. But if I can get it to a point where I'm pretty happy with it before then, I'll see what I can do getting you access to an AMI.

OTOH, there is also a possibility that I'm happy enough within the next 2 weeks that I may just publish it. But there is nothing stopping you from letting me know about any shortcomings, bugs or improvements we can make. If they're critical bugs, we can re-release ASAP. If they're not so critical, we can bundle them into the next release (which we should be able to do within a month).

One major benefit of the install via Omnibus package will be that generally it will be a breeze to rebuild with an updated version of GitLab! :)

Good luck with your networking woes. Hopefully Telstra comes to the party!

Jeremy Davis's picture

Just a quick update.

The install of GitLab appears to work ok and everything seems to be running as it should. However, the initial interactive inithooks that I have devised do not work as they should. The WebUI still requests that you set a password when you first access it. The email/password set by the inithook doesn't allow login and the password set via WebUI also fails... :(

So lots more work left to do, but definitely progress. FWIW I've pushed my changes back to the WIP GitHub repo (in my personal GH account).

Sytko's picture

Hello. When can I download a ready iso for testing?
Jeremy Davis's picture

Hi there and thanks for your interest and willingness to be involved. I really appreciate it.

Unfortunately, I've had a few other things I've needed to prioritise this week, but I have made a little more progress since last time I posted. I think it's pretty close to being ready for further testing, but it seems unlikely that I'll get there today (so may not be until next week).

One thing that would be incredibly helpful would be to get an understanding of what areas of the filesystem might need to be included in a backup? Anyone who is already using the Omnibus install (Tim? Royce?) might already have some ideas!? It's something I haven't looked at yet at all, but will need to before we do an official release. Any pointers that might save me time would be really appreciated.

Otherwise, if you'd like to build your own ISO using TKLDev, then that's an option (so you don't need to wait for me if you're super keen). There is a doc page which should help get you up to speed. To build the "new" GitLab (dev) appliance, when you get to building your second iso (it's recommended to just build Core first to ensure that you understand what is going on and ensure everything is working properly), then you can use the 'omnibus-pkg-install' branch of my fork of the gitlab repo. I.e.:

cd products
git clone --branch omnibus-pkg-install https://github.com/JedMeister/gitlab.git
cd gitlab
make

(Please also see my notes in this post above).

I have been testing the code (committed as of late last week) and have just committed and pushed the (minor) only changes that I have made to the code since I built the instance I've been testing. It's also worth noting, that I plan to remove the "schema" setup from the inithook to a confconsole plugin, as it triggers the Let's Encrypt cert generation. IMO it doesn't really make sense to do it at firstboot as it's unlikely that most users will have everything in place at that point and choosing to generate Let's Encrypt certs without DNS pre-configured will cause failure.

At this point, I doubt I will get any further this week, but hope to have something ready for others to test early next week.

OnePressTech's picture

Hey Jed,

Sorry I'm not being much help. I have extracted myself finally from Optus but am now on the Telstra joyride. We'll see how that goes.

I expect to be back on this next week.

Regarding Backup, are we trying to identify volatile / non-volatile folders / files to know which should / should not be backed up by TKLBAM?

GitLab backup may provide some helpful guidance:

https://docs.gitlab.com/ce/raketasks/backup_restore.html

IDEA:

Since we are already following an unconventional journey with this particular VM by installing via omnibus rather than source...why stop there :-)

We could just create a local folder on the GitLab VM, configure / trigger a GitLab backup that dumps everything into that folder, and configure TKLBam to just back up that backup folder and consider everything else as poart of the mirrored image.

So on a restore, we do the opposite, TKLBAM restores the backup folder on a freshly created GitLab VM and then a GitLab restore is triggered. I assume we would not touch TKLBAM so it would probably be a cron script that triggers a restore if the folder date changes...or something like that.

Thoughts?

Cheers,

Tim (Managing Director - OnePressTech)

Jeremy Davis's picture

I'm appreciative of any input I get! So it's all good. Plus I totally understand the telco pain...

Regarding Backup, are we trying to identify volatile / non-volatile folders / files to know which should / should not be backed up by TKLBAM?

Exactly! :)

GitLab backup may provide some helpful guidance: https://docs.gitlab.com/ce/raketasks/backup_restore.html

Fantastic, thanks! That will help tons!

We could just create a local folder on the GitLab VM, configure / trigger a GitLab backup that dumps everything into that folder, and configure TKLBam to just back up that backup folder and consider everything else as poart of the mirrored image.

Hmm, that sounds like a very reasonable suggestion. We'd then look to avoid backing up all/most of the rest of GitLab. Although FWIW generally anything included in an apt package (with the exception of the config in /etc and sometimes data within /var), would not be desirable to include anyway.

FWIW, it appears that re GitLab, much (if not all?) of the data in /var (/var/opt/gitlab) is generated by the 'gitlab-ctl reconfigure' command (generated from settings in the /etc/gitlab/gitlab.rb).

Regarding triggering the GitLab backup (and restore), assuming that i can be done via commandline, TKLBAM hooks could easily cope with that!

My only concern regarding your idea would be; what happens if the GitLab version that that creates the backup, is different to the version it is being restored to? I have no idea how the Omnibus package might cope with that?!

And between GitLab having a rapid release cycle and the ease of update that the Omnibus package will provide, there is a very real chance that the data will be for an alternate version of Gitab than what it ws created from. FWIW, the current backup somewhat works around that, but including GitLab itself (which is a double edged sword as it requires the user to manually keep the install up to date, and also makes it likely that it won't "just work" when migrating between major versions).

I guess we could consider a restore hook to try to match the installed version with the backup version. But the more stuff scripted during restore, the more factors need to be considered, and the greater the risk of something breaking...

Note though, that this isn't really a specific concern related to your suggestion. It would apply to many other appliances and a "normal" TKLBAM style backup too. We have a few other appliances that have 3rd party repos. But the fact that the GitLab Omnibus includes so much software, I anticipate that the implications would be significantly larger and due to the shear volume of software installed, significantly more likely for issues to occur.

Having said that, TKLBAM's primary aim is to provide reliable backup and restore for a specific single server. Usage as a data migration tool, is certainly in scope, but is secondary to the primary objective. Plus is not guaranteed without the potential need for manual intervention. Still I think it requires some thought.

The other thing that occurs to me is that ideally we would want an uncompressed backup, stored within a consistent place (e.g. no date stamped directories). Otherwise the daily diffs (i.e. daily incremental backups) would potentially be ridiculously large. I am unsure whether the GitLab backup tool would support that, but I'll have a look sometime soon.

Regardless, awesome suggestion and definitely food for thought! :)

OnePressTech's picture

Thanks for everything Jed.

Regarding cross-version backup / restore or stale restores, this is always an issue.

To mitigate the risk the GitLab backup could be set on a daily schedule so that the backup TKLBAM stores is always current.

Cheers,

Tim (Managing Director - OnePressTech)

Jeremy Davis's picture

If we run the backup as a pre-backup hook, then the backup should always be up to date. Although we need to consider how we might handle things, if say the GItLab backup fails. I assume the best course would be for the TKLBAM backup to also fail (ideally loudly) in that case.

Anther consideration that has just occurred to me too, is that by default TKLBAM will possibly want to backup the (Postgres) DB. I assume that a GitLab backup would include the relevant DB data? If so, we won't want the DB being included separately as well. TBH I'm not sure how TKLBAM determines which DB is in use? We might be lucky and because it's not in the default place (i.e. installed by Omnibus) it doesn't even try to back it up. Worst case though, we could explicitly exclude it.

PS, it doesn't look like I'm going to get any more time to play with it today, but so far the basic install and firstboot seems fairly reliable. The inithooks are still disabled by default, but the interactive one I've been working on appears to work well. And I've tested just manually removing the secrets (from /var/opt/gitlab/)and (re)running 'gitlab-ctl reconfigure' seems quite happy to regenerate them. So that looks like it will work fine. I think I mentioned that the inithooks currently give the option to do the Let's Encrypt at first boot, but I plan to move that out to Confconsole I reckon (unless you have a good reason not to).

OnePressTech's picture

If TKLBAM triggers a GitLab backup before it does its diffs that would work.

LetsEncrypt from the console is a good plan. Some DevOps may want to use their own cert rather than LetsEncrypt. Cert-upload should eventually be added as an option in confconsole so the DevOps supplied cert can be auto-installed as part of the installation process.

Cheers,

Tim (Managing Director - OnePressTech)

Jeremy Davis's picture

Awesome, sounds like we're on the same page then. That's great.

Unfortunately, it looks like I have another (programming related) task that I'm going to need to prioritise early next week. Whilst that should be quite fun, I was hoping to get GitLab wrapped up (at least the initial "rc" of the rebuilt appliance) early next week. It seems likely that that will not be a realistic goal now and probably mid-week will be the earliest. Regardless, I'll certainly see what I can do. TBH, it's really close I reckon.

Take care mate and chat more soon.

Sytko's picture

Hi. According to the instructions above - iso created. I will test further. The question is how to further update the virtual machine to the new version of gitlab?
Jeremy Davis's picture

Thanks for helping with testing! :)

When you build the iso, it should pre-install the latest version of GitLab-CE by default. From then on, you can follow the GitLab documentation on upgrading (use the instructions for Debian).

If you update regularly, then updating GitLab to the latest version should be as simple as running this:

apt update
apt install gitlab-ce

The only additional thing that you may need to do is update the config file (/etc/gitlab/gitlab.rb). I.e add/update any options that have changed for the new version.

However, it's important to note that if you fall behind the upstream updates, you will need to update to the latest release of the major version you are on, before updating the new major version. GitLab versioning is explained on this page. Please specifically see the section on upgrade recommendations . Please note that page is for GitLab-EE but also applies to GitLab-CE. (links updated to point to CE docs)

OnePressTech's picture

Gitlab docos are symmetrical...just change /ee/ to /ce/ in url

So in Jed's email above for CE the upgrade recommendations for CE is at:

https://docs.gitlab.com/ce/policy/maintenance.html#upgrade-recommendations

AND for EE is at:

https://docs.gitlab.com/ee/policy/maintenance.html#upgrade-recommendations

JED: Please adjust your post...it is not always true that EE instructions apply to CE.

Cheers,

Tim (Managing Director - OnePressTech)

Jeremy Davis's picture

Thanks mate. I'll update my post above. I just followed the links that generic GitLab docs provided. They obviously favour the EE docs when they need to provide links from generic docs to specific docs (which makes sense. I'll keep your point in mind for future links! :)

Jeremy Davis's picture

I have made a bit of a start on migration documentation. But it didn't take very long for me to start hitting details that required further elaboration and consideration. It's already chewed up fair bit of time and there is no end in sight...

So I've had to put that aside for now. I won't bother posting what I have yet as it's incomplete and a little contradictory (as I discovered more details). But I already have some quite useful insight into how it might work under different scenarios.

If anyone wants pointers on how I anticipate migration from a version of our existing appliance (or any source install for that matter), to the upcoming (Omnibus install) GitLab appliance might work, please ask. To provide relevant and specific thoughts, I'll need info on the TurnKey version (or Debian/Ubuntu version if not TurnKey) that you currently have and the version of GitLab that you are running on it.

I'm almost certain that providing info and pointers for a specific scenario will be much easier that needing to consider all possibilities...!

OnePressTech's picture

I agree Jed. Though I am sure TKLX clients would all love fully life-cycled VMs, that is a huge  investment that no VM supplier (Bitnami, AWS, Google, Azure, etc) has committed to. TKLX provides shrink-wrapped VMs...it is up to the VM users to life-cycle them. This is the same no matter who people source their VMs from.

So other than some basic guidance on migrating from old GitLab VM to new GitLab VM I think your approach makes sense. Keep it simple and let the community backfill :-)

Cheers,

Tim (Managing Director - OnePressTech)

Add new comment