Your encouragement and kind words are warmly welcomed! :) Also those resources look like they'll be handy once we get to that bit.

Unfortunately, I don't have much to report yet (other than I REALLY hate Omnibus now I know more about it! Bloody Chef and Ruby and ... Argh!). I've already spent hours and can't yet get GitLab to install to the build environment (chroot). I keep thinking I'm close, but no cigar yet... :(

The more time I spend with web apps written in Ruby, the more I wonder why the hell anybody in their right mind would do that!? Anyway, that's enough whinging... Hopefully, I'll have a breakthrough soon and have something to share...

Hi there and thanks for your interest and willingness to be involved. I really appreciate it.

Unfortunately, I've had a few other things I've needed to prioritise this week, but I have made a little more progress since last time I posted. I think it's pretty close to being ready for further testing, but it seems unlikely that I'll get there today (so may not be until next week).

One thing that would be incredibly helpful would be to get an understanding of what areas of the filesystem might need to be included in a backup? Anyone who is already using the Omnibus install (Tim? Royce?) might already have some ideas!? It's something I haven't looked at yet at all, but will need to before we do an official release. Any pointers that might save me time would be really appreciated.

Otherwise, if you'd like to build your own ISO using TKLDev, then that's an option (so you don't need to wait for me if you're super keen). There is a doc page which should help get you up to speed. To build the "new" GitLab (dev) appliance, when you get to building your second iso (it's recommended to just build Core first to ensure that you understand what is going on and ensure everything is working properly), then you can use the 'omnibus-pkg-install' branch of my fork of the gitlab repo. I.e.:

cd products
git clone --branch omnibus-pkg-install https://github.com/JedMeister/gitlab.git
cd gitlab
make

(Please also see my notes in this post above).

I have been testing the code (committed as of late last week) and have just committed and pushed the (minor) only changes that I have made to the code since I built the instance I've been testing. It's also worth noting, that I plan to remove the "schema" setup from the inithook to a confconsole plugin, as it triggers the Let's Encrypt cert generation. IMO it doesn't really make sense to do it at firstboot as it's unlikely that most users will have everything in place at that point and choosing to generate Let's Encrypt certs without DNS pre-configured will cause failure.

At this point, I doubt I will get any further this week, but hope to have something ready for others to test early next week.

If we run the backup as a pre-backup hook, then the backup should always be up to date. Although we need to consider how we might handle things, if say the GItLab backup fails. I assume the best course would be for the TKLBAM backup to also fail (ideally loudly) in that case.

Anther consideration that has just occurred to me too, is that by default TKLBAM will possibly want to backup the (Postgres) DB. I assume that a GitLab backup would include the relevant DB data? If so, we won't want the DB being included separately as well. TBH I'm not sure how TKLBAM determines which DB is in use? We might be lucky and because it's not in the default place (i.e. installed by Omnibus) it doesn't even try to back it up. Worst case though, we could explicitly exclude it.

PS, it doesn't look like I'm going to get any more time to play with it today, but so far the basic install and firstboot seems fairly reliable. The inithooks are still disabled by default, but the interactive one I've been working on appears to work well. And I've tested just manually removing the secrets (from /var/opt/gitlab/)and (re)running 'gitlab-ctl reconfigure' seems quite happy to regenerate them. So that looks like it will work fine. I think I mentioned that the inithooks currently give the option to do the Let's Encrypt at first boot, but I plan to move that out to Confconsole I reckon (unless you have a good reason not to).

Awesome, sounds like we're on the same page then. That's great.

Unfortunately, it looks like I have another (programming related) task that I'm going to need to prioritise early next week. Whilst that should be quite fun, I was hoping to get GitLab wrapped up (at least the initial "rc" of the rebuilt appliance) early next week. It seems likely that that will not be a realistic goal now and probably mid-week will be the earliest. Regardless, I'll certainly see what I can do. TBH, it's really close I reckon.

Take care mate and chat more soon.

Thanks for helping with testing! :)

When you build the iso, it should pre-install the latest version of GitLab-CE by default. From then on, you can follow the GitLab documentation on upgrading (use the instructions for Debian).

If you update regularly, then updating GitLab to the latest version should be as simple as running this:

apt update
apt install gitlab-ce

The only additional thing that you may need to do is update the config file (/etc/gitlab/gitlab.rb). I.e add/update any options that have changed for the new version.

However, it's important to note that if you fall behind the upstream updates, you will need to update to the latest release of the major version you are on, before updating the new major version. GitLab versioning is explained on this page. Please specifically see the section on upgrade recommendations . Please note that page is for GitLab-EE but also applies to GitLab-CE. (links updated to point to CE docs)

I was hoping to get a bit further today, but at this stage it looks unlikely...

Hopefully you'll get a chance to check it out over the weekend. Please let me know how it goes and any thoughts you have (good, bad or otherwise).

FWIW as the tklbam hook has grown (in size and complexity) I have been considering whether I should persevere with the bash hook I have, or whether it might be better to rewrite it in python?! If you have any thoughts at all on that (language and/or functionality and/or other ideas) please share.

Also, if you need more info about using the TKLBAM profile, please hit me up via text message or something. Otherwise, I'll be back onto it on Monday.

FWIW the gitlab_restore function that I have constructed is within the TKLBAM hook script. To save you from decoding my bash, here is the essence of what it does:

gitlab-ctl reconfigure
gitlab-ctl stop unicorn
gitlab-ctl stop sidekiq
gitlab-rake gitlab:backup:restore BACKUP="relevant_backup"
gitlab-ctl restart
gitlab-rake gitlab:check SANITIZE=true

So it looks like I'm not actually re-running 'gitlab-ctl reconfigure' AFTER restoring the backup! I suspect that's the issue! Also TBH, I don't recall why I'm running 'gitlab-ctl reconfigure' BEFORE I run the restore?! Seems a bit redundant in retrospect...

Armed with your input and my reflection, I'll try tweaking the script a little and see how we go.

To be explicit, I'll try moving the 'gitlab-ctl reconfigure' from before the restore, to afterwards (between the restore step and the restart step as you suggest).

Also do you have any thoughts on the value of running the check? Perhaps it's not really of value there and just slows things down?

Awesome stuff. That sounds good. I'm really hoping moving that gitlab-ctl reconfigure does the trick (I suspect that it will). On reflection, I'm not really sure why that didn't occur to me previously (nor why any of the many threads I've read haven't double checked with users experiencing the issue). Anyway...

FWIW the whole /etc/gitlab dir (i.e. gitlab-secrets.json & gitlab.rb, plus any TLS certs that have been generated) are included in the backup by TKLBAM itself. All the rest of the GitLab directories (i.e. /opt/gitlab & /var/opt/gitlab) are explicitly excluded. The GitLab backup runs prior to TKLBAM doing anything. The file is then transferred out of the GitLab backup directory (/var/opt/gitlab/backups by default) to a location that is included in the backup (currently /var/cache/tklbam-gitlab). Then TKLBAM does it's thing...

I've also renamed the backup file (and stored the original name in a text file to support the restore process). That way, assuming that GitLab tars up the files in the same order, unless lots changes, the daily incremental backups should still be quite small. The rationale is that if the file is named as per GL naming convention, as far as TKLBAM is concerned, it will be a new file every day. Giving it a consistent name, makes TKLBAM realise that it's essentially the same file, but it's not exactly the same (so does a binary diff). TBH, I haven't actually double checked that my understanding is correct, which I should do. Because if it makes no difference, it's probably better to just move it, rather than renaming it too.

It has also occurred to me to untar the GitLab backup before doing the TKLBAM backup. TKLBAM already tars everything up prior to upload, so there is no increase in backup size by doing that (it may actually decrease the size of the backup?!) If we went that way, you could be assured that the daily incremental backup size would only increase directly relative to the new files, etc. OTOH, for many users it may not make much difference and it means an additional operation at backup time (untarring the GL backup) and restore time (tarring it back up so GL will recognise it). All that additional process, means additional opportunities for stuff to go wrong though, so I'm inclined to leave it be... (Simply rename it to a generic name).

As per always, interested in any input you (or anybody else) has.

Also, unfortunately I've been dragged away onto other stuff now. So it seems unlikely I'll be able to get back to this until next week... I think I'm really close now though! :)

With a few explcit exclusions, the whole /etc directory (including /etc/gitlab) is included in the TKLBAM backup as part of the normal profile. So the gitlab-secrets.json file (and the config file, plus TLS certs) are all automagically restored by TKLBAM before any of the rest of this stuff happens. I appreciate the reminder though as I hadn't committed and pushed back the other required changes to the TKLBAM profile repo. So to avoid the risk of forgetting that, I've just done that now. :)

And actually, I wonder if re-running gitlab-ctl reconfigure with a different gitlab-secrets.json file (and then not re-running it after the restore) is perhaps part of the issue in the first place? TBH, I hadn't considered that before, but it actually seems plausible...

Anyway mate. Thanks again for your input. Hopefully I'll be able to tidy this up early next week. Cheers.

I understand that neither gitlab-secrets.json nor the gitlab.rb config file, are included when gitlab-rake gitlab:backup:restore BACKUP="relevant_backup" is run.

But because TKLBAM already includes most of /etc (with some exclusions) and gitlab-secrets.json (and gitlab.rb) are stored in /etc/gitlab, they are automagically included in the normal TKLBAM backup. I.e. no additional command/inclusion/etc is required to include them.

That too may have been part of the issue with the 500 errors. The restore process that I've scripted runs post TKLBAM restore. So the backed up gitlab-secrets.json file has already been restored when the restore component of the Gitlab specific hook script runs. But then I was running gitlab-ctl reconfigure (i.e. original data, with restored gitlab-secrets.json). Then restoring the data and not re-running gitlab-ctl reconfigure.

Hopefully I should be able to get back to this today. Armed with this additional info, I'm really confident that it won't take much more work to get it going. Then it'll just require the docs to be written. That may take a little more time, but hopefully shouldn't be too bad.

Thanks for trying it out, but as Tim notes GitLab is a bit of a beast, so has fairly high resource requirements (at least for a headless Linux server anyway).

Regardless, to double check, I just downloaded the ISO and launched it into a (KVM) VM with 2 vCPUs and 4GB RAM. And as it turns out, I can reproduce your 500 Error! Argh!

I'm pretty pissed TBH considering the time and energy that I put in and I did test the latest build code just before I built it (for release). I'm not 100% sure what has gone wrong, but clearly there is an issue!

FWIW it looks like it's something to do with the encryption keys (although I don't understand why). Here's how I worked that much out:

Open an SSH session and set the log to output to the session in real time:

tail -f /var/log/gitlab/gitlab-rails/production.log

Then I tried to log into the webUI, using username 'root' and the password I set at firstboot. This is what the log showed:

Started POST "/users/sign_in" for 127.0.0.1 at 2019-03-29 02:10:23 +0000
Processing by SessionsController#create as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "user"=>{"login"=>"root", "password"=>"[FILTERED]", "remember_me"=>"0"}}
Completed 500 Internal Server Error in 214ms (ActiveRecord: 19.6ms)
  
OpenSSL::Cipher::CipherError ():
  
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:55:in `get_token'
app/models/concerns/token_authenticatable_strategies/base.rb:27:in `ensure_token'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:42:in `ensure_token'
app/models/concerns/token_authenticatable.rb:38:in `block in add_authentication_token_field'
app/services/application_settings/update_service.rb:18:in `execute'
lib/gitlab/metrics/instrumentation.rb:161:in `block in execute'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `execute'
app/controllers/application_controller.rb:467:in `disable_usage_stats'
app/controllers/application_controller.rb:453:in `set_usage_stats_consent_flag'
lib/gitlab/middleware/rails_queue_duration.rb:24:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:55:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/correlation_id.rb:15:in `use_id'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'

I'll work out an a workaround and post back ASAP. Then I guess, I'll have to rebuild it...!

It's disabled by default, but I did include version matching software to the TKLBAM hook script (for restoring a backup). However, it didn't occur to me that that might be valuable for users initially.

It's actually pretty easy to do. And seeing as I already have a fix to push, I'll see if I can include that in the next (bugfix release too). We'll see (it makes a lot of sense).

In the meantime, here's how to do it. Assuming that you want v10.0.6 (just picked a version at random), it's simply a case of this:

apt update
apt remove gitlab-ce
apt install gitlab-ce=10.0.6-ce.0

From what I can gather, the "version" string (i.e. "10.0.6-ce.0" in my example above) is:

<VERSION>-<RELEASE_CHANNEL>.<DEB_BUILD>
Where:
<VERSION&gt = 3 part version number; i.e. 10.0.6
<RELEASE_CHANNEL> - 'ce' (or 'ee')
<DEB_BUILD> - generally '0', but if they rebuild a package with the same GitLab version, this number is incremented

Re your comment:

GitLab updates versions on the 22 day of each month. When an O/S reaches end-of-life GitLab won't update to a higher version. That means everyone only has 1 month to migrate a GitLab to a new server with an updated Debian before their TKLX GitLab becomes out-of-synch with their current version.

Perhaps I'm missing some context, or misunderstand your statement, but are you sure about this? I note that both the Debian Jessie (i.e. base of v14.x - already well past "standard" EOL; but still maintained via "LTS" for about another year) and Debian Stretch (i.e. base of v15.x - current stable; "standard" EOL 1 year after Buster release, then "LTS" support until 5 years after Buster release) have omnibus packages for v11.9.1 (see Jessie here and Stretch here) which by my understanding is the latest GitLab release

If you look down that page you linked to a little, under Supported Operating Systems both Debian 8 (Jessie - basis of v14.x) and Debian 9 (Stretch - basis of v15.x) are currently supported (Jessie until mid 2020 & Stretch until 2022). I'm not sure, but I think that the top table (with mention of "Raspbian Jessie" just above "Debian Wheezy") may have thrown you?

Regarding your request/suggestion, I'm potentially open to being a bit more savage in clearing out the pre-installed version on firstboot (if the user chooses to install a different version), but I'm not really keen on shipping with no version installed at all. Some users run servers offline (or at least without direct internet connection) so requiring an internet connection on firstboot doesn't seem like a good idea.

It's also worth noting that if a new/different version is installed clean, then without some tweaking, the firstboot password won't work and a new password would need to be set via the webUI. I would be inclined to leverage the tweak I do within the conf script to disable that, and set a password via the firstboot scripts (even if a new/different version is being installed).

Also the forum.gitlab.com link you provided doesn't work for me. When I first browsed there it said I needed to log in to view it. Then when I logged in, it said I didn't have adequate permissions?!

Sorry I totally missed that.

FWIW Wheezy was EOL 2018-06-01 (Debian moved it from "LTS" to "archive") so as far as Debian are concerned, there haven't been any OS level updates since then. Having said that Freeaxian, have been providing an ELTS for Wheezy, so there probably have been OS level updates, at least for important stuff. And seeing as you are getting GitLab via Omnibus, most things would be being updated via that I guess...

I was hoping to have a fixed ISO available already, but then I tried to also implement Tim's request re selecting the version to use at first boot. I decided to do that via python (with intention of making it somewhat generic so that it could be used elsewhere too). But I'm still relatively new to "proper" python programming (I've been using python for glue code for years, but it's not quite the same...). Needless to say, I got a bit bogged down with that and unfortunately, I've since been dragged aside to some other high priority tasks.

Unless I get a chance to get back to GitLab early next week, I probably should just rebuild it (with the fix for this particular issue) and get that published ASAP. The pressure to complete the version selection stuff will be off then and I can complete the code I've started when I get a chance. In the meantime, we can just document how to change versions. Actually GitLab have docs that pretty much cover that. Also FWIW my reading suggests that there is no issue with doing this on a Omnibus install with no data. However, it's worth noting, that unless you plan to restore a backup, if you it's not a good idea to downgrade GitLab version.. and include the version selection option in a future release.

In the meantime, you have a few options if you wish to persevere with GitLab on TurnKey:

• Recommended: Use the new v15.2 GitLab and apply the issue fix/workaround (as noted both on the issue and in my previous post above)
• Use the older v15.1 release (GitLab installed from source) - Not recommended
• Alternatively, do what Tim has been doing for years - and install GitLab yourself (on Core probably).

As I note above, I recommend that you simply apply the fix. I'll repost it here so it's easier to find:

mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.bak
gitlab-rails runner -e production "ApplicationSetting.current.reset_runners_registration_token!"
/usr/lib/inithooks/bin/gitlab.py

Great work on that. Out of interest, how did you build your ISO? Did you use our buildtool/infrastructure TKLDev? If so, generally the best way to go is to open a pull request on GitHub for the relevant buildcode (obviously our GitLab repo in this case). Then we can rebuild the appliance with the updated buildcode and publish in all our usual formats.

If you used some alternate method to build your ISO, then that's fine, and your efforts are still appreciated, but we only distribute appliances that we build ourselves (from the relevant GitHub repo buildcode. That's in place because that's the only way we can be assured that it complies with our standards. So you'd need to share your file via some sort of filesharing app, such as DropBox, Google Drive or one of the many online filesharing apps.

Also as it turns out in this case, it looks like I beat you to it. Whilst I haven't actually written up a relevant "release" blog post yet, I have uploaded our updated (and fixed) GitLab appliance (v15.3 - downloadable from the appliance page) and ours also includes GitLab v11.10.4. I'm sorry that I wasn't a bit quicker with it (both building it and the announcement) but at least you know how it works now for future reference. Please feel free to give it a spin and share your feedback.

Thanks again for your contribution. And sorry that I wasn't a bit more public on what I was up to...

What? Even the 15.3 one?! I tested it quite extensively and it was working fine when I published it?!

I'll have another look at it myself ASAP.

And/or please feel free to share your code that is working. The build code repo is here.

First up, it's likely worth noting that this won't be specific to the GitLab appliance. The default Postfix (the mail transfer agent (MTA) that we preinstall) config is common to all TurnKey appliances. It's perhaps also worth further noting that by default on the GitLab appliance, GitLab should be configured to send emails via Postfix (on localhost). So if you haven't adjusted the GitLab email settings and mails from GitLab are working, then the rest of this post is possibly irrelevant or wrong.

Anyway, I'm not 100% sure, but by my understanding, the emails will still go to "root@gitlab" (i.e. root@localhost), but should also be forwarded to the email you have noted in the /root/.forward file.

I suspect that if mail is not being forwarded to the email address in your forwards file, then it's about email deliverability rather than anything else.

Depending on where your server is running, Postfix may be being blocked completely (e.g. if you are self hosting via a normal consumer ISP plan). Alternatively the IP address of your server may have been blacklisted on one of the many Spam blacklists sites. That is quite common for the IP addresses provided by Cloud/VPS companies (because spammers often abuse these services).

As a general rule, the easiest way to work around that is to configure Postfix to send emails via an SMTP relay (rather than directly). We have a Confconsole plugin which provides and easy way to configure an SMTP Mail relay.

If you've already configured GitLab emails, this is likely irrelevant, but FWIW by default GitLab is configured to send emails via Postfix on localhost. So setting up the SMTP relay for Postfix should also allow GitLab email sending via the same SMTP relay (without need to configure GitLab email sending separately).

Finally, if you are sure everything is right and the emails aren't sending as expected, it's likely worth checking the Postfix logs. They should be found within /var/log and IIRC are called mail.log and mail.err (but I could be wrong...).