Veronica Charlton's picture

 

Initially when I install the appliance it boots and runs.

 

I use turnkey-init to setup the information needed for gitlab.

 

Gitlab runs but when I try to login on the web interface I get a 500 error.

 

If I reboot gitlab is not started in the bootup provess.

 

help!!

Forum: 
Jeremy Davis's picture

Argh! Bloody GitLab! One of the most painful appliances to maintain...!

Ok, sorry for my rant. I'll take a few deep breaths...

I don't yet have a fix, but I can confirm the bug. FWIW, on face value, it appears that between my final test of the new appliance and the final build (~1-2 hours) they released a new version which was either buggy or at least not compatible with our setup. Argh!

I want to blame GitLab (because I really do hate it) but really it's my fault. I should have done a final test (after doing the final build, destined for publishing). But I'd spent so long working on it, it was so late on Friday when I finally got it working and so resource intensive and so slow to build/start and therefore test (plus, I'd tested my local build so intensively), that I skipped that step. I can only assume that within that 1-2 hour window, they released a new version that broke things for us... Following my previous experience with GitLab, I should have known that would bite me...

I can't yet confirm, but I suspect that installing (i.e. updating to) the latest GitLab version should fix it. I had hoped to be able to post back to confirm (or not as the case maybe) by now, but after 20 minutes, it's still only 60%. In the meantime, here's the command to update GitLab CE (run as root user, or prefix with 'sudo'):

apt update
apt install gitlab-ce
Jeremy Davis's picture

Sorry, I can now confirm that even updating GitLab-CE doesn't work. I'm still not clear what the issue is, but it appears to run deeper than I suspected...

In the meantime, I've reverted the GitLab appliance page back to the previous v16.0.

Veronica Charlton's picture

I understand how things go sometimes. Been around the block a few times. 

My environment is on a closed network. I will grab the current one linked and bring it up. Theen move it over and check it out.

 

Thanks!

Veronica Charlton's picture

I installed the 1.60 version and it appears to go through the initializatino process properly. When I check using gitlab-ctl status I see everything I would expect running.

When I attemtp to access it through the web interface it does not connect. I can ping the VM so I know that part is working.

When I do a netstat I do not see any connections to port 80. Gitlab seems to be only connection is the localhost address (127.0.0.1). I have an external DNS that is resolving the IP correctly.

I am also able to access the webadmin interface for the server itself.

That is where things stand at the moment. 

badco's picture

Can you show the error you get when you try to access the webUI? (screenshot would be good)

Veronica Charlton's picture

I appologize for not getting back to you.

 

In all honesty I installed Centos in a new appliance and then gitlab onto that.

 

That is working and I have not gone back to try and figure out why the turnkey one isn't.

badco's picture

Alternatively you could use TKLCore LXC and install gitlab into that, then you get webmin and other features.

Michael's picture

I have the same problem like the first comment. I can give a small Workaround for one part of the problem:

For restart GitLab after reboot you can do following on console:

systemctl enable gitlab-runsvdir
systemctl start gitlab-runsvdir

 

But for the 500-Error i have no solution yet.

Alex's picture

I'm trying to understand the gitlab 500 error but it's not easy.

The command `gitlab-rake gitlab:check --trace` report no issue:

root@git .../log/gitlab#  gitlab-rake gitlab:check --trace
** Invoke gitlab:check (first_time)
** Invoke gitlab_environment (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute gitlab_environment
** Execute gitlab:check
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.17.0 ? ... OK (13.17.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ... 
GitLab Instance / Monitoring ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.31.0 ? ... yes (2.31.1)
Git user has default SSH configuration? ... yes
Active users: ... 1
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

root@git .../log/gitlab#

 

The command `gitlab-ctl tail` report the error:

==> /var/log/gitlab/gitlab-rails/production.log <==
  
OpenSSL::Cipher::CipherError ():
  
lib/gitlab/database.rb:342:in `block in transaction'
lib/gitlab/database.rb:341:in `transaction'
app/services/application_settings/update_service.rb:50:in `update_settings'
lib/gitlab/metrics/instrumentation.rb:160:in `block in update_settings'
lib/gitlab/metrics/method_call.rb:27:in `measure'
lib/gitlab/metrics/instrumentation.rb:160:in `update_settings'
app/services/application_settings/update_service.rb:12:in `execute'
lib/gitlab/metrics/instrumentation.rb:160:in `block in execute'
lib/gitlab/metrics/method_call.rb:27:in `measure'
lib/gitlab/metrics/instrumentation.rb:160:in `execute'
app/controllers/application_controller.rb:531:in `disable_usage_stats'
app/controllers/application_controller.rb:517:in `set_usage_stats_consent_flag'
app/controllers/application_controller.rb:463:in `block in set_current_context'
lib/gitlab/application_context.rb:70:in `block in use'
lib/gitlab/application_context.rb:70:in `use'
lib/gitlab/application_context.rb:27:in `with_context'
app/controllers/application_controller.rb:454:in `set_current_context'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/transaction.rb:56:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:172:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:21:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

 

I also find an error in the postgresql which was not present in the previous package:

2021-07-21_21:05:08.72639 LOG:  configuration file "/var/opt/gitlab/postgresql/data/postgresql.conf" contains errors; unaffected changes were applied
2021-07-21_21:05:08.95919 LOG:  no match in usermap "gitlab" for user "gitlab" authenticated as "root"
2021-07-21_21:05:08.95921 FATAL:  Peer authentication failed for user "gitlab"
2021-07-21_21:05:08.95921 DETAIL:  Connection matched pg_hba.conf line 70: "local   all         all                               peer map=gitlab"
2021-07-21_21:05:09.04222 LOG:  no match in usermap "gitlab" for user "gitlab" authenticated as "root"
2021-07-21_21:05:09.04223 FATAL:  Peer authentication failed for user "gitlab"
2021-07-21_21:05:09.04224 DETAIL:  Connection matched pg_hba.conf line 70: "local   all         all                               peer map=gitlab"
2021-07-21_21:05:09.10794 LOG:  no match in usermap "gitlab" for user "gitlab" authenticated as "root"
2021-07-21_21:05:09.10796 FATAL:  Peer authentication failed for user "gitlab"
2021-07-21_21:05:09.10796 DETAIL:  Connection matched pg_hba.conf line 70: "local   all         all                               peer map=gitlab"

But the config line seems identical to the preivous package.

 

I will continue to investigate a bit.

 

3piece (James Moyle)'s picture

I think I have solved it. It was an issue with cyper as can be seen by running the following:
sudo gitlab-rake gitlab:doctor:secrets VERBOSE=1
  Notice there is an issue with Application Settings and Projects  
sudo gitlab-rails dbconsole SELECT runners_token, runners_token_encrypted FROM projects; UPDATE projects SET runners_token = null, runners_token_encrypted = null; SELECT runners_registration_token_encrypted, encrypted_ci_jwt_signing_key FROM application_settings; UPDATE application_settings SET runners_registration_token_encrypted = null; UPDATE application_settings SET encrypted_ci_jwt_signing_key = null;
  I hope this works for others. After completing this I was able to run Gitlab.   I also upgraded to a newer version, first upgrading to v14.0.14, then to v14.1.3   BTW, there appears to be an issue with locales on this app.
3piece (James Moyle)'s picture

The problem appears to be with two keys in the database that already encrypted and causing issues for cypher. Follow from here: https://docs.gitlab.com/ee/raketasks/backup_restore.html#verify-that-all-values-can-be-decrypted To check this log into the console:
sudo gitlab-rake gitlab:doctor:secrets VERBOSE=1
  Notice there is an issue with Application Settings and Projects
sudo gitlab-rails dbconsole SELECT runners_token, runners_token_encrypted FROM projects; UPDATE projects SET runners_token = null, runners_token_encrypted = null; SELECT runners_registration_token_encrypted, encrypted_ci_jwt_signing_key FROM application_settings; UPDATE application_settings SET runners_registration_token_encrypted = null; UPDATE application_settings SET encrypted_ci_jwt_signing_key = null;
  Gitlab was working after a reset.   I then upgraded Gitlab, to v14.0.14 then to v14.1.3 without issues. There is also an issue with Locales on the system.
kamaradski's picture

ok this explains why I couldn't get it to work the last couple of days. Thanks in advance for not giving up on this hard to maintain appliance.

just to confirm, even the `apt install gitlab-ce` is now broken:

root@gitlab ~# apt install gitlab-ce
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be upgraded:
  gitlab-ce
1 upgraded, 0 newly installed, 0 to remove and 9 not upgraded.
Need to get 940 MB of archives.
After this operation, 68.0 MB of additional disk space will be used.
Get:1 https://packages.gitlab.com/gitlab/gitlab-ce/debian buster/main amd64 gitlab-ce amd64 14.2.1-ce.0 [940 MB]
Fetched 940 MB in 14s (66.7 MB/s)                                                                                                                                                                  
[master fe5a4f2] saving uncommitted changes in /etc prior to apt run
 4 files changed, 0 insertions(+), 0 deletions(-)
 rename rc2.d/{S01turnkey-init-fence => K01turnkey-init-fence} (100%)
 rename rc3.d/{S01turnkey-init-fence => K01turnkey-init-fence} (100%)
 rename rc4.d/{S01turnkey-init-fence => K01turnkey-init-fence} (100%)
 rename rc5.d/{S01turnkey-init-fence => K01turnkey-init-fence} (100%)
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 107097 files and directories currently installed.)
Preparing to unpack .../gitlab-ce_14.2.1-ce.0_amd64.deb ...
gitlab preinstall: It seems you are upgrading from major version 13 to major version 14.
gitlab preinstall: It is required to upgrade to the latest 14.0.x version first before proceeding.
gitlab preinstall: Please follow the upgrade documentation at https://docs.gitlab.com/ee/update/index.html#upgrade-paths
dpkg: error processing archive /var/cache/apt/archives/gitlab-ce_14.2.1-ce.0_amd64.deb (--unpack):
 new gitlab-ce package pre-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/gitlab-ce_14.2.1-ce.0_amd64.deb
Enumerating objects: 1240, done.
Counting objects: 100% (1240/1240), done.
Delta compression using up to 4 threads
Compressing objects: 100% (782/782), done.
Writing objects: 100% (1240/1240), done.
Total 1240 (delta 69), reused 1234 (delta 66)
E: Sub-process /usr/bin/dpkg returned an error code (1)

Gitlab is way over my head, as it's a beast to maintain, so unfortunately I will be no help to you on this one :(

Jeremy Davis's picture

If you look closely at the output you posted and you can see this line:

[...] Please follow the upgrade documentation at https://docs.gitlab.com/ee/update/index.html#upgrade-paths

Between the error message and looking at that doc page, it appears that you need to do some incremental updates before upgrading to latest v14.x. I.e. So try this:

apt install -y gitlab-ce=13.12.9-ce.0

Please note that I have guessed the version number format. Hopefully I've got it right, but it may be formatted slightly differently. If that command fails, please check the specific version number format like this:

apt policy gitlab-ce | grep 13.12.9

(If feeling unsure, please post the output and I'll help you interpret it).

Then next up, upgrade to v14.0.x, then v14.1.x (adjust version numbers as need be as per above) and finally the latest:

apt install -y gitlab-ce=14.0.7-ce.0
apt install -y gitlab-ce=14.1.2-ce.0
apt install -y gitlab-ce
Darrin's picture

Are there any plans on releasing a new LXC with these fixes implemented, and the upgrades done?  I have to admit, I am mad at myself for not googling for the answer quicker.  I had just built a Ubuntu VM and installed GitLab, but for other reasons wanted to run it as a container on Proxmox so installed again.  After doing the upgrade dance hoping to fix the problem, turned out a simple update query was all that was needed.  Would be nice if a new container was released.

Jeremy Davis's picture

The current build code has issues so building a new container won't fix it (the new container will be broken). I put some time aside to work on it but was unable to resolve the latest issues with certificates in the time allocated so had to move to other things.

Once we have the base of our upcoming v17.0 release underway, I will circle back to GitLab and try again. I'm confident that I'll be able to work it out, but I need to get these other higher priority items resolved first.

Webdug's picture

Everything works (for me) now... Thanks everyone for the advice

Add new comment