Veronica Charlton's picture
Veronica Charlton - Thu, 2021/06/03 - 22:21

 

Initially when I install the appliance it boots and runs.

 

I use turnkey-init to setup the information needed for gitlab.

 

Gitlab runs but when I try to login on the web interface I get a 500 error.

 

If I reboot gitlab is not started in the bootup provess.

 

help!!

Forum: 
Support
Tags: 
turnkey gitlab
Jeremy Davis's picture

Jeremy Davis - Fri, 2021/06/04 - 05:04

Argh! Bloody GitLab! One of the most painful appliances to maintain...!

Ok, sorry for my rant. I'll take a few deep breaths...

I don't yet have a fix, but I can confirm the bug. FWIW, on face value, it appears that between my final test of the new appliance and the final build (~1-2 hours) they released a new version which was either buggy or at least not compatible with our setup. Argh!

I want to blame GitLab (because I really do hate it) but really it's my fault. I should have done a final test (after doing the final build, destined for publishing). But I'd spent so long working on it, it was so late on Friday when I finally got it working and so resource intensive and so slow to build/start and therefore test (plus, I'd tested my local build so intensively), that I skipped that step. I can only assume that within that 1-2 hour window, they released a new version that broke things for us... Following my previous experience with GitLab, I should have known that would bite me...

I can't yet confirm, but I suspect that installing (i.e. updating to) the latest GitLab version should fix it. I had hoped to be able to post back to confirm (or not as the case maybe) by now, but after 20 minutes, it's still only 60%. In the meantime, here's the command to update GitLab CE (run as root user, or prefix with 'sudo'):

apt update
apt install gitlab-ce
Jeremy Davis's picture

Jeremy Davis - Fri, 2021/06/04 - 05:33

Sorry, I can now confirm that even updating GitLab-CE doesn't work. I'm still not clear what the issue is, but it appears to run deeper than I suspected...

In the meantime, I've reverted the GitLab appliance page back to the previous v16.0.

Veronica Charlton's picture

Veronica Charlton - Fri, 2021/06/04 - 13:57

I understand how things go sometimes. Been around the block a few times. 

My environment is on a closed network. I will grab the current one linked and bring it up. Theen move it over and check it out.

 

Thanks!

Veronica Charlton's picture

Veronica Charlton - Fri, 2021/06/04 - 18:12

I installed the 1.60 version and it appears to go through the initializatino process properly. When I check using gitlab-ctl status I see everything I would expect running.

When I attemtp to access it through the web interface it does not connect. I can ping the VM so I know that part is working.

When I do a netstat I do not see any connections to port 80. Gitlab seems to be only connection is the localhost address (127.0.0.1). I have an external DNS that is resolving the IP correctly.

I am also able to access the webadmin interface for the server itself.

That is where things stand at the moment. 

badco's picture

badco - Mon, 2021/06/07 - 09:12

Can you show the error you get when you try to access the webUI? (screenshot would be good)

Veronica Charlton's picture

Veronica Charlton - Fri, 2021/06/18 - 23:33

I appologize for not getting back to you.

 

In all honesty I installed Centos in a new appliance and then gitlab onto that.

 

That is working and I have not gone back to try and figure out why the turnkey one isn't.

badco's picture

badco - Sun, 2021/06/20 - 05:52

Alternatively you could use TKLCore LXC and install gitlab into that, then you get webmin and other features.

Michael's picture

Michael - Thu, 2021/07/15 - 17:36

I have the same problem like the first comment. I can give a small Workaround for one part of the problem:

For restart GitLab after reboot you can do following on console:

systemctl enable gitlab-runsvdir
systemctl start gitlab-runsvdir

 

But for the 500-Error i have no solution yet.

Alex's picture

Alex - Thu, 2021/07/22 - 16:27

I'm trying to understand the gitlab 500 error but it's not easy.

The command `gitlab-rake gitlab:check --trace` report no issue:

root@git .../log/gitlab#  gitlab-rake gitlab:check --trace
** Invoke gitlab:check (first_time)
** Invoke gitlab_environment (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute gitlab_environment
** Execute gitlab:check
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.17.0 ? ... OK (13.17.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ... 
GitLab Instance / Monitoring ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.31.0 ? ... yes (2.31.1)
Git user has default SSH configuration? ... yes
Active users: ... 1
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

root@git .../log/gitlab#

 

The command `gitlab-ctl tail` report the error:

==> /var/log/gitlab/gitlab-rails/production.log <==
  
OpenSSL::Cipher::CipherError ():
  
lib/gitlab/database.rb:342:in `block in transaction'
lib/gitlab/database.rb:341:in `transaction'
app/services/application_settings/update_service.rb:50:in `update_settings'
lib/gitlab/metrics/instrumentation.rb:160:in `block in update_settings'
lib/gitlab/metrics/method_call.rb:27:in `measure'
lib/gitlab/metrics/instrumentation.rb:160:in `update_settings'
app/services/application_settings/update_service.rb:12:in `execute'
lib/gitlab/metrics/instrumentation.rb:160:in `block in execute'
lib/gitlab/metrics/method_call.rb:27:in `measure'
lib/gitlab/metrics/instrumentation.rb:160:in `execute'
app/controllers/application_controller.rb:531:in `disable_usage_stats'
app/controllers/application_controller.rb:517:in `set_usage_stats_consent_flag'
app/controllers/application_controller.rb:463:in `block in set_current_context'
lib/gitlab/application_context.rb:70:in `block in use'
lib/gitlab/application_context.rb:70:in `use'
lib/gitlab/application_context.rb:27:in `with_context'
app/controllers/application_controller.rb:454:in `set_current_context'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/transaction.rb:56:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:172:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:21:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

 

I also find an error in the postgresql which was not present in the previous package:

2021-07-21_21:05:08.72639 LOG:  configuration file "/var/opt/gitlab/postgresql/data/postgresql.conf" contains errors; unaffected changes were applied
2021-07-21_21:05:08.95919 LOG:  no match in usermap "gitlab" for user "gitlab" authenticated as "root"
2021-07-21_21:05:08.95921 FATAL:  Peer authentication failed for user "gitlab"
2021-07-21_21:05:08.95921 DETAIL:  Connection matched pg_hba.conf line 70: "local   all         all                               peer map=gitlab"
2021-07-21_21:05:09.04222 LOG:  no match in usermap "gitlab" for user "gitlab" authenticated as "root"
2021-07-21_21:05:09.04223 FATAL:  Peer authentication failed for user "gitlab"
2021-07-21_21:05:09.04224 DETAIL:  Connection matched pg_hba.conf line 70: "local   all         all                               peer map=gitlab"
2021-07-21_21:05:09.10794 LOG:  no match in usermap "gitlab" for user "gitlab" authenticated as "root"
2021-07-21_21:05:09.10796 FATAL:  Peer authentication failed for user "gitlab"
2021-07-21_21:05:09.10796 DETAIL:  Connection matched pg_hba.conf line 70: "local   all         all                               peer map=gitlab"

But the config line seems identical to the preivous package.

 

I will continue to investigate a bit.

 

kamaradski's picture

kamaradski - Wed, 2021/08/25 - 03:41

ok this explains why I couldn't get it to work the last couple of days. Thanks in advance for not giving up on this hard to maintain appliance.

just to confirm, even the `apt install gitlab-ce` is now broken:

root@gitlab ~# apt install gitlab-ce
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be upgraded:
  gitlab-ce
1 upgraded, 0 newly installed, 0 to remove and 9 not upgraded.
Need to get 940 MB of archives.
After this operation, 68.0 MB of additional disk space will be used.
Get:1 https://packages.gitlab.com/gitlab/gitlab-ce/debian buster/main amd64 gitlab-ce amd64 14.2.1-ce.0 [940 MB]
Fetched 940 MB in 14s (66.7 MB/s)                                                                                                                                                                  
[master fe5a4f2] saving uncommitted changes in /etc prior to apt run
 4 files changed, 0 insertions(+), 0 deletions(-)
 rename rc2.d/{S01turnkey-init-fence => K01turnkey-init-fence} (100%)
 rename rc3.d/{S01turnkey-init-fence => K01turnkey-init-fence} (100%)
 rename rc4.d/{S01turnkey-init-fence => K01turnkey-init-fence} (100%)
 rename rc5.d/{S01turnkey-init-fence => K01turnkey-init-fence} (100%)
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 107097 files and directories currently installed.)
Preparing to unpack .../gitlab-ce_14.2.1-ce.0_amd64.deb ...
gitlab preinstall: It seems you are upgrading from major version 13 to major version 14.
gitlab preinstall: It is required to upgrade to the latest 14.0.x version first before proceeding.
gitlab preinstall: Please follow the upgrade documentation at https://docs.gitlab.com/ee/update/index.html#upgrade-paths
dpkg: error processing archive /var/cache/apt/archives/gitlab-ce_14.2.1-ce.0_amd64.deb (--unpack):
 new gitlab-ce package pre-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/gitlab-ce_14.2.1-ce.0_amd64.deb
Enumerating objects: 1240, done.
Counting objects: 100% (1240/1240), done.
Delta compression using up to 4 threads
Compressing objects: 100% (782/782), done.
Writing objects: 100% (1240/1240), done.
Total 1240 (delta 69), reused 1234 (delta 66)
E: Sub-process /usr/bin/dpkg returned an error code (1)

Gitlab is way over my head, as it's a beast to maintain, so unfortunately I will be no help to you on this one :(

Jeremy Davis's picture

Jeremy Davis - Wed, 2021/08/25 - 06:39

If you look closely at the output you posted and you can see this line:

[...] Please follow the upgrade documentation at https://docs.gitlab.com/ee/update/index.html#upgrade-paths

Between the error message and looking at that doc page, it appears that you need to do some incremental updates before upgrading to latest v14.x. I.e. So try this:

apt install -y gitlab-ce=13.12.9-ce.0

Please note that I have guessed the version number format. Hopefully I've got it right, but it may be formatted slightly differently. If that command fails, please check the specific version number format like this:

apt policy gitlab-ce | grep 13.12.9

(If feeling unsure, please post the output and I'll help you interpret it).

Then next up, upgrade to v14.0.x, then v14.1.x (adjust version numbers as need be as per above) and finally the latest:

apt install -y gitlab-ce=14.0.7-ce.0
apt install -y gitlab-ce=14.1.2-ce.0
apt install -y gitlab-ce
Darrin's picture

Darrin - Tue, 2021/10/12 - 02:31

Are there any plans on releasing a new LXC with these fixes implemented, and the upgrades done?  I have to admit, I am mad at myself for not googling for the answer quicker.  I had just built a Ubuntu VM and installed GitLab, but for other reasons wanted to run it as a container on Proxmox so installed again.  After doing the upgrade dance hoping to fix the problem, turned out a simple update query was all that was needed.  Would be nice if a new container was released.

Jeremy Davis's picture

Jeremy Davis - Tue, 2021/10/12 - 04:57

The current build code has issues so building a new container won't fix it (the new container will be broken). I put some time aside to work on it but was unable to resolve the latest issues with certificates in the time allocated so had to move to other things.

Once we have the base of our upcoming v17.0 release underway, I will circle back to GitLab and try again. I'm confident that I'll be able to work it out, but I need to get these other higher priority items resolved first.

Webdug's picture

Webdug - Sat, 2021/10/16 - 15:54

Everything works (for me) now... Thanks everyone for the advice

Dario Balboni's picture

Dario Balboni - Thu, 2023/08/31 - 12:18
Just a quick note to everyone that experiences said problems, namely 50X errors after trying to change admin settings. From the command line: 
gitlab-rake gitlab:doctor:secrets VERBOSE=1 # Check that the errors are indeed on ApplicationSetting
gitlab-rails runner "ApplicationSetting.find(1).destroy" # Here you want to find the latest one if you have upgraded from earlier versions
 

Add new comment