Gitlab Turnkey - 502 Bad Gateway

Hi Guys,

I deployed the ova template of the latest turnkey-gitlab and everything was working fine. If we leave it for some time (possibly overnight) then when accessing the site we receive 502 Bad Gateway. Running with 4Gb RAM and most issues i have seen relate to memory issues when runngin with 512MB or 1GB RAM so not sure this would be the issue.

Anyone experienced this with the latest version and could possibly shed some light?

Thanks,

Pete

Replies (5)

Hmm that is weird...

Reply 1 2015-12-05

At first glance I was going to suggest more RAM but 4GB should be plenty... And the fact that this doesn't happen immediately suggests that that may not be the problem.

Obviously the Nginx front end is running (otherwise you wouldn't be able to connect at all). Probably the first thing to check is whether GitLab itslef is running. something like:

service gitlab-unicorn status
service gitlab-sidekiq status

Assuming one of those is not running the GitLab log (/home/git/gitlab/log) may give you some hints on what the issue is. If those services are running then I'd check the Nginx error log (/var/log/nginx/gitlab.error.log) for clues.

Do you have lots of users?

Reply 2 2015-12-07

AFAIK the more users you have the more RAM you need. 8GB seems like a lot for it to require; although maybe thats reasonable if you have a lot of users? Actually I just checked the GitLab docs and they say that 4GB should support up to 1000 users!

If RAM is a precious commodity; then you could always add more swap (instead or as well as) and see how that affects performance; suggestion here

upping RAM to 8GB seems to have fixed it for now

Reply 3 2015-12-07

I doubled the RAM on Friday night as a test and everything is running smoothly this morning when I checked so unable to check service status. I do see the below in the gitlab.error.log file around the time it failed on Friday

2015/12/04 17:44:57 [error] 608#0: *2057 connect() to unix:/home/git/gitlab/tmp/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream, client: 10.181.24.154, server: , request: "GET / HTTP/1.1", upstream: "http://unix:/home/git/gitlab/tmp/sockets/gitlab.socket:/", host: "ccgitlab01.domain.local"

also looking through the GitLab Logs (/home/git/gitlab/log/unicorn.stderr.log) I see a large number of "exceeds memory limit" errors ... interestingly enough these errors were logged when the system failed and also now while the system is working?

W, [2015-12-07T16:21:24.098020 #27732]  WARN -- : #<Unicorn::HttpServer:0x00000001ea43d0>: worker (pid: 27732) exceeds memory limit (232906752 bytes > 220520031 bytes)
W, [2015-12-07T16:21:24.098114 #27732]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 27732) alive: 662 sec (trial 1)
I, [2015-12-07T16:21:24.320918 #1194]  INFO -- : reaped #<Process::Status: pid 27732 exit 0> worker=1
I, [2015-12-07T16:21:24.324009 #27815]  INFO -- : worker=1 ready

I will post some more information if it fails again with 8GB RAM

Thanks very much

Pete

Only 2 users via LDAP

Reply 4 2015-12-07

Only change from vanilla install was to enable LDAP integration and then 2 users were auto-created when they logged in before we started seeing the issue

I will let you know if I see any failures after upgrading to 8GB RAM

Where are you running this?

Reply 5 2016-01-14

Unfortunately GitLab v14.0 won't run inside a container (with the exception of the Docker build). Is that were you have this running?