Pete's picture

Hi Guys,

 

I deployed the ova template of the latest turnkey-gitlab and everything was working fine.  If we leave it for some time (possibly overnight) then when accessing the site we receive 502 Bad Gateway.  Running with 4Gb RAM and most issues i have seen relate to memory issues when runngin with 512MB or 1GB RAM so not sure this would be the issue.

Anyone experienced this with the latest version and could possibly shed some light?

 

Thanks,

Pete

Forum: 
Jeremy Davis's picture

At first glance I was going to suggest more RAM but 4GB should be plenty... And the fact that this doesn't happen immediately suggests that that may not be the problem.

Obviously the Nginx front end is running (otherwise you wouldn't be able to connect at all). Probably the first thing to check is whether GitLab itslef is running. something like:

service gitlab-unicorn status
service gitlab-sidekiq status
Assuming one of those is not running the GitLab log (/home/git/gitlab/log) may give you some hints on what the issue is. If those services are running then I'd check the Nginx error log (/var/log/nginx/gitlab.error.log) for clues.
Pete's picture

I doubled the RAM on Friday night as a test and everything is running smoothly this morning when I checked so unable to check service status.  I do see  the below in the gitlab.error.log file around the time it failed on Friday

2015/12/04 17:44:57 [error] 608#0: *2057 connect() to unix:/home/git/gitlab/tmp/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream, client: 10.181.24.154, server: , request: "GET / HTTP/1.1", upstream: "http://unix:/home/git/gitlab/tmp/sockets/gitlab.socket:/", host: "ccgitlab01.domain.local"

 

also looking through the GitLab Logs (/home/git/gitlab/log/unicorn.stderr.log) I see a large number of "exceeds memory limit" errors ... interestingly enough these errors were logged when the system failed and also now while the system is working?

W, [2015-12-07T16:21:24.098020 #27732]  WARN -- : #<Unicorn::HttpServer:0x00000001ea43d0>: worker (pid: 27732) exceeds memory limit (232906752 bytes > 220520031 bytes)
W, [2015-12-07T16:21:24.098114 #27732]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 27732) alive: 662 sec (trial 1)
I, [2015-12-07T16:21:24.320918 #1194]  INFO -- : reaped #<Process::Status: pid 27732 exit 0> worker=1
I, [2015-12-07T16:21:24.324009 #27815]  INFO -- : worker=1 ready

I will post some more information if it fails again with 8GB RAM

Thanks very much

Pete

Jeremy Davis's picture

AFAIK the more users you have the more RAM you need. 8GB seems like a lot for it to require; although maybe thats reasonable if you have a lot of users? Actually I just checked the GitLab docs and they say that 4GB should support up to 1000 users!

If RAM is a precious commodity; then you could always add more swap (instead or as well as) and see how that affects performance; suggestion here

Pete's picture

Only change from vanilla install was to enable LDAP integration and then 2 users were auto-created when they logged in before we started seeing the issue

I will let you know if I see any failures after upgrading to 8GB RAM

Jeremy Davis's picture

Unfortunately GitLab v14.0 won't run inside a container (with the exception of the Docker build). Is that were you have this running?

Add new comment