Ivan D Marin's picture

Hey,

I am very new at these kind of things.

I was able to start the server and make it run, however, while configuring the SMTP from the confconsole and later restarted to see if it took the new configuration but now I get a constant 502 error when I try to access either the DNS or the IP itself.

I tried multiple things inside the forums but either got permission errors even when using sudo or simply path non existing.

These posts are from years ago for versions 13 or even later. Would love some version 16 help here.

Thanks

Ivan.

Forum: 
Jeremy Davis's picture

A 502 error suggests that the Mattermost service is not running. That could be because there is something wrong, or it might just need to be started (generally it should auto start though, so something wrong seems likely).

I wouldn't have suspected that doing the SMTP config from Confconsole would be the cause as that adjusts system settings and should not interfere with Mattermost at all. So I suspect that is coincidental (although I may be wrong...).

Thanks for noting that you are running the v16.x version.

Ideally it'd be good to try to understand what happened, so I suggest checking the status of the Mattermost service. Do that like this (if you aren't running as the 'root' user, prefix commands with 'sudo').

systemctl status mattermost

Hopefully that might give some insight into what went wrong? Please post the output here and I can give some help interpreting it.

If there isn't any obvious error message there, or you feel like trying to restart it anyway, assuming that it's currently stopped, you could try starting it and see what happens:

systemctl start mattermost

If it fails to start, please post back the output of this:

journalctl -u mattermost
Ivan D Marin's picture

Hi Jeremy! 

Thank you for your quick response.

I went ahead and used all the commands that you mentioned and the service shows as active but exited. And starting the service throws an error. As requested, below are the outputs of all the commands:

 

systemctl status mattermost
* mattermost.service - LSB: Mattermost Group Chat
   Loaded: loaded (/etc/init.d/mattermost; generated)
   Active: active (exited) since Tue 2020-11-03 23:41:13 UTC; 5h 53min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 789 ExecStart=/etc/init.d/mattermost start (code=exited, status=0/SUC

systemctl start mattermost
Failed to start mattermost.service: Access denied
See system logs and 'systemctl status mattermost.service' for details.

journalctl -u mattermost
Hint: You are currently not seeing messages from other users and the system.
      Users in the 'systemd-journal' group can see all messages. Pass -q to
      turn off this notice.
No journal files were opened due to insufficient permissions.

 

Looking forward to your reply and thanks again.

Jeremy Davis's picture

From the output of those commands I assume that you aren't logged in as the root user. So can you please try again with a sudo prefix (plus I've added some additional commands). FWIW I did note that if you aren't running as the 'root' user, prefix commands with 'sudo' but obviously I didn't make it clear enough.

So please try again, this time with a 'sudo' prefix:

sudo systemctl status mattermost
sudo systemctl restart mattermost
sudo journalctl -u mattermost
sudo netstat -tlnp
sudo ps aux | grep mattermost

Also, you should find at least one (perhaps more) log file in /var/log/mattermost. Check for files like this (you may need need sudo, but just in case...):

sudo ls -l /var/log/mattermost

The file names will be the last part. Then use the 'cat' command to show the contents of the main log file. My guess is that it's probably called either just 'log', or perhaps it's 'mattermost.log'. So assuming 'mattermost.log':

sudo cat /var/log/mattermost/mattermost.log
Ivan D Marin's picture

Thank you for your reply,

Here is all the findings and commands you asked me about. You can probably see what happens without me explaining it. But here is what happened so far.

 

sudo systemctl status mattermost
* mattermost.service - LSB: Mattermost Group Chat
   Loaded: loaded (/etc/init.d/mattermost; generated)
   Active: active (exited) since Tue 2020-11-03 23:41:13 UTC; 15h ago
     Docs: man:systemd-sysv-generator(8)
  Process: 789 ExecStart=/etc/init.d/mattermost start (code=exited, status=0/SUCCESS)

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.


After using restart mattermost.
sudo systemctl status mattermost
* mattermost.service - LSB: Mattermost Group Chat
   Loaded: loaded (/etc/init.d/mattermost; generated)
   Active: active (exited) since Wed 2020-11-04 15:16:36 UTC; 52s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 21724 ExecStart=/etc/init.d/mattermost start (code=exited, status=0/SUCCESS)

Nov 04 15:16:36 3 systemd[1]: mattermost.service: Succeeded.
Nov 04 15:16:36 3 systemd[1]: Stopped LSB: Mattermost Group Chat.
Nov 04 15:16:36 3 systemd[1]: Starting LSB: Mattermost Group Chat...
Nov 04 15:16:36 3 mattermost[21724]: Starting Mattermost: mattermost.
Nov 04 15:16:36 3 systemd[1]: Started LSB: Mattermost Group Chat.

 

But the gateway error is there.

sudo journalctl -u mattermost
-- Logs begin at Wed 2020-11-04 06:49:49 UTC, end at Wed 2020-11-04 15:20:18 UTC. --
Nov 04 15:16:36 3 systemd[1]: Stopping LSB: Mattermost Group Chat...
Nov 04 15:16:36 3 mattermost[21718]: Stopping Mattermost: mattermost.
Nov 04 15:16:36 3 systemd[1]: mattermost.service: Succeeded.
Nov 04 15:16:36 3 systemd[1]: Stopped LSB: Mattermost Group Chat.
Nov 04 15:16:36 3 systemd[1]: Starting LSB: Mattermost Group Chat...
Nov 04 15:16:36 3 mattermost[21724]: Starting Mattermost: mattermost.
Nov 04 15:16:36 3 systemd[1]: Started LSB: Mattermost Group Chat.

 sudo netstat -tlnp


Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      619/mysqld
tcp        0      0 127.0.0.1:10000         0.0.0.0:*               LISTEN      822/perl
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      478/nginx: master p
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      465/sshd
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      496/postgres
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      478/nginx: master p
tcp        0      0 127.0.0.1:12319         0.0.0.0:*               LISTEN      627/shellinaboxd
tcp        0      0 0.0.0.0:12320           0.0.0.0:*               LISTEN      415/stunnel4
tcp        0      0 0.0.0.0:12321           0.0.0.0:*               LISTEN      414/stunnel4
tcp        0      0 0.0.0.0:12322           0.0.0.0:*               LISTEN      478/nginx: master p
tcp6       0      0 :::22                   :::*                    LISTEN      465/sshd

 

sudo ps aux | grep mattermost
admin    21863  0.0  0.0   3084   884 pts/0    S+   15:22   0:00 grep --color=auto mattermost

 

sudo ls -l /var/log/mattermost
lrwxrwxrwx 1 root root 20 May 19 09:06 /var/log/mattermost -> /opt/mattermost/logs

 

Here are the logs after I found out about the server not sending the emails to registered users (I guessed that it could be due to the SMTP not being configured, if you could confirm that would me as an extra that would be amazing)

 

sudo cat /var/log/mattermost/mattermost.log

{"level":"error","ts":1604416110.8391392,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604416224.777207,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604416226.4958405,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604416230.0570052,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604416239.603888,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604416260.9385293,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604416401.1343484,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604416670.2690942,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604417260.1147933,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"craig@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604417264.8273256,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"craig@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604417278.4053977,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"craig@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604417417.0314841,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"craig@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604420010.8870022,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604422897.718368,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604423061.2708125,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604423063.983516,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604423844.6949112,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604424368.2029579,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604424372.951456,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604424382.7043073,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604424384.3050628,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604424386.4853554,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604424391.248235,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604424397.1774147,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604424399.060931,"caller":"app/notification_email.go:103","msg":"Error while sending the email","user_email":"chevonne@sirius-support.com","error":"SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided"}
{"level":"error","ts":1604425002.5016952,"caller":"mlog/log.go:175","msg":"Connection unsuccessful: SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided","path":"/api/v4/email/test","request_id":"wg1bf6nykpyrbgm3wywihn3mnw","ip_addr":"190.140.146.94","user_id":"msi4ot3iqidn7xcprdpotujqqe","method":"POST","err_where":"testEmail","http_code":500,"err_details":""}
{"level":"error","ts":1604425014.9563532,"caller":"mlog/log.go:175","msg":"Connection unsuccessful: SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided","path":"/api/v4/email/test","request_id":"y893wajn6i8zdphtyu6mymt9zy","ip_addr":"190.140.146.94","user_id":"msi4ot3iqidn7xcprdpotujqqe","method":"POST","err_where":"testEmail","http_code":500,"err_details":""}
{"level":"error","ts":1604425018.6332467,"caller":"mlog/log.go:175","msg":"Connection unsuccessful: SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided","path":"/api/v4/email/test","request_id":"a6zh6d8zqbbbtj1jpgiuqb7xar","ip_addr":"190.140.146.94","user_id":"msi4ot3iqidn7xcprdpotujqqe","method":"POST","err_where":"testEmail","http_code":500,"err_details":""}
{"level":"error","ts":1604425022.134368,"caller":"mlog/log.go:175","msg":"Connection unsuccessful: SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided","path":"/api/v4/email/test","request_id":"kq8oadmuhfgjddjgj6kw8pjqse","ip_addr":"190.140.146.94","user_id":"msi4ot3iqidn7xcprdpotujqqe","method":"POST","err_where":"testEmail","http_code":500,"err_details":""}
{"level":"error","ts":1604425038.3730745,"caller":"mlog/log.go:175","msg":"Connection unsuccessful: SendMail: Failed to open TLS connection., context deadline exceeded","path":"/api/v4/email/test","request_id":"e5yeyjj1tpgu9x71g9w8ijtuma","ip_addr":"190.140.146.94","user_id":"msi4ot3iqidn7xcprdpotujqqe","method":"POST","err_where":"testEmail","http_code":500,"err_details":""}
{"level":"error","ts":1604425347.1586845,"caller":"mlog/log.go:175","msg":"Connection unsuccessful: SendMail: Failed to open TLS connection., context deadline exceeded","path":"/api/v4/email/test","request_id":"hysexfpzdifgfkp3xro95atwzr","ip_addr":"190.140.146.94","user_id":"msi4ot3iqidn7xcprdpotujqqe","method":"POST","err_where":"testEmail","http_code":500,"err_details":""}
{"level":"error","ts":1604425363.740589,"caller":"mlog/log.go:175","msg":"Connection unsuccessful: SendMail: Failed to open TLS connection., context deadline exceeded","path":"/api/v4/email/test","request_id":"kqtgq6ww1pfpudbhyxfdzm7t4o","ip_addr":"190.140.146.94","user_id":"msi4ot3iqidn7xcprdpotujqqe","method":"POST","err_where":"testEmail","http_code":500,"err_details":""}
{"level":"error","ts":1604425386.4115517,"caller":"mlog/log.go:175","msg":"Connection unsuccessful: SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided","path":"/api/v4/email/test","request_id":"x31d5wu8h3rixeb9ejaisfqama","ip_addr":"190.140.146.94","user_id":"msi4ot3iqidn7xcprdpotujqqe","method":"POST","err_where":"testEmail","http_code":500,"err_details":""}
{"level":"error","ts":1604425389.8293517,"caller":"mlog/log.go:175","msg":"Connection unsuccessful: SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided","path":"/api/v4/email/test","request_id":"r4a7xnwz57ripkmf4xdjejx3ww","ip_addr":"190.140.146.94","user_id":"msi4ot3iqidn7xcprdpotujqqe","method":"POST","err_where":"testEmail","http_code":500,"err_details":""}
{"level":"error","ts":1604425397.7048059,"caller":"mlog/log.go:175","msg":"Connection unsuccessful: SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided","path":"/api/v4/email/test","request_id":"1c1kgcrhcjyddyrzz1jz1kfcfy","ip_addr":"190.140.146.94","user_id":"msi4ot3iqidn7xcprdpotujqqe","method":"POST","err_where":"testEmail","http_code":500,"err_details":""}
{"level":"error","ts":1604426020.4446352,"caller":"mlog/log.go:175","msg":"Failed to send password reset email successfully.","path":"/api/v4/users/password/reset/send","request_id":"ugykse3prjfapgi9j63bgjibma","ip_addr":"190.140.146.94","user_id":"","method":"POST","err_where":"SendPasswordReset","http_code":500,"err_details":"err=Error setting \"From Address\""}
{"level":"error","ts":1604426153.952183,"caller":"mlog/log.go:175","msg":"Connection unsuccessful: SendMail: Error setting \"From Address\", 501 Invalid MAIL FROM address provided","path":"/api/v4/email/test","request_id":"mdqawxpmspd3tpqn59iy95rs5h","ip_addr":"190.140.146.94","user_id":"msi4ot3iqidn7xcprdpotujqqe","method":"POST","err_where":"testEmail","http_code":500,"err_details":""}
{"level":"error","ts":1604426168.7113433,"caller":"mlog/log.go:175","msg":"Failed to send password reset email successfully.","path":"/api/v4/users/password/reset/send","request_id":"8yij5ukojffn9r8q51amp1c8wy","ip_addr":"190.140.146.94","user_id":"","method":"POST","err_where":"SendPasswordReset","http_code":500,"err_details":"err=Failed to open connection."}
{"level":"error","ts":1604426171.252347,"caller":"mlog/log.go:175","msg":"Failed to send password reset email successfully.","path":"/api/v4/users/password/reset/send","request_id":"h1fi1xxs53gq7b8mu7isy1sfuh","ip_addr":"190.140.146.94","user_id":"","method":"POST","err_where":"SendPasswordReset","http_code":500,"err_details":"err=Failed to open connection."}
{"level":"info","ts":1604426206.2101674,"caller":"app/server.go:402","msg":"Stopping Server..."}
{"level":"info","ts":1604426206.210225,"caller":"app/web_hub.go:130","msg":"stopping websocket hub connections"}
{"level":"info","ts":1604426206.2215881,"caller":"app/plugin.go:279","msg":"Shutting down plugins"}
{"level":"info","ts":1604426206.2389686,"caller":"jobs/workers.go:184","msg":"Stopped workers"}
{"level":"info","ts":1604426206.2390046,"caller":"jobs/schedulers.go:145","msg":"Stopping schedulers."}
{"level":"info","ts":1604426206.239022,"caller":"jobs/schedulers.go:77","msg":"Schedulers stopped."}
{"level":"info","ts":1604426206.2421365,"caller":"app/server.go:453","msg":"Server stopped"}
{"level":"info","ts":1604426337.6004832,"caller":"utils/i18n.go:83","msg":"Loaded system translations","for locale":"en","from locale":"/opt/mattermost/i18n/en.json"}
{"level":"info","ts":1604426337.622714,"caller":"app/server_app_adapters.go:58","msg":"Server is initializing..."}
{"level":"info","ts":1604426337.6624696,"caller":"sqlstore/supplier.go:224","msg":"Pinging SQL","database":"master"}
{"level":"error","ts":1604426338.1087706,"caller":"app/server_app_adapters.go:139","msg":"Mail server connection test is failed: Could not connect to SMTP server, check SMTP server settings."}
{"level":"info","ts":1604426338.1202345,"caller":"mlog/log.go:167","msg":"Starting up plugins"}
{"level":"info","ts":1604426338.1222703,"caller":"app/plugin.go:202","msg":"Syncing plugins from the file store"}
{"level":"info","ts":1604426338.15529,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/jitsi.tar.gz"}
{"level":"info","ts":1604426338.1556354,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/zoom.tar.gz"}
{"level":"info","ts":1604426338.1573002,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/com.mattermost.aws-sns.tar.gz"}
{"level":"info","ts":1604426346.7355423,"caller":"mlog/sugar.go:19","msg":"Ensuring Surveybot exists","plugin_id":"com.mattermost.nps"}
{"level":"info","ts":1604426346.7700264,"caller":"app/server.go:255","msg":"Current version is 5.23.0 (5.23.0/Thu May 14 12:18:07 UTC 2020/ed1576d007a09b07574b18784b9eabb3d8c9db53/none)","current_version":"5.23.0","build_number":"5.23.0","build_date":"Thu May 14 12:18:07 UTC 2020","build_hash":"ed1576d007a09b07574b18784b9eabb3d8c9db53","build_hash_enterprise":"none"}
{"level":"info","ts":1604426346.7700636,"caller":"app/server.go:266","msg":"Team Edition Build","enterprise_build":false}
{"level":"info","ts":1604426346.7700849,"caller":"app/server.go:270","msg":"Printing current working","directory":"/opt/mattermost"}
{"level":"info","ts":1604426346.7700982,"caller":"app/server.go:271","msg":"Loaded config","source":"file:///opt/mattermost/config/config.json"}
{"level":"info","ts":1604426346.83523,"caller":"sqlstore/post_store.go:1524","msg":"Post.Message has size restrictions","max_characters":16383,"max_bytes":65535}
{"level":"info","ts":1604426346.9446394,"caller":"jobs/workers.go:68","msg":"Starting workers"}
{"level":"info","ts":1604426346.9553895,"caller":"jobs/schedulers.go:74","msg":"Starting schedulers."}
{"level":"info","ts":1604426346.980512,"caller":"app/web_hub.go:75","msg":"Starting websocket hubs","number_of_hubs":2}
{"level":"info","ts":1604426347.0750244,"caller":"app/server.go:509","msg":"Starting Server..."}
{"level":"error","ts":1604426347.0751793,"caller":"commands/server.go:81","msg":"listen tcp 3.250.25.199:443: bind: cannot assign requested address"}
{"level":"info","ts":1604426347.0751991,"caller":"app/server.go:402","msg":"Stopping Server..."}
{"level":"info","ts":1604426347.0752115,"caller":"app/web_hub.go:130","msg":"stopping websocket hub connections"}
{"level":"info","ts":1604426347.0759432,"caller":"app/plugin.go:279","msg":"Shutting down plugins"}
{"level":"info","ts":1604426360.5722115,"caller":"jobs/workers.go:184","msg":"Stopped workers"}
{"level":"info","ts":1604426360.572269,"caller":"jobs/schedulers.go:145","msg":"Stopping schedulers."}
{"level":"info","ts":1604426360.5722885,"caller":"jobs/schedulers.go:77","msg":"Schedulers stopped."}
{"level":"info","ts":1604426360.572482,"caller":"app/server.go:453","msg":"Server stopped"}
{"level":"info","ts":1604426862.2643986,"caller":"utils/i18n.go:83","msg":"Loaded system translations","for locale":"en","from locale":"/opt/mattermost/i18n/en.json"}
{"level":"info","ts":1604426862.2671406,"caller":"app/server_app_adapters.go:58","msg":"Server is initializing..."}
{"level":"info","ts":1604426862.2868862,"caller":"sqlstore/supplier.go:224","msg":"Pinging SQL","database":"master"}
{"level":"error","ts":1604426862.8360224,"caller":"app/server_app_adapters.go:139","msg":"Mail server connection test is failed: Could not connect to SMTP server, check SMTP server settings."}
{"level":"info","ts":1604426862.924489,"caller":"mlog/log.go:167","msg":"Starting up plugins"}
{"level":"info","ts":1604426862.9444673,"caller":"app/plugin.go:202","msg":"Syncing plugins from the file store"}
{"level":"info","ts":1604426863.0006466,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/com.mattermost.aws-sns.tar.gz"}
{"level":"info","ts":1604426863.0199018,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/jitsi.tar.gz"}
{"level":"info","ts":1604426863.0401433,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/zoom.tar.gz"}
{"level":"info","ts":1604426872.4984405,"caller":"mlog/sugar.go:19","msg":"Ensuring Surveybot exists","plugin_id":"com.mattermost.nps"}
{"level":"info","ts":1604426872.5172126,"caller":"app/server.go:255","msg":"Current version is 5.23.0 (5.23.0/Thu May 14 12:18:07 UTC 2020/ed1576d007a09b07574b18784b9eabb3d8c9db53/none)","current_version":"5.23.0","build_number":"5.23.0","build_date":"Thu May 14 12:18:07 UTC 2020","build_hash":"ed1576d007a09b07574b18784b9eabb3d8c9db53","build_hash_enterprise":"none"}
{"level":"info","ts":1604426872.517247,"caller":"app/server.go:266","msg":"Team Edition Build","enterprise_build":false}
{"level":"info","ts":1604426872.517267,"caller":"app/server.go:270","msg":"Printing current working","directory":"/opt/mattermost"}
{"level":"info","ts":1604426872.5172796,"caller":"app/server.go:271","msg":"Loaded config","source":"file:///opt/mattermost/config/config.json"}
{"level":"info","ts":1604426872.5660417,"caller":"sqlstore/post_store.go:1524","msg":"Post.Message has size restrictions","max_characters":16383,"max_bytes":65535}
{"level":"info","ts":1604426872.637497,"caller":"jobs/workers.go:68","msg":"Starting workers"}
{"level":"info","ts":1604426872.6508627,"caller":"jobs/schedulers.go:74","msg":"Starting schedulers."}
{"level":"info","ts":1604426872.7312126,"caller":"app/web_hub.go:75","msg":"Starting websocket hubs","number_of_hubs":2}
{"level":"info","ts":1604426872.828919,"caller":"app/server.go:509","msg":"Starting Server..."}
{"level":"error","ts":1604426872.8290577,"caller":"commands/server.go:81","msg":"listen tcp 3.250.25.199:443: bind: cannot assign requested address"}
{"level":"info","ts":1604426872.8290777,"caller":"app/server.go:402","msg":"Stopping Server..."}
{"level":"info","ts":1604426872.8290925,"caller":"app/web_hub.go:130","msg":"stopping websocket hub connections"}
{"level":"info","ts":1604426872.8296573,"caller":"app/plugin.go:279","msg":"Shutting down plugins"}
{"level":"info","ts":1604426884.1010628,"caller":"jobs/workers.go:184","msg":"Stopped workers"}
{"level":"info","ts":1604426884.1011212,"caller":"jobs/schedulers.go:145","msg":"Stopping schedulers."}
{"level":"info","ts":1604426884.1011395,"caller":"jobs/schedulers.go:77","msg":"Schedulers stopped."}
{"level":"info","ts":1604426884.1013343,"caller":"app/server.go:453","msg":"Server stopped"}
{"level":"info","ts":1604427343.8532548,"caller":"utils/i18n.go:83","msg":"Loaded system translations","for locale":"en","from locale":"/opt/mattermost/i18n/en.json"}
{"level":"info","ts":1604427343.8556435,"caller":"app/server_app_adapters.go:58","msg":"Server is initializing..."}
{"level":"info","ts":1604427343.8704126,"caller":"sqlstore/supplier.go:224","msg":"Pinging SQL","database":"master"}
{"level":"error","ts":1604427344.0206149,"caller":"app/server_app_adapters.go:139","msg":"Mail server connection test is failed: Could not connect to SMTP server, check SMTP server settings."}
{"level":"info","ts":1604427344.0460048,"caller":"mlog/log.go:167","msg":"Starting up plugins"}
{"level":"info","ts":1604427344.0478194,"caller":"app/plugin.go:202","msg":"Syncing plugins from the file store"}
{"level":"info","ts":1604427344.096813,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/zoom.tar.gz"}
{"level":"info","ts":1604427344.1025205,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/com.mattermost.aws-sns.tar.gz"}
{"level":"info","ts":1604427344.1227489,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/jitsi.tar.gz"}
{"level":"info","ts":1604427353.252266,"caller":"mlog/sugar.go:19","msg":"Ensuring Surveybot exists","plugin_id":"com.mattermost.nps"}
{"level":"info","ts":1604427353.2675548,"caller":"app/server.go:255","msg":"Current version is 5.23.0 (5.23.0/Thu May 14 12:18:07 UTC 2020/ed1576d007a09b07574b18784b9eabb3d8c9db53/none)","current_version":"5.23.0","build_number":"5.23.0","build_date":"Thu May 14 12:18:07 UTC 2020","build_hash":"ed1576d007a09b07574b18784b9eabb3d8c9db53","build_hash_enterprise":"none"}
{"level":"info","ts":1604427353.2676096,"caller":"app/server.go:266","msg":"Team Edition Build","enterprise_build":false}
{"level":"info","ts":1604427353.267633,"caller":"app/server.go:270","msg":"Printing current working","directory":"/opt/mattermost"}
{"level":"info","ts":1604427353.2676458,"caller":"app/server.go:271","msg":"Loaded config","source":"file:///opt/mattermost/config/config.json"}
{"level":"info","ts":1604427353.320063,"caller":"sqlstore/post_store.go:1524","msg":"Post.Message has size restrictions","max_characters":16383,"max_bytes":65535}
{"level":"info","ts":1604427353.4164343,"caller":"jobs/workers.go:68","msg":"Starting workers"}
{"level":"info","ts":1604427353.4262745,"caller":"jobs/schedulers.go:74","msg":"Starting schedulers."}
{"level":"info","ts":1604427353.4352193,"caller":"app/web_hub.go:75","msg":"Starting websocket hubs","number_of_hubs":2}
{"level":"info","ts":1604427353.5321176,"caller":"app/server.go:509","msg":"Starting Server..."}
{"level":"error","ts":1604427353.5322793,"caller":"commands/server.go:81","msg":"listen tcp 3.250.25.199:443: bind: cannot assign requested address"}
{"level":"info","ts":1604427353.5322976,"caller":"app/server.go:402","msg":"Stopping Server..."}
{"level":"info","ts":1604427353.5323098,"caller":"app/web_hub.go:130","msg":"stopping websocket hub connections"}
{"level":"info","ts":1604427353.5332553,"caller":"app/plugin.go:279","msg":"Shutting down plugins"}
{"level":"info","ts":1604427359.4715729,"caller":"jobs/workers.go:184","msg":"Stopped workers"}
{"level":"info","ts":1604427359.471673,"caller":"jobs/schedulers.go:145","msg":"Stopping schedulers."}
{"level":"info","ts":1604427359.47171,"caller":"jobs/schedulers.go:77","msg":"Schedulers stopped."}
{"level":"info","ts":1604427359.4719331,"caller":"app/server.go:453","msg":"Server stopped"}
{"level":"info","ts":1604428114.362519,"caller":"utils/i18n.go:83","msg":"Loaded system translations","for locale":"en","from locale":"/opt/mattermost/i18n/en.json"}
{"level":"info","ts":1604428114.3641436,"caller":"app/server_app_adapters.go:58","msg":"Server is initializing..."}
{"level":"info","ts":1604428114.3848944,"caller":"sqlstore/supplier.go:224","msg":"Pinging SQL","database":"master"}
{"level":"error","ts":1604428114.7451699,"caller":"app/server_app_adapters.go:139","msg":"Mail server connection test is failed: Could not connect to SMTP server, check SMTP server settings."}
{"level":"info","ts":1604428114.809761,"caller":"mlog/log.go:167","msg":"Starting up plugins"}
{"level":"info","ts":1604428114.817676,"caller":"app/plugin.go:202","msg":"Syncing plugins from the file store"}
{"level":"info","ts":1604428114.862728,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/com.mattermost.aws-sns.tar.gz"}
{"level":"info","ts":1604428114.8647976,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/jitsi.tar.gz"}
{"level":"info","ts":1604428114.8651912,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/zoom.tar.gz"}
{"level":"info","ts":1604428122.945352,"caller":"mlog/sugar.go:19","msg":"Ensuring Surveybot exists","plugin_id":"com.mattermost.nps"}
{"level":"info","ts":1604428122.962241,"caller":"app/server.go:255","msg":"Current version is 5.23.0 (5.23.0/Thu May 14 12:18:07 UTC 2020/ed1576d007a09b07574b18784b9eabb3d8c9db53/none)","current_version":"5.23.0","build_number":"5.23.0","build_date":"Thu May 14 12:18:07 UTC 2020","build_hash":"ed1576d007a09b07574b18784b9eabb3d8c9db53","build_hash_enterprise":"none"}
{"level":"info","ts":1604428122.9622805,"caller":"app/server.go:266","msg":"Team Edition Build","enterprise_build":false}
{"level":"info","ts":1604428122.962328,"caller":"app/server.go:270","msg":"Printing current working","directory":"/opt/mattermost"}
{"level":"info","ts":1604428122.9623451,"caller":"app/server.go:271","msg":"Loaded config","source":"file:///opt/mattermost/config/config.json"}
{"level":"info","ts":1604428123.0743418,"caller":"sqlstore/post_store.go:1524","msg":"Post.Message has size restrictions","max_characters":16383,"max_bytes":65535}
{"level":"info","ts":1604428123.1461809,"caller":"jobs/workers.go:68","msg":"Starting workers"}
{"level":"info","ts":1604428123.162566,"caller":"jobs/schedulers.go:74","msg":"Starting schedulers."}
{"level":"info","ts":1604428123.1869218,"caller":"app/web_hub.go:75","msg":"Starting websocket hubs","number_of_hubs":2}
{"level":"info","ts":1604428123.3318455,"caller":"app/server.go:509","msg":"Starting Server..."}
{"level":"error","ts":1604428123.331992,"caller":"commands/server.go:81","msg":"listen tcp 3.250.25.199:443: bind: cannot assign requested address"}
{"level":"info","ts":1604428123.3320117,"caller":"app/server.go:402","msg":"Stopping Server..."}
{"level":"info","ts":1604428123.3320227,"caller":"app/web_hub.go:130","msg":"stopping websocket hub connections"}
{"level":"info","ts":1604428123.3325915,"caller":"app/plugin.go:279","msg":"Shutting down plugins"}
{"level":"info","ts":1604428124.860737,"caller":"jobs/workers.go:184","msg":"Stopped workers"}
{"level":"info","ts":1604428124.8607852,"caller":"jobs/schedulers.go:145","msg":"Stopping schedulers."}
{"level":"info","ts":1604428124.8608015,"caller":"jobs/schedulers.go:77","msg":"Schedulers stopped."}
{"level":"info","ts":1604428124.8609877,"caller":"app/server.go:453","msg":"Server stopped"}
{"level":"info","ts":1604429128.4616988,"caller":"utils/i18n.go:83","msg":"Loaded system translations","for locale":"en","from locale":"/opt/mattermost/i18n/en.json"}
{"level":"info","ts":1604429128.4834657,"caller":"app/server_app_adapters.go:58","msg":"Server is initializing..."}
{"level":"info","ts":1604429128.5225499,"caller":"sqlstore/supplier.go:224","msg":"Pinging SQL","database":"master"}
{"level":"error","ts":1604429128.785076,"caller":"app/server_app_adapters.go:139","msg":"Mail server connection test is failed: Could not connect to SMTP server, check SMTP server settings."}
{"level":"info","ts":1604429128.8169346,"caller":"mlog/log.go:167","msg":"Starting up plugins"}
{"level":"info","ts":1604429128.8187513,"caller":"app/plugin.go:202","msg":"Syncing plugins from the file store"}
{"level":"info","ts":1604429128.846923,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/zoom.tar.gz"}
{"level":"info","ts":1604429128.8530617,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/com.mattermost.aws-sns.tar.gz"}
{"level":"info","ts":1604429128.8733,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/jitsi.tar.gz"}
{"level":"info","ts":1604429137.5827897,"caller":"mlog/sugar.go:19","msg":"Ensuring Surveybot exists","plugin_id":"com.mattermost.nps"}
{"level":"info","ts":1604429137.5973246,"caller":"app/server.go:255","msg":"Current version is 5.23.0 (5.23.0/Thu May 14 12:18:07 UTC 2020/ed1576d007a09b07574b18784b9eabb3d8c9db53/none)","current_version":"5.23.0","build_number":"5.23.0","build_date":"Thu May 14 12:18:07 UTC 2020","build_hash":"ed1576d007a09b07574b18784b9eabb3d8c9db53","build_hash_enterprise":"none"}
{"level":"info","ts":1604429137.5973566,"caller":"app/server.go:266","msg":"Team Edition Build","enterprise_build":false}
{"level":"info","ts":1604429137.5973942,"caller":"app/server.go:270","msg":"Printing current working","directory":"/opt/mattermost"}
{"level":"info","ts":1604429137.5974095,"caller":"app/server.go:271","msg":"Loaded config","source":"file:///opt/mattermost/config/config.json"}
{"level":"info","ts":1604429137.7237258,"caller":"sqlstore/post_store.go:1524","msg":"Post.Message has size restrictions","max_characters":16383,"max_bytes":65535}
{"level":"info","ts":1604429137.8062377,"caller":"jobs/workers.go:68","msg":"Starting workers"}
{"level":"info","ts":1604429137.8171308,"caller":"jobs/schedulers.go:74","msg":"Starting schedulers."}
{"level":"info","ts":1604429137.8363667,"caller":"app/web_hub.go:75","msg":"Starting websocket hubs","number_of_hubs":2}
{"level":"info","ts":1604429137.9748762,"caller":"app/server.go:509","msg":"Starting Server..."}
{"level":"error","ts":1604429137.9750233,"caller":"commands/server.go:81","msg":"listen tcp 3.250.25.199:443: bind: cannot assign requested address"}
{"level":"info","ts":1604429137.9750419,"caller":"app/server.go:402","msg":"Stopping Server..."}
{"level":"info","ts":1604429137.9750526,"caller":"app/web_hub.go:130","msg":"stopping websocket hub connections"}
{"level":"info","ts":1604429137.9757032,"caller":"app/plugin.go:279","msg":"Shutting down plugins"}
{"level":"info","ts":1604429145.8360863,"caller":"jobs/workers.go:184","msg":"Stopped workers"}
{"level":"info","ts":1604429145.8361397,"caller":"jobs/schedulers.go:145","msg":"Stopping schedulers."}
{"level":"info","ts":1604429145.83617,"caller":"jobs/schedulers.go:77","msg":"Schedulers stopped."}
{"level":"info","ts":1604429145.8363652,"caller":"app/server.go:453","msg":"Server stopped"}
{"level":"info","ts":1604429461.4267051,"caller":"utils/i18n.go:83","msg":"Loaded system translations","for locale":"en","from locale":"/opt/mattermost/i18n/en.json"}
{"level":"info","ts":1604429461.4303513,"caller":"app/server_app_adapters.go:58","msg":"Server is initializing..."}
{"level":"info","ts":1604429461.4757917,"caller":"sqlstore/supplier.go:224","msg":"Pinging SQL","database":"master"}
{"level":"error","ts":1604429461.9268403,"caller":"app/server_app_adapters.go:139","msg":"Mail server connection test is failed: Could not connect to SMTP server, check SMTP server settings."}
{"level":"info","ts":1604429461.9427073,"caller":"mlog/log.go:167","msg":"Starting up plugins"}
{"level":"info","ts":1604429461.9473717,"caller":"app/plugin.go:202","msg":"Syncing plugins from the file store"}
{"level":"info","ts":1604429461.9703689,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/zoom.tar.gz"}
{"level":"info","ts":1604429461.9706087,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/com.mattermost.aws-sns.tar.gz"}
{"level":"info","ts":1604429461.970942,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/jitsi.tar.gz"}
{"level":"info","ts":1604429470.497553,"caller":"mlog/sugar.go:19","msg":"Ensuring Surveybot exists","plugin_id":"com.mattermost.nps"}
{"level":"info","ts":1604429470.5109978,"caller":"app/server.go:255","msg":"Current version is 5.23.0 (5.23.0/Thu May 14 12:18:07 UTC 2020/ed1576d007a09b07574b18784b9eabb3d8c9db53/none)","current_version":"5.23.0","build_number":"5.23.0","build_date":"Thu May 14 12:18:07 UTC 2020","build_hash":"ed1576d007a09b07574b18784b9eabb3d8c9db53","build_hash_enterprise":"none"}
{"level":"info","ts":1604429470.5110288,"caller":"app/server.go:266","msg":"Team Edition Build","enterprise_build":false}
{"level":"info","ts":1604429470.5110493,"caller":"app/server.go:270","msg":"Printing current working","directory":"/opt/mattermost"}
{"level":"info","ts":1604429470.5110614,"caller":"app/server.go:271","msg":"Loaded config","source":"file:///opt/mattermost/config/config.json"}
{"level":"info","ts":1604429470.5557423,"caller":"sqlstore/post_store.go:1524","msg":"Post.Message has size restrictions","max_characters":16383,"max_bytes":65535}
{"level":"info","ts":1604429470.6251202,"caller":"jobs/workers.go:68","msg":"Starting workers"}
{"level":"info","ts":1604429470.637855,"caller":"jobs/schedulers.go:74","msg":"Starting schedulers."}
{"level":"info","ts":1604429470.6469295,"caller":"app/web_hub.go:75","msg":"Starting websocket hubs","number_of_hubs":2}
{"level":"info","ts":1604429470.7295532,"caller":"app/server.go:509","msg":"Starting Server..."}
{"level":"error","ts":1604429470.7297103,"caller":"commands/server.go:81","msg":"listen tcp 3.250.25.199:443: bind: cannot assign requested address"}
{"level":"info","ts":1604429470.7297523,"caller":"app/server.go:402","msg":"Stopping Server..."}
{"level":"info","ts":1604429470.729766,"caller":"app/web_hub.go:130","msg":"stopping websocket hub connections"}
{"level":"info","ts":1604429470.7304606,"caller":"app/plugin.go:279","msg":"Shutting down plugins"}
{"level":"info","ts":1604429485.4422927,"caller":"jobs/workers.go:184","msg":"Stopped workers"}
{"level":"info","ts":1604429485.4423506,"caller":"jobs/schedulers.go:145","msg":"Stopping schedulers."}
{"level":"info","ts":1604429485.4423685,"caller":"jobs/schedulers.go:77","msg":"Schedulers stopped."}
{"level":"info","ts":1604429485.4425561,"caller":"app/server.go:453","msg":"Server stopped"}
{"level":"info","ts":1604434775.3020291,"caller":"utils/i18n.go:83","msg":"Loaded system translations","for locale":"en","from locale":"/opt/mattermost/i18n/en.json"}
{"level":"info","ts":1604434775.304195,"caller":"app/server_app_adapters.go:58","msg":"Server is initializing..."}
{"level":"info","ts":1604434775.3296223,"caller":"sqlstore/supplier.go:224","msg":"Pinging SQL","database":"master"}
{"level":"error","ts":1604434775.5579157,"caller":"app/server_app_adapters.go:139","msg":"Mail server connection test is failed: Could not connect to SMTP server, check SMTP server settings."}
{"level":"info","ts":1604434775.5981095,"caller":"mlog/log.go:167","msg":"Starting up plugins"}
{"level":"info","ts":1604434775.600105,"caller":"app/plugin.go:202","msg":"Syncing plugins from the file store"}
{"level":"info","ts":1604434775.6361303,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/zoom.tar.gz"}
{"level":"info","ts":1604434775.6377783,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/com.mattermost.aws-sns.tar.gz"}
{"level":"info","ts":1604434775.6479664,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/jitsi.tar.gz"}
{"level":"info","ts":1604434784.0297575,"caller":"mlog/sugar.go:19","msg":"Ensuring Surveybot exists","plugin_id":"com.mattermost.nps"}
{"level":"info","ts":1604434784.0461872,"caller":"app/server.go:255","msg":"Current version is 5.23.0 (5.23.0/Thu May 14 12:18:07 UTC 2020/ed1576d007a09b07574b18784b9eabb3d8c9db53/none)","current_version":"5.23.0","build_number":"5.23.0","build_date":"Thu May 14 12:18:07 UTC 2020","build_hash":"ed1576d007a09b07574b18784b9eabb3d8c9db53","build_hash_enterprise":"none"}
{"level":"info","ts":1604434784.0462272,"caller":"app/server.go:266","msg":"Team Edition Build","enterprise_build":false}
{"level":"info","ts":1604434784.046248,"caller":"app/server.go:270","msg":"Printing current working","directory":"/opt/mattermost"}
{"level":"info","ts":1604434784.0462613,"caller":"app/server.go:271","msg":"Loaded config","source":"file:///opt/mattermost/config/config.json"}
{"level":"info","ts":1604434784.097042,"caller":"sqlstore/post_store.go:1524","msg":"Post.Message has size restrictions","max_characters":16383,"max_bytes":65535}
{"level":"info","ts":1604434784.1731412,"caller":"jobs/workers.go:68","msg":"Starting workers"}
{"level":"info","ts":1604434784.1835594,"caller":"jobs/schedulers.go:74","msg":"Starting schedulers."}
{"level":"info","ts":1604434784.2169747,"caller":"app/web_hub.go:75","msg":"Starting websocket hubs","number_of_hubs":2}
{"level":"info","ts":1604434784.408227,"caller":"app/server.go:509","msg":"Starting Server..."}
{"level":"error","ts":1604434784.4083896,"caller":"commands/server.go:81","msg":"listen tcp 3.250.25.199:443: bind: cannot assign requested address"}
{"level":"info","ts":1604434784.408411,"caller":"app/server.go:402","msg":"Stopping Server..."}
{"level":"info","ts":1604434784.4084234,"caller":"app/web_hub.go:130","msg":"stopping websocket hub connections"}
{"level":"info","ts":1604434784.4091427,"caller":"app/plugin.go:279","msg":"Shutting down plugins"}
{"level":"info","ts":1604434795.9067254,"caller":"jobs/workers.go:184","msg":"Stopped workers"}
{"level":"info","ts":1604434795.9068084,"caller":"jobs/schedulers.go:145","msg":"Stopping schedulers."}
{"level":"info","ts":1604434795.906837,"caller":"jobs/schedulers.go:77","msg":"Schedulers stopped."}
{"level":"info","ts":1604434795.9070408,"caller":"app/server.go:453","msg":"Server stopped"}
{"level":"info","ts":1604436847.6193056,"caller":"utils/i18n.go:83","msg":"Loaded system translations","for locale":"en","from locale":"/opt/mattermost/i18n/en.json"}
{"level":"info","ts":1604436847.6475034,"caller":"app/server_app_adapters.go:58","msg":"Server is initializing..."}
{"level":"info","ts":1604436847.6616678,"caller":"sqlstore/supplier.go:224","msg":"Pinging SQL","database":"master"}
{"level":"error","ts":1604436847.9720871,"caller":"app/server_app_adapters.go:139","msg":"Mail server connection test is failed: Could not connect to SMTP server, check SMTP server settings."}
{"level":"info","ts":1604436847.9937174,"caller":"mlog/log.go:167","msg":"Starting up plugins"}
{"level":"info","ts":1604436847.998041,"caller":"app/plugin.go:202","msg":"Syncing plugins from the file store"}
{"level":"info","ts":1604436848.0236943,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/com.mattermost.aws-sns.tar.gz"}
{"level":"info","ts":1604436848.02405,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/jitsi.tar.gz"}
{"level":"info","ts":1604436848.0243948,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/zoom.tar.gz"}
{"level":"info","ts":1604436856.287942,"caller":"mlog/sugar.go:19","msg":"Ensuring Surveybot exists","plugin_id":"com.mattermost.nps"}
{"level":"info","ts":1604436856.3033535,"caller":"app/server.go:255","msg":"Current version is 5.23.0 (5.23.0/Thu May 14 12:18:07 UTC 2020/ed1576d007a09b07574b18784b9eabb3d8c9db53/none)","current_version":"5.23.0","build_number":"5.23.0","build_date":"Thu May 14 12:18:07 UTC 2020","build_hash":"ed1576d007a09b07574b18784b9eabb3d8c9db53","build_hash_enterprise":"none"}
{"level":"info","ts":1604436856.3033912,"caller":"app/server.go:266","msg":"Team Edition Build","enterprise_build":false}
{"level":"info","ts":1604436856.3034122,"caller":"app/server.go:270","msg":"Printing current working","directory":"/opt/mattermost"}
{"level":"info","ts":1604436856.3034256,"caller":"app/server.go:271","msg":"Loaded config","source":"file:///opt/mattermost/config/config.json"}
{"level":"info","ts":1604436856.3707144,"caller":"sqlstore/post_store.go:1524","msg":"Post.Message has size restrictions","max_characters":16383,"max_bytes":65535}
{"level":"info","ts":1604436856.469041,"caller":"jobs/workers.go:68","msg":"Starting workers"}
{"level":"info","ts":1604436856.4810128,"caller":"jobs/schedulers.go:74","msg":"Starting schedulers."}
{"level":"info","ts":1604436856.554253,"caller":"app/web_hub.go:75","msg":"Starting websocket hubs","number_of_hubs":2}
{"level":"info","ts":1604436856.6492832,"caller":"app/server.go:509","msg":"Starting Server..."}
{"level":"error","ts":1604436856.6494484,"caller":"commands/server.go:81","msg":"listen tcp 3.250.25.199:443: bind: cannot assign requested address"}
{"level":"info","ts":1604436856.6494796,"caller":"app/server.go:402","msg":"Stopping Server..."}
{"level":"info","ts":1604436856.6494937,"caller":"app/web_hub.go:130","msg":"stopping websocket hub connections"}
{"level":"info","ts":1604436856.650056,"caller":"app/plugin.go:279","msg":"Shutting down plugins"}
{"level":"info","ts":1604436863.2606115,"caller":"jobs/workers.go:184","msg":"Stopped workers"}
{"level":"info","ts":1604436863.2606666,"caller":"jobs/schedulers.go:145","msg":"Stopping schedulers."}
{"level":"info","ts":1604436863.2606847,"caller":"jobs/schedulers.go:77","msg":"Schedulers stopped."}
{"level":"info","ts":1604436863.2608943,"caller":"app/server.go:453","msg":"Server stopped"}
{"level":"info","ts":1604446873.7518744,"caller":"utils/i18n.go:83","msg":"Loaded system translations","for locale":"en","from locale":"/opt/mattermost/i18n/en.json"}
{"level":"info","ts":1604446873.75628,"caller":"app/server_app_adapters.go:58","msg":"Server is initializing..."}
{"level":"info","ts":1604446873.7711787,"caller":"sqlstore/supplier.go:224","msg":"Pinging SQL","database":"master"}
{"level":"error","ts":1604446873.9633024,"caller":"app/server_app_adapters.go:139","msg":"Mail server connection test is failed: Could not connect to SMTP server, check SMTP server settings."}
{"level":"info","ts":1604446873.9813762,"caller":"mlog/log.go:167","msg":"Starting up plugins"}
{"level":"info","ts":1604446873.98555,"caller":"app/plugin.go:202","msg":"Syncing plugins from the file store"}
{"level":"info","ts":1604446874.0160615,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/zoom.tar.gz"}
{"level":"info","ts":1604446874.0163643,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/com.mattermost.aws-sns.tar.gz"}
{"level":"info","ts":1604446874.0167124,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/jitsi.tar.gz"}
{"level":"info","ts":1604446882.660603,"caller":"mlog/sugar.go:19","msg":"Ensuring Surveybot exists","plugin_id":"com.mattermost.nps"}
{"level":"info","ts":1604446882.678561,"caller":"app/server.go:255","msg":"Current version is 5.23.0 (5.23.0/Thu May 14 12:18:07 UTC 2020/ed1576d007a09b07574b18784b9eabb3d8c9db53/none)","current_version":"5.23.0","build_number":"5.23.0","build_date":"Thu May 14 12:18:07 UTC 2020","build_hash":"ed1576d007a09b07574b18784b9eabb3d8c9db53","build_hash_enterprise":"none"}
{"level":"info","ts":1604446882.6785936,"caller":"app/server.go:266","msg":"Team Edition Build","enterprise_build":false}
{"level":"info","ts":1604446882.678614,"caller":"app/server.go:270","msg":"Printing current working","directory":"/opt/mattermost"}
{"level":"info","ts":1604446882.678626,"caller":"app/server.go:271","msg":"Loaded config","source":"file:///opt/mattermost/config/config.json"}
{"level":"info","ts":1604446882.724058,"caller":"sqlstore/post_store.go:1524","msg":"Post.Message has size restrictions","max_characters":16383,"max_bytes":65535}
{"level":"info","ts":1604446882.792694,"caller":"jobs/workers.go:68","msg":"Starting workers"}
{"level":"info","ts":1604446882.8075445,"caller":"jobs/schedulers.go:74","msg":"Starting schedulers."}
{"level":"info","ts":1604446882.8212526,"caller":"app/web_hub.go:75","msg":"Starting websocket hubs","number_of_hubs":2}
{"level":"info","ts":1604446882.9150665,"caller":"app/server.go:509","msg":"Starting Server..."}
{"level":"error","ts":1604446882.9152265,"caller":"commands/server.go:81","msg":"listen tcp 3.250.25.199:443: bind: cannot assign requested address"}
{"level":"info","ts":1604446882.9152458,"caller":"app/server.go:402","msg":"Stopping Server..."}
{"level":"info","ts":1604446882.915258,"caller":"app/web_hub.go:130","msg":"stopping websocket hub connections"}
{"level":"info","ts":1604446882.916,"caller":"app/plugin.go:279","msg":"Shutting down plugins"}
{"level":"info","ts":1604446896.6636872,"caller":"jobs/workers.go:184","msg":"Stopped workers"}
{"level":"info","ts":1604446896.663745,"caller":"jobs/schedulers.go:145","msg":"Stopping schedulers."}
{"level":"info","ts":1604446896.663763,"caller":"jobs/schedulers.go:77","msg":"Schedulers stopped."}
{"level":"info","ts":1604446896.6639605,"caller":"app/server.go:453","msg":"Server stopped"}
{"level":"info","ts":1604502996.50675,"caller":"utils/i18n.go:83","msg":"Loaded system translations","for locale":"en","from locale":"/opt/mattermost/i18n/en.json"}
{"level":"info","ts":1604502996.5068674,"caller":"app/server_app_adapters.go:58","msg":"Server is initializing..."}
{"level":"info","ts":1604502996.5334883,"caller":"sqlstore/supplier.go:224","msg":"Pinging SQL","database":"master"}
{"level":"error","ts":1604502996.591641,"caller":"app/server_app_adapters.go:139","msg":"Mail server connection test is failed: Could not connect to SMTP server, check SMTP server settings."}
{"level":"info","ts":1604502996.5972037,"caller":"mlog/log.go:167","msg":"Starting up plugins"}
{"level":"info","ts":1604502996.5972428,"caller":"app/plugin.go:202","msg":"Syncing plugins from the file store"}
{"level":"info","ts":1604502996.6256766,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/zoom.tar.gz"}
{"level":"info","ts":1604502996.635172,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/com.mattermost.aws-sns.tar.gz"}
{"level":"info","ts":1604502996.6678982,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/jitsi.tar.gz"}
{"level":"info","ts":1604503004.9164195,"caller":"mlog/sugar.go:19","msg":"Ensuring Surveybot exists","plugin_id":"com.mattermost.nps"}
{"level":"info","ts":1604503004.9305973,"caller":"app/server.go:255","msg":"Current version is 5.23.0 (5.23.0/Thu May 14 12:18:07 UTC 2020/ed1576d007a09b07574b18784b9eabb3d8c9db53/none)","current_version":"5.23.0","build_number":"5.23.0","build_date":"Thu May 14 12:18:07 UTC 2020","build_hash":"ed1576d007a09b07574b18784b9eabb3d8c9db53","build_hash_enterprise":"none"}
{"level":"info","ts":1604503004.930632,"caller":"app/server.go:266","msg":"Team Edition Build","enterprise_build":false}
{"level":"info","ts":1604503004.9306526,"caller":"app/server.go:270","msg":"Printing current working","directory":"/opt/mattermost"}
{"level":"info","ts":1604503004.9306653,"caller":"app/server.go:271","msg":"Loaded config","source":"file:///opt/mattermost/config/config.json"}
{"level":"info","ts":1604503004.9614573,"caller":"sqlstore/post_store.go:1524","msg":"Post.Message has size restrictions","max_characters":16383,"max_bytes":65535}
{"level":"info","ts":1604503005.027653,"caller":"jobs/workers.go:68","msg":"Starting workers"}
{"level":"info","ts":1604503005.0361476,"caller":"app/web_hub.go:75","msg":"Starting websocket hubs","number_of_hubs":2}
{"level":"info","ts":1604503005.0446563,"caller":"jobs/schedulers.go:74","msg":"Starting schedulers."}
{"level":"info","ts":1604503005.073258,"caller":"app/server.go:509","msg":"Starting Server..."}
{"level":"error","ts":1604503005.0733886,"caller":"commands/server.go:81","msg":"listen tcp 3.250.25.199:443: bind: cannot assign requested address"}
{"level":"info","ts":1604503005.073407,"caller":"app/server.go:402","msg":"Stopping Server..."}
{"level":"info","ts":1604503005.0734181,"caller":"app/web_hub.go:130","msg":"stopping websocket hub connections"}
{"level":"info","ts":1604503005.0740275,"caller":"app/plugin.go:279","msg":"Shutting down plugins"}
{"level":"info","ts":1604503016.48188,"caller":"jobs/workers.go:184","msg":"Stopped workers"}
{"level":"info","ts":1604503016.4819355,"caller":"jobs/schedulers.go:145","msg":"Stopping schedulers."}
{"level":"info","ts":1604503016.4819534,"caller":"jobs/schedulers.go:77","msg":"Schedulers stopped."}
{"level":"info","ts":1604503016.4821455,"caller":"app/server.go:453","msg":"Server stopped"}

Jeremy Davis's picture

Looking at the Mattermost log, this bit appears to be the problem:

[...] listen tcp 3.250.25.199:443: bind: cannot assign requested address [...]

Basically what appears to be happening is that Mattermost is trying to bind to port 443 (the default HTTPS port), but it can't because Nginx is already bound to that port (and only one application can bind to a port).

After a bit of googling it appears that can happen when Mattermost is upgraded (although the Mattermost devs don't appear to understand how that happens). Regardless, I'm guessing that you did that.

The fix is to adjust the config file (by default that should be /opt/mattermost/config/config.json) to disable it from binding to port 443 and instead revert to the default config where Nginx acts as a reverse proxy for Mattermost.

The lines that need to be adjusted in the config file and their desired/required values are:

"ListenAddress": "127.0.0.1:8065"
"UseLetsEncrypt": false
"Forward80To443": false

If you want to manually make the adjsutments as per above, that's fine, but here is some code that you can copy/paste:

sudo su
CONF=/opt/mattermost/config/config.json
sed -i 's|ListenAddress":.*|ListenAddress": "127.0.0.1:8065",|' $CONF
sed -i 's|UseLetsEncrypt":.*|UseLetsEncrypt": false,|' $CONF
sed -i 's|Forward80To443":.*|Forward80To443": false,|' $CONF
exit

Once that's been done, then hopefully restarting Mattermost should now work:

sudo systemctl start mattermost

Please let me know if that gets you up and running again. If not, please post your full config file:

sudo cat /opt/mattermost/config/config.json
Ivan D Marin's picture

Hi Jeremy 

Thanks again for your reply. I went ahead and used all your commands, there were a couple I noticed had a | missing and was able to get the configuration in. However, this unfortunatelly didn't fix the issue and I can see that the settings were in the config.json file. 

Below is the full info on the config file.

{
    "ServiceSettings": {
        "SiteURL": "https://mattermost.sirius-support.com",
        "WebsocketURL": "",
        "LicenseFileLocation": "",
        "ListenAddress": "127.0.0.1:8065",
        "ConnectionSecurity": "",
        "TLSCertFile": "",
        "TLSKeyFile": "",
        "TLSMinVer": "1.2",
        "TLSStrictTransport": false,
        "TLSStrictTransportMaxAge": 63072000,
        "TLSOverwriteCiphers": [],
        "UseLetsEncrypt": false,
        "LetsEncryptCertificateCacheFile": "./config/letsencrypt.cache",
        "Forward80To443": false,
[...]

Jeremy Davis's picture

[...] there were a couple I noticed had a | missing [...]

Oops, sorry about the typos and glad to hear that you could work out what was meant to be going on. I've edited my post to fix those typos.

Also sorry for the overactive spam filter blocking your posts. I've now ttweaked your user account so hopefully that won't happen again...

Thanks too for posting the config file. Although, in retrospect, I probably should have edited the config file in your post before I published it. It's only just occurred to me that it includes some "secrets" (e.g. API keys and "salts"). Having these published on the internet is not good... I have now edited your post to remove those, but I did publish it a few hours ago.

Unfortunately I'm not super familiar with Mattermost and can't be clear on which of those settings have been set by you and which might be default (some are generated at install time and regenerated by scripts we provide on firstboot). So assuming that you are using this in production, it's probably a good idea to make sure that these "secrets" are regenerated. I would say that particularly any 3rd API keys that are included (e.g. Zoom API key) be updated ASAP!

Also, I must admit that beyond checking the lines that my sed commands were meant to edit and a quick "by eye" check that it's valid json, I'm not familiar enough with Mattermost config to really know what to be looking for...

So to see why it's not working, we'll need to again check the logs. Judging from the output you've posted previously, my guess is that the Mattermost log itself will be most useful.

To ensure that we catch the tail end of the (attempted) start process, let's try starting it again and then just post the last 40 lines of that with these commands:

sudo systemctl start mattermost
sudo tail -40 /opt/mattermost/config/config.json

Having said all that though, I took the liberty of checking the URL that was in your config file and it appears to be working?! Was there something else that you discovered yourself? I'm curious what made it work...

Ivan D Marin's picture

Hi Jeremy, 

Actually I configured a different Mattermost server and left this one to fix and to learn. Right now the ip with the problem would be 3.250.25.199. Which is still giving the error. I apologize for the misunderstanding there.

Thanks for the heads up in the secrets and such but right now you could say that this is only a Dev instance and Zoom ID and Secret were removed and no longer accessible.

I went ahead and checked the logs and the config file if it has any secret or confidential info.

Below are the config files and the mattermost logs. I am still trying to figure out a way to fix it with no luck. However, on the new mattermost server I set up, once I rebooted I got an ngix error too, however this one went away after some time. I was reading that it could be linked to let'sencrypt but I don't have the knowledge to check that or investigate that.

 

 sudo tail -40 /opt/mattermost/logs/mattermost.log
{"level":"info","ts":1604446882.724058,"caller":"sqlstore/post_store.go:1524","msg":"Post.Message has size restrictions","max_characters":16383,"max_bytes":65535}
{"level":"info","ts":1604446882.792694,"caller":"jobs/workers.go:68","msg":"Starting workers"}
{"level":"info","ts":1604446882.8075445,"caller":"jobs/schedulers.go:74","msg":"Starting schedulers."}
{"level":"info","ts":1604446882.8212526,"caller":"app/web_hub.go:75","msg":"Starting websocket hubs","number_of_hubs":2}
{"level":"info","ts":1604446882.9150665,"caller":"app/server.go:509","msg":"Starting Server..."}
{"level":"error","ts":1604446882.9152265,"caller":"commands/server.go:81","msg":"listen tcp 3.250.25.199:443: bind: cannot assign requested address"}
{"level":"info","ts":1604446882.9152458,"caller":"app/server.go:402","msg":"Stopping Server..."}
{"level":"info","ts":1604446882.915258,"caller":"app/web_hub.go:130","msg":"stopping websocket hub connections"}
{"level":"info","ts":1604446882.916,"caller":"app/plugin.go:279","msg":"Shutting down plugins"}
{"level":"info","ts":1604446896.6636872,"caller":"jobs/workers.go:184","msg":"Stopped workers"}
{"level":"info","ts":1604446896.663745,"caller":"jobs/schedulers.go:145","msg":"Stopping schedulers."}
{"level":"info","ts":1604446896.663763,"caller":"jobs/schedulers.go:77","msg":"Schedulers stopped."}
{"level":"info","ts":1604446896.6639605,"caller":"app/server.go:453","msg":"Server stopped"}
{"level":"info","ts":1604502996.50675,"caller":"utils/i18n.go:83","msg":"Loaded system translations","for locale":"en","from locale":"/opt/mattermost/i18n/en.json"}
{"level":"info","ts":1604502996.5068674,"caller":"app/server_app_adapters.go:58","msg":"Server is initializing..."}
{"level":"info","ts":1604502996.5334883,"caller":"sqlstore/supplier.go:224","msg":"Pinging SQL","database":"master"}
{"level":"error","ts":1604502996.591641,"caller":"app/server_app_adapters.go:139","msg":"Mail server connection test is failed: Could not connect to SMTP server, check SMTP server settings."}
{"level":"info","ts":1604502996.5972037,"caller":"mlog/log.go:167","msg":"Starting up plugins"}
{"level":"info","ts":1604502996.5972428,"caller":"app/plugin.go:202","msg":"Syncing plugins from the file store"}
{"level":"info","ts":1604502996.6256766,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/zoom.tar.gz"}
{"level":"info","ts":1604502996.635172,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/com.mattermost.aws-sns.tar.gz"}
{"level":"info","ts":1604502996.6678982,"caller":"app/plugin.go:262","msg":"Syncing plugin from file store","bundle":"plugins/jitsi.tar.gz"}
{"level":"info","ts":1604503004.9164195,"caller":"mlog/sugar.go:19","msg":"Ensuring Surveybot exists","plugin_id":"com.mattermost.nps"}
{"level":"info","ts":1604503004.9305973,"caller":"app/server.go:255","msg":"Current version is 5.23.0 (5.23.0/Thu May 14 12:18:07 UTC 2020/ed1576d007a09b07574b18784b9eabb3d8c9db53/none)","current_version":"5.23.0","build_number":"5.23.0","build_date":"Thu May 14 12:18:07 UTC 2020","build_hash":"ed1576d007a09b07574b18784b9eabb3d8c9db53","build_hash_enterprise":"none"}
{"level":"info","ts":1604503004.930632,"caller":"app/server.go:266","msg":"Team Edition Build","enterprise_build":false}
{"level":"info","ts":1604503004.9306526,"caller":"app/server.go:270","msg":"Printing current working","directory":"/opt/mattermost"}
{"level":"info","ts":1604503004.9306653,"caller":"app/server.go:271","msg":"Loaded config","source":"file:///opt/mattermost/config/config.json"}
{"level":"info","ts":1604503004.9614573,"caller":"sqlstore/post_store.go:1524","msg":"Post.Message has size restrictions","max_characters":16383,"max_bytes":65535}
{"level":"info","ts":1604503005.027653,"caller":"jobs/workers.go:68","msg":"Starting workers"}
{"level":"info","ts":1604503005.0361476,"caller":"app/web_hub.go:75","msg":"Starting websocket hubs","number_of_hubs":2}
{"level":"info","ts":1604503005.0446563,"caller":"jobs/schedulers.go:74","msg":"Starting schedulers."}
{"level":"info","ts":1604503005.073258,"caller":"app/server.go:509","msg":"Starting Server..."}
{"level":"error","ts":1604503005.0733886,"caller":"commands/server.go:81","msg":"listen tcp 3.250.25.199:443: bind: cannot assign requested address"}
{"level":"info","ts":1604503005.073407,"caller":"app/server.go:402","msg":"Stopping Server..."}
{"level":"info","ts":1604503005.0734181,"caller":"app/web_hub.go:130","msg":"stopping websocket hub connections"}
{"level":"info","ts":1604503005.0740275,"caller":"app/plugin.go:279","msg":"Shutting down plugins"}
{"level":"info","ts":1604503016.48188,"caller":"jobs/workers.go:184","msg":"Stopped workers"}
{"level":"info","ts":1604503016.4819355,"caller":"jobs/schedulers.go:145","msg":"Stopping schedulers."}
{"level":"info","ts":1604503016.4819534,"caller":"jobs/schedulers.go:77","msg":"Schedulers stopped."}
{"level":"info","ts":1604503016.4821455,"caller":"app/server.go:453","msg":"Server stopped"}


sudo tail -40 /opt/mattermost/config/config.json
            "com.mattermost.aws-sns": {
                "Enable": false
            },
            "com.mattermost.custom-attributes": {
                "Enable": false
            },
            "com.mattermost.nps": {
                "Enable": true
            },
            "jitsi": {
                "Enable": true
            },
            "zoom": {
                "Enable": false
            }
        },
        "EnableMarketplace": true,
        "EnableRemoteMarketplace": true,
        "AutomaticPrepackagedPlugins": true,
        "RequirePluginSignature": false,
        "MarketplaceUrl": "https://api.integrations.mattermost.com",
        "SignaturePublicKeyFiles": []
    },
    "DisplaySettings": {
        "CustomUrlSchemes": [],
        "ExperimentalTimezone": false
    },
    "GuestAccountsSettings": {
        "Enable": false,
        "AllowEmailAccounts": true,
        "EnforceMultifactorAuthentication": false,
        "RestrictCreationToDomains": ""
    },
    "ImageProxySettings": {
        "Enable": false,
        "ImageProxyType": "local",
        "RemoteImageProxyURL": "",
        "RemoteImageProxyOptions": ""
    }
 

Jeremy Davis's picture

Look at these lines in the log:

{"level":"info","ts":1604503005.073258,"caller":"app/server.go:509","msg":"Starting Server..."}
{"level":"error","ts":1604503005.0733886,"caller":"commands/server.go:81","msg":"listen tcp 3.250.25.199:443: bind: cannot assign requested address"}
{"level":"info","ts":1604503005.073407,"caller":"app/server.go:402","msg":"Stopping Server..."}

If I pull out the important bits, it might make more sense to you:

Starting Server...
listen tcp 3.250.25.199:443: bind: cannot assign requested address
Stopping Server...

The middle line means that it's trying to bind to the external IP address (I assume that 3.250.25.199 is the server's public IP address) on port 443. That won't work because Nginx is already listening there. The config change that we made should have fixed that, but for some reason it hasn't. TBH, I'm at a bit of a loss as to why the config change that you applied hasn't worked?! Your previous post that included the cofnig contents appeared to be correct to me?! (FWIW, this most recent config output isn't that useful as the command you used "tail -40" will just give the last 40 lines; very useful with log files, not so useful with config files, unless the lines you need are within the last 40...).

The most important line is the "ListenAddress". As per the config that you previously displayed, it should be "127.0.0.1:8065". If you want to double check, you can use a text search tool called 'grep' (you shouldn't need sudo, but just in case, may as well use it):

grep "ListenAddress" /opt/mattermost/config/config.json

It should return this:

        "ListenAddress": "127.0.0.1:8065",

It may perhaps be worth double checking the the config file is valid (I suspect that it would complain in the logs if that was the issue, but just incase...). You should be able to do that like this:

cd /opt/mattermost
bin/mattermost config validate

Also browsing through the Mattermost docs, I had a look for any config settings that may be relevant (see here). Not counting the "ListenAddress" as per above, the ones that jumped out at me as perhaps relevant (and what I think they probably should be) are:

"Forward80To443": false,
"ConnectionSecurity": "",
"TLSCertFile": "",
"TLSKeyFile": "",
"UseLetsEncrypt": false,

If you have those settings configured and it still doesn't start, then I'm fast running out of ideas...

Perhaps if you can explain what you did to get to where you are, maybe I can try recreate your issue?

Ivan D Marin's picture

Thanks again for your reply, I appreciate all your assistance.

It is as you mention, using the grep the listening address was this one.

"ListenAddress": "127.0.0.1:8065".

However, I did get that twice, not sure if that is a problem. 

Continuing with this I tried to use one of the commands and I got this, maybe this has to do with the issue?

$ bin/mattermost config validate
Error: failed to initialize config store: failed to load: failed to unmarshal config with env overrides: parsing error at line 409, character 6: invalid character '}' looking for beginning of object key string
Usage:
  mattermost config validate [flags]

Flags:
  -h, --help   help for validate

Global Flags:
  -c, --config string        Configuration file to use. (default "config.json")
      --disableconfigwatch   When set config.json will not be loaded from disk when the file is changed.

From what I know so far, when a program doesn't have an { at the beginning its a code problem which is the error we just received. Does this give any idea to you or you need something else?

In regards to how I got to this point. I installed TurnKey Mattermost and was trying to validate the SMTP by moving the settings here and there (I later learned that the issue was on aws smtp) however, I tried to google the commands on how to configure it either with commands or inside mattermost itself. 

All failed and then I read that I needed to change a setting inside the confconsole which was the mail relaying feature. Once I configured that one I rebooted the EC2 Instance and that is when this happened. At first, someone said it was the service on one of the forums and then I realized it was something else which brought me here. 

 

Jeremy Davis's picture

The config file is in a format called "json" (pronounced jason) and valid json requires that items (usually key-value pairs) be separated by commas. And it looks like your ListenAddress key-value pair has a full stop/period/dot ('.') at the end, rather than a comma (','). I.e. this:

"ListenAddress": "127.0.0.1:8065".

Should instead be this:

"ListenAddress": "127.0.0.1:8065",

Unfortunately, it's not obvious whether that is the issue that is noted in the error message that you noted when you ran the config validation tool. I suspect that it may be, but either way, it's definitely an issue that you'll need to resolve. FYI sometimes a formatting error in one place, won't actually cause an error until later. This is is a frustrating, but not uncommon issue with syntax parsers.

So with specific regards to the error message you got:

Error: failed to initialize config store: failed to load: failed to unmarshal config with env overrides: parsing error at line 409, character 6: invalid character '}' looking for beginning of object key string

My guess is that it's a missing comma prior to line 409. Hence why I suspect that the line you noted may be the issue. Please try fixing that and re-run the validation tool.

Thanks too for the additional info. I was just trying to get a feel for the likelihood of it being a bug, or something that you have accidentally broken. My guess is the latter... :)

Add new comment