deutrino's picture
deutrino - Sat, 2021/05/15 - 06:07

Something killed postgres. In /var/log/postgresql

2021-05-14 09:44:49.532 UTC [585] LOG:  received fast shutdown request
2021-05-14 09:44:49.533 UTC [585] LOG:  aborting any active transactions
[various specific connections terminated]
2021-05-14 09:44:49.571 UTC [585] LOG:  background worker "logical replication launcher" (PID 614) exited with exit code 1
2021-05-14 09:44:49.574 UTC [609] LOG:  shutting down
2021-05-14 09:44:49.605 UTC [585] LOG:  database system is shut down

This is on two separate hosts, two separate providers in different countries. The events did not happen at the exact same time of day.

 

Forum: 
Support
Tags: 
mattermost
postgres
postgresql
deutrino's picture

deutrino - Sat, 2021/05/15 - 06:17

I'm pretty sure a recent unattended update to postgresql somehow eventually caused postgresql to exit and not be restarted. I rebooted both servers and they came back up and appear to be functioning normally now.

Jeremy Davis's picture

Jeremy Davis - Sun, 2021/05/16 - 01:51

It looks like you are right. There was a recent security update to PostgreSQL (11.12-0+deb10u1) See Debian Security Announcement (DSA) 4915-1.

Although it should have restarted PostgreSQL cleanly. It appears that it was stopped (relatively) cleanly (I don't think that the "logical replication launcher" message is a problem). It just wasn't restarted?!

When I'm back at my desk, I'll launch one without installing the sec updates on firstboot and see how it goes.

Thanks for posting.

deutrino's picture

deutrino - Fri, 2022/05/13 - 17:39

Something like this just happened again, on the same two mattermost servers. I'm still investigating what the cause may have been.

Jeremy Davis's picture

Jeremy Davis - Mon, 2022/05/16 - 01:31

It looks like there was another sec update (DSA-5135-1 - to patch CVE-2022-1552).

Still it's not ideal that it crashes your servers! I wonder why it crashes Postgres and doesn't just gracefully restart it? I wonder if there is something we could do to make it restart gracefully on sec update?

deutrino's picture

deutrino - Fri, 2022/05/13 - 17:45

log snippet:

	2022-05-13 09:03:09.441 UTC [541] LOG:  received fast shutdown request
	2022-05-13 09:03:09.442 UTC [541] LOG:  aborting any active transactions
	[removed various web apps generating errors as a result of this]
	2022-05-13 09:03:09.490 UTC [541] LOG:  background worker "logical replication launcher" (PID 599) exited with exit code 1
	2022-05-13 09:03:09.499 UTC [594] LOG:  shutting down
	2022-05-13 09:03:09.533 UTC [541] LOG:  database system is shut down
Jeremy Davis's picture

Jeremy Davis - Mon, 2022/05/16 - 01:34

It looks like it literally just got shut down and not restarted. I'm assuming that just starting it works ok? If so, then I wonder if secupdates checking for PostgreSQL updates and just restarting PostgreSQL (if it's not running) would resolve it?

deutrino's picture

deutrino - Mon, 2022/05/16 - 05:13

You know, I probably could have tried to just start it up again, but in both cases now I ended up rebooting the VM "to be sure."

I suppose if it happens again and I end up back at this thread, hopefully I'll see to try a 'systemctl start' and report back! lol

Add new comment