deutrino's picture

Something killed postgres. In /var/log/postgresql

2021-05-14 09:44:49.532 UTC [585] LOG:  received fast shutdown request
2021-05-14 09:44:49.533 UTC [585] LOG:  aborting any active transactions
[various specific connections terminated]
2021-05-14 09:44:49.571 UTC [585] LOG:  background worker "logical replication launcher" (PID 614) exited with exit code 1
2021-05-14 09:44:49.574 UTC [609] LOG:  shutting down
2021-05-14 09:44:49.605 UTC [585] LOG:  database system is shut down

This is on two separate hosts, two separate providers in different countries. The events did not happen at the exact same time of day.


deutrino's picture

I'm pretty sure a recent unattended update to postgresql somehow eventually caused postgresql to exit and not be restarted. I rebooted both servers and they came back up and appear to be functioning normally now.

Jeremy Davis's picture

It looks like you are right. There was a recent security update to PostgreSQL (11.12-0+deb10u1) See Debian Security Announcement (DSA) 4915-1.

Although it should have restarted PostgreSQL cleanly. It appears that it was stopped (relatively) cleanly (I don't think that the "logical replication launcher" message is a problem). It just wasn't restarted?!

When I'm back at my desk, I'll launch one without installing the sec updates on firstboot and see how it goes.

Thanks for posting.

deutrino's picture

Something like this just happened again, on the same two mattermost servers. I'm still investigating what the cause may have been.

Jeremy Davis's picture

It looks like there was another sec update (DSA-5135-1 - to patch CVE-2022-1552).

Still it's not ideal that it crashes your servers! I wonder why it crashes Postgres and doesn't just gracefully restart it? I wonder if there is something we could do to make it restart gracefully on sec update?

deutrino's picture

log snippet:

	2022-05-13 09:03:09.441 UTC [541] LOG:  received fast shutdown request
	2022-05-13 09:03:09.442 UTC [541] LOG:  aborting any active transactions
	[removed various web apps generating errors as a result of this]
	2022-05-13 09:03:09.490 UTC [541] LOG:  background worker "logical replication launcher" (PID 599) exited with exit code 1
	2022-05-13 09:03:09.499 UTC [594] LOG:  shutting down
	2022-05-13 09:03:09.533 UTC [541] LOG:  database system is shut down
Jeremy Davis's picture

It looks like it literally just got shut down and not restarted. I'm assuming that just starting it works ok? If so, then I wonder if secupdates checking for PostgreSQL updates and just restarting PostgreSQL (if it's not running) would resolve it?

deutrino's picture

You know, I probably could have tried to just start it up again, but in both cases now I ended up rebooting the VM "to be sure."

I suppose if it happens again and I end up back at this thread, hopefully I'll see to try a 'systemctl start' and report back! lol

Add new comment