Forum archive
2 TKL mattermost boxes, 2 different hosts, dead on same day
Something killed postgres. In /var/log/postgresql
2021-05-14 09:44:49.532 UTC [585] LOG: received fast shutdown request 2021-05-14 09:44:49.533 UTC [585] LOG: aborting any active transactions [various specific connections terminated] 2021-05-14 09:44:49.571 UTC [585] LOG: background worker "logical replication launcher" (PID 614) exited with exit code 1 2021-05-14 09:44:49.574 UTC [609] LOG: shutting down 2021-05-14 09:44:49.605 UTC [585] LOG: database system is shut down
This is on two separate hosts, two separate providers in different countries. The events did not happen at the exact same time of day.
Replies (7)
Hmm, yeah looks like there was a recent PostgreSQL sec update!
Reply 1I'm pretty sure a recent
Reply 2I'm pretty sure a recent unattended update to postgresql somehow eventually caused postgresql to exit and not be restarted. I rebooted both servers and they came back up and appear to be functioning normally now.
Looks like there was another sec update
Reply 3It looks like there was another sec update (DSA-5135-1 - to patch CVE-2022-1552).
Still it's not ideal that it crashes your servers! I wonder why it crashes Postgres and doesn't just gracefully restart it? I wonder if there is something we could do to make it restart gracefully on sec update?
Something like this just
Reply 4Something like this just happened again, on the same two mattermost servers. I'm still investigating what the cause may have been.
It looks like it literally just got shut down and not restarted?
Reply 5It looks like it literally just got shut down and not restarted. I'm assuming that just starting it works ok? If so, then I wonder if secupdates checking for PostgreSQL updates and just restarting PostgreSQL (if it's not running) would resolve it?
log snippet:
Reply 6log snippet:
2022-05-13 09:03:09.441 UTC [541] LOG: received fast shutdown request
2022-05-13 09:03:09.442 UTC [541] LOG: aborting any active transactions
[removed various web apps generating errors as a result of this]
2022-05-13 09:03:09.490 UTC [541] LOG: background worker "logical replication launcher" (PID 599) exited with exit code 1
2022-05-13 09:03:09.499 UTC [594] LOG: shutting down
2022-05-13 09:03:09.533 UTC [541] LOG: database system is shut down
You know, I probably could
Reply 7You know, I probably could have tried to just start it up again, but in both cases now I ended up rebooting the VM "to be sure."
I suppose if it happens again and I end up back at this thread, hopefully I'll see to try a 'systemctl start' and report back! lol
It looks like you are right. There was a recent security update to PostgreSQL (11.12-0+deb10u1) See Debian Security Announcement (DSA) 4915-1.
Although it should have restarted PostgreSQL cleanly. It appears that it was stopped (relatively) cleanly (I don't think that the "logical replication launcher" message is a problem). It just wasn't restarted?!
When I'm back at my desk, I'll launch one without installing the sec updates on firstboot and see how it goes.
Thanks for posting.