host:
pve-manager/7.3-4/d69b70d4 (running kernel: 5.15.83-1-pve)

turnkey template:
nextcloud 17.1.1

expected behaviour;
no issues

actual behaviour;

when running for the first time as soon as I try and enter a password as part of the first setup procedure. the screen freezes.

Only when I hit backspace the following screenshot aapears.

 

 

 

Forum: 

it does not matter if I use nesting or not, or wether or not I use custom mount points or not.

 

It does matter if I use nesting or not, even though I said it did not matter earlier.

when I enable nesting for the container then the redis server in the container is able to start.
Without nesting the syslog contains something like this

systemd[135]: systemd-resolved.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc: Permission denied
Mar 10 16:06:40 debian systemd[135]: systemd-resolved.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-resolved: Permission denied

But then not systemd-resolved.service:but redis.server

 

I think the whoe lxc implementation on debian is rather broken

Jeremy Davis's picture

I guess as an end user you could consider the current behavior "broken", but I would argue that it's not as black and white as that. IMO it's not that it's "broken", it's just that security is a tradeoff. The tighter you wind the security screws, the less "user friendly" the system tends to be and the more things don't "just work". That's pretty much always the case with security. If you want to take it to the extreme, to make your system "ultimately" secure, unplug it! :) Obviously that's impractical if you intend to actually use it, but hopefully you get my point that security is always a trade off.

To give a little more context, over the last few years Debian have been slowly tightening the security screws and have hardened services like Redis. These measures lock the services down more and reduce possible attack vectors (aiming to pre-emptively mitigate and minimize as yet unknown security issues that may come to light in the future). The application of those security hardening measures requires certain system capabilities from the OS. Conversely, in order to isolate LXC guests and secure the LXC host system as much as possible, lots of system capabilities are limited in an unprivileged LXC guest. Thus the system capabilities required for the security hardened services aren't available - thus the services won't run as intended.

So to get security hardened services like Redis running in an unprivileged container, you are left with either needing to (manually) wind back the service security hardening measures, or provide the required system capabilities to the container (e.g. via enabling "nesting"). So essentially the trade off is guest security vs host security.

FWIW service security hardening measures are almost always the cause of systemd "namespace" issues as you detail here. It's essentially systemd failing to start the service as it doesn't have the required system capabilities to lock down the service as per it's configuration.

Generally these issues only apply to LXC guests (and can usually be worked around as noted above). FWIW if you run the server as a "proper" VM (rather than a LXC container) you shouldn't hit them.

I tried uploading a screenshot of the issue, but I am thinking now I never actually did.

It will be difficult for me to wade through the workflow of this particular software to get my screenshot accross

Jeremy Davis's picture

Thanks for your report and apologies for the issues posting the screenshot. I have posted your screenshot for you. I don't know why, but it won't publicly display the files you attach to the post, but they are publicly accessible. So if you upload/attach a picture, you can embed/display it within your post. I agree that it's a completely unintuitive and unnecessarily convoluted process. We're just going to have to put up with it for now though. We have plans to move to alternate software this year.

Beyond the weirdness of our forums, looking at your screenshot, that looks a bit broken! I'm not completely sure but I think that perhaps the DB isn't running for some reason?

My host isn't quite as up to date as yours (and I'm not in a position to update it currently) but it is close. I just downloaded the same TurnKey Nextcloud template as you and launched it. On auto pilot I gave it 2 vCPUs and 2GB RAM (my normal defaults - I mostly use it for testing and there's only a couple of TurnKey appliances that need more than that). It worked fine for me, so I retried just using PVE defaults (1x vCPU & 512MB RAM) as my first guess was running out of RAM (and causing MySQL/MariaDB to crash). But that worked fine too (and didn't go above 200MB RAM usage), so I suspect that it's something on your end. Although I'm not sure what it might be?

The first thing I'd look at is the container resource usage (in the PVE UI). My CPU was maxing out (my server has a low power CPU), particularly when using only one vCPU, but that didn't seem to cause a problem (insufficient CPU resources usually only makes things run slow).

If that looks ok, then another thing to check is how much disk space you have on the PVE host volume - where your container root volume is. I've had vaguely similar weirdness when I over-allocated disk space on my host. Another thing to check is RAM (and CPU) usage on your host (running out of RAM is most likely to cause DB crash in my experience).

If you get this far and still haven't found a clue, then I recommend either trying to diagnose it, or try launching a new instance (depending on how involved you want to get and/or your knowledge and/or interest).

If you want to just try a new one, I would probably just leave that container as is for a moment and launch a completely new one. If you first logged in through the NoVNC window, perhaps try doing it via SSH instead? My host has a fairly slow CPU so I often open the NoVNC window and wait for it to get to a prompt. But instead of logging in there, I log in via SSH to run through the firstboot questions. IMO SSH using a native client is the best experience. FYI I add my key and set a static IP during container setup in the PVE UI to support my workflow.

If you want to take a deeper dive trying to work out the issue, you could try entering the container from the host - via 'pct enter VMID'.

Once inside the container, you should go straight to a prompt and I'd start by looking at the journal:

journalctl -n 50 --no-pager

That will show the last 50 lines of the journal and output it in raw text. FYI by default journalctl outputs to a pager so you can scroll back and forth through it - which can be handy, but without is better for copy/pasting IMO.

Feel free to post that (and any/all other output) back here if you wish and I can assist interpreting it.

If you want to check if my guess (about the DB not running) is correct. check the status of MariaDB:

systemctl status mariadb

In the output, look for the line that starts "Active:" (should be 3rd line down). If it says anything other than "active (running)", please post back all the output.

If it's not "active (running)" (e.g. "active (exited)"), you could try restarting it?:

systemctl restart mariadb

Recheck it's status. Either way, check the journal for the log history:

journalctl -u mariadb -n 30 --no-pager

As per above, that will output the last 30 lines in raw text. If it wasn't running but restarted ok, you might want to increase that number to go back more lines to ensure that you capture the original failure (so we can try to work out why).

looks like i can paste screenshots now ;)

The container virtual disk is on a sfs pool with 3.4 TB available.

zfs list LTData
NAME     USED  AVAIL     REFER  MOUNTPOINT
LTData  1.91T  3.42T      272K  /LTData

 

I have tried recreating the container about 6 or 7 times by now.

It worked 2 out of those times, as in I was able to get through the turnkey-init process and actually use the application for a while.
both times though after a while I was no longer able to log in stating username/password was incorrect.

This leads me to believe that access tot he database stopped working again, these times much later than the other times.

I will not try another container anymore, I'd rather go deep into the one I have or try antoher container with some kind of special diagnostics going on while creating it.

journalctl -n 50 --no-pager

Feb 18 13:30:01 nextcloud CRON[28538]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Feb 18 13:30:01 nextcloud CRON[28539]: (www-data) CMD (^I/usr/bin/php -f /var/www/nextcloud/occ /dev/null 2>&1)
Feb 18 13:30:02 nextcloud CRON[28538]: pam_unix(cron:session): session closed for user www-data
Feb 18 13:30:02 nextcloud postfix/pickup[28451]: 0BCC3A4C: uid=33 from=<www-data>
Feb 18 13:30:02 nextcloud postfix/cleanup[28543]: 0BCC3A4C: message-id=<20230218133002.0BCC3A4C@nextcloud>
Feb 18 13:30:02 nextcloud postfix/qmgr[3636]: 0BCC3A4C: from=<www-data@nextcloud>, size=5134, nrcpt=1 (queue active)
Feb 18 13:30:03 nextcloud postfix/local[28545]: 0BCC3A4C: to=<www-data@nextcloud>, orig_to=<www-data>, relay=local, delay=1.8, delays=1.2/0.01/0/0.52, dsn=2.0.0, status=sent (delivered to mailbox)
Feb 18 13:30:03 nextcloud postfix/qmgr[3636]: 0BCC3A4C: removed
Feb 18 13:39:01 nextcloud CRON[28548]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Feb 18 13:39:01 nextcloud systemd[1]: Starting Clean php session files...
Feb 18 13:39:01 nextcloud CRON[28549]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Feb 18 13:39:01 nextcloud systemd[1]: phpsessionclean.service: Succeeded.
Feb 18 13:39:01 nextcloud CRON[28548]: pam_unix(cron:session): session closed for user root
Feb 18 13:39:01 nextcloud systemd[1]: Finished Clean php session files.
Feb 18 13:45:01 nextcloud CRON[28599]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Feb 18 13:45:01 nextcloud CRON[28600]: (www-data) CMD (^I/usr/bin/php -f /var/www/nextcloud/occ /dev/null 2>&1)
Feb 18 13:45:02 nextcloud CRON[28599]: pam_unix(cron:session): session closed for user www-data
Feb 18 13:45:02 nextcloud postfix/pickup[28451]: 05519A4C: uid=33 from=<www-data>
Feb 18 13:45:02 nextcloud postfix/cleanup[28604]: 05519A4C: message-id=<20230218134502.05519A4C@nextcloud>
Feb 18 13:45:02 nextcloud postfix/qmgr[3636]: 05519A4C: from=<www-data@nextcloud>, size=5134, nrcpt=1 (queue active)
Feb 18 13:45:03 nextcloud postfix/local[28606]: 05519A4C: to=<www-data@nextcloud>, orig_to=<www-data>, relay=local, delay=2, delays=1.3/0.01/0/0.68, dsn=2.0.0, status=sent (delivered to mailbox)
Feb 18 13:45:03 nextcloud postfix/qmgr[3636]: 05519A4C: removed
Feb 18 14:00:01 nextcloud CRON[28609]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Feb 18 14:00:01 nextcloud CRON[28610]: (www-data) CMD (^I/usr/bin/php -f /var/www/nextcloud/occ /dev/null 2>&1)
Feb 18 14:00:01 nextcloud CRON[28609]: pam_unix(cron:session): session closed for user www-data
Feb 18 14:00:01 nextcloud postfix/pickup[28451]: 18267A4C: uid=33 from=<www-data>
Feb 18 14:00:01 nextcloud postfix/cleanup[28614]: 18267A4C: message-id=<20230218140001.18267A4C@nextcloud>
Feb 18 14:00:01 nextcloud postfix/qmgr[3636]: 18267A4C: from=<www-data@nextcloud>, size=5134, nrcpt=1 (queue active)
Feb 18 14:00:01 nextcloud postfix/local[28616]: 18267A4C: to=<www-data@nextcloud>, orig_to=<www-data>, relay=local, delay=0.09, delays=0.06/0/0/0.03, dsn=2.0.0, status=sent (delivered to mailbox)
Feb 18 14:00:01 nextcloud postfix/qmgr[3636]: 18267A4C: removed
Feb 18 14:05:56 nextcloud systemd[1]: Starting Daily apt download activities...
Feb 18 14:05:56 nextcloud systemd[1]: apt-daily.service: Succeeded.
Feb 18 14:05:56 nextcloud systemd[1]: Finished Daily apt download activities.
Feb 18 14:09:01 nextcloud CRON[28670]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Feb 18 14:09:01 nextcloud CRON[28671]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Feb 18 14:09:01 nextcloud systemd[1]: Starting Clean php session files...
Feb 18 14:09:01 nextcloud CRON[28672]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Feb 18 14:09:01 nextcloud CRON[28673]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Feb 18 14:09:01 nextcloud CRON[28671]: pam_unix(cron:session): session closed for user root
Feb 18 14:09:01 nextcloud CRON[28670]: pam_unix(cron:session): session closed for user root
Feb 18 14:09:01 nextcloud systemd[1]: phpsessionclean.service: Succeeded.
Feb 18 14:09:01 nextcloud systemd[1]: Finished Clean php session files.
Feb 18 14:15:01 nextcloud CRON[28736]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Feb 18 14:15:01 nextcloud CRON[28737]: (www-data) CMD (^I/usr/bin/php -f /var/www/nextcloud/occ /dev/null 2>&1)
Feb 18 14:15:01 nextcloud CRON[28736]: pam_unix(cron:session): session closed for user www-data
Feb 18 14:15:01 nextcloud postfix/pickup[28741]: C9CC9A56: uid=33 from=<www-data>
Feb 18 14:15:01 nextcloud postfix/cleanup[28742]: C9CC9A56: message-id=<20230218141501.C9CC9A56@nextcloud>
Feb 18 14:15:02 nextcloud postfix/qmgr[3636]: C9CC9A56: from=<www-data@nextcloud>, size=5134, nrcpt=1 (queue active)
Feb 18 14:15:02 nextcloud postfix/local[28744]: C9CC9A56: to=<www-data@nextcloud>, orig_to=<www-data>, relay=local, delay=1.7, delays=1.1/0.01/0/0.54, dsn=2.0.0, status=sent (delivered to mailbox)
Feb 18 14:15:02 nextcloud postfix/qmgr[3636]: C9CC9A56: removed

above is the output just after having tried entering a password again during terunkey-init. it seems clues to the issue at hand are not being logged.

yes you seem to be on to something.

maria db looks dead

systemctl status mariadb
● mariadb.service - MariaDB 10.5.15 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Fri 2023-02-17 16:20:57 UTC; 21h ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
   Main PID: 510 (code=exited, status=0/SUCCESS)
     Status: "MariaDB server is down"
        CPU: 170ms

Feb 17 16:20:55 nextcloud mariadbd[510]: 2023-02-17 16:20:55 0 [Note] Event Scheduler: Purging the queue. 0 events
Feb 17 16:20:55 nextcloud mariadbd[510]: 2023-02-17 16:20:55 0 [Note] InnoDB: FTS optimize thread exiting.
Feb 17 16:20:55 nextcloud debian-start[2839]: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/run/mysqld/mysqld.sock' (2)
Feb 17 16:20:56 nextcloud mariadbd[510]: 2023-02-17 16:20:56 0 [Note] InnoDB: Starting shutdown...
Feb 17 16:20:56 nextcloud mariadbd[510]: 2023-02-17 16:20:56 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
Feb 17 16:20:56 nextcloud mariadbd[510]: 2023-02-17 16:20:56 0 [Note] InnoDB: Buffer pool(s) dump completed at 230217 16:20:56
Feb 17 16:20:57 nextcloud mariadbd[510]: 2023-02-17 16:20:57 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
Feb 17 16:20:57 nextcloud mariadbd[510]: 2023-02-17 16:20:57 0 [Note] InnoDB: Shutdown completed; log sequence number 884519; transaction id 1675
Feb 17 16:20:57 nextcloud mariadbd[510]: 2023-02-17 16:20:57 0 [Note] /usr/sbin/mariadbd: Shutdown complete
Feb 17 16:20:57 nextcloud systemd[1]: mariadb.service: Succeeded.

after having restarted mariadb this happens when entering the admin password during turnkey-init

journalctl -u mariadb -n 30 --no-pager
-- Journal begins at Fri 2023-02-17 16:20:44 UTC, ends at Sat 2023-02-18 14:19:33 UTC. --
Feb 18 14:19:32 nextcloud mariadbd[28857]: 2023-02-18 14:19:32 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-0+deb11u1) starting as process 28857 ...
Feb 18 14:19:32 nextcloud mariadbd[28857]: 2023-02-18 14:19:32 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/
Feb 18 14:19:32 nextcloud mariadbd[28857]: 2023-02-18 14:19:32 0 [Note] InnoDB: Uses event mutexes
Feb 18 14:19:32 nextcloud mariadbd[28857]: 2023-02-18 14:19:32 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Feb 18 14:19:32 nextcloud mariadbd[28857]: 2023-02-18 14:19:32 0 [Note] InnoDB: Number of pools: 1
Feb 18 14:19:32 nextcloud mariadbd[28857]: 2023-02-18 14:19:32 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Feb 18 14:19:32 nextcloud mariadbd[28857]: 2023-02-18 14:19:32 0 [Note] InnoDB: Using Linux native AIO
Feb 18 14:19:32 nextcloud mariadbd[28857]: 2023-02-18 14:19:32 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
Feb 18 14:19:32 nextcloud mariadbd[28857]: 2023-02-18 14:19:32 0 [Note] InnoDB: Completed initialization of buffer pool
Feb 18 14:19:32 nextcloud mariadbd[28857]: 2023-02-18 14:19:32 0 [Note] InnoDB: 128 rollback segments are active.
Feb 18 14:19:32 nextcloud mariadbd[28857]: 2023-02-18 14:19:32 0 [Note] InnoDB: Creating shared tablespace for temporary tables
Feb 18 14:19:32 nextcloud mariadbd[28857]: 2023-02-18 14:19:32 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
Feb 18 14:19:32 nextcloud mariadbd[28857]: 2023-02-18 14:19:32 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
Feb 18 14:19:32 nextcloud mariadbd[28857]: 2023-02-18 14:19:32 0 [Note] InnoDB: 10.5.15 started; log sequence number 884555; transaction id 1674
Feb 18 14:19:32 nextcloud mariadbd[28857]: 2023-02-18 14:19:32 0 [Note] Plugin 'FEEDBACK' is disabled.
Feb 18 14:19:32 nextcloud mariadbd[28857]: 2023-02-18 14:19:32 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Feb 18 14:19:32 nextcloud mariadbd[28857]: 2023-02-18 14:19:32 0 [Note] InnoDB: Buffer pool(s) load completed at 230218 14:19:32
Feb 18 14:19:33 nextcloud mariadbd[28857]: 2023-02-18 14:19:33 0 [Note] Server socket created on IP: '127.0.0.1'.
Feb 18 14:19:33 nextcloud mariadbd[28857]: 2023-02-18 14:19:33 0 [Note] Reading of all Master_info entries succeeded
Feb 18 14:19:33 nextcloud mariadbd[28857]: 2023-02-18 14:19:33 0 [Note] Added new Master_info '' to hash table
Feb 18 14:19:33 nextcloud mariadbd[28857]: 2023-02-18 14:19:33 0 [Note] /usr/sbin/mariadbd: ready for connections.
Feb 18 14:19:33 nextcloud mariadbd[28857]: Version: '10.5.15-MariaDB-0+deb11u1'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Debian 11
Feb 18 14:19:33 nextcloud systemd[1]: Started MariaDB 10.5.15 database server.
Feb 18 14:19:33 nextcloud /etc/mysql/debian-start[28880]: Upgrading MySQL tables if necessary.
Feb 18 14:19:33 nextcloud /etc/mysql/debian-start[28883]: Looking for 'mysql' as: /usr/bin/mysql
Feb 18 14:19:33 nextcloud /etc/mysql/debian-start[28883]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Feb 18 14:19:33 nextcloud /etc/mysql/debian-start[28883]: This installation of MariaDB is already upgraded to 10.5.15-MariaDB.
Feb 18 14:19:33 nextcloud /etc/mysql/debian-start[28883]: There is no need to run mysql_upgrade again for 10.5.15-MariaDB.
Feb 18 14:19:33 nextcloud /etc/mysql/debian-start[28883]: You can use --force if you still want to run mysql_upgrade
Feb 18 14:19:33 nextcloud /etc/mysql/debian-start[28891]: Checking for insecure root accounts.

any ideas one might have a greatly welcomed.

I am not sure I understood how to get screenshots in posts. I am motivated to jump through the hoops as I think turnkey is awesome and I'd like to help identify issues

this is just after having started mariadb, which was confirmed to have started up.

and the last journal log posted earlier is just after that

caught exception

dialog.py line 3130

...

...

self.wait_for_program_termination    raise DialogError

...

leading and trailing whitespace stripped, was:

 

Jeremy Davis's picture

In future it's probably best to just paste all the text rather than trying to trim it. It's easier for me to get full context from raw text.

Regardless, that shows that it died waiting for something to happen. So some sort of timeout, possibly waiting for MySQL - that had crashed?!

Trying to capture the logs at the point that it died would be really useful to me. Perhaps don't limit the 'journalctl -u mariadb' output? Maybe write it to a txt file and upload to your OP?

To write to a file:

journalctl -u mariadb --no-pager > /mariadb.txt

waiting for MySQL?

I am far from qualified so please forgive my confusion.

The database used for the next cloud system is MySQL or MariaDB?

Jeremy Davis's picture

It's MariaDB. But it replaces MySQL in Debian, so the terms have been used interchangeably. As of now though it's actually MariaDB (and think MariaDB anytime MySQL is used in context with TurnKey - official MySQL is not available).

then I would no longer have need for screenshots

Jeremy Davis's picture

There is, but it's minimal and likely won't be a lot f help (but might be some?). You should be able to find it at /var/log/inithooks.log

Jeremy Davis's picture

Our current software only allows the first post in a thread to have files attached/uploaded. So if you want to upload files, edit your first post. A bug means that attached files aren't publicly displayed (only you and I can see them - when editing the post). Images can be displayed within any post - but the image needs to be "embedded" (i.e. it needs to already be on the internet somewhere). So displaying a screenshot is a 2 step process, upload (here via editing your OP, or anywhere else that can host an image), then embedding the picture in your post (via it's URL).

Screenshots are really useful for displaying anything that is rendered, e.g. visual issues with web content, visual issues with CLI rendering, etc), but generally aren't that great for general log and CLI output.

In the case of text (e.g. log output) it's probably best copy/pasted anyway, rather than screenshots. Using raw text should be quicker and easier for you, plus easier reading for me. It also makes research much easier for me (because I can in turn copy/paste from the log output you provide).

But back to your problem at hand


Ok, so it seems like I was potentially onto something and MySQL/MariaDB is dying/being killed. (FYI I use MySQL and MariaDB interchangeably although on TurnKey it's specifically MariaDB, not MySQL - ask if you want more context).

Unfortunately, none of the log/journal output you've shared shows the failure. I can see it starting ok and I can see it shutting down. But it's not completely clear even if the shutdown is from a reboot (when it should shut down as part of the poweroff process) or if something else told it to shutdown or it's doing that itself due to some failure condition?

It might be useful for me to understand where the failures occur a little better? On these recent runs, could you please clarify on the unsuccessful times, whether it dies on exactly the same place and same result (e.g. as per the screenshot in your first post). Or did it seem to happen at a somewhat random point every time? If different, was it completely different every time? Or vaguely in the same place? Also, does it always only occur after entering data and hitting enter? Or does it sometimes occur mid data entry (e.g. become unresponsive while you are entering data - before you have finished)?

FWIW my previous thoughts were following a mental thread, thinking that the firstboot issues may have just been a symptom of the DB failure. That may still be the case as the MySQL question (set adminer user password) and the Nextcloud specific questions (email, domain, password) all depend on MySQL running. So if the lockup only occurs after hitting enter after one of those, I may have been on the right track. However, the fact that it appears to work ok occasionally, but then still has issues later (i.e. quite intermittent, but ultimately ends in a similar way) - suggests to me that even if the direct cause is MySQL crashing, the underlying cause is something to do with your host. E.g. some resource limit being hit.

When you say:

[when it worked,] both times though after a while I was no longer able to log in stating username/password was incorrect.

Which log in are you referring to? In a particular web app? Adminer? Webmin? SSH? Via the NoVNC window? Any additional info related to which specific interfaces you tried and the results would be invaluable! E.g. if it all worked fine in the NoVNC window but not Adminer, or Webmin stopped working, etc.

Also, what sort of existing workload(s) do you have on your Proxmox currently? Have you experienced any even vaguely similar issues with other containers - or VMs? Have you recently tried any non-TurnKey containers - or VMs? Obviously I can't rule out something TurnKey specific yet, but I can't help but think that there is something not quite right with your host and/or the environment that the containers are running in. Perhaps double check RAM availability on the host (assuming you have swap set up on your host, that should only cause it to run really slow though, I wouldn't expect crashes).

indeed it has nothing noteworthy in it.

is multifold as was both no longer able to log into adminer and the general web interface.
Both stopped accepting my credentials after some time.

Jeremy Davis's picture

Both the main web app (e.g. WordPress on our WordPress appliance) and Adminer will depend on MariaDB, so neither will work if the DB crashes. The main web app would likely just error if it can't access the DB

I just wanted to clarify whether this issue was affecting any other components? I'm gathering from your feedback that you can still log in ok via SSH, so it seems likely that the issue is limited to MariaDB crashing.

But why is it crashing?

One of the few test instances I launched back when you first reported this is still running flawlessly?!

Jeremy Davis's picture

Beyond CPU, you have very similar resource specs to me (mine is a low power Atom CPU - 8x 2.4GHz). I note that you don't have any swap set up. I highly doubt that is relevant here, but I recommend setting up swap. Although I tend to run RAM heavy workloads, rather than CPU heavy ones. I have 8GB swap configured and even though my RAM usage is currently ~65% (of 32GB; i.e. ~20GB) it still uses swap (currently ~3GB = ~35%). I also note that you don't have any KSM shared memory. If you have similar VMs running, I'd be inclined to investigate why that's not working? It allows similar OS to share host memory and will reduce overall RAM load quite a bit (e.g. I currently have nearly 5GB shared KSM - shared between my ~10 running VMs).

None of that should be the cause of issues you've reported, but I thought was worth sharing.

What sort of workloads do you have running? I.e. are they mostly VMs or containers (or a blend of both)? What OS are you running as guests? Do you have any other LXC containers running reliably with MariaDB (or MySQL)? Have you tried launching any non-TurnKey containers?

Regardless, we need to work out why MariaDB is dying/being killed. Can you please dump the complete journal (i.e. all of it) for MariaDB:

journalctl -u mariadb --no-pager > mariadb.log.txt

You can edit your original post and attach the file there if you want (but please let me know if you've done that so I know to look for it).

Another thing you could try is launching TurnKey into a VM instead of a container. ISOs can be downloaded from the website.

Also, if you have some non-ZFS storage set up, perhaps try putting the container root volume on that. I have no reason to think that ZFS is an issue here, but it's the only other difference of significance between my set up and yours. (FWIW I have only had issues with ZFS myself, but I always assumed that it was a failing hard drive that caused all my pain).

given that you are passionate and really involved I can give you access to my desktop. You can create the container and see it go down the drain from there.

 

Yes sure this is like unpaid support.

Don't worry I am considering going for paid support later down the line.

Jeremy Davis's picture

I've sent you an email with my public key. Add that to your /root/.ssh/authorized_keys file on the server you're giving me access to and share it's public IP (and ensure that it's publicly accessible).

Let me know (inc the public IP) via email when you're ready to go and we'll take it from there. Probably best to keep that communication via email and we can report back here once I've had a poke around.

And can log in again.

But this is probably temporary as I have been here before and then a few hours/days later I can no longer login.

If/when that happens again I'll continue the hunt for the root cause

this is after like a day or soner and I want to use the web interface.

 

Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

...

the moral of the story is as this/

it's a hit or miss wether one can get through the turnkey-init process.

and even when one does then it is a matter of time before one can no longer log in

and now we have even internal server issues

Jeremy Davis's picture

I wonder why MariaDB is so unstable on your system? There must be some reason why!? The logs you've shown me so far don't indicate any crash though! They just show that it shut down?!

Re-reading through this thread, I'm wondering if it's just simply running out of RAM? How much RAM does the guest have? (Apologies if you've already noted that somewhere and I missed it on my re-read). On my Proxmox v7.x system new containers default to 512MB RAM (with 512MB swap). I would expect that to be enough for a fairly simple setup with minimal traffic. However perhaps it's not enough in your config for some reason?

If you haven't already, perhaps try with more RAM allocated? I'd try 2GB (i.e. 2048MB). (Again sorry if I've already mentioned that and I missed it on reread).

FWIW I have one running (with 1GB RAM allocated) that has been up for 2 weeks now and is still running fine.

Something else that might be worth a try is dumping the whole MariaDB journal and uploading it (edit your first post and attach the file there - let me know when you've done that as they won't be immediately visible to anyone but you). Dump all MariaDB journal entries to a file like this:

journalctl -u mariadb -no-pager > /root/mariadb-journal.txt

In case it's not obvious, the file will be found at /root/mariadb-journal.txt

I have the same issues with the redmine turnkey. Vanilla lxc's seem to not misbehave.

The nextcloud container has

CPU usage 0.01% of 4 CPU(s)
Memory usage 2.01% (164.43 MiB of 8.00 GiB)
SWAP usage 0.00% (0 B of 512.00 MiB)
Bootdisk size 21.02% (1.68 GiB of 8.00 GiB
 
Mariadb seems to be running fine;

systemctl status mariadb
● mariadb.service - MariaDB 10.5.15 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2023-03-11 19:19:14 UTC; 1 weeks 0 days ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 247 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
    Process: 262 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    Process: 291 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq >
    Process: 582 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    Process: 586 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
   Main PID: 450 (mariadbd)
     Status: "Taking your SQL requests now..."
      Tasks: 9 (limit: 38288)
     Memory: 72.9M
        CPU: 44.951s
     CGroup: /system.slice/mariadb.service
             └─450 /usr/sbin/mariadbd

Mar 11 19:19:14 nextcloud mariadbd[450]: Version: '10.5.15-MariaDB-0+deb11u1'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Debian 11
Mar 11 19:19:14 nextcloud systemd[1]: Started MariaDB 10.5.15 database server.
Mar 11 19:19:14 nextcloud /etc/mysql/debian-start[594]: Upgrading MySQL tables if necessary.
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Note] InnoDB: Buffer pool(s) load completed at 230311 19:19:14
Mar 11 19:19:14 nextcloud /etc/mysql/debian-start[597]: Looking for 'mysql' as: /usr/bin/mysql
Mar 11 19:19:14 nextcloud /etc/mysql/debian-start[597]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Mar 11 19:19:14 nextcloud /etc/mysql/debian-start[597]: This installation of MariaDB is already upgraded to 10.5.15-MariaDB.
Mar 11 19:19:14 nextcloud /etc/mysql/debian-start[597]: There is no need to run mysql_upgrade again for 10.5.15-MariaDB.
Mar 11 19:19:14 nextcloud /etc/mysql/debian-start[597]: You can use --force if you still want to run mysql_upgrade
Mar 11 19:19:14 nextcloud /etc/mysql/debian-start[640]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables

 

 

 

 

journalctl -u mariadb --no-pager
-- Journal begins at Fri 2023-03-10 15:16:08 UTC, ends at Sun 2023-03-19 11:09:05 UTC. --
Mar 10 15:16:11 nextcloud systemd[1]: Starting MariaDB 10.5.15 database server...
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-0+deb11u1) starting as process 586 ...
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Note] InnoDB: Uses event mutexes
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Note] InnoDB: Number of pools: 1
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Note] InnoDB: Using Linux native AIO
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Note] InnoDB: Completed initialization of buffer pool
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Note] InnoDB: 128 rollback segments are active.
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Note] InnoDB: Creating shared tablespace for temporary tables
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Note] InnoDB: 10.5.15 started; log sequence number 884507; transaction id 1674
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Note] Plugin 'FEEDBACK' is disabled.
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Note] Server socket created on IP: '127.0.0.1'.
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Note] Reading of all Master_info entries succeeded
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Note] Added new Master_info '' to hash table
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Note] /usr/sbin/mariadbd: ready for connections.
Mar 10 15:16:12 nextcloud mariadbd[586]: Version: '10.5.15-MariaDB-0+deb11u1'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Debian 11
Mar 10 15:16:12 nextcloud mariadbd[586]: 2023-03-10 15:16:12 0 [Note] InnoDB: Buffer pool(s) load completed at 230310 15:16:12
Mar 10 15:16:12 nextcloud systemd[1]: Started MariaDB 10.5.15 database server.
Mar 10 15:16:12 nextcloud /etc/mysql/debian-start[1348]: Upgrading MySQL tables if necessary.
Mar 10 15:16:12 nextcloud /etc/mysql/debian-start[1401]: Checking for insecure root accounts.
Mar 10 15:16:12 nextcloud /etc/mysql/debian-start[1422]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
Mar 10 15:19:11 nextcloud systemd[1]: Stopping MariaDB 10.5.15 database server...
Mar 10 15:19:11 nextcloud mariadbd[586]: 2023-03-10 15:19:11 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
Mar 10 15:19:11 nextcloud mariadbd[586]: 2023-03-10 15:19:11 0 [Note] Event Scheduler: Purging the queue. 0 events
Mar 10 15:19:11 nextcloud mariadbd[586]: 2023-03-10 15:19:11 0 [Note] InnoDB: FTS optimize thread exiting.
Mar 10 15:19:11 nextcloud mariadbd[586]: 2023-03-10 15:19:11 0 [Note] InnoDB: Starting shutdown...
Mar 10 15:19:11 nextcloud mariadbd[586]: 2023-03-10 15:19:11 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
Mar 10 15:19:11 nextcloud mariadbd[586]: 2023-03-10 15:19:11 0 [Note] InnoDB: Buffer pool(s) dump completed at 230310 15:19:11
Mar 10 15:19:11 nextcloud mariadbd[586]: 2023-03-10 15:19:11 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
Mar 10 15:19:11 nextcloud mariadbd[586]: 2023-03-10 15:19:11 0 [Note] InnoDB: Shutdown completed; log sequence number 884519; transaction id 1676
Mar 10 15:19:11 nextcloud mariadbd[586]: 2023-03-10 15:19:11 0 [Note] /usr/sbin/mariadbd: Shutdown complete
Mar 10 15:19:11 nextcloud systemd[1]: mariadb.service: Succeeded.
Mar 10 15:19:11 nextcloud systemd[1]: Stopped MariaDB 10.5.15 database server.
-- Boot 494a822467674585866bc8d8c3c039bd --
Mar 10 15:19:16 nextcloud systemd[1]: Starting MariaDB 10.5.15 database server...
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-0+deb11u1) starting as process 508 ...
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Note] InnoDB: Uses event mutexes
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Note] InnoDB: Number of pools: 1
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Note] InnoDB: Using Linux native AIO
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Note] InnoDB: Completed initialization of buffer pool
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Note] InnoDB: 128 rollback segments are active.
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Note] InnoDB: Creating shared tablespace for temporary tables
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Note] InnoDB: 10.5.15 started; log sequence number 884519; transaction id 1674
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Note] Plugin 'FEEDBACK' is disabled.
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Note] Server socket created on IP: '127.0.0.1'.
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Note] Reading of all Master_info entries succeeded
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Note] Added new Master_info '' to hash table
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Note] /usr/sbin/mariadbd: ready for connections.
Mar 10 15:19:16 nextcloud mariadbd[508]: Version: '10.5.15-MariaDB-0+deb11u1'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Debian 11
Mar 10 15:19:16 nextcloud mariadbd[508]: 2023-03-10 15:19:16 0 [Note] InnoDB: Buffer pool(s) load completed at 230310 15:19:16
Mar 10 15:19:16 nextcloud systemd[1]: Started MariaDB 10.5.15 database server.
Mar 10 15:19:16 nextcloud /etc/mysql/debian-start[609]: Upgrading MySQL tables if necessary.
Mar 10 15:19:34 nextcloud systemd[1]: Stopping MariaDB 10.5.15 database server...
Mar 10 15:19:34 nextcloud mariadbd[508]: 2023-03-10 15:19:34 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
Mar 10 15:19:34 nextcloud mariadbd[508]: 2023-03-10 15:19:34 0 [Note] Event Scheduler: Purging the queue. 0 events
Mar 10 15:19:34 nextcloud mariadbd[508]: 2023-03-10 15:19:34 0 [Note] InnoDB: FTS optimize thread exiting.
Mar 10 15:19:34 nextcloud mariadbd[508]: 2023-03-10 15:19:34 0 [Note] InnoDB: Starting shutdown...
Mar 10 15:19:34 nextcloud mariadbd[508]: 2023-03-10 15:19:34 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
Mar 10 15:19:34 nextcloud mariadbd[508]: 2023-03-10 15:19:34 0 [Note] InnoDB: Buffer pool(s) dump completed at 230310 15:19:34
Mar 10 15:19:34 nextcloud mariadbd[508]: 2023-03-10 15:19:34 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
Mar 10 15:19:34 nextcloud mariadbd[508]: 2023-03-10 15:19:34 0 [Note] InnoDB: Shutdown completed; log sequence number 884531; transaction id 1675
Mar 10 15:19:34 nextcloud mariadbd[508]: 2023-03-10 15:19:34 0 [Note] /usr/sbin/mariadbd: Shutdown complete
Mar 10 15:19:34 nextcloud systemd[1]: mariadb.service: Succeeded.
Mar 10 15:19:34 nextcloud systemd[1]: Stopped MariaDB 10.5.15 database server.
-- Boot f13f864e3c274324b83eedf4e8e28e4f --
Mar 10 15:20:50 nextcloud systemd[1]: Starting MariaDB 10.5.15 database server...
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-0+deb11u1) starting as process 498 ...
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Note] InnoDB: Uses event mutexes
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Note] InnoDB: Number of pools: 1
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Note] InnoDB: Using Linux native AIO
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Note] InnoDB: Completed initialization of buffer pool
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Note] InnoDB: 128 rollback segments are active.
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Note] InnoDB: Creating shared tablespace for temporary tables
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Note] InnoDB: 10.5.15 started; log sequence number 884531; transaction id 1674
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Note] Plugin 'FEEDBACK' is disabled.
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Note] Server socket created on IP: '127.0.0.1'.
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Note] InnoDB: Buffer pool(s) load completed at 230310 15:20:50
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Note] Reading of all Master_info entries succeeded
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Note] Added new Master_info '' to hash table
Mar 10 15:20:50 nextcloud mariadbd[498]: 2023-03-10 15:20:50 0 [Note] /usr/sbin/mariadbd: ready for connections.
Mar 10 15:20:50 nextcloud mariadbd[498]: Version: '10.5.15-MariaDB-0+deb11u1'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Debian 11
Mar 10 15:20:50 nextcloud systemd[1]: Started MariaDB 10.5.15 database server.
Mar 10 15:20:50 nextcloud /etc/mysql/debian-start[647]: Looking for 'mysql' as: /usr/bin/mysql
Mar 10 15:20:50 nextcloud /etc/mysql/debian-start[647]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Mar 10 15:20:50 nextcloud /etc/mysql/debian-start[647]: This installation of MariaDB is already upgraded to 10.5.15-MariaDB.
Mar 10 15:20:50 nextcloud /etc/mysql/debian-start[647]: There is no need to run mysql_upgrade again for 10.5.15-MariaDB.
Mar 10 15:20:50 nextcloud /etc/mysql/debian-start[647]: You can use --force if you still want to run mysql_upgrade
Mar 10 15:20:50 nextcloud /etc/mysql/debian-start[660]: Checking for insecure root accounts.
Mar 10 15:58:12 nextcloud systemd[1]: Stopping MariaDB 10.5.15 database server...
Mar 10 15:58:12 nextcloud mariadbd[498]: 2023-03-10 15:58:12 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
Mar 10 15:58:12 nextcloud mariadbd[498]: 2023-03-10 15:58:12 0 [Note] Event Scheduler: Purging the queue. 0 events
Mar 10 15:58:12 nextcloud mariadbd[498]: 2023-03-10 15:58:12 0 [Note] InnoDB: FTS optimize thread exiting.
Mar 10 15:58:12 nextcloud mariadbd[498]: 2023-03-10 15:58:12 0 [Note] InnoDB: Starting shutdown...
Mar 10 15:58:12 nextcloud mariadbd[498]: 2023-03-10 15:58:12 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
Mar 10 15:58:12 nextcloud mariadbd[498]: 2023-03-10 15:58:12 0 [Note] InnoDB: Buffer pool(s) dump completed at 230310 15:58:12
Mar 10 15:58:12 nextcloud mariadbd[498]: 2023-03-10 15:58:12 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
Mar 10 15:58:12 nextcloud mariadbd[498]: 2023-03-10 15:58:12 0 [Note] InnoDB: Shutdown completed; log sequence number 884543; transaction id 1676
Mar 10 15:58:12 nextcloud mariadbd[498]: 2023-03-10 15:58:12 0 [Note] /usr/sbin/mariadbd: Shutdown complete
Mar 10 15:58:12 nextcloud systemd[1]: mariadb.service: Succeeded.
Mar 10 15:58:12 nextcloud systemd[1]: Stopped MariaDB 10.5.15 database server.
-- Boot 4051c24d39764fb8be09aa187f4b501b --
Mar 10 16:00:31 nextcloud systemd[1]: Starting MariaDB 10.5.15 database server...
Mar 10 16:00:31 nextcloud mariadbd[478]: 2023-03-10 16:00:31 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-0+deb11u1) starting as process 478 ...
Mar 10 16:00:31 nextcloud mariadbd[478]: 2023-03-10 16:00:31 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/
Mar 10 16:00:31 nextcloud mariadbd[478]: 2023-03-10 16:00:31 0 [Note] InnoDB: Uses event mutexes
Mar 10 16:00:31 nextcloud mariadbd[478]: 2023-03-10 16:00:31 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Mar 10 16:00:31 nextcloud mariadbd[478]: 2023-03-10 16:00:31 0 [Note] InnoDB: Number of pools: 1
Mar 10 16:00:31 nextcloud mariadbd[478]: 2023-03-10 16:00:31 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Mar 10 16:00:32 nextcloud mariadbd[478]: 2023-03-10 16:00:32 0 [Note] InnoDB: Using Linux native AIO
Mar 10 16:00:32 nextcloud mariadbd[478]: 2023-03-10 16:00:32 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
Mar 10 16:00:32 nextcloud mariadbd[478]: 2023-03-10 16:00:32 0 [Note] InnoDB: Completed initialization of buffer pool
Mar 10 16:00:32 nextcloud mariadbd[478]: 2023-03-10 16:00:32 0 [Note] InnoDB: 128 rollback segments are active.
Mar 10 16:00:32 nextcloud mariadbd[478]: 2023-03-10 16:00:32 0 [Note] InnoDB: Creating shared tablespace for temporary tables
Mar 10 16:00:32 nextcloud mariadbd[478]: 2023-03-10 16:00:32 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
Mar 10 16:00:32 nextcloud mariadbd[478]: 2023-03-10 16:00:32 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
Mar 10 16:00:32 nextcloud mariadbd[478]: 2023-03-10 16:00:32 0 [Note] InnoDB: 10.5.15 started; log sequence number 884543; transaction id 1674
Mar 10 16:00:32 nextcloud mariadbd[478]: 2023-03-10 16:00:32 0 [Note] Plugin 'FEEDBACK' is disabled.
Mar 10 16:00:32 nextcloud mariadbd[478]: 2023-03-10 16:00:32 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Mar 10 16:00:32 nextcloud mariadbd[478]: 2023-03-10 16:00:32 0 [Note] Server socket created on IP: '127.0.0.1'.
Mar 10 16:00:32 nextcloud mariadbd[478]: 2023-03-10 16:00:32 0 [Note] Reading of all Master_info entries succeeded
Mar 10 16:00:32 nextcloud mariadbd[478]: 2023-03-10 16:00:32 0 [Note] Added new Master_info '' to hash table
Mar 10 16:00:32 nextcloud mariadbd[478]: 2023-03-10 16:00:32 0 [Note] /usr/sbin/mariadbd: ready for connections.
Mar 10 16:00:32 nextcloud mariadbd[478]: Version: '10.5.15-MariaDB-0+deb11u1'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Debian 11
Mar 10 16:00:32 nextcloud systemd[1]: Started MariaDB 10.5.15 database server.
Mar 10 16:00:32 nextcloud /etc/mysql/debian-start[674]: Upgrading MySQL tables if necessary.
Mar 10 16:00:32 nextcloud mariadbd[478]: 2023-03-10 16:00:32 0 [Note] InnoDB: Buffer pool(s) load completed at 230310 16:00:32
Mar 10 16:00:32 nextcloud /etc/mysql/debian-start[679]: Looking for 'mysql' as: /usr/bin/mysql
Mar 10 16:00:32 nextcloud /etc/mysql/debian-start[679]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Mar 10 16:00:32 nextcloud /etc/mysql/debian-start[679]: This installation of MariaDB is already upgraded to 10.5.15-MariaDB.
Mar 10 16:00:32 nextcloud /etc/mysql/debian-start[679]: There is no need to run mysql_upgrade again for 10.5.15-MariaDB.
Mar 10 16:00:32 nextcloud /etc/mysql/debian-start[679]: You can use --force if you still want to run mysql_upgrade
Mar 10 16:00:32 nextcloud /etc/mysql/debian-start[717]: Checking for insecure root accounts.
-- Boot 60b4a22d31b64e618bf12ab48cbc2b9e --
Mar 10 16:12:12 nextcloud systemd[1]: Starting MariaDB 10.5.15 database server...
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-0+deb11u1) starting as process 517 ...
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] mariadbd: Aria engine: starting recovery
Mar 10 16:12:13 nextcloud mariadbd[517]: recovered pages: 0% 49% 100% (0.0 seconds); tables to flush: 1 0
Mar 10 16:12:13 nextcloud mariadbd[517]:  (0.0 seconds);
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] mariadbd: Aria engine: recovery done
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] InnoDB: Uses event mutexes
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] InnoDB: Number of pools: 1
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] InnoDB: Using Linux native AIO
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] InnoDB: Completed initialization of buffer pool
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=884543,884543
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] InnoDB: 128 rollback segments are active.
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] InnoDB: Creating shared tablespace for temporary tables
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] InnoDB: 10.5.15 started; log sequence number 884555; transaction id 1674
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] Plugin 'FEEDBACK' is disabled.
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] Server socket created on IP: '127.0.0.1'.
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] Reading of all Master_info entries succeeded
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] Added new Master_info '' to hash table
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] /usr/sbin/mariadbd: ready for connections.
Mar 10 16:12:13 nextcloud mariadbd[517]: Version: '10.5.15-MariaDB-0+deb11u1'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Debian 11
Mar 10 16:12:13 nextcloud systemd[1]: Started MariaDB 10.5.15 database server.
Mar 10 16:12:13 nextcloud mariadbd[517]: 2023-03-10 16:12:13 0 [Note] InnoDB: Buffer pool(s) load completed at 230310 16:12:13
Mar 10 16:12:13 nextcloud /etc/mysql/debian-start[665]: Looking for 'mysql' as: /usr/bin/mysql
Mar 10 16:12:13 nextcloud /etc/mysql/debian-start[665]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Mar 10 16:12:13 nextcloud /etc/mysql/debian-start[665]: This installation of MariaDB is already upgraded to 10.5.15-MariaDB.
Mar 10 16:12:13 nextcloud /etc/mysql/debian-start[665]: There is no need to run mysql_upgrade again for 10.5.15-MariaDB.
Mar 10 16:12:13 nextcloud /etc/mysql/debian-start[665]: You can use --force if you still want to run mysql_upgrade
Mar 10 19:48:28 nextcloud systemd[1]: Stopping MariaDB 10.5.15 database server...
Mar 10 19:48:28 nextcloud mariadbd[517]: 2023-03-10 19:48:28 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
Mar 10 19:48:28 nextcloud mariadbd[517]: 2023-03-10 19:48:28 0 [Note] Event Scheduler: Purging the queue. 0 events
Mar 10 19:48:28 nextcloud mariadbd[517]: 2023-03-10 19:48:28 0 [Note] InnoDB: FTS optimize thread exiting.
Mar 10 19:48:28 nextcloud mariadbd[517]: 2023-03-10 19:48:28 0 [Note] InnoDB: Starting shutdown...
Mar 10 19:48:28 nextcloud mariadbd[517]: 2023-03-10 19:48:28 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
Mar 10 19:48:28 nextcloud mariadbd[517]: 2023-03-10 19:48:28 0 [Note] InnoDB: Buffer pool(s) dump completed at 230310 19:48:28
Mar 10 19:48:28 nextcloud mariadbd[517]: 2023-03-10 19:48:28 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
Mar 10 19:48:28 nextcloud mariadbd[517]: 2023-03-10 19:48:28 0 [Note] InnoDB: Shutdown completed; log sequence number 1292026; transaction id 2537
Mar 10 19:48:28 nextcloud mariadbd[517]: 2023-03-10 19:48:28 0 [Note] /usr/sbin/mariadbd: Shutdown complete
Mar 10 19:48:28 nextcloud systemd[1]: mariadb.service: Succeeded.
Mar 10 19:48:28 nextcloud systemd[1]: Stopped MariaDB 10.5.15 database server.
Mar 10 19:48:28 nextcloud systemd[1]: mariadb.service: Consumed 2.523s CPU time.
-- Boot f3b957cb83394adca5da27ef2f8af477 --
Mar 10 19:48:33 nextcloud systemd[1]: Starting MariaDB 10.5.15 database server...
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-0+deb11u1) starting as process 483 ...
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Note] InnoDB: Uses event mutexes
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Note] InnoDB: Number of pools: 1
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Note] InnoDB: Using Linux native AIO
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Note] InnoDB: Completed initialization of buffer pool
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Note] InnoDB: 128 rollback segments are active.
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Note] InnoDB: Creating shared tablespace for temporary tables
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Note] InnoDB: 10.5.15 started; log sequence number 1292026; transaction id 2538
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Note] Plugin 'FEEDBACK' is disabled.
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Note] Server socket created on IP: '127.0.0.1'.
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Note] InnoDB: Buffer pool(s) load completed at 230310 19:48:33
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Note] Reading of all Master_info entries succeeded
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Note] Added new Master_info '' to hash table
Mar 10 19:48:33 nextcloud mariadbd[483]: 2023-03-10 19:48:33 0 [Note] /usr/sbin/mariadbd: ready for connections.
Mar 10 19:48:33 nextcloud mariadbd[483]: Version: '10.5.15-MariaDB-0+deb11u1'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Debian 11
Mar 10 19:48:33 nextcloud systemd[1]: Started MariaDB 10.5.15 database server.
Mar 10 19:48:33 nextcloud /etc/mysql/debian-start[632]: Looking for 'mysql' as: /usr/bin/mysql
Mar 10 19:48:33 nextcloud /etc/mysql/debian-start[632]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Mar 10 19:48:33 nextcloud /etc/mysql/debian-start[632]: This installation of MariaDB is already upgraded to 10.5.15-MariaDB.
Mar 10 19:48:33 nextcloud /etc/mysql/debian-start[632]: There is no need to run mysql_upgrade again for 10.5.15-MariaDB.
Mar 10 19:48:33 nextcloud /etc/mysql/debian-start[632]: You can use --force if you still want to run mysql_upgrade
Mar 10 19:48:33 nextcloud /etc/mysql/debian-start[652]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
Mar 11 19:18:30 nextcloud systemd[1]: Stopping MariaDB 10.5.15 database server...
Mar 11 19:18:30 nextcloud mariadbd[483]: 2023-03-11 19:18:30 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
Mar 11 19:18:30 nextcloud mariadbd[483]: 2023-03-11 19:18:30 0 [Note] Event Scheduler: Purging the queue. 0 events
Mar 11 19:18:30 nextcloud mariadbd[483]: 2023-03-11 19:18:30 0 [Note] InnoDB: FTS optimize thread exiting.
Mar 11 19:18:30 nextcloud mariadbd[483]: 2023-03-11 19:18:30 0 [Note] InnoDB: Starting shutdown...
Mar 11 19:18:30 nextcloud mariadbd[483]: 2023-03-11 19:18:30 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
Mar 11 19:18:30 nextcloud mariadbd[483]: 2023-03-11 19:18:30 0 [Note] InnoDB: Buffer pool(s) dump completed at 230311 19:18:30
Mar 11 19:18:31 nextcloud mariadbd[483]: 2023-03-11 19:18:31 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
Mar 11 19:18:31 nextcloud mariadbd[483]: 2023-03-11 19:18:31 0 [Note] InnoDB: Shutdown completed; log sequence number 5595809; transaction id 10803
Mar 11 19:18:31 nextcloud mariadbd[483]: 2023-03-11 19:18:31 0 [Note] /usr/sbin/mariadbd: Shutdown complete
Mar 11 19:18:31 nextcloud systemd[1]: mariadb.service: Succeeded.
Mar 11 19:18:31 nextcloud systemd[1]: Stopped MariaDB 10.5.15 database server.
Mar 11 19:18:31 nextcloud systemd[1]: mariadb.service: Consumed 14.495s CPU time.
-- Boot 366c73e06aac4101988c710636c85b8a --
Mar 11 19:19:13 nextcloud systemd[1]: Starting MariaDB 10.5.15 database server...
Mar 11 19:19:13 nextcloud mariadbd[450]: 2023-03-11 19:19:13 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-0+deb11u1) starting as process 450 ...
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Note] InnoDB: Uses event mutexes
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Note] InnoDB: Number of pools: 1
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Note] InnoDB: Using Linux native AIO
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Note] InnoDB: Completed initialization of buffer pool
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Note] InnoDB: 128 rollback segments are active.
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Note] InnoDB: Creating shared tablespace for temporary tables
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Note] InnoDB: 10.5.15 started; log sequence number 5595809; transaction id 10804
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Note] Plugin 'FEEDBACK' is disabled.
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Note] Server socket created on IP: '127.0.0.1'.
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Note] Reading of all Master_info entries succeeded
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Note] Added new Master_info '' to hash table
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Note] /usr/sbin/mariadbd: ready for connections.
Mar 11 19:19:14 nextcloud mariadbd[450]: Version: '10.5.15-MariaDB-0+deb11u1'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Debian 11
Mar 11 19:19:14 nextcloud systemd[1]: Started MariaDB 10.5.15 database server.
Mar 11 19:19:14 nextcloud /etc/mysql/debian-start[594]: Upgrading MySQL tables if necessary.
Mar 11 19:19:14 nextcloud mariadbd[450]: 2023-03-11 19:19:14 0 [Note] InnoDB: Buffer pool(s) load completed at 230311 19:19:14
Mar 11 19:19:14 nextcloud /etc/mysql/debian-start[597]: Looking for 'mysql' as: /usr/bin/mysql
Mar 11 19:19:14 nextcloud /etc/mysql/debian-start[597]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Mar 11 19:19:14 nextcloud /etc/mysql/debian-start[597]: This installation of MariaDB is already upgraded to 10.5.15-MariaDB.
Mar 11 19:19:14 nextcloud /etc/mysql/debian-start[597]: There is no need to run mysql_upgrade again for 10.5.15-MariaDB.
Mar 11 19:19:14 nextcloud /etc/mysql/debian-start[597]: You can use --force if you still want to run mysql_upgrade
Mar 11 19:19:14 nextcloud /etc/mysql/debian-start[640]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables

 

b

Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

Leads me again to the redis server that is not running anylonger.

cat /var/log/apache2/error.log

----- start snippet
[Sat Mar 11 21:22:29.415969 2023] [php7:error] [pid 1166] [client 10.4.0.247:42100] PHP Fatal error:  Uncaught RedisException: No such file or directory in /var/www/nextcloud/lib/private/RedisFactory.php:137\nStack trace:\n#0 /var/www/nextcloud/lib/private/RedisFactory.php(137): Redis->pconnect()\n#1 /var/www/nextcloud/lib/private/RedisFactory.php(178): OC\\RedisFactory->create()\n#2 /var/www/nextcloud/lib/private/Memcache/Redis.php(43): OC\\RedisFactory->getInstance()\n#3 /var/www/nextcloud/lib/private/Memcache/Factory.php(118): OC\\Memcache\\Redis->__construct()\n#4 /var/www/nextcloud/lib/private/Server.php(1120): OC\\Memcache\\Factory->createLocking()\n#5 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(162): OC\\Server->OC\\{closure}()\n#6 /var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(122): OC\\AppFramework\\Utility\\SimpleContainer->OC\\AppFramework\\Utility\\{closure}()\n#7 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(129): Pimple\\Container->offsetGet()\n#8 /var/www/nextcloud/lib/private/ServerContainer.php(136): OC\\AppFramework\\Utility\\SimpleContainer->que in /var/www/nextcloud/lib/private/RedisFactory.php on line 137

----- end snippet

 

This error looks similar to the error I get when running the container without nesting (which I really want).
Having the container with nesting on this time enabled me to get through the turnkey-init procedure. But like I predicted it was only a matter of time.

systemctl status redis-server
● redis-server.service - Advanced key-value store
     Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sat 2023-03-11 19:19:14 UTC; 1 weeks 0 days ago
       Docs: http://redis.io/documentation,
             man:redis-server(1)
    Process: 820 ExecStart=/usr/bin/redis-server /etc/redis/redis.conf --supervised systemd --daemonize no (code=exited, status=226/NAMESPACE)
   Main PID: 820 (code=exited, status=226/NAMESPACE)
        CPU: 7ms

Mar 11 19:19:14 nextcloud systemd[1]: redis-server.service: Scheduled restart job, restart counter is at 5.
Mar 11 19:19:14 nextcloud systemd[1]: Stopped Advanced key-value store.
Mar 11 19:19:14 nextcloud systemd[1]: redis-server.service: Start request repeated too quickly.
Mar 11 19:19:14 nextcloud systemd[1]: redis-server.service: Failed with result 'exit-code'.
Mar 11 19:19:14 nextcloud systemd[1]: Failed to start Advanced key-value store.

systemctl start redis-server
Job for redis-server.service failed because the control process exited with error code.
See "systemctl status redis-server.service" and "journalctl -xe" for details.

But he interesting part is in the syslog, not the journal

Mar 19 11:21:22 nextcloud systemd[1]: Starting Advanced key-value store...
Mar 19 11:21:22 nextcloud systemd[34831]: redis-server.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc: Permission denied
Mar 19 11:21:22 nextcloud systemd[34831]: redis-server.service: Failed at step NAMESPACE spawning /usr/bin/redis-server: Permission denied
Mar 19 11:21:22 nextcloud systemd[1]: redis-server.service: Main process exited, code=exited, status=226/NAMESPACE
Mar 19 11:21:22 nextcloud systemd[1]: redis-server.service: Failed with result 'exit-code'.
Mar 19 11:21:22 nextcloud systemd[1]: Failed to start Advanced key-value store.
Mar 19 11:21:23 nextcloud systemd[1]: redis-server.service: Scheduled restart job, restart counter is at 1.
Mar 19 11:21:23 nextcloud systemd[1]: Stopped Advanced key-value store.
Mar 19 11:21:23 nextcloud systemd[1]: Starting Advanced key-value store...
Mar 19 11:21:23 nextcloud systemd[34834]: redis-server.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc: Permission denied
Mar 19 11:21:23 nextcloud systemd[34834]: redis-server.service: Failed at step NAMESPACE spawning /usr/bin/redis-server: Permission denied
Mar 19 11:21:23 nextcloud systemd[1]: redis-server.service: Main process exited, code=exited, status=226/NAMESPACE
Mar 19 11:21:23 nextcloud systemd[1]: redis-server.service: Failed with result 'exit-code'.
Mar 19 11:21:23 nextcloud systemd[1]: Failed to start Advanced key-value store.
Mar 19 11:21:23 nextcloud systemd[1]: redis-server.service: Scheduled restart job, restart counter is at 2.
Mar 19 11:21:23 nextcloud systemd[1]: Stopped Advanced key-value store.
Mar 19 11:21:23 nextcloud systemd[1]: Starting Advanced key-value store...
Mar 19 11:21:23 nextcloud systemd[34837]: redis-server.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc: Permission denied
Mar 19 11:21:23 nextcloud systemd[34837]: redis-server.service: Failed at step NAMESPACE spawning /usr/bin/redis-server: Permission denied
Mar 19 11:21:23 nextcloud systemd[1]: redis-server.service: Main process exited, code=exited, status=226/NAMESPACE
Mar 19 11:21:23 nextcloud systemd[1]: redis-server.service: Failed with result 'exit-code'.
Mar 19 11:21:23 nextcloud systemd[1]: Failed to start Advanced key-value store.
Mar 19 11:21:23 nextcloud systemd[1]: redis-server.service: Scheduled restart job, restart counter is at 3.
Mar 19 11:21:23 nextcloud systemd[1]: Stopped Advanced key-value store.
Mar 19 11:21:23 nextcloud systemd[1]: Starting Advanced key-value store...
Mar 19 11:21:23 nextcloud systemd[34840]: redis-server.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc: Permission denied
Mar 19 11:21:23 nextcloud systemd[34840]: redis-server.service: Failed at step NAMESPACE spawning /usr/bin/redis-server: Permission denied
Mar 19 11:21:23 nextcloud systemd[1]: redis-server.service: Main process exited, code=exited, status=226/NAMESPACE
Mar 19 11:21:23 nextcloud systemd[1]: redis-server.service: Failed with result 'exit-code'.
Mar 19 11:21:23 nextcloud systemd[1]: Failed to start Advanced key-value store.
Mar 19 11:21:23 nextcloud systemd[1]: redis-server.service: Scheduled restart job, restart counter is at 4.
Mar 19 11:21:23 nextcloud systemd[1]: Stopped Advanced key-value store.
Mar 19 11:21:23 nextcloud systemd[1]: Starting Advanced key-value store...
Mar 19 11:21:23 nextcloud systemd[34843]: redis-server.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc: Permission denied
Mar 19 11:21:23 nextcloud systemd[34843]: redis-server.service: Failed at step NAMESPACE spawning /usr/bin/redis-server: Permission denied
Mar 19 11:21:23 nextcloud systemd[1]: redis-server.service: Main process exited, code=exited, status=226/NAMESPACE
Mar 19 11:21:23 nextcloud systemd[1]: redis-server.service: Failed with result 'exit-code'.
Mar 19 11:21:23 nextcloud systemd[1]: Failed to start Advanced key-value store.
Mar 19 11:21:24 nextcloud systemd[1]: redis-server.service: Scheduled restart job, restart counter is at 5.
Mar 19 11:21:24 nextcloud systemd[1]: Stopped Advanced key-value store.
Mar 19 11:21:24 nextcloud systemd[1]: redis-server.service: Start request repeated too quickly.
Mar 19 11:21:24 nextcloud systemd[1]: redis-server.service: Failed with result 'exit-code'.
Mar 19 11:21:24 nextcloud systemd[1]: Failed to start Advanced key-value store.

 

 

I can't be sure but I have been reading somewhere that

systemd[34831]: redis-server.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc: Permission denied

Is because apparmor is set too tight on the host or that the container is not configured to work around the expectations of the host.

I'll do some more research

wow,

I found that the nesting feature was turned off again which explains whay I have gotten these apparmor related errors again.

What I do not understand is why that happend all of a sudden a day or 2 after things were running fine.

But that could have been me and that I simply forgot doing that. Unlikely but it is a possibility.

Anyway I am really not happy needing nesting for a container. There seems to be a solution available but I lack the skills to see what they mean.

please see

https://github.com/lxc/lxc/issues/4127

Is that anything we could put into the next turnkey template to avoid the needs for nesting?

Just checked again on a vanila debian container I installed as a test.

syslog

systemd[2978]: logrotate.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc: Permission denied
Mar 20 00:00:45 debian systemd[2978]: logrotate.service: Failed at step NAMESPACE spawning /usr/sbin/logrotate: Permission denied

 

The same things happens there aswell.

 

So we are barking up the wrong tree here.

I will take this up with the lxc people first

Jeremy Davis's picture

I agree that the issue is not TurnKey specific, but whether the core issue is in LXC, AppArmor or systemd depends who you listen to. The systemd dev is adamant that it's a LXC and/or Apparmor issue. The LXC devs have suggested that the issue is the way that systemd implements cgroups. And I'm not sure what the AppArmor devs think.

Like I think I noted earlier, the core issue appears to be the Debian security systemd service hardening and/or AppArmor that causes the issue. If you relax the service file hardening, then it's highly likely that it will "just work". So like I said previously, the tradeoff is to reduce the hardening of the guest (i.e. make the guest less secure) or reduce the isolation of the guest via allowing "nested" containerization (i.e potentially make the host less secure).

My personal take is that if you control both the host and the guest, so long as you ensure that both your host and your guest are kept up to date and the guest is as locked down as possible, that the reduced isolation of the guest (i.e. via enabling nesting) is the better of the 2 evils. It's also the more convenient option as you don't need to manually make changes to the Debian default files. If you only control the host (and others control the guest) then compromising the security of the guest would be the better path. That maximizes the security of the host but offloads the hassle of tweaking services to whoever is maintaining the guest. Obviously which path you take is up to you.

If you do want to have a go at winding back the guest's security hardening, then you can do that by copying the default systemd service file and editing it. The service hardening options that you will need to disable are things like:

NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
DevicePolicy=closed
ProtectSystem=strict
ProtectHome=read-only
ProtectControlGroups=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
MemoryDenyWriteExecute=yes
LockPersonality=yes

That list isn't exhaustive, but should cover the most common/likely options. As I noted, all those options are security hardening, but they relay on system capabilities that an unnested and unprivileged container doesn't have access to (hence the failures).

The way to change them is to either use 'systemctl edit SERVICE_NAME' or copy the original service file (from /usr/lib/systemd/system) to /etc/systemd/system. The reason for that is that the original service files (in /usr/lib) are handled by package management. Edited original files will be overwritten (with the original file) next time you update the package. Files in /etc/systemd/system are user managed and will (generally) not be changed by package updates (if they are, you will be asked if you want to accept the changes or keep your existing files) and will override the same name files in /usr/lib/systemd/system.

After making changes to service files, you'll need to reload systemd like this:

systemctl daemon-reload

Thank you Jeremy,

It's a good thing I control both the host and the container.

The top priority is keeping the host secure and the container can be relaxed a the least amount it needs to stay functional ;)

So for now I will use nesting then.

In case I find a solution that has no tradeoff's and requires no tinkering I will let you know.

Jeremy Davis's picture

I wouldn't hold your breath for a solution without tradeoffs. One of the fundamental aspects of LXC (and Docker for that matter) is that it leverages the host system at a fairly low level. That means that by definition, it either is inextricably linked to the hosts resources (more-or-less like a process running on the host), or it is isolated and has limited capabilities. You can trade between these 2 by varying degrees, but there will always be some sort of trade-off. It's possible that I'm missing something, but I can not see how it can be any other way.

That's security in general really. the tighter the security screws, the more things won't "just work". Security is always a trade-off (usually against usability) and I don't see how LXC (and/or Docker) will ever be able to surmount that.

If you want servers that are (more) isolated from the host, then use a "proper" VM (where the tradeoff is resource usage).

Roy Cordero's picture

I just installed my nginx and everything is fine but to use it from the internet I have my subdomain with cloudflare and with my nginx I point that domain and direct it to the ip of the service, but I don't know what port to use, What port should I point it to? I tried 80 and it doesn't work, then in config.php I add the ip with the port '10.100.32.9:80' and it doesn't work either... does anyone know how to get this nextcloud from turnkeycore by nginx?

 

Jeremy Davis's picture

If you set the Nextcloud "domain" to be the domain that you intend to access it with and ensure that the domain points to the IP address of your server, then it should "just work". You shouldn't need a port, but if you do need to specify one, then port 443 is recommended (HTTPS). Port 80 in the other option but is only plain HTTP, so not recommended these days.

Add new comment