L. Arnold's picture
L. Arnold - Fri, 2023/09/22 - 03:41

Hello TKL Team!

Unfortunately my LetsEncrypt stopped working and I can't migrate (first try anyway) my Joomla 3 Server running TKL 15.5 - Stretch.

I mainly want to get Let's encrypt going again on this Joomla Server./

Since I am asking also need to move TLS on a TKL Odoo 14.1 server to support TLS 1.2 or 1.3.  Google and Firefox are rejecting https on this server.  This needs to run Odoo 8 so stuck with TKL 14.

Sorry for 2 questions.  One project on my side.  Working through TKLDEV or some manual migrations otherwise will be needed I expect.

Thank you for your help!

Forum: 
Support
Jeremy Davis's picture

Jeremy Davis - Tue, 2023/09/26 - 06:07

You have a few options (in no particular order):

  • Migrate data (should work ok with joomla, odoo will be trickier)
  • Update dehydrated - may resolve the issue if it's purely to do with ACME protocol; won't work if the underlying cause is OpenSSL too old
  • Put your server(s) behind a reverse proxy - i.e. do HTTPS termination on a newer server and reverse proxy traffic to your existing server(s)
  • run a new server purely to get LE certs - and sync certs with a cron job

Migrate Joomla data

Migrating your Joomla with TKLBAM should work ok, especially if you just restore the joomla files and DB. The same can't be said of Odoo, as that requires a specific version of python. But for Joomla, try this on a new Joomla3 server (where BACKUP_ID is the actual backup ID number of your Joomla3 backup):

mkdir -p /tklbam-dump
tklbam-restore BACKUP_ID --raw-download=/tklbam-dump
tklbam-restore /tklbam-dump --skip-packages --limits="/var/www/joomla3 mysql:joomla3"

That will restore just your Joomla data dir and DB. To reset the Joomla DB password (so Joomla can connect to it's DB) just re-run the firstboot script:

/usr/lib/inithooks/firstboot.d/20regen-joomla-secrets

If you want/need other data/files, you'll find them in /tklbam-dump - relative to /. I.e. /etc/some/file will be found at /tklbam-dump/etc/some/file. Note though that manually moving files will change permissions (to the user who moves the files - probably root). To keep permissions consistent with their original source, use tklbam with limits (e.g. 'tklbam-restore /tklbam-dump --skip-packages --limits="/file/to/restore /dir/to/restore/"').

Update Dehydrated

I'd need to know more about the failure to be sure that this would work, but in theory, it should be fairly easy to update Dehydrated. There is a possibility that the failure is caused by an old version of OpenSSL (rather than Dehydrated). If that's the case, then this won't work. But if the issue is the old version of Dehydrated, then this will work.

I have a suspicion that removing the current dehydrated might break things, so instead just rename the executable (this is generally considered bad practice, but you gotta do what you gotta do sometimes and seeing as Stretch won't have any updates it's fairly safe). So (as root):

mv /usr/bin/dehydrated /usr/bin/dehydrated.old

Then install the latest (Dehydrated is a bash script, so this is not dangerous either):

cd /usr/local/src
git clone https://github.com/dehydrated-io/dehydrated.git
ln -s /usr/local/src/dehydrated/dehydrated /usr/local/bin/dehydrated

Then double check that it's set up properly:

which dehydrated

It should return (note the local dir in the path):

/usr/local/bin/dehydrated

If it still points to /usr/bin/dehydrated (note no 'local' in the path) then something isn't right.

Now update the "hook script":

url=https://raw.githubusercontent.com/turnkeylinux/confconsole/master/share/letsencrypt/dehydrated-confconsole.hook.sh
file=/etc/dehydrated/confconsole.hook.sh
wget -O $file $url

Also update the config file (add new variable; CHALLENGETYPE):

echo "CHALLENGETYPE='http-01'" >> /etc/dehydrated/confconsole.config

If it's failing because of the old Dehydrated, it should now work. If it still fails, then my guess is that it needs newer OpenSSL - but that's a whole lot harder to do...

Other options

I have to dash now but I'll try to circle back to this soon and give detail on the other options. If you notice I haven't responded in a few days time, please feel free to bump me. Also please share how you go with either of the above.

L. Arnold's picture

L. Arnold - Wed, 2023/09/27 - 05:04

So I will go through and use your system above.  Thank you for that.

I thought I would try first on a simple TKLBAM Restore # server which is not starting.

Error tells me that I should also try "turnkey-init" to get MySQL password working post restore.  I did just do that on another RAW system which at least got me into that one.
(but yes, I will  try your very good instruction above.  Seems the best path)

 

root@joomla3 ~# /usr/lib/inithooks/firstboot.d/20regen-joomla-secrets
Traceback (most recent call last):
	File "/usr/lib/inithooks/bin/mysqlconf.py", line 145, in <module>
		main()
	File "/usr/lib/inithooks/bin/mysqlconf.py", line 124, in main
		m = MySQL()
	File "/usr/lib/inithooks/bin/mysqlconf.py", line 41, in __init__
		self._start()
	File "/usr/lib/inithooks/bin/mysqlconf.py", line 64, in _start
		raise Error("could not start mysqld")
	__main__.Error: could not start mysqld

Followup:
Still won't start at least the web after Turnkey-Init

Also the Same error on Joomla Secrets Regen.

I will follow your step by step on a new server.  
Thank you for the guidance.  notes to follow

L. Arnold's picture

L. Arnold - Wed, 2023/09/27 - 05:42

The Table Restore brings error:

table: mysql/tables_priv
table: mysql/index_stats
table: mysql/db
table: mysql/innodb_index_stats
table: mysql/time_zone_name
table: mysql/time_zone_leap_second
table: mysql/event
table: mysql/proc
table: mysql/time_zone_transition
table: mysql/proxies_priv
table: mysql/slow_log
table: mysql/time_zone_transition_type
table: mysql/table_stats
table: mysql/help_category
table: mysql/help_topic
table: mysql/columns_priv
table: mysql/ndb_binlog_index
table: mysql/gtid_slave_pos
table: mysql/roles_mapping
table: mysql/plugin
table: mysql/func
table: mysql/host
table: mysql/innodb_table_stats
table: mysql/general_log
table: mysql/help_relation
table: mysql/time_zone
table: mysql/column_stats
table: mysql/user
table: mysql/servers
table: mysql/procs_priv
table: mysql/help_keyword
ERROR 1050 (42S01) at line 6219: Table 'user' already exists We're done. You may want to reboot now to reload all service configurations.

 

Not sure if that is ok but hey.
More to follow.

L. Arnold's picture

L. Arnold - Wed, 2023/09/27 - 06:04

The Web Interface gave me the same "Error"  (only that)

I went into Webmin and checked MariaDB Server

Had to provide "Adminer + Password"  and got in.
I see these 4 Database:

information_schema

mysql

performance_schema

sites_joomla

When I look at "Table Data", then Users.
I see my email as Super User.  Then a user and a Manager.
Does not appear that more data came in. 

---

When I look at Apache (I had a rather complicated Apache having the old Server host perhaps 6 different domains).  I don't think we restored that in the process above.

In apache,  yes, we are serving Joomla defaults which is "ok".  (My apache was quite complicated)...

-----

Just Browsed the var/www/joomla directory and I am not seeing any of the restore.
I think the Restore data is in some parrallel directories.

I see tklbam-dump  (fair bit of stuff there) (etc, root, TKLBAM, usr, var)

and tklbam-dumptklbam-restore  (which is Empty)

----------

Let me work on that restore process again.  I don't think it hit the live data on the server.  Basically made it into the TKLBAM folder but did not then get out (theories anyway)
 

Jeremy Davis's picture

Jeremy Davis - Fri, 2023/10/06 - 05:30

Hey Landis, apologies on the slow response, as usual I have way too many plates spinning... :)

Regardless, if you are ok to share your data with me, then I would be happy to have a try. I'm really trying to push the v18.0 release out as fast as we can, but you're a long term TurnKey community member and supporter of TurnKey, so it would be severely remiss of me to not offer you a bit of extra assistance! :)

Here's how to gather your latest backup into a password protected archive. Note that it requires CLI (feel free to use some other method if you prefer, but this should "just work"):

mkdir -p /tklbam-joomla3-dump
# change BACKUP_ID below for actual backup ID number
tklbam-restore BACKUP_ID --raw-download=/tklbam-joomla3-dump
tar czf - /tklbam-joomla3-dump | gpg -o landis-joomla3.tgz.gpg --symmetric

That should ask you to enter a password, and will result in an encrypted archive file of your backup data called landis-joomla3.tgz.gpg. You'll then need to download the file from your server. My guess is that it will be fairly large (probably too big for email) so you'll probably have to upload it somewhere. I notice you have a gmail address, so perhaps upload to Google drive and send me a link? Please note that my turnkey email is not linked to google, so better to use my personal email (which is linked to google - jeremy AT jeremydavis.org). Email the password to my TurnKey email. Alternatively I'll email you my phone number and you can send it to me via What's App or Signal text message.

Alternatively, if you'd rather me log into your server and have a look, please add my public key to your server like this (it's a REALLY long, single line):

echo "ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAABAQCvBup9/16HsrT4a+WxkjNScbYCueCJVCkxHtLRnUEOYH7HTQfybmy4JSRkPiyS4ig7Jk9lAoFLiLtEfI4l93iQsr/gp3SAjKftxso5Wf/BtOVinkzue66tq+4EZcji+cezg3JfqKz2NlCnmcFh3/8ohTk+i8EM10fzaN41cYGL4/x/G0I2wA1bfPR4dZe2ybUIoHzfzYJzOqgv9knNIxXBACQyXiN801hNdKhq7Snk8kBrc+IxoLZIvqTYXIFeSDLSn5L9n8plLQnhWhEg72cUyoJGBHofy+RGpHcaKBnuEXtLrXpqzpg23X+L2zwry9a/GMeMS9u+sXB2PD5fym4N jeremy@turnkeylinux.org" >> /root/.ssh/authorized_keys

Then email or text me the public IP address.

Jeremy Davis's picture

Jeremy Davis - Wed, 2023/10/11 - 06:49

Ok, fingers crossed, I think I have a fix!

FWIW it appears that the issue is related to this MariaDB bug. FWIW TKL v16.x had MariaDB v10.3; TKL v17.x has MariaDB v10.5.x - so restoring backups from v16.x or earlier (that include a MySQL/MariaDB database) will fail when restoring to v17.0 or newer.

To work around the issue, I've created a TKLBAM hook script - TBH, because it affects about 70% of the library (any app backup that includes MySQL/MariaDB from v16.x or prior, being restored to a v17.0 or newer) I'm not 100% sure where it should go. But for now I've created a new repo in my personal GitHub account. For full details (including how to get it on your server) please see the Readme.

Please note that the script could do with some tidying, but should be fully functional. I'd appreciate any feedback you have (good, bad or ugly).

I'm not sure, but perhaps it'd be a good idea to try something similar for your Odoo restore. I.e. bundle up your backup and send it through so I can have a play with it?

L. Arnold's picture

L. Arnold - Mon, 2023/10/02 - 17:32

Strange,  I have been able to get a Odoo 8 TKL 14.2 Server fired up on Oracle Cloud

Trying to migrate a Linode Hosted version of the same.  All works but the main data file.

The "Old" Linode Hosted version is Postgresql 9.4.26

the "New" Oracle Hosted version is Postgresql 9.4.12

I get database, wrong version, in TKLBAM.
I just get failure in Odoo Database Manager

and I can't seem to upgrade Postrgresql within Shell.

I know I know,  move forward.  Really just trying to archive this.  Might just do a data extract and move on.

Jeremy Davis's picture

Jeremy Davis - Fri, 2023/10/06 - 06:48

Hmm, that seems weird!? Are you using default PostrgreSQL? (from Debian apt repo?)

Assuming that you are, I suggest that you double check the package version via apt:

apt-cache policy postgresql

Run that on both servers and compare. My guess is that there is an update (to at least 9.4.26) available in the repos.

If you upgrade (install) postgresql via apt, hopefully that should resolve it. I.e. (on the new Oracle server):

apt-get update
apt-get install postgresql -y
Jeremy Davis's picture

Jeremy Davis - Mon, 2023/10/09 - 10:19

Ok, so I've had a closer look at this Odoo v14.2 one and I think I have a workaround. The issue is that the Debian Jessie repos no longer exist. A workaround is to use the Freeaxian ELTS (Extended Long Term Support - Freeaxian are a company that provide Debian based services). If you are doing anything important with this server, I strongly recommend that you consider subscribing to Freeaxian. Whilst it's not required (the updates they provide are provided free to anyone), if you are a paying subscriber, they will support the packages that you are using.

To enable the Freeaxian Jessie repos:

# download the key
apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com A07310D369055D5A

# write the sources list for Freeaxian ELTS
cat > /etc/apt/sources.list.d/freeaxian-elts.list <<EOF
deb http://deb.freexian.com/extended-lts jessie main contrib non-free
deb http://deb.freexian.com/extended-lts jessie-lts main contrib non-free
EOF

# disable old repos
mv /etc/apt/sources.list.d/sources.list /etc/apt/sources.list.d/sources.list.disabled
mv /etc/apt/sources.list.d/security.sources.list /etc/apt/sources.list.d/security.sources.list.disabled
mv /etc/apt/sources.list.d/backports.sources.list /etc/apt/sources.list.d/backports.sources.list.disabled

# run apt update
apt-get update

# install postgres
apt-get install -y postgresql

I'm not 100% sure that will be enough, but it should definitely get you to a point where you have the same version of PostgreSQL on both v14.2 servers. FWIW here's my Odoo v14.2 server after I ran the above:

# apt-cache policy postgresql
postgresql:
  Installed: 9.4+165+deb8u4
  Candidate: 9.4+165+deb8u4
  Version table:
 *** 9.4+165+deb8u4 0
        500 http://deb.freexian.com/extended-lts/ jessie/main amd64 Packages
        500 http://deb.freexian.com/extended-lts/ jessie-lts/main amd64 Packages
        100 /var/lib/dpkg/status
L. Arnold's picture

L. Arnold - Mon, 2023/10/09 - 20:50

that is cool Jeremy.  Nice work with the Freeaxian.

I got a small error and I am not sure If I got all the way to Postgresql 9.4.26  (ancient history I know) which seems to be my hangup.

-------

I got this error at the end of several freeaxian updates....

apt-get update

W: Failed to fetch http://http.debian.net/debian/dists/jessie-backports/main/binary-amd64/Packages  404  Not Found [IP: 151.101.26.132 80]

which then led to
E: Some index files failed to download. They have been ignored, or old ones used instead.

---

was a typo in the "policy" line ("r" instead of "t" in "postgresql

but I seem t o be in the same place as you now..  Not sure if that is fully 9.4.26 - I am readin 165


 

root@odoo ~# apt-cache policy postgresql

postgresql:

Installed: 9.4+165+deb8u4

Candidate: 9.4+165+deb8u4

Version table:

*** 9.4+165+deb8u4 0

        500 http://deb.freexian.com/extended-lts/ jessie/main amd64 Packages                                                                                            

        500 http://deb.freexian.com/extended-lts/ jessie-lts/main amd64 Packages                                                                                        

100 /var/lib/dpkg/status

root@odoo ~#

 

However.  the old server,  though "Webmin" is telling me  version "9.4.26" in Postgresql

When I go to Shell,  this is my Log of that last set


 

root@odoo ~# apt-cache policy postgresql

postgresql:

Installed: 9.4+165+deb8u4

Candidate: 9.4+165+deb8u4

Version table:

*** 9.4+165+deb8u4 0

        500 http://security.debian.org/ jessie/updates/main amd64 Packages                                                            

100 /var/lib/dpkg/status

9.4+165+deb8u3 0

        500 http://http.debian.net/debian/ jessie/main amd64 Packages   

 

When I go into Webmin it is still showing "9.4.12"  (not 9.4.26, as on the other server) but I don't really trust Webmin to be reading the situation.  Perhaps that is a "Webmin rendering of PostgreSQL".  not sure.

I will try to move the database over with the Odoo Database tool, which is where I was getting the wrong version file error in the past.

----

Tried that again through Odoo, but just got a vague statement that the database cannot be restored

The old server so the database should be transportable..  Alas not just now.

----

Just logged in with Adminer..
It looks like the "Posggresql version we are looking at is saying:

PostgreSQL version: 9.4.26 through PHP extension PgSQL

Anyway,

Perhaps there is a way, from either the Shell or Adminer to do a Database Dump then, on the new Oracle Server bring it in in such a way that Odoo does not reject it.  I know this is an Odoo issue as much as anything.  

Let me see what I can find.

 

L. Arnold's picture

L. Arnold - Wed, 2023/10/11 - 17:25

It is interesting that the 2 issues here (sorry to have 2 issues in one thread) are different but related.

Odoo is running Postgresql

Joomla is running MariaSQL

both, with different generations of TKL have different database versions.  Both have database migration issues owing to the database generations.

Joomla3:  Anyway,  running now on a new hub.joomla server.

Complications arose from both Linode and the Hub requiring different password formats..  hopefully I can get through any differential and use turnkey-init later to get things syncronized.  Mainly difficult keeping track of these things through generations and the fact that TKLBAM itself has a tendency to reset some of them.

speaking of which,  I just got the following error:


HOOK /etc/tklbam/hooks.d/maria_db-changes :: op=restore state=post pwd=/root ::
TKLBAM_RESTORE_PROFILE_ID: turnkey-joomla3-15.5-stretch-amd64
[maria_db-changes] INFO: hook invoked after backup restore. Extras path = /root
[maria_db-changes] INFO: Restore of MySQL/MariaDB database detected; ensuring th                                                    at MySQL/MariaDB is running properly
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run                                                    /mysqld/mysqld.sock' (2)
2023-10-11  8:20:14 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-0+deb11u                                                    1) starting as process 4330 ...
FATAL: /var/run/mysqld/mysqld.sock not found after waiting 11 seconds
Traceback (most recent call last):
  File "/usr/bin/tklbam", line 46, in <module>
    CliWrapper.main()
  File "/usr/lib/tklbam/cliwrapper.py", line 88, in main
    commands[command].main()
  File "/usr/lib/tklbam/cmd_restore.py", line 533, in main
    hooks.restore.post()
  File "/usr/lib/tklbam/hooks.py", line 81, in post
    self._run("post")
  File "/usr/lib/tklbam/hooks.py", line 73, in _run
    _run_hooks(self.LOCAL_HOOKS, (self.name, state))
  File "/usr/lib/tklbam/hooks.py", line 42, in _run_hooks
    (fpath, " ".join(args), e.exitcode))
hooks.HookError: `/etc/tklbam/hooks.d/maria_db-changes restore post` non-zero ex                                                    itcode (1)
 

So...  Earlier before a crash on the previous server, I saw some "user updates" right out of the box.  This time it seemed to come at the end...

Anyway, I am sure we are getting close..
Sorry for the trouble.

Landis

 

Jeremy Davis's picture

Jeremy Davis - Thu, 2023/10/12 - 01:00

Looking at your output, it looks like MySQL/MariaDB isn't/wasn't running. You could try to start the service like this:

systemctl restart mariadb

If this is an existing server that you've already tried, and the systemctl command fails, then my guess is that MySQL/MariaDB has already been broken. It's fixable, but in my recent experience, it's a PITA. One thing that you could try that isn't too hard, is wiping out the current DBs (back to Debian default) and try again? This should do it:

systemctl stop mariadb
pkill mariadb # just in case an orphan process is running
mv /var/lib/mysql /var/lib/mysql.bak
mkdir /var/lib/mysql
chown mysql:mysql /var/lib/mysql
mariadb-install-db
systemctl start mariadb

If that doesn't fix MySQL/MariaDB and it's still not running, in scenarios like this, it's often quicker and easier to start again from scratch.

I recall when we chatted, you were having troubles getting it running locally on VirtualBox. For this sort of stuff a local install will likely make your feedback loop tighter, so perhaps it's worth exploring what the issue is with that not working? Then you can continue your testing and trial and error work locally!?

L. Arnold's picture

L. Arnold - Wed, 2023/12/13 - 20:56

So I got the same error to a new server, after having run the MariaDB Bugfix Update you posted in another thread. https://github.com/JedMeister/tklbam-restore-mariadb-bugfix

From Joomla 3 TKL15.4
tklbam restore (164)
 

Got to the same point again and errored out as above:
Basically Maria.db not running.


HOOK /etc/tklbam/hooks.d/maria_db-changes :: op=restore state=post pwd=/root ::
TKLBAM_RESTORE_PROFILE_ID: turnkey-joomla3-15.5-stretch-amd64
[maria_db-changes] INFO: hook invoked after backup restore. Extras path = /root
[maria_db-changes] INFO: Restore of MySQL/MariaDB database detected; ensuring th                                   at MySQL/MariaDB is running properly
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run                                   /mysqld/mysqld.sock' (2)
2023-12-13 11:38:56 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-0+deb11u                                   1) starting as process 3054 ...
FATAL: /var/run/mysqld/mysqld.sock not found after waiting 11 seconds
Traceback (most recent call last):
  File "/usr/bin/tklbam", line 46, in <module>
    CliWrapper.main()
  File "/usr/lib/tklbam/cliwrapper.py", line 88, in main
    commands[command].main()
  File "/usr/lib/tklbam/cmd_restore.py", line 533, in main
    hooks.restore.post()
  File "/usr/lib/tklbam/hooks.py", line 81, in post
    self._run("post")
  File "/usr/lib/tklbam/hooks.py", line 73, in _run
    _run_hooks(self.LOCAL_HOOKS, (self.name, state))
  File "/usr/lib/tklbam/hooks.py", line 42, in _run_hooks
    (fpath, " ".join(args), e.exitcode))
hooks.HookError: `/etc/tklbam/hooks.d/maria_db-changes restore post` non-zero ex                                   itcode (1)
 

Then, following that I ran:
systemctl restart mariadb

got: 

root@joomla3 ~# 
Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.
 

I am thinking perhaps the MariaDB Password.  Perhaps I can envoke
turnkey init to reset passwords, then restart..  Not sure.

Anyway, I can see you have gotten this to work for others.  I appreciate the efforts for sure Jeremy!
 

Jeremy Davis's picture

Jeremy Davis - Thu, 2023/12/14 - 04:43

Was this on a freshly launched, clean server? I.e. without any previous data or previous attempted data migration/restore. My guess is that it wasn't, but it'd be good to know.

Also, did you install the new package I built or the manual way (as detailed in the link your posted)? FWIW either should work, the main difference is that that rebuilt TKLBAM includes the fix already; plus a PostgreSQL fix as well (which I'm pretty sure you originally reported too - so thanks again for that).

It seems clear from the error message you've posted that MySQL/MariaDB wasn't running when you tried the restore. I'm not clear on why that is, but considering the additional info you've provided, my guess is that you had already been doing stuff on this server before this attempt and whatever is/was wrong with MySQL/MariaDB was an issue prior to trying this restore.

So I suggest trying your restore again on a clean instance. Here is a suggested workflow:

  • Start a new instance of the desired size/type
  • Install the updated/bugfixed TKLBAM (from here)
  • Double check that MySQL/MariaDB is running and all is well, e.g. 'systemctl status mariadb'
  • Run the restore and fingers crossed it should "just work"

If it doesn't, then let's speak more and I'll get a complete copy of your backup data and try again myself.

L. Arnold's picture

L. Arnold - Wed, 2023/12/13 - 21:11

I ended up running 

turnkey-init
that gave me root password I could access the system with.

TKLBAM also reset my "cert" login which I need to find which one I had aiming at the system.
I can see most of my settings went across (as I did not do a limited tklbam which perhaps I should of).  My goal with this is getting let's encrypt running (and getting the whole system running on 17.1 seems a good idea)

But it did not get Maria Running again.  I have verified also after restart, as well as the Webmin restart server.

It seems I am close.  Need to jumpstart MariaDB it would appear.

Jeremy Davis's picture

Jeremy Davis - Thu, 2023/12/14 - 04:46

Yeah, we need to work out what the MariaDB issue is.

As I noted in my previous post, my suspicion is that whatever is wrong with it, was already an issue before this restore. But perhaps there is something else I'm overlooking?

L. Arnold's picture

L. Arnold - Fri, 2023/12/15 - 18:53

I got this error:

MariaDB error message The full MariaDB error message was : connect to server at 'localhost' failed error: 'Can't connect to local MySQL server through socket '/run/mysqld/mysqld.sock' (2)' Check that mysqld is running and that the socket: '/run/mysqld/mysqld.sock' exists!
 

and when I look, there is no file there named mysqld.sock

Looking in the old server the mysqld.sock file is there along with mysqld.pid > PID file basically shows "814"  (likely the process ID #)
the .sock file I am not able to "edit" in Webmin.  When I download it, it is completely empty when I open in notepad++.

Anyway, I tried to simply upload those 2 files to the new server but Mysql (maria) is still not starting.
I am looking next at what is there on a Fresh Install (no tklbam) of Joomla3. (turnkey-init again required to access after hub rollout)

mysqld.pid

I see the PID # is different : 624.

mysqld.sock  appears empty

both however show different ownership than the old system:  mysql:root  in the new and mysql:mysql in the old  (from my file upload).  Changed to mysql:mysql on the Restored server.
Maria still not starting:

One batch of feedback:

root@joomla3 ~# systemctl status mariadb.service
* mariadb.service - MariaDB 10.5.15 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2023-12-15 09:47:14 MST; 1min 11s ago
Docs: man:mariadbd(8) https://mariadb.com/kb/en/library/systemd/
Process: 15791
ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Process: 15792
ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 15794
ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] &>
Process: 15964
ExecStart=/usr/sbin/mariadbd $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=7)
Main PID: 15964 (code=exited, status=7)
Status: "MariaDB server is down" CPU: 160ms
Dec 15 09:47:13 joomla3 systemd[1]: Starting MariaDB 10.5.15 database server...
Dec 15 09:47:14 joomla3 mariadbd[15964]: 2023-12-15 9:47:14 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-0+deb11u1) starting as process 15964 ...
Dec 15 09:47:14 joomla3 systemd[1]: mariadb.service: Main process exited, code=exited, status=7/NOTRUNNING
Dec 15 09:47:14 joomla3 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Dec 15 09:47:14 joomla3 systemd[1]: Failed to start MariaDB 10.5.15 database server.

Second batch of feedback might indicate the old SSL certs were restored or simply that I have not set SSL on the new server:

root@joomla3 ~# journalctl -xe
Dec 15 09:49:11 joomla3 stunnel[458]: LOG3[1893]: SSL_accept: ../ssl/record/rec_layer_s3.c:1543: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert c>
Dec 15 09:49:11 joomla3 stunnel[458]: LOG5[1893]: Connection reset: 0 byte(s) sent to TLS, 0 byte(s) sent to socket
Jeremy Davis's picture

Jeremy Davis - Sun, 2023/12/17 - 23:33

The sock and pid files should be automatically created when MariaDB starts. If it's running (properly), they should be there. If they're not there, then the service isn't running properly.

As you guessed, a pid file is a file that contains a process's ID number (that's what a PID is - a Process ID). PID file were required under the old init system (called sysvinit), so that the init system knew which process to kill when it was time to stop. The current init system; systemd can track pids itself, so generally pid files are no longer needed. However, they are still used sometimes. Without checking on a server of the same release that is working as it should, I can't tell you whether or not MariaDB needs a PID file. But as a general rule these days, unless there are log messages complaining about a missing PID file, just because one doesn't exist doesn't necessarily mean things are bad.

It's no surprise that you couldn't do anything with the socket file in Webmin. Socket files aren't a normal file. They're a special file that is used to communicate between processes. In the case of MariaDB, it's how the client talks to the server. The socket file is created by the service when it starts.

Look at the last 3 lines from the output you posted for 'systemctl status mariadb.service':

Dec 15 09:47:14 joomla3 systemd[1]: mariadb.service: Main process exited, code=exited, status=7/NOTRUNNING
Dec 15 09:47:14 joomla3 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Dec 15 09:47:14 joomla3 systemd[1]: Failed to start MariaDB 10.5.15 database server.

So for some reason, MariaDB is failing to start. Unfortunately there isn't enough information there to know why, but there is definitely something wrong with MariaDB. My guess is that your restore was interrupted at some point and the DB changes (that I scripted) have only been partially completed and left your server in a broken state, halfway between the default and your DB backup.

The snippet you showed from the journal is irrelevant here as that is stunnel stuff - not related to the MariaDB issues. If we just look in the journal for MariaDB related messages, we might get more insight into the MariaDB issues:

journalctl -b -u mariadb.service

I wonder if there is something in your MySQL/MariaDB config that is upsetting MariaDB? (Perhaps something from a restored config?). Hopefully the above journalctl command might show it?

Another thing that might be the issue (if there is no clear cause in the error messages) is that during the DB migration (that hook script I wrote) MariaDB is killed, then started manually (i.e. without using the service - it's required to update the permissions). That process should in turn be killed later and the service restarted, but perhaps killing it failed for some reason and it's still running in the background, causing issues?

Rebooting should resolve that, or alternatively try killing any/all MariaDB/MySQL processes (then try starting the service again):

pkill mariadb
pkill mysql

In summary, the issue you are hitting is that MariaDB is not running/starting. The missing sock file is a symptom of the problem, not the cause. Like I said in my previous post, I suggest a clean start (with a working MariaDB service). This same issue may reoccur, but IMO a clean start and then noting each and every command you run and (ideally capturing all) the output is the only way to diagnose what is causing this issue. It is possible that it's a combination of something specific in your backup and something not working as intended in my hook script.

I'm pretty snowed under at the moment (as per always), but if you can give me a copy of your backup (I'm pretty sure that I detailed how to do that?) I'm happy to try again.

L. Arnold's picture

L. Arnold - Tue, 2023/12/19 - 04:42

Yester day I actually started, but it turns out I ran your "instructions" on the server I was trying to migrate.  Didn't kill it at least.
Basically mixed the "local restore folder (not default) with your mysql update.  

New server now (the last one was new also)..  Let me go over your post (2 posts above).  I might also try the "semi simple process" that I mention in Paragraph 1 here.  

You are right.  The trick is going to be getting mariadb starting.  Must be a way.

Thanks for your notes Jeremy.  Sorry that I only get back to this every few days.

 

L. Arnold's picture

L. Arnold - Tue, 2023/12/19 - 04:59

Started the system up again and the same.  Not killing the system just but shutting it down for now.

On to the new one (maybe two of them).

L. Arnold's picture

L. Arnold - Tue, 2023/12/19 - 18:19 
  1. mkdir -p /tklbam-dump
  2. tklbam-restore BACKUP_ID --raw-download=/tklbam-dump
  3. URL=https://raw.githubusercontent.com/JedMeister/tklbam-restore-mariadb-bugfix/master/maria_db-changes
  4. wget -O /etc/tklbam/hooks.d/maria_db-changes $URL
  5. chmod +x /etc/tklbam/hooks.d/maria_db-changes
  6. (RESTART SERVER.  I don't see if anything was applied so doing a restart.)
    reboot now -h
  7. tklbam-restore /tklbam-dump --skip-packages --limits="/var/www/joomla3 mysql:joomla3"
  8. /usr/lib/inithooks/firstboot.d/20regen-joomla-secrets
  9. reboot now -h

Damn.  Error still.

I get this in Webmin which seems promising.

The full MariaDB error message was : connect to server at 'localhost' failed error: 'Access denied for user 'adminer'@'localhost''

I then ran

  1. turnkey-init
    promising in that I saw the Joomla ICON in the Tab,  but still an error

Same Error as earlier today (but more different than posted earlier):

 Warning!  Webmin needs to know your MariaDB administration login and password in order to manage your database. Please enter your administration username (usually root) and password below.

MariaDB error message

The full MariaDB error message was : connect to server at 'localhost' failed error: 'Access denied for user 'adminer'@'localhost''

 

I tried a few more times with that Login page in Webmin and got in.

  • I am not seeing the Joomla MYSQL Database, but Maria is running.
  • Maybe I will try that restore command again.
  • Proceed slowly...
    progress anyway.

 

Jeremy Davis's picture

Jeremy Davis - Wed, 2023/12/20 - 04:00

Ok, so Adminer password should be pretty easy. Just run this:

/usr/lib/inithooks/bin/mysqlconf.py --user=adminer

And enter your desired password interactively, or if you want to do it non-interactively:

/usr/lib/inithooks/bin/mysqlconf.py --user=adminer --pass="YOUR_AWESOME_PASSWORD"

If you get any error message(s), please post them.

When you say "I am not seeing the Joomla MYSQL Database, but Maria is running." how did you check for the joomla DB? Here's how I would do it from the commandline:

mysql -e "show databases;"

When I first started replying, I started from the start. On re-reading your post, it seems that there has actually been a fair bit of progress and telling you to "start again" probably wasn't that useful or contructive (perhaps it might be, but you seem really close!). So I started again from where you are up. Seeing as I had already written all the below, I decided to keep it. Not sure how much value it has, but here is is anyway:

Firstly I recommend doing this on a fresh, clean server. If you've already tried restoring a backup, you could try winding that back via:

tklbam-restore-rollback

Although please be aware that only one rollback is saved, so if you've run 'tklbam-restore' more than once (without any rollbacks in between) you're probably best off starting again with a clean server.

If you don't want to keep starting from scratch, you could take a snapshot of the server (before doing the restore) and just restore that if you want to rerun.

Here's my suggested workflow:

First install the updated TKLBAM package:

wget https://github.com/turnkeylinux/tklbam/releases/download/v1.4.3.2/tklbam_1.4.3+4+g745ecc2_all.deb
apt install ./tklbam_1.4.3+4+g745ecc2_all.deb

(Using that, you don't need to download the hook script separately)

Then restore your backup. Start downloading it (as you were):

rm -rf /tklbam-dump
mkdir /tklbam-dump
tklbam-restore BACKUP_ID --raw-download=/tklbam-dump

It's probably unnecessary, but considering the issues you've been hitting, double check that MySQL/MariaDB is running:

systemctl status mariadb

Assuming that is running fine, if you want to take a snapshot for use later, now is probably the time to do it. Then actually try the restore:

tklbam-restore /tklbam-dump --skip-packages --limits="/var/www/joomla3 mysql:joomla3"
 | tee ~/tklbam-restore.log

That will leave you with a file called tklbam-restore.log in your home directory. Please feel free to share that with me (either attach to your top post in the thread and mention it; or copy paste the contents, or email it to me at my personal email - you should have it already).

After the restore has run, double check that MariaDB is running:

systemctl status mariadb

If it is, try restarting it and double check it's status:

systemctl restart mariadb
systemctl status mariadb

If it's not, then something has gone wrong - but I'm not sure what... Please share the log output.

Assuming that it is running ok, then re-run the firstboot script:

/usr/lib/inithooks/firstboot.d/20regen-joomla-secrets

Now circle back to the top of this post for the last few bits...

L. Arnold's picture

L. Arnold - Wed, 2023/12/20 - 04:50

Hi Jeremy,

There is a "sites-joomla" db which as far as I can see is the "example" database.  Pretty empty etc.

Looking a bit more..  The old server database is just called "joomla", not "joomla3" as is in the restore command.  So, I will try that first right over what I have.

I expect that the old Joomla config is pointing at that database (joomla).
I do also have a custom named database on the old server but looking at the data it is the "joomla" database being used.  I should look also for the joomla.cfg file (imagining/remembering there is one)...

got an error about "no password"... ?? :

(somehow not able to copy paste here)

Let me give it a spin on a new one.

L. Arnold's picture

L. Arnold - Wed, 2023/12/20 - 05:02 
systemctl status mariadb

Seems I can copy / paste the last line here at a time.
Main last line for that server (not with the full new TKLBAM package on it) reads:


Dec 20 02:53:39 joomla3 systemd[1]: Failed to start MariaDB 10.5.15 database server.

 

I have a screenshot but not seeing how I can post it..  Basically though MariaDB is still wacked.

But we are getting closer.  

L. Arnold's picture

L. Arnold - Wed, 2023/12/20 - 19:46
  1. create new server on hub
  2. try to login via webshell
  3. login via putty with hub/aws cert for aws location
  4. run turnkey-init
  5. set passwords to same as specified in hub
  6. verify login in webshell (works)

Second set of Steps:

  1. create manual snapshot
  2. wget https://github.com/turnkeylinux/tklbam/releases/download/v1.4.3.2/tklbam_1.4.3+4+g745ecc2_all.deb
  3. /tklbam_1.4.3+4+g745ecc2_all.deb
    1. Which Grub Install Devices?  
      /dev/xvda  (checked in red)
    2. or
      /dev/xvda2  (not checked)
      Choosing xvda  (need to do a few steps to get it checked)
      1. Is this really about GRUB???
      2. Not sure if this an error?

        1. N: Download is performed unsandboxed as root as file '/root/tklbam_1.4.3+4+g745ecc2_all.deb' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
    3. mkdir /tklbam-dump
    4. tklbam-restore BACKUP_ID --raw-download=/tklbam-dump  (my ID/ not Backup_ID)
      1. Was not very Verbose somehow.  I did not see anything about SQL... but let's look a bit more.
    5. systemctl status mariadb
      1. seems "ok"..  some yellow marked in Putty but...
    6. systemctl restart mariadb
    7. systemctl status mariadb
      1. I have to "Copy" the last line to "exit" somehow.  A bit strange
    8. Login to Webmin
      1. some fits and starts
    9. Check Webmin Servers Mysql
      1. get the following which really does not seem correct as nothing changed pw wise:

        MariaDB Database Server
        MySQL version 10.5.15-MariaDB

         

         Warning!  Webmin needs to know your MariaDB administration login and password in order to manage your database. Please enter your administration username (usually root) and password below.

        MariaDB error messageThe full MariaDB error message was : connect to server at 'localhost' failed error: 'Access denied for user 'adminer'@'localhost' (using password: NO)'

         

        MariaDB Login
        Login
        Password
          Force override the given password, if lost or forgotten 

         Save 

      2. gave it the Mysql password set with turnkey-init

      3. I see the same databases that I saw last time around:

        1. Database name Tables Indexes Views

            

          		 information_schema 82 0 0

            

          		 mysql 30 8 1
            Database name Tables Indexes Views

            

          		 performance_schema 80 0 0

            

          		 sites_joomla 78 107 0
      4. So I will work on the restore some more.
      5. Joomla is starting.  I have not crashed it completely
      6. Seeing the "Default" Turnkey Joomla website in https (let's encrypt not configured)
      7. Just need to move the old Joomla Config and Database over
    10. I now have a ghost in the editor (box below)
      1. exiting for now.

L. Arnold's picture

L. Arnold - Fri, 2023/12/22 - 05:10

This is my note as to process.  Thank you for the help Jeremy!
 

Process to Move Joomla from 15.x to 17.x with Jeremy Fix's

 

1. New Joomla3 Server

2:  Verify it is running and you can get in with Webshell or Shell (putty).

hint:  use your hub key and putty if you cannot and run turnkey-init to reset passwords

As of today (6-21-2023)

need to install Install TKLBAM FIXES:  (here: https://github.com/turnkeylinux/tklbam/releases/tag/v1.4.3.3 )

wget https://github.com/turnkeylinux/tklbam/releases/download/v1.4.3.3/tklbam...

 then

 apt install ./tklbam_1.4.3.3_all.deb

Then to do a 2 step TKLBAM RESTORE

1. mkdir /tklbam-dump

2. tklbam-restore BACKUP_ID --raw-download=/tklbam-dump  (my ID/ not Backup_ID)

then load that restore

3. tklbam-restore /tklbam-dump --no-rollback --skip-packages --limits="/var/www/joomla mysql:joomla" | tee tklbam-restore.log

NOTE that my build of Joomla3 had the Database just as "joomla" not "joomla3" which later TKL builds seem to be using..  Important to watch the datase names if specifying the mysql:database name.


and as Jeremy Says:

That should "just work" ...

It did for me 
Joomla3 (TKL ver 15.x to TKL ver 17.x)
 

Again, THANK YOU JEREMY!

 

Add new comment