L. Arnold's picture
L. Arnold - Fri, 2023/09/22 - 03:41

Hello TKL Team!

Unfortunately my LetsEncrypt stopped working and I can't migrate (first try anyway) my Joomla 3 Server running TKL 15.5 - Stretch.

I mainly want to get Let's encrypt going again on this Joomla Server./

Since I am asking also need to move TLS on a TKL Odoo 14.1 server to support TLS 1.2 or 1.3.  Google and Firefox are rejecting https on this server.  This needs to run Odoo 8 so stuck with TKL 14.

Sorry for 2 questions.  One project on my side.  Working through TKLDEV or some manual migrations otherwise will be needed I expect.

Thank you for your help!

Forum: 
Support
Jeremy Davis's picture

Jeremy Davis - Tue, 2023/09/26 - 06:07

You have a few options (in no particular order):

  • Migrate data (should work ok with joomla, odoo will be trickier)
  • Update dehydrated - may resolve the issue if it's purely to do with ACME protocol; won't work if the underlying cause is OpenSSL too old
  • Put your server(s) behind a reverse proxy - i.e. do HTTPS termination on a newer server and reverse proxy traffic to your existing server(s)
  • run a new server purely to get LE certs - and sync certs with a cron job

Migrate Joomla data

Migrating your Joomla with TKLBAM should work ok, especially if you just restore the joomla files and DB. The same can't be said of Odoo, as that requires a specific version of python. But for Joomla, try this on a new Joomla3 server (where BACKUP_ID is the actual backup ID number of your Joomla3 backup):

mkdir -p /tklbam-dump
tklbam-restore BACKUP_ID --raw-download=/tklbam-dump
tklbam-restore /tklbam-dump --skip-packages --limits="/var/www/joomla3 mysql:joomla3"

That will restore just your Joomla data dir and DB. To reset the Joomla DB password (so Joomla can connect to it's DB) just re-run the firstboot script:

/usr/lib/inithooks/firstboot.d/20regen-joomla-secrets

If you want/need other data/files, you'll find them in /tklbam-dump - relative to /. I.e. /etc/some/file will be found at /tklbam-dump/etc/some/file. Note though that manually moving files will change permissions (to the user who moves the files - probably root). To keep permissions consistent with their original source, use tklbam with limits (e.g. 'tklbam-restore /tklbam-dump --skip-packages --limits="/file/to/restore /dir/to/restore/"').

Update Dehydrated

I'd need to know more about the failure to be sure that this would work, but in theory, it should be fairly easy to update Dehydrated. There is a possibility that the failure is caused by an old version of OpenSSL (rather than Dehydrated). If that's the case, then this won't work. But if the issue is the old version of Dehydrated, then this will work.

I have a suspicion that removing the current dehydrated might break things, so instead just rename the executable (this is generally considered bad practice, but you gotta do what you gotta do sometimes and seeing as Stretch won't have any updates it's fairly safe). So (as root):

mv /usr/bin/dehydrated /usr/bin/dehydrated.old

Then install the latest (Dehydrated is a bash script, so this is not dangerous either):

cd /usr/local/src
git clone https://github.com/dehydrated-io/dehydrated.git
ln -s /usr/local/src/dehydrated/dehydrated /usr/local/bin/dehydrated

Then double check that it's set up properly:

which dehydrated

It should return (note the local dir in the path):

/usr/local/bin/dehydrated

If it still points to /usr/bin/dehydrated (note no 'local' in the path) then something isn't right.

Now update the "hook script":

url=https://raw.githubusercontent.com/turnkeylinux/confconsole/master/share/letsencrypt/dehydrated-confconsole.hook.sh
file=/etc/dehydrated/confconsole.hook.sh
wget -O $file $url

Also update the config file (add new variable; CHALLENGETYPE):

echo "CHALLENGETYPE='http-01'" >> /etc/dehydrated/confconsole.config

If it's failing because of the old Dehydrated, it should now work. If it still fails, then my guess is that it needs newer OpenSSL - but that's a whole lot harder to do...

Other options

I have to dash now but I'll try to circle back to this soon and give detail on the other options. If you notice I haven't responded in a few days time, please feel free to bump me. Also please share how you go with either of the above.

L. Arnold's picture

L. Arnold - Wed, 2023/09/27 - 05:04

So I will go through and use your system above.  Thank you for that.

I thought I would try first on a simple TKLBAM Restore # server which is not starting.

Error tells me that I should also try "turnkey-init" to get MySQL password working post restore.  I did just do that on another RAW system which at least got me into that one.
(but yes, I will  try your very good instruction above.  Seems the best path)

 

root@joomla3 ~# /usr/lib/inithooks/firstboot.d/20regen-joomla-secrets
Traceback (most recent call last):
	File "/usr/lib/inithooks/bin/mysqlconf.py", line 145, in <module>
		main()
	File "/usr/lib/inithooks/bin/mysqlconf.py", line 124, in main
		m = MySQL()
	File "/usr/lib/inithooks/bin/mysqlconf.py", line 41, in __init__
		self._start()
	File "/usr/lib/inithooks/bin/mysqlconf.py", line 64, in _start
		raise Error("could not start mysqld")
	__main__.Error: could not start mysqld

Followup:
Still won't start at least the web after Turnkey-Init

Also the Same error on Joomla Secrets Regen.

I will follow your step by step on a new server.  
Thank you for the guidance.  notes to follow

L. Arnold's picture

L. Arnold - Wed, 2023/09/27 - 05:42

The Table Restore brings error:

table: mysql/tables_priv
table: mysql/index_stats
table: mysql/db
table: mysql/innodb_index_stats
table: mysql/time_zone_name
table: mysql/time_zone_leap_second
table: mysql/event
table: mysql/proc
table: mysql/time_zone_transition
table: mysql/proxies_priv
table: mysql/slow_log
table: mysql/time_zone_transition_type
table: mysql/table_stats
table: mysql/help_category
table: mysql/help_topic
table: mysql/columns_priv
table: mysql/ndb_binlog_index
table: mysql/gtid_slave_pos
table: mysql/roles_mapping
table: mysql/plugin
table: mysql/func
table: mysql/host
table: mysql/innodb_table_stats
table: mysql/general_log
table: mysql/help_relation
table: mysql/time_zone
table: mysql/column_stats
table: mysql/user
table: mysql/servers
table: mysql/procs_priv
table: mysql/help_keyword
ERROR 1050 (42S01) at line 6219: Table 'user' already exists We're done. You may want to reboot now to reload all service configurations.

 

Not sure if that is ok but hey.
More to follow.

L. Arnold's picture

L. Arnold - Wed, 2023/09/27 - 06:04

The Web Interface gave me the same "Error"  (only that)

I went into Webmin and checked MariaDB Server

Had to provide "Adminer + Password"  and got in.
I see these 4 Database:

information_schema

mysql

performance_schema

sites_joomla

When I look at "Table Data", then Users.
I see my email as Super User.  Then a user and a Manager.
Does not appear that more data came in. 

---

When I look at Apache (I had a rather complicated Apache having the old Server host perhaps 6 different domains).  I don't think we restored that in the process above.

In apache,  yes, we are serving Joomla defaults which is "ok".  (My apache was quite complicated)...

-----

Just Browsed the var/www/joomla directory and I am not seeing any of the restore.
I think the Restore data is in some parrallel directories.

I see tklbam-dump  (fair bit of stuff there) (etc, root, TKLBAM, usr, var)

and tklbam-dumptklbam-restore  (which is Empty)

----------

Let me work on that restore process again.  I don't think it hit the live data on the server.  Basically made it into the TKLBAM folder but did not then get out (theories anyway)
 

Jeremy Davis's picture

Jeremy Davis - Fri, 2023/10/06 - 05:30

Hey Landis, apologies on the slow response, as usual I have way too many plates spinning... :)

Regardless, if you are ok to share your data with me, then I would be happy to have a try. I'm really trying to push the v18.0 release out as fast as we can, but you're a long term TurnKey community member and supporter of TurnKey, so it would be severely remiss of me to not offer you a bit of extra assistance! :)

Here's how to gather your latest backup into a password protected archive. Note that it requires CLI (feel free to use some other method if you prefer, but this should "just work"):

mkdir -p /tklbam-joomla3-dump
# change BACKUP_ID below for actual backup ID number
tklbam-restore BACKUP_ID --raw-download=/tklbam-joomla3-dump
tar czf - /tklbam-joomla3-dump | gpg -o landis-joomla3.tgz.gpg --symmetric

That should ask you to enter a password, and will result in an encrypted archive file of your backup data called landis-joomla3.tgz.gpg. You'll then need to download the file from your server. My guess is that it will be fairly large (probably too big for email) so you'll probably have to upload it somewhere. I notice you have a gmail address, so perhaps upload to Google drive and send me a link? Please note that my turnkey email is not linked to google, so better to use my personal email (which is linked to google - jeremy AT jeremydavis.org). Email the password to my TurnKey email. Alternatively I'll email you my phone number and you can send it to me via What's App or Signal text message.

Alternatively, if you'd rather me log into your server and have a look, please add my public key to your server like this (it's a REALLY long, single line):

echo "ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAABAQCvBup9/16HsrT4a+WxkjNScbYCueCJVCkxHtLRnUEOYH7HTQfybmy4JSRkPiyS4ig7Jk9lAoFLiLtEfI4l93iQsr/gp3SAjKftxso5Wf/BtOVinkzue66tq+4EZcji+cezg3JfqKz2NlCnmcFh3/8ohTk+i8EM10fzaN41cYGL4/x/G0I2wA1bfPR4dZe2ybUIoHzfzYJzOqgv9knNIxXBACQyXiN801hNdKhq7Snk8kBrc+IxoLZIvqTYXIFeSDLSn5L9n8plLQnhWhEg72cUyoJGBHofy+RGpHcaKBnuEXtLrXpqzpg23X+L2zwry9a/GMeMS9u+sXB2PD5fym4N jeremy@turnkeylinux.org" >> /root/.ssh/authorized_keys

Then email or text me the public IP address.

Jeremy Davis's picture

Jeremy Davis - Wed, 2023/10/11 - 06:49

Ok, fingers crossed, I think I have a fix!

FWIW it appears that the issue is related to this MariaDB bug. FWIW TKL v16.x had MariaDB v10.3; TKL v17.x has MariaDB v10.5.x - so restoring backups from v16.x or earlier (that include a MySQL/MariaDB database) will fail when restoring to v17.0 or newer.

To work around the issue, I've created a TKLBAM hook script - TBH, because it affects about 70% of the library (any app backup that includes MySQL/MariaDB from v16.x or prior, being restored to a v17.0 or newer) I'm not 100% sure where it should go. But for now I've created a new repo in my personal GitHub account. For full details (including how to get it on your server) please see the Readme.

Please note that the script could do with some tidying, but should be fully functional. I'd appreciate any feedback you have (good, bad or ugly).

I'm not sure, but perhaps it'd be a good idea to try something similar for your Odoo restore. I.e. bundle up your backup and send it through so I can have a play with it?

L. Arnold's picture

L. Arnold - Mon, 2023/10/02 - 17:32

Strange,  I have been able to get a Odoo 8 TKL 14.2 Server fired up on Oracle Cloud

Trying to migrate a Linode Hosted version of the same.  All works but the main data file.

The "Old" Linode Hosted version is Postgresql 9.4.26

the "New" Oracle Hosted version is Postgresql 9.4.12

I get database, wrong version, in TKLBAM.
I just get failure in Odoo Database Manager

and I can't seem to upgrade Postrgresql within Shell.

I know I know,  move forward.  Really just trying to archive this.  Might just do a data extract and move on.

Jeremy Davis's picture

Jeremy Davis - Fri, 2023/10/06 - 06:48

Hmm, that seems weird!? Are you using default PostrgreSQL? (from Debian apt repo?)

Assuming that you are, I suggest that you double check the package version via apt:

apt-cache policy postgresql

Run that on both servers and compare. My guess is that there is an update (to at least 9.4.26) available in the repos.

If you upgrade (install) postgresql via apt, hopefully that should resolve it. I.e. (on the new Oracle server):

apt-get update
apt-get install postgresql -y
Jeremy Davis's picture

Jeremy Davis - Mon, 2023/10/09 - 10:19

Ok, so I've had a closer look at this Odoo v14.2 one and I think I have a workaround. The issue is that the Debian Jessie repos no longer exist. A workaround is to use the Freeaxian ELTS (Extended Long Term Support - Freeaxian are a company that provide Debian based services). If you are doing anything important with this server, I strongly recommend that you consider subscribing to Freeaxian. Whilst it's not required (the updates they provide are provided free to anyone), if you are a paying subscriber, they will support the packages that you are using.

To enable the Freeaxian Jessie repos:

# download the key
apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com A07310D369055D5A

# write the sources list for Freeaxian ELTS
cat > /etc/apt/sources.list.d/freeaxian-elts.list <<EOF
deb http://deb.freexian.com/extended-lts jessie main contrib non-free
deb http://deb.freexian.com/extended-lts jessie-lts main contrib non-free
EOF

# disable old repos
mv /etc/apt/sources.list.d/sources.list /etc/apt/sources.list.d/sources.list.disabled
mv /etc/apt/sources.list.d/security.sources.list /etc/apt/sources.list.d/security.sources.list.disabled
mv /etc/apt/sources.list.d/backports.sources.list /etc/apt/sources.list.d/backports.sources.list.disabled

# run apt update
apt-get update

# install postgres
apt-get install -y postgresql

I'm not 100% sure that will be enough, but it should definitely get you to a point where you have the same version of PostgreSQL on both v14.2 servers. FWIW here's my Odoo v14.2 server after I ran the above:

# apt-cache policy postgresql
postgresql:
  Installed: 9.4+165+deb8u4
  Candidate: 9.4+165+deb8u4
  Version table:
 *** 9.4+165+deb8u4 0
        500 http://deb.freexian.com/extended-lts/ jessie/main amd64 Packages
        500 http://deb.freexian.com/extended-lts/ jessie-lts/main amd64 Packages
        100 /var/lib/dpkg/status
L. Arnold's picture

L. Arnold - Mon, 2023/10/09 - 20:50

that is cool Jeremy.  Nice work with the Freeaxian.

I got a small error and I am not sure If I got all the way to Postgresql 9.4.26  (ancient history I know) which seems to be my hangup.

-------

I got this error at the end of several freeaxian updates....

apt-get update

W: Failed to fetch http://http.debian.net/debian/dists/jessie-backports/main/binary-amd64/Packages  404  Not Found [IP: 151.101.26.132 80]

which then led to
E: Some index files failed to download. They have been ignored, or old ones used instead.

---

was a typo in the "policy" line ("r" instead of "t" in "postgresql

but I seem t o be in the same place as you now..  Not sure if that is fully 9.4.26 - I am readin 165


 

root@odoo ~# apt-cache policy postgresql

postgresql:

Installed: 9.4+165+deb8u4

Candidate: 9.4+165+deb8u4

Version table:

*** 9.4+165+deb8u4 0

        500 http://deb.freexian.com/extended-lts/ jessie/main amd64 Packages                                                                                            

        500 http://deb.freexian.com/extended-lts/ jessie-lts/main amd64 Packages                                                                                        

100 /var/lib/dpkg/status

root@odoo ~#

 

However.  the old server,  though "Webmin" is telling me  version "9.4.26" in Postgresql

When I go to Shell,  this is my Log of that last set


 

root@odoo ~# apt-cache policy postgresql

postgresql:

Installed: 9.4+165+deb8u4

Candidate: 9.4+165+deb8u4

Version table:

*** 9.4+165+deb8u4 0

        500 http://security.debian.org/ jessie/updates/main amd64 Packages                                                            

100 /var/lib/dpkg/status

9.4+165+deb8u3 0

        500 http://http.debian.net/debian/ jessie/main amd64 Packages   

 

When I go into Webmin it is still showing "9.4.12"  (not 9.4.26, as on the other server) but I don't really trust Webmin to be reading the situation.  Perhaps that is a "Webmin rendering of PostgreSQL".  not sure.

I will try to move the database over with the Odoo Database tool, which is where I was getting the wrong version file error in the past.

----

Tried that again through Odoo, but just got a vague statement that the database cannot be restored

The old server so the database should be transportable..  Alas not just now.

----

Just logged in with Adminer..
It looks like the "Posggresql version we are looking at is saying:

PostgreSQL version: 9.4.26 through PHP extension PgSQL

Anyway,

Perhaps there is a way, from either the Shell or Adminer to do a Database Dump then, on the new Oracle Server bring it in in such a way that Odoo does not reject it.  I know this is an Odoo issue as much as anything.  

Let me see what I can find.

 

L. Arnold's picture

L. Arnold - Wed, 2023/10/11 - 17:25

It is interesting that the 2 issues here (sorry to have 2 issues in one thread) are different but related.

Odoo is running Postgresql

Joomla is running MariaSQL

both, with different generations of TKL have different database versions.  Both have database migration issues owing to the database generations.

Joomla3:  Anyway,  running now on a new hub.joomla server.

Complications arose from both Linode and the Hub requiring different password formats..  hopefully I can get through any differential and use turnkey-init later to get things syncronized.  Mainly difficult keeping track of these things through generations and the fact that TKLBAM itself has a tendency to reset some of them.

speaking of which,  I just got the following error:


HOOK /etc/tklbam/hooks.d/maria_db-changes :: op=restore state=post pwd=/root ::
TKLBAM_RESTORE_PROFILE_ID: turnkey-joomla3-15.5-stretch-amd64
[maria_db-changes] INFO: hook invoked after backup restore. Extras path = /root
[maria_db-changes] INFO: Restore of MySQL/MariaDB database detected; ensuring th                                                    at MySQL/MariaDB is running properly
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run                                                    /mysqld/mysqld.sock' (2)
2023-10-11  8:20:14 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-0+deb11u                                                    1) starting as process 4330 ...
FATAL: /var/run/mysqld/mysqld.sock not found after waiting 11 seconds
Traceback (most recent call last):
  File "/usr/bin/tklbam", line 46, in <module>
    CliWrapper.main()
  File "/usr/lib/tklbam/cliwrapper.py", line 88, in main
    commands[command].main()
  File "/usr/lib/tklbam/cmd_restore.py", line 533, in main
    hooks.restore.post()
  File "/usr/lib/tklbam/hooks.py", line 81, in post
    self._run("post")
  File "/usr/lib/tklbam/hooks.py", line 73, in _run
    _run_hooks(self.LOCAL_HOOKS, (self.name, state))
  File "/usr/lib/tklbam/hooks.py", line 42, in _run_hooks
    (fpath, " ".join(args), e.exitcode))
hooks.HookError: `/etc/tklbam/hooks.d/maria_db-changes restore post` non-zero ex                                                    itcode (1)
 

So...  Earlier before a crash on the previous server, I saw some "user updates" right out of the box.  This time it seemed to come at the end...

Anyway, I am sure we are getting close..
Sorry for the trouble.

Landis

 

Jeremy Davis's picture

Jeremy Davis - Thu, 2023/10/12 - 01:00

Looking at your output, it looks like MySQL/MariaDB isn't/wasn't running. You could try to start the service like this:

systemctl restart mariadb

If this is an existing server that you've already tried, and the systemctl command fails, then my guess is that MySQL/MariaDB has already been broken. It's fixable, but in my recent experience, it's a PITA. One thing that you could try that isn't too hard, is wiping out the current DBs (back to Debian default) and try again? This should do it:

systemctl stop mariadb
pkill mariadb # just in case an orphan process is running
mv /var/lib/mysql /var/lib/mysql.bak
mkdir /var/lib/mysql
chown mysql:mysql /var/lib/mysql
mariadb-install-db
systemctl start mariadb

If that doesn't fix MySQL/MariaDB and it's still not running, in scenarios like this, it's often quicker and easier to start again from scratch.

I recall when we chatted, you were having troubles getting it running locally on VirtualBox. For this sort of stuff a local install will likely make your feedback loop tighter, so perhaps it's worth exploring what the issue is with that not working? Then you can continue your testing and trial and error work locally!?

Add new comment