Marcos's picture
I hope this tutorial is useful for the community. For those who don’t know me, I’m Marcos Méndez. I’ve been managing systems for nearly 15 years, but some of the more “essential” processes in the Linux lifecycle only became solidified in my understanding over the past few years.

Inevitably, when any software reaches its well-known EOL (End of Life), inexperienced young administrators tend to panic when it comes to migrating to the current, more secure version. For a long time, this caused me a lot of stress because I didn’t have enough experience to know the "way of the stones" (as we say in Brazilian Portuguese).

The truth is that despite everything, I decided to dedicate some time to writing about these processes since they’ve evolved over the years. I found in the TKL (TurnKey Linux) community a true haven of stability, reliability, and people willing to help.

That said, I know some may find what I’m sharing here trivial, but for me, dealing with sensitive hospital management data at a national level is something that even the most experienced professionals handle with extreme caution. That moment when you gather courage often becomes the moment things “move forward,” even if sometimes it feels like you’re taking a step back in the learning process—it’s still progress.

Nextcloud, being PHP-based, is widely used on the internet and for web services. Personally, I hate PHP; I find it poorly written and, most importantly, extremely vulnerable due to its widespread use and, consequently, its frequent exploitation. As a result, it’s one of the "most hacked" systems out there.

Nevertheless, one of my clients has an enormous amount of data stored on Nextcloud, and I’m obligated to maintain it.

The Challenges: Challenge 1: We’re dealing with hospital data, which is always in production. My downtime needs to be minimal—if there’s any downtime at all.

Challenge 2: A Kubernetes pipeline is over-engineered for me, being a one-man band. It would add unnecessary complexity and create additional problems (I have Kubernetes applications for other functionalities, managed by a team, so I know what I’m talking about).

Challenge 3: This environment has been in production since TKL 16.0 for Nextcloud, meaning we’re jumping 2 major versions up to version 12.

So, where to start? The first thing is to always keep Nextcloud as up-to-date as possible within the limits of the operating system and its supported PHP packages.

The logic is straightforward: NEXTCLOUD < Stable PHP < Supported OS

For example, PHP 7.2 doesn’t run on Debian 12 but does on Debian 10. You should always check the Nextcloud changelogs to see the minimum supported PHP version to determine which components need upgrading and when. Capiche?

Let’s say we have Nextcloud 25 (which has gone through several PHP upgrades from 7.2 to the more stable 8.1).

However, Nextcloud 26 is not supported on Debian 10 because neither the required PHP version nor the SURY repositories (which I based my last script on) are supported.

What to do? The good old $ apt full-upgrade! \o/

I know it can be scary, but as long as you have a backup ready to go, the worst-case scenario is having to call management and reschedule today’s tests—just kidding (kind of).

The principle is simple: Go to /etc/apt/sources.list.d/source.list and replace every instance of buster with bullseye.

Then run:

bash

$ apt update $ apt full-upgrade At this point, you’ll notice that SURY packages aren’t found for Debian 10. So, we also need to update /etc/apt/sources.list.d/php.list.

This file comes natively with TKL Nextcloud. Replace all instances of buster with bullseye here as well.

Lastly, update /etc/apt/sources.list.d/security.sources.list. There’s a small trick here, which I learned from this documentation: simply replace buster/updates with bullseye-security.

Once that’s done, run:

bash Copiar código $ apt update At this point, you’ll notice that the 404 errors for SURY packages are gone.

Next, execute:

bash Copiar código $ apt full-upgrade Final Notes: This process takes time—a lot of time. It may seem like something went wrong, but the reality is that you’re replacing 100% of the operating system. Even if it’s a container, there’s still a significant amount of work happening under the hood.

Stay patient, and everything will (probably) work out fine!

Some errors are shown

$/usr/bin/deb-systemd-helper: error: systemctl preset failed on resolvconf-pull-resolved.service: No such file or directory Failed to reload daemon: Transport endpoint is not connected

I forgot to resize the LXC Disk to support the upgrade process.

Ask some help to CHATGPT

Nothing helps

Removing mariadb-server-10.3 (1:10.3.39-0+deb10u2) ... Failed to stop mariadb.service: Transport endpoint is not connected See system logs and 'systemctl status mariadb.service' for details. dpkg: error processing package mariadb-server-10.3 (--remove): installed mariadb-server-10.3 package pre-removal script subprocess returned error exit status 1 Failed to reload daemon: Transport endpoint is not connected

Okay, so it didn’t work, everything went wrong.

Linux seems to be more complicated than I thought. In the end, we have a production server and a staging mirror (which I hadn’t mentioned earlier that I commissioned) offline.

I try a:

bash

$apt update $apt --fix-broken install Only to finally arrive at:

The following packages were automatically installed and are no longer required: cpp-8 gyp libapache2-mod-php7.3 libasan5 libc-ares2 libidn11 libip4tc0 libip6tc0 libiptc0 libisl19 libjs-is-typedarray libmpdec2 libmpx2 libperl5.28 libprocps7 libpython3.7-minimal libpython3.7-stdlib libssl-dev libuv1 libuv1-dev node-abbrev node-ansi node-ansi-align node-ansistyles node-aproba node-archy node-are-we-there-yet node-asn1 node-assert-plus node-asynckit node-aws-sign2 node-aws4 node-bcrypt-pbkdf node-bluebird node-boxen node-builtin-modules node-builtins node-cacache node-call-limit node-caseless node-chalk node-chownr node-cli-boxes node-combined-stream node-concat-stream node-config-chain node-console-control-strings node-copy-concurrently node-cyclist node-dashdash node-decompress-response node-deep-extend node-delayed-stream node-delegates node-detect-indent node-detect-newline node-duplexer3 node-duplexify node-ecc-jsbn node-editor node-encoding node-end-of-stream node-escape-string-regexp node-extsprintf node-flush-write-stream node-forever-agent node-form-data node-from2 node-fs-vacuum node-fs-write-stream-atomic node-fs.realpath node-gauge node-getpass node-glob node-got node-har-schema node-har-validator node-has-symbol-support-x node-has-to-string-tag-x node-has-unicode node-hosted-git-info node-http-signature node-iconv-lite node-iferr node-import-lazy node-imurmurhash node-inflight node-ini node-is-builtin-module node-is-npm node-is-object node-is-plain-obj node-is-retry-allowed node-is-typedarray node-isstream node-isurl node-jsbn node-json-parse-better-errors node-json-schema node-json-stringify-safe node-jsonparse node-jsonstream node-jsprim node-latest-version node-lazy-property node-libnpx node-lockfile node-lowercase-keys node-mime-types node-mimic-response node-minimist node-mississippi node-move-concurrently node-mute-stream node-node-uuid node-nopt node-normalize-package-data node-npm-package-arg node-npmlog node-oauth-sign node-once node-opener node-osenv node-p-cancelable node-p-timeout node-package-json node-parallel-transform node-path-is-inside node-performance-now node-prepend-http node-promise-inflight node-promzard node-proto-list node-pump node-pumpify node-qs node-qw node-rc node-read node-read-package-json node-registry-auth-token node-registry-url node-request node-resolve-from node-retry node-rimraf node-run-queue node-semver node-semver-diff node-sha node-slash node-slide node-sorted-object node-spdx-correct node-spdx-expression-parse node-spdx-license-ids node-sshpk node-ssri node-stream-each node-stream-iterate node-stream-shift node-strip-json-comments node-tar node-term-size node-text-table node-through node-through2 node-timed-out node-tough-cookie node-tunnel-agent node-tweetnacl node-typedarray node-uid-number node-unique-filename node-unpipe node-url-parse-lax node-url-to-options node-uuid node-validate-npm-package-license node-validate-npm-package-name node-verror node-wide-align node-widest-line node-wrappy node-write-file-atomic node-xdg-basedir perl-modules-5.28 python-pkg-resources python3.7-minimal A whole bunch of Node.js packages that don’t work.

I forgot... NextCloud is so heterogeneous that it relies on JavaScript rendering, which I had also forgotten depends on Node.js. Another headache full of vulnerabilities and problems.

In the end, I’m going to have to start over.

My wife has already gone to the port to sail with her friends on the boat I built over the last four years... and here I am, on a Sunday, writing this completely frustrated text. I took 2x Valsartan (180 mg) to keep my blood pressure normal (I can’t let the stress take over my body).

It feels like the story of Donald Crowhurst: Donald Crowhurst - Wikipedia

But I won’t surrender.

Ok lets do what i should have done a long time ago

Install Nextcloud 17.0 from TKL and migrate database and files

For some reason i don't know why proxmox (my hypervisor) doesn't show the 17.1 version from http://mirror.turnkeylinux.org/turnkeylinux/images/proxmox/debian-11-tur...

For all of you trying to survive please keep this link saved http://mirror.turnkeylinux.org/turnkeylinux/images/proxmox

This will sabe you life later.

I also don't know why the 16.0 version is not here, maybe because it was discontinued. Archive.org maybe can save you in this case?

No worries.

Downloading the 17.1 Nextcloud TKL images and starting a new LXC for migrating

The download is limited by 750kb/s I remember i promised David to comission a mirror for Brazil with all the TKL Images, I also remember that i don't know how to do that. I cry.

Another 30 minutes had pass

(In the mean while my NC/TKL app with Version 18.1 is Rsyncing in the 4th LXC i comisisoned for this mission the files, after finishing i'll try to import the DB, maybe it works, but i don't have faith)

My wife sends a message: When do you think you leave the office to join us in the Sunday afternoon? I Cry again. a Sunday with a sunny day.

I answer her, I have no CLUE

I think: (They are paying me 3rd world countries salary (200US$ for this upgrade) why i'm submitting my self to that? I remember, if i don't do it it will be worse!) Captalism sucks.

The 3rd backup restored, I try to backup the database to migrate it to a 17.2 TKL/NextCloud LXC

I install the TKL 17.2 NextCloud with the same adminer and admin password ad the 16.1 version (I fear this the if I change the passwords for newer one this might crash the migration)

In the meanwhile i tar -czvf nextcloud-www-folder.tar.gz www to copy it to the 17.2 version, hoping it will work

I forget to resize the container ( it got stuck ) i stop the process and kill dumm stuff, resize the disk in proxmox and continue the process. TASK OK next page

While tar is compressing the 16NC files i go to the 17 and import the DB. Lets cross fingers

I the image after installation doesn't response, the NC/TKL17.2 might be broken or untested as it should in PVE8.3.0 i dont know.

Thinks are scaling up im 3 days straight with no sleep trying to fix this stuff

Lets try again

Go to the node7 where the LXC is hosted, PCT enter lxc-id

Voila i have the yellow letters.

$service apache2 status

!FAILED!

Try to recomission the LXC

turnkey-init the LXC17 is not responding. 45 seconds to respond? What is going on? Everything is slowing down? password authentication takes another 50seconds.

Turnkey init showd the d-live (blue hell screen) im filling the forms

I Skip the TKLBAP key, i don't have money neither my costumer for that (WHY!!!)

tkl-init works, Webmin responds, i'm able to import the SQL? lets find out

The database has the same name, but the hash that tkl-init uses is differant, i have to drop all the tables and import the SQL manually

8 or 9 prompt to GPT to help me and he provides me this:

SET FOREIGN_KEY_CHECKS = 0;

DROP TABLE IF EXISTS oc_collres_collections; DROP TABLE IF EXISTS oc_filecache; DROP TABLE IF EXISTS oc_file_locks; DROP TABLE IF EXISTS oc_oauth2_access_tokens; DROP TABLE IF EXISTS oc_calendars; DROP TABLE IF EXISTS oc_dav_shares; DROP TABLE IF EXISTS oc_open_local_editor; DROP TABLE IF EXISTS oc_circles_mount; DROP TABLE IF EXISTS oc_comments; DROP TABLE IF EXISTS oc_storages_credentials; DROP TABLE IF EXISTS oc_group_admin; DROP TABLE IF EXISTS oc_text_sessions; DROP TABLE IF EXISTS oc_calendar_invitations; DROP TABLE IF EXISTS oc_dav_cal_proxy; DROP TABLE IF EXISTS oc_notifications; DROP TABLE IF EXISTS oc_reactions; DROP TABLE IF EXISTS oc_calendar_rooms_md; DROP TABLE IF EXISTS oc_schedulingobjects; DROP TABLE IF EXISTS oc_vcategory; DROP TABLE IF EXISTS oc_calendar_reminders; DROP TABLE IF EXISTS oc_whats_new; DROP TABLE IF EXISTS oc_users; DROP TABLE IF EXISTS oc_profile_config; DROP TABLE IF EXISTS oc_storages; DROP TABLE IF EXISTS oc_filecache_extended; DROP TABLE IF EXISTS oc_calendarobjects_props; DROP TABLE IF EXISTS oc_files_trash; DROP TABLE IF EXISTS oc_activity_mq; DROP TABLE IF EXISTS oc_file_metadata; DROP TABLE IF EXISTS oc_privacy_admins; DROP TABLE IF EXISTS oc_circles_remote; DROP TABLE IF EXISTS oc_photos_albums; DROP TABLE IF EXISTS oc_twofactor_backupcodes; DROP TABLE IF EXISTS oc_login_flow_v2; DROP TABLE IF EXISTS oc_authorized_groups; DROP TABLE IF EXISTS oc_addressbookchanges; DROP TABLE IF EXISTS oc_mimetypes; DROP TABLE IF EXISTS oc_jobs; DROP TABLE IF EXISTS oc_photos_albums_collabs; DROP TABLE IF EXISTS oc_circles_circle; DROP TABLE IF EXISTS oc_addressbooks; DROP TABLE IF EXISTS oc_circles_token; DROP TABLE IF EXISTS oc_share; DROP TABLE IF EXISTS oc_circles_share_lock; DROP TABLE IF EXISTS oc_preferences; DROP TABLE IF EXISTS oc_comments_read_markers; DROP TABLE IF EXISTS oc_flow_checks; DROP TABLE IF EXISTS oc_mounts; DROP TABLE IF EXISTS oc_flow_operations; DROP TABLE IF EXISTS oc_systemtag_object_mapping; DROP TABLE IF EXISTS oc_notifications_pushhash; DROP TABLE IF EXISTS oc_calendarobjects; DROP TABLE IF EXISTS oc_group_user; DROP TABLE IF EXISTS oc_federated_reshares; DROP TABLE IF EXISTS oc_direct_edit; DROP TABLE IF EXISTS oc_circles_member; DROP TABLE IF EXISTS oc_vcategory_to_object; DROP TABLE IF EXISTS oc_calendarchanges; DROP TABLE IF EXISTS oc_user_transfer_owner; DROP TABLE IF EXISTS oc_files_versions; DROP TABLE IF EXISTS oc_trusted_servers; DROP TABLE IF EXISTS oc_notifications_settings; DROP TABLE IF EXISTS oc_collres_accesscache; DROP TABLE IF EXISTS oc_properties; DROP TABLE IF EXISTS oc_authtoken; DROP TABLE IF EXISTS oc_cards; DROP TABLE IF EXISTS oc_cards_properties; DROP TABLE IF EXISTS oc_calendar_resources_md; DROP TABLE IF EXISTS oc_calendar_resources; DROP TABLE IF EXISTS oc_text_steps; DROP TABLE IF EXISTS oc_recent_contact; DROP TABLE IF EXISTS oc_circles_event; DROP TABLE IF EXISTS oc_groups; DROP TABLE IF EXISTS oc_flow_operations_scope; DROP TABLE IF EXISTS oc_bruteforce_attempts; DROP TABLE IF EXISTS oc_circles_membership; DROP TABLE IF EXISTS oc_oauth2_clients; DROP TABLE IF EXISTS oc_text_documents; DROP TABLE IF EXISTS oc_calendarsubscriptions; DROP TABLE IF EXISTS oc_systemtag; DROP TABLE IF EXISTS oc_directlink; DROP TABLE IF EXISTS oc_circles_mountpoint; DROP TABLE IF EXISTS oc_user_status; DROP TABLE IF EXISTS oc_known_users; DROP TABLE IF EXISTS oc_webauthn; DROP TABLE IF EXISTS oc_migrations; DROP TABLE IF EXISTS oc_share_external; DROP TABLE IF EXISTS oc_photos_albums_files; DROP TABLE IF EXISTS oc_collres_resources; DROP TABLE IF EXISTS oc_systemtag_group; DROP TABLE IF EXISTS oc_ratelimit_entries; DROP TABLE IF EXISTS oc_twofactor_providers; DROP TABLE IF EXISTS oc_appconfig; DROP TABLE IF EXISTS oc_calendar_rooms;

SET FOREIGN_KEY_CHECKS = 1;



IT WORKS!

Now i have to import the NC/TKL 16 Database

I go to webmin, MariaDB, Nextcloud Database, Execute SQL, Run SQL from File, chose the SQL in my downloads folder and wait.

Lets wait,

and a little more wait

Small steps and solid footprints this is the concept, don't forget, this is a compass in a sysadmin life.

$Output from uploaded SQL commands .. No output generated 134 tables were created. 214 records were inserted.

Everything seems working.

Im going to rsync the files now. Fingerscross again. I Think again, rsync is to slow tar compress and scp will be faster. It works, another 1h45 minutes to transfer the files.

Tar.gz -xzvf is taking over 2hrs I really think that I should start contribuiting to TKLdev comunnity with a more, let's say human approach, why this nextcloud is monolith? I could try to make several LXCs with a SMB for files a SQL for the DB and PHP for the webserver and another separate one for redis, maybe i could sync to HAProxy and KeepAlived to make it highly available, just ideas, waiting to decompress information.

So after extracting I realize i just have to move the nextcloud-data folder, as the sql was already imported and the nextcloud folder has the newer 26.0.3 version of NC

After doing that it WORKS!!! WOW crying for being happy after several errors and 3 days working.

I try to login and it generates a 500 error. I realize i can check in the nextcloud-data/data/nextcloud.log to check what is going on. Apperantly is a secret error!

I realize when i installed the new NEXTCLOUD the random encryption generated is totally different to the one registered in the SQL database

I go to the nextcloud/config/config.php and copy the secret from the OLD Nc instance to the new one.

AWESOME! I log-in and al the files are available.

I just have to repeat the process to the TKL18 of NC and after that resync everything to the Hospital production.

I Hope this helps people to realize 2 things

TKL makes things easier as they are standarized and well documented by a effortless and unbreakable community

Second and most important, maintain anything is a PITA (Pa!n in d as) do not never be dimotivated, things have to be done no matter what (in this case hospitals with low budget in Brazil)



Just do what every body does, i'm not a superhuman neither anyone else. Upgrading in the painfull manner is the only reliable and straightforward way we have found. We all want a magicall script that does all for us. But neither GPT or anyone else has managed to transform this text in a html readable, imagine a complex script like that. LOL. Good 2025 to every one. Godspeed all the sysadmins here.



Gratefull and thankyou for this beautifull community!
Forum: 
Jeremy Davis's picture

Happy New Year to you too! :)

Thanks for your kind words. I agree with you on our "unbreakable community" but I'm not sure that "well documented" is a realistic description! :) We try, but as per many open source projects, documentation is perhaps one of our biggest challenges. I don't want to raise expectations too high, but we have plans to consolidate and organize our documentation better. The structure of the updated website (background work in progress - no ETA) will hopefully make docs easier to find what you are looking for and/or contribute to make it better. It will allow us to have a "single source of truth" without the way that information is quite scattered and not always consistent.

I'm glad to hear that despite all the false starts and dead ends you managed to get there in the end. The story of your Nextcloud maintenance/upgrade experience here is an epic tale. It has plenty of woe, but also shows good humor and lots of tenacity! TBH it actually read a lot like a good short story. For a moment I forgot that it was your real world frustration maintaining a Turnkey server and was just a cool and interesting story. It had all the ups, downs and the happy ending that many good fiction stories have. But then I crashed back to reality...

Beyond that I'd like to take this opportunity to explicitly to thank you for all that you have contributed over the years. You are a valuable community member and we're really glad to have you involved. TurnKey community members like you are what keep us going. It is fantastic to get insight into both the good and the bad parts of using TurnKey in production. I have no doubt that your contributions to TurnKey over the years have helped many other users - perhaps without them (or you) even knowing.

Your Nextcloud experience is particularly useful. It is one of our most popular appliances but I have not maintained it in production myself. Vicarious experience from long term users such as yourself is invaluable. Despite my extensive experience maintaining the appliance library (there is not a single appliance that I have not touched at one time or other) I have only maintained a few TurnKey servers in production. My "real world" experience maintaining a production TKL server is limited. We only "dogfood" a few (inc this website and the master mirror) and I run some for personal use (Gitea, Mediaserver among them). So there are many that I have never used other than to spin up to troubleshoot bug and issues users report.

I started writing a heap more, but I'm not going to get it finished, so figured that I'd post this bit first. Hopefully I'll get back to it next week.

Take care bro and thanks again for everything :)

Add new comment