I was thinking that you were trying to install TurnKey as a VM onto to Proxmox, rather than installing to bare metal and installing Proxmox (I assume packages) onto that...

As to why it didn't work, TBH, I can't be sure, but as TurnKey is primarily aimed at running as a VM, the hardware support is not as good as it ideally should be (e.g. TurnKey doesn't support UEFI and/or secure boot; there is lots of networking hardware it doesn't support OOTB, etc). Considering that Proxmox is designed to be installed to hardware, I suggest that you've done the right thing!

If you want to install TKLBAM though, that should be possible. You'll need to add the TurnKey repos. Probably the easiest way to do that would be to just copy the info from an installed TurnKey appliance. On any TurnKey appliance check the /etc/apt/sources.list.d/sources.list and /etc/apt/sources.list.d/security.sources.list files for the "archive.turnkeylinux.org" lines to add to the apt config to your Proxmox server (I suggest creating a new file in /etc/apt/sources.list.d/ called "turnkey.list" or similar). You'll find the required TurnKey "main" and "security" keyfiles in /usr/share/keyrings/tkl-buster-main.gpg & /usr/share/keyrings/tkl-buster-security.gpg. Copy them across to the same location on your Proxmox server.

Once you've done that, hopefully this should do the trick to install tklbam:

apt update
apt install tklbam

Then link to your Hub account. Seeing as there isn't a profile for Proxmox, I suggest that you just use the one for v16.0 Core:

tklbam-init YOUR_HUB_API_KEY --force-profile=core-16.0-buster-amd64

(Where YOUR_HUB_API_KEY is your actual Hub API key).

You can then do a dry run to get an idea of what will be backed up like this:

tklbam-backup --simulate

Then you can have a poke around in /TKLBAM to see what is being backed up. You can then add (or exclude) specific directories via the /etc/tklbam/overrides. To add new directories or files, put the full path onto a new line (you can add as many as you like). To exclude a directory or file(s) preceded the full path with a dash/minus (i.e. '-'). You can mix and match inclusions and exclusions to pin point specific files/directories.

You can remove the /TKLBAM directory and re-run to check your updated config. I.e.:

rm -rf /TKLBAM
tklbam-backup --simulate

Once you are happy, then you can just run the full backup normally:

tklbam-backup

If you want to enable automated backups, then you can just add a cron job, or copy across the default cronjob from a TurnKey instance. You can find it at /etc/cron.daily/tklbam-backup or create it manually (from a SSH session) like this:

cat > /etc/cron.daily/tklbam-backup >>EOF
#!/bin/sh

set \${JITTER:=3600}

randomsleep() {
    MAXSLEEP=\$1

    if [ -n "\$MAXSLEEP" ] ; then
        if [ \$MAXSLEEP -gt 0 ] ; then
            if [ -z "\$RANDOM" ] ; then
                # A fix for shells that do not have this bash feature.
                RANDOM=$(dd if=/dev/urandom count=1 2> /dev/null | cksum | cut -c"1-5")
            fi
            TIME=\$((\$RANDOM % $MAXSLEEP))
            sleep \$TIME
        fi
    fi
}

# prevent everyone from hitting the Hub API at exactly the same time...
randomsleep \$JITTER
tklbam-backup --quiet
EOF

And make sure it's executable:

chmod +x /etc/cron.daily/tklbam-backup