Psilospiral's picture

I am trying to setup TKL Syncthing on a r720xd server running Proxmox VE 5.4-11.  I have the Syncthing CT installed and running.

On the r720xd, I have a ZFS pool created, which is currently only used for TKL File Server as a NAS.  All shares in TKL File Server are working great - r/w accessible by clients on the LAN.

The ZFS pool is mounted to the PVE host as /r720xd1

/r720xd1 has two subdirectories for two containers running under PVE:

/r720xd1/subvol-104-disk-0  # For TKL Syncthing

/r720xd1/subvol-108-disk-0  # For TKL File Server

subvol-108 was created when I added a mountpoint of /srv/storage for the TKL File Server.  This was done to utilize the ZFS file system as the storage for TKL File Server shares.

subvol-104 was created when I added a mountpoint /r720xd1/subvol-108-disk-0/syncthing .  This was done in an attempt to directly access the shares in my TKL File Server directory.

I logged in to the Syncthing webGUI from a client on the LAN and changed Actions>Settings>Default Folder Path to: r720xd1/subvol-108-disk-0/syncthing

When I try to create a new 'test' Folder to share via Synthing from the webGUI I receive:

Failed to create folder root directory mkdir /r720xd1/subvol-108-disk-0/syncthing/test: permission denied.

So I have to assume the default user 'syncthing' (or the entire 104 container) does not have permission to write to the /r720xd1/subvol-108-disk-0 - presumably because this is the subvol for a different container.  I am checking in the PVE forums to see if it is possible for an OS on one subvol to access another.

When TKL Syncthing is installed on an OS like PVE, what is the best practice for allowing TKL Syncthing to access the PVE OS host storage space?



Jeremy Davis's picture

I'm unfamiliar with ZFS, so I'm not sure how much assistance I can provide regarding that side of things. Although I would assume that from the context of what you are trying to achieve, the filesystem type is essentially irrelevant?! If I'm right, then hopefully I'll be able to assist.

I'll assume that your Syncthing appliance is a container. If it's a full VM, then you'll need to go a different route...

The first thing I would note is that AFAIK, from the Proxmox side, it's not possible to mount a disk/partition/directory that belongs to one container directly into another container. It is possible using generic tools (e.g. Samba, SSHFS, NFS, etc - that would allow you to mount a remote filesystem within a VM or any other remote server), but I'm guessing that's not what you're wanting.

AFAIK, if you wish to have a common directory mounted within 2 (or more) containers, then what you are likely looking for is bind mount point. I.e. the storage is mounted to the Proxmox host, then it is bind mounted within both containers.

If I'm misunderstood and that's what you've done already, then my guess is that it's a permissions issue. To trouble shoot that, I'd first log into the Synthing server as root (either directly via SSH or from Proxmox via 'pct'). First thing to check is that the directory is mounted and exists within the container:

ls /r720xd1/subvol-108-disk-0/syncthing/

If that gives an error, then it seems that it's not mounted (or at least not mounted where you think it is).

If it does exist, then next I'd check that root has write access. I'd do that by trying to create a file. I.e.: assuming that the directory is mounted as you've noted, try this:

touch /r720xd1/subvol-108-disk-0/syncthing/testfile

If that doesn't work, then you may need to adjust the permissions and/or the way it's being mounted by Proxmox. If root doesn't have write access, then you'll almost certainly have issues allowing other users (e.g. the syncthing user) to have access.

If root has write access, then I'd guess that it's purely a syncthing permissions issue. As with most things Linux, there are a myriad of ways to resolve that, although my inclination would be to add the syncthing user to the samba users group (or whatever group owns the shared files). Note that the samba group won't exist by default on the Syncthing appliance, so you'll need to create it. It's worth noting, that under the hood, it's actually the GUID (group user ID) that matters, not the actual name. So you'll need to ensure that the 'samba" group (that you create) has the same GUID as the samba group on the fileserver.

Once you think that you have everything set up, you can check that the syncthing user has write access like this:

su - syncthing -s /bin/sh -c 'touch /r720xd1/subvol-108-disk-0/syncthing/testfile2'

If that completes without error and the file exists, then you should be good to go...

Psilospiral's picture


Yes, I am sure the problem is how I am attempting to reach/mount the drives - nothing specifically to do with ZFS itself.  I can use cd, cp, mv commands on the ZFS drive contents like any other.  It is also available at /r720xd1 on the PVE host.

You are correct: My TKL Syncthing appliance is a container, not a VM.  I only have two current containers (CTs) - TKL File Server (which I am growing more and more familiar with!) and now TKL Syncthing.

A Proxmox member suggested I install Syncthing in the same container (CT) as my TKL File Server as a work around.  However, I am leaning toward keeping each TKL appliance installed in its own CT separately.  My thought is that keeping each appliance separate decentralizes available services to guests, reducing the probability that multiple services could go down at once.  If my TKL File Server CT were to not boot for some reason, I would also loose my Syncthing services if they were both running within the same CT.  Your opinion here is appreciated.

Regarding Proxmox: one container subvol not able to mount another container's subvol.  I never got an answer to this in Proxmox forum, but it seems plausible from a security standpoint.  I'm not experienced enough with mounting to say for certain it cannot be done.  I can experiment with this more later, but for now it looks like it's not possible.

I cannot see the TKL File Server container's subvol from within the TKL Syncthing container.  So, it is not mounted (and thinking it is not possible per above).  Assuming this is not possible/desirable for security reasons, then I will need to mount my TKL File Server share '/syncthing' within the Syncthing CT.  The goal here is to utilize the ZFS pool currently bind mounted in TKL File Server CT to the host ZFS drive for storage so that I do not use the Syncthing CT root file system for storage.

I tried addressing this just like my TKL File Server: by adding a mount point to the Syncthing CT from within PVE webUI via Resources>Add>Mount Point, selecting the ZFS file system under storage, and setting the path to /home/syncthing - which is TKL Syncthing's default storage path.  However, then when I access the Syncthing webUI, I receive a 502 Bad Gateway error.  Deleting the mountpoint from within PVE webUI then allows me to login normally again in Syncthing webUI.  Adding the mount point again results in 502 again.  This also occurs when I try the 'mount --bind...' method within TKL Syncthing.

So now I am just trying to mount my TKL File Server share '/syncthing' from within the TKL Syncthing CT.  Then I will update the "default folder" in Syncthing to /mnt/syncthing and it should then utilize my TKL File Server for storage.  But I cannot get the TKL Syncthing CT to mount my /syncthing share!

I can read and write the /syncthing share from a Win guest on the LAN easily via Samba, so I just want to try mounting it from within TKL Syncthing too...

Within TKL Syncthing CLI:

	mkdir /mnt/syncthing
	mount /mnt/syncthing

And receive:

root@syncthing /mnt/syncthing# mount /mnt/syncthing
	mount: wrong fs type, bad option, bad superblock on, missing codepage or helper program, or other error
	(for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program)
	In some cases useful info is found in syslog - try dmesg | tail or so.

I saw nothing in dmesg|tail that looked relevant.

I also tried:

mount -t cifs /mnt/syncthing

And received the same error message.

When trying '-t samba' or '-t smbfs' I receive:

mount: unknown filesystem type 'samba' (or 'smbfs')

I have Options>Features>CIFS and NFS checked within the PVE webUI for the Syncthing CT - so I should be able to mount from within there just fine.  I am thinking my mount syntax is wrong for trying to mount the Samba share.  What should be the exact command?


Psilospiral's picture


Sorry for the verbose troubleshooting, but hopefully this will save somebody else the agony I am going through today!  FWIW, I plan to create a detailed tutorial on how to setup a PVE based TKL File Server NAS, along with Syncthing (and likely a few other appliances) when I have all of this ironed out.  I have a very long and detailed installation log!

First, the GOOD news:

I have successfully mounted my /syncthing TKL File Server share from within a separate container running TKL Syncthing!!!  Essentially, I am just trying to utilize the ZFS storage on the host PVE server to serve as a NAS via TKL File Server (which works great so far!).  Now I want to utilize this same ZFS storage (via TKL File Server) as a storage share for my Syncthing file syncs.  I have several hundred GiB of current Syncthing storage, so it needs to be accessable from the TKL File Server, not from within the small 8 GiB CT (obviously). To get these two CTs to talk, I have to first go through PVE:

First mount the share /syncthing within PVE webUI:
Enter the TKL File Server IP address
Enter username and password
Click Share and select the desired TKL File Server share (in my case, /syncthing)
Content: Disk image (Although this appears to only create a subdirectory called "image" under /syncthing, you cannot click "ok" to proceed without selecting something - so I accepted this default from the list).

This creates /mnt/pve/syncthing in the PVE host file system.
Browsing /mnt/pve/syncthing in PVE CLI displays files shared as /syncthing in TKL File Server, as expected.  (I now have a /backuppve share hosted by the TKL File Server CT to backup all of my CTs and VMs via PVE CLI!)

Note the ID# of the Syncthing CT within PVE.
nano /etc/pve/lxc/[ID#].conf
Add to the last line:
mp0: /mnt/pve/syncthing,mp=/mnt/syncthing

Restart the Syncthing CT and in its CLI:
ls /mnt/syncthing
will list the comtents of the shared directory /syncthing from the TKL File Server CT.  SUCCESS!

I've reached this point by finding help witin the PVE forum; specifically:

Now the BAD news: I still have permission issues related to Syncthing.

From within the TKL Syncthing CLI, I can

touch /mnt/syncthing testfromwithinsyncthingcli.txt

and the file creates without a problem and is visible in WebDAV under TKL File Server.  (Which I thought was total success...)

However, when I go to the Syncthing webUI and try to create a shared folder utilizing the new /mnt/syncthing default storage directory, I receive the following 'Notice' in the webUI:

	Failed to create folder root directory mkdir /mnt/syncthing/testfolderfromwithinsyncthing: permission denied.
	Error on folder testfolderfromwithinsyncthing: folder path missing

When I delete the Folder ID portion under 'Folder Path' when attempting to add a new folder, it even displays the "/mnt/syncthing/images/" subdirectory created within PVE.  So I know it is connecting and seeing the shared directory properly...  It's just not letting Syncthing write to it.  This appears to be isolated only to the Syncthing webUI user, because the root user in the TKL Syncthing CT can write to the same mount point with no issue. 

What am I missing?

Jeremy Davis's picture

It sounds like you are so close... Great work on the troubleshooting and it sounds like you've had some significant progress, so well done on that.

So essentially it sounds like you are now simply hitting a permissions issue when trying to write to the directory as the Syncthing user. As a general rule, assuming a filesystem is mounted read/write, root always has full access to the filesystem (there are some edge/corner cases, but they require explicit configuration so we'll ignore the possibility of those as it seems irrelevant here).

(As per pretty much all Linux problems) There are a few workarounds you could employ. Although probably the first thing to do would be to check who owns the directory when it's mounted. You can do that with 'ls', using the '-l' (long list) switch (FWIW I often also add in the '-A' (almost all) switch which will also show hidden files). E.g. on my local system (my work laptop) here's is what I see when I list the /mnt directory:

user@ninjux ~$ ls -lA /mnt
total 4
drwxr-xr-x 2 user root 4096 Jun 25 08:33 isos

The first set of characters 'drwxr-xr-x' in this case means 'd' for directory, the 'rwxr-xr-x' is the permissions, 3 sets of 3 characters referring to user ('rwx'), group ('r-x') and everybody ('r-x'). These letters represent read, write and execute. FWIW, these map to the octal (numerical) permission values which you often see, r=4, w=2, x=1. The octal permissions are calculated by adding the values - e.g. rwx = 7; r-x = 5, etc. The other values you'll need to consider from my example are 'user' and 'root' (i.e. user and group; the "normal" user account on my system is 'user' and this particular directory is owned by the 'root' group).

So if you run 'ls -lA /mnt', that will show the user and group ownership of files/directories contained within /mnt and also show the permissions associated. As I think I noted previously, it's actually the ID of the user/group that is important rather than the name, but if there is a UID/GID that matches a user on the system 'ls' will display the username/group. If a user ID is not known to the local system, then 'ls' will fallback to UID/GID.

So once you have that information, then you can adjust the permissions to allow the syncthing user to access them. Depending on how you want to have things configured, you can adjust the directory permissions and/or the syncthing user account (e.g. do you want read only access via Samba, with Syncthing having the primary ownership? Or do you want something else?). For example, you could change the primary syncthing user group to match the current permissions (and ensure that the group has full read/write access).

I hope that heads you in the right direction...

Psilospiral's picture


Well, I have certainly had a fun time paging through 'man mount', 'man chown', 'man id', 'man gpasswd', 'man usermod' and a host of other nix commands lately!  I appreciate the ownership primer...

From within the TKL Syncthing container CLI:

	root@syncthing /home/syncthing# ls -Al /home/syncthing/Sync
	total 4
	drwxr-xr-x 2 syncthing syncthing 4096 Aug 24 20:13 .stfolder

So I can see that user 'syncthing' in group 'syncthing' owns /home/syncthing/Sync .  My bind mounted TKL file server share /syncthing is mounted to /mnt/syncthing , where I am having permission errors.  So:

	root@syncthing /home/syncthing# ls -Al /mnt/syncthing
	total 2048
	drwxr-xr-x 2 root root 0 Aug 25 18:50 images
	-rwxr-xr-x 1 root root 0 Aug 25 18:35 touchtestfrompveroot.txt
	-rwxr-xr-x 1 root root 0 Aug 25 19:10 touchtestfromsyncthingroot.txt

Fair enough - user 'root' seems to own /mnt/syncthing and not user 'syncthing'.  Well, let's try changing that:

	chown syncthing:syncthing /mnt/syncthing

results in no error messages - which makes me thing I'm on to something.  But adding a folder in Syncthing webUI fails again.  so:

	root@syncthing /home/syncthing# ls -Al /mnt/syncthing
	total 2048
	drwxr-xr-x 2 root root 0 Aug 25 18:50 images
	-rwxr-xr-x 1 root root 0 Aug 25 18:35 touchtestfrompveroot.txt
	-rwxr-xr-x 1 root root 0 Aug 25 19:10 touchtestfromsyncthingroot.txt

Looks like my chown command had no effect.

Even though I know this is not best practice, for troubleshooting sake I decided to add user 'syncthing' to the root group and make it the primary:

	root@syncthing /home/syncthing# usermod -a -G root syncthing
	root@syncthing /home/syncthing# groups syncthing
	syncthing : syncthing root
	root@syncthing /home/syncthing# usermod -g root syncthing
	syncthing : root

Trying to add a folder again in syncthing webUI results in the same 'permission denied' error.  Rrrrrrr.  Let's get user 'syncthing' out of the 'root' group.

	root@syncthing /home/syncthing# gpasswd -d syncthing root
	Removing user syncthing from group root
	root@syncthing /home/syncthing# groups syncthing
	syncthing : syncthing

I changed ownership of syncthing's default local directory of /home/syncthing/Sync to user 'root' successfully and then back to 'syncthing' without issue.  So, I'm getting the feeling that I need to change something in PVE again!  I'll be back.


Jeremy Davis's picture

Unless you explicitly tell it to recursively take ownership, it will only affect the immediate directory. In the case of chown, the switch you are looking for is '-R'. So give this a go:

chown -R syncthing:syncthing /mnt/syncthing

Re-reading your post, it's possibly worth adding that the reason why it didn't work when you added syncthing to the root group, is that according to what you've posted, the group permissions for that directory are 'r-x' i.e. read/execute (no write).

I agree that that isn't a good idea so try the recursive option on chown first. But FWIW, to allow the root group to have write access, you'd need to add group write permissions. In this case it would go something like this:

chmod -R g+w /mnt/syncthing

I.e. recursively ('-R') add ('+') group ('g') write ('w') permissions to /mnt/syncthing. As we agree though, that isn't the best plan. I just wanted to add that for completeness...

Psilospiral's picture


It appears I can neither alter ownership of /mnt/syncthing nor grant write access with chown:

root@syncthing /mnt/syncthing# ls -Al
total 2048
drwxr-xr-x 2 root root 0 Aug 25 18:50 images
-rwxr-xr-x 1 root root 0 Aug 25 18:35 testfrompveroot.txt
-rwxr-xr-x 1 root root 0 Aug 25 19:10 testfromsyncthingroot.txt
root@syncthing /mnt/syncthing# chown -R syncthing:syncthing /mnt/syncthing
root@syncthing /mnt/syncthing# ls -Al
total 2048
drwxr-xr-x 2 root root 0 Aug 25 18:50 images
-rwxr-xr-x 1 root root 0 Aug 25 18:35 testfrompveroot.txt
-rwxr-xr-x 1 root root 0 Aug 25 19:10 testfromsyncthingroot.txt
root@syncthing /mnt/syncthing# chmod -R g+w /mnt/syncthing
root@syncthing /mnt/syncthing# ls -Al
total 2048
drwxr-xr-x 2 root root 0 Aug 25 18:50 images
-rwxr-xr-x 1 root root 0 Aug 25 18:35 testfrompveroot.txt
-rwxr-xr-x 1 root root 0 Aug 25 19:10 testfromsyncthingroot.txt

So if I cannot grant write permission or change ownership as root, this must be controlled at the PVE level, right?

Jeremy Davis's picture

That does seem strange! TBH, I'm a little surprised by that. FWIW, I currently have a local fileserver running on Proxmox which has a separate HDD mounted within the container for storage. It's a independent physical drive, with LVM setup and the LV mounted in Proxmox (as /media/fileserver-storage FWIW). That Proxmox directory is then directly mounted at /srv/storage within my LXC fileserver container. Permissions within that work exactly as I have described and would expect a similar result on your end. Although perhaps there is some detail I've missed?

Out of interest, what happens if you create a new subfolder within the mount from within the container. And then try adjusting the ownership of that? E.g.:

mkdir /mnt/syncthing/data
chown -R syncthing:syncthing /mnt/syncthing/data

The only other thing that occurs to me is that my container is "privileged". If your is "unprivileged", then that may explain things!? FWIW, in an "unprivileged" container, the root user inside the container, does not map to the root user outside the container. Obviously, that provides an additional layer of security (a root user inside a "privileged" container potentially has some degree of influence over the host). So if that's the case perhaps you need to grant ownership to a non-root user outside the container?

Although having said that, if I understand correctly, you previously noted that you could create files within the mount as the root user, but just not as the syncthing user...

Psilospiral's picture


root@syncthing /mnt/syncthing# ls -hAl 
total 2.0M 
drwxr-xr-x 2 root root 0 Aug 25 18:50 images 
-rwxr-xr-x 1 root root 0 Aug 25 18:35 testfrompveroot.txt 
-rwxr-xr-x 1 root root 0 Aug 25 19:10 testfromsyncthingroot.txt 
root@syncthing /mnt/syncthing# mkdir data 
root@syncthing /mnt/syncthing# ls -hAl 
total 2.0M 
drwxr-xr-x 2 root root 0 Aug 27 01:11 data 
drwxr-xr-x 2 root root 0 Aug 25 18:50 images 
-rwxr-xr-x 1 root root 0 Aug 25 18:35 testfrompveroot.txt 
-rwxr-xr-x 1 root root 0 Aug 25 19:10 testfromsyncthingroot.txt 
root@syncthing /mnt/syncthing# chown -R syncthing:syncthing /mnt/syncthing/data 
root@syncthing /mnt/syncthing# ls -hAl 
total 2.0M 
drwxr-xr-x 2 root root 0 Aug 27 01:11 data 
drwxr-xr-x 2 root root 0 Aug 25 18:50 images 
-rwxr-xr-x 1 root root 0 Aug 25 18:35 testfrompveroot.txt 
-rwxr-xr-x 1 root root 0 Aug 25 19:10 testfromsyncthingroot.txt 

No change!  My TKL Syncthing container has Unprivileged container: No


Jeremy Davis's picture

I'm quite perplexed by this issue TBH. A quick google suggests that you are not alone, and unfortunately, no one has has posted an answer there... Other than usage of ZFS, the other factor that may contribute, is that both my Proxmox host and my container are quite old, so perhaps there is some legacy config which has endured the updates and continues to allow it to work in my scenario?

Anyway, the next thing that I would be inclined to try is granting ownership to the syncthing user on the PVE host. As it's unlikely that the syncthing user exists on the PVE host (and even if it did, it may not align with the UID within the container; as I noted previously, it's actually the UID/GID that matters, not the name).

So you'll need to get the syncthing user UID/GID within the container. To do that, when logged into your syncthing container:

id syncthing

That should return a string somewhat like this (using my desktop user account 'user' as an example, note that I edited it to remove all the additional groups that 'user' belongs to):

uid=1000(user) gid=1000(user) groups=1000(user)

So in my case, the UID is 1000 (GID is also 1000). It's quite likely that your UID/GID is the same (UIDs below 1000 are "reserved" for system use, so the first non-system user is often 1000), but if not substitute your results. To use my UID/GID as an example (i.e. 1000). On your PVE host, try chowning the directory there. I.e.:

chown -R 1000:1000 /path/to/mnt/syncthing

Then check it out within the container. Hopefully it's now owned by the synthing user... If not, TBH I'm out of ideas...

Psilospiral's picture


User syncthing is UID/GID 999.  When I 'chown -R 999:999 /mnt/syncthing' , and then ls -hl, root still owns the directory!  So it occured to me, let me try to mount the directory form within /home/syncthing instead!  I switched to user 'syncthing' and made a new directory to mount from within the user syncthing home:

root@syncthing ~# su - syncthing
$ pwd
$ mkdir share
$ ls -hAl
total 36K
-rw-r--r-- 1 syncthing syncthing  220 May 15  2017 .bash_logout
-rw-r--r-- 1 syncthing syncthing 2.1K Oct 17  2018 .bashrc
drwxr-xr-x 2 syncthing syncthing 4.0K Oct 17  2018 .bashrc.d
drwx------ 3 syncthing syncthing 4.0K Aug 27 19:11 .config
-rw-r--r-- 1 syncthing syncthing  746 Oct 17  2018 .profile
drwxr-xr-x 2 syncthing syncthing 4.0K Oct 17  2018 .profile.d
drwx------ 2 syncthing syncthing 4.0K Oct 17  2018 .ssh
drwxr-xr-x 3 syncthing syncthing 4.0K Aug 27 19:11 Sync
drwxr-xr-x 2 syncthing syncthing 4.0K Aug 27 19:20 share

After this, I changed my mountpoint in the PVE host /etc/fstab from 'mp0: /mnt/pve/syncthing,mp=/mnt/syncthing' to 'mp0: /mnt/pve/syncthing,mp=/home/syncthing/share and restarted the container.

Then in syncthing CLI:

root@syncthing ~# su - syncthing
$ ls -hAl
total 36K
-rw-r--r-- 1 syncthing syncthing  220 May 15  2017 .bash_logout
-rw-r--r-- 1 syncthing syncthing 2.1K Oct 17  2018 .bashrc
drwxr-xr-x 2 syncthing syncthing 4.0K Oct 17  2018 .bashrc.d
drwx------ 3 syncthing syncthing 4.0K Aug 27 19:11 .config
-rw-r--r-- 1 syncthing syncthing  746 Oct 17  2018 .profile
drwxr-xr-x 2 syncthing syncthing 4.0K Oct 17  2018 .profile.d
drwx------ 2 syncthing syncthing 4.0K Oct 17  2018 .ssh
drwxr-xr-x 3 syncthing syncthing 4.0K Aug 27 19:11 Sync
drwxr-xr-x 2 root      root      4.0K Aug 27 19:20 share

Ownership of user syncthing's /home/syncthing/share is changed to root on reboot!

Is there a way I can remove the syncthing user and let the syncthing app run from user root in the TKL Syncthing appliance?  (At least for testing???)


Jeremy Davis's picture

As I was reading through, I thought that you had it! But then I got to the second last line... :(

I just double checked my Proxmox server and 100% confirmed how it is set up. Here is my Container config (from PVE):

arch: amd64
cores: 4
hostname: dafileserver
memory: 2048
mp0: /media/fileserver-storage,mp=/srv/storage
net0: name=eth0,bridge=vmbr0,gw=,hwaddr=D6:4D:AD:7E:B2:49,ip=,type=veth
onboot: 1
ostype: debian
protection: 1
rootfs: local-lvm:vm-108-disk-1,size=8G
swap: 1024

The relevant line from Proxmox fstab ('cat /etc/fstab | grep storage' - IIRC I didn't manually add that fstab entry, I think it was done by Proxmox when I added it via the WebUI; but I don't recall, it was so long ago...):

/dev/xtra/storage /media/fileserver-storage ext4 errors=remount-ro 0 1

Note that 'mount' is showing the LVM location using the "mapper" location which may be a little confusing, but FWIW /dev/xtra/storage and /dev/mapper/xtra-storage are (symlinks to) the same actual device. The output of 'mount | grep storage' (on Proxmox):

/dev/mapper/xtra-storage on /media/fileserver-storage type ext4 (rw,relatime,errors=remount-ro,data=ordered)

The mount point on PVE ('ls -lA /media | grep fileserver-storage'):

drwxr-xr-x  7 root root 4096 Oct  7  2012 fileserver-storage

The mount point within the container ('ls -lA /srv'):

drwxrwxr-x 51 media media 4096 Aug 10 18:24 storage

I have been doing some maintenance relatively recently and I noticed the datestamp on my storage dir is pretty recent, so I rebooted the container to double check that it wasn't just because I hadn't rebooted recently, but the /srv/directory is still owned by media after a reboot...

FWIW the fstab of my fileserver is empty, so it's not something weird going on there.

So TBH, I'm a bit lost... Perhaps ZFS is a factor after all?

Re your question on running syncthing as root, (as per your note re testing) obviously that's not a great idea (especially within a privileged container), but I'm sure that it'd be possible. I just had a quick browse through the Syncthing appliance build code and whilst we create the user at build time, it doesn't appear that we are explicitly telling syncthing to use it, so you'll probably have to dig around in the syncthing config to find out how to run it as root.

This really does seem to be something weird with the way Promxox/LXC is doing things. Did you have any joy asking on the Proxmox forums?

Having said that, seeing as you did manage to get the permissions set as you desired, just that they didn't survive a reboot, perhaps a workaround would be to create a "service" which sets the desired permissions at boot time? If I were to do that, it would likely be a bash script to set the permissions at build time, and a systemd service file that calls that script (before syncthing starts). I'd have to double check the systemd docs to recall exactly how to do that, but OTTOMH (completely untested) it might look something like this:

Description=Fileserver storage permissions hack



And save it as /etc/systemd/system/perms-fix.service (with the bash script that fixes the permissions saved as /usr/local/bin/fix-perms). The bash script needs to be executable (i.e. 'chmod +x /usr/local/bin/fix-perms') and the service needs to be enabled ('systemctl enable perms-fix.service'). Also, I'm not 100% sure on the name of the syncthing service, you'd need to double check that is right.

But TBH, I'm still at a loss as to why this is so hard... IMO it should "just work"!

Psilospiral's picture


I was thinking it was something specific to the TKL Syncthing appliance, so I just finished installing debian-9.0-standard_9.7-1.amd64.tar.gz in a fresh CT, adding user 'syncthing', updating the password, apt-get install syncthing, ran syncthing, systemctl enable syncthing@syncthing.service, start, status to verify it was all running, modified the /home/syncthing/.config/syncthing/config.xml <gui enabled...> to to go to it on my LAN, added the mountpoint in PVE config for the CT to utilize the default /home/syncthing/Sync, checked that user syncthing owned it, verified I was seeimg files within the actual share, rebooted the CT - and guess what?  The ownership goes back to root and syncthing cannot write to the share.  SO, it doesn't look like it is anything specific to TKL Syncthing either. 

I have posted on PVE forum and will report back if I get anywhere.  I've got a headache now and calling it quits for the night!


Jeremy Davis's picture

I was fairly certain that it was a PVE/LXC issue (rather than TurnKey specific) but thanks for confirming that... As I also noted, perhaps ZFS is a factor after all? Maybe it's worth trying to mount a "normal" directory from the host (rather than a separate ZFS path)? Even if it still doesn't work, at least then you are narrowing down the possible causes.

The only other thing that has occurred to me is that I haven't updated my Proxmox server for a while (I'm currently running PVE v5.4; based on Debian 9/Stretch). If you're using a different (newer) version, then perhaps it's something that has been introduced relatively recently and I'll have to contend with it too (once I upgrade)?

Re your Proxmox thread, please feel free to cross post a link to that here if you wish.

I don't blame you for having a headache; this whole ordeal sounds like one big headache! Hope you feel better soon...

Psilospiral's picture


Sorry for the delay - I've been out a week on vacation and finally got back around to working on this issue again.  I decided to take a fresh look at mounting the cifs share within a standard qemu VM instead of the lxc CT.  PVE has no special "Features" settings for VMs to allow cifs shares, so I thought it would be a good way to troubleshoot as I have had no issues mounting my TKL File Server shares on physical Win10 and Debian guests on my LAN.

During my troubleshooting, I also had difficulty in mounting the cifs shares, but finally got things working with this entry in my fstab:

//<ip of TKL File Server>/syncthing /mnt/syncthing cifs username=<username>,password=<password>,noperm,rw 0 0

SO, I had to try the same thing on a new TKL Syncthing CT.  The interesting thing here is that the share does not even need to be mounted on the PVE host!  Further, the only modification I had to make on the PVE side was Datacenter>(node name)>syncthing CT>Options>Features>CIFS:checked.  Then, I simply entered the above line in my fstab, tested it with 'mount -a' and was able to touch /mnt/syncthing/root-test as user 'root' as well as touch /mnt/syncthing/syncthing-test as user 'syncthing'!  There were no modifications required to the original /etc/pve/lxc/xxx.conf file, (such as a custom mountpoint).

Now, the only thing I did different on the TKL File Server side was selecting the "syncthing" user as "Create with owner:" and "users" in "Create with group" when initially creating the file share.  SO, to test, I went back, deleted the share, deleted the /srv/storage/syncthing directory completely and recreated the syncthing share again with the default owner and group, which are both root.  I then made the share writable, clicked "Restart Samba Servers" as well as restarted the TKL Syncthing CT.  Guess what?  A 'touch /mnt/syncthing/root-test' failed with Permission denied.  Of course, the same for user 'syncthing' as well.  What did I have to do to fix it without deleting the share and starting over?  On CLI in TKL File Server:

chown -R syncthing:users /srv/storage/syncthing

Upon 'ls -hal /srv/storage/syncthing' I could see that user 'syncthing' owned the share again and as soon as I tried a touch test form within the TKL Syncthing CT, I was back in business.   FINALLY.

So, to recap, from the PVE side only Options>Features>CIFS:checked needs to be done to allow the TKL Syncthing CT to mount the TKL File Server CT share.  In the TKL Syncthing CT, include the mount line above in /etc/fstab.  In TKL File Server, make sure the user 'syncthing' is created and included as the owner (even though on the TKL Syncthing side it's UID=999 and on the TKL File Server side it's 1000 or above) when creating the share (or chown it later).

'noperm' was necessary in the mount line.  I tried altering the 'syncthing' user UID=999 in TKL File Server, as well as 'usermod -u 1000 syncthing' in TKL Syncthing to match up the UIDs on both sides, but ran into the permission denied error every time.

So for now, I'm living with noperm, UID's not matching exactly, and finally obtaining rw access on my share!

Jeremy Davis's picture

Glad to hear that you finally found a solution that works for you!

Regarding use of CIFS shares, that's a pretty clever way of working around it! :) TBH, use of them never occurs to me because I don't have any Windows computers on my network. But now I think about it, your experience makes sense. So long as the samba-client is installed on your Syncthing server, then what you've discovered adds up. Let me explain a little...

Samba CIFS (or technically more correctly as of Samba4; "Server Message Block v3" - i.e. SMB) is a reverse engineering of Windows SMB3 protocol and it's essentially a protocol that provides a "network filesystem". So now thinking about it, it doesn't surprise me that it doesn't need to be mounted on the host. After all, it's a network filesystem! So any clients on the network can access it - assuming they have the appropriate credentials as per the authentication configured on the SMB server; Samba in this case.

As it's not a native Linux protocol, it knows nothing about Linux users and UIDs/GIDs (as you've noted). Samba3 worked around that by support for linking Samba (SMB) users to Linux users. Much of this functionality made it into Samba4 too (what we leverage in the Fileserver appliance). Although it's worth noting that the part that allowed synchronisation of Linux and SMB user passwords has been removed (it had some quite serious security issues which couldn't be easily resolved). So there is currently not an easy way to syncronise Linux/SMB password (after creation). The way that Samba4 works around that issue by default, is to keep them completely separate. On modern Samba4 systems with AD style domain integration (e.g. how our Domain Controller appliance is configured; the Fileserver still uses the older "stand alone" Samba3 style SMB setup) all the SMB users are mapped to a single Linux user and ACLs (essentially a much more fine grained filesystem permissions system) is used to allow/block access to filesystem objects.

So assuming that you installed the samba client software on your Syncthing server, and have a SMB syncthing user that maps to the SMB syncthing user, that would explain why the UIDs/GIDs don't match up, but it "just works".

Good stuff! :)

Psilospiral's picture


I must have failed to mention that I 'apt-get install cifs-utils -y' on the TKL Syncthing CT before attempting to mount the cifs share - otherwise I would not have gotten very far!

In addition, I failed to mention that when I was altering the UID number of user 'syncthing' in the TKL Syncthing CT, as well as altering the UID number of user 'syncthing' in TKL File Server CT, that I was doing so without the 'noperm' option entered in my TKL Syncthing /etc/fstab.  I wanted to see if I could obtain rw access without the noperm option, but nothing I tried worked.  This was with a privileged CT, so maybe if I complete a backup and restore it as an unprivileged CT it may react differently...

Until I have time to play with that possibility, it is good to know that the below line placed in the /etc/fstab of any privileged TKL Syncthing lxc container installation will allow the appliance to utilize rw access of a cifs "Samba" share served by a TKL File Server CT - all running on a PVE host:

//<ip of TKL File Server>/syncthing /mnt/syncthing cifs username=<username>,password=<password>,noperm,rw 0 0

I presume this will work just as well with all other privileged TKL CT appliances on PVE as well, so long as the username has write access to the share configured properly in TKL File Server and the username and password are entered correctly.  Nothing at all needs to be done on the PVE side with the single exception of clicking the "CIFS" feature 'on' in the CT's Options>Features section!

I will likely be experimenting with ZoneMinder or OpenVPN next...

A big thanks again for all your help.  This has been a great learning experience!

Add new comment