Ritchie McGrath's picture

So I'm using TurnKey Linux File server and I was deleting my UNIX user. When I pressed delete, I accidently pressed delete all the users files. As soon as i did this, I unplugged and turned off the server. I can't read the hard drive when I take it out and put it into another PC (I assume turnkey uses a weird file system) and when I turn the server back on, it just say there is a SMB daemon task in progress and I can't cancel it. how can I cancel this process? Is it possible for me to get my files back?


Please, any help appreciated

Jeremy Davis's picture

Ok strap yourself in, this is gonna be a big one...!

First up, you probably already realise this, but this is why backups are so important! It's also well worth noting, that data loss, is a matter of when, not if! Whilst SSDs are (at least on face value) more reliable that "spinney" hard drives, they too can suffer from data loss and corruption. So if you only store your important files in a single location, losing them at some point is inevitable...

Regarding, TurnKey using a "weird file system", no by default TurnKey uses the normal Linux filesystem, ext4 (or possibly ext3 if it's a really old install). Although it does by default also leverage LVM (this can be chosen at install, but is the default option). ext3/4 drivers are available for other OS, but the support isn't necessarily that great. ALso, I'm not sure about LVM support on other OS, so that may add some complexity if using an alternate OS.

Recovery of the files is quite likely considering that it sounds like it was accidental deletion, rather than hardware failure (and even with hardware failure, recovery is sometimes possible). This is because, under normal circumstance, when a file is deleted, it isn't actually deleted on the disk. Just the reference to the file in the disk's index is deleted. It's a bit like if a disk were a book, removing a chapter from the contents page. The content of the chapter still exists in the book, just you don't know how to go straight to it!

So whilst the file data still exists (at least until it's overwritten with new data), none of the normal tools will be able to see it. But as it's no longer referenced, the system assumes that it's free space and will happily overwrite it with new data (when writing data is next done). If the files were still open when deleted (and you knew the path to them) then you may have been able to recover them straight away, although depending on the volume of data, that may have been impractical; besides it's irrelevant now as you turned it off.

Despite that, stopping it was probably a good move! That is because, there is often lots of filesystem activity (such as writing log files, etc) even when you aren't creating new files. And deleted files (or at least parts of them) might be overwritten quite quickly.

So there are a few different ways to recover deleted files on ext filesystems, but the only ones I would recommend are all Linux based. So I would only recommend that you attempt recovery from another Linux system. There are a few reasons for that. One is that you'll have all the drivers necessary to access the filesystem and they'll be stable (important when recovering files). Also, it reduces the risk of accidentally overwriting the data you wish to recover. Windows (and possibly other OS) will try to automount any disks it finds that it can read, and metadata will be auto-written, perhaps wiping out some of your deleted files. With the exception of removable media on Desktop systems, Linux tends to only mount drives which you explicitly tell it to.

If you don't have another Linux system, then I suggest that you install one (TurnKey, Debian or Ubuntu would all be reasonable options, although any Linux should be near enough). I would personally go for a commandline only system (generally there will be no automounting at all) but YMMV. The tools I'm talking about are all commandline tools anyway, so a GUI isn't of any help... A live CD/USB is another option, but you'll still need plenty of free space on something you can write to to save the recovered files (assuming there are a few).

So you'll also need another drive, ideally as large as the one you wish to recover from, but at least as large as the files that you need to recover (to save the recovered files to). If you have a much larger drive spare (or multiple drives) then personally, I think that it's good practice to clone the disk first, so that worst case (if you muck something up) you can have another go at recovery. That is especially important if it's a physical failure (simply reading from the disk might corrupt data further) but in your case, probably isn't required (just nice if it's an option).

I use the dd tool for that job (should be available in any Linux by default) but will require you to make sure that you are copying from the right drive, to the right drive (if you get mixed up, you might accidentally erase the drive you wish to recover from, making file recovery MUCH harder, if not impossible). So be super careful (and I won't give any instructions here, you can google for them, although please feel free to ask for clarity if you wish to double check...).

Once you have cloned your disk (or using the original if you skip that step) you'll need to check whether it is using LVM or not. Assuming you are running from Linux and you are running as root (if not you may need to prefix 'sudo' for some/all of the commands I note for the rest of the post), the best way to do that is by running 'fdisk -l'. That should show you all the drives that are attached to your system and how they are partitioned. Here's an example of one of the systems I have handy:

Disk /dev/sdb: 77 GiB, 82678120448 bytes, 161480704 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: A81DEA40-46AD-49DD-8DF2-42942BFCE748

Device     Start       End   Sectors Size Type
/dev/sdb1   2048 161480670 161478623  77G Linux filesystem

Disk /dev/sda: 32 GiB, 34359738368 bytes, 67108864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x797b495f

Device     Boot Start      End  Sectors Size Id Type
/dev/sda1        2048 67106815 67104768  32G 8e Linux LVM

Disk /dev/mapper/turnkey-root: 24.8 GiB, 26621247488 bytes, 51994624 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/turnkey-swap_1: 4 GiB, 4290772992 bytes, 8380416 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

That shows one disk (/dev/sdb - 77 GiB) has one partition (/dev/sdb1) with a "Linux filesystem" (ext2/3/4, although in my case, almost certainly ext4 - see the line that starts '/dev/sdb1'). The other disk (/dev/sda - 36GiB) has one partition which is LVM (see the line that starts '/dev/sda1' and ends 'Linux LVM'). Finally, you can see that there are 2 "disks" in "/dev/mapper/", these are the LVs (logical volumes) from /dev/sda. You can check which ones are in use (and where they are mounted) run 'df -h'. Here's my output again:

Filesystem                Size  Used Avail Use% Mounted on
udev                      2.0G     0  2.0G   0% /dev
tmpfs                     396M   13M  383M   4% /run
/dev/mapper/turnkey-root   25G   14G  9.3G  60% /
tmpfs                     2.0G     0  2.0G   0% /dev/shm
tmpfs                     5.0M     0  5.0M   0% /run/lock
tmpfs                     2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/sdb1                  76G  5.6G   66G   8% /mnt

That shows that my 'turnkey-root' LV is mounted as the root filesystem (i.e. '/') so that the one my system is running from. The other drive (specifically the /dev/sdb1 partition) is mounted at '/mnt'. Ideally in your scenario, the drive that you wish to recover from is not mounted (although if you are recovering from a clone, then it doesn't matter quite so much). But regardless, Linux file recovery tools generally work on unmounted filesystems you'll need to mount it to recover files. AFAIK, the exception is if your disk is using LVM. AFAIK, you do need to mount the LV to recover files, but I recommend mounting it as read only.

If you are using LVM, mount it read only like this:

vgchange -ay VOLUME_NAME # e.g. in my case, it'd be turnkey-root
mkdir /media/fileserver-disk
mount /dev/mapper/VOLUME_NAME /media/fileserver-disk -o ro
# in my case, /dev/mapper/turnkey-root

Then first, I'd try 'extundelete'. It's likely not installed, so make sure it is:

apt update
apt install extundelete

Then change directory to the drive where you wish to restore the files to (if it's not the drive that has Linux installed, you may need to mount that one too; although this time, skip the '-o ro' bit to mount it read-write - and obviously mount it somewhere else!). Then run extundelete and hopefully you will get back your files!

For example, assuming I have an additional drive to recover files mounted as /media/recovered_files:

cd /media/recovered_files
extundelete /dev/mapper/VOLUME_NAME --restore-all

ALternatively, if you know the path where the deleted files were stored, e.g. /home/username, then you can exclude all other deleted files like this:

cd /media/recovered_files
extundelete /dev/mapper/VOLUME_NAME --restore-file /home/username/*

Once that's done, you should find all your recovered files in a directory named "RECOVERED_FILES".

If that didn't work, or there are still some files missing that you'd like to try recovering, then the next option is a tool called 'photorec'. It's named that because it started life as a SD card photo recovery tool, but does much more than that now! Check the list of different files it can recover here. In TurnKey/Debian/Ubuntu it's part of the 'testdisk' package and also likely isn't installed. To install:

apt install testdisk

Once installed, then launch it like this:

photorec /path/to/unmounted/volume

And follow the prompts... FWIW, here is a fairly detailed walkthrough.

Add new comment