James's picture
 
I'm using a wordpress VM appliance and I "think" I told it to do the exact same update.
 
A few months ago when having this issue, I read over the posts on how to attempt to update TKL and not have a boot issue. Eventually, I got it to work. What I didn't see in the forums was how to RECOVER/RESTORE an appliance with this issue. Now I'm in a situation where there's content which hasn't yet been backed up. I need to recover it.
 
I'm a Linux Newbie. I've copied my VM so that I can use it for data retrieval. So far, I've tried using Knoppix and Ubuntu LiveCD in hopes to see the HDD files and possibly manually recover/restore the Wordpress site onto another VM. Ubuntu 11.10 doesn't even load (possibly an issue with that and Vmserver).
 
I can now view drive 'dm-0' using Knoppix and see the wordpress files. I can't acces 'var/lib/mysql'. I keep getting a "Permission denied" error.
 
I've also tried using ExploreFS and Linux Reader to view the drives but everytime I try and open the partitions, I get read errors.

Some threads mentioned doing a chroot with udev using a LiveCD to roll back the update. I have no clue what that means or how to properly do it.
 
Can someone give me some help on this? How do I grab the mysql database and migrate it over to a new TKL Wordpress appliance?
 
I've also attached a screen shot of the Linux boot error.
Forum: 
James's picture

Ok, I found the answer and how to recover the system after this horrid update. Below, I've pasted the text from another forum. Thank you shakaran for this one.

https://bugs.launchpad.net/ubuntu/+source/udev/+bug/433943

I received 2 errors from the below solution. But still, was able to get my VM working.

First Issue

Once I got to Step 6:

sudo cp /etc/resolv.conf /media/karmic/etc/resolv.conf

The error I received from the above command was:

cp: not writing through dangling symlink" and then the directory/file which I believe was the 'media/karmic/etc/resolv.conf

I couldn't figure anyway around this error and ignored it.

Second Issue

The second error I received was at Step 8 which command was:

sudo chroot /media/karmic apt-get dist-upgrade

I don't have the exact text from this error...Sorry.....I remember it asking me to do 'dpkg configure -a'. Unfortunately, I can't remember the full command I had to use to resolve this issue. Tomorrow, I'll follow the below steps again and update this part of the solution.

Once I rebooted the VM, TKL booted up just fine and I was able to retrieve all of my data!

1. - Download/grab a live cd.

2. - Start live cd and when it loads the desktop open a gnome-terminal.

3. - Create this dirs:

sudo mkdir /media/karmic
sudo mkdir /media/karmic/proc /media/karmic/dev /media/karmic/etc

4. - Mount your linux partition (for me sda6):

sudo mount /dev/sda6 /media/karmic

5. - Bind the dirs:

sudo mount -o bind /proc /media/karmic/proc
sudo mount -o bind /dev /media/karmic/dev/
sudo mount -o bind /dev/pts /media/karmic/dev/pts

6. - Copy this file

sudo cp /etc/resolv.conf /media/karmic/etc/resolv.conf

7. - Update your real linux partition with chroot:

sudo chroot /media/karmic apt-get update

8. - Upgrade your real linux partition with chroot:

sudo chroot /media/karmic apt-get dist-upgrade

9. - If you have some broken package:

sudo chroot /media/karmic apt-get -f install

10. - Reboot and voilá! Fixed for me.

James's picture

UPDATE

Ok, so I followed the instructions again so I may list the exact issues I had and how I overcame them. Please be sure that you can connect your system to the internet. I chose to use NAT knowing that the system would get an IP address via DHCP.

1. My linux partition within Knoppix was 'dm-0' (zero not the letter 'o'). Be sure to verify which partition belongs to your TKL.

2. When having to perform:
sudo cp /etc/resolv.conf /media/karmic/etc/resolv.conf
I received an error of:
cp: not writing through dangling symlink '/media/karmic/etc/resolv.conf'
I ended up ignoring this error because I didn't know what it meant or how to fix it.

3. When having to perform:
sudo chroot /media/karmic apt-get update
I received a whole bunch of errors stating that it couldn't fetch or download files. I was suggested to do 'sudo dpkg --configure -a' to correct the problem. So, I performed
sudo chroot /media/karmic dpkg --configure -a
I was then prompted with a new screen (blue background/black text) asking me if I wanted to replace the files and I did so. Once it was finished I performed the previous step 'sudo chroot /media/karmic apt-get update' and that went through just fine.

And that's that! Once I got through those 3 hurdles and finished the steps, a reboot allowed TKL to boot up properly!

Jeremy Davis's picture

Sorry to be a bit slow but glad you fixed it and thanks for posting your full solution. Your issue is a more intense version of the issue you link to in your OP. I suspect others have not come across this one (as I haven't personally either) so good job on fixing it!

The only further recommendation I would make is to put a hold on udev so it doesn't happen again if udev gets updated:

echo udev hold | dpkg --set-selections

Also I tidied up your middle post too.

James's picture

Jeremy,

Thanks for the response.

What do you mean by, more intense version? I thought the issues were the same?

Could you explain to me why the udev update fails and caused this issue? Do you think the developers of udev know about this issue? It's seemed to have plagued people for quite sometime.

If I put a hold on udev, does this mean that I can turn on automatic updates and should be fine in the future or is it still probably better to do manual updates? Perhaps, update a dummy TKL just to see how that reacts to an update?

With Windows, I know there's an option to install Windows ontop of an existing install. Could I have done this with TKL? I'm not sure if I saw this option after booting from the TKL Wordpress ISO.

Lastly, do you know why I couldn't use these Windows Linux partition readers to view the TKL disk drive? That really through me off.

Thanks!

Jeremy Davis's picture

When I initially encountered this bug (and the few times since that I've run apt-get upgrade) I rebooted as a step in resolving the issue. I never had the issues you had trying to get your appliance to boot after experiencing these issues so I consider it a more intense version - or probably more correctly, a more intense result of this bug.

Why issue occurs: TBH I have no idea. For some reason the udev process hangs when trying to restart (after installing an updated version). And yes the upstream maintainer (Ubuntu) know about it - have a look at the bug here and you can see that it also "Affects udev (Ubuntu)". It seems that they haven't fixed it yet.

Auto updates: Auto security updates are already installed in TKL every day by default. IMO there is generally not any reason to apply other updates (unless of course you have a reason - eg find a bug in a package and want to check if it's been fixed). Personally, as a general rule with Linux servers I don't apply updates (other than security ones) and I certainly wouldn't recommend auto applying anything other than security updates. If it's not a mission critical server (eg a dev box running locally) perhaps it wouldn't be such an issue, but generally if it's not broken, why fix it!? Linux land is a very different place to Windows!

Install "on top": Not 100% sure what you are referring to here, but I'll assume that you mean like a "recovery install' (I don't recall the proper name either) where you basically reinstall Windows but without losing all of your data. FYI in my experience that has a habit of leaving you with a fairly broken system - the core OS is repaired and user data remains, but often apps are broken. Better to just backup your data and install Linux! :) Sorry I digress... No, TKL does not support that sort of (unreliable IMO) method of reinstall (part of the TKL install partitions the HDD and thus destroys all data already on there). If you use TKLBAM though a clean reinstall with all your data restored is pretty easy.

"Windows Linux partition readers": TBH I have no idea what you are referring to here. I assume you have some Win app that can read some Linux filesystems? TKL uses a new(ish) software filesystem technology called Logical Volume Management (LVM) which does things quite differently to traditional filesystems. It's sort of (almost) a little bit like RAID (but not really). One of the advantages is that allows adding of additional harddrives to expand the system without needing to alter anything dramatically (ie like if you could add another HDD to the C drive in windows). It also allows taking 'snapshots' of the filesystem of a server (even if it running). Hopefully TKL will take greater advantage of this in the future (currently this functionality is not used by default and not documented by TKL). Incidentally, most modern versions of Linux should be able to read the LVM filesystem no worries.

James's picture

Jeremy

Sorry for the late response. You are right about what I meant about install "on top". I did not want to use the repaired system as a solution but instead, for recovery. I did not know how to pull the wordpress files or database from the damaged system. Recovering the OS I believed should had been an option, much like Windows has. Obviously, certain services may not have contined to work but as long as I could get to the data, that was fine with me. 

In terms of "Windows Linux partition readers", you were right again! As an alternative method to accessing my data, I wanted to slave my virtual drive onto another system (Linux or Windows). When doing it with Linux, I couldn't read the LVM so I was installing/evaluating these different Linux partition readers to do it. Each one supported a different file system and I could've sworn one or two of them could read LVM. Slaving the drive to a Linux system failed most of the time. I thought that perhaps the file system was badly corrupted. Eventually, I got it. :)

Jeremy Davis's picture

Possibly the first step I would've taken would be to try to repair the filesystem (with a Linux LiveCD/ISO) insitu.

And IMO backup is preferable to data recovery. Data loss is inevitable realisticly (just a matter of when, not if). So multiple backups (ideally onsite as well as offsite) is the way to go I reckon. And TKLBAM is really easy to use and pretty cheap too (~$0.14/GB/mth) so for the data contained in a WordPress appliance the cost would be negligable.

Add new comment