EGY's picture

Hello,

granted that I'm writing about a ubuntu-10.04-turnkey-lamp_11.3-1_i386 template, virtualized and contained in a CT of a Proxmox Host, I'm facing in a strange issue after a monthly-scheduled upgrade of this distro happened two months ago (December 2013).

First of all, I've to say that I've token notice of this issue, which I'm about to expose, just after a power-failure happened 3 weeks ago. Then, networking service wasn't able to get start up at the system boot, whereas it was able to do it manually.

After a lof of "googling" and comparing this distro upgrated with an identical one but not upgrated, I've understood that it depends on the upgrade of two booting managements present in my distro,SysVInit and UpStart, which leads them in conflict with each other.

By an empirical method, I tried to bring init.d directory back at its original state recreating symbolic links to the upstart-job. So, it appears as follow:

root@VMLAMP01 /etc/init.d# ls -l
total 368
-rw-r--r-- 1 root root  2427 Oct 15  2012 README
lrwxrwxrwx 1 root root    21 Mar  8 16:02 acpid -> /lib/init/upstart-job
-rw-r--r-- 1 root root  2227 Apr 15  2013 acpid.orig
-rwxr-xr-x 1 root root  7820 Jan 31 19:35 apache2
-rwxr-xr-x 1 root root  2341 Sep  7  2009 bootlogd
-rwxr-xr-x 1 root root  1276 Oct 15  2012 bootlogs
-rwxr-xr-x 1 root root  1281 Jul 14  2013 bootmisc.sh
-rwxr-xr-x 1 root root  4027 Jun 28  2010 casper
-rwxr-xr-x 1 root root  3816 Jul 14  2013 checkfs.sh
-rwxr-xr-x 1 root root  1099 Jul 14  2013 checkroot-bootclean.sh
-rwxr-xr-x 1 root root  9673 Jul 14  2013 checkroot.sh
-rwxr-xr-x 1 root root  1780 Mar 29  2013 confconsole
lrwxrwxrwx 1 root root    21 Mar  8 16:11 console-setup -> /lib/init/upstart-job
-rw-r--r-- 1 root root  1379 Dec  9  2011 console-setup.orig
lrwxrwxrwx 1 root root    21 Mar  8 16:03 cron -> /lib/init/upstart-job
-rw-r--r-- 1 root root  3033 Jul  3  2012 cron.orig
-rwxr-xr-x 1 root root  2182 Mar 29  2013 di-live
lrwxrwxrwx 1 root root    21 Mar  8 16:11 dmesg -> /lib/init/upstart-job
-rwxr-xr-x 1 root root  1105 Nov 23  2010 grub-common
-rwxr-xr-x 1 root root  1329 Sep  7  2009 halt
-rwxr-xr-x 1 root root 10577 Jun 24  2012 hdparm
lrwxrwxrwx 1 root root    21 Mar  8 16:12 hostname -> /lib/init/upstart-job
-rw-r--r-- 1 root root  1423 Oct 15  2012 hostname.sh.orig
-rwxr-xr-x 1 root root  1359 Jun 21  2011 hubdns
lrwxrwxrwx 1 root root    21 Mar  8 16:14 hwclock -> /lib/init/upstart-job
lrwxrwxrwx 1 root root    21 Mar  8 16:14 hwclock-save -> /lib/init/upstart-job
-rwxr-xr-x 1 root root  3882 Feb 24 12:52 hwclock.sh.orig
-rwxr-xr-x 1 root root  1164 Nov  1 05:11 inithooks
-rwxr-xr-x 1 root root  1019 Feb 25 14:27 jboss5
-rwxr-xr-x 1 root root 12550 Sep 21  2012 jetty
-rwxr-xr-x 1 root root 12357 Apr 25  2012 jetty.dpkg-dist
-rwxr-xr-x 1 root root  7592 Apr 28  2012 kbd
-rwxr-xr-x 1 root root  1591 Oct  1  2012 keyboard-setup
-rwxr-xr-x 1 root root  1293 Sep  7  2009 killprocs
lrwxrwxrwx 1 root root    21 Aug 13  2012 module-init-tools -> /lib/init/upstart-job
-rwxr-xr-x 1 root root   995 Oct 15  2012 motd
-rwxr-xr-x 1 root root   670 Feb 24  2013 mountall-bootclean.sh
-rwxr-xr-x 1 root root  2128 Feb 24  2013 mountall.sh
-rwxr-xr-x 1 root root  1508 Jul 14  2013 mountdevsubfs.sh
-rwxr-xr-x 1 root root  1414 Feb 24 15:07 mountkernfs.sh
-rwxr-xr-x 1 root root   678 Feb 24  2013 mountnfs-bootclean.sh
-rwxr-xr-x 1 root root  2440 Oct 15  2012 mountnfs.sh
-rwxr-xr-x 1 root root  1731 Jul 14  2013 mtab.sh
-rw-r--r-- 1 root root  5437 Dec 13 18:24 mysql.orig
lrwxrwxrwx 1 root root    21 Mar  8 16:16 network-interface -> /lib/init/upstart-job
lrwxrwxrwx 1 root root    21 Mar  8 16:16 network-interface-security -> /lib/init/upstart-job
-rw-r--r-- 1 root root  4323 Feb 25 18:45 networking
-rwxr-xr-x 1 root root  1818 Feb  3  2010 ntp
-rwxr-xr-x 1 root root   882 Sep  7  2009 ondemand
lrwxrwxrwx 1 root root    21 Mar  8 16:17 plymouth -> /lib/init/upstart-job
lrwxrwxrwx 1 root root    21 Mar  8 16:18 plymouth-log -> /lib/init/upstart-job
-rw-r--r-- 1 root root   725 Jun 30  2012 plymouth-log.orig
lrwxrwxrwx 1 root root    21 Mar  8 16:18 plymouth-splash -> /lib/init/upstart-job
lrwxrwxrwx 1 root root    21 Mar  8 16:18 plymouth-stop -> /lib/init/upstart-job
-rw-r--r-- 1 root root  1313 Jun 30  2012 plymouth.orig
-rwxr-xr-x 1 root root  7358 Mar 11  2013 postfix
lrwxrwxrwx 1 root root    21 Aug 13  2012 procps -> /lib/init/upstart-job
-rwxr-xr-x 1 root root  3093 Jan 18  2010 quota
-rwxr-xr-x 1 root root  1823 Jan 18  2010 quotarpc
-rwxr-xr-x 1 root root  6120 Oct 15  2012 rc
-rwxr-xr-x 1 root root   782 Oct 15  2012 rc.local
-rwxr-xr-x 1 root root   117 Oct 15  2012 rcS
-rwxr-xr-x 1 root root   639 Sep  7  2009 reboot
-rwxr-xr-x 1 root root  1074 Jul 14  2013 rmnologin
-rwxr-xr-x 1 root root  4395 Dec  2  2012 rsync
lrwxrwxrwx 1 root root    21 Mar  8 16:19 rsyslog -> /lib/init/upstart-job
-rw-r--r-- 1 root root  3054 Mar  5  2013 rsyslog.orig
-rwxr-xr-x 1 root root  1163 Sep 16  2012 screen-cleanup
-rwxr-xr-x 1 root root  3200 Oct 15  2012 sendsigs
-rwxr-xr-x 1 root root  4770 Apr  2  2013 shellinabox
-rwxr-xr-x 1 root root   590 Sep  7  2009 single
-rw-r--r-- 1 root root  4290 Oct 15  2012 skeleton
-rwxr-xr-x 1 root root  3881 Feb  8  2013 ssh
-rwxr-xr-x 1 root root   519 Sep  7  2009 stop-bootlogd
-rwxr-xr-x 1 root root  1095 Sep  7  2009 stop-bootlogd-single
-rwxr-xr-x 1 root root   714 Jun 28  2012 sudo
-rwxr-xr-x 1 root root  4572 Nov  1 05:11 turnkey-init-fence
lrwxrwxrwx 1 root root    21 Mar  8 16:19 udev -> /lib/init/upstart-job
lrwxrwxrwx 1 root root    21 Mar  8 16:20 udev-finish -> /lib/init/upstart-job
-rwxr-xr-x 1 root root  1179 Aug 19  2012 udev-mtab
-rw-r--r-- 1 root root  8827 Mar  7 17:01 udev.orig
lrwxrwxrwx 1 root root    21 Mar  8 16:20 udevmonitor -> /lib/init/upstart-job
lrwxrwxrwx 1 root root    21 Mar  8 16:20 udevtrigger -> /lib/init/upstart-job
-rwxr-xr-x 1 root root  2721 Apr 10  2013 umountfs
-rwxr-xr-x 1 root root  2195 Apr 10  2013 umountnfs.sh
-rwxr-xr-x 1 root root  1122 Oct 15  2012 umountroot
lrwxrwxrwx 1 root root    21 Mar  8 16:17 upstart-job -> /lib/init/upstart-job
-rwxr-xr-x 1 root root  3111 Oct 15  2012 urandom
-rwxr-xr-x 1 root root  1359 Jul 28  2013 webmin
-rwxr-xr-x 1 root root  1342 Apr 11  2012 winbind
-rwxr-xr-x 1 root root  2666 Mar  2  2012 x11-common

where *.orig are files I've replaced by a sym link.

Now, I achieve networking to start up at system boot. But, there is another strange behavior of mysql service: sometime it start up at boot system but sometime it doesn't, and I don't know why!!!

In fine, after this distro upgrade, I have issues with python too. Here is an example:

root@VMLAMP01 /etc/init.d# turnkey-sysinfo
Traceback (most recent call last):
  File "/usr/bin/turnkey-sysinfo", line 19, in <module>
    import netinfo
ImportError: No module named netinfo

I searched this module, netinfo, and I found:

root@VMLAMP01 /etc/init.d# find /usr/lib/ -name "net*"
/usr/lib/tklbam/deps/lib/python2.7/dist-packages/boto/ec2/networkinterface.py
/usr/lib/jvm/java-6-openjdk/jre/lib/net.properties
/usr/lib/python2.7/dist-packages/netinfo.pyc
/usr/lib/python2.7/dist-packages/netinfo.py
/usr/lib/python3.2/__pycache__/netrc.cpython-32.pyc
/usr/lib/python3.2/netrc.py
/usr/lib/python2.6/netrc.pyc
/usr/lib/python2.6/netrc.py

But, my system uses 2.6 version of python:

root@VMLAMP01 /etc/init.d# which python
/usr/bin/python
root@VMLAMP01 /etc/init.d# ls -l /usr/bin/python
lrwxrwxrwx 1 root root 9 Aug 13  2012 /usr/bin/python -> python2.6

thus far, I've exposed what are my issues... Now, I write what are my questions:

  • how can I know why my mysql service sometime starts up and sometime not?
  • how can I exactly know which services I have to start up by Up Start system and which by SysVInit?
  • how can I fix python issues? (bearing in mind that not all modules of 2.6 version are in 2.7 version)

Please, help me.

Forum: 
Jeremy Davis's picture

But I recall that there was a bug with the MySQL upstart on the default 10.04. I vaguely recall that TKL implemented a workaround although I don't recall any details...

Sorry it's probably not a great help, but hopefully might be a useful hint...?!

EGY's picture

I think it doesn't depend on a bug of MySQL upstart on the default 10.04... because I created this CT in ProXMoX more than 1 year ago and I faced this issue just after last DISTRO upgrade: I've never had any problems before.

Anyway, could you suggest me how can I trace UpStart and SysInit processes at the boot of S.O.?

I have a idea, but I can test it because I can't able to get trace of UpStart jobs.

I read somewhere that was written a job named openvz.conf just to make UpStart service able to run correct.

root@VMLAMP01 /etc/init# cat openvz.conf
# OpenVZ - Fix init sequence to have OpenVZ working with upstart
# Source: http://blog.bodhizazen.net/linux/ubuntu-10-04-openvz-templates/

description "Fix OpenVZ"

start on startup

task

pre-start script

        mount -t devpts devpts /dev/pts || true
        mount -t tmpfs varrun /var/run
        mount -t tmpfs varlock /var/lock
        mkdir -p /var/run/network

        if [ ! -e /etc/mtab ]; then
                cat /proc/mounts > /etc/mtab
        fi

        touch /var/run/utmp
        chmod 664 /var/run/utmp
        chown root.utmp /var/run/utmp

        if [ "$(find /etc/network/ -name upstart -type f)" ]; then
                chmod -x /etc/network/*/upstart || true
        fi

end script

script

        start networking
        initctl emit filesystem --no-wait
        initctl emit local-filesystems --no-wait
        initctl emit virtual-filesystems --no-wait
        initctl emit net-device-up --no-wait
        init 2

end script

And here it is wrote

init 2

command! So, if we go in mysql.conf,

root@VMLAMP01 /etc/init# cat mysql.conf
# MySQL Service

description     "MySQL Server"
author          "Mario Limonciello <superm1@ubuntu.com>"

start on (net-device-up
          and local-filesystems
          and runlevel [2345])
stop on runlevel [016]

we can see that it start when system goes in runlevel 2. Therefore, this is my idea:

could it be that upstart-job openvz is not able, sometime, to force the system to go in runlevel 2??

This can be a breakthrough, but I need to know how I can trace init process before.

I also read that it is possible debugging of Up-Start by adding "--verbose" at kernel, but it is an impossible modification in a OpenVZ Virtual Machine.

Could you help me, please?

 

 

Best Regards.
 

Jeremy Davis's picture

I'm inclined to disagree that your experience rules out the possibility of the MySQL upstart bug reappearing due to your latest upgrade. My suspicion is/was that this upgrade (for whatever reason...) has undone whatever the MySQL bug workaround was and reverted to the original setup. Perhaps your previous upgrades didn't mess with the fix, but this one did...? But really it's academic because I'm only guessing...

I haven't had any contact with upstart since TKL v11 (which was a while ago now...) so realistically I'm not going to be much use to you. And even back then I probably wouldn't have been much help to you as I have learned as I have gone... Personally I would just migrate your content to a new v13.0 server and be done with it (IMO Ubuntu is a bit flakey; Debian is much more solid and stable)...

Otherwise, if you think that it is an Ubuntu issue then you're probably more likely to find someone who can meaningfully help you over on the Ubuntu forums (not meaning to fob you off, just being realistic).

Add new comment