John Carver's picture

I came across your TurnKey website yesterday. Since I was already experimenting with VirtualBox, I decided to download and test the Drupal appliance. The process went smoothly and in a few minutes I had a working website. One of Drupal's shortcomings (IMHO) is the difficulty of updating modules and the core software. The situation improves with each new version, but right now the best tool I've found is drush, short for drupal shell. Drush update will download and apply module updates, and hopefully soon be able to upgrade core. Drush has a number of useful administrative commands. I was able to install Drush on the Drupal appliance and so far it appears to work quite well. I would highly recommend you consider including Drush in future additions of the Drupal appliance. I've documented the necessary steps if you are interested. Drush can be found at http://drupal.org/project/drush.

Note: Edited InstallDrush.txt on 1/23/10 to change php-cli to php5-cli, use /tmp instead of /opt, and cleanup after.

Note: Edited on 4/25/10 to correct path error reported by Johnny Canuck.

Note: Edited on 6/6/10 to correct symlink error reported by Dan.

Forum: 
Neil Aggarwal's picture

You can contribute a patch to the appliance that enables drush.

See this page: http://www.turnkeylinux.org/docs/tklpatch

John Carver's picture

I'll have to study the tklpatch creation process. In the meantime, I've attached the steps I used to install Drush on a freshly installed Drupal appliance to the original post.

1. # apt-get update
2. # apt-get install wget php5-cli
3. Install Drush
	# cd /tmp
	# wget http://ftp.drupal.org/files/projects/drush-All-Versions-2.1.tar.gz
	# tar -zxvf drush-All-Versions-2.1.tar.gz
	# chown -R root.root drush
	# mv drush /usr/share
        # ln -s /usr/share/drush/drush /usr/bin/drush
	# cp /usr/share/drush/examples/example.drushrc.php ~/.drushrc.php
   Edit ~/.drushrc.php replacing
	# $options['r'] = '/home/USER/workspace/drupal-6'; 
   with
	$options['r'] = '/usr/share/drupal6/';
4. Test
	# drush help
	# drush status
	# drush statusmodules
5. Cleanup
	# rm /tmp/drush-All-Versions-2.1.tar.gz

Note: Edited on 1/23/10 to change php-cli to php5-cli, use /tmp instead of /opt, and cleanup after.

Note: Edited on 4/25/10 to correct path error reported by Johnny Canuck.

Note: Edited on 6/6/10 to correct symlink error reported by Dan.

Information is free, knowledge is acquired, but wisdom is earned.

dman's picture

  I tried out the appliance as a ready-made home for an existing site - I want to bundle a fully working snapshot as an archive.

drush is THE tool for folk that will be doing many Drupal deployments, and the setup of it is a hassle each time, so it would be lovely if it were in this distro.

NOTE that it requires php-cli - which turnkey-drupal6-2009.10-hardy-x86-virtualbox did not have available.

Anyway, with drush, the deployment was a breeze (save for one annoyance).

I could 

1. define a site-alias that described the 'turnkey' appliance (its paths and IP)

2. push the db from my dev site to the appliance

drush sql-sync devsite turnkey

3. push the files and modules across

 

drush rsync devsite\:\!site turnkey\:\!site 

drush rsync devsite\:sites/all/ turnkey\:sites/all/

 

And the site almost worked.

(... I spent a long while trying to find why it kept wanting to 'install' even though all the bits were configured. Eventually I found the custom code to delete the /tmp/install_redirect.php thing)

Apart from that, it was three steps to a working, cloned drupal site on a stand-alone VM,

... anyway. drush is a key part of drupal management now.  Would be cool to use it,

John Carver's picture

Good point on php-cli being required to run Drush. I obviously installed it but forgot to add it to my notes. I've corrected the InstallDrush.txt attachment and instructions above. I believe Drush requires both wget and php-cli. If there are other dependencies, I haven't discovered them yet, but I haven't fully tested all of the commands either.

Information is free, knowledge is acquired, but wisdom is earned.

John Carver's picture

dman,

Does the drush sql-sync command have an option for syncing just the table structure between the two databases while leaving the records unchanged?  I'm thinking it would be handy to remove any tables in the dest that don't exit in the source.  Normally I rely on drush updatedb to add and initialize any new tables, but sometimes old tables from removed modules get left behind.

Information is free, knowledge is acquired, but wisdom is earned.

dman's picture

 Yes, it does choose some tables to update structure-only, to leave cache and stuff out.

No, AFAIK it doesn't yet contain remote selective deletes of missing stuff, that would be scary.

But I think there is a 'drop-database' (or rebuild or something) in the works (I have one in my own custom additions) that could become part of the process. Check the issue queue I guess.

 

Edit: It's in already. Sql sync - drop target database and recreate 

John Carver's picture

The drush sql sync command does need to drop the target database before syncing the new data, otherwise the result would be a merge of the old and new databases.  I hope the decision was to drop all tables and not just those in the source.  My problem is just a bit different.  I have a website that was originally started in Drupal 4.7 and is now at 5.21.  Over the course of several years experimenting with different modules and upgrading versions, a fair amount of 'cruft' has crept into my database.  As I prepare to upgrade to 6.xx, I'm trying to figure out a strategy to identify what obsolete tables have been left behind by modules that didn't clean up when they were uninstalled or crept in when data was copied from one database to another without clearing (dropping all tables) the target database.

My thought at this point is create a sandbox with an empty database, then using drush, make a list of installed_modules and a list of enabled_modules from the production site and use those lists to install a fresh copy of Drupal 5.21 with the identical modules installed and enabled.  Then I can use phpMyAdmin to compare the two databases and remove any obsolete tables i.e. tables that exist in production but not in the 'clean' copy.  If I could figure out how to automate this last step, I could create a 'drush sql clean' command.

Do you think this would be a useful effort?  I would probably only use it once, so it wouldn't be worth the effort unless others could benefit as well.

Information is free, knowledge is acquired, but wisdom is earned.

Liraz Siri's picture

Thanks for everybody who weighed in. I had fun exploring drush a while back but never got around to incorporating it into my workflow. From the discussion it sounds more useful and widely used than I realized. It's now on our todo list for the next version of TurnKey Drupal.
John Carver's picture

I've been working on testing a tklpatch for adding drush-All-Versions-2.1, to TurnKey Drupal.  I wasn't aware of the debian package for drush that dman mentioned.  I'll have to check it out to see if it makes sense to use that instead.

My testing so far hasn't revealed any problems with drush that are unique to TurnKey Drupal.  Some of the drush commands don't work because they require certain modules like simpletest to be installed, but that is a drush issue and not related to the TurnKey install.

I've delayed posting the patch because I wanted to include some scripts that I've written to supplement drush including updating Drupal core.  Before committing to using TKL Drupal in a virtual machine, I wanted to make sure that I have all the functionality of my old site.  What I've found is the use of symlinks for sites, profiles, files, etc broke my original scripts which expected everything to be stored under the drupal root.  The last remaining issue I have to solve is to figure out how to use tar to backup and restore all the relevant parts while preserving the symlinks.  If anyone has experience with some of the more esoteric tar commands, I could use some advice.

Information is free, knowledge is acquired, but wisdom is earned.

Liraz Siri's picture

Since tar preserves symlinks by default I'm assuming you are looking for a more exotic command that would make symlinks relative or something like that. To the best of my knowledge no such command exists. Symlinks are just short snippets of data to tar. Maybe look into modifying your symlink structure so that it is more general?

BTW, a tklpatch for adding drush to TurnKey Drupal sounds great!

John Carver's picture

Well, yes tar preserves symlinks by default, but it doesn't follow them either.  If you backup using tar from the drupal root, then the contents of sites, profiles, and files won't be included because they are symlinks.  If you use tar's -h option to dereference symlinks, then all paths are included, but when you restore, the symlinks are overwritten which would break the site upon the next debian update.  I've come to the conclusion that I need to backup core, sites, profiles, and files as separate tar files then dump the database(s) and finally tar the whole collection with compression.  Special handling will be needed for .htaccess, robots.txt, and favicon.ico if it exists.  Those are the files in drupal core that are likely to modified by webmasters.

I hope to develop a system robust enough to backup my current system with a single file tree, and restore it into a TKL Drupal appliance and have everything end up where it should.  I'll post the results here when I'm done.

Information is free, knowledge is acquired, but wisdom is earned.

John Carver's picture

My attempt to build a tklpatch using drush_2.0-6_all.deb failed as follows:

# tklpatch turnkey-drupal6-2009.10-hardy-x86.iso drush_2.0-6_all_tkl_drupal6.patch
# extracting root filesystem and isolinux from ISO

created 28730 files
created 2956 directories
created 1665 symlinks
created 41 devices
created 0 fifos
TKLPATCH_ISOLABEL: drush_2.0-6_all_tkl_drupal6.patch
# applying patch drush_2.0-6_all_tkl_drupal6.patch
# installing 1 debs from drush_2.0-6_all_tkl_drupal6.patch/debs
# chroot execute: dpkg -i /*.deb
Selecting previously deselected package drush.
(Reading database ... 23373 files and directories currently installed.)
Unpacking drush (from /drush_2.0-6_all.deb) ...
dpkg: dependency problems prevent configuration of drush:
 drush depends on php5-cli (>= 5.2); however:
  Package php5-cli is not installed.
dpkg: error processing drush (--install):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 drush

Apparently tklpatch processes the /debs before executing the conf file.  My conf file looks like this:


#!/bin/bash -ex
install()
{
    apt-get update
    DEBIAN_FRONTEND=noninteractive apt-get -y \
        -o DPkg::Options::=--force-confdef \
        -o DPkg::Options::=--force-confold \
        install $@
}

# Install drush (Drupal shell) - a highly useful package for managing Drupal
# drush_2.0-6

# Install required packages
install wget php5-cli;

# Let udhcpc set hostname
sed -i -e 's/#SET_HOSTNAME/SET_HOSTNAME/' /etc/udhcpc/*;

# Tell the world what we've done!
echo 'Installed drush_2.0-6_all (Drupal shell) on TurnKey Drupal!' >> /etc/issue

Is there a possibility for a pre-install and post-install script ala Debian?  I suppose I can download the php5-cli deb and add it to the /debs folder.  Will try again tomorrow.

Information is free, knowledge is acquired, but wisdom is earned.

Liraz Siri's picture

You make a really good argument for installing debs/ after the conf script is executed. I've just pushed the update to github. It's a very simple fix so if you want you can just edit tklpatch-apply so it applies the patch elements in the correct order and try again.
[ -d $patch_dir/overlay ] && tklpatch-apply-overlay $rootfs $patch_dir/overlay
[ -x $patch_dir/conf ] && tklpatch-apply-conf $rootfs $patch_dir/conf
[ -d $patch_dir/debs ] && tklpatch-apply-debs $rootfs $patch_dir/debs
Alon Swartz's picture

I agree there is room for improvement, but I don't think the proposed fix is the right way to go. Moving the debs/ installation after the conf doesn't allow for any customization via the overlay or the conf itself.

Until we fix this issue, the easiest way would be to include the deb in the overlay and install it during conf.

Update: I just had a quick chat with Liraz and we decided to revert the changes back in github. I'll be working on a fix and try get it out asap.
John Carver's picture

Sorry guys, it was late last night and I wasn't thinking too clearly.  I didn't mean to imply that the order of applying debs, overlays, and the conf script should be changed.  As Alon points out, its important to be able to make changes after the debs are installed, and changing the order would break tklpatches that others have prepared.

I must have been channeling previous experience in building rpms.  I just took another look at the Debian Binary Package Building HOWTO and noted that Debian only provides the 'postinst' and 'prerm' scripts.  The rpm spec file, on the other hand, is much more fine grained, providing macros that are executed in order.  They include %prep, %setup, %patch, %build, %install, %clean, %files, among others.  The %setup, %patch, and %build roughly correspond to the installation of the /debs.  %install and %clean correspond to the tklpatch conf script, and %files to the /overlay.  Neither Debian nor tklpatch have the equivalent of %prep.  What I was thinking last night was the potential for adding an additional script, 'preconf', that is optional but if it exists, is executed before installing the debs and applying the overlay.  This would add some additional flexibility without breaking previous patches.  Just my 2 cents.

Update:  I edited tklpatch-apply and copied the apply-conf line to before apply-debs, then changed the line to install 'preconf' if it exists.  I then moved the install of the prerequisite, php5-cli to a 'preconf' file in the patch_dir.  Worked like a champ.  Both 'conf' and 'preconf' were applied in the expected sequence.

[ -x $patch_dir/preconf ] && tklpatch-apply-conf $rootfs $patch_dir/preconf
[ -d $patch_dir/debs ] && tklpatch-apply-debs $rootfs $patch_dir/debs
[ -d $patch_dir/overlay ] && tklpatch-apply-overlay $rootfs $patch_dir/overlay
[ -x $patch_dir/conf ] && tklpatch-apply-conf $rootfs $patch_dir/conf

tklpatch-apply-debs could be modified to identify and install any prerequisites, but that would make it more complicated and might lead to some nasty surprises.  Better IMHO to let the patch developer identify any required packages and install them via 'preconf'.

Information is free, knowledge is acquired, but wisdom is earned.

Alon Swartz's picture

It took a while, but I have just released tklpatch v0.92 (github, package archive). The new version includes several enhancements, the main one being support for configuration hooks.

I am excited about the new functionality, it provides a very powerful interface to support any use case, and is fully backwards compatible. Take a look at the advanced documentation, and the changelog if you're interested.

As always, feedback is welcome!
John Carver's picture

Thanks Alon,  I couldn't have asked for any more flexibility.  I'll use the new version to create a new patch for Drush and then submit it here when I'm done testing.

Information is free, knowledge is acquired, but wisdom is earned.

Johnny Canuck's picture

Hi,

Following the instructions at:

http://www.turnkeylinux.org/files/attachments/InstallDrush_1.txt

there may be a couple of issues:

1) I think the lines:

# mv drush /usr/share
# cp example.drushrc.php ~/.drushrc.php

should be:

# cp drush/example.drushrc.php ~/.drushrc.php
# mv drush /usr/share

 

2) After completing the instructions, drush is not executing, because of a path issue. In my case I edited ~/.bashrc and added the following line:

PATH=$PATH:/usr/share/drush

 

Am a noob, so these may not be the best practices. At least it now works for me.


John Carver's picture

That's what I get for relying on memory and the command line history.  The lines should have read:

# mv drush /usr/share
# cp /usr/share/drush/example.drushrc.php ~/.drushrc.php

Instead of adding /usr/share/drush to the PATH, I've been creating a link as follows:

# ln -s /usr/share/drush/drush.php /usr/bin/drush

That is consistent with the installation of the Debian version of Drush.  http://packages.debian.org/sid/drush

Information is free, knowledge is acquired, but wisdom is earned.

Add new comment