TurnKey Linux Virtual Appliance Library

Drush (drupal shell) on TurnKey Drupal appliance

John Carver's picture

I came across your TurnKey website yesterday. Since I was already experimenting with VirtualBox, I decided to download and test the Drupal appliance. The process went smoothly and in a few minutes I had a working website. One of Drupal's shortcomings (IMHO) is the difficulty of updating modules and the core software. The situation improves with each new version, but right now the best tool I've found is drush, short for drupal shell. Drush update will download and apply module updates, and hopefully soon be able to upgrade core. Drush has a number of useful administrative commands. I was able to install Drush on the Drupal appliance and so far it appears to work quite well. I would highly recommend you consider including Drush in future additions of the Drupal appliance. I've documented the necessary steps if you are interested. Drush can be found at http://drupal.org/project/drush.

Note: Edited InstallDrush.txt on 1/23/10 to change php-cli to php5-cli, use /tmp instead of /opt, and cleanup after.

Note: Edited on 4/25/10 to correct path error reported by Johnny Canuck.

Note: Edited on 6/6/10 to correct symlink error reported by Dan.

subscribe

my first idea when reading the drupal specs was ... drush is missing :) so please ... add it :) i will be glad to help 

You can contribute a patch with drush

You can contribute a patch to the appliance that enables drush.

See this page: http://www.turnkeylinux.org/docs/tklpatch

http://UnmeteredVPS.net/tkl
Try our TurnKey Linux appliance hosting FREE!

John Carver's picture

Steps to install Drush on TurnKey Drupal appliance

I'll have to study the tklpatch creation process. In the meantime, I've attached the steps I used to install Drush on a freshly installed Drupal appliance to the original post.

1. # apt-get update
2. # apt-get install wget php5-cli
3. Install Drush
	# cd /tmp
	# wget http://ftp.drupal.org/files/projects/drush-All-Versions-2.1.tar.gz
	# tar -zxvf drush-All-Versions-2.1.tar.gz
	# chown -R root.root drush
	# mv drush /usr/share
        # ln -s /usr/share/drush/drush /usr/bin/drush
	# cp /usr/share/drush/examples/example.drushrc.php ~/.drushrc.php
   Edit ~/.drushrc.php replacing
	# $options['r'] = '/home/USER/workspace/drupal-6'; 
   with
	$options['r'] = '/usr/share/drupal6/';
4. Test
	# drush help
	# drush status
	# drush statusmodules
5. Cleanup
	# rm /tmp/drush-All-Versions-2.1.tar.gz

Note: Edited on 1/23/10 to change php-cli to php5-cli, use /tmp instead of /opt, and cleanup after.

Note: Edited on 4/25/10 to correct path error reported by Johnny Canuck.

Note: Edited on 6/6/10 to correct symlink error reported by Dan.

Information is free, knowledge is acquired, but wisdom is earned.

 i just found the note on it

 i just found the note on it beeing an "apt version" I think we rather create a new thing out of the box ? Or is there any special reason for using APT ?

 

thanks :-)

 I'm not an authority here,

 I'm not an authority here, but it appears the turnkey distro does some very funky things with the paths used. I'm deducing that this is following the apt/debian convention of putting code, configs, and content on three different disk partitions. See the symlinks within sites/*

This seems to be OK and working, although disconcerting to a standard Drupal user. In short, a clean Drupal checkout would still need these changes made to it.

I don't know if these changes are applied by the apt distro or just turnkey.

As a developer, I'd personally prefer a clean sync with Drupal core ... but I can see why these changes were chosen in this environment.

(I may be misguided or wrong - this is just from inspection of the distro, I have no inside knowledge)

John Carver's picture

Volatile and Non-Volatile Files

I think it has to do with the separation of volatile and non-volatile files.  I'm following the discussion over at <link>.  I've heard of people using version control to maintain their Drupal site, but I've never tried it.  I think separating the files, profiles and sites may be a good idea, but it does cause confusion for those of us who have used the standard file structure for awhile.  I have a drupal update script based on drush that I tried in the TurnKey appliance.  It mangled the softlinks so now I'm trying to rewrite and debug it so it works in either environment.  Even if git can be used to manage the Drupal files, there is still the issue of managing and backing up the databases.  I think Drush is the best tool to handle the complexities of updating Drupal.

Information is free, knowledge is acquired, but wisdom is earned.

Alon Swartz's picture

TurnKey uses the deb package from Debian

TurnKey uses the deb package from Debian (its more up to date than the one available in the Ubuntu repositories), and auto-updated to install security patches.

All deb packages adhere to the Debian policy, which inturn follow the Filesystem Hierarchy Standard (FHS):
The FHS standard consists of a set of requirements and guidelines for
file and directory placement under UNIX-like operating systems. The 
guidelines are intended to support interoperability of applications, 
system administration tools, development tools, and scripts as well 
as greater uniformity of documentation for these systems.
I hope the above sheds some light on the subject.
John Carver's picture

Best Practices

Alon, thanks for providing that insight.  As someone coming from a RedHat/Fedora/CentOS background, I sometimes have trouble adjusting to Ubuntu/Debian.  I'm not complaining, I just have to learn the rules of the road.  May I suggest that you consider adding a 'Best Practices' section under Help and placing information like this where new members can easlly locate it.  See Drupal's Best Practices for an example of what I mean.  When I first started using Ubuntu, I tried installing the Drupal package but found it so outdated it wasn't usable.  I went back to what I was familiar with, installing native Drupal manually.  How timely are the Debian updates?  Version 6.15 of Drupal was released on December 16th to fix a Cross Site Scripting vulnerability in core.  I still haven't seen an update from either Debian or TurnKey.  Up to now, I've been relying on Drush and some custom scripts to update Drupal modules and core, which is why I started this thread recommending including Drush in the TurnKey Drupal appliance.  There are still some problems to be resolved in getting my scripts to deal with the embedded softlinks for /sites /profiles, and /files.  I also see a potential for problems if updating directly from source interfers with the apt update process.

I think it's great that you are open to new members getting involved with the development process.  I'm looking into a local need for a Drupal+CiviCRM appliance.  I will know more next week. 

Information is free, knowledge is acquired, but wisdom is earned.

+1 for drush

  I tried out the appliance as a ready-made home for an existing site - I want to bundle a fully working snapshot as an archive.

drush is THE tool for folk that will be doing many Drupal deployments, and the setup of it is a hassle each time, so it would be lovely if it were in this distro.

NOTE that it requires php-cli - which turnkey-drupal6-2009.10-hardy-x86-virtualbox did not have available.

Anyway, with drush, the deployment was a breeze (save for one annoyance).

I could 

1. define a site-alias that described the 'turnkey' appliance (its paths and IP)

2. push the db from my dev site to the appliance

drush sql-sync devsite turnkey

3. push the files and modules across

 

drush rsync devsite\:\!site turnkey\:\!site 

drush rsync devsite\:sites/all/ turnkey\:sites/all/

 

And the site almost worked.

(... I spent a long while trying to find why it kept wanting to 'install' even though all the bits were configured. Eventually I found the custom code to delete the /tmp/install_redirect.php thing)

Apart from that, it was three steps to a working, cloned drupal site on a stand-alone VM,

... anyway. drush is a key part of drupal management now.  Would be cool to use it,

John Carver's picture

Good point on php-cli being

Good point on php-cli being required to run Drush. I obviously installed it but forgot to add it to my notes. I've corrected the InstallDrush.txt attachment and instructions above. I believe Drush requires both wget and php-cli. If there are other dependencies, I haven't discovered them yet, but I haven't fully tested all of the commands either.

Information is free, knowledge is acquired, but wisdom is earned.

Is that your quote?

Hello John,

I've just discovered Turnkey Linux from a note I saw on the Drupal home page. It's a fantastic idea and I'll be giving it a run on Amazon this week.

I'm new to Drupal, but from what I gather, I'll be another one voting for Drush. (Big fan of the comman line here.)

One quick question that's off topic. Is the quote "Information is free, knowledge is acquired, but wisdom is earned." yours? It's great too.

 

 

John Carver's picture

I think so

Some years ago, I wrote some notes on the subject of information (data), knowledge, wisdom, and the differences between them.  My tag line was an attempt to distill the essence of those thoughts.  I did google to see if I was subliminally quoting something I'd read earlier, but didn't find any exact matches.  I owe credit to those sources on which my notes were based, but sadly I can no longer identify who they were.

I also found TurnKey Linux through the link on the Drupal page.  Serendipity.

Thanks for your kind comment.

 

Information is free, knowledge is acquired, but wisdom is earned.

John Carver's picture

drush sql-sync

dman,

Does the drush sql-sync command have an option for syncing just the table structure between the two databases while leaving the records unchanged?  I'm thinking it would be handy to remove any tables in the dest that don't exit in the source.  Normally I rely on drush updatedb to add and initialize any new tables, but sometimes old tables from removed modules get left behind.

Information is free, knowledge is acquired, but wisdom is earned.

 Yes, it does choose some

 Yes, it does choose some tables to update structure-only, to leave cache and stuff out.

No, AFAIK it doesn't yet contain remote selective deletes of missing stuff, that would be scary.

But I think there is a 'drop-database' (or rebuild or something) in the works (I have one in my own custom additions) that could become part of the process. Check the issue queue I guess.

 

Edit: It's in already. Sql sync - drop target database and recreate 

John Carver's picture

Sql sync

The drush sql sync command does need to drop the target database before syncing the new data, otherwise the result would be a merge of the old and new databases.  I hope the decision was to drop all tables and not just those in the source.  My problem is just a bit different.  I have a website that was originally started in Drupal 4.7 and is now at 5.21.  Over the course of several years experimenting with different modules and upgrading versions, a fair amount of 'cruft' has crept into my database.  As I prepare to upgrade to 6.xx, I'm trying to figure out a strategy to identify what obsolete tables have been left behind by modules that didn't clean up when they were uninstalled or crept in when data was copied from one database to another without clearing (dropping all tables) the target database.

My thought at this point is create a sandbox with an empty database, then using drush, make a list of installed_modules and a list of enabled_modules from the production site and use those lists to install a fresh copy of Drupal 5.21 with the identical modules installed and enabled.  Then I can use phpMyAdmin to compare the two databases and remove any obsolete tables i.e. tables that exist in production but not in the 'clean' copy.  If I could figure out how to automate this last step, I could create a 'drush sql clean' command.

Do you think this would be a useful effort?  I would probably only use it once, so it wouldn't be worth the effort unless others could benefit as well.

Information is free, knowledge is acquired, but wisdom is earned.

Liraz Siri's picture

Next version of TurnKey Drupal will include drush

Thanks for everybody who weighed in. I had fun exploring drush a while back but never got around to incorporating it into my workflow. From the discussion it sounds more useful and widely used than I realized. It's now on our todo list for the next version of TurnKey Drupal.

drush debian package

There may be convergence coming

debian package for drush

would fit with the approach so far!

John Carver's picture

debian package for drush

I checked the Ubuntu site and found that indeed drush 2.0.6 will be included in Lucid.

http://packages.ubuntu.com/lucid/php/drush

It's a little older than the 2.1 version I've been testing, but maybe there is a reason.  It's possible there are problems with 2.1 on Ubuntu that I haven't yet discovered.  Anyway it seems like the safest course is to use this package as the basis for a tklpatch.  That will also ease the upgrade path to Lucid LTS.  I doubt this will ever be backported to Hardy, but if Liraz wants to look into adding this package to the TurnKey repository, that would be great.  In the meantime, I'll just download and put the drush_2.0-6_all.deb in the patch debs directory.

Information is free, knowledge is acquired, but wisdom is earned.

John Carver's picture

drush testing

I've been working on testing a tklpatch for adding drush-All-Versions-2.1, to TurnKey Drupal.  I wasn't aware of the debian package for drush that dman mentioned.  I'll have to check it out to see if it makes sense to use that instead.

My testing so far hasn't revealed any problems with drush that are unique to TurnKey Drupal.  Some of the drush commands don't work because they require certain modules like simpletest to be installed, but that is a drush issue and not related to the TurnKey install.

I've delayed posting the patch because I wanted to include some scripts that I've written to supplement drush including updating Drupal core.  Before committing to using TKL Drupal in a virtual machine, I wanted to make sure that I have all the functionality of my old site.  What I've found is the use of symlinks for sites, profiles, files, etc broke my original scripts which expected everything to be stored under the drupal root.  The last remaining issue I have to solve is to figure out how to use tar to backup and restore all the relevant parts while preserving the symlinks.  If anyone has experience with some of the more esoteric tar commands, I could use some advice.

Information is free, knowledge is acquired, but wisdom is earned.

Liraz Siri's picture

Tar preserves symlinks by default

Since tar preserves symlinks by default I'm assuming you are looking for a more exotic command that would make symlinks relative or something like that. To the best of my knowledge no such command exists. Symlinks are just short snippets of data to tar. Maybe look into modifying your symlink structure so that it is more general?

BTW, a tklpatch for adding drush to TurnKey Drupal sounds great!

John Carver's picture

symlinks

Well, yes tar preserves symlinks by default, but it doesn't follow them either.  If you backup using tar from the drupal root, then the contents of sites, profiles, and files won't be included because they are symlinks.  If you use tar's -h option to dereference symlinks, then all paths are included, but when you restore, the symlinks are overwritten which would break the site upon the next debian update.  I've come to the conclusion that I need to backup core, sites, profiles, and files as separate tar files then dump the database(s) and finally tar the whole collection with compression.  Special handling will be needed for .htaccess, robots.txt, and favicon.ico if it exists.  Those are the files in drupal core that are likely to modified by webmasters.

I hope to develop a system robust enough to backup my current system with a single file tree, and restore it into a TKL Drupal appliance and have everything end up where it should.  I'll post the results here when I'm done.

Information is free, knowledge is acquired, but wisdom is earned.

John Carver's picture

tklpatch failure

My attempt to build a tklpatch using drush_2.0-6_all.deb failed as follows:

# tklpatch turnkey-drupal6-2009.10-hardy-x86.iso drush_2.0-6_all_tkl_drupal6.patch
# extracting root filesystem and isolinux from ISO

created 28730 files
created 2956 directories
created 1665 symlinks
created 41 devices
created 0 fifos
TKLPATCH_ISOLABEL: drush_2.0-6_all_tkl_drupal6.patch
# applying patch drush_2.0-6_all_tkl_drupal6.patch
# installing 1 debs from drush_2.0-6_all_tkl_drupal6.patch/debs
# chroot execute: dpkg -i /*.deb
Selecting previously deselected package drush.
(Reading database ... 23373 files and directories currently installed.)
Unpacking drush (from /drush_2.0-6_all.deb) ...
dpkg: dependency problems prevent configuration of drush:
 drush depends on php5-cli (>= 5.2); however:
  Package php5-cli is not installed.
dpkg: error processing drush (--install):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 drush

Apparently tklpatch processes the /debs before executing the conf file.  My conf file looks like this:


#!/bin/bash -ex
install()
{
    apt-get update
    DEBIAN_FRONTEND=noninteractive apt-get -y \
        -o DPkg::Options::=--force-confdef \
        -o DPkg::Options::=--force-confold \
        install $@
}

# Install drush (Drupal shell) - a highly useful package for managing Drupal
# drush_2.0-6

# Install required packages
install wget php5-cli;

# Let udhcpc set hostname
sed -i -e 's/#SET_HOSTNAME/SET_HOSTNAME/' /etc/udhcpc/*;

# Tell the world what we've done!
echo 'Installed drush_2.0-6_all (Drupal shell) on TurnKey Drupal!' >> /etc/issue

Is there a possibility for a pre-install and post-install script ala Debian?  I suppose I can download the php5-cli deb and add it to the /debs folder.  Will try again tomorrow.

Information is free, knowledge is acquired, but wisdom is earned.

Liraz Siri's picture

Installing debs after conf

You make a really good argument for installing debs/ after the conf script is executed. I've just pushed the update to github. It's a very simple fix so if you want you can just edit tklpatch-apply so it applies the patch elements in the correct order and try again.
[ -d $patch_dir/overlay ] && tklpatch-apply-overlay $rootfs $patch_dir/overlay
[ -x $patch_dir/conf ] && tklpatch-apply-conf $rootfs $patch_dir/conf
[ -d $patch_dir/debs ] && tklpatch-apply-debs $rootfs $patch_dir/debs
Alon Swartz's picture

I agree there is room for improvement, but I don't think...

I agree there is room for improvement, but I don't think the proposed fix is the right way to go. Moving the debs/ installation after the conf doesn't allow for any customization via the overlay or the conf itself.

Until we fix this issue, the easiest way would be to include the deb in the overlay and install it during conf.

Update: I just had a quick chat with Liraz and we decided to revert the changes back in github. I'll be working on a fix and try get it out asap.
John Carver's picture

pre-install vs post-install script

Sorry guys, it was late last night and I wasn't thinking too clearly.  I didn't mean to imply that the order of applying debs, overlays, and the conf script should be changed.  As Alon points out, its important to be able to make changes after the debs are installed, and changing the order would break tklpatches that others have prepared.

I must have been channeling previous experience in building rpms.  I just took another look at the Debian Binary Package Building HOWTO and noted that Debian only provides the 'postinst' and 'prerm' scripts.  The rpm spec file, on the other hand, is much more fine grained, providing macros that are executed in order.  They include %prep, %setup, %patch, %build, %install, %clean, %files, among others.  The %setup, %patch, and %build roughly correspond to the installation of the /debs.  %install and %clean correspond to the tklpatch conf script, and %files to the /overlay.  Neither Debian nor tklpatch have the equivalent of %prep.  What I was thinking last night was the potential for adding an additional script, 'preconf', that is optional but if it exists, is executed before installing the debs and applying the overlay.  This would add some additional flexibility without breaking previous patches.  Just my 2 cents.

Update:  I edited tklpatch-apply and copied the apply-conf line to before apply-debs, then changed the line to install 'preconf' if it exists.  I then moved the install of the prerequisite, php5-cli to a 'preconf' file in the patch_dir.  Worked like a champ.  Both 'conf' and 'preconf' were applied in the expected sequence.

[ -x $patch_dir/preconf ] && tklpatch-apply-conf $rootfs $patch_dir/preconf
[ -d $patch_dir/debs ] && tklpatch-apply-debs $rootfs $patch_dir/debs
[ -d $patch_dir/overlay ] && tklpatch-apply-overlay $rootfs $patch_dir/overlay
[ -x $patch_dir/conf ] && tklpatch-apply-conf $rootfs $patch_dir/conf

tklpatch-apply-debs could be modified to identify and install any prerequisites, but that would make it more complicated and might lead to some nasty surprises.  Better IMHO to let the patch developer identify any required packages and install them via 'preconf'.

Information is free, knowledge is acquired, but wisdom is earned.

Alon Swartz's picture

TKLPatch advanced configuration hook scripts

It took a while, but I have just released tklpatch v0.92 (github, package archive). The new version includes several enhancements, the main one being support for configuration hooks.

I am excited about the new functionality, it provides a very powerful interface to support any use case, and is fully backwards compatible. Take a look at the advanced documentation, and the changelog if you're interested.

As always, feedback is welcome!
John Carver's picture

Very nice job!

Thanks Alon,  I couldn't have asked for any more flexibility.  I'll use the new version to create a new patch for Drush and then submit it here when I'm done testing.

Information is free, knowledge is acquired, but wisdom is earned.

drush not in the path

Hi,

Following the instructions at:

http://www.turnkeylinux.org/files/attachments/InstallDrush_1.txt

there may be a couple of issues:

1) I think the lines:

# mv drush /usr/share
# cp example.drushrc.php ~/.drushrc.php

should be:

# cp drush/example.drushrc.php ~/.drushrc.php
# mv drush /usr/share

 

2) After completing the instructions, drush is not executing, because of a path issue. In my case I edited ~/.bashrc and added the following line:

PATH=$PATH:/usr/share/drush

 

Am a noob, so these may not be the best practices. At least it now works for me.


John Carver's picture

Opps! My Bad

That's what I get for relying on memory and the command line history.  The lines should have read:

# mv drush /usr/share
# cp /usr/share/drush/example.drushrc.php ~/.drushrc.php

Instead of adding /usr/share/drush to the PATH, I've been creating a link as follows:

# ln -s /usr/share/drush/drush.php /usr/bin/drush

That is consistent with the installation of the Debian version of Drush.  http://packages.debian.org/sid/drush

Information is free, knowledge is acquired, but wisdom is earned.

Error

Just followed the installation instructions and get the following error when I try to run drush help. Any help would be appreciated.

root@drupal6:/tmp# drush help
/usr/bin/drush: line 1: ?php: No such file or directory
/usr/bin/drush: line 2: //: is a directory
/usr/bin/drush: line 4: /bin: is a directory
/usr/bin/drush: line 5: drush-All-Versions-2.1.tar.gz: command not found
/usr/bin/drush: line 6: drush-All-Versions-2.1.tar.gz: command not found
/usr/bin/drush: line 7: drush-All-Versions-2.1.tar.gz: command not found
/usr/bin/drush: line 8: drush-All-Versions-2.1.tar.gz: command not found
/usr/bin/drush: line 9: */: No a file or directory
/usr/bin/drush: line 11: //: is a directory
/usr/bin/drush: line 12: //: syntax error near unexpected token ')'
root@drupal6:/tmp#

error

Found the error the symbolic link is wrong it should be:

ln -s /usr/share/drush/drush /usr/bin/drush

 

Dan

John Carver's picture

Opps Again!

The symlink to /usr/share/drush/drush.php happens to work in my installation, but may not work in all cases.  According to the header in /usr/share/drush/drush

# This script is a simple wrapper that will run Drush with the most appropriate
# php executable it can find.

Somewhere I saw an example linking directly to drush.php and used it without asking why.  I'll modify the instructions to use the more general case.  Thanks Dan, for letting me know.

Information is free, knowledge is acquired, but wisdom is earned.

The drush command 'pm-update' could not be executed

I got the turnkey-drupal6-11.0-lucid-x86 and installed it as a virtual machine using VMWare Workstation 7.1.3. It installs without a hitch.

The problem is when I try to run drush pm-update on the shellinabox:

root@drupal6 /etc/drush# drush pm-update 
Command pm-update needs the following modules installed/enabled to run: update. [error] 
The drush command 'pm-update' could not be executed.

It's the same with the pm-refresh:

root@drupal6 /etc/drush# drush rf 
Command pm-refresh needs the following modules installed/enabled to run: update. [error] 
The drush command 'rf' could not be executed. 

Anyone got a fix?

John Carver's picture

drush pm-enable update

Try the following:

drush pm-enable update

Then try your other commands.  The update module is now part of Drupal core so it doesn't need to be installed.  I thought it was now enabled by default in TKL Drupal appliance, but maybe not in the version you have.  In any case, it needs to be enabled before the Drush update commands will work.

Information is free, knowledge is acquired, but wisdom is earned.

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)