TurnKey Linux Virtual Appliance Library

TKLPatch for Drupal 7

John Carver's picture

Drupal 7.0 was officially released on Jan 5, 2011, yet there is still no Ubuntu package available.  There is a Debian package available for Sid, but attempts to incorporate it into a TurnKey LAMP appliance proved problematic.  I can understand concerns about relying on upstream developers for support, but in this case the availabilty of Drupal 7's built-in updates and Drush's site-install, provide the ability to obtain updates directly from the source.

A new strategy of installing Drush first using git, and then using the power of Drush to download, install, and configure the Drupal site has proven effective.   A TKLPatch to add Drupal 7 to the TurnKey LAMP appliance is now available in the Development Wiki, turnkey-drupal7-11.x-lucid-x86-patch-beta1 and also on GitHub.

<snip>

Moved the remainder of the original post to the Development Wiki where it probably should have been all along.

Jeremy's picture

Excellent work!

I don't use Drupal myself but this looks fantastic! And the beauty of someone such as yourself taking this initiative is that it frees up the core devs for other stuff (because no doubt they would've been planning on releasing a Drupal 7 appliance in the next release anyway).

Good work! :)

Drush Installs

I tried to respond to the original comment but couldn't for some reason.

Most of what I do is Drupal so I'm glad to be able to help out here.  I'm a core contributor (with a bunch of more talented folks for sure).  

I don't install Drupal sites using Drush.  I've been using it too long to try installing it that way, but I think it's possible.  There are some guides here:

There are actually 2 versions of Drush to consider using.  It might be easier to install Drush with PEAR as they recommend here:

  http://drupal.org/project/drush

Drush is certainly a very powerful tool.

Jeremy, who is maintaining the current Drupal 6 appliance? It's really not that much different between D6 & D7 other than that with D7 you can safely use PHP 5.3.

Liraz Siri's picture

The core developers are

The core developers are currently maintaining the Drupal 6 TurnKey appliance. But we're in the process of building out a community development infrastructure that we hope will eventually allow trusted community members to adopt the maintenance of specific appliances like Debian developers adopt packages.

With the community's support we could scale TurnKey into hundreds, maybe even thousands of pre-integrated open source solutions...

John Carver's picture

PEAR installation of Drush

I'll take a look at the PEAR installation of Drush.  I choose the git installation method because it clearly has long term support from Drupal.org.  I'm not thrilled, however, that git clone stores a complete copy of the git repository in the appliance; for Drush that's about 3.6 MB of extra data.  I'm a git noob, so if there is a better way to download a specific version without cloning the repository, please let me know.

I'm puzzled by your reference to 2 versions of Drush.  I choose to use the latest stable release which is currently 4.5.  Now that I'm planning to add drush_make, I'm reconsidering and may use a 5.0 dev release instead to simplify the update process.  If I install Drush 4.5 + drush_make now, then the update scripts have to detect when 5.0 is released and remove drush_make.  Overall, it may simpler to start with 5.0 dev which includes drush_make and use Drush's self-update to handle updates.

Information is free, knowledge is acquired, but wisdom is earned.

Drush Project Page

I'm just going by what's placed in their project description page and our own experience.  I'd suggest asking those questions in the Drush issue queue.

John Carver's picture

Upload Drupal7 ISO

Does anyone have any suggestions about where I can upload the Drupal7-beta1 iso, so testers don't have to setup and patch the LAMP appliance?  The iso is about 300 MB.

Information is free, knowledge is acquired, but wisdom is earned.

Jeremy's picture

Hi John

What about TKL Community Downloads (on SourceForge)? If you're happy with that idea shoot me an email with your SF username and we'll go from there.

Did it get uploaded?

I can probably test it out? I don't have any experience bringing ISO's into VirtualBox, but that's ultimately what I'd want to do to check it out.

A really good starting point for D7 is http://drupal.org/project/quickstart

John Carver's picture

Not Yet

I sent a private message to Jeremy with the information he requested, but I haven't heard back yet.  I'll upload an iso as soon as I have permission for my SourceForge account.

Information is free, knowledge is acquired, but wisdom is earned.

John Carver's picture

Drupal7 beta1 ISO uploaded to SourceForge

The beta1 version of turnkey-drupal7-11.3-lucid-x86-beta1.iso has been uploaded to SourceForge.

http://sourceforge.net/projects/turnkeylinuxcom/files/turnkey-drupal7/tu...

Information is free, knowledge is acquired, but wisdom is earned.

Thanks a lot John, your LAMP

Thanks a lot John, your LAMP appliance is absolutely perfect for people like me who wants to focus on Drupal dev and not on Linux installation/configuration.

Liraz Siri's picture

This is awesome work John!

Happy new year John and sorry for the late response. I've been following the work you've been doing for the TurnKey Drupal 7 tklpatch (on the wiki) and it seems to be really good stuff. When we come out with Drupal 7 officially in the next release this will be a big help and we'll give you credit for pushing Drupal 7 forward.

We're a bit swamped with everything we have going on at the moment so it sometimes takes us a bit of time to get a response out of us but we still stand up and take notice when community members contribute back to the project. Many many thanks!

John Carver's picture

Your Welcome

I started the project because I needed a Drupal 7 base to start experimenting with Drupal's VoIP module.  I decided that if I were doing the work for my personal use, I should release it back to the community as a way of saying thanks for all that the TurnKey developers and community have contributed.

I must say that I deeply appreciate the openness and willingness to accept input that I find here.  Examples include adding Drush to the Drupal appliance and reworking TKLPatch to meet some of my earlier requirements.  What goes around, comes around and so I predict good things for TurnKey's future.

I think most people would agree that Drupal has a steep learning curve that often discourages potential users.  My vision for the Drupal appliance is to collect best-practices from the Drupal.org community and package them in an easy to use appliance that will encourage a wider audience to give Drupal a try.

I'm currently working on a beta2 release of the Drupal 7 patch, which will add drush_make and the ability to select from a list of profiles during firstboot.  In the spirit of openness, I'm still looking for suggestions about which modules and themes should be included in the appliance. 

Information is free, knowledge is acquired, but wisdom is earned.

Ok, some related TKL

Ok, some related TKL Links:

Don't worry too much about bundling modules.  You can do something simple like that if you want, but there are some great Drupal distrobutions out there which can really help speed up the process both of choosing good combinations of modules but also allowing for optimized configurations.

You could possibly include all of these distrobutions re-using the same base Drupal code base:

Many of these install profiles are also available via drush_make but not all.

I don't know that there is an existing VOIP Distro, but looking from the module to the discussion around integration it looks like it would be a good idea, some links here too:

I haven't been involved at all in this other than peeking at development every now and again.

I look forward to hearing about your progress learning Drupal.  Like any complex tool it takes time to learn.

Drupal 7, Varnish & Git

Ok, I've got a Drupal 7.10 instance up and running now and am finally serving live sites from it. 

How would I contribute this back? Is there a way to export a diff of the server environment from the initial Drupal 6 instance? 

If there's a way to export a list of added packages & configs I'd love to contribute that back.

John Carver's picture

Try Drush Make

For the list of packages, you can use drush_make to generate a makefile specific to your site.

drush generate-makefile yoursite.make

That gives you a makefile that can be used by drush_make to install the same packages, libraries, and patches on another site.  Unfortunately, it does not include configs that are stored in the database.  For those, you will have to consider the context and features modules.  For more details see the Development Seed Blogs:

http://developmentseed.org/blog/2009/oct/27/drupal-distributions-drush-m...

http://developmentseed.org/blog/2009/jul/09/development-staging-producti...

Information is free, knowledge is acquired, but wisdom is earned.

Lots of options

There are always lots of options in Drupal.  We've not chosen to use drush_make but there's nothing wrong with it for many. 

Would be a nice way to set up a bunch of default modules for a Drupal 7 application for sure!

John Carver's picture

drush-4.5 vs drush-5.0-dev

I had hoped to take advantage of some drush-5.0 features, specifically better handling of file permissions and built-in drush make, but I've found that drush-5.0-dev is still a moving target (no stable release) and new issues have cropped up, e.g.
Make cache directory selection more robust in situations where selected location is not writable

For now, I've decided to stick with the latest stable release of drush, drush-4.5, and separately install drush_make-6.x-2.3, although it will have to be removed later when upgrading to drush-5.0.

Information is free, knowledge is acquired, but wisdom is earned.

John Carver's picture

drush download vs drush make

For years, drush download is the method I've used to build websites and it's the method used in the beta1 version of the Turnkey Drupal7 patch.  In the past several weeks, I've been experimenting with an alternative method of building sites using features, profiler, and drush_make described by Dmitri Gaskin at DrupalCon Chicago, "From Zero to Distribution using Features, Profiler, and Drush Make".  Dmitri is the author of Drush Make.  Drush Make has been adopted by the Drush team and is part of Drush 5.0 core, so it's safe to assume it will be around for the long term.  I thought it would be useful to document here some of the differences between the two methods.

drush download drupal

  • downloads and unpacks drupal core to a specified location, e.g. /usr/share/drupal7
  • any special arrangement of files & directories, ala the debian package, must be handled after the download
  • if destination location exists, drush download offers to overwrite it
  • the settings.php file must be created with rw permissions, if using install.php
  • ownership of sites/default/files must be changed to www-data
  • extra packages containing modules and themes can be downloaded and unpacked
  • * ownership of modules & themes defaults to that set by the package developer, usually uid=6226, and must be changed after each download
  • patch installation must be individually scripted
  • does not create a site database and db_user

* fixed in Drush-5.x

drush make site.make </path/to/destination> (drush make version 2.3)

  • recursively downloads and unpacks drupal core and any profiles, modules or themes specified in site.make or any profile.make found therein
  • if /path/to/destination exists, even if empty, drush make refuses to continue
  • any special arrangement of files & directories, ala the debian package, must be handled after the drush make
  • package versions and patches can be specified in .make files
  • ownership of files & directories set to user running drush make
  • drush make seems to lack a local file option for profiles.  Custom profiles & patches must be hosted externally or via http://localhost
  • --prepare-install option does not appear to work in version 2.3
    • the settings.php file must be created with rw permissions
    • ownership of sites/default/files must be changed to www-data
  • does not create a site database and db_user

Information is free, knowledge is acquired, but wisdom is earned.

John Carver's picture

install.php vs drush site-install

Downloading and unpacking Drupal core and the desired contrib modules using drush download or drush make is only the first step in building a Drupal site.  Next the database must be created, populated with data, and the initial configuration applied. Once again we have two choices, Drupal's built-in install.php or Drush's site-install.

install.php

  • prompts user for information about the site and the site administrator
  • allows user to chose from a list of profiles
  • install.php cannot create the site database and dbuser
  • settings.php must exist and be writable during install and manually changed to read-only after
  • webserver must have write access to sites/default/files

drush site-install

  • information about site, profile, and site administrator input via command-line
  • can create the site database and dbuser, if supplied the MySQL root password
  • creates settings.php from information supplied on command-line
  • leaves settings.php world readable
  • creates sites/default/files owned by the uid running drush, usually root

Information is free, knowledge is acquired, but wisdom is earned.

John Carver's picture

site-builder

For the past several weeks, I've been working on a site-builder script based on 'drush make' and 'drush site-install'.  The strategy was to move all the site building parts from the TKLpatch scripts of beta1 into a common script that could be called during firstboot.  As the script began to take shape, I realized that with a few changes, it could be structured to handle multi-site installations and perhaps a mix of Drupal 6 and Drupal 7 sites.

Initially I had hoped it would be as simple as:

drush download profile
drush site-install profile ...

but of course reality is never that easy.

Currently, the core of site-builder has been tested on several profiles and is able to deliver a complete functioning website.  It does, however need some work on handling error cases.  I also have to learn enough python to write the user interface which I've been avoiding until now.  I hope to have this completed and rolled into a beta2 patch in about two weeks.  I thought I should post a progress report just in case anyone is wondering.

Information is free, knowledge is acquired, but wisdom is earned.

John Carver's picture

Roadmap

Roadmap for the Drupal 7 TKLpatch

beta1: Proof of concept and laying the foundation for later development.

beta2: Develop site-builder, a script which can download a profile and build a website on-the-fly.  Multi-site capabilty was moved ahead and incorporated in the initial design of site-builder.

beta3: Focus on security and file ownership/permissions.  Define the roles of server-admin and site-admin.

beta4: Focus on performance and SEO incorporating best-practices where possible.

Information is free, knowledge is acquired, but wisdom is earned.

Drupal 7 patch

I am trying to use your Drupal TKL patch on the Turnkey-lamp appliance.   It typically fails with a "no such directory error".   I have tried patching an exiting installation and an iso file with similar results.

I am using the root id in the /tmp directory.   What can I do different?  Thank you for your help.

Jim

 

 

 

root@lamp /tmp# tklpatch turnkey-lamp-11.3-lucid-x86.iso turnkey-drupal7-11.x-lucid-x86-patch-beta1.tar.gz
# extracting root filesystem and isolinux from ISO
Parallel unsquashfs: Using 2 processors
31815 inodes (33205 blocks) to write
 
[==================================================================================================/] 33205/33205 100%
created 29260 files
created 3217 directories
created 1958 symlinks
created 86 devices
created 0 fifos
TKLPATCH_ISOLABEL: turnkey-drupal7-11.x-lucid-x86-patch-beta1
# extracting patch turnkey-drupal7-11.x-lucid-x86-patch-beta1.tar.gz to /tmp/tmp.0GlHPQqfUG
fatal: no such directory: /tmp/tmp.0GlHPQqfUG/turnkey-drupal7-11.x-lucid-x86-patch-beta1
root@lamp /tmp# ls -l
total 205008
drwx------  3 root root      4096 May  9 15:22 tmp.0GlHPQqfUG
-rw-r--r--  1 root root      6887 May  9 14:37 turnkey-drupal7-11.x-lucid-x86-patch-beta1.tar.gz
drwxr-xr-x  3 root root      4096 May  9 15:22 turnkey-lamp-11.3-lucid-x86.cdroot
-rw-r--r--  1 root root 209907712 May  9 15:18 turnkey-lamp-11.3-lucid-x86.iso
drwxr-xr-x 20 root root      4096 Dec  5 13:53 turnkey-lamp-11.3-lucid-x86.rootfs
root@lamp /tmp# cd tmp.*
root@lamp /tmp/tmp.0GlHPQqfUG# ls -l
total 4
drwxr-xr-x 5 root root 4096 Dec 23 17:05 drupal7
root@lamp /tmp/tmp.0GlHPQqfUG# cd drupal7
root@lamp tmp.0GlHPQqfUG/drupal7# ls -l
total 20
-rw-r--r-- 1 root root  323 Dec 16 00:26 README.txt
-rw-r--r-- 1 root root  908 Dec 23 17:05 changelog
drwxr-xr-x 2 root root 4096 Dec 22 18:30 conf
drwxr-xr-x 2 root root 4096 Dec  3 04:43 debs
drwxr-xr-x 5 root root 4096 Dec 16 02:44 overlay
John Carver's picture

My fault for not catching

My fault for not catching this.  In my development environment, I keep all my patches in /opt/patches/... and the iso files in /opt/iso/...  While testing I used the patch directory, /opt/patches/drupal7, instead of the tar.gz file which I didn't create until ready to release and apparently didn't test the final time.  The problem is that the tar.gz is extracting to tmp.0GlHPQqfUG/drupal7 instead of tmp.0GlHPQqfUG/turnkey-drupal7-11.x-lucid-x86-patch-beta1 where tklpatch expects to find the extracted patch.

Probably the easiest workaround is to create /opt/patches and manually extract the patch file there.  Then issue the tklpatch command as

/tmp# tklpatch turnkey-lamp-11.3-lucid-x86.iso /opt/patches/drupal7

If that doesn't work, let me know.  In the meantime I'll try to rework and test the patch and issue a new release.

Information is free, knowledge is acquired, but wisdom is earned.

Jeremy's picture

Easily done....

I have done similar stuff myself. For future reference the best way to avoid this in future is to use the tklpatch-bundle command instead of manually tar.gz the files.

So:

cd /opt/patches
tklpatch-bundle drupal7

The only thing is that you won't get the nice version info, but it encourages you to rename the folder when you are working on it which probably isn't a bad thing.

John Carver's picture

Actually I did create the

Actually I did create the patch using tklpatch-bundle as you suggested, but that resulted in a patch named drupal7.tar.gz.  I didn't like the fact there was no version information so I renamed the file to fit what looked like a standard way of naming patches, i.e. turnkey-drupal7-11.x-lucid-x86-patch-beta1.tar.gz.  Unfortunately, I never thought to test after making the name change.

I use git for tracking changes so renaming the folder for each release isn't an attractive option.  I'd like to have tklpatch-apply ignore the name of the patch folder.

/usr/bin/tklpatch-apply:

add
name="$(basename $patch_tmp/*)"
  before
patch_dir=$patch_tmp/$name

This assumes that the patch tarball will always have exactly one base folder that contains the patch files.  I thought about eliminating the base folder entirely, but that would break compatibility with older patches.  This worked for the one test case I tried, which was to apply my renamed patch.

Perhaps the larger question is there a preferred naming scheme for patches that indicates which core or appliance they should be applied to, and what is the version of the result?  The name I chose doesn't effectively convey that the patch should be applied to a turnkey-lamp-11.x-lucid-x86 appliance and results in a turnkey-drupal7-11.x-lucid-x86-beta1 appliance.  Should a patch specify which appliance(s) and versions it can be applied to, and how could tklpatch make use of that information?


Information is free, knowledge is acquired, but wisdom is earned.

Jeremy's picture

Nice one John!

Ahh, you're about 5 steps in front of me by the look of it! :)

I too use git for tracking patches, but I copy the patch folder when I create a patch to share. To use your example as an example:

cp -r drupal7/* turnkey-drupal7-11.x-lucid-x86-patch-beta1/

then:

tklpatch-bundle turnkey-drupal7-11.x-lucid-x86-patch-beta1

But I quite like your workaround.

On the other hand, another way would be to include a switch in tklpatch-bundle which copies the patch folder to a (temp) folder of the desired name prior to bundling. It's usage might look something like:

tklpatch-bundle --name=turnkey-drupal7-11.x-lucid-x86-patch-beta1 drupal7/

I don't know... Just thinking out loud really...

Regardless, TKLPatch is on the TKL GitHub account so feel free to have a play. If you have a GitHub account then you can fork it, then issue a pull request and if the core devs like you're work they'll probably include it in future releases of TKLPatch.

As for naming convention, there certainly isn't an official one. Many people have simply named there patches like 'drupal7.tar.gz' (to use your example). I too like to include some version info, however mine has been lazier than yours. Mine have been more along the lines of 'drupal7-0.1.tar.gz'. Your more extensive naming is definately superior as you can see at a glance what it is intended for (and hence get a good idea off the bat whether it should work or not).

patch workaround

Yes John this works!  I now have my drupal7 appliance working.  thank you.  Jim

John Carver's picture

Glad to hear you got it

Glad to hear you got it working.  Now that I know my mistake, I'll be more careful when releasing future patches.  Watch for the beta2 version which will have multi-site capability.

Information is free, knowledge is acquired, but wisdom is earned.

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)