TurnKey Linux Virtual Appliance Library

TKLPatch for Gitorious

This TKLPatch will convert turnkey-core-beta-lucid-x86 image to a Gitorious appliance setup to work in a private network. I'm not a git user yet but I'm learning quick (to be a cool rails dev). That said, it's important for me that if any community member has experience with git, please give this appliance a test drive and feel free to post bugs / recommendations. 

IMPORTANT NOTE: you must add gitorious.localdomain to your OS hosts file to access this appliance using this url. Hosts file is located:
  Windows: C:\windows\system32\drivers\etc\hosts
  Linux: /etc/hosts
  Mac: ???
  
Just add a line with the appliance IP and the hostname. Ex:
        192.168.1.100   gitorious.localdomain
 

Features:

Deployed on apache using:
  Ruby Enterprise 1.8.7
  Passenger 2.2.15
Apache ActiveMQ
Postfix MTA 
Access url: http://gitorious.localdomain/
            https://gitorious.localdomain/
User: admin@example.org
Pass: turnkey
 

What it does:

 
1. Sets hostname to gitorious.localdomain
 
# Set hostname
HOSTNAME="gitorious"
echo "$HOSTNAME" > /etc/hostname
sed -i "s|127.0.1.1 \(.*\)|127.0.1.1 $HOSTNAME $HOSTNAME.localdomain|" /etc/hosts
hostname gitorious
 
2. Updates packages information
 
apt-get update
 
3. Download and installs Ruby Enterprise & RubyGems 1.3.7. Creates symlinks to usr/bin
 
# Install Ruby Enterprise and RubyGems
wget http://rubyforge.org/frs/download.php/71100/ruby-enterprise_1.8.7-2010.0...
dpkg -i ruby-enterprise_1.8.7-2010.02_i386_ubuntu10.04.deb
rm ruby-enterprise_1.8.7-2010.02_i386_ubuntu10.04.deb
ln -s /usr/local/bin/ruby /usr/bin/ruby
wget http://production.cf.rubygems.org/rubygems/rubygems-1.3.7.tgz
tar xf rubygems-1.3.7.tgz
cd rubygems-1.3.7/
ruby setup.rb
ln -s /usr/local/bin/gem /usr/bin/gem
cd /root/
rm -Rf rubygems-1.3.7/
4. Install apache, git, openjdk (for ActiveMQ), sphinx, mysql-server, Xsendfile apache module (needed by gitorious to work with tars), apg (for random key creation), Memcached for better performance, and other requirements. Gitorious uses sphinx to accelerate database searches, through a gem called ultrasphinx. 
 
# Install required packages
install build-essential apache2-mpm-prefork apache2-prefork-dev libapr1-dev libaprutil1-dev git-core git-svn git-doc git-cvs sphinxsearch postfix memcached mysql-server imagemagick libmagickwand-dev aspell libaspell-dev libapache2-mod-xsendfile jsvc openjdk-6-jre-headless apg
5. Install passenger and apache modules. Configure passenger to use Ruby Enterprise.
 
gem install --no-ri --no-rdoc --version 2.2.15 passenger
/usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.15/bin/passenger-install-apache2-module -a
echo "LoadModule passenger_module /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.15/ext/apache2/mod_passenger.so" > /etc/apache2/mods-available/passenger.load
echo "<IfModule mod_passenger.c>" > /etc/apache2/mods-available/passenger.conf
echo "   PassengerRoot /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.15" >> /etc/apache2/mods-available/passenger.conf
echo "   PassengerRuby /usr/local/bin/ruby" >> /etc/apache2/mods-available/passenger.conf
echo "</IfModule>" >> /etc/apache2/mods-available/passenger.conf
a2enmod passenger
a2enmod rewrite  
a2enmod ssl 
echo "ServerName gitorious.localdomain" > /etc/apache2/httpd.conf
service apache2 restart
6. Download & install Active MQ. Configured to use STOMP as comunication protocol. To use as a service, activemq file is overlayed to /etc/init.d
# Install ActiveMQ
wget -O apache-activemq-5.3.2-bin.tar.gz http://www.carfab.com/apachesoftware/activemq/apache-activemq/5.3.2/apac...
tar xf apache-activemq-5.3.2-bin.tar.gz -C /usr/share/
mv /usr/share/apache-activemq-5.3.2/ /usr/share/activemq
rm apache-activemq-5.3.2-bin.tar.gz
cp /usr/share/activemq/conf/activemq-stomp.xml /usr/share/activemq/conf/activemq.xml
chmod +x /etc/init.d/activemq
7. Install required gems. All gems installed without documentation as this is a production system. Also made a symlink to rake in /usr/bin
# Install required gems
gem install rack --no-rdoc --no-ri --version 1.0.1
gem install rails --no-rdoc --no-ri --version 2.3.8
gem install rdiscount --version "= 1.3.1.1" --no-ri --no-rdoc
gem install stomp --version "= 1.1" --no-ri --no-rdoc
gem install ruby-openid --version ">= 2.0.4" --no-ri --no-rdoc
gem install --no-ri --no-rdoc ultrasphinx raspell mysql rake chronic rmagick json ruby-hmac geoip daemons echoe ruby-yadis ruby-openid mime-types diff-lcs
ln -s /usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/bin/rake /usr/bin/rake
8. Create user and group git. Added apache user (www-data) to git group.
 
#Create git user and group
adduser --system --home /var/www/gitorious/ --no-create-home --group --shell /bin/bash git   
usermod -a -G git www-data
 
9. Install gitorious by cloning the gitorious repository. Also configured the 3 main files (database.yml, broker.yml, and gitorios.yml, which was previously overlayed to /root/ and it's completed in this step adding the cookie_secret property as recommended by gitorious (using apg)
 
# Install gitorious
git clone git://gitorious.org/gitorious/mainline.git /var/www/gitorious
cd /var/www/gitorious

echo "production:" > config/broker.yml
echo "  adapter: stomp" >> config/broker.yml

echo "production:" > config/database.yml
echo "  adapter: mysql" >> config/database.yml
echo "  database: gitorious_production" >> config/database.yml
echo "  username: root" >> config/database.yml
echo "  password: " >> config/database.yml
echo "  host: localhost" >> config/database.yml
echo "  encoding: utf8" >> config/database.yml

export SECRET=`apg -m 64 | tr -d '\n'`
cat /root/gitorious.yml | sed \
  -e "s/cookie_secret\:.*$/cookie_secret\: $SECRET/g" \
  > /var/www/gitorious/config/gitorious.yml
rm /root/gitorious.yml
10. Migrate production database, bootstrap ultrasphinx gem, and create initial user/pass. The script /root/create_admin_turnkey is a copy of script/create_admin script, modified to add the admin@example.org user and pass turnkey. 
 
sed -i 's/0.0.0.0/localhost/' config/ultrasphinx/default.base
RAILS_ENV=production rake db:create
RAILS_ENV=production rake db:migrate
RAILS_ENV=production rake ultrasphinx:bootstrap
RAILS_ENV=production /root/create_admin_turnkey
rm /root/create_admin_turnkey
 
11. Create various directories needed by the app and set correct ownership and permissions. Also create the gitorious symlink in /usr/bin.
mkdir -p /var/git/git-repos
mkdir -p /var/git/tarballs-cache
mkdir -p /var/git/tarballs-work
mkdir /var/www/gitorious/tmp/pids
mkdir /var/www/gitorious/.ssh    
touch /var/www/gitorious/.ssh/authorized_keys    
chown -R git:git /var/www/gitorious
chown -R git:git /var/git
chmod -R ug+rwx /var/git
chmod 700 /var/www/gitorious/.ssh    
chmod 600 /var/www/gitorious/.ssh/authorized_keys
ln -s /var/www/gitorious/script/gitorious /usr/bin
 
12. Disable default apache website and enable gitorious and gitorious-ssl websites (previously overlayed)
 
a2dissite default
a2ensite gitorious    
a2ensite gitorious-ssl
service apache2 reload
13.Configure overlayed scripts to start at system startup
 
#Make init scripts start on boot
chown git:git /etc/init.d/git-ultrasphinx
chown git:git /etc/init.d/git-daemon
chown git:git /etc/init.d/git-poller
chmod 755 /etc/init.d/git-ultrasphinx /etc/init.d/git-daemon /etc/init.d/git-poller
update-rc.d git-daemon defaults  
update-rc.d git-ultrasphinx defaults  
update-rc.d git-poller defaults
update-rc.d activemq defaults
14. Stop services and cleanup apt cache.
 
#Stop services and cleanup
service mysql stop
service apache2 stop
cleanup_apt

 

Enjoy it! Please send some feedback in this forum. 

Attached: 

Somehow doesn't work for

Somehow doesn't work for me.... First run on a fresh unsquashfs fails with "passanger.so not found", consecutive runs fail with "can't connect to mysql through socket"....

Please give more info to help you

Hi Rtreffer, thanks for trying the patch. Could you please provide more info on the patching process? yo can do it by issuing the following command:

tklpatch turnkey-core-beta-lucid-x86.iso gitorious.tar.gz | tee tklgit.log

And attach tklgit.log file to see what's happening during the patching process. 

I'll try that, I've so far

I'll try that, I've so far patched an unsquashfs'ed 10root.squashfs with tklapply-patch /mnt/ /home/root/gitorious/

Direkt tklpatch is slightly better, here is the output: http://pastebin.com/G8yTqg8j

I should point out that the

I should point out that the mysql user creation fails, this is critical as it prevents any logins.

Jeremy's picture

Just had a quick browse

And it looks like it must be a pretty complex patch (way above my pay grade)! I'm not sure if I'm headed in the right direction or not (very newb) but I found this (starting on line 2747):
	Warning: The home dir /var/www/gitorious/ you specified can't be accessed: No such file or directory
Adding system user `git' (UID 107) ...
Adding new group `git' (GID 111) ...
Adding new user `git' (UID 107) with group `git' ...
Not creating home directory `/var/www/gitorious/'.
Initialized empty Git repository in /var/www/gitorious/.git/
Couldn't create database for {"encoding"=>"utf8", "username"=>"root", "adapter"=>"mysql", "database"=>"gitorious_production", "host"=>"localhost", "password"=>nil}, charset: utf8, collation: utf8_unicode_ci (if you set the charset manually, make sure you have a matching collation)
(in /var/www/gitorious)
rake aborted!
Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

Could it be as simple as the fact its not creating /var/www/gitorious ? Why would it not be creating the dir? I'm assuming that it should be created as the user 's git home dir?

Sorry more questions than answers :)

Problem solved after re-installing mysql

Re-install mysql-server and mysql-client using this command:
 

sudo apt-get install mysql-server mysql-client

and then some libraries you need to install to make MySQL available to ruby:
 

sudo apt-get install libmysql-ruby

This all solved my problem. Try it !!! :)

The problem is at the MySQL database creation.

Looking at the output you posted, the errors start to show from the moment of migrating Gitorious database 


(Couldn't create database for {"encoding"=>"utf8", "username"=>"root", "adapter"=>"mysql", "database"=>"gitorious_production", "host"=>"localhost", "password"=>nil}, charset: utf8, collation: utf8_unicode_ci (if you set the charset manually, make sure you have a matching collation)
(in /var/www/gitorious)
rake aborted!
Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

I'll run again this patch in my dev environment. Mysql server seems to have started correctly, as seen in line:

1514: mysql start/running, process 21744

The user home is not created at the moment of the git user creation, and it also doesn't exists at that moment, but after that, the gitorious repository is being cloned at /var/www/gitorious creating that home directory for the user. So that's not the problem, I could create the user after cloning Gitorious and the warning would disappear. I'll update it. Let me check and I'll come back to you in a moment. 

Just for curiosity: did you upgrade tklpatch to its lastest version? Are you patching from within Turnkey Core Lucid or in another environment? 

This is indeed a very complex patch. So many things I was going crazy with each test. Thanks for trying it. 

Jeremy's picture

I saw the MySQL err and thought it may have related to path err

So I was off track but I appreciate you taking the time to explain where :)

I haven't actually tested your patch, just sticking my newb nose in and getting a bit down and dirty with stuff way over my head! I had a scan through Guest's error code link (above) and found that bit (there may be more?) I did notice that there were references to sub dirs of /var/www/gitorious after the errors but thought perhaps the MySQL failure was somehow related to the path error (not sure how though).

The appliance looks really cool and I will defiantely have a go at it soon hopefully. I think it could be a really cool tool for projects I work on, both at home and at work. Oh so many things I want to do and so little time!

I reckon you'd be going crazy with it! I can handle a bit of basic code/commandline but serious stuff like that does my head in!

It's working for me

Here's my log when running the patch: http://pastebin.com/vBbtZkqY

Take a look at this differences:

Yours (starting mysql-server) lines  1514-1515

mysql start/running, process 21744
 

Mine (starting mysql-server) lines 798-801 (fewer lines because I only posted tklpatch-apply output)

 

* Starting MySQL database server mysqld       
[ OK ]
 * Checking for corrupt, not cleanly closed and upgrade needing tables.
The other difference, of course, is during the migration of the DB. Take a look at my output in lines 1339-2153
 
(in /var/www/gitorious)
(in /var/www/gitorious)
==  CreateUsers: migrating ====================================================
-- create_table("users", {:force=>true})
   -> 0.0365s
-- add_index(:users, :login)
   -> 0.0183s
   .... blah blah blah
And finally line 2153:
Admin user created successfully.
 
So there may be something different in your patching environment. I'm using tklpatch version 
 
	dpkg -l tklpatch
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                              Version                           Description
+++-=================================-=================================-==================================================================================
ii  tklpatch                          0.93+5+ga2bd433                   TurnKey Linux Customization Mechanism
Also, try again in a clean (recently booted) system. It could be that your previous attempts left some locked file in the fs that could be causing problems. I hope this helps, please post your results.

Thanks for the suggestions,

Thanks for the suggestions, I'm retrying the patch right now.

mysql is indeed installed on the host but was offline (ps aux|grep [m]y -> no results), I've just rebooted and will try to patch again. Host OS is maverick right now, with tklpatch 0.93 (from the lucid repository, no +5-ga... though).

Liraz Siri's picture

Awesome work. A few minor comments

Finally had a chance to sit down and investigate this patch a bit more thoroughly.

First off, excellent work. This is a complex patch and it looks like a lot of work went into getting everything right. The only things that caught my attention were a few violations of the FHS (Filesystem Hierarchy Standard). This is the standard Debian-based systems are advised to follow with respect to what belongs where (e.g., data, configuration files, local binaries, stuff controlled by package management, etc.).

For example:

  • Re-linking /usr/bin/ruby to a path in /usr/local. Ideally you would do that as a last resort if you can't achieve the same effect by manipulating PATH in the environment.
  • Putting various helper scripts in /root. I'd put them into /tmp if you want to delete them later (I.e., they're temporary), or if it's useful to keep them around /usr/local/bin.

In case you're wondering if this technical bureaucracy serves any other point besides slowing down system integrators, yes it actually does make sense when you take package management into consideration. If you replace /usr/bin/ruby for example, the next package update for ruby will overwrite your change and break the system. Also, if you're making an efficient backup of the system (e.g., TKLBAM) you can save a lot of space by just backing up those directories which contain your local customizations / data and recreating the rest of the system by syncing package management states.

Ruby Enterprise Edition in BrightBox.Net APT Repository

There is actually no need at all to install Ruby Enterprise Edition by hand and symlinking it. BrightBox (a Ruby on Rails hosting company) provides an APT Repository at http://APT.BrightBox.Net/ with Ruby Enterprise Edition and many Rails-related Debian packages (like the Passenger module for Apache or a patched version of NGinX with the Passenger module NGinX). Their REE package installs into /usr/bin just like any other Debian package. And it provides the ruby package, which means you can simply install it with aptitude install ruby and it will get REE, since you can be pretty much guaranteed that BrightBox’s packages will always be newer than Debian’s.

This is awesome

This is a very good tutorial after searching ing google just too many times for a working recipe for gitorious finally.

 

Excellent work man!

Thanks!

I know what you're talking about google! I read almost 3 or 4 to come to this. Just remember that being a TKLPatch, some files must be overlayed (copied) to the file system before running this steps. You could wait to see if this patch gets converted to an official appliance in the library, which will make it easier to install.

How can I apply the patch?

How can I apply the patch?

There is no documentation about it but it's fairly easy

I checked the documentation and it seems there's no tutorial to apply a patch. But it's easy, follow the directions of JedMeister in this thread http://www.turnkeylinux.org/forum/general/20100819/turnkey-linux-and-tklpatch and if you find yourself lost feel free to ask for more help. 

Liraz Siri's picture

Wrote a tutorial on how to apply a patch

I wrote a tutorial on how to apply a TKLPatch. Seems lots of people have had trouble applying patches so that (along with Adrian's comment) made me suspect the existing documentation is inadequate. Hope this will clear things up.

Jeremy's picture

Certainly doesn't hurt to tidy it up a bit

But why the recommendation to uncompress prior to application of patch? Personally I think it would be a better idea to clarify in the patch construction and the importance of using the tklpatch-bundle command to create your patch. This way the potential pitfalls for users applying the patch are avoided and it makes it less daunting for TKL/Linux newbs (all you have to do is tklpatch tklimage.iso tklpatch.tar.gz). Perhaps it'd be better to create a new file extension for tklpatches such as .tklp or .patch (or similar etc)? Obviously it'd still be a .tar.gz behind the scenes, just with an alternate extension. I know its only one small line of code, I just think newbs can be turned off pretty easily.

Also for ease of use (and reduce the likelihood of issues) I'd be inclined to keep the tutorial really basic with simple instructions for patching ISOs. All other options can be listed in the full documentation. Basically keep the 'quick ref for newbs' really basic and clean, with fuller explanation held back for the 'full' documentation.

Also while I'm meandering off topic all over the forums tonight (my time) I'd like to add that I'd really love to see some sort of TKLPatch database/table type page where all the patches are in the one place and basically outlined (like they are/were on the dev wiki) but with them available for download irght there too. This could also mention if they have been published as an appliance or not and users could add info on which patches they had successfully applied to which appliances (sort of mix and match as many have asked for here previously). I would suggest that this usage scenario not be 'officially' supported but troubleshooting etc could be welcome in the forums for those interested (similar to what Proxmox do for certain features they do not wish to support - discussion is welcome on the forums, but the devs rarely involve themselves).

Alon Swartz's picture

Removed the extract section

The extra doc page was definitely needed. I also noticed the "extract tklpatch" part, and agree it's not necessary. We don't want to encourage bundling and patching of tklpatches adhoc style (that's why tklpatch-bundle exists, and patching from the tarball as well).

Weird errors during installation

I get these errors when the install procedure reaches something between 70 and 80% idk..

http://image-upload.de/image/rIoIsV/f315cac258.png

After waiting some time, the install finished though. And it "seems" to work. But I wanted to mention it.

I used this command, sorry I don't have the tklgit.log anymore :(

 

tklpatch turnkey-core-beta-lucid-x86.iso gitorious.tar.gz | tee tklgit.log

What was your patching environment?

Maybe it has something to do with your patching environment, do you have a postfix running on the system where you applied the patch? It's preferably to patch in an isolate environment (a VM with TKL Lucid Core) to prevent errors. You can also try (if you have a postfix running) to stop the service before patching. 

Just for peace of mind, it's better to repatch without errors so if you get into any issue in the future, we don't suspect of these errors during patching. 

Thanks for reporting!

Almost works: I've repatched it

I've repatched it in the same vm (no postfix) and it works now.

hmm I encounter one problem when I actually want to use gitorious in production.

So when I enter "http://myurl.com" it redirects me to "http://gitorious.localdomain".

Could you help me solving that please?

Yes, check these points:

First be sure that your host fqdn is properly set. Issue the following command in a terminal: hostname -f (or cat /etc/hosts) correct the appropiate entries if needed. (check step 1 if you need help)

Second, change "ServerName gitorious.localdomain" in /etc/apache2/httpd.conf to myurl.com. You can do this with

sed -i "s/ServerName gitorious.localdomain/ServerName myurl.com/" /etc/apache2/httpd.conf

Third, you have to change the following settings in /var/www/gitorious/config/gitorious.yml:

gitorious_client_host: gitorious.localdomain
gitorious_host: gitorious.localdomain

You can do both changes with:

sed -i "s/gitorious.localdomain/myurl.com/" /var/www/gitorious/config/gitorious.yml

Be sure to backup all files in case anything goes wrong. Please tell me if this works for you. If you loose repos is normal, as gitorious is able to server in multiple domains, and when you make a change, it won't reconize the other domain projects/repos. At least this is what I've read. Thanks for testing the appliance, your feedback is welcome. 

P.S. Restart apache once all changes are made:

service apache2 restart

Found it.

OK. I found it. For all others you have to edit /var/www/gitorious/config/gitorious.yml

 

gitorious_client_host: YOUR_URL_HERE
gitorious_host: YOUR_URL_HERE
public_mode: true
 
I think making this configurable over the webinterface would save some time for newbies.
 
By the way: Really great work Adrian!!
I highly appreciate and recommend it!
 
Cheers
Fernandos

Thanks!

Be warned that public mode will let anyone open an account and create repos. Be sure that's what you want. Private mode means an administrator create accounts/repos. I'll think I'll add a helper script for this

For what its worth, i would

For what its worth, i would LOVE to see this as a appliance, perhaps as part of an updated http://www.turnkeylinux.org/revision-control

Would a bounty or donation help speed that along? Does it have a estimate on when it might be turned into a full appliance? 

Hi a_c_m

It is very possible that this patch become an official appliance, for the new release of Turnkey Linux based on Ubuntu Lucid. But there is no official date when this is going to happen, the Core Devs are in the middle of producing a second beta of the Core, then they'll update the existing appliances, and finally convert some of the patches to official appliances. I'm hoping too this one gets in. 

I encourage you to make a donation if it is in your possibilities, but cannot guarantee that would speed things up, as the team is small and the work to be done is immense. But it surely helps the project to be maintained and this kind of appliances to be finished and available to you and all the community. 

Keep in touch for the evolution of this patch and let's hope it's ready before the year ends. 

Any Updates?

I've been following this thread in hopes of the full appliance appearing.  Just wondering if there are any updates on this?  I also, would really LOVE to see this as a full appliance.  GREAT WORK ADRIAN!

Soon...

Be patient, Turnkey-lucid already have been published and next week a version with some fixes will be available. The second part of the releases will include Gitorious. I'm also waiting for some of my other patches to get to the second batch of releases, I know the tkl guys are working hard to make this happen. 

Meanwhile, have you tryed this version? There's still time for some feedback before the final appliance goes live. 

are we there yet?

Hi there,

I was going to try applying your patches on the released turnkey-core-11.1-lucid but it's been 9 days since your "be patient" post, so I'm thinking I should just wait for the Gitorious appliance to show up.

Is there a beta release we can help test?

Reid

Jeremy's picture

Hopefully soon

But soon is relative. Our beloved TKL devs do not like to give clear timeframes as they don't like to dissapoint. Whilst you could try it out (by patching TKL Core) it would be better to wait if you plan to put it into production.

couldn't wait.. :-)

I am running "tklpatch turnkey-core-11.1-lucid-x86.iso gitorious.tar.gz" even as I type...

I'll report here if there are any problems.

Reid

Gitorious fails to run

I uploaded a screenshot to Flickr:

http://www.flickr.com/photos/clith/5388084035/sizes/o/

Looks like I got the exact same result as a_c_m back on Jan 2 <https://img.skitch.com/20110102-duj7ifmku3uafbcph3sde37qqd.jpg>. No patch update over the last 4 weeks?

Reid

Jeremy's picture

That sucks :(

I've got no idea why it's doing that (well out of my league). Not sure whether Adrian has plans to update his patch or not though seeing as the TKL guys have started working on Part 2 of the v11 release (which will include Gitorious). I'm guessing from his comments above that he may not be updating it, if that's the case then you'll either need to try debugging it yourself, or wait for the official release.

Sorry I've been busy today to follow your comments

Please do the following: when applying the patch, add an extra > gitorious.log and post it to check for errors during patch.

 

tklpatch turnkey-core-11.1-lucid-x86.iso gitorious.tar.gz > gitorious.log

Also: you can try what the error is suggesting, that is, go to /var/www/gitorious and run 

rake gems:install

This patch was done a while ago, and I haven't tested with the stable releases of turnkey-core. I'm also not updating it as the core tkl-devs will release this appliance soon as part2 of turnkey-lucid. But I have no problem giving you some support in the meantime. 

Tried installing missing gem

 

It was missing the "hoe" gem, so I did a "gem install hoe" and now I'm getting a screen saying:
 
We are sorry, but something went wrong.
We have been notified about this issue and we will take a look at it shortly.
 
I looked in /var/log/apache2/error.log and it's saying:
 
sendmail: fatal: git(107): No recipient addresses found in message header
postdrop: warning: stdin: unexpected EOF in data, record type 78 length 124
postdrop: fatal: uid=107: malformed input
 
I will try re-doing the patch and save the output for you.
 
Oh, I also did a "rake gems:install" but actually, since there was only a production environment, I had to do "env RAILS_ENV=production rake gems:install" instead :-)
 
Reid

How to fix sendmail: fatal: git(107): No recipient addresses err

Hello, 

While registering a user in gitorious web panel got 500 internal server error. 

 I looked in /var/log/apache2/error.log and it's saying:

sendmail: fatal: git(1003): No recipient addresses found in message header
postdrop: warning: stdin: unexpected EOF in data, record type 78 length 86
postdrop: fatal: uid=1003: malformed input

I applied this patch to and

I applied this patch to and installed  turnkey-core-beta-lucid-x86 and it worked fine. Great work!

It applied cleanly to the new

It applied cleanly to the new core-11, but when you try to view gitorious.localdomain you get : https://skitch.com/a-c-m/r8m9j/ruby-on-rails-application-could-not-be-started

Output from tklpatch

(Posted separately to avoid scrunching into the right side of the page!)

I have saved the output from using zsh to execute:

(date; tklpatch turnkey-core-11.1-lucid-x86.iso gitorious.tar.gz; date) |& tee tklpatch-out.txt

You can see it at http://rae.pastebin.com/1AzyJUEK

There are some errors that start around line 2628 and 2634 that might be the problem.

Reid

There are lots of errors in this log

It surprises me it worked at the end for you. For what I can see, the system is unable to start apache as port 80 is in use (1920). Did you run this patch over the previously patched iso? I'll make my test here anyway.

Also, the patch was unable to clone gitorious (2633-2636)

And then, all errors are because of the /var/www/gitorious directory not exists (2637-2668).

Let me run this one locally and I'll come back to you.

Nope, patches were always clean

I always started with a clean ISO, so that wasn't it.

I'm ok for now, since I got it working with the beta release of the base ISO.

Reid

Got it working with the beta..

Thinking it might be a beta-vs-release issue, I tried using turnkey-core-beta-lucid-x86. However, I also had to do the following:

% export RAILS_ENV=production
% rake gems:install ; # installed "hoe"
% rake db:create
% rake db:migrate
% rake ultrasphinx:bootstrap
% crontab -e

* * * * * cd /var/www/gitorious && /usr/bin/rake ultrasphinx:index RAILS_ENV=production

% /etc/init.d/apache2 restart
% ruby script/create_admin

To get script/console working (for deleiting users, etc) you will also need libreadline

% apt-get install libreadline5

After all that, I had a running Gitorious. I'm not sure I needed *all* of it - I was copying it off of a Gitorious setup page at http://gitorious.org/gitorious/pages/UbuntuInstallation ; I suspect I needed to do some of that with the release version. Maybe I'll give it a try tomorrow.

Reid

[updates:
- did a minor edit on order of commands - had a duplicated db:migreate (it failed because I needed to do the gems:install, after which I did db:migrate again)
- added libreadline5 ]

[Much later update:
- if you change the hostname, along with editing config/citorious.yml be sure to edit /etc/hosts if the domain name is not a real one for the machine. I keep hitting this and forgetting! ]

not quite as successful as I had thought

The web front-end seems to work fine, but git operations are failing. Creating a repository never finishes, with the spiner perpetually spinning, and any attempt to access the repository failing as well.

Trying to use the ssh method of access results in being asked for "git"s password.

So.. it looks like more work is needed.

Oh, I did perform the same commands on the release'd base OS as on the beta and I got the same successul results. So it seems that when the OS boots, a number of gitorious initialization steps need to be performed. Perhaps my problems relate to more steps that need to be followed?

Reid

scratch that! A reboot was needed to make it all work

I figured out that the problem with git operations was that ~git/script/poller wasn't running. When I ran it manually like this, everything worked:

% cd ~git
% export RAILS_ENV=production
% script/poller run

There is a script at /etc/init.d/git-poller that runs this as a daemon on startup, so after a reboot, it ran automatically and now everything is running correctly. So yay!

Reid

Tried to build patch a VM today

Hi, I tried applying this patch to a running turnkey-core-11.1-lucid-x86-vmdk, here are the logs:

http://pastebin.com/kGp7WDrL

I have highlighted what seems like the first occurance of errors, though some are obviously benin.

In the end, the /var/www/gitorious/config/gitorious.yml was absent, and hoe refused to install, I needed to specify the version with gem install hoe --version 2.6.0.

It looks that everything works after rebuilding the config, and using Reid Ellis's various posts. (Thanks!)

Nice it worked

This patch is getting old, probably need some updating. But the official appliance is comming soon, is just a matter of waiting. 

nice , can't wait! thx

nice , can't wait! thx turnkey :) 

Can not login in after build

First thank you for this gitorious install, I really would like to get this running as I am sure alot of folks would. Things went well with the patch, but I had to install the gem bundler before I could run the any db stuff could take place. After that a few odd little tweaks and looky looky I have the home page of the gitorious site..... although you can login, comes back with an internal server error.

Any ideas on this or better yet is there a VM applicance already configured ;-)

BTW build with 11.1 turnkey core.

Thanks for any guidance in advance.

Bill

Jeremy's picture

Still no official appliance released yet

So still have to wait for the TKL core devs (Alon & Liraz). As Adrian (the dev of this patch) says just above, this patch is getting quite old now so may require some tweaking to get it to work properly.

Good luck.

Updated Patch for turnkey-core-11.1-lucid-x86.iso

Hello, 

I modified the gitorious patch so that it will build a functioning gitorious server from the turnkey-core-11.1-lucid-x86.iso image.

I modified it to use: 

  • ruby-enterprise_1.8.7-2011.03_1386
  • rubygems-1.4.1 (to install hoe)
  • apache-activemq-5.5.0-bin
  • bundler (to install required gems for gitorious)

I also modified it to use bundle exec to perform the rake commands.

I modified git-poller to use "bundle exec" because a gem thats required was the incorrect version and the poller would just shut down.  This was causing the ssh_keys to never be activated and the repository to never be created.

However, I did notice that my call to "/root/create_admin_turnkey" is out of order and it needs to be run manually.  Tomorrow I will try to fix and test that problem.

I wasn't sure how to post files in the comments, feel free to download the file from http://dl.dropbox.com/u/193989/gitorious-11.1-lucid-x86.tar.gz and give it a shot.

If you have any problems let me know.

Thx for the update!

 

Thx for the update!

Keep us posted pls

Jeremy's picture

Nice work Scott

You can only attach files to the first post of a new thread so just start a new thread and attach your updated patch to it. Then just post a link at the bottom of this thread pointing to your new thread.

Good to know

Thanks Jeremy, 

I will create a new thread when I think I have all the bugs worked out of the patch.

I am currently reading the webmin module development documentation.  I am considering creating a webmin module to include in the patch so that we can look in webmin to see the poller, sphinx, git-daemon and activemq service status. I think it would also be nice to be able to change the hostname, rotate the logs, etc. all from the webmin module.

Jeremy's picture

A Webmin module sounds awesome!

That'd be great if you developed a custom Webmin module. The only thing that springs to mind would be when you develop this perhaps consider rather than making just one module; create 2 modules: One with Gitorious specific settings and another a more generic module - that could potentially be included with Core (and thus all TKL appliances).

New Patch from Scott

Just tried the patch and during patching you get an error, it creates a tmp directory and then it creates a gitorious-lucid file instead of a gitorious-11.1-lucid.tar.gz where can I change this?

 

Thanks

 

Bill

Oops

I renamed the tar file before uploading it, I guess you cant do that 

I have renamed the patch folder. I will bundle, test and push it out to my dropbox shortly.

Update:  I guess my employer blocks the port that apt-get uses.. so I can't test the change until this evening.

In the meantime try:

  • mv gitorious-11.1-lucid-x86.tar.gz gitorious-lucid.tar.gz 
  • tklpatch turnkey-core-11.1-lucid-x86.iso gitorious-lucid.tar.gz

or 

  • tar -xf gitorious-11.1-lucid-x86.tar.gz
  • tklpatch-bundle gitorious-lucid
  • tklpatch turnkey-core-11.1-lucid-x86.iso gitorious-lucid.tar.gz

The directory in the gz file

The directory in the gz file is incorrect for what is being looked for after uncompressed in the /tmp directory

Re tared the gitorious-lucid directory to gitorious-11.1-lucid.x86 and the patching starts to work

create admin error

Scott

Here is the error on create admin

/var/www/gitorious/config/environment.rb:26: Your config/gitorious.yml does not have an entry for your current Rails environment. Please consult config/gitorious.sample.yml for instructions. (Runtim
eError)                                                                                                                                                                                               
        from /var/www/gitorious/config/../vendor/rails/railties/lib/initializer.rb:111:in `run'                                                                                                       
        from /var/www/gitorious/config/environment.rb:24                                                                                                                                              
        from ./create_admin_turnkey:2:in `require'                                                                                                                                                    
        from ./create_admin_turnkey:2

 

Any sugguestions, sorry not too familiar with ruby and rails.

Specify the RAILS_ENV

You should be able to run it by specifying the rails environment

RAILS_ENV=production /root/create_admin_turnkey

Also, I just pushed an updated version to dropbox at the same url.

http://dl.dropbox.com/u/193989/gitorious-11.1-lucid-x86.tar.gz

It should fix the folder structure problem and take care of creating the admin for you as well. (although I haven't had a chance to test it)

I take it out for a spin....

I take it out for a spin.... will let you know what happens

 

Thanks

New Try

Scott,

Just so you know, the dreaded 500 server error upon login. This seems to be the only function gitorious has LOL. Never can get past this.

 

Just letting you know. BTW running inside a VM .

Error details?

Can you give me the details of what its complaining about?  I am not an expert, but I will see what I can figure out.

Well logging in it won't take

Well logging in it won't take you anywhere but the Internal server error. Looking at the running processes the ultrasphinx process is not starting. You also have to uninstall and downgrade two gems.

ultrasphinx is just for searching so I don't think that is a problem, since you can not log in. You apparently have this running I wish I knew the secret.

 

Thanks

Give this one a shot

So there were a couple small problems with that last one.  I have pushed a new version up to my dropbox at the same url:  http://dl.dropbox.com/u/193989/gitorious-11.1-lucid-x86.tar.gz

Just to make sure that I am not doing anything out of the ordinary.. here are the steps I used to patch and build my system.

Here are the steps I use to patch the iso:

  1. Start the turnkey-core-11.1-lucid-x86 VM that I got from here
  2. ssh root@turnkey-core-ip-address
  3. apt-get update
  4. apt-get install tklpatch
  5. mkdir /work
  6. cd /work
  7. wget -O turnkey-core-11.1-lucid-x86.iso http://downloads.sourceforge.net/project/turnkeylinux/turnkey-core/11.1-lucid-x86/turnkey-core-11.1-lucid-x86.iso
  8. wget -O gitorious-11.1-lucid-x86.tar.gz http://dl.dropbox.com/u/193989/gitorious-11.1-lucid-x86.tar.gz
  9. tklpatch turnkey-core-11.1-lucid-x86.iso gitorious-11.1-lucid-x86.tar.gz
  10. exit (exit the ssh session)
  11. scp root@turnkey-core-ip-address:/work/turnkey-core-11.1-lucid-x86-patched.iso ~/Documents/Parallels/turnkey-gitorious-11.1-lucid-x86.iso

 

Then I create a new Parallels Virtual Machine booting from the ISO and follow the installation screens.
 
When it has started and I am presented with the Turnkey Configuration Console
  • I open chrome and browse to gitorious.localdomain
  • I log in with 'admin@example.org' and 'turnkey'
  • After the terms of service screen
    • I click on Adminstration
    • I click on Create New User and add my user
    • I then suspend the Admin login and logout
  • Then I login with my user and
    • I click on ~myuser
    • I click on "go to your dashboard"
    • I click on "Manage SSH Keys"
    • I click on "Add SSH Key"
    • I add my key and then refresh the page until the key becomes active.
    • I click on "Projects"
    • I click on "Create a new project"
    • I add the project and git repository following the steps

All of those steps worked fine for me, however I did notice that the sphinx service was not running.  Its because /var/www/gitorious/db/sphinx/* is owned by root.

To fix this problem:

  • ssh root@gitorious.localdomain
  • chown git:git /var/www/gitorious/db/sphinx/
  • chown git:git /var/www/gitorious/db/sphinx/*
  • /etc/init.d/git-ultrasphinx start

After I verified everything is working then I change my hostname to match my fully qualified domain name as Adrian outlined above.

I figure some people will not want this to be an internet facing appliance.  If that is the case, then I recommend changing the hostname to whatever you want and then setup ZeroConf to enable browsing by name without the need to hardcode the hostfile.

ZeroConf is as easy as apt-get install avahi-daemon.  Then you can access your appliance by browsing to https://hostname.local without the need to modify your hosts file or adding a dns server.

Hope all of this works for you

Scott

*******************  You shouldn't need any of this  *********************

Here are some common problems that I have ran into while messing around with gitorious the last 2 days.

  1. ruby error  undefined method `[]' for false:FalseClass
    • usually caused by empty or missing yml files.
  2. SSH Key never activates or Repository never gets created
    • usually caused by poller not running. (shouldn't happen.. I modified /etc/init.d/git-poller to use bundler to start the service which should ensure the correct gems)
  3. misc other issues... just double check the yml files and the permissions.  /var/www/gitorious/** should all be owned by git:git

Scot Success.....!!!!!

Success .... Nice Job ... now just need to recreate this and face the internet.

Thanks for all your work,responses and help.

Cheers,

Bill

Any chance of an official appliance?

After all this time, all these patches, and the promises of an official appliance .. is there any news on it? Maybe an upload to the SourceForge Turnkey project? If not yet, is there anything we can do to help?

 

P.S. Is there a way to add more tags to this post? Like maybe "git" and/or "gitorious"?

How can I help?

I would be more than happy to push my iso to the Turnkey Community sf project. Who do I contact to give them the iso?

Thanks Scott! Some suggestions

First of all thanks for taking time to upgrade the patch and contribute back. That's what makes opensource great! You seem to know very well about the app, contrary to my initial patch, which was a bit of a try-and-fail patch. I'll point you places where you can contribute with the project:

- Webmin modules would be a great addition, if they are simple to implement. But for this appliance, I think it would be more important to code an inithook script to configure the admin password. You can use the create_admin_turnkey as a base for the implementation, and maybe add the hostname setting. For information on this, you can check the inithooks documentation in the site.

- Feel free to update the dev's wiki page for this patch. Update it to point to the new thread were you'll attach your final version of the patch. Also, you gave a lot of useful documentation on how to setup the appliance once is running. That would make a great addition to the documentation wiki. Add it now to the dev's page until the official appliance is out. With this update, surely the core team will have less to worry when implementing the official appliance. BTW, they are currently working hard to make the project a bit more open and accept more outside collaboration, so the lucid releases second batch will probably take some time.

- When you feel your patch is complete, you can contact Jed or myself, and we'll happily create the iso and upload it to the community appliances. I'll suggest Jed right now as my laptop died and I've been a bit with my hands tied up, but I'll hope to get a new one by the end of july. 

Again, thanks for your contribution, it's nice to see someone jump in and give a hand. If you have further questions the general forum is the right place to ask.

Newbie question

Now that I have this running and such, I am having trouble getting a local git repository into the gitorious repository... I should know this but....if it was svn I would be in but its not and I need help.

Thanks for any info or point me in the right direction, trying to make this KISS princple like LOL for us git virgins.

 

Thanks again Scott for the working instructions and patch.

Reply to myself

Figured it out ... I am lil dense sometimes... Thanks.. btw had the wrong ssh key installed for my laptop that is why the errors not pushing repository.

Any news on the turnkey gitorious offical release?

Any news on the turnkey gitorious offical release?

Jeremy's picture

TKL devs 'release when done'

So no word on when, although considering part 2 of v11.x release is well overdue hopefully we'll see it sometime soonish.

This will be epic when it is done

I have spent weeks of my like trying to have gitorious installed. I was depressed and defeated, but this gives me hope.

Keep up the good work!

I am buildding another Gitorious server using the latest patch..

I'm trying to create another Gitorious server (the first is running nicely under VirtualBox on an iMac).

First try didn't go well. I'm using Scott Beardsley's latest patch at http://dl.dropbox.com/u/193989/gitorious-11.1-lucid-x86.tar.gz. There seems to be no /var/www/gitorious directory -- I think the "git clone" failed.

Is there any means for these scripts to handle errors like this gracefully? i.e. tell the user "Can't connect to git server, patch failed"?

"create_admin_turnkey" doesn't work

The "create_admin_turnkey" script in root's home directory doesn't work any more. Gitorious now stores encrypted passwords with salt. So now I have to figure out how to make the password encrypted so that the admin user can be created.

Migrating an old repo to a new machine

Took me a while to figure out I needed to run this to upgrade my gitorious MySQL database.

bundle exec rake db:migrate:up VERSION=20110223130355

That's specific to the version of gitorious I was at (20110223130355). egad.

How to migrate your gitorious server to a new instance

I've finished a successful test run of moving our Gitorious instance to a new install of Gitorious (on machine "newgitorious") using this patch. Here are the key steps I used:

As root on the old VM/machine:

mysqldump --all-databases --add-drop-database --add-drop-table --add-locks | gzip > dev-sql-dump.gz
scp dev-sql-dump.gz root@newgitorious:
rsync -az -e ssh --delete /var/git/ newgitorious:/var/git

On the newgitorious machine:

As root:

apt-get install avahi-daemon
zcat dev-sql-dump.gz | mysql

avahi-daemon allowed me to use ZeroConf so I could use the hostname newgitorious.local without any mucking about with /etc/hosts or similar. I think this should be included with all TKL distributions. Using "applianceName.localhost" is sub-optimal to say the least.

As user "git":

bundle exec rake db:migrate:up VERSION=20110223130355

As mentioned before, this is related to the specific version of Gitorious I was running on the old machine. I really wish the Gitorious dev team would update their real revision number more often. By now it should really be at version 2.7 or something.

Hope this all helps someone else.

Upgrade

Reid,

Kinda confused, where did you find the version number for your old gitorious?

Also, when you moved you database from your old version to the version 2.0 that is out there now what exactly where your steps and you also moved the repositories I assume. I am trying to update so I can be at this new versioning level from gitorious.org to make the updating easier I  hope.

Thanks

Bill

Version numbers

Hi Bill,

The version number was the least obvious part of all this. It involves the directory db/migrate. On the old machine, the last file in there was 20100316124803_create_archived_events.rb. Looking on the new machine, the very next file was 20110223130355_add_suspended_at_to_projects.rb - I ended up using the "20110223130355" string as my argument to "bundle exec rake db:migrate:up" since otherwise, Rails didn't seem able to migrate. I would have expected to simply do a "bundle exec rake db:migrate" and have everything taken care of, but alas, such was not the case.

I found the "bundle exec rake db:migrate:up" command through a series of google searches frankly.

Hope this helps,

Reid

Versioning

Hey Ried,

Thanks for the info.

Curious how you made the move of repositories (copy the data I am sure) but the update to the MySql, its not just a restore or is it really that simple.

Bill

mysql dump

The rsync command took care of copying over the git repositories, and the only other thing I used was mysqldump. I added flags so that I could do it over and over until I got it right though.

One thing that may not be finished are the ssh keys. I think there is a rails command needed to push them into ~git/.ssh ...

Reid

5 months later and no Gitorious appliance

Looks like TKL isn't interested in appliances any more. Maybe we need a tkl-users.org or something to host user-generated appliances?

Jed created a site to host community appliances

I wouldn't say TKL don't have an interest in appliances. What I would say is that they became overwhelmed with the work. I asked the main team to be more open for the project's health, after all, it's an opensource project. What I understood is that the main infraestructure for creating appliances isn't community friendly, and they decided to make an effort to enhance the whole thing so they can be more open to contributions in the future. 

The fact is that as the second batch of appliances (a lot of them almost a year old since they got published), there is no info on timeline/features being developed. I'm sure they are working, and I expect to see the results by the Precise Pangolin release. Meanwhile, we can upload them to the community site.  

I also suggested a site dedicated to tklpatches with a voting system, as I understand it's going to be impossible to review/officialize all patches into appliances, but at least as you suggest, give a space to community to post appliances and be reviewed by other community members. I'm sure we can push as community to see some of these things done, or start something ourselves and later we could get the official support.

Alon Swartz's picture

Definitely interested in expanding TurnKey...

We are definitely interested in expanding the TurnKey library, but as Adrian mentioned we decided to focus our efforts on moving our build infrastructure to the cloud, which will make it much easier to open up the development process to others and ease our current bottle neck.

We also have plans/ideas of creating a more "developers" orientated section for developing and sharing appliances (and later implement reviews/voting/etc).

BTW, I just came across GitLab (open source clone of GitHub). Would be interesting to see how it compares to Gitorious.

Great idea

both on the GitLab appliance, more more importantly on the idea of 'Community TKLs'.  

Gitlab?

After all that work, you went with Gitlab instead of Gitorious? Can you tell us why? Does Gitlab have more/better features? On the surface, it seems to use the same components as Gitorious..

Liraz Siri's picture

We followed Adrian Moya's recommendation

Adrian Moya, that nice chap who developed the Gitorious TKLPatch recommended we do Gitlab after using both. He said he found Gitlab more powerful. I think it's based on Gitorious BTW...

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)