TKLPatch for Gitorious

Hi Rtreffer, thanks for trying the patch. Could you please provide more info on the patching process? yo can do it by issuing the following command:

tklpatch turnkey-core-beta-lucid-x86.iso gitorious.tar.gz | tee tklgit.log

And attach tklgit.log file to see what's happening during the patching process. 

So I was off track but I appreciate you taking the time to explain where :)

I haven't actually tested your patch, just sticking my newb nose in and getting a bit down and dirty with stuff way over my head! I had a scan through Guest's error code link (above) and found that bit (there may be more?) I did notice that there were references to sub dirs of /var/www/gitorious after the errors but thought perhaps the MySQL failure was somehow related to the path error (not sure how though).

The appliance looks really cool and I will defiantely have a go at it soon hopefully. I think it could be a really cool tool for projects I work on, both at home and at work. Oh so many things I want to do and so little time!

I reckon you'd be going crazy with it! I can handle a bit of basic code/commandline but serious stuff like that does my head in!

Looking at the output you posted, the errors start to show from the moment of migrating Gitorious database 

(Couldn't create database for {"encoding"=>"utf8", "username"=>"root", "adapter"=>"mysql", "database"=>"gitorious_production", "host"=>"localhost", "password"=>nil}, charset: utf8, collation: utf8_unicode_ci (if you set the charset manually, make sure you have a matching collation)
(in /var/www/gitorious)
rake aborted!
Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

I'll run again this patch in my dev environment. Mysql server seems to have started correctly, as seen in line:

1514: mysql start/running, process 21744

The user home is not created at the moment of the git user creation, and it also doesn't exists at that moment, but after that, the gitorious repository is being cloned at /var/www/gitorious creating that home directory for the user. So that's not the problem, I could create the user after cloning Gitorious and the warning would disappear. I'll update it. Let me check and I'll come back to you in a moment. 

Just for curiosity: did you upgrade tklpatch to its lastest version? Are you patching from within Turnkey Core Lucid or in another environment? 

This is indeed a very complex patch. So many things I was going crazy with each test. Thanks for trying it. 

Here's my log when running the patch: http://pastebin.com/vBbtZkqY

Take a look at this differences:

Yours (starting mysql-server) lines  1514-1515

mysql start/running, process 21744

Mine (starting mysql-server) lines 798-801 (fewer lines because I only posted tklpatch-apply output)

* Starting MySQL database server mysqld       [157G
[151G[ OK ]
 * Checking for corrupt, not cleanly closed and upgrade needing tables.

(in /var/www/gitorious)
(in /var/www/gitorious)
==  CreateUsers: migrating ====================================================
-- create_table("users", {:force=>true})
   -> 0.0365s
-- add_index(:users, :login)
   -> 0.0183s
   .... blah blah blah

Admin user created successfully.

    dpkg -l tklpatch
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                              Version                           Description
+++-=================================-=================================-==================================================================================
ii  tklpatch                          0.93+5+ga2bd433                   TurnKey Linux Customization Mechanism

Finally had a chance to sit down and investigate this patch a bit more thoroughly.

First off, excellent work. This is a complex patch and it looks like a lot of work went into getting everything right. The only things that caught my attention were a few violations of the FHS (Filesystem Hierarchy Standard). This is the standard Debian-based systems are advised to follow with respect to what belongs where (e.g., data, configuration files, local binaries, stuff controlled by package management, etc.).

For example:

• Re-linking /usr/bin/ruby to a path in /usr/local. Ideally you would do that as a last resort if you can't achieve the same effect by manipulating PATH in the environment.
• Putting various helper scripts in /root. I'd put them into /tmp if you want to delete them later (I.e., they're temporary), or if it's useful to keep them around /usr/local/bin.

In case you're wondering if this technical bureaucracy serves any other point besides slowing down system integrators, yes it actually does make sense when you take package management into consideration. If you replace /usr/bin/ruby for example, the next package update for ruby will overwrite your change and break the system. Also, if you're making an efficient backup of the system (e.g., TKLBAM) you can save a lot of space by just backing up those directories which contain your local customizations / data and recreating the rest of the system by syncing package management states.

First be sure that your host fqdn is properly set. Issue the following command in a terminal: hostname -f (or cat /etc/hosts) correct the appropiate entries if needed. (check step 1 if you need help)

Second, change "ServerName gitorious.localdomain" in /etc/apache2/httpd.conf to myurl.com. You can do this with

sed -i "s/ServerName gitorious.localdomain/ServerName myurl.com/" /etc/apache2/httpd.conf

Third, you have to change the following settings in /var/www/gitorious/config/gitorious.yml:

gitorious_client_host: gitorious.localdomain
gitorious_host: gitorious.localdomain

You can do both changes with:

sed -i "s/gitorious.localdomain/myurl.com/" /var/www/gitorious/config/gitorious.yml

Be sure to backup all files in case anything goes wrong. Please tell me if this works for you. If you loose repos is normal, as gitorious is able to server in multiple domains, and when you make a change, it won't reconize the other domain projects/repos. At least this is what I've read. Thanks for testing the appliance, your feedback is welcome. 

P.S. Restart apache once all changes are made:

service apache2 restart

Be warned that public mode will let anyone open an account and create repos. Be sure that's what you want. Private mode means an administrator create accounts/repos. I'll think I'll add a helper script for this

I've got no idea why it's doing that (well out of my league). Not sure whether Adrian has plans to update his patch or not though seeing as the TKL guys have started working on Part 2 of the v11 release (which will include Gitorious). I'm guessing from his comments above that he may not be updating it, if that's the case then you'll either need to try debugging it yourself, or wait for the official release.

Please do the following: when applying the patch, add an extra > gitorious.log and post it to check for errors during patch.

tklpatch turnkey-core-11.1-lucid-x86.iso gitorious.tar.gz > gitorious.log

Also: you can try what the error is suggesting, that is, go to /var/www/gitorious and run 

rake gems:install

This patch was done a while ago, and I haven't tested with the stable releases of turnkey-core. I'm also not updating it as the core tkl-devs will release this appliance soon as part2 of turnkey-lucid. But I have no problem giving you some support in the meantime. 

I uploaded a screenshot to Flickr:

http://www.flickr.com/photos/clith/5388084035/sizes/o/

Looks like I got the exact same result as a_c_m back on Jan 2 <https://img.skitch.com/20110102-duj7ifmku3uafbcph3sde37qqd.jpg>. No patch update over the last 4 weeks?

Reid

I am running "tklpatch turnkey-core-11.1-lucid-x86.iso gitorious.tar.gz" even as I type...

I'll report here if there are any problems.

Reid

But soon is relative. Our beloved TKL devs do not like to give clear timeframes as they don't like to dissapoint. Whilst you could try it out (by patching TKL Core) it would be better to wait if you plan to put it into production.

Hi there,

I was going to try applying your patches on the released turnkey-core-11.1-lucid but it's been 9 days since your "be patient" post, so I'm thinking I should just wait for the Gitorious appliance to show up.

Is there a beta release we can help test?

Reid

Be patient, Turnkey-lucid already have been published and next week a version with some fixes will be available. The second part of the releases will include Gitorious. I'm also waiting for some of my other patches to get to the second batch of releases, I know the tkl guys are working hard to make this happen. 

Meanwhile, have you tryed this version? There's still time for some feedback before the final appliance goes live. 

I've been following this thread in hopes of the full appliance appearing.  Just wondering if there are any updates on this?  I also, would really LOVE to see this as a full appliance.  GREAT WORK ADRIAN!

It is very possible that this patch become an official appliance, for the new release of Turnkey Linux based on Ubuntu Lucid. But there is no official date when this is going to happen, the Core Devs are in the middle of producing a second beta of the Core, then they'll update the existing appliances, and finally convert some of the patches to official appliances. I'm hoping too this one gets in. 

I encourage you to make a donation if it is in your possibilities, but cannot guarantee that would speed things up, as the team is small and the work to be done is immense. But it surely helps the project to be maintained and this kind of appliances to be finished and available to you and all the community. 

Keep in touch for the evolution of this patch and let's hope it's ready before the year ends. 

I always started with a clean ISO, so that wasn't it.

I'm ok for now, since I got it working with the beta release of the base ISO.

Reid

It surprises me it worked at the end for you. For what I can see, the system is unable to start apache as port 80 is in use (1920). Did you run this patch over the previously patched iso? I'll make my test here anyway.

Also, the patch was unable to clone gitorious (2633-2636)

And then, all errors are because of the /var/www/gitorious directory not exists (2637-2668).

Let me run this one locally and I'll come back to you.

(Posted separately to avoid scrunching into the right side of the page!)

I have saved the output from using zsh to execute:

(date; tklpatch turnkey-core-11.1-lucid-x86.iso gitorious.tar.gz; date) |& tee tklpatch-out.txt

You can see it at http://rae.pastebin.com/1AzyJUEK

There are some errors that start around line 2628 and 2634 that might be the problem.

Reid

Thinking it might be a beta-vs-release issue, I tried using turnkey-core-beta-lucid-x86. However, I also had to do the following:

% export RAILS_ENV=production
% rake gems:install ; # installed "hoe"
% rake db:create
% rake db:migrate
% rake ultrasphinx:bootstrap
% crontab -e

* * * * * cd /var/www/gitorious && /usr/bin/rake ultrasphinx:index RAILS_ENV=production

% /etc/init.d/apache2 restart
% ruby script/create_admin

To get script/console working (for deleiting users, etc) you will also need libreadline

% apt-get install libreadline5

After all that, I had a running Gitorious. I'm not sure I needed *all* of it - I was copying it off of a Gitorious setup page at http://gitorious.org/gitorious/pages/UbuntuInstallation ; I suspect I needed to do some of that with the release version. Maybe I'll give it a try tomorrow.

Reid

[updates:
- did a minor edit on order of commands - had a duplicated db:migreate (it failed because I needed to do the gems:install, after which I did db:migrate again)
- added libreadline5 ]

[Much later update:
- if you change the hostname, along with editing config/citorious.yml be sure to edit /etc/hosts if the domain name is not a real one for the machine. I keep hitting this and forgetting! ]

I figured out that the problem with git operations was that ~git/script/poller wasn't running. When I ran it manually like this, everything worked:

% cd ~git
% export RAILS_ENV=production
% script/poller run

There is a script at /etc/init.d/git-poller that runs this as a daemon on startup, so after a reboot, it ran automatically and now everything is running correctly. So yay!

Reid

The web front-end seems to work fine, but git operations are failing. Creating a repository never finishes, with the spiner perpetually spinning, and any attempt to access the repository failing as well.

Trying to use the ssh method of access results in being asked for "git"s password.

So.. it looks like more work is needed.

Oh, I did perform the same commands on the release'd base OS as on the beta and I got the same successul results. So it seems that when the OS boots, a number of gitorious initialization steps need to be performed. Perhaps my problems relate to more steps that need to be followed?

Reid

This patch is getting old, probably need some updating. But the official appliance is comming soon, is just a matter of waiting. 

So still have to wait for the TKL core devs (Alon & Liraz). As Adrian (the dev of this patch) says just above, this patch is getting quite old now so may require some tweaking to get it to work properly.

Good luck.

That'd be great if you developed a custom Webmin module. The only thing that springs to mind would be when you develop this perhaps consider rather than making just one module; create 2 modules: One with Gitorious specific settings and another a more generic module - that could potentially be included with Core (and thus all TKL appliances).

Thanks Jeremy, 

I will create a new thread when I think I have all the bugs worked out of the patch.

I am currently reading the webmin module development documentation.  I am considering creating a webmin module to include in the patch so that we can look in webmin to see the poller, sphinx, git-daemon and activemq service status. I think it would also be nice to be able to change the hostname, rotate the logs, etc. all from the webmin module.

You can only attach files to the first post of a new thread so just start a new thread and attach your updated patch to it. Then just post a link at the bottom of this thread pointing to your new thread.

Hello, 

I modified the gitorious patch so that it will build a functioning gitorious server from the turnkey-core-11.1-lucid-x86.iso image.

I modified it to use: 

• ruby-enterprise_1.8.7-2011.03_1386
• rubygems-1.4.1 (to install hoe)
• apache-activemq-5.5.0-bin
• bundler (to install required gems for gitorious)

I also modified it to use bundle exec to perform the rake commands.

I modified git-poller to use "bundle exec" because a gem thats required was the incorrect version and the poller would just shut down.  This was causing the ssh_keys to never be activated and the repository to never be created.

However, I did notice that my call to "/root/create_admin_turnkey" is out of order and it needs to be run manually.  Tomorrow I will try to fix and test that problem.

I wasn't sure how to post files in the comments, feel free to download the file from http://dl.dropbox.com/u/193989/gitorious-11.1-lucid-x86.tar.gz and give it a shot.

If you have any problems let me know.

I renamed the tar file before uploading it, I guess you cant do that 🙂

I have renamed the patch folder. I will bundle, test and push it out to my dropbox shortly.

Update:  I guess my employer blocks the port that apt-get uses.. so I can't test the change until this evening.

In the meantime try:

• mv gitorious-11.1-lucid-x86.tar.gz gitorious-lucid.tar.gz 
• tklpatch turnkey-core-11.1-lucid-x86.iso gitorious-lucid.tar.gz

or 

• tar -xf gitorious-11.1-lucid-x86.tar.gz
• tklpatch-bundle gitorious-lucid
• tklpatch turnkey-core-11.1-lucid-x86.iso gitorious-lucid.tar.gz

Can you give me the details of what its complaining about?  I am not an expert, but I will see what I can figure out.

So there were a couple small problems with that last one.  I have pushed a new version up to my dropbox at the same url:  http://dl.dropbox.com/u/193989/gitorious-11.1-lucid-x86.tar.gz

Just to make sure that I am not doing anything out of the ordinary.. here are the steps I used to patch and build my system.

Here are the steps I use to patch the iso:

1. Start the turnkey-core-11.1-lucid-x86 VM that I got from here
2. ssh root@turnkey-core-ip-address
3. apt-get update
4. apt-get install tklpatch
5. mkdir /work
6. cd /work
7. wget -O turnkey-core-11.1-lucid-x86.iso http://downloads.sourceforge.net/project/turnkeylinux/turnkey-core/11.1-...
8. wget -O gitorious-11.1-lucid-x86.tar.gz http://dl.dropbox.com/u/193989/gitorious-11.1-lucid-x86.tar.gz
9. tklpatch turnkey-core-11.1-lucid-x86.iso gitorious-11.1-lucid-x86.tar.gz
10. exit (exit the ssh session)
11. scp root@turnkey-core-ip-address:/work/turnkey-core-11.1-lucid-x86-patched.iso ~/Documents/Parallels/turnkey-gitorious-11.1-lucid-x86.iso

• I open chrome and browse to gitorious.localdomain
• I log in with 'admin@example.org' and 'turnkey'
• After the terms of service screen
  • I click on Adminstration
  • I click on Create New User and add my user
  • I then suspend the Admin login and logout
• Then I login with my user and
  • I click on ~myuser
  • I click on "go to your dashboard"
  • I click on "Manage SSH Keys"
  • I click on "Add SSH Key"
  • I add my key and then refresh the page until the key becomes active.
  • I click on "Projects"
  • I click on "Create a new project"
  • I add the project and git repository following the steps

All of those steps worked fine for me, however I did notice that the sphinx service was not running.  Its because /var/www/gitorious/db/sphinx/* is owned by root.

To fix this problem:

• ssh root@gitorious.localdomain
• chown git:git /var/www/gitorious/db/sphinx/
• chown git:git /var/www/gitorious/db/sphinx/*
• /etc/init.d/git-ultrasphinx start

After I verified everything is working then I change my hostname to match my fully qualified domain name as Adrian outlined above.

I figure some people will not want this to be an internet facing appliance.  If that is the case, then I recommend changing the hostname to whatever you want and then setup ZeroConf to enable browsing by name without the need to hardcode the hostfile.

ZeroConf is as easy as apt-get install avahi-daemon.  Then you can access your appliance by browsing to https://hostname.local without the need to modify your hosts file or adding a dns server.

Hope all of this works for you

Scott

*******************  You shouldn't need any of this  *********************

Here are some common problems that I have ran into while messing around with gitorious the last 2 days.

1. ruby error  undefined method `[]' for false:FalseClass
   • usually caused by empty or missing yml files.
2. SSH Key never activates or Repository never gets created
   • usually caused by poller not running. (shouldn't happen.. I modified /etc/init.d/git-poller to use bundler to start the service which should ensure the correct gems)
3. misc other issues... just double check the yml files and the permissions.  /var/www/gitorious/** should all be owned by git:git

You should be able to run it by specifying the rails environment

RAILS_ENV=production /root/create_admin_turnkey

Also, I just pushed an updated version to dropbox at the same url.

http://dl.dropbox.com/u/193989/gitorious-11.1-lucid-x86.tar.gz

It should fix the folder structure problem and take care of creating the admin for you as well. (although I haven't had a chance to test it)

First of all thanks for taking time to upgrade the patch and contribute back. That's what makes opensource great! You seem to know very well about the app, contrary to my initial patch, which was a bit of a try-and-fail patch. I'll point you places where you can contribute with the project:

- Webmin modules would be a great addition, if they are simple to implement. But for this appliance, I think it would be more important to code an inithook script to configure the admin password. You can use the create_admin_turnkey as a base for the implementation, and maybe add the hostname setting. For information on this, you can check the inithooks documentation in the site.

- Feel free to update the dev's wiki page for this patch. Update it to point to the new thread were you'll attach your final version of the patch. Also, you gave a lot of useful documentation on how to setup the appliance once is running. That would make a great addition to the documentation wiki. Add it now to the dev's page until the official appliance is out. With this update, surely the core team will have less to worry when implementing the official appliance. BTW, they are currently working hard to make the project a bit more open and accept more outside collaboration, so the lucid releases second batch will probably take some time.

- When you feel your patch is complete, you can contact Jed or myself, and we'll happily create the iso and upload it to the community appliances. I'll suggest Jed right now as my laptop died and I've been a bit with my hands tied up, but I'll hope to get a new one by the end of july. 

Again, thanks for your contribution, it's nice to see someone jump in and give a hand. If you have further questions the general forum is the right place to ask.

I would be more than happy to push my iso to the Turnkey Community sf project. Who do I contact to give them the iso?

After all this time, all these patches, and the promises of an official appliance .. is there any news on it? Maybe an upload to the SourceForge Turnkey project? If not yet, is there anything we can do to help?

P.S. Is there a way to add more tags to this post? Like maybe "git" and/or "gitorious"?

So no word on when, although considering part 2 of v11.x release is well overdue hopefully we'll see it sometime soonish.

The "create_admin_turnkey" script in root's home directory doesn't work any more. Gitorious now stores encrypted passwords with salt. So now I have to figure out how to make the password encrypted so that the admin user can be created.

Took me a while to figure out I needed to run this to upgrade my gitorious MySQL database.

bundle exec rake db:migrate:up VERSION=20110223130355

That's specific to the version of gitorious I was at (20110223130355). egad.

I've finished a successful test run of moving our Gitorious instance to a new install of Gitorious (on machine "newgitorious") using this patch. Here are the key steps I used:

As root on the old VM/machine:

mysqldump --all-databases --add-drop-database --add-drop-table --add-locks | gzip > dev-sql-dump.gz
scp dev-sql-dump.gz root@newgitorious:
rsync -az -e ssh --delete /var/git/ newgitorious:/var/git

On the newgitorious machine:

As root:

apt-get install avahi-daemon
zcat dev-sql-dump.gz | mysql

avahi-daemon allowed me to use ZeroConf so I could use the hostname newgitorious.local without any mucking about with /etc/hosts or similar. I think this should be included with all TKL distributions. Using "applianceName.localhost" is sub-optimal to say the least.

As user "git":

bundle exec rake db:migrate:up VERSION=20110223130355

As mentioned before, this is related to the specific version of Gitorious I was running on the old machine. I really wish the Gitorious dev team would update their real revision number more often. By now it should really be at version 2.7 or something.

Hope this all helps someone else.

I'm trying to create another Gitorious server (the first is running nicely under VirtualBox on an iMac).

First try didn't go well. I'm using Scott Beardsley's latest patch at http://dl.dropbox.com/u/193989/gitorious-11.1-lucid-x86.tar.gz. There seems to be no /var/www/gitorious directory -- I think the "git clone" failed.

Is there any means for these scripts to handle errors like this gracefully? i.e. tell the user "Can't connect to git server, patch failed"?

Hi Bill,

The version number was the least obvious part of all this. It involves the directory db/migrate. On the old machine, the last file in there was 20100316124803_create_archived_events.rb. Looking on the new machine, the very next file was 20110223130355_add_suspended_at_to_projects.rb - I ended up using the "20110223130355" string as my argument to "bundle exec rake db:migrate:up" since otherwise, Rails didn't seem able to migrate. I would have expected to simply do a "bundle exec rake db:migrate" and have everything taken care of, but alas, such was not the case.

I found the "bundle exec rake db:migrate:up" command through a series of google searches frankly.

Hope this helps,

Reid

The rsync command took care of copying over the git repositories, and the only other thing I used was mysqldump. I added flags so that I could do it over and over until I got it right though.

One thing that may not be finished are the ssh keys. I think there is a rails command needed to push them into ~git/.ssh ...

Reid

both on the GitLab appliance, more more importantly on the idea of 'Community TKLs'.  

We are definitely interested in expanding the TurnKey library, but as Adrian mentioned we decided to focus our efforts on moving our build infrastructure to the cloud, which will make it much easier to open up the development process to others and ease our current bottle neck.

We also have plans/ideas of creating a more "developers" orientated section for developing and sharing appliances (and later implement reviews/voting/etc).

BTW, I just came across GitLab (open source clone of GitHub). Would be interesting to see how it compares to Gitorious.

I wouldn't say TKL don't have an interest in appliances. What I would say is that they became overwhelmed with the work. I asked the main team to be more open for the project's health, after all, it's an opensource project. What I understood is that the main infraestructure for creating appliances isn't community friendly, and they decided to make an effort to enhance the whole thing so they can be more open to contributions in the future. 

The fact is that as the second batch of appliances (a lot of them almost a year old since they got published), there is no info on timeline/features being developed. I'm sure they are working, and I expect to see the results by the Precise Pangolin release. Meanwhile, we can upload them to the community site.  

I also suggested a site dedicated to tklpatches with a voting system, as I understand it's going to be impossible to review/officialize all patches into appliances, but at least as you suggest, give a space to community to post appliances and be reviewed by other community members. I'm sure we can push as community to see some of these things done, or start something ourselves and later we could get the official support.

Adrian Moya, that nice chap who developed the Gitorious TKLPatch recommended we do Gitlab after using both. He said he found Gitlab more powerful. I think it's based on Gitorious BTW...