Alex's picture

I installed the bugzilla appliance, and it worked great.

Now I am working on the Canvas appliance, and things are not going as well.

I have tried version 13 both 32 bit and 64 bit, using both VMWare Player as well as VirtualBox.

 

Each time I point a browser to the site, it redirects me to the logon page, but nothing is displayed except for the "Oops something broke." error message.  Looking into the canvas production error log, I see the following in all environments:

 

Status: 500 Internal Server Error
  ERR unknown command 'evalsha'
    /var/www/canvas/vendor/bundle/ruby/1.9.1/gems/redis-3.0.1/lib/redis/client.rb:79:in `call'
    /var/www/canvas/vendor/bundle/ruby/1.9.1/gems/redis-3.0.1/lib/redis.rb:2122:in `block in _eval'
    /var/www/canvas/vendor/bundle/ruby/1.9.1/gems/redis-3.0.1/lib/redis.rb:36:in `block in synchronize'
    /usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
    /var/www/canvas/vendor/bundle/ruby/1.9.1/gems/redis-3.0.1/lib/redis.rb:36:in `synchronize'
    /var/www/canvas/vendor/bundle/ruby/1.9.1/gems/redis-3.0.1/lib/redis.rb:2121:in `_eval'
    /var/www/canvas/vendor/bundle/ruby/1.9.1/gems/redis-3.0.1/lib/redis.rb:2173:in `evalsha'
    /var/www/canvas/vendor/bundle/ruby/1.9.1/gems/redis-scripting-1.0.1/lib/redis/scripting/script.rb:20:in `run'
    /var/www/canvas/vendor/bundle/ruby/1.9.1/gems/redis-scripting-1.0.1/lib/redis/scripting/module.rb:22:in `run'
    /var/www/canvas/lib/canvas/request_throttle.rb:246:in `increment'
    /var/www/canvas/lib/canvas/request_throttle.rb:224:in `ensure in reserve_capacity'
    /var/www/canvas/lib/canvas/request_throttle.rb:224:in `reserve_capacity'
    /var/www/canvas/lib/canvas/request_throttle.rb:54:in `call'
    /var/www/canvas/app/middleware/stats_timing.rb:8:in `block in call'
    /var/www/canvas/vendor/bundle/ruby/1.9.1/bundler/gems/rails-e86daf8ff727/activesupport/lib/active_support/core_ext/benchmark.rb:17:in `block in ms'
    /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
    /var/www/canvas/vendor/bundle/ruby/1.9.1/bundler/gems/rails-e86daf8ff727/activesupport/lib/active_support/core_ext/benchmark.rb:17:in `ms'
    /var/www/canvas/app/middleware/stats_timing.rb:8:in `call'
    /var/www/canvas/app/middleware/load_account.rb:12:in `call'
    /var/www/canvas/app/middleware/sessions_timeout.rb:24:in `call'
    /var/www/canvas/vendor/bundle/ruby/1.9.1/gems/encrypted_cookie_store-instructure-1.0.4/lib/encrypted_cookie_store.rb:33:in `call'
    /var/www/canvas/vendor/bundle/ruby/1.9.1/bundler/gems/rails-e86daf8ff727/actionpack/lib/action_controller/failsafe.rb:26:in `call'
    /var/www/canvas/vendor/bundle/ruby/1.9.1/gems/rack-1.1.3/lib/rack/lock.rb:11:in `block in call'
    <internal:prelude>:10:in `synchronize'
    /var/www/canvas/vendor/bundle/ruby/1.9.1/gems/rack-1.1.3/lib/rack/lock.rb:11:in `call'
    /var/www/canvas/vendor/bundle/ruby/1.9.1/bundler/gems/rails-e86daf8ff727/actionpack/lib/action_controller/dispatcher.rb:106:in `call'
    /var/lib/gems/1.9.1/gems/passenger-4.0.20/lib/phusion_passenger/rack/thread_handler_extension.rb:77:in `process_request'

 

Any help or pointers would be greatly appreciated!

Forum: 
Tomaž's picture

Hello!

Been trying to solve this one for a few days already. Anyway, solution is quick and easy:

 

1. apt-get remove redis-server

2. Put this in /etc/apt/preferences:

Package: *
Pin: release o=packages.dotdeb.org
Pin-Priority: 100

Package: redis-server
Pin: release o=packages.dotdeb.org
Pin-Priority: 900

3. Create file with this content in /etc/apt/sources.list.d/:

deb http://packages.dotdeb.org wheezy all
deb-src http://packages.dotdeb.org wheezy all

And run this command to import their key:

wget http://www.dotdeb.org/dotdeb.gpg
apt-key add dotdeb.gpg

4. Then you can simply:

apt-get install redis-server

 

Yiou can test it with /etc/init.d/redis-server restart.

 

Now things are working!

 

 

 

Alex's picture

Followed your instructions, all commands appeared successful.

redis-server -v shows:

version 2.4.14 (00000000:0)

Still getting the same error.  Thanks for the try though.

Jeremy Davis's picture

Thanks very much for the workaround! I have confirmed that it works! You rock!

And not only that, I went to lodge a bug and it was done already! :)

Alex's picture

Thank you Tomaž!  Amazing products, love what you guys are doing.

Jeremy Davis's picture

Have any of you Canvas using people encountered any of the bugs listed here? From what I can gather the v13.0 TKL appliance uses the same version of Canvas as v12.1 but I am not sure... Be great if someone could confirm whether or not these bugs are still relevant.

Thanks! :)

Alex's picture

Looking through, I am not able to reproduce any of the bugs in the v13 applicance.

Jeremy Davis's picture

Thanks for the feedback Alex. I will close the bug on the issue tracker but if you do find any issues, don't hesitate to post them and hopefully we can at least try to find a work around...

Cheers.

Jeremy Davis's picture

You'll need at least 1GB RAM available, it also takes quite a while for it to start up initially.

Regardless, I'm not 100% sure what the issue you are experiencing is. I just launched a fresh v13.0 Canvas appliance and the initial error is quite different to yours. I get a page that says (something like) "Oops something has gone wrong". When I enter these 3 lines, then it works straight away...

Jeremy Davis's picture

Or only after you update redis-server?

When I tried to reproduce your issue, I can't.

As I posted above, when I first start the appliance I get a message like "Oops, something has gone wrong", then after adding Debian wheezy-backports repo and updating redis-server it works fine.

I am using an OpenVZ template (rather than the VM build you are using) but I doubt that is the issue (both of them are built from the same source - ISO). I have 1GB RAM allocated + 1GB swap and it fluctuates between 700MB-950MB RAM usage and has about 200MB-300MB in swap. It runs quite sluggishly but otherwise ok (it takes a few minutes to get started, both initially and after updating Redis-Server)...

I suggest that you start again from the start. Firstly I suggest that you verify the image isn't corrupt (don't just download it again - I have had occasional issue with a corrupt SourceForge mirror, just downloading it again won't confirm that the file isn't corrupt!)

Double check that you are getting the "Oops..." message in your web browser when you first launch it. Then from the commandline run these three lines:

echo -e '# Backports repository\ndeb http://ftp.debian.org/debian wheezy-backports main' >> /etc/apt/sources.list.d/wheezy-backports.list
apt-get update
apt-get install -t wheezy-backports redis-server

Let me know how that goes...

Jeremy Davis's picture

Hopefully it might help some others... :)

BTW we intend to resolve this issue with the 13.1 maintenance release because really TurnKey products should just work OOTB without users needing to fiddle around with stuff like this!

Peter's picture

I'm trying to set up tkl Canvas 13.0 image on

- VirtualBox (v.4.3.2) on a Win7 host o/s

- 1.5Gb RAM allocated. can't find a page with minimum requirements, or md5 checksum.

I've uninstalled and re-installed redis (v2.8.9) as instructed above.

Any advice would be appreciated.

Error log:

/var/www/canvas/vendor/bundle/ruby/1.9.1/bundler/gems/rails-e86daf8ff727/activesupport/lib/active_support/inflector.rb:3:in `<top (required)>': iconv will be deprecated in the future, use String#encode instead.
NOTE: Gem.source_index is deprecated, use Specification. It will be removed on or after 2011-11-01.
Gem.source_index called from /var/www/canvas/vendor/bundle/ruby/1.9.1/bundler/gems/rails-e86daf8ff727/railties/lib/rails/gem_dependency.rb:21.
/var/www/canvas/config/initializers/ruby_version_compat.rb:43: warning: already initialized constant HALF_DAYS_IN_DAY
Jeremy Davis's picture

I.e. from this post? The reason I ask is because you used the words 'uninstalled and re-installed redis' whereas the best path IMO is to update redis.

I'm guessing that you may have followed the original instructions as you have v2.8.9 (the version I recommend is v2.8.6 from Wheezy backports). Having said that I would expect that the outcome would (should) be pretty much the same... (i.e. a working server).

Out of interest, what is the response you are getting in your browser? Are you getting the same as I was (i.e. the "oops" message) or something else? Also where is that log info coming from? I had a quick look on my local test server (which I had already applied the fix to) and couldn't find anything similar in any of the logs...

Also the md5 & sha1 checksums and the gpg sig can be found here (you can find them from the appliance page by following the link that says "Manifest & Sigs"). Assuming you are using the VMDK build, this is the one you want. And there isn't actually a description of 'minimum requirements' (which is probably something for us to consider...) However 1.5GB RAM should be adequate, mine is running with 1GB RAM + 1GB swap. Generally in my experience Ruby (and Java/Tomcat) apps require 1-1.5GB min.

Peter's picture

Thanks for the quick reply, Jeremy.

I'm not sure if this is a redis problem since the error was identical before and after the update.

I tried a again from scratch using the backports (v2.8.6).

The browser says:

Web application could not be started
Phusion Passenger has listed more information about the error below

it's the same info as

$ tail /var/log/apache2/error.log

My md5 and SHA-1 of the turnkey-canvas-13.0-wheezy-amd64-vmdk.zip file is:

md5:    C4491C45C6E0A4B107C82797CD809513
SHA-1: 52D79D610BA2F1D385DCD48D63DAC9D3C19C17BB

These are NOT the same as the ones listed on your site. I downloaded it again and got the same numbers with the new files.

I can't find anything in the VirtualBox GUI that mentions swap space.

I will try it on a ubuntu host next.

Jeremy Davis's picture

If the MD5 and SHA1 are wrong then I'm 99.9% sure that the image must be corrupt. I'm using the OpenVZ template but FWIW the md5 checks out fine...

I have on occasion had an issue with SourceForge mirrors with corrupt files (on their end). I would suggest downloading from a different mirror to what it defaults to (you should be able to manually select a mirror).

Peter's picture

I checked  mirrors http://turnkey.interhost.co.il/vmdk/turnkey-canvas-13.0-wheezy-amd64-vmd...

http://kartolo.sby.datautama.net.id/turnkeylinux/vmdk/turnkey-canvas-13....

Both the md4 and SHA-1 are the same as what I have. Perhaps the vmdk.zip got corrupted before it was pushed to the mirrors?

 

 

Jeremy Davis's picture

Well I don't know about those mirrors, but I just downloaded the VMDK from Skylink.de (when I click the direct download link in SourceForge the link is https://downloads.sourceforge.net/project/turnkeylinux/vmdk/turnkey-canv...). And FWIW the md5 matches.

I then loaded it up and got the same "Oops" message. I ran the 3 lines of code from my previous post and it works...

I suggest that you try the SkyLink mirror...

Jeremy Davis's picture

I'm not actually that familiar with Canvas. It might be worth asking that question of people that know a bit more about Canvas than I do. I just did a quick google and it seems that Instructure (the makers of Canvas) have something of a forum here, although TBH the google group here looks perhaps a little more promising?

Even if they think that it is a TurnKey issue, then perhaps they might have some ideas.

Please feel free to post back if/when you do that. Also if they want/need further info from us on the resolution then I'd be more than happy to post wherever relevant. Having said that though, perhaps we should start a new thread for this discussion as it's actually a bit off topic here (vaguely relevant but I suspect the cause is quite different...)

Peter's picture

The new download file checksums match yours now. I followed your backport instructions for redis-server v2.8.6

I still get an error.

$ tail /var/log/apache2/error.log

     in 'virtual Passenger::ApplicationPool2::ProcessPtr Passenger::ApplicationPool2::SmartSpawner::spawn(const Passenger::ApplicationPool2::Options&)' (SmartSpawner.h:744)
     in 'void Passenger::ApplicationPool2::Group::spawnThreadRealMain(const SpawnerPtr&, const Passenger::ApplicationPool2::Options&, unsigned int)' (Implementation.cpp:782)

[ 2014-05-12 15:58:39.3563 2096/7f9038264700 agents/HelperAgent/RequestHandler.h:1975 ]: [Client 20] Cannot checkout session. An error occurred while starting up the preloader: it did not write a startup response in time.
Error page:

/var/www/canvas/vendor/bundle/ruby/1.9.1/bundler/gems/rails-e86daf8ff727/activesupport/lib/active_support/inflector.rb:3:in `<top (required)>': iconv will be deprecated in the future, use String#encode instead.
NOTE: Gem.source_index is deprecated, use Specification. It will be removed on or after 2011-11-01.
Gem.source_index called from /var/www/canvas/vendor/bundle/ruby/1.9.1/bundler/gems/rails-e86daf8ff727/railties/lib/rails/gem_dependency.rb:21.

The browser says:

Web Application could not be started.

Phusion Passenger has listed more information about the error below.

/var/www/canvas/vendor/bundle/ruby/1.9.1/bundler/gems/rails-e86daf8ff727/activesupport/lib/active_support/inflector.rb:3:in `<top (required)>': iconv will be deprecated in the future, use String#encode instead.
NOTE: Gem.source_index is deprecated, use Specification. It will be removed on or after 2011-11-01.
Gem.source_index called from /var/www/canvas/vendor/bundle/ruby/1.9.1/bundler/gems/rails-e86daf8ff727/railties/lib/rails/gem_dependency.rb:21.
Jeremy Davis's picture

How frustrating...! That is so annoying!

Well I guess at least we know that it's not corrupt now...

I wonder what the problem might be. When I tested last night I wasn't actually using VirtualBox (I was using KVM). So I'm about to fire it up in VirtualBox (on Windows) to see what happens then...

Peter's picture

I just switched over to my ubuntu partition. Everything works fine. I guess its a VirtualBox Windows edition bug.

The noise in error.log about HALF_DAYS_IN_DAY and Gem.source_index is the same.

Unfortunately, my job forces me to live in Windows. Stepping through the instructure github repo install instructions.

Thanks for trying, Jeremy.
 

Jeremy Davis's picture

Sorry I didn't get back sooner. I don't currently have access to a Windows machine so I wouldn't have been able to beat you to this discovery anyway. Pity we couldn't get you going with the TurnKey appliance but I understand. Sometimes you just have to cut your losses and move on...

Frustrating that there must be a bug in Windows VBox that is causing these issues. I wonder if the issue is something in the VMDK itself or whether it's something to do with the interaction between VBox and Canvas? I.e. whether the bug still remains if it is installed from the ISO rather than the VMDK...? Guess I'll have to get on a Win box sometime and see...

Could you please let me know what version Windows and what version of VBox and I'll try to recreate it when I get a chance.

Thanks for your patience and perseverance while we tried... :)

Peter's picture

My HP nc400 notebook has an Intel T7200 CPU

Windows 7 Ultimate. Service Pack 1. 32-bit
VirtualBox 4.3.2 r90405

I just discovered that my host is 32-bit while my vmdk was 64bit. This might be the problem. I'll give it a shot when i have the chance.

Jeremy Davis's picture

I have read that theoretically it is possible, but TBH I have never managed to get it to work...

Liraz Siri's picture

You'll need to apply this fix to update redis-server:

echo -e '# Backports repository\ndeb http://ftp.debian.org/debian wheezy-backports main' >> /etc/apt/sources.list.d/wheezy-backports.list
apt-get update
apt-get install -t wheezy-backports redis-server

Since it's broken out of the box, I thought about unpublishing TurnKey Canvas altogether but since the fix is relatively simple then I think there's still value in having it in there. I updated the page to document the issue and the fix.

The build pulls the latest version of Canvas from upstream. When we tested this, Canvas required the version of redis server in Debian Wheezy but a few days later when the final build went through Canvas was updated to require a newer version of Redis. I looked into this and there's no easy way to fix this in a hotpatch because the APT configuration has to be updated and it's tricky to do that from within a package.

The way to prevent this sort of thing from happening in the future is to test the final build version rather than the development version. Preferably, automatically...

Add new comment