Mystic's picture

The files tab is greyed out in Canvas RCE, and it appears that Turnkey Linux does not come bundled with the RCE server. Therefore, no teachers can upload files into their assignments. The issue is identical to the one posted on Bitnami's forum: https://community.bitnami.com/t/rich-content-editor-and-passenger/70780/4

Is there any proposed date on this solution?

Forum: 
Jeremy Davis's picture

FWIW the current TurnKey appliance does include the RCE.

The Canvas documentation is particularly lacking in detail regarding the setup of the RCE API server. There have a number of issues lodged with Canvas (including a patch to improve the integration) but they are yet to do anything with it. I guess that as their revenue comes from their hosted service, there isn't a huge priority in supporting "self hosted" instances.

We are aware of the issue and we're pretty sure that we have a solution. However, seeing as the current appliance is outdated, we're currently working on a completely updated appliance with the latest version of Canvas and RCE API fully functional. I'm not sure when that will be ready but soon hopefully!

Sorry that that doesn't help you right now. I haven't worked on it and don't have the full resolution documented.

Jeremy Davis's picture

Apologies on my such delayed response. Because I didn't hear back from you, following this up slipped down my list of priorities (the squeaky wheel gets the oil and all that...). Regardless, that was pretty poor form on my behalf and I'm really sorry about that...

I checked with my colleague who is more familiar with this and there are a few things. Firstly, you need a "proper" SSL/TLS certificate, configured for your domain (a free Let's Encrypt cert is fine). Then there is an issue with the Canvas RCE API. It appears t o be unpatched currently (see the issue here). Yo ucna find a few different workarounds noted in that thread, but we hope to produce a fully functional (i.e. patched so it works OOTB) Canvas appliance within coming weeks.

Mystic's picture

Hi Jeremy,

Thank you for your prompt response. Is there any update as to when the patched version will be released? I am wary of trying to attempt those attempted fixes until an official version that fixes the issue has been released first.

Does this issue also affect file uploads? Whenever I upload a file into the "files" section, the progress bar never works. It uploads it in the background and then it magically finishes.

I am also suddently getting weird 502 errors when uploading files to some courses. Any suggestions? The instructors cannot upload files anymore with this error.

Mystic's picture

Jeremy:

In regards to the file upload, I had to retry twice (so three attempts total) to get the file uploaded. There is nothing wrong with the quota, as the user is only using 20%. I don't know what else I could do as there is not much support and documentation regarding this problem.

Mystic's picture

Jeremy:

I found the error here:

request_parameters: ''{"name"=>"sample_video.mp4", "size"=>"77131086", "content_type"=>"video/mp4",
 "on_duplicate"=>"rename", "parent_folder_id"=>"163", "no_redirect"=>"true", "_method"=>"POST",
 "authenticity_token"=>"XrDwMzGvlqV1+A3eKIyQ1JoI8fH4Pbxua3HxzGtlBFBmiJRUGp3l5iO8RrpO9OG77UWXwpRRjAAHRtr8IjZ8BQ=="}''
exception_message: ''ERR Error running script (call to f_55328419e7888a8fff3c59901e25d48909168320):
 @user_script:30: @user_script: 30: -MISCONF Redis is configured to save RDB snapshots,
 but is currently not able to persist on disk. Commands that may modify the data
 set are disabled. Please check Redis logs for details about the error. caused by
 ERR Error running script (call to f_55328419e7888a8fff3c59901e25d48909168320): @user_script:30:
 @user_script: 30: -MISCONF Redis is configured to save RDB snapshots, but is currently
 not able to persist on disk. Commands that may modify the data set are disabled.
 Please check Redis logs for details about the error.''
hostname: www.mywebsite.com
pid: 10967
 

Jeremy Davis's picture

The updated/patched appliance is really close. We just need to do the last batch of testing on it. Hopefully it should be ready in about a week and a half.

Re the error that you've noted. That seems really weird that it's failing to run your script and complaining about Redis config - namely that it's not being saved to disk?!

TBH, I suspect that the error message is a bit of a red herring. AFAIK Canvas does not need Redis to cache to disk. I don't know for sure, but my guess is that your server might be running out of RAM? FWIW Redis runs in RAM by default - as the message hints. You can change Redis so that it caches to disk, but it shouldn't be required - running in RAM is generally recommended as that's part of what makes it so fast. Making it save to disk will likely slow things down significantly and make your server less responsive.

Mystic's picture

Jeremy: That sounds great! I look forward to the patched version very soon. What is the Debian command to process the upgrade once it is released?

As for the Redis error, I ruled out memory usage. I doubled the memory on the server from 8 to 16 and monitored the memory usage at around 20%. Even uploading small video files in the range of 70-100 MB still triggers this error.

I visited this StackOverflow topic at: https://stackoverflow.com/questions/19581059/misconf-redis-is-co and followed the second post and tried disabling the snapshot using "config set stop-writes-on-bgsave-error no". However the error still persists even after this.

Just for good measure, I increased the disk space on the / root directory but I'm still getting the error. It is causing quite a bit of headache for some teachers trying to upload files onto their courses. They have to retry at least 3-4 times before the upload successfully completes. I don't know what else I can do. Any possible suggestions?

As always, thank you for your help Jeremy!

Jeremy Davis's picture

It sounds like a strange issue!

The fact that you say that it often works eventually, suggests that it may be some sort of race condition?! Intermittent issues can be notoriously hard to resolve in my experience.

Next I was going to say that it's probably also worth double checking free disk space. But you beat me to that and it sounds like you've ruled that out...

Re the StackOverflow Q&A that you refer to, unfortunately the answer order isn't guaranteed to be the same, and many answers include the command you note. FWIW, you're best referring to a particular answer via clicking the "share" button below the relevant answer and then copy/pasting the URL.

Anyway, I'm no Redis expert, but after doing a but more reading, I'm wondering if it's a permissions issue? I.e. Redis can't write to disk? (TBH, seeing as it's intermittent, I'm really not sure but perhaps?). According to what I've read now about the setting you tweaked though, that should suppress the error message, but not necessarily fix the underlying issue. See this answer on the Q&A you linked to. Although I suggest that you keep reading before you race off and try that blidly.

This answer (again on the same Q&A you linked to) sounds like a sensible approach and has quite a few upvotes. I'm not recommending that you necessarily follow the suggestion to modify /etc/sysctl.conf, but starting at the log file (probably /var/log/redis/redis-server.log) or checking via journalctl seems like a really good idea!

I suggest that you recreate the issue, and get the last few lines of the redis log so you can try to see exactly what it's up to. Assuming that the log file is /var/log/redis/redis-server.log then to get just the last 30 lines, do this:

tail -30 /var/log/redis/redis-server.log

Feel free to post here if you want my 2c on it.

Regarding getting further help. Canvas have a support Q&A, sort of like forums, although my experience suggests that it's not so great for technical self-hosting related issues. The Canvas Google group has been quite good and contains some really good info, but getting timely assistance for specific issues can be a bit hit and miss sometimes. Still it may be worth posting there? If you do, please feel free to cross post a link to that here.

Mystic's picture

Hi Jeremy,

Thank you so much for your detailed response. Over the past week, I have been carefully monitoring the server and doing some test uploads. The issue disappeared once I upgraded our server to the next EC2 tier with increased memory.  I am pretty sure now that it was a low memory issue. I tried to stress the server with multiple running Canvax sessions and I was pretty shocked at how quickly the free memory dropped. If you couple that with a user uploading a video file, I can see how the instance would be short on memory very quickly.

I was aware that Canvas is a resource-intensive application, but I wasn't expecting this quick of a drop. Anyway, for now it seems to be ok! :)

In the meantime, I am looking forward to the update with the RCE issue fixed!

Jeremy Davis's picture

I'm really glad to hear that you worked it out and thanks for reporting back to let us know. I'm almost certain that this will be useful for other users.

TBH, I'm not completely surprised that it ended up being system resources. Canvas is a beast of a thing and the hardware requirements seem to just escalate over time!

As for the update, I haven't yet announced it, but the new Canvas release should now be available on AWS via the Hub (and I hope to have it available from the Marketplace really soon - but unfortunately no clear ETA). It's possibly irrelevant to you, but I'll update the appliance page soon with the ISO, OVA & VMDK.

Regardless, there are 2 primary pathways open to you. One is to manually upgrade/update your existing server (both OS and Canvas itself). The other is to migrate your data to a new server. Unfortunately in the case of the Canvas appliance, neither are going to be super clean and easy and both will require you to get your hands a bit dirty and may need some trial and error.

If you choose the upgrade path, then you will need to create a snapshot of your current server. I personally advise that you launch a new server from your snapshot and work within that. I also advise that you document as you go, so worst case, you can trash the snapshot server and start again (and be able to easily recreate the steps that worked). That will also allow you to limit the downtime. You could even leave the current server untouched and do a trial run with a snapshot and then replay your documentation later. Essentially, the OS migration is migrating from Debian 9/Stretch (the basis of TurnKey v15.x) to Debian 10/Buster (the basis of TurnKey v16.x). The Debian docs are quite extensive, plus you'll find plenty of tutorials online. As for Canvas, generally the Canvas upgrade docs will be the primary source of info in that regard. There may be some idiosyncracies (such as our last Canvas release included a fork of the RCE code repo in an effort to work around an issue in the upstream source code). But mostly it should be a case of following their recommendations.

The other option is to migrate your data from your existing instance to a newer one. In theory, this path would be my preferred option as then you just move what you need to and everything else is fresh and shiny (e.g. newer version of Ruby without needing to make any changes) and with no old config and files laying around. Unfortunately, I'm not super familiar with the inner workings of Canvas, so am not aware of everything that might need to be moved. You would obviously need to move the database (i.e. dump your current DB and upload to the new server). Your config files will also need to be copied, plus wherever the uploaded files are stored (a quick google suggests it may be /var/www/canvas/tmp perhaps?). You would then need to update the config files as per the Canvas upgrade docs; as well as running the migrations (to upgrade your database).

Our TKLBAM backup tool may be useful, but unfortunately, in the case of Canvas, it almost certainly won't be a simple case of backing up your current server and restoring your backup to a new TurnKey server...

Whichever way you go, I would urge you to work within a "disposable" server and document as you go. That will relieve some pressure as worst case you can trash it and start again. Also, you won't be getting so much pressure from others as they won't be affected any more than needed i.e. when you actually do the migration or upgrade "for real". If you go the data migration pathway, you could even download the ISO or VM build and do your initial explorations in a local VM.

Anyway, I'm happy to assist as best I can. So please feel free to ask if you have questions. I can also elaborate further if you need more info.

Mystic's picture

Hi Jeremy,

No problem at all and thank you so much for your descriptive post. I think I would rather hold off until the latest version is available on marketplace. Once it is available on marketplace, I will attempt the upgrade. I will be checking this post everyday until I hear your update, and then I will go ahead and try the migration route (second option) that you suggested. 

Again, thank you so much and look forward to your update!

xeoto's picture

Does this issue also affect file uploads? Whenever I upload a file into the "files" section, the progress bar never works. It uploads it in the background and then it magically finishes.

Mystic's picture

Hi Jeremy,

I did the update (used the latest image that was released on 6/26 and I am still getting the same error. The files tab is greyed out, and when you click on Assignments to embed an assignment within a page, it says "Loading Failed". The update did not seem to fix this issue :(. See the below image.

Jeremy Davis's picture

Investigating another issue with another Canvas user, we've discovered a weird intermittent problem. You can read more on that thread and/or on the issue. Essentially it's a race condition between the "Passenger standalone" service and mod_passenger. The "standalone" config does not include serving the API so if it runs under that, the API won't be working. That explains why it worked when I tested it, but others have reported ongoing issues (or not - as the case maybe).

As noted, the fix is to disable the "passenger" service and restart Apache. I.e.:

systemctl stop passenger
systemctl disable passenger
systemctl restart apache2

It's also worth noting that there are 2 other requirements for the RCE API to work properly. You may already be fulfilling these, but for the sake of completeness:

  • Your server must be configured to be served and accessible via a "proper" domain name which you own.
  • Your server must have a valid TLS (aka SSL) certificate (Let's Encrypt certs are fine, but self signed certs won't work).

We'll be releasing an updated Canvas appliance with the required changes included ASAP. However, following the instructions above should get your current server up and running properly.

If you continue to have issues, please confirm that you have fulfilled the other points as well. If you want further assistance, please share the URL so that I can have a look from my end to double check the DNS config and cert. Further investigation will require you create a log in for me on your Canvas server (it doesn't need to be admin, so long as you can guide me on how to recreate the issue from a non-admin user account).

Mystic's picture

Hi Jeremy,

Thanks so much for your fast response. I have PM'ed you with my information.

Jeremy Davis's picture

I've just logged in and had a quick look. On face value everything appears to be working ok. Although I can't see the page that you show in your screenshot and I can't work out how to get to it?

Although, one thing I did notice is that a lot of the content (including cookies) is being served from your server's IP address, rather than the domain name. I'm not sure, but I suspect that this may be part of the issue?! The RCE API is served via HTTPS on port 3000.2 It uses CORS to allow the usage of resources from different sources (different ports are considered different sources). For security, CORS has a same origin policy. So to allow the use of an alternate origin (i.e. port 3000) a special header is provided by the web server allowing the use of DOMAIN_NAME:3000. However, my guess is that instead of providing DOMAIN_NAME:3000, the header is providing IP_ADDRESS:3000 so it doesn't match what Canvas is pre-configured to accept.

I could be wrong, but my guess is that when you first set this up you were using the IP address. But then you switched to using the domain. And somewhere along the line the IP address has been stored somewhere (perhaps in the DB? perhaps in a a config file?) and although it appears to work ok via the domain, it's actually serving a lot of the content directly from the IP address...

Also FWIW (and I'm fairly sure this isn't related, but is worth fixing) a few things (such as images) appear to be being served via vanilla HTTP (even when browsing via an https URL). A dirty but easy way to work around that would be to redirect http to https. The better way to do that would be to fix what is explictly using http, and redirect to https anyway...

Jeremy Davis's picture

Ok so I worked out how to get to where you were. Part of the issue was that I had "Chrome developer tools" enabled (which allows you to see javascript errors which is invaluable for resolving these sort of issues, but was sort of a little confusing in this case). I disabled that and browsed to Courses >> [selected a course] >> Pages >> Add page.

That showed the same issue as per your screenshot. I then re-enabled the dev tools and I can see that your Canvas instance is trying to load the API from localhost:3000 ?! Clearly that's not going to work - unless of course you had installed Canvas locally...!

So I had a bit more of a look. I'm not sure why or how, but my bet is that for some reason, the relevant config file wasn't updated on firstboot. The "dynamic settings" file ships with "localhost:3000" as the rce-api location, but it should be updated when you run the firstboot scripts. Regardless, please try manually adjust that. The file should be at /var/www/canvas/config/dynamic_settings.yml and the line that you'll need to update should currently look like something this:

     app-host: localhost:3000

Change it to this (instead of YOUR_FULL_DOMAIN use your actual full domain name for your Canvas site):

     app-host: YOUR_FULL_DOMAIN:3000

Then restart the relevant services:

systemctl restart canvas_init
systemctl restart apache2

And hopefully you should be all good now! :)

Please let me know regardless...

Mystic's picture

Hi Jeremy,

That works and thank you so much! I would have never figured out to modify that dynamic_settings file without your file. Shouldn't the Turnkey setup change that URL during the initial setup process?

However the biggest issue is in the migration. I tried to dump and restore the .rdb database from the old Canvas server to the new one (following this guide: https://community.pivotal.io/s/article/How-to-Backup-and-Restore-Open-Source-Redis?language=en_US), but after I do this, the server has an internal 500 error and won't connect. The Canvas upgrade docs (https://github.com/instructure/canvas-lms/wiki/Upgrading) weren't of any help because it is not pulling from any git repository and the commands all fail. 

I'm fine with getting a quote to do this job because I don't see any other way, and re-creating all of the courses and users manually would be a lot of time. I'm so surprised that with such an advanced piece of software that you cannot do a simple backup and restore. Where to go from here?

Jeremy Davis's picture

So I don't think it's necessary to backup Redis (although I could be wrong... I certainly don't consider myself a Canvas expert). Regardless, have you tried troubleshooting the 500 error? That usually refers to a background service that has crashed and/or isn't running.

The bulk of the data is in the PostgreSQL 'canvas' DB. and the rest should be within a particular directory within the Canvas dir (/var/www/canvas). I forget what it's called, but perhaps "files"?

Personally, I think migrating to the newer version is a better path (then you get the updated OS as well), but upgrading your current install should be possible. You note that "it is not pulling from any git repository and the commands all fail". What version of the appliance are you using? TurnKey Canvas appliance has been installing Canvas via git fro some time. Although one v15.x release was using a third party repo for the RCE API because of a bug in the Instructure repo. So my guess is that perhaps you were in the wrong dir (or perhaps it's a really old TurnKey Canvas server)?

I don't know about any other modules/plugins/addons (or even if Canvas has those?) but the DB and the files should be enough to transfer your data from the old server to the new one. Our built in blackup tool; TKLBAM should be able to collect everything you need from your old server and transfer it to your new server. Although I doubt that a vanilla restore across a major version will just work, so I'd recommend what we call a "staged restore". If you're interested in hearing more about that, then let me know.

Regardless of how, if/when you migrate your data, you will still need to run some of the "upgrade" steps from the Canvas docs. Particularly the database migrations, but some of the other steps too perhaps...

Ultimately, they make money out of hosting it themselves. It's open source and I don't think that they go out of their way to make it any harder to self host than they have to. But their business model depends on the fact that it's often easier to pay them than it is to host and maintain yourself. So there is not much motivation for them to make it even easier to self host/self manage as it would likely cannibalise their sales.

Jeremy Davis's picture

I just realised that you raised a question which I hadn't directly addressed:

Shouldn't the Turnkey setup change that URL during the initial setup process?

Yes, it should- and it should have! I'm not at all clear why it didn't in your case but I will investigate as it's possibly a bug (which may have already been addressed by my colleague in the upcoming update and if not it will be)? The new appliance (which I hope to release next week) will certainly be more robust and should "just work".

Mystic's picture

 

Jeremy,

Sorry to bother you again. I've started doing using the Export Course and Import Course modules to import the course content but all of the import jobs are queud and not moving. I'm not sure why. You can login and see what I mean.

 
 
Jeremy Davis's picture

I am unaware of those modules, but the background task runner runs as a separate server called "canvas_init". So I suggest that you check the status of that. I.e.:

systemctl status cavnas_init
Mystic's picture

Jeremy,

I followed the upgrade instructions on https://github.com/instructure/canvas-lms/wiki/Upgrading and I got the git repo all updated, the gems updated, the node modules updated, and recompiled all the assets and it didn't actually upgrade it into the next version (was still in 15.3 so I don't know why).

I tried TKLBAM but it also didn't work (got some weird internal errors and wasn't working). I think I may just wait for you to release your new version on AWS until I do the next upgrade.

In regards to checking the canvas_init status, it is running normally but the imports are still all queued. 

Mystic's picture

Hi Jeremy,

Sorry for going back and forth. I finally fixed the issue using the documentation provided in:

https://github.com/instructure/canvas-rce-api/issues/6#issuecomment-5949...

And...

https://www.programmersought.com/article/3729940188/

I won't be doing an upgrade for now until I really understand how to do the backup in the least painful way possible. Thank you for all your help!

Add new comment