You are here
I wanted to play with writing a patch, so I started playing with creating one for SABnzbd and a few addons. I've got the patch working but need to do a little more testing on it before releasing it into the wild. This testing should be done in the next day or two.
For those unaware, SABnzbd is an Open Source Binary Newsreader written in Python.
SABnzbd makes Usenet as simple and streamlined as possible by automating everything we can. All you have to do is add an .nzb. SABnzbd takes over from there, where it will be automatically downloaded, verified, repaired, extracted and filed away with zero human interaction.
There are two very popular addons, SickBeard and CouchPotato which help automate the rerieval of nzb files for TV Shows and Movies (respectively).
I've created this patch mostly for myself but thought I'd query here to see if there is a general interest in this or not. If you're interested, respond and I'll put the patch somewhere people can get to it.
Sounds awesome!
I for one will be very interested in this!
Out of interest, what are you basing it on? Are you just using Core? Or are you using the Fileserver appliance (IMO that would be the best one). Also it may even be worth considering the Torrent server appliance? An AIO torrent/nzb/fileserver appliance!? Despite how cool that sounds, I guess I'd probably be inclined to stick with the fileserver appliance and consider the AIO server separately... Just my 2c though! :)
My work so far is based upon
My work so far is based upon the LAMP stack, as I'll eventually want a mysql and apache2 instance for a newznab instance. That said, I'm not doing anything yet that requires more than core so maybe I'll look at the fileserver appliance as well.
It is probably easier to add LAMP functionality
Than to install eXtplorer etc. Although it wouldn't be unsurmountable. I'm going to be away for a few weeks but when I get back I'm happy to chip in my 2c on your progress if you'd like some feedback.
Almost there...
I just took a look at fileserver, and it does have some nifty features. I migrated to use it as the base and now have the system mostly setup (all services startup correctly and configure themselves as best as possible, accounts/passwords are created and set, I've sucessfully been able to download several binaries, and retrieve/view them with eXtplorer).I've got a little more testing to do as well as some code cleanup, but I'm getting very close. Next steps:
Brilliant!
Sounds like this is going to be a killer patch/appliance! And hopefully it'll make it into the part 2 of the v11.x release ie become an official appliance.
I'm leaving tomorrow and won't be back for a few weeks but I'll forward to this when I get back!
Just a brief update... I
Just a brief update...
I have just released version 0.1 of this tklpatch and iso image (based upon fileserver). They're both available on my webserver at http://www.aceshome.com
BIG NOTE: Pulling a lot of usenet articles down will consume a lot of disk space. It's best to either create an LVM (so you can add disk later) or mount /srv/storage on a large disk. I installed on a 120GB virtual disk and then copy files off when I want to keep them.
Once the appliance has been deployed, you must configure a few things:
The major things to configure are your usenet credentials and search providers.
Note that to use this appliance, you should have the following:
I just released version 0.2,
I just released version 0.2, after adding MediaFrontPage to it.
Version 0.3 is up now (well,
Version 0.3 is up now (well, it has been for a few days). I've seen about 15 downloads of the patch and 15 of the iso, if you have taken a look and have some feedback... I am all ears.
Get it at http://www.aceshome.com
This looks friggin awesome!
Please post the TKLPatch somewhere so we can add this officially to the next release of the TurnKey library. If you can attach it to your forum post that would be great.
I have been putting the
I have been putting the source to my tklpatches and source on github at https://github.com/tssgery
There are currently 4 of them available:
Some of the prebuilt patches and iso images are on my personal website at http://www.aceshome.com
Sounds like you've made great progress on this!
I've not been back long from my holiday, but I'll be having a much closer look at this one the moment I have time (I also have a lot of other projects to tidy up). Good work! :)
With Liraz's glowing endorsement above you can be quite sure that your patch will make it into the v11.x part 2 release (when it finally sees the light of day - hopefully not too long now...)!
Bit of feedback
I have just installed it and nice touch with the setting of password on first boot. Good work! (I must admit that I have just had default passwords in the patches I have made - hoping that the core devs will fix that for me when they create the official appliance).
I have yet to do anything much with it yet but the first thing I'd like to note is a slight mistake in your original docs (SABnzbd seems to be running on 8080 - eXtplorer is on 80).
Also I'm not quite sure what's going on. I haven't done much troubleshooting yet but it seems that SickBeard and CouchPotato aren't running (or at least aren't listening). SABnzbd is working fine. I did a quick netstat -na |grep LISTEN and there is only the default Fileserver stuff + SABnzbd. I'll do some more testing and/or have a look at the patch and get back to you.
Finally I'd like to suggest customising the default services.txt for confconsole with these details:
Here is one I just threw together (it's a very small thing but makes it a little more finished). It needs to go in /etc/confconsole/services.txt IMO it's best put in the overlay (rather than mucking around with sed).
Also I think it's a good idea to have all the interfaces available via https. Obviously that won't be an issue when run locally although perhaps users may wish to access it remotely and IMO it's definately an issue if run in the cloud.
[update] Just checking your patch and it seems that sed is meant to be adding to services.txt - not sure why it's not working. I'll investigate further. Obviously there are a few things not woring as they should...
[another update] I just checked my TKLPatch log and it seems that none of the commands after installing SABnzbd ran!? So SickBeard and CouchPotato aren't even installed! Not sure what's going on there...
I'll check when I get home on
I'll check when I get home on Sunday but suspect it's something trivial. I did most of my testing via installing the patched iso and assume you actually installed fileserver and then tried patching the running VM?
Thanks
I'm at work now so I won't get a chance to play any more until I get home. After a little more investigation last night it seemed that the other apps where installed (despite the fact that my log didn't show it - not sure why...?). The big issue seems to be that the customised LigHTTPd conf file wasn't copied across, but even when I did it manually it still didn't work. I think I'll download your ISO and install from that so I can see how it's meant to be. Then I can compare and see why the patch didn't stick.
BTW I patched a vanila TKL Fileserver ISO using TKLPatch running on TKL Core (rather than installing in a running instance of TKL Fileserver).
If you just tklpatched a
If you just tklpatched a filserver iso and then installed a VM from the new patched iso, that's almost exactly what I do (I patch the iso on a LAMP server instead of core). I'll look this evening, US time, and see what I find. If you can put your iso somewhere that I can get it, I can diff the contents of yours and mine and get a clue or two. That'd be easier than you looking and telling me what you find.
Almost there!
I've downloaded your ISO And it's definately closer (and different - it's nearly 20MB bigger for starters). But it's still not quite working as it should.
The confconsole screen doesn't display all the info (no mention of SABnzbd, SickBeard or CouchPotato - I worked out why that was, see below) and trying to log in via the web browser (using Firefox 7.0.1 under Bodhi Linux 1.2.0 - based on Ubuntu 10.04 but with lots of updated stuff) causes some strange behaviour. When I type the full url/IP into the address bar (ie http://192.168.1.103/) and hit go, first the http and slashes disapear (leaving just 192.168.1.103) then it brings up a search for 'nzbapp/mfp' (currently my local DNS server forwards DNS requests to OpenDNS - and when it receives a request that it thinks isn't a url it does a OpenDNS web search - see guide.opendns.com/main).
http://192.168.1.103/extplorer/ loads extplorer ok and when I use port 8085 it redirects successfully to http://192.168.1.103:8085/movie/ (CouchPotato) 8080 brings up SABnzbd wizard (http://192.168.1.103:8080/wizard/) and 8081 brings up SickBeard (http://192.168.1.103:8081/home/).
But now I have just found MediaFrontPage at http://192.168.1.103/mfp/ and it all becomes clear why you don't mention SABnzd, SickBeard or CouchPotato on the confconsole screen! Wow! That is seriously cool! I like all those cool widget things for the different services they're awesome!
Only thing is that eXplorer is the only one that works :( The other tabs cause similar OpenDNS search behaviour as for the IP address mentioned above: searches for nzbapp:<port> - except they're in a frame of MFP. I guess that if I had a local DNS entry for nzbapp all would be well, but you shouldn't need that IMO.
Also during the time it's taken to write this post (testing as I write) my browser has crashed 3 times (thankfully I've got the Lazaras addon so I don't have to keep typing!) I only updated a few days ago so perhaps it's coincidental but it's the first time I've had any issues.
Sorry to sound so critical. I think it's a fantastic idea and you've obviously put a lot of thought and work into it so good job! :)
Seriously strange
What you describe is seriously strange. I've deployed that iso about 100 times and not experienced this, but I suspect it is dns related. I use OpenDNS myself and don't suspect that but instead think I have some flawed logic that I assume your hostnames are resolvable (for example, I use dnsmasque and have configured it to resolve 'nzbapp' to the static IP I setup the VM with. Is your VM statically or dynamically addressed? I find it hard to believe that anything in this could cause a browser crash but if you send me the specifics of the browser version and plugins you have, I'll try and recreate it.
You can try...
... Adding an entry for 'nzbapp' into yout /etc/hosts file and see what happens. That would validate if it's a dns problem or something else.
just tested it a bit, and...
you should add 'nzbapp' to your hosts file and set it to resolve to the IP assigned to the appliance. There are several places where the configuration files need to point to applications and the only reliable way is by setting the hostname. If your dhcp server updates dns with the hostname you may be able to get by without the hosts file change but I assume that not all routers are configured that way.
I should have checked this before releasing the patch but I always set my servers to static addressing so didn't catch it.
Don't sweat it :)
Its all good. It's hardly mission critical stuff and I think you've done an awesome job with this appliance! I still need to work out why the patch didn't stick when I did it myself but I'll leave that for another day...
Out of interest, first I tried adding an entry to my local (Linux) DNS and that didn't work either (it seems to just return a FQDN which doesn't work).
It works now (via editing the hosts file), but I think it's something worth further consideration. I imagine most users will want to use this appliance locally at home in which case a hosts edit isn't that big a deal, but what if a user wants to access it remotely (or even just have a play with it in the cloud)? What if a user is using a device that doesn't easily allow that level of configuration (no access to root or admin account/game console/tablet/smart phone/etc?) IMO needing to add to the hosts file is a little clunky and ultimately may be a showstopper for some.
[update] I just realised that I should only have to update the hostname (to a FQDN) to get it to work with my DNS shouldn't I!? I haven't tried yet but I reckon that'll fix it!
you'll still need to be able to resolve the full hostname
I have made some assumptions in my appliance that the hostname is 'nzbapp', if you change the hostname (via editing /etc/hostname and such), you'll need to change configuration files as well.
What I should probably do is to prompt the user for the hostname during deployment and use that for configuration. That way, if they deploy to amazon they can use the real resovable name.
I think that's a great idea
If you create a inithook script and set it to run on first boot that would be a winner IMO. I'm assuming that's how you ask the password question on install?
If you were to go that way then users could choose to use a simple hostname, a FQDN or even just an IP. The beauty of an inithook script too is that if they change their mind they can just manually relaunch the script, enter new details and everything would be updated.
I've got it asking for a hostname
I'm asking for a hostname now and setting the config files properly. Re-running the inithook script sets the value properly. One thing I would advise against though, is setting the hostname to be the IP address. In a DHCP environment, the IP address can change at any time (yes, the DHCP server will almost always re-assign the same IP but it's not guaranteed). This will lead to all sorts of confusion on the behalf of users.
I am going to take a look at your patch issue next.
Awesome work
Good job! :)
As for DHCP & setting hostname as IP: agreed! Because I always set static IPs I forget about the idiosyncracies of DHCP. So yes you are right, setting the hostname as IP and not setting a static IP is a very bad plan!
patch issue
I found one issue with the patch. I was renaming the patch file for redistribution (adding the release number to the patch name) so it was easier to identify what version of the patch it is. When running tklpatch against this file it couldn't probperly extract and apply the patch, throwing some errors and failing to create the patched iso file. I am not sure this is what you saw but I will upload a new nzbapp.tar.gz patch later today.
I took my new patch and have applied it to the fileserver base on a core appliance without issue.
If you see this error again, please capture the stdout/stderr of the tklpatch command and let me know.
I don't think that was my problem
Because when that happens it basically just ends up with an unchanged ISO. So still not sure why it didn't work for me and unfortunately I haven't had time to have a look yet (too much work and too many other projects on the go). As I think I said earlier it may be my patching environment. I have a installation of Core that I use for patching, perhaps it needs a refresh!?
As for the TKLPatch issue you speak of, when I create a TKLPatch I find the best way to avoid that easily made mistake is instead of using tar to create the patch.tar.gz I just use the tklpatch-bundle command (it basically does the same thing - just avoids the risk of that mistake).
I think the idea of including a version number in the patch name is a good one. It allows users to see at a glance whether they are using the latest version.
version 0.4 is uploaded to
version 0.4 is uploaded to http://www.aceshome.com
Please note that the only changes are:
Excellent work - you rock! :)
Nice one. I'll download and test sometime when I get a chance. Got some other stuff I need to get onto over the next few days but I'll definately be back to check this out!
I think that this is a really good candidate for release as an official TKL appliance! :)
Some more feedback...
Hi there. I'm having another look at your NZBapp. I must say I really like it, although I admit, I'm still yet to take it for a proper road test yet.
I was much more successful patching the Fileserver ISO this time. Not quite sure what happened last time... I haven't yet actually set it up to download any NZBs so I am only in the early stages of testing really but thought I'd post what I've found so far (and some questions too). :)
I now have an instance running on KVM on my PVE server and one on OVZ too. What sort of specs do you recommend for this? Here's what I've found so far (I have other VMs idling too so not really scientific...)
My KVM instance has 1GB RAM and 2 cores allocated and is running quite well. I initially started with 512MB and one core and that was not much good at all. It was really laggy. The RAM boost (which I did first) noticably helped but enabling the second core was what made the big difference. I did try increasing RAM some more, but that made no difference (no surprise really as it's only using 530MB at idle).
OTOH it's running under OVZ much nicer. I has only one core allocated and 512MB RAM (and is using ~60MB swap too - strange that the OVZ version is using more RAM) and is still more responsive than the KVM instance. I guess that more comes down to the idiosyncracies of the virtualisation tech though.
One question for you... Is there a good reason why you don't just make /mfp the web root? On both instances it takes noticably longer to load the MFP front page and settings tab (in comparison to SABnzb, SickBeard and CouchPotato - those 3 are quite snappy) and I can't help but think it's the redirects. What do you think?
One strange thing about the OVZ instance too is that you have to manually point it to /mfp or otherwise it redirects to OpenDNS search (and it searches for 'nzbapp/mfp'). eg nzbovz.home.local redirects to OpenDNS (despite the fact I have the DNS working now) but nzbovz.home.local/mfp works fine. I'm assuming that there's a setting somewhere that's not applying from the first boot scripts. (I made the OVZ image from the ISO that I patched).
One other little thing... The confconsole page is really hard to read. It seems that it doesn't have the correct line breaks, so the lines are all just jumbled up together. Out of interest, is it like that on your ISO too? IMO it's not worth redoing your patch just for that. Personally I prefer to just include the custom services.txt in the patch overlay, rather than writing to it with sed. In my experience using overlay files is preferred whenever possible as it seems to be much more reliable than sed (although obviously sometimes you have to use sed). Although I'm not good with sed anyway so I have an aversion to using it (regular expressions do my head in!!)
Thanks for the
Lot's of comments/questions, let me try to address them.
Firstly, I'm glad that your patching worked better this time. :)
As far as resources, I run VMware ESXi 4.1 and have devoted 1vCPU and 4GB of memory to it. I don't have great numbers for what memory I am actually using though. It should run pretty small while not downloading any content but will definitely spike while downloading. 1vCPU and 1GB should be sufficient for most uses. As a note, SABnzbd is the heaviest user of CPU/memory and many people run it on embedded systems so they have taken a lot of pain to make it resource light. I just happen to have 16GB of memory in that ESXi server so gave it a lot.
The redirect to /mfp for the webroot does not take a lot of time. It's the actually loading of MediaFrontPage that takes a while. Firstly, let me say that I am not 100% happy to have used mfp. I like the functionality but have problems with the way it is designed and configured (it is the best thing I have found though, and I think it will continue to improve). What mfp is doing upon load is contacting SABnzbd to get a list of downloads occurring as well as pulling down rss feed content and such. Simply put, this takes a while. I can go more into why it takes a while but I will leave that for another day :)
The strangeness within DNS you are seeing is due to a couple of things: 1) The way mfp is configured, it needs to know the ip address/hostname of the SAB, sickbeard, and couchpotato instances so it can pass them to the browser running the mfp client. 2) The way my patch was working was to assume the hostname was 'nzbapp'. I placed the 'nzbapp' hostname in the configuration files that mfp needs (/var/www/mfp/config.ini I believe). In version 0.4 of the patch, I changed that logic and now ask the user during deployment via iso what their hostname should be. I then use that value in the mfp configuration. BTW, version 0.4 is on www.aceshome.com
My confconsole looks correct. I switched from sed to just echoing the lines out for v0.3 of my patch (I'd rather create the service.txt file from the conf script than an overlay). Which version of the patch did you apply?
BTW, I will probably release another version of the nzbapp within the next few days. There have been some changes to mfp that I'd like include as they fix problems I have seen (but are not mentioned here).
I take that last comment
Edit: After looking into this, it appears that confconsole does not like the occurrence of a '\n' string in /etc/confconsole/services.txt. because our hostnames both started with an 'n', I was echoing the SMB connection string as "\\hostname" and causing the problem.
I have tried every possible thing I can think of to escape the backslashes properly and have not found one that works. I'll keep looking in my spare time but for now will eliminate the "\" character from the file.
Even more feedback...
I've had a bit more of a play and all I can say is that this is AWESOME!
This would have to be the coolest thing ever! You have done an excellent job with this and setup is so easy.
I look forward to playing some more. I think I'll have to set myself up with a XBMC-PC now (it's been on the cards for a while anyway...)
Can't thank you enough!
Glad you like it
I'm working on one last version (fixing the confconsole text and moving to a newer MediaFrontPage), and once that is done... I'll call this a "wrap". It started as a learning experience with TKL and since I don't use this software at home... it's time to move onto something else :)
Sounds good
So you going to do another TKLpatch? If so, looking forward to what you next come up with!
version 0.5 available
This is likely the last release of this from me and addresses a couple of issues:
patch and iso can be downlaoded from www.aceshome.com
Version 0.6 available
One more release...
I have added the HeadPhones addon which let's you search for music.
Version 0.6 is available at http://www.aceshome.com
Check which ports are being listened on
To do that, from the commandline (using either SSH or WebShell) use this command:
The meaning of the switches:
-p Tell me the name of the program and it's PID
-l that is listening
-u on UDP ports
-t and TCP ports
But OTTOMH it's 80, 8080, 8081 & 8085 for the webapps plus 12320 (WebShell) and 12321 (Webmin). Also if you want to use the Samba fileshares you'll need ports 139 and 445.
Thanks Jeremy. version
Thanks Jeremy.
version 0.6 also added HeadPhones which sits on port 8181, so the complete list for v0.6 is:
This does bring up a good point though... I DID NOT set iptables up to start automatically nor did I configure the rules to allow these ports. As with anything other system, I would make sure that you either configure/start iptables OR put this appliance behind another firewall. If you are not familiar with iptables, the easiest way to configure it may be to use webmin.
You can, of course, change the webapp ports if you would like. You would need to change the webapp configuration AND the lighttpd config file.
MFP isn't perfect
As you have found out, MFP isn't perfect. I does take some time to load and query the other services as to their status.
On my configuration (the VM has 1 vCPU and 4GB memory), most of MFP comes up within a second but the "System Info" and "SABnzbd queue" take between 5 and 10 seconds as well. I wish I could offer you some black magic that solves this but I can't.
One more comment: Development on MFP has all but stopped. The developers of MFP are working on Maraschino now, which I plan to look at this week to see if it's a better solution than MFP.
Wow Maraschino looks really nice! :)
And I notice that Maraschino is written in Python... So may be a little more responsive? It still looks like a pretty young project though (only just released v0.1 ~a month ago).
Anyway, glad we got you here mate, with your finger on the pulse with this media type stuff! Regardless of the imperfections of MFP I still think this would have to be one of my favourite appliances ATM! :)
I'll be interested to see/hear how you go with Maraschino.
Yeah it's a little laggy especially on first load
As I said above it works pretty nice for me with the same allowance as you - although my environment is a little different: running under KVM on headless Linux (PVE v1.9) 'server' (actually an old E6300/2.3GHz C2D desktop with GPU removed and RAM boosted to 8GB) rather than in VBox under Win. Still I would've thought that your gruntier CPU would've been a noticable improvement.
Mine still lags a bit on first access (although not as bad as you are reporting) but is noticably faster if I go back to the main or settings screens (perhaps because of browser caching?) Actually IIRC the widgets themselves still take a little while to load (but the rest of the page, the tabs etc all load pretty quick).
I have a few ideas on tweaks you could try. I'm not sure what (if any) tweaks tssgery has done to the standard TKL LAMP appliance in his patch/ISO (and haven't got access to my server ATM to check) but some or all of the following are perhaps worth a try? Be great if you could keep track of what you do and post back with results. Hopefully this will help tweak this for default optimal performance (for when it becomes an official TKL appliance (as I hope it does).
So here goes a few ideas...:
[update] tssgerry has already done this, see post below.
Enable image caching (from MFP website) - It seems that it's just meant to cache images (so no idea if it will help) and there is also an unresolved bug listed against it, so not even sure if it works at all...?Increase php memory allocation - AFAIK MFP is written in php so this may help. The config file should be found /etc/php5/apache2/php.ini and the item to change the vaue of is memory_limit, OTTOMH the default value is 32MB (in a standard TKL LAMP appliance), try boosting that up a bit. Maybe try 64MB and see if that helps, and if not enough, keep going - although if you haven't noticed any improvement by 128MB its probably not going to do it.
Apache caching - It's not something I've really played with but could be worth a shot. Unfortunately I can't offer you any guidance there so you're on your own (but there should be plenty of info online about that).
A final word - There are probably lots of other php and Apache tweaks that could be worth a try but it'll be a case of step-by-step trial and error; search then test. If you're googling keep in mind that TKL v11.x is based on Ubuntu 10.04 so anything that applies to Lucid Server (Ubuntu 10.04) should apply to TKL v11.x. Just a word of warning with googling Apache related info, that there are some idiosyncracies to the Debian/Ubuntu versions of Apache (so many instructions for other versions of Linux probably won't apply). For most tweaks you'll need to restart Apache before any changes will take effect:
Good luck and hope to hear back from you! :)
sbpcache is already enabled
Thanks for the ideas.
Here's my 2 cents... I have already enabled the sbpcache in the patch and I really doubt (99.9% certainty) that caching (this or apache) is not the problem here.
Tuning php would be interesting to try, though I think even this would yield very minor gains. Php is not a great performing language and it's used a lot in MFP but the amount of processing is still very low.
The way MFP works is to have a gadget either use the SABnzbd, Sickbeard, or CouchPotato API or screen-scrape to pull the data. I suspect it's the communication between these components an dprocessing of the responses that's causing the delays in loading some of the widgets. There are likely several ways to improve this by going into MFP and fixing some of the architectual and coding issues. I had started to do this but life got in the way and I've been unable to spend any time on it Given that MFP seems to be dying off, I'd rather spend time on looking for alternatives.
All that said, if other people take a look and find some enhancements to make... please let me know or fork my github repo and make changes. I'd be happy to roll them in :)
You're probably right re caching
I was just throwing ideas around. :)
As for php memory allocation, in my (limited) understanding the implications (and the potential for performance improvement) of increasing memory allocation depend a lot on how the php coding is done. Regardless it's probably worth a try. I might even give it a crack when I get home and see if it makes any difference).
And yes I agree, if MFP is a stalled/dying project (which it seems it is - after your last post I had a bit of a look and saw that there haven't been any commits for a while) you're better off putting your energy elsewhere. Besides I think Python is better anyway! :)
As I said, that other one you mentioned looks pretty sweet too so could be worth a crack.
Another version on the way
It's been a while since I released a version of this but wanted people to know that there is a new version in the works. Hopefully, I'll get it out sometime this week.
I have switched from MediaFrontPage to Maraschino and things are looking really good. Maraschino is much more actively developed now, works better, and is [IMHO] more visually appealing.
Awesome!
I'll be keen to test it once you release it.
TBH I'm still using v0.3 but looking to update and consolidate this server with my old (TKL v2009.x - Ubuntu Hardy based) Fileserver anyway. Currently I run both (as separate VMs on my PVE server) but it seems crazy not to have them consolidated and I'm getting sick of having to either move files out of the NZB server to my Fileserver and/or keep increasing the disk size of my NZB server... Especially now that I have started using an XBMC HTPC setup, it would be much nicer to have the server backend set up a bit better.
The only reason why I haven't consolidated them sooner is for a few reasons. Firstly it took me a while to work out how to mount a separate physical HDD in an OVZ template (which was part of the process of migrating to a single server model - I didn't have room to copy my files from one VM to another) and secondly I discovered SSHFS (after unsuccessfully wrestling with NFS in OVZ containers under PVE). SSHFS makes it really easy to mount remote filesystems across Linux and for them to appear as local. Also the fact that I never successfully got your server to run under OVZ was a bit of a pain. I don't recall what the issue was but I couldn't get it to work. When you make your new release I'll have another bash at it.
Sounds good...
I have been able to make some headway today by totally ignoring everything else I need to do around the house so this week is looking good (possibly even later today).
I've personally not tried OVZ but can not image why this would not work as long as your name resolution is working properly. When using a front-end like MFP or Maraschino, it's really important that the hostname you specify when deploying the image is resolvable by clients.
The other think I should mention is that I have ONLY tested this by installing from the ISO image. I have not tested it by deploying the patch onto a running system. Maybe you saw something there?
Yeah I think that it was/is a hostname thing
But i didn't play with it enough to be sure. I just installed from ISO to KVM and all was good, so I've just been using it since. And actually, now I see that you have just release v0.9 I think i must be using a more recent version than v0.3!? Perhaps it is v0.4 or even v0.5? (Doesn't have Headphones).
Anyway, I'll pull my finger out and give this a test drive. Been loving the one I have running! Definately interested to check out the MFP replacement! :)
BTW is there an easy way to transfer my existing SickBeard and CouchPotato settings (like queued movies/tv shows etc) across? I'm sure I could work it out, but any pointers off the top of your head?
I should really test/document migrating
Unfortunately, I am away from my system right now... but I think SickBeard, CouchPotato, and HeadPhones all keep a database (just a sqlite database) in the respective directories (/home/nzb/.sickbeard, /home/nzb/.couchpotato, /home/nzb/.headphones).
If I remember correctly, you can setup the new system and then copy old databes files over the new and your settings are retained. I would shut down the servers before copying the files over and then restarted afterwards.
That said... you're on your own here :)
If I were to do this, I would create a vSphere snapshot (or make a copy of the databases before overlaying them) of my machine before doing anything and revert my snapshot when/if it breaks.
Thanks heaps
Thanks for the pointers. I'll have a play, see how I go & post back.
Version 0.9 has been released
The iso and TKLpatch can be downloaded from: http://www.aceshome.com/
v0.9 is a winner! :)
Just thought I'd let you know that I have successfully patched the TKL Fileserver OVZ template with your v0.9 patch and it looks like it works a treat! I haven't yet migrated my settings etc from my old nzbapp server, but will have a crack at that in the next day or 2. Actually I now intend to move all my media/files from my existing fileserver into this too, so I will just have one consolidated appliance.
So nice work and I'll post back on my progress with transferring my settings across, thanks for the tips.
Only other thing of note that may be useful for others is that if you wish to customise the front page at all, hover over the top left corner and a little cog will appear (took me a little while to work this out so may save others a few moments of wondering...)
Thanks again! :)
Excellent!
I'm glad it worked for you. Let me know if you run across any issues.
Thank you
Good catch on the port number and for letting me know. I've modified the port in source and it'll be included in my next release
I use samba to mount a share
I use samba to mount a share from my NAS onto my appliance and it works well. If smbclient works for you then you should have no problems. Are you getting errors with the cifs mount or what?
New version coming soon
I am almost, about 99% done, updating my patch to woork with the new TKL appliances based upon Debian Squeeze. I've got a little more testing to do but should have a new version out soon.
Cool
Looking forward to it. I made a start myself and got it mostly running, but got sidetracked by other things so haven't finished... Can you please update the patch on GitHub too when you release? Thanks :)
my github should already be updated
I've already pushed my changes to the master branch on github, so you should be able to create the patch :)
BTW... the biggest change is that Maraschino now can update itself. It'll tell you when an update is available and ask you to update... just like sickbeard, headphones, and couchpotato. Very cool indeed
Awesome thanks Eric
Sounds very cool!
:)
Just like to say it's running sweet!
I applied the patch (from your git repo) to a fresh fileserver OVZ template and fired it up on Proxmox. Other than a few start up issues (nothing to do with your patch - the firstboot scripts shouldn't run in an OVZ template but forgot to fix that prior to running) it runs awesome!
Thanks again for such a great patch.
That's good news, thanks for
That's good news, thanks for testing it out. I've been meaning to test it more myself but had a little snafu when upgrading my vSphere instance from 4.0 to 5.1 a few weeks ago and have spent time fixing that instead.
I'm going to try to get the patch and iso made available early this week for anyone else interested.
Post processing of Sick Beard not working quite properly...
There seems to be an issue with the post processing for SickBeard (I haven't got that far in testing CouchPotato yet). The downloads are collecting in /srv/storage/downloads/completed/TV/<name>.<SnEn>.<other-info>/<name>.<SnEn>.<other-info>.<file-extension> I'm sure previously they went into /srv/storage/TV/<show-name>/Season<number>/<episode-files>. Unfortunately I lost my old server config due to harddrive failure (and I hadn't backed it up) so I can't confirm what the differences are. SickBeard happily found all my other (previous) downloads and added the shows (they were all on another HDD that I bind mount into the container).
Here is the error that SABnzb is throwing:
Loading config from /home/nzb/.sickbeard/autoProcessTV/autoProcessTV.cfg Traceback (most recent call last): File "/home/nzb/.sickbeard/autoProcessTV/sabToSickBeard.py", line 29, in <module> autoProcessTV.processEpisode(sys.argv[1], sys.argv[2]) File "/home/nzb/.sickbeard/autoProcessTV/autoProcessTV.py", line 56, in processEpisode config.readfp(fp) File "/usr/lib/python2.6/ConfigParser.py", line 305, in readfp self._read(fp, filename) File "/usr/lib/python2.6/ConfigParser.py", line 482, in _read raise MissingSectionHeaderError(fpname, lineno, line) ConfigParser.MissingSectionHeaderError: File contains no section headers. file: /home/nzb/.sickbeard/autoProcessTV/autoProcessTV.cfg, line: 1 'host=localhost\n'Or if there something I needed to configure first for this to work? I can't recall what i did last time when I set it all up so perhaps there is a step that I'm missing?
Update...
I checked the config example and added '[SickBeard]' to the first line. SABnazb now no longer complains. /home/nzb/.sickbeard/autoProcessTV/autoProcessTV.cfg now looks like this:
but the files are put in /srv/storage/tv/<name>/<SnEn>.<other-info>.<file-extension> rather than /srv/storage/tv/<name>/<Season number>/<SnEn>.<other-info>.<file-extension> which I'm sure it was doing before. But perhaps there's some config I need to do somewhere (in SickBeard) that I missed...? I'll have a look now.Doh! I just needed to adjust the SickBeard settings (Config>>Post Processing) to put the episodes into the correct folders...
I'll take a look anyway
I'll have a look anyway, once life gets out of the way, just to make sure everything is work as expected.
Yeah the autoProcessTV.cfg needs a little tweak
It just needs the [SickBeard] at the top of the /home/nzb/.sickbeard/autoProcessTV/autoProcessTV.cfg file. The other issue I was having was clearly a PEBKAC error! :)
Eric, Were you able to
Eric,
Were you able to integrate Sphinx search engine into the newznab ?
I did it folowing the howto links around the web and works fine
http://newznab.readthedocs.org/en/latest/guides/install_ubuntu-11.10/
For the sab,sickbeard,couch they can be proxied behind apache
I don't know about newznab behind apache It probably works too
I have not
I have not put sphinx on the newznab appliance. IMHO, it's not needed for a small newznab index. I'm glad to hear you got it working though, I'll take a look at that link.
Right now, sab, sickbeard, couchpotato, and headphones all sit behind a lighttpd instance on the nzbapp appliance. NewzNab site behind apache.
Try TurnKey Torrentserver appliance as a base
Eric hasn't updated this for a while so it probably would be best to just manually install the additional components on top. It would be really cool if you documented it though. Perhaps start a new thread. Then with the info you discover could become the basis of a new appliance.
Then we can get this (or something like it) into the official TurnKey library! :)
Fair enough...
I agree that it was a pity that Eric's work was never officially adopted by TurnKey. But in fairness this thread is ~3+ years old! Back then TurnKey didn't have the infrastructure to make it easier for the community to contribute appliance code. But all that has changed now...
Also I'm not totally clear on what you mean by "all this work to make such a simple appliance". For a single person, creating/updating a single appliance is IMO not a massive amount of work - at least not much more than installing it yourself. Once you understand how our build infrastructure (TKLDev) works it's not that hard IMO. And we are always happy to help newbs that are interested learn how to use it. I would be willing to personally mentor you if you were interested.
I have long had plans to update/rejig Eric's work and create an appliance myself but I have so much on my plate that it keeps getting pushed back. Just supporting and updating the appliances we have is a full time job; let alone developing new ones (and developing better ways of creating them in the first place...). I have (almost) completed the v14.0 appliance release of our current library (about 100 appliances). It's still not quite finished and I didn't do it alone; but I have personally spent in excess of 700 hours on it to date! And this is for something that you (or anyone) can download for free!
As for the modularity issue I agree that it would be nice to have a better, more modular way to add and remove server functionality. It is certainly something the core devs (and leading community members) are very keen to have in place. But the reality is that we are a small team and way more good ideas that we have resources to fulfil. Currently our build tools and infrastructure aren't perfect but they are robust and they work. IMO the learning curve isn't too steep either...
Obviously I'm biased but IMO we have an awesome community here. Surely you can't think that it's legitimate to criticise our community and all the hard work they do because they don't work on the thing that you want!?
Don't get me wrong, we love to get feedback even if it's not flattering. But I think that your comments are overly harsh and not very respectful! If you have found another project that fulfils your needs then that's great. If you want to tell us why you're going elsewhere then that's fine too; but please keep your criticism respectful.
I think this is essentially dead
TBH I think that starting again may be the best plan. Eric's work could be a useful starting point and/or bit of a guide but a lot has changed since this was released...
Re your questions:
If you want to have a crack at updating/rebuilding this and releasing as an official turnkey appliance I am more that happy to provide some guidance!
I am still around. Kind of...
I know it's been a while since I posted but I do occasioanlly pop in here and see what's going on. I've been doing a lot more work with Docker and PaaS layers like Rancher than with virtual machines so my involvement here has waned. A lot.
I am going to try and update the patch to newer versions of software and sickrage (instead of sickbeard) but have no plans to add mylar.
IMHO, if you want plugable services then you want to use containers.
Hey Eric! :)
We anticipate that our next major release will leverage docker style containers. FWIW some of the work has already started. We've called the new appliance model TKLX.
Currently there isn't a lot to see other than a GitHub repo with a few containers. Although IMO Alon did a pretty good job getting our (Debian based) base container down to 12MB! (compressed).
Unfortunately, we've had to put that aside to tidy some other stuff up, but we hope to get back to it soon.
The plan at this point is to probably have some overlap between our existing monolithic appliances and the new container based ones. We plan to provide a host OS (essentially like Core but with container support built in) and then all other components can be added as containers. Ideally, we'd also like to provide some sort of OS agnostic agent to manage it all so users wouldn't even need a TurnKey host OS. E.g. you could run TurnKey in Docker or CoreOS etc. Perhaps directly within OSX and Windows even (I hear that Windows now supports Docker!?).
How do your plans compare to...
Interesting, but how does this compare to things like Kubernetes, Mesos, Rancher, Swarm, etc?
I would skip the host OS and the agnostic agent unless you see that no other PaaS layers are viable (which I find hard to believe). Why is CoreOS not good enough (or Photon OS or RedHat Atomic or Debian or CentOS, fot that matter).
I'd like to see turnkey do a few things:
- Provide a REALLY good base image. Something like Phusion/baseimage-docker with a clean and simple to use init system. I feel that Phusion/baseimage-docker is really good but I'd like to see versions based off various Ubuntu versions (14.04 and 16.04 for starters)
- Provide a nice library of containers and 'wrappers' around them. An exmaple of a 'wrapper' would be something like a Rancher catalog, where TKL has done the work of composing the entries. For a reading of what I have done with Rancher, take a look at http://www.aceshome.com
Thanks for the feedback Eric
We've decided to stick with Debian as a base, primarily for stability and security. However additional alternatives are not completely off the table. I hadn't come across Phusion/baseimage-docker before, but it looks like they've done a pretty good job. FWIW our existing base image can be found on GitHub and/or Docker Hub.
AFAIK Kubernetes integration is on the cards. Rancher, I hadn't heard of before but it looks cool. The others I'm not sure about at all (I've heard of them, but not sure how/if they fit into the plans). Thanks for pointing me to your blog. I certainly ping Alon with a link to that!
Part of our issue is that we want to transition in such a way that we don't leave our existing userbase stranded. Many users have come to rely on TurnKey and we want to take them with us when we move. So providing a base host OS (probably just a tweaked version of our existing Core appliance with Docker pre-installed) seems like a must. OTOH we want to broaden the appeal of TurnKey to new users (who want containers, not monolithic appliances). So we want to ensure that we can support both use cases.
The idea is that for users who just want a new "WordPress server" (for example) can still just download a TurnKey iso/ova/whatever and away they go. They don't need to understand how the container components slot together under the hood. But other users who just want a "WordPress docker container" can just use our container as part of their existing workflow and on their existing OS.
The idea of the agent is somewhat multipurpose. Although AFAIK part of the rationale is so we can still support some sort of TKLBAMish type backup and migration functionality, outside of the containers themselves. Ideally that will support at least migration from existing TKLBAM backups to TKLX. But perhaps more...
Sorry I can't be a bit more informative and/or specific. Thanks tons for your feedback though!
Patch is updated
Updated the patch to work with turnkey 14.1, commit id of f991077
Note that a final reboot is needed after setting up the services, as couchpotato fails to start on the first attempt
https://github.com/tssgery/nzbapp
Awesome! Thanks!
Add new comment