TurnKey Linux Virtual Appliance Library

tklpatch for SABnzbd, CouchPotato, and SickBeard

Eric (tssgery)'s picture

I wanted to play with writing a patch, so I started playing with creating one for SABnzbd and a few addons. I've got the patch working but need to do a little more testing on it before releasing it into the wild. This testing should be done in the next day or two.

For those unaware, SABnzbd is an Open Source Binary Newsreader written in Python.

SABnzbd makes Usenet as simple and streamlined as possible by automating everything we can. All you have to do is add an .nzb. SABnzbd takes over from there, where it will be automatically downloaded, verified, repaired, extracted and filed away with zero human interaction.

There are two very popular addons, SickBeard and CouchPotato which help automate the rerieval of nzb files for TV Shows and Movies (respectively).

I've created this patch mostly for myself but thought I'd query here to see if there is a general interest in this or not. If you're interested, respond and I'll put the patch somewhere people can get to it.

Jeremy's picture

Sounds awesome!

I for one will be very interested in this!

Out of interest, what are you basing it on? Are you just using Core? Or are you using the Fileserver appliance (IMO that would be the best one). Also it may even be worth considering the Torrent server appliance? An AIO torrent/nzb/fileserver appliance!? Despite how cool that sounds, I guess I'd probably be inclined to stick with the fileserver appliance and consider the AIO server separately... Just my 2c though! :)

Eric (tssgery)'s picture

My work so far is based upon

My work so far is based upon the LAMP stack, as I'll eventually want a mysql and apache2 instance for a newznab instance. That said, I'm not doing anything yet that requires more than core so maybe I'll look at the fileserver appliance as well.

 


Jeremy's picture

It is probably easier to add LAMP functionality

Than to install eXtplorer etc. Although it wouldn't be unsurmountable. I'm going to be away for a few weeks but when I get back I'm happy to chip in my 2c on your progress if you'd like some feedback.

Eric (tssgery)'s picture

Almost there...

I just took a look at fileserver, and it does have some nifty features. I migrated to use it as the base and now have the system mostly setup (all services startup correctly and configure themselves as best as possible, accounts/passwords are created and set, I've sucessfully been able to download several binaries, and retrieve/view them with eXtplorer).I've got a little more testing to do as well as some code cleanup, but I'm getting very close. Next steps:

  1. Testing
  2. Pull as much configuration data as possible from SABnzbd into SickBeard and CochPotato (they all use similar entries for search engines)
  3. Testing
  4. Documentation
  5. Ensure SABnzbd, CouchPotato, and SickBeard are up to date at firstboot time
  6. Addition of Newznab, or possibly placing it in it's own tklpatch.
  7. Addition of Headphones, when I feel it's stable enough to do so.

Jeremy's picture

Brilliant!

Sounds like this is going to be a killer patch/appliance! And hopefully it'll make it into the part 2 of the v11.x release ie become an official appliance.

I'm leaving tomorrow and won't be back for a few weeks but I'll forward to this when I get back!

Eric (tssgery)'s picture

Just a brief update... I

Just a brief update...

I have just released version 0.1 of this tklpatch and iso image (based upon fileserver). They're both available on my webserver at http://www.aceshome.com

BIG NOTE: Pulling a lot of usenet articles down will consume a lot of disk space. It's best to either create an LVM (so you can add disk later) or mount /srv/storage on a large disk. I installed on a 120GB virtual disk and then copy files off when I want to keep them.

Once the appliance has been deployed, you must configure a few things:

The major things to configure are your usenet credentials and search providers.

Note that to use this appliance, you should have the following:

  • premium news hosting service (such as giganews). If you'd like a free 14 day trial, you can get one by clicking here
  • An account or access to a usenet indexing site, such as nzbmatrix or nzb.su. Alternatively, you can setup a newznab instance yourself (there is a soon to be forthcoming tklpatch for this as well).

Eric (tssgery)'s picture

I just released version 0.2,

I just released version 0.2, after adding MediaFrontPage to it.


Eric (tssgery)'s picture

Version 0.3 is up now (well,

Version 0.3 is up now (well, it has been for a few days). I've seen about 15 downloads of the patch and 15 of the iso, if you have taken a look and have some feedback... I am all ears.

 

Get it at http://www.aceshome.com


Liraz Siri's picture

This looks friggin awesome!

Thanks for taking the time to put a TKLPatch for this together. To be perfectly honest I hadn't previously heard of any of these projects, but I've been looking for something like this forever now. It looks awesome! This is the sort of totally unexpected, excellent contribution we had in mind when we developed TKLPatch.

Please post the TKLPatch somewhere so we can add this officially to the next release of the TurnKey library. If you can attach it to your forum post that would be great.

Eric (tssgery)'s picture

I have been putting the

I have been putting the source to my tklpatches and source on github at https://github.com/tssgery

There are currently 4 of them available:

  • nzbapp - SABnzbd/CouchPotato/Sickbeard/MediaFrontPage
  • newznabapp - preinstalled newznab unsenet indexing engine
  • mcapp - Minecraft/McMyAdmin Server :)
  • virtboxapp - VirtualBox/PHPVirtualBox (has not undergone much testing, yet)

Some of the prebuilt patches and iso images are on my personal website at http://www.aceshome.com


Jeremy's picture

Sounds like you've made great progress on this!

I've not been back long from my holiday, but I'll be having a much closer look at this one the moment I have time (I also have a lot of other projects to tidy up). Good work! :)

With Liraz's glowing endorsement above you can be quite sure that your patch will make it into the v11.x part 2 release (when it finally sees the light of day - hopefully not too long now...)!

Jeremy's picture

Bit of feedback

I have just installed it and nice touch with the setting of password on first boot. Good work! (I must admit that I have just had default passwords in the patches I have made - hoping that the core devs will fix that for me when they create the official appliance).

I have yet to do anything much with it yet but the first thing I'd like to note is a slight mistake in your original docs (SABnzbd seems to be running on 8080 - eXtplorer is on 80).

Also I'm not quite sure what's going on. I haven't done much troubleshooting yet but it seems that SickBeard and CouchPotato aren't running (or at least aren't listening). SABnzbd is working fine. I did a quick netstat -na |grep LISTEN and there is only the default Fileserver stuff + SABnzbd. I'll do some more testing and/or have a look at the patch and get back to you.

Finally I'd like to suggest customising the default services.txt for confconsole with these details:

Here is one I just threw together (it's a very small thing but makes it a little more finished). It needs to go in /etc/confconsole/services.txt IMO it's best put in the overlay (rather than mucking around with sed).

SABnzbd:      http://$ipaddr
CouchPotato:  http://$ipaddr:8085
SickBeard:    http://$ipaddr:8081
File manager: https://$ipaddr
Web shell:    https://$ipaddr:12320
Webmin:       https://$ipaddr:12321
SMB/CIFS:     \\$ipaddr (ports 139/445)
SSH/SFTP:     root@$ipaddr (port 22)

Also I think it's a good idea to have all the interfaces available via https. Obviously that won't be an issue when run locally although perhaps users may wish to access it remotely and IMO it's definately an issue if run in the cloud.

[update] Just checking your patch and it seems that sed is meant to be adding to services.txt - not sure why it's not working. I'll investigate further. Obviously there are a few things not woring as they should...

[another update] I just checked my TKLPatch log and it seems that none of the commands after installing SABnzbd ran!? So SickBeard and CouchPotato aren't even installed! Not sure what's going on there...

Eric (tssgery)'s picture

I'll check when I get home on

I'll check when I get home on Sunday but suspect it's something trivial. I did most of my testing via installing the patched iso and assume you actually installed fileserver and then tried patching the running VM?


Jeremy's picture

Thanks

I'm at work now so I won't get a chance to play any more until I get home. After a little more investigation last night it seemed that the other apps where installed (despite the fact that my log didn't show it - not sure why...?). The big issue seems to be that the customised LigHTTPd conf file wasn't copied across, but even when I did it manually it still didn't work. I think I'll download your ISO and install from that so I can see how it's meant to be. Then I can compare and see why the patch didn't stick.

BTW I patched a vanila TKL Fileserver ISO using TKLPatch running on TKL Core (rather than installing in a running instance of TKL Fileserver).

Eric (tssgery)'s picture

If you just tklpatched a

If you just tklpatched a filserver iso and then installed a VM from the new patched iso, that's almost exactly what I do (I patch the iso on a LAMP server instead of core). I'll look this evening, US time, and see what I find. If you can put your iso somewhere that I can get it, I can diff the contents of yours and mine and get a clue or two. That'd be easier than you looking and telling me what you find.


Jeremy's picture

Almost there!

I've downloaded your ISO And it's definately closer (and different - it's nearly 20MB bigger for starters). But it's still not quite working as it should.

The confconsole screen doesn't display all the info (no mention of SABnzbd, SickBeard or CouchPotato - I worked out why that was, see below) and trying to log in via the web browser (using Firefox 7.0.1 under Bodhi Linux 1.2.0 - based on Ubuntu 10.04 but with lots of updated stuff) causes some strange behaviour. When I type the full url/IP into the address bar (ie http://192.168.1.103/) and hit go, first the http and slashes disapear (leaving just 192.168.1.103)  then it brings up a search for 'nzbapp/mfp' (currently my local DNS server forwards DNS requests to OpenDNS - and when it receives a request that it thinks isn't a url it does a OpenDNS web search - see guide.opendns.com/main).

http://192.168.1.103/extplorer/ loads extplorer ok and when I use port 8085 it redirects successfully to http://192.168.1.103:8085/movie/ (CouchPotato) 8080 brings up SABnzbd wizard (http://192.168.1.103:8080/wizard/) and 8081 brings up SickBeard (http://192.168.1.103:8081/home/).

But now I have just found MediaFrontPage at http://192.168.1.103/mfp/ and it all becomes clear why you don't mention SABnzd, SickBeard or CouchPotato on the confconsole screen! Wow! That is seriously cool! I like all those cool widget things for the different services they're awesome!

Only thing is that eXplorer is the only one that works :( The other tabs cause similar OpenDNS search behaviour as for the IP address mentioned above: searches for nzbapp:<port> - except they're in a frame of MFP. I guess that if I had a local DNS entry for nzbapp all would be well, but you shouldn't need that IMO.

Also during the time it's taken to write this post (testing as I write) my browser has crashed 3 times (thankfully I've got the Lazaras addon so I don't have to keep typing!) I only updated a few days ago so perhaps it's coincidental but it's the first time I've had any issues.

Sorry to sound so critical. I think it's a fantastic idea and you've obviously put a lot of thought and work into it so good job! :)

Eric (tssgery)'s picture

Seriously strange

What you describe is seriously strange. I've deployed that iso about 100 times and not experienced this, but I suspect it is dns related. I use OpenDNS myself and don't suspect that but instead think I have some flawed logic that I assume your hostnames are resolvable (for example, I use dnsmasque and have configured it to resolve 'nzbapp' to the static IP I setup the VM with. Is your VM statically or dynamically addressed? I find it hard to believe that anything in this could cause a browser crash but if you send me the specifics of the browser version and plugins you have, I'll try and recreate it.


Eric (tssgery)'s picture

You can try...

... Adding an entry for 'nzbapp' into yout /etc/hosts file and see what happens. That would validate if it's a dns problem or something else.


Eric (tssgery)'s picture

just tested it a bit, and...

you should add 'nzbapp' to your hosts file and set it to resolve to the IP assigned to the appliance. There are several places where the configuration files need to point to applications and the only reliable way is by setting the hostname. If your dhcp server updates dns with the hostname you may be able to get by without the hosts file change but I assume that not all routers are configured that way.

 

I should have checked this before releasing the patch but I always set my servers to static addressing so didn't catch it.


Jeremy's picture

Don't sweat it :)

Its all good. It's hardly mission critical stuff and I think you've done an awesome job with this appliance! I still need to work out why the patch didn't stick when I did it myself but I'll leave that for another day...

Out of interest, first I tried adding an entry to my local (Linux) DNS and that didn't work either (it seems to just return a FQDN which doesn't work).

It works now (via editing the hosts file), but I think it's something worth further consideration. I imagine most users will want to use this appliance locally at home in which case a hosts edit isn't that big a deal, but what if a user wants to access it remotely (or even just have a play with it in the cloud)? What if a user is using a device that doesn't easily allow that level of configuration (no access to root or admin account/game console/tablet/smart phone/etc?) IMO needing to add to the hosts file is a little clunky and ultimately may be a showstopper for some.

[update] I just realised that I should only have to update the hostname (to a FQDN) to get it to work with my DNS shouldn't I!? I haven't tried yet but I reckon that'll fix it!

Eric (tssgery)'s picture

you'll still need to be able to resolve the full hostname

I have made some assumptions in my appliance that the hostname is 'nzbapp', if you change the hostname (via editing /etc/hostname and such), you'll need to change configuration files as well.

What I should probably do is to prompt the user for the hostname during deployment and use that for configuration. That way, if they deploy to amazon they can use the real resovable name.


Jeremy's picture

I think that's a great idea

If you create a inithook script and set it to run on first boot that would be a winner IMO. I'm assuming that's how you ask the password question on install?

If you were to go that way then users could choose to use a simple hostname, a FQDN or even just an IP. The beauty of an inithook script too is that if they change their mind they can just manually relaunch the script, enter new details and everything would be updated.

Eric (tssgery)'s picture

I've got it asking for a hostname

I'm asking for a hostname now and setting the config files properly. Re-running the inithook script sets the value properly. One thing I would advise against though, is setting the hostname to be the IP address. In a DHCP environment, the IP address can change at any time (yes, the DHCP server will almost always re-assign the same IP but it's not guaranteed). This will lead to all sorts of confusion on the behalf of users.

I am going to take a look at your patch issue next.


Jeremy's picture

Awesome work

Good job! :)

As for DHCP & setting hostname as IP: agreed! Because I always set static IPs I forget about the idiosyncracies of DHCP. So yes you are right, setting the hostname as IP and not setting a static IP is a very bad plan!

Eric (tssgery)'s picture

patch issue

I found one issue with the patch. I was renaming the patch file for redistribution (adding the release number to the patch name) so it was easier to identify what version of the patch it is. When running tklpatch against this file it couldn't probperly extract and apply the patch, throwing some errors and failing to create the patched iso file. I am not sure this is what you saw but I will upload a new nzbapp.tar.gz patch later today.

I took my new patch and have applied it to the fileserver base on a core appliance without issue.

If you see this error again, please capture the stdout/stderr of the tklpatch command and let me know.


Jeremy's picture

I don't think that was my problem

Because when that happens it basically just ends up with an unchanged ISO. So still not sure why it didn't work for me and unfortunately I haven't had time to have a look yet (too much work and too many other projects on the go). As I think I said earlier it may be my patching environment. I have a installation of Core that I use for patching, perhaps it needs a refresh!?

As for the TKLPatch issue you speak of, when I create a TKLPatch I find the best way to avoid that easily made mistake is instead of using tar to create the patch.tar.gz I just use the tklpatch-bundle command (it basically does the same thing - just avoids the risk of that mistake).

I think the idea of including a version number in the patch name is a good one. It allows users to see at a glance whether they are using the latest version.

Eric (tssgery)'s picture

version 0.4 is uploaded to

version 0.4 is uploaded to http://www.aceshome.com

 

Please note that the only changes are:

  • allowing the user to specify a hostname during installation
  • fix to rename the nzbapp.tar.gz patch to allow proper patching of the fileserver iso

Jeremy's picture

Excellent work - you rock! :)

Nice one. I'll download and test sometime when I get a chance. Got some other stuff I need to get onto over the next few days but I'll definately be back to check this out!

I think that this is a really good candidate for release as an official TKL appliance! :)

Jeremy's picture

Some more feedback...

Hi there. I'm having another look at your NZBapp. I must say I really like it, although I admit, I'm still yet to take it for a proper road test yet.

I was much more successful patching the Fileserver ISO this time. Not quite sure what happened last time... I haven't yet actually set it up to download any NZBs so I am only in the early stages of testing really but thought I'd post what I've found so far (and some questions too). :)

I now have an instance running on KVM on my PVE server and one on OVZ too. What sort of specs do you recommend for this? Here's what I've found so far (I have other VMs idling too so not really scientific...)

My KVM instance has 1GB RAM and 2 cores allocated and is running quite well. I initially started with 512MB and one core and that was not much good at all. It was really laggy. The RAM boost (which I did first) noticably helped but enabling the second core was what made the big difference. I did try increasing RAM some more, but that made no difference (no surprise really as it's only using 530MB at idle).

OTOH it's running under OVZ much nicer. I has only one core allocated and 512MB RAM (and is using ~60MB swap too - strange that the OVZ version is using more RAM) and is still more responsive than the KVM instance. I guess that more comes down to the idiosyncracies of the virtualisation tech though.

One question for you... Is there a good reason why you don't just make /mfp the web root? On both instances it takes noticably longer to load the MFP front page and settings tab (in comparison to SABnzb, SickBeard and CouchPotato - those 3 are quite snappy) and I can't help but think it's the redirects. What do you think?

One strange thing about the OVZ instance too is that you have to manually point it to /mfp or otherwise it redirects to OpenDNS search (and it searches for 'nzbapp/mfp'). eg nzbovz.home.local redirects to OpenDNS (despite the fact I have the DNS working now) but nzbovz.home.local/mfp works fine. I'm assuming that there's a setting somewhere that's not applying from the first boot scripts. (I made the OVZ image from the ISO that I patched).

One other little thing... The confconsole page is really hard to read. It seems that it doesn't have the correct line breaks, so the lines are all just jumbled up together. Out of interest, is it like that on your ISO too? IMO it's not worth redoing your patch just for that. Personally I prefer to just include the custom services.txt in the patch overlay, rather than writing to it with sed. In my experience using overlay files is preferred whenever possible as it seems to be much more reliable than sed (although obviously sometimes you have to use sed). Although I'm not good with sed anyway so I have an aversion to using it (regular expressions do my head in!!)

Eric (tssgery)'s picture

Thanks for the

Thanks for the feedback.

Lot's of comments/questions, let me try to address them.

Firstly, I'm glad that your patching worked better this time. :)

As far as resources, I run VMware ESXi 4.1 and have devoted 1vCPU and 4GB of memory to it. I don't have great numbers for what memory I am actually using though. It should run pretty small while not downloading any content but will definitely spike while downloading. 1vCPU and 1GB should be sufficient for most uses. As a note, SABnzbd is the heaviest user of CPU/memory and many people run it on embedded systems so they have taken a lot of pain to make it resource light. I just happen to have 16GB of memory in that ESXi server so gave it a lot.

The redirect to /mfp for the webroot does not take a lot of time. It's the actually loading of MediaFrontPage that takes a while. Firstly, let me say that I am not 100% happy to have used mfp. I like the functionality but have problems with the way it is designed and configured (it is the best thing I have found though, and I think it will continue to improve). What mfp is doing upon load is contacting SABnzbd to get a list of downloads occurring as well as pulling down rss feed content and such. Simply put, this takes a while. I can go more into why it takes a while but I will leave that for another day :)

The strangeness within DNS you are seeing is due to a couple of things: 1) The way mfp is configured, it needs to know the ip address/hostname of the SAB, sickbeard, and couchpotato instances so it can pass them to the browser running the mfp client. 2) The way my patch was working was to assume the hostname was 'nzbapp'. I placed the 'nzbapp' hostname in the configuration files that mfp needs (/var/www/mfp/config.ini I believe). In version 0.4 of the patch, I changed that logic and now ask the user during deployment via iso what their hostname should be. I then use that value in the mfp configuration. BTW, version 0.4 is on www.aceshome.com

My confconsole looks correct. I switched from sed to just echoing the lines out for v0.3 of my patch (I'd rather create the service.txt file from the conf script than an overlay). Which version of the patch did you apply?

BTW, I will probably release another version of the nzbapp within the next few days. There have been some changes to mfp that I'd like include as they fix problems I have seen (but are not mentioned here).


Eric (tssgery)'s picture

I take that last comment

I take that last comment back... my confconsole does seem to have lost the line feeds :( Guess now I have a real reason to release v0.5

Edit: After looking into this, it appears that confconsole does not like the occurrence of a '\n' string in /etc/confconsole/services.txt. because our hostnames both started with an 'n', I was echoing the SMB connection string as "\\hostname" and causing the problem.

I have tried every possible thing I can think of to escape the backslashes properly and have not found one that works. I'll keep looking in my spare time but for now will eliminate the "\" character from the file.


Jeremy's picture

Even more feedback...

I've had a bit more of a play and all I can say is that this is AWESOME!

This would have to be the coolest thing ever! You have done an excellent job with this and setup is so easy.

I look forward to playing some more. I think I'll have to set myself up with a XBMC-PC now (it's been on the cards for a while anyway...)

Can't thank you enough!

Eric (tssgery)'s picture

Glad you like it

I'm working on one last version (fixing the confconsole text and moving to a newer MediaFrontPage), and once that is done... I'll call this a "wrap". It started as a learning experience with TKL and since I don't use this software at home... it's time to move onto something else :)


Jeremy's picture

Sounds good

So you going to do another TKLpatch? If so, looking forward to what you next come up with!

Eric (tssgery)'s picture

version 0.5 available

This is likely the last release of this from me and addresses a couple of issues:

  • The console text was somewhat garbled dependent upon the hostname chosen during installation.
  • I have moved back to DejaVu’s MediaFrontPage repository instead of my own fork. My fork will be removed/deleted at some point in the near future.
  • Minor change to the /etc/hosts file

patch and iso can be downlaoded from www.aceshome.com


Eric (tssgery)'s picture

Version 0.6 available

One more release...

I have added the HeadPhones addon which let's you search for music.

 

Version 0.6 is available at http://www.aceshome.com


ports

Could we get a list of all ports that need to be opened for access?

Jeremy's picture

Check which ports are being listened on

To do that, from the commandline (using either SSH or WebShell) use this command:

netstat -plut

The meaning of the switches:

-p Tell me the name of the program and it's PID
-l  that is listening
-u on UDP ports
-t and TCP ports

But OTTOMH it's 80, 8080, 8081 & 8085 for the webapps plus 12320 (WebShell) and 12321 (Webmin). Also if you want to use the Samba fileshares you'll need ports 139 and 445.

Eric (tssgery)'s picture

Thanks Jeremy.   version

Thanks Jeremy.

 

version 0.6 also added HeadPhones which sits on port 8181, so the complete list for v0.6 is:

  • 80, 8080, 8081, 8085, 8181 for the webapps
  • 12320 (WebShell) and 12321 (Webmin).
  • Also if you want to use the Samba fileshares you'll need ports 139 and 445.
  • SSH is on 22

 

This does bring up a good point though... I DID NOT set iptables up to start automatically nor did I configure the rules to allow these ports. As with anything other system, I would make sure that you either configure/start iptables OR put this appliance behind another firewall. If you are not familiar with iptables, the easiest way to configure it may be to use webmin.

You can, of course, change the webapp ports if you would like. You would need to change the webapp configuration AND the lighttpd config file.


mediafrontpage very slow to load......

Hi,

First, great project!  I am running this currently on a virtualbox machine and everything seems to run more or less as expected in the vbox environment (I am defining a 1Gb machine with 2cpu for the vbox, and the host machine is an i7 with 6Gb running Windows 7).  I have even mounted some windows shares into the /srv paths so that I can download from sabnzb straight to my windows machine's physical disks.  All works fine, but....

MFP is VERY slow, can take circa 10+ seconds to load the widgets home page (and the settings page).  The links that open up the other apps (for sab, couch etc) are all very responsive and appear virtually instantly.

The mfp system appears to be configured correctly, i.e. the expected and correct things do get displayed in the widget information, but it just takes a long time for the widgets to appear (NB: the page actually stays blank/slate coloured for circa 10s+, then the widgets appear quite rapidly).

Any ideas?  Any other info you might need to try and solve this?

Thanks

Eric (tssgery)'s picture

MFP isn't perfect

As you have found out, MFP isn't perfect. I does take some time to load and query the other services as to their status.

On my configuration (the VM has 1 vCPU and 4GB memory), most of MFP comes up within a second but the "System Info" and "SABnzbd queue" take between 5 and 10 seconds as well. I wish I could offer you some black magic that solves this but I can't.

One more comment: Development on MFP has all but stopped. The developers of MFP are working on Maraschino now, which I plan to look at this week to see if it's a better solution than MFP.


Jeremy's picture

Wow Maraschino looks really nice! :)

And I notice that Maraschino is written in Python... So may be a little more responsive? It still looks like a pretty young project though (only just released v0.1 ~a month ago).

Anyway, glad we got you here mate, with your finger on the pulse with this media type stuff! Regardless of the imperfections of MFP I still think this would have to be one of my favourite appliances ATM! :)

I'll be interested to see/hear how you go with Maraschino.

Jeremy's picture

Yeah it's a little laggy especially on first load

As I said above it works pretty nice for me with the same allowance as you - although my environment is a little different: running under KVM on headless Linux (PVE v1.9) 'server' (actually an old E6300/2.3GHz C2D desktop with GPU removed and RAM boosted to 8GB) rather than in VBox under Win. Still I would've thought that your gruntier CPU would've been a noticable improvement.

Mine still lags a bit on first access (although not as bad as you are reporting) but is noticably faster if I go back to the main or settings screens (perhaps because of browser caching?) Actually IIRC the widgets themselves still take a little while to load (but the rest of the page, the tabs etc all load pretty quick).

I have a few ideas on tweaks you could try. I'm not sure what (if any) tweaks tssgery has done to the standard TKL LAMP appliance in his patch/ISO (and haven't got access to my server ATM to check) but some or all of the following are perhaps worth a try? Be great if you could keep track of what you do and post back with results. Hopefully this will help tweak this for default optimal performance (for when it becomes an official TKL appliance (as I hope it does).

So here goes a few ideas...:

[update] tssgerry has already done this, see post below.
Enable image caching (from MFP website) - It seems that it's just meant to cache images (so no idea if it will help) and there is also an unresolved bug listed against it, so not even sure if it works at all...?

Optional:

  1. Speed up image loading times.
  • Create a folder named sbpcache
    			sudo mkdir /var/www/sbpcache
  • Give MFP write permissions to the cache folder
    			sudo chmod 777 /var/www/spbcache

Increase php memory allocation - AFAIK MFP is written in php so this may help. The config file should be found /etc/php5/apache2/php.ini and the item to change the vaue of is memory_limit, OTTOMH the default value is 32MB (in a standard TKL LAMP appliance), try boosting that up a bit. Maybe try 64MB and see if that helps, and if not enough, keep going - although if you haven't noticed any improvement by 128MB its probably not going to do it.

Apache caching - It's not something I've really played with but could be worth a shot. Unfortunately I can't offer you any guidance there so you're on your own (but there should be plenty of info online about that).

A final word - There are probably lots of other php and Apache tweaks that could be worth a try but it'll be a case of step-by-step trial and error; search then test. If you're googling keep in mind that TKL v11.x is based on Ubuntu 10.04 so anything that applies to Lucid Server (Ubuntu 10.04) should apply to TKL v11.x. Just a word of warning with googling Apache related info, that there are some idiosyncracies to the Debian/Ubuntu versions of Apache (so many instructions for other versions of Linux probably won't apply). For most tweaks you'll need to restart Apache before any changes will take effect:

service apache2 restart

Good luck and hope to hear back from you! :)

Eric (tssgery)'s picture

sbpcache is already enabled

Thanks for the ideas.

Here's my 2 cents... I have already enabled the sbpcache in the patch and I really doubt (99.9% certainty) that caching (this or apache) is not the problem here. 

Tuning php would be interesting to try, though I think even this would yield very minor gains. Php is not a great performing language and it's used a lot in MFP but the amount of processing is still very low.

The way MFP works is to have a gadget either use the SABnzbd, Sickbeard, or CouchPotato API or screen-scrape to pull the data. I suspect it's the communication between these components an dprocessing of the responses that's causing the delays in loading some of the widgets. There are likely several ways to improve this by going into MFP and fixing some of the architectual and coding issues. I had started to do this but life got in the way and I've been unable to spend any time on it Given that MFP seems to be dying off, I'd rather spend time on looking for alternatives.

All that said, if other people take a look and find some enhancements to make... please let me know or fork my github repo and make changes. I'd be happy to roll them in :)


Jeremy's picture

You're probably right re caching

I was just throwing ideas around. :)

As for php memory allocation, in my (limited) understanding the implications (and the potential for performance improvement) of increasing memory allocation depend a lot on how the php coding is done. Regardless it's probably worth a try. I might even give it a crack when I get home and see if it makes any difference).

And yes I agree, if MFP is a stalled/dying project (which it seems it is - after your last post I had a bit of a look and saw that there haven't been any commits for a while) you're better off putting your energy elsewhere. Besides I think Python is better anyway! :)

As I said, that other one you mentioned looks pretty sweet too so could be worth a crack.

Eric (tssgery)'s picture

Another version on the way

It's been a while since I released a version of this but wanted people to know that there is a new version in the works. Hopefully, I'll get it out sometime this week.

I have switched from MediaFrontPage to Maraschino and things are looking really good. Maraschino is much more actively developed now, works better, and is [IMHO] more visually appealing.


Jeremy's picture

Awesome!

I'll be keen to test it once you release it.

TBH I'm still using v0.3 but looking to update and consolidate this server with my old (TKL v2009.x - Ubuntu Hardy based) Fileserver anyway. Currently I run both (as separate VMs on my PVE server) but it seems crazy not to have them consolidated and I'm getting sick of having to either move files out of the NZB server to my Fileserver and/or keep increasing the disk size of my NZB server... Especially now that I have started using an XBMC HTPC setup, it would be much nicer to have the server backend set up a bit better.

The only reason why I haven't consolidated them sooner is for a few reasons. Firstly it took me a while to work out how to mount a separate physical HDD in an OVZ template (which was part of the process of migrating to a single server model - I didn't have room to copy my files from one VM to another) and secondly I discovered SSHFS (after unsuccessfully wrestling with NFS in OVZ containers under PVE). SSHFS makes it really easy to mount remote filesystems across Linux and for them to appear as local. Also the fact that I never successfully got your server to run under OVZ was a bit of a pain. I don't recall what the issue was but I couldn't get it to work. When you make your new release I'll have another bash at it.

Eric (tssgery)'s picture

Sounds good...

I have been able to make some headway today by totally ignoring everything else I need to do around the house so this week is looking good (possibly even later today).

I've personally not tried OVZ but can not image why this would not work as long as your name resolution is working properly. When using a front-end like MFP or Maraschino, it's really important that the hostname you specify  when deploying the image is resolvable by clients.

The other think I should mention is that I have ONLY tested this by installing from the ISO image. I have not tested it by deploying the patch onto a running system. Maybe you saw something there?

 


Jeremy's picture

Yeah I think that it was/is a hostname thing

But i didn't play with it enough to be sure. I just installed from ISO to KVM and all was good, so I've just been using it since. And actually, now I see that you have just release v0.9 I think i must be using a more recent version than v0.3!? Perhaps it is v0.4 or even v0.5? (Doesn't have Headphones).

Anyway, I'll pull my finger out and give this a test drive. Been loving the one I have running! Definately interested to check out the MFP replacement! :)

BTW is there an easy way to transfer my existing SickBeard and CouchPotato settings (like queued movies/tv shows etc) across? I'm sure I could work it out, but any pointers off the top of your head?

Eric (tssgery)'s picture

I should really test/document migrating

Unfortunately, I am away from my system right now... but I think SickBeard, CouchPotato, and HeadPhones all keep a database (just a sqlite database) in the respective directories (/home/nzb/.sickbeard, /home/nzb/.couchpotato, /home/nzb/.headphones).

 

If I remember correctly, you can setup the new system and then copy old databes files over the new and your settings are retained. I would shut down the servers before copying the files over and then restarted afterwards.

 

That said... you're on your own here :)

If I were to do this, I would create a vSphere snapshot (or make a copy of the databases before overlaying them) of my machine before doing anything and revert my snapshot when/if it breaks. 


Jeremy's picture

Thanks heaps

Thanks for the pointers. I'll have a play, see how I go & post back.

Eric (tssgery)'s picture

Version 0.9 has been released

The iso and TKLpatch can be downloaded from: http://www.aceshome.com/


Jeremy's picture

v0.9 is a winner! :)

Just thought I'd let you know that I have successfully patched the TKL Fileserver OVZ template with your v0.9 patch and it looks like it works a treat! I haven't yet migrated my settings etc from my old nzbapp server, but will have a crack at that in the next day or 2. Actually I now intend to move all my media/files from my existing fileserver into this too, so I will just have one consolidated appliance.

So nice work and I'll post back on my progress with transferring my settings across, thanks for the tips.

Only other thing of note that may be useful for others is that if you wish to customise the front page at all, hover over the top left corner and a little cog will appear (took me a little while to work this out so may save others a few moments of wondering...)

Thanks again! :)

Eric (tssgery)'s picture

Excellent!

I'm glad it worked for you. Let me know if you run across any issues.


Thanks!

Thanks for this, I found it most useful and am using it in my  MediaServer8 project.

One small issue I noted in that blog;

The port setting in /home/nzb/.sickbeard/autoProcessTV/autoPorcessTV.cfg is incorrect out of the box. I needed to change this from 8080 to 8081 to make the post-processing work properly.

Eric (tssgery)'s picture

Thank you

Good catch on the port number and for letting me know. I've modified the port in source and it'll be included in my next release


another implementation

Hey guys, I thought I'd share my twist on the above.

SABnzbd+ http://github.com/MercyfulFate/tkl-sabnzbd

Sickbeard http://github.com/MercyfulFate/tkl-sickbeard

Headphones http://github.com/MercyfulFate/tkl-headphones

CouchPotato http://github.com/MercyfulFate/tkl-couchpotato

PVR (Integrated the above) http://github.com/MercyfulFate/tkl-pvr

Some of it is based of tssgery's patch, but ive moved configuration into /etc/ and the appcode into /opt/.

logs in /var/log/* as standard.

 

Its all work in progress so feel free to add comments. This is for my own personal use, so its not gonna get gold star maintaince, but add an issue in github for anything that should be fixed(under the appropriate patch, pvr is just the integration of the others) and it will go onto my bucket list.

If you want anything added, add an issue to tkl-pvr.

MF

Shared AFP or SMB share on my Mac will not mount

I am trying to mount some shared from the host machine (Mac OS X 10.8) to /media/xxx, but I am not able to do so.  It seems AFP is not support out of the box and cifs just doesn't/  smbclient works fine, but being a linux noob I do not which to use for the best.

I am trying to mount these shares so that SABnzb can use the location to download and process files.  Sickbeard  and the like will then use another share to copy the post processed files into the relevant libraries and then update my Plex Server interface runnin on my mac.

Has anyone got any ideas?

Eric (tssgery)'s picture

I use samba to mount a share

I use samba to mount a share from my NAS onto my appliance and it works well. If smbclient works for you then you should have no problems. Are you getting errors with the cifs mount or what?


Eric (tssgery)'s picture

New version coming soon

I am almost, about 99% done, updating my patch to woork with the new TKL appliances based upon Debian Squeeze. I've got a little more testing to do but should have a new version out soon.


Jeremy's picture

Cool

Looking forward to it. I made a start myself and got it mostly running, but got sidetracked by other things so haven't finished... Can you please update the patch on GitHub too when you release? Thanks :) 

Eric (tssgery)'s picture

my github should already be updated

I've already pushed my changes to the master branch on github, so you should be able to create the patch :)

 

BTW... the biggest change is that Maraschino now can update itself. It'll tell you when an update is available and ask you to update... just like sickbeard, headphones, and couchpotato. Very cool indeed


Jeremy's picture

Awesome thanks Eric

Sounds very cool!

:)

Jeremy's picture

Just like to say it's running sweet!

I applied the patch (from your git repo) to a fresh fileserver OVZ template and fired it up on Proxmox. Other than a few start up issues (nothing to do with your patch - the firstboot scripts shouldn't run in an OVZ template but forgot to fix that prior to running) it runs awesome!

Thanks again for such a great patch.

Eric (tssgery)'s picture

That's good news, thanks for

That's good news, thanks for testing it out. I've been meaning to test it more myself but had a little snafu when upgrading my vSphere instance from 4.0 to 5.1 a few weeks ago and have spent time fixing that instead.

I'm going to try to get the patch and iso made available early this week for anyone else interested.


Jeremy's picture

Post processing of Sick Beard not working quite properly...

There seems to be an issue with the post processing for SickBeard (I haven't got that far in testing CouchPotato yet). The downloads are collecting in /srv/storage/downloads/completed/TV/<name>.<SnEn>.<other-info>/<name>.<SnEn>.<other-info>.<file-extension> I'm sure previously they went into /srv/storage/TV/<show-name>/Season<number>/<episode-files>. Unfortunately I lost my old server config due to harddrive failure (and I hadn't backed it up) so I can't confirm what the differences are. SickBeard happily found all my other (previous) downloads and added the shows (they were all on another HDD that I bind mount into the container).

Here is the error that SABnzb is throwing:

Loading config from /home/nzb/.sickbeard/autoProcessTV/autoProcessTV.cfg
Traceback (most recent call last):
  File "/home/nzb/.sickbeard/autoProcessTV/sabToSickBeard.py", line 29, in <module>
    autoProcessTV.processEpisode(sys.argv[1], sys.argv[2])
  File "/home/nzb/.sickbeard/autoProcessTV/autoProcessTV.py", line 56, in processEpisode
    config.readfp(fp)
  File "/usr/lib/python2.6/ConfigParser.py", line 305, in readfp
    self._read(fp, filename)
  File "/usr/lib/python2.6/ConfigParser.py", line 482, in _read
    raise MissingSectionHeaderError(fpname, lineno, line)
ConfigParser.MissingSectionHeaderError: File contains no section headers.
file: /home/nzb/.sickbeard/autoProcessTV/autoProcessTV.cfg, line: 1
'host=localhost\n'

Or if there something I needed to configure first for this to work? I can't recall what i did last time when I set it all up so perhaps there is a step that I'm missing?

Jeremy's picture

Update...

I checked the config example and added '[SickBeard]' to the first line. SABnazb now no longer complains. /home/nzb/.sickbeard/autoProcessTV/autoProcessTV.cfg now looks like this:

[SickBeard]
host=localhost
port=8081
username=
password=
web_root=

but the files are put in /srv/storage/tv/<name>/<SnEn>.<other-info>.<file-extension> rather than /srv/storage/tv/<name>/<Season number>/<SnEn>.<other-info>.<file-extension> which I'm sure it was doing before. But perhaps there's some config I need to do somewhere (in SickBeard) that I missed...? I'll have a look now.

Doh! I just needed to adjust the SickBeard settings (Config>>Post Processing) to put the episodes into the correct folders...

Eric (tssgery)'s picture

I'll take a look anyway

I'll have a look anyway, once life gets out of the way, just to make sure everything is work as expected.


Jeremy's picture

Yeah the autoProcessTV.cfg needs a little tweak

It just needs the [SickBeard] at the top of the /home/nzb/.sickbeard/autoProcessTV/autoProcessTV.cfg file. The other issue I was having was clearly a PEBKAC error! :)

Mihai Petre's picture

Eric,   Were you able to

Eric,

 

Were you able to integrate Sphinx search engine into the newznab ?

I did it folowing the howto links around the web and works fine

http://newznab.readthedocs.org/en/latest/guides/install_ubuntu-11.10/

For the sab,sickbeard,couch they can be proxied behind apache

I don't know about newznab behind apache It probably works too


Eric (tssgery)'s picture

I have not

I have not put sphinx on the newznab appliance. IMHO, it's not needed for a small newznab index. I'm glad to hear you got it working though, I'll take a look at that link.

Right now, sab, sickbeard, couchpotato, and headphones all sit behind a lighttpd instance on the nzbapp appliance. NewzNab site behind apache.


Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)