TurnKey Linux Virtual Appliance Library

wget-friendly URLs

When it comes to downloading large files such as the Turnkey VM or ISO image, I much prefer using wget for a couple of reasons:

1) Very often the computer on which I'm running the web browser is not the same as the one to which I want to download the file; very often that system is in an open ssh terminal on my desktop.
2) It has a very useful auto-restart feature in case of an interruption.

Would it be possible to provide friendlier URLs for downloads on this site? wget doesn't really like the redirect URLs that the TurnKey Linux website uses.

Thanks.

Liraz Siri's picture

You can wget images on sourceforge

I like using wget too, and in fact I use it all the time with the sourceforge links.

Like so:

wget http://downloads.sourceforge.net/project/turnkeylinux/turnkey-lamp/11.0r...
Also, we're going to be experimenting soon with hosting a few images directly on a CDN and paying for the bandwidth with advertising on the download page. If it pays for itself we'll move from sourceforge to hosting the images directly. A nice bonus is that the CDN is much faster than sourceforge. I downloaded an ISO image to one of our servers at 30MB/s!

Jeremy's picture

Is there some way you could still use SourceForge as a mirror?

Here in Australia most people have download quotas but my ISP (and a few others) mirror SourceForge so its quota free. Besides downloads from SourceForge max out my connection (possibly because its mirrored by my ISP) so faster is no use to me!

Liraz Siri's picture

We'll keep sourceforge as a mirror

I didn't realize there was a special situation with SourceForge in Australia. Don't worry about it, we'll keep SourceForge updated as a mirror as long as it's useful to you and maybe others in a similar situation.

We probably wouldn't want to drop SourceForge entirely anyhow as hosting all build types on the CDN would be expensive.

Please keep me informed

TKLDevEnv's tklpatch-getimage command does it's magic to list and download files from sourceforge. If there are plans for moving them, please drop me an email so I can prepare the code for the changes. Maybe in that time we could agree in a better way to get the images (but did I told you I was proud of my code in tklpatch-getimage? wink)

Look at this beauty (tklpatch-getimage --update):

echo "Updating image list, please wait" 
wget -q -O /tmp/imagelist http://sourceforge.net/projects/turnkeylinux/files/ 
cat /tmp/imagelist | sed -n 's|.*href="\(http://sourceforge.net/projects/turnkeylinux/files/.*\.iso/download\).*|\1|p' | sort | uniq > /usr/local/share/tklpatch/baseimagelist 
COUNT=`wc -l < /usr/local/share/tklpatch/baseimagelist` 
echo "Done. $COUNT images found." 
exit 0

Liraz Siri's picture

Publishing to sourceforge is no big deal

Don't worry guys, we're not going to delete anything from sourceforge so all the existing links will continue to work indefinitely. Also, transferring to sourceforge is pretty easy from "the cloud" so if there is demand for it we could continue to publish new releases there. It's no big deal.

BTW JedMeister, the CDN we are going to be experimenting with (e.g., SimpleCDN) has a presence in Sydney so that might work just as well for you.

Here is a test link:

wget http://e1.simplecdn.net/testbucket.turnkeylinux.org/turnkey-lamp-11.0rc-...

Jeremy's picture

Another idea for downloads

I perhaps should have started a new topic but thought I'd just do a quick post here as its somewhat relevant to discussion (although only vaguely relevant to the OP).

I've been thinking, what about having TKL images also available as torrents? I for one would be happy to seed TKL images and I'm sure there are plenty of other TKL users who would do likewise. This could be a low cost and alternative way to distribute TKL and another way that users could give back to the project (by seeding the TKL images).

What do you think?

Liraz Siri's picture

I'd be willing to experiment but I'm skeptical

Torrents work best for very popular files which hundreds/thousands of people will want to download simultaneously. A CDN served HTTP download will usually provide a better user experience in other cases. But it might be a good idea to give it a shot, maybe run a test case with just LAMP...

I gave torrents a shot with the Ampache material

I put the Ampache material we produced on a tracker and seeded for a month. I thought it was a grand idea and applauded me profusely. I did get one download over the course of the month, all told. Granted, I could have shopped around for trackers a little.


Jeremy's picture

Skeptisim duly noted

but I think the main problem with torrents is a sort of chicken-egg type scenario.

They don't work as an effective download option unless a lot of people are seeding it, but you won't get a lot of people seeding it unless there are quite a few downloading it. Hence your note on popularity Liraz.

If TKL were to have a tracker and regularly updated scrape available on the torrent download page so it would be obvious to downloaders the situation - that would avoid or at least reduce the potential for a poor user experience. I don't know but I would suspect that some may use it and support it just as a way of reducing load/costs to TKL, sort of like a way of giving back, by taking a little bit. As long as we could have at least a dozen or so consistent seeders it should be a relevant option (even if not the preferred one). I'm not sure how the tracker/quota thing all works, but if it gets off the ground, perhaps there is a way of giving karma for seeders? Not that I probably need too much more karma :)

I can provide at least 2 seed locations for starters (from home and from work) and it sounds like Rik may be able to have his arm twisted, and perhaps could convince his work to seed it too. Then if both you dev guys seed one each from home and one from work then we have 7 seeders already.

I can commit for sure.

I absolutely have the resources to seed. I guess I would have to learn how to mirror. But I'll commit what I've got. Only limitation I see is bandwidth.

Can't seed from work, but can have students contribute by seeding at home. Get the message across that if they already know about torrents, they know the wrong things; if they don't know, they'll learn legit use.


Jeremy's picture

Ahh yes... Students, excellent!

That's a great thought, getting the students involved. Yes I think it's fantastic to be able to teach them about the legitimate and legal use of technology. They could have a go at the TKL Torrent Server appliance and/or look at how it could be tweaked/improved etc or even look at alternatives to how it could be done. Lots of great learning opportunities!

Ok then, I'll have to grab the bull by the horns then and get a bit proactive and try this experiement out.

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)