A fresh FlashGot user emailed me yesterday asking for help: he had previously loaded thousands of URLs in FlashGet (the popular Chinese download manager), but later he found they were unusable for batch downloading because they where redirected through the anonym.to service: FlashGet was unable to follow the redirection and always downloaded the useless anonym.to interstitial page, instead of the real content.

He had mass-imported them before hearing of FlashGot, using Internet Explorer's "Download all by FlashGet" menu item. In facts, FlashGot automatically works around many kinds of redirection and "link protection" services, either simple like tinyurl.com, which just uses a 302 HTTP response, or quite complex like link-protector.com, which deploys a JavaScript-based obfuscation schema.
The best part, from my point of view at least, is that FlashGot doesn't need JavaScript to be enabled on the interstitial page, and does not execute any code embedded in the redirecting page itself, therefore it plays nice with NoScript and prevents users from meeting invasive ads, popups or even malware.
In the anonym.to case, we've got a META refresh, whose main task is hiding the REFERER header, thus "anonymizing" the navigation path. FlashGot handles this situation gracefully, stripping away the header as intended, parsing the destination URL and sending it to the download manager.
But since these "thousands of URLs" had not been collected through FlashGot, and since download managers usually don't parse the content they download, FlashGet could not "see" the META refresh and stopped on the anonym.to page, rather than reaching the real destination.

So this user wanted me to use my supposed knowledge of FlashGet internals to "hack" its database and replace the anonym.to URLs with the actual target addresses "post mortem", since in the meanwhile the page where he had grabbed the links originally had disappeared from the web, so he couldn't use "FlashGot Selection" anymore.
I frankly don't own that knowledge, nor I was in the spirit of hacking a closed binary file format -- not for this purpose, at least: FlashGet is a proprietary software not affiliated to FlashGot, the open source Firefox extension which supports several download managers, and incidentally FlashGet itself -- right, right, I'll never curse enough my name-choosing skills...

But luckily enough, there's a fairly easy and general solution to this problem, thanks to the very simple structure of anonym.to URLs. They always come in the form

http://anonym.to?http://www.destination-site.com

, therefore they can easily be rewritten on the fly by a local proxy.
So I instructed him to do the following:

  • Download Muffin, a quite old but handy open source visual HTTP proxy. No installation needed, just put the JAR file in a writable folder of its own and click it, provided that you've got Java installed.
  • Open Muffin's Edit|Filters menu item.
  • Select Rewrite on the top list and click the Enable button.
  • Select Rewrite on the bottom list and click the Preferences button
  • Add the following rule:
    \?.*(https?://.*) $1
  • Click Apply and Save.
  • Configure your download manager to use Muffin. In FlashGet, FlashGet|Tools|Preferences|Proxy|Add, Title "Muffin", Server "localhost", port "51966", type "HTTP", "Always use default proxy".

I kept the rewrite rule extremely generic, working with every kind of redirected links having the destination HTTP URL somewhere in their query string -- not just anonym.to.
Of course, this trivial trick can also work with any other download manager supporting proxies, i.e. all the ones I know.
Happy downloads!

7 Responses to “One Rule to Redirect Them All”

  1. #1 Steve says:

    If you look on the Flashgot site:

    "Batch download support for many anonymize/redirection services (anonym.to, tinyurl.com, moourl.com, anonymz.com, shorten.ws, tubeurl.com)"

    Also.. why don't you just do a simple find and replace on the list?

    "http://anonym.to?" > ""

  2. #2 Aerik says:

    That really simple kind of redirect is also easily taken care of the "redirect remover" firefox extension. With linkification enabled, it even corrects the text-link you've written on this page when you click it or hover over it.

    I also use Flashget with Flashgot. I never, ever have this kind of problem.

  3. #3 Giorgio says:

    @Steve, Aerik:
    The point of the whole article is that if he used FlashGot's commands to grab those URL everything would have worked just fine, because FlashGot resolves the redirections, including anonym.to, before sending the link(s) to the download manager.

    "Find a replace on the list" was not an option, because the list was not in a plain text file but inside the binary, proprietary and opaque FlashGEt database.
    The same goes for the "redirect remover" extension, because the original page(s) where the links were hosted was not available anymore.

  4. #4 Aerik says:

    Well, another reason I advocate redirect remover is that when you're using it, you click a redirect link and instead of contacting the first site's server and then redirecting to the second site's server, you just go to the site you want. Along with the refcontrol or disabling referrers with web developer, you can go ahead and click links without giving unnecessary hits and clickcounts to websites which are basically blogspamming by contributing nothing to the media they want to be a pseudo-mirror for but want to gain ad revenue simply for the clicks.

  5. #5 Giorgio says:

    @Aerik
    Good point!

  6. #6 Kaoper says:

    Im a fan of the firefox addon for this.

    I also use www.referhush.com on a few sites.

  7. #7 Eric says:

    I was sooo hoping you had a hack for the proprietary binary database that FlashGet uses. I have a completely reasonable, although twisted logically, reason. I have had a couple of instances of FlashGet crashing on me and this last one took out a slew of downloads lined up to run tonight. Sadly, I will perhaps attempt hacking the (ahem, yea, proprietary) database in the meager attempt to recover that list.

    Much thanks to the gurus here and around the world who make this tech worth while.

Bad Behavior has blocked 2420 access attempts in the last 7 days.