Archive for May 19th, 2008

A fresh FlashGot user emailed me yesterday asking for help: he had previously loaded thousands of URLs in FlashGet (the popular Chinese download manager), but later he found they were unusable for batch downloading because they where redirected through the service: FlashGet was unable to follow the redirection and always downloaded the useless interstitial page, instead of the real content.

He had mass-imported them before hearing of FlashGot, using Internet Explorer's "Download all by FlashGet" menu item. In facts, FlashGot automatically works around many kinds of redirection and "link protection" services, either simple like, which just uses a 302 HTTP response, or quite complex like, which deploys a JavaScript-based obfuscation schema.
The best part, from my point of view at least, is that FlashGot doesn't need JavaScript to be enabled on the interstitial page, and does not execute any code embedded in the redirecting page itself, therefore it plays nice with NoScript and prevents users from meeting invasive ads, popups or even malware.
In the case, we've got a META refresh, whose main task is hiding the REFERER header, thus "anonymizing" the navigation path. FlashGot handles this situation gracefully, stripping away the header as intended, parsing the destination URL and sending it to the download manager.
But since these "thousands of URLs" had not been collected through FlashGot, and since download managers usually don't parse the content they download, FlashGet could not "see" the META refresh and stopped on the page, rather than reaching the real destination.

So this user wanted me to use my supposed knowledge of FlashGet internals to "hack" its database and replace the URLs with the actual target addresses "post mortem", since in the meanwhile the page where he had grabbed the links originally had disappeared from the web, so he couldn't use "FlashGot Selection" anymore.
I frankly don't own that knowledge, nor I was in the spirit of hacking a closed binary file format -- not for this purpose, at least: FlashGet is a proprietary software not affiliated to FlashGot, the open source Firefox extension which supports several download managers, and incidentally FlashGet itself -- right, right, I'll never curse enough my name-choosing skills...

But luckily enough, there's a fairly easy and general solution to this problem, thanks to the very simple structure of URLs. They always come in the form

, therefore they can easily be rewritten on the fly by a local proxy.
So I instructed him to do the following:

  • Download Muffin, a quite old but handy open source visual HTTP proxy. No installation needed, just put the JAR file in a writable folder of its own and click it, provided that you've got Java installed.
  • Open Muffin's Edit|Filters menu item.
  • Select Rewrite on the top list and click the Enable button.
  • Select Rewrite on the bottom list and click the Preferences button
  • Add the following rule:
    \?.*(https?://.*) $1
  • Click Apply and Save.
  • Configure your download manager to use Muffin. In FlashGet, FlashGet|Tools|Preferences|Proxy|Add, Title "Muffin", Server "localhost", port "51966", type "HTTP", "Always use default proxy".

I kept the rewrite rule extremely generic, working with every kind of redirected links having the destination HTTP URL somewhere in their query string -- not just
Of course, this trivial trick can also work with any other download manager supporting proxies, i.e. all the ones I know.
Happy downloads!

Discovered the ultimate cure for the NIMBY syndrome!

Reported by Beppe Grillo (popular Italian blog), but ignored by mainstream television news as usual: with an urgent decree effective since May 1st 2008, Italian Government allows toxic/nuclear waste storage sites, polluting power plants, incinerators and similar tourist attractions to be covered by State Secret.
Information about their existence, location and environmental impact can be declared "classified": anybody revealing them risks up to 5 years in prison.
Even the official Public Health agencies are banned from exercising their ordinary monitoring powers: in other words, no common people can actually measure, know or tell if a certain place in the sun-blessed Italian seaside or countryside is being actively poisoned by a government-blessed shit factory.

In the embedded Youtube movie clip, former Minister of Culture and Tourism Francesco Rutelli (of the cabinet which wrote the aforementioned decree) invites you to visit the Best Country in the World®.
Where are you going to spend your summer vacations?
Any relocation hint for me and my family?

Bad Behavior has blocked 924 access attempts in the last 7 days.