A fresh FlashGot user emailed me yesterday asking for help: he had previously loaded thousands of URLs in FlashGet (the popular Chinese download manager), but later he found they were unusable for batch downloading because they where redirected through the anonym.to service: FlashGet was unable to follow the redirection and always downloaded the useless anonym.to interstitial page, instead of the real content.
He had mass-imported them before hearing of FlashGot, using Internet Explorer’s “Download all by FlashGet” menu item. In facts, FlashGot automatically works around many kinds of redirection and “link protection” services, either simple like tinyurl.com, which just uses a 302 HTTP response, or quite complex like link-protector.com, which deploys a JavaScript-based obfuscation schema.
The best part, from my point of view at least, is that FlashGot doesn’t need JavaScript to be enabled on the interstitial page, and does not execute any code embedded in the redirecting page itself, therefore it plays nice with NoScript and prevents users from meeting invasive ads, popups or even malware.
In the anonym.to case, we’ve got a META refresh, whose main task is hiding the REFERER header, thus “anonymizing” the navigation path. FlashGot handles this situation gracefully, stripping away the header as intended, parsing the destination URL and sending it to the download manager.
But since these “thousands of URLs” had not been collected through FlashGot, and since download managers usually don’t parse the content they download, FlashGet could not “see” the META refresh and stopped on the anonym.to page, rather than reaching the real destination.
So this user wanted me to use my supposed knowledge of FlashGet internals to “hack” its database and replace the anonym.to URLs with the actual target addresses “post mortem”, since in the meanwhile the page where he had grabbed the links originally had disappeared from the web, so he couldn’t use “FlashGot Selection” anymore.
I frankly don’t own that knowledge, nor I was in the spirit of hacking a closed binary file format — not for this purpose, at least: FlashGet is a proprietary software not affiliated to FlashGot, the open source Firefox extension which supports several download managers, and incidentally FlashGet itself — right, right, I’ll never curse enough my name-choosing skills…
But luckily enough, there’s a fairly easy and general solution to this problem, thanks to the very simple structure of anonym.to URLs. They always come in the form http://anonym.to?http://www.destination-site.com, therefore they can easily be rewritten on the fly by a local proxy.
So I instructed him to do the following:
- Download Muffin, a quite old but handy open source visual HTTP proxy. No installation needed, just put the JAR file in a writable folder of its own and click it, provided that you’ve got Java installed.
- Open Muffin’s Edit|Filters menu item.
- Select Rewrite on the top list and click the Enable button.
- Select Rewrite on the bottom list and click the Preferences button
- Add the following rule:
?.*(https?://.*) $1
- Click Apply and Save.
- Configure your download manager to use Muffin. In FlashGet, FlashGet|Tools|Preferences|Proxy|Add, Title “Muffin”, Server “localhost”, port “51966″, type “HTTP”, “Always use default proxy”.
I kept the rewrite rule extremely generic, working with every kind of redirected links having the destination HTTP URL somewhere in their query string — not just anonym.to.
Of course, this trivial trick can also work with any other download manager supporting proxies, i.e. all the ones I know.
Happy downloads!
