#opensourcehn

Agree. Scrapy is great for this sort of thing. Takes care of little details you don't want to be worrying about.

I coupled scrapy with Stem to control Tor on the odd chance your IP gets blocked. Works great.

This may be hypocritical given what we are discussing, but please don't use Tor for this kind of activity.

There are hundreds of proxy vendors, and other curators of open proxies if you have a tight budget, which are designed to spread out the origin of the load and do some minor masking of your traffic.

But it would make me very sad to have an exit node blocked for being involved with this.