Some open source software developed by me you can use on your own web
crawling solutions
Proxy Service
An API to do a smarter use of proxy lists
Repository
Proxy Service (Scrapy Middleware)
A Scrapy middleware to use Proxy Service with Scrapy
Repository
Proxy Service (PSDash)
A dashboard for manage Proxy Service
Repository
User Agent Service
Service to store and return user agent strings to use in your web
crawling project
Repository
User Agent Service (Scrapy Middleware)
A Scrapy middleware to use User Agent Service with Scrapy
Repository
Scrapy Object Cache
Object cache solution for Scrapy
Repository