Python framework for web scraping and web crawling projects
Web crawling & scraping framework
scrapy
$ scrapy startproject myproject
$ scrapy genspider myspider example.com
$ scrapy crawl myspider