Scrapy

Short description: Python web-crawling framework

Scrapy

Developer(s)| Zyte (formerly Scrapinghub)
Initial release| 26 June 2008 (2008-06-26)
Written in| Python
Operating system| Windows, macOS, Linux
Type| Web crawler
License| BSD License

Scrapy (/ˈskreɪpaɪ/[1] SKRAY-peye) is a free and open-source web-crawling framework written in Python and developed in Cambuslang. Originally designed for web scraping, it can also be used to extract data using APIs or as a general-purpose web crawler.[2] It is currently maintained by Zyte (formerly Scrapinghub), a web-scraping development and services company.

Scrapy project architecture is built around "spiders", which are self-contained crawlers that are given a set of instructions. Following the spirit of other don't repeat yourself frameworks, such as Django,[3] it makes it easier to build and scale large crawling projects by allowing developers to reuse their code.

The Scrapy framework provides you with powerful features such as auto-throttle, rotating proxies and user-agents, allowing you scrape virtually undetected across the net. Scrapy also provides a web-crawling shell, which can be used by developers to test their assumptions on a site’s behavior.[4]

Some well-known companies and products using Scrapy are: Lyst,[5][6] Parse.ly,[7] Sayone Technologies,[8] Sciences Po Medialab,[9] Data.gov.uk’s World Government Data site.[10]

## History

Scrapy was born at London-based web-aggregation and e-commerce company Mydeco, where it was developed and maintained by employees of Mydeco and Insophia (a web-consulting company based in Montevideo, Uruguay). The first public release was in August 2008 under the BSD license, with a milestone 1.0 release happening in June 2015.[11] In 2011, Zyte (formerly Scrapinghub) became the new official maintainer.[12][13]

## References

1. ↑ Commit 975f150
2. ↑ Scrapy at a glance.
3. ↑ "Frequently Asked Questions". http://doc.scrapy.org/en/latest/faq.html#did-scrapy-steal-x-from-django.
4. ↑ "Scrapy shell". http://doc.scrapy.org/en/latest/topics/shell.html.
5. ↑ Bell, Eddie; Heusser, Jonathan. "Scalable Scraping Using Machine Learning". http://talks.lystit.com/dsl-scraping-presentation/#/4.
6. ↑ Scrapy | Companies using Scrapy
7. ↑ Montalenti, Andrew. "Web Crawling & Metadata Extraction in Python". https://speakerdeck.com/amontalenti/web-crawling-and-metadata-extraction-in-python.
8. ↑ "Scrapy Companies". https://scrapy.org/companies/.
9. ↑ Hyphe v0.0.0: the first release of our new webcrawler is out!
10. ↑ Ben Firshman [@bfirsh] (21 January 2010). "World Govt Data site uses Django, Solr, Haystack, Scrapy and other exciting buzzwords bit.ly/5jU3La #opendata #datastore". https://twitter.com/bfirsh/status/8025368963.
11. ↑ Medina, Julia (19 June 2015). "Scrapy 1.0 official release out!". scrapy-users (Mailing list).
12. ↑ Pablo Hoffman (2013). List of the primary authors & contributors. https://github.com/scrapy/scrapy/blob/master/AUTHORS. Retrieved 18 November 2013.
13. ↑ Interview Scraping Hub.

## External links

* Scrapy Tutorial Series

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Scrapy. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Software:Scrapy&oldid=2681369"