Unrivaled Web Crawling Performance

Our unique technology gives you easy, fully-customizable access to the world’s fastest web crawler.

The 80legs web crawling platform is like no other web crawling service available. It enables access to unrivaled web crawling performance that you can easily customize to meet your unique web scraping needs. Follow through with the steps below to learn how we make this possible.


Arhcitecture Diagram

1
Step 1: You use the 80legs website or API to create a web crawl. We give you the ability to customize your crawl by providing a handful of options that specific how your crawl will run. These options are:
  1. A list of URLs which tell 80legs where to start the crawl
  2. An 80pp which tells 80legs what data to scrape from each crawled URL, as well as what URLs to crawl next
  3. Other options to control the crawl, such as the # of total URLs you want to crawl.
In order to make the process of building the web crawl easy on you, we provide several pre-built 80apps for you to use. However, if you want to do something more specific, we also give you the option of building your own 80app. Learn how...
2 Step 2: 80legs takes over from here and does the heavy lifting for you. Your URLs are run through a variety of sanity checks to make sure they can be crawled. If they pass, their sent to a URL queue, where they’ll be wait to be picked up. 80legs will automatically rate-limit how fast you crawl certain URLs so your crawl doesn’t overwhelm any websites. This is one of the ways we make sure your web crawl doesn’t get blocked by anyone.
3 Step 3: Your URLs, along with the 80app, are sent out to our massive pool of crawling nodes. Each crawling node will fetch the HTML content of a URL, run the 80app on that HTML, and return the resulting data to 80legs. This massive collection of crawling nodes is a key reason 80legs can provide such amazingly-fast web crawling.
4 Step 4: As your crawl runs, the results from each URL crawled will be packed up and delivered to your account, where they’ll wait for you to download them. Again, you can use the website or API to do this.

If you’d like to see this in action, try running a web crawl now.
It’s free!
Build a Web Crawl