Custom Web Crawling

1. Use one of the default 80legs apps or build your own using 80legs app templates.

You can customize your app to scrape and crawl any public web page. 80legs crawling apps use Javascript methods that you can update to meet any scraping requirements you have. You can customize what data gets scraped and which links are followed from each URL crawled.

You can also use default 80legs crawling apps to collect common HTML data, like links, keywords, meta tags, and more.

2. Enter the URLs or websites you want to crawl.

Enter one or more (up to several thousand) URLs you want to crawl. These are the URLs where the web crawl will start. Links from these URLs will be followed automatically, depending on the settings of your web crawl.

3. Run the web crawl.

Once you’ve finished configuring your web crawl, you submit it for processing in our cloud platform. 80legs will begin working on the web crawl and updating its progress as the crawl proceeds.

80legs will automatically handle all of the heavy lifting required to properly run a web crawl, including de-duplicating crawled URLs, utilizing multiple IP addresses, auto rate-limiting crawl speeds, and more.

4. Download your results.

80legs will post results as the web crawl runs. Once the crawl has finished, all of the results will be available, and you can download them to your computer or local environment.

Use Custom Web Crawls To:

Additional Features

Rate Limiting

80legs automatically rate limits requests to web pages, using response times to run crawls faster or slower.

Multiple IP Addresses

80legs distributes request to web pages across a large collection of servers with different IP addresses.

RESTful API

The 80legs API lets you easily automate web crawl creation, result retrieval, and more.

Scraper Customization

Use the 80legs Javascript-based app framework to fully customize behavior for your web crawling.

Custom Web Crawling Customers

Ready to start?

Sign Up Now