Custom Web Crawling
1. Use one of the default 80legs apps or build your own using 80legs app templates.
You can also use default 80legs crawling apps to collect common HTML data, like links, keywords, meta tags, and more.
2. Enter the URLs or websites you want to crawl.
Enter one or more (up to several thousand) URLs you want to crawl. These are the URLs where the web crawl will start. Links from these URLs will be followed automatically, depending on the settings of your web crawl.
3. Run the web crawl.
Once you’ve finished configuring your web crawl, you submit it for processing in our cloud platform. 80legs will begin working on the web crawl and updating its progress as the crawl proceeds.
80legs will automatically handle all of the heavy lifting required to properly run a web crawl, including de-duplicating crawled URLs, utilizing multiple IP addresses, auto rate-limiting crawl speeds, and more.
4. Download your results.
80legs will post results as the web crawl runs. Once the crawl has finished, all of the results will be available, and you can download them to your computer or local environment.