The Giant Web Crawl service crawls over 600,000 different domains and over 600 million URLs each month. Our customers use this service to find common content or data elements found on millions of domains.
Example use-cases include:
- Competitive intelligence on adoption of website plugins
- Collecting keywords for content analysis
- Scanning websites for pirated music
- Collecting email addresses to fight email spam
Here's how the Giant Web Crawl works:
We process a list of every registered domain and feed this list into a set of 80legs web crawls.
These web crawls run a series of data extraction methods on each URL crawled.
As a Giant Web Crawl customer, you receive the data that interests you in your own 80legs account.
You download your Giant Web Crawl data from your own account.
Monotype ImagingScans websites for competitive intelligence on font usage.
MailChimpCrawls the entire web to help prevent spam emailing.
Giant Web Crawl Pricing
$1,000 per month for up to 50,000 websites (50 million URLs) crawled every month.
Contact Us to get access to the Giant Web Crawl.