About Monotype & Their Needs
Monitoring websites for font usage and market intelligence
Monotype Solutions produces and sells a wide variety of fonts for use on the web. They compete with other companies to provide these creative assets.
Checking millions of websites for HTML codes like fonts requires enterprise-scale web crawling
In order to gain market intelligence on font usage on the web, Monotype needed to scan millions of websites on a regular basis. Doing this at scale meant overcoming various technical challenges. Specifically, they would have to setup their own web crawling infrastructure to handle the web crawling volume required for their needs.
How it Worked
Monotype used 80legs to power web crawling at scale
By using the 80legs web crawling platform, Monotype easily setup several automated web crawls to the millions of websites they wanted to monitor. Our simple, RESTful API let them automate the process of crawl creation, URL list updating, and result downloading. They only needed to handle configuration of the web crawls, and 80legs handled everything else, including rate limiting and utilizing multiple IP addresses.