80legs

Menu
  • Products
    • Custom Web Crawling
    • Giant Web Crawl
    • Datafiniti
  • Resources
    • Customers
    • Developers
  • Pricing
  • About
  • Contact
  • Signup or Login

The 80legs Web Crawler

What is voltron?

If you’ve come to this page, then you’re probably interested in learning more about our web crawler, identified as user-agent “voltron.”

Voltron runs on the 80legs web crawling platform, which consists of many servers, which is why you may see our web crawler access your site from many different IP addresses.

Why is voltron crawling my website?

Voltron is the user-agent used by 80legs, a web crawling service provider. 80legs allows its users to design and run custom web crawls. So, if voltron is crawling your website, it means that one or more 80legs users created a web crawl that went (eventually) to your website.

People use 80legs for a variety of reasons, including providing data to their own search engines, monitoring trends in online opinions, and for other interesting applications.

Help us crawl your website properly

If you feel that voltron is crawling your website too quickly, please let us know what an appropriate crawl rate is for you. If you’d like us to stop crawling your website, the best thing to do is to block our web crawler using the robots.txt specification. To do this, add the following to your robots.txt:

User-agent: voltron
Disallow: /

If you block voltron using robots.txt, you will see crawl requests die down gradually, rather than immediately. This happens because of our distributed architecture. Our computers only periodically receive robots.txt information for domains they are crawling.

Blocking us by IP address

Blocking our web crawler by IP address will not work. Due to the distributed nature of our infrastructure, we have numerous constantly changing IP addresses. We strongly recommend you don’t try to block our web crawler by IP address, as you’ll most likely spend several hours of futile effort and be in a very bad mood at the end of it. The best way to save you time and frustration it just to add us in your robots.txt or contact us directly.

Learn more

To read more about the inner workings of voltron, please take a look at our knowledge base.

To learn more about 80legs, please check out the rest of the site. If you’d like to ask us any questions, don’t hesitate to contact us.

Follow us on Social Media
80legs

Headquarters

2815 Manor Rd #100
Austin, TX 78722
844-493-2328

Products

  • Custom Crawling
  • Giant Web Crawl
  • Datafiniti

Resources

  • Customers
  • Developers

Terms

  • Terms of Use
  • Privacy Policy
  • Request Your Data

Pricing

About

Signup or Login

Contact

Follow Us Copyright © 2023 Datafiniti, LLC. All rights reserved. | Terms of Use

We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.OkRead our Privacy Policy