Crawler failures occur when our bot is blocked.
Common Fixes
- Firewalls: Cloudflare or ModSecurity might flag us. Whitelist User Agent:
SERPHubBot/1.0. - Robots.txt: Ensure you aren't blocking
User-agent: *from entering. - Speed: Slow servers might timeout.
Was this article helpful?