Broken Link CheckerAbout our crawler (BrokenLinkCheckerBot)
If you found this page in your server logs: BrokenLinkCheckerBot is the crawler behind brokenlinkchecker.io, a suite of SEO tools site owners use to check their own websites.
A visit means someone asked us to scan a site that is, or links to, one of your pages.
01 · Crawler details
How does the bot identify itself?
Every request carries this user agent string, with a link back to this page:
BrokenLinkCheckerBot/1.0 (+https://brokenlinkchecker.io/bot)
When does it crawl?
Only on demand. The bot visits a site when its owner or a visitor starts a scan, or on the schedule a paying customer configured.
It powers the link scans, site audits and internal link analysis in the suite. It does not roam the web building an index.
What about IP addresses?
Our requests are routed through rotating residential ISP addresses, so there is no fixed list of crawler IPs to publish.
Identify the bot by its exact user agent string above. Every request carries it, and a scanned site keeps a consistent address for the duration of a scan.
How do we handle robots.txt?
Crawling, meaning discovering pages: we fetch and respect your robots.txt. Paths you disallow are not crawled. Users can only relax this for sites they scan themselves. The default is on.
Checking a single link: when a page we scan links to your site, we verify that one URL with a single HEAD or GET request. Like a browser following the link, this individual check does not consult robots.txt.
How do I block or allow the bot?
To keep us from crawling your site, add this to your robots.txt:
User-agent: BrokenLinkCheckerBot Disallow: /
To block us entirely, including single link checks, add a rule in your firewall or CDN that matches the user agent string above.
To make sure link checks against your site always work, allowlist the same user agent instead.
If you see abusive traffic claiming to be our bot, email us the timestamps and IPs from your logs and we will confirm whether the requests were ours.
How fast does it crawl?
At most 3 concurrent requests per host, with a delay between requests. We back off automatically when a server answers 429 or 403.
Check results are cached, so popular URLs are verified at most once every 6 hours across all scans. Anonymous scans are capped at 250 pages, with internal safety limits on total links per scan.
Crawler questions
Report unexpected crawl behavior to bot@brokenlinkchecker.io with the timestamps and IPs from your logs. We reply within one business day.
Scan with the crawler yourself
Free for 250 pages. No account needed.