Search Engine Spider Checker
Look up any crawler, prove whether a visiting bot is genuine, and see exactly what your robots.txt is telling it.
Paste a User-Agent string from your log to see which crawler it belongs to — and what the string alone can and cannot prove.
This is the only reliable test. A User-Agent can be copied by anyone, but the IP address a real crawler uses is published by its owner — so we look up its reverse DNS and then resolve that name back to confirm it.
Reads the live robots.txt of any site, shows which crawlers it names, and tells you whether the one you pick is actually allowed to fetch the path you type.
Paste raw access-log lines (Apache or Nginx, combined format) and get a break-down of which crawlers really hit your site. Everything is parsed in your browser — the log never leaves this page.
Crawler names, User-Agent examples and reverse-DNS domains come from each vendor's own documentation — open the documentation link on any card to check the current details. Passwords, cookies and page content are never read: the robots.txt check fetches that one public file, and nothing about your site is stored unless logging is switched on.