Free tool by SearchVector
See what search engines can crawl and index
Try:
Fetch
Pull the site’s robots.txt.
Parse
Read every crawl directive.
Rules
Match rules per user-agent.
Index
Confirm indexability.
Sitemap
Detect the sitemap.
Report
Show the full verdict.
example.com/robots.txt
Crawlable
Response Time
0.42s
Crawl Health
92
Healthy
Indexable Pages
248
+12%
Crawl Access
PATH
ACCESS
Googlebot
/
Allowed
Bingbot
/
Allowed
*
/admin
Blocked
AhrefsBot
/
Blocked
example.com/robots.txt
Crawlable
Response Time
0.42s
Crawl Health
92
Healthy
Indexable Pages
248
+12%
Crawl Access
PATH
ACCESS
Googlebot
/
Allowed
Bingbot
/
Allowed
*
/admin
Blocked
AhrefsBot
/
Blocked
example.com/robots.txt
Crawlable
Response Time
0.42s
Crawl Health
92
Healthy
Indexable Pages
248
+12%
Crawl Access
PATH
ACCESS
Googlebot
/
Allowed
Bingbot
/
Allowed
*
/admin
Blocked
AhrefsBot
/
Blocked
Fetch
Pull the site’s robots.txt.
Parse
Read every crawl directive.
Rules
Match rules per user-agent.
Index
Confirm indexability.
Sitemap
Detect the sitemap.
Report
Show the full verdict.
Analyze your robots.txt and verify that Google and Bing can crawl and index your pages — before a blocked directive quietly costs you rankings.
Make sure search engines can find your content
If search engines can't crawl your pages, they never get indexed — and pages that aren't indexed can't rank. Our crawler reads your robots.txt and surfaces the directives that decide what bots are allowed to see.
This free tool gives instant crawlability checks. For continuous technical-SEO monitoring and full-site audits, connect the SearchVector dashboard.
What you get
Robots.txt analysis
Included
Parse crawl directives for every user-agent in seconds.
Indexability check
Included
Confirm Google and Bing are allowed to index the page.
Instant results
Included
No queue, no waiting — status the moment you hit analyze.
Free forever
Included
No signup required. Part of our free SEO toolkit.
How it works
Check crawlability in three steps
From URL to a clear allow/block verdict in seconds.
01
Enter your URL
Paste any website URL. We fetch and read its robots.txt automatically.
02
Get the analysis
We parse the file and check crawl and index rules for Google and Bing.
03
Review results
See a clear crawlability verdict plus the full robots.txt for review.
Everything you need to check crawlability
A focused technical-SEO check that answers one question fast: can bots reach this page?
Robots.txt parser
Fetch and parse robots.txt to understand every crawl directive.
Multi-engine check
Verify crawlability for both Google and Bing at once.
Instant analysis
Get results immediately without lengthy site-wide crawls.
Index status
Confirm your pages are allowed to be indexed in search.
Full robots.txt view
Read the complete file for a careful manual review.
Free & unlimited
Check as many URLs as you like — no registration.
Website crawlability Questions
Everything you need to know about crawlability and robots.txt.