Sitemap & robots.txt
Finds the sitemap (from robots.txt or the usual addresses), parses it, counts the URLs, checks a sample of them, and reads robots.txt line by line for the mistakes that quietly hide a site from search.
20 of 20 free runs left this hour. Used them up? An extra run is $1.49 and works for every tool here, no account needed.
How it works
- We fetch /robots.txt and parse it: user-agents, Disallow and Allow rules, Sitemap lines, syntax slips.
- The sitemap is taken from robots.txt, or /sitemap.xml and /sitemap_index.xml are tried.
- The XML is parsed: a plain sitemap or an index of sitemaps, URL count, lastmod dates, size limits.
- A sample of listed URLs is requested to see that they answer 200, are not redirected and are not blocked by robots.txt.
Questions people ask
Do I need a sitemap?
For a site with more than a handful of pages, yes. It is the list search engines start from, and the lastmod dates tell them what to re-crawl. Without one they discover pages only by following links.
Why does robots.txt matter if the sitemap is fine?
robots.txt wins. A page listed in the sitemap but disallowed in robots.txt is not crawled — a very common leftover from a launch when "Disallow: /" was meant to keep the staging site out.
What are the limits for a sitemap?
50,000 URLs and 50 MB uncompressed per file. Above that, split it and list the parts in a sitemap index.
Does noindex belong in robots.txt?
No — Google stopped honouring noindex in robots.txt in 2019. Use a meta robots tag or an X-Robots-Tag header on the page itself.
My sitemap has URLs that redirect. Is that bad?
Yes: every entry should be the final, canonical address that answers 200. Redirected or 404 entries waste crawl budget and make the whole file look unreliable.
Free, no signup. Runs on our servers, nothing is installed. Want all of it at once? The full site check runs 30+ tests in one go.