Robots.txt

Why it matters

The robots.txt file sits at the root of your domain and tells search engine crawlers which parts of the site to avoid. It works by convention, not enforcement. Most major crawlers respect it. Bad actors do not.

Blocking a section of your site in robots.txt is correct for staging directories, admin panels, and dynamically generated pages with no search value. What causes problems is blocking sections that should be indexed, or failing to update robots.txt after a site restructure that moved important content into a directory that was previously blocked.

The difference between robots.txt and noindex matters here. Robots.txt prevents a page from being crawled. Noindex prevents a page from being indexed. A page blocked in robots.txt may still appear in search results if external links point to it — Google can list a URL it knows about without having crawled the content.

Common mistakes: a Disallow: / that blocks Googlebot from the entire site, a disallowed assets directory that prevents CSS or JavaScript from loading, or a robots.txt reset to defaults by a CMS update.

How we check it

We fetch the robots.txt file for your domain and evaluate whether the monitored page URL is blocked by any Disallow rule applying to Googlebot or the universal wildcard. We store the result as a yes/no flag.

If the robots.txt status for a page changes between crawls, we flag it.

What to watch for

A page that was previously accessible becoming robots-blocked is high priority. Check your robots.txt file directly and verify whether the change was intentional. Deployments that reset a robots.txt to a default state are a known cause.