How to monitor robots.txt or sitemap changes

robots.txt and sitemap.xml are public files. A deploy or plugin can add Disallow: / or point the sitemap at staging. You will not notice from the homepage. Monitor the files themselves and read the diff when they change.

Why these two URLs

• **/robots.txt** — crawler rules. A mistaken disallow blocks sections of the site.
• **/sitemap.xml** (or the sitemap index your robots.txt names) — the list of URLs you claim. Emptying it or switching hostnames is a silent SEO foot-gun.

Launch hygiene already checks them once — website launch checklist. Monitoring catches the regression later.

Setup

SignalWatch (opens in a new tab) fetches a public HTTP(S) URL and compares visible text (for XML, the response body text). Free: 3 monitors every 15 minutes, no card.

1. Create a monitor for https://yoursite.example/robots.txt.
2. Create another for the sitemap URL you actually ship.
3. Skip heavy CSS selectors; these files are usually plain. Use ignore rules only if a generator injects a changing timestamp on every request.
4. Alert to Slack or email. Treat any diff as suspicious until proven intentional.

You still want a SiteCheck inspection for whether noindex and sitemap issues exist on HTML pages — monitoring the file is not a full crawl. Pair with how to inspect a website for health problems.

Checklist

• Both URLs bookmarked and monitored (two of your three free monitors is a good spend).
• Diff reviewed on every alert.
• After intentional SEO changes, expect an alert — that confirms the monitor works.