robots.txt
A file telling crawlers which parts of your site to skip. Advisory, not security — well-behaved bots honor it; bad ones read it for a map.
Free · no signup · last reviewed August 29, 2026
The definition
A file telling crawlers which parts of your site to skip. Advisory, not security — well-behaved bots honor it; bad ones read it for a map.
Why it matters to an owner
The classic self-inflicted wound: a developer blocks the whole site during a redesign and forgets to unblock at launch. If rankings vanish post-launch, look here first.
Keep going
XML SitemapA machine-readable list of your site's pages that search engines use to discover content. …SEO (Search Engine Optimization)Making your site the best available answer for searches your customers make, so it ranks w…Organic TrafficVisitors from unpaid search results. Earned by content and reputation rather than bought p…
This is the work. Want it done?
Everything here is free to use yourself — that’s the point. If you’d rather a senior marketer just handle it, in your accounts and your name: 20-minute call, no pitch deck.