Definition
A robots.txt file sits at the root of a domain and tells crawlers which paths they may or may not request. It controls crawling, not indexing: a blocked URL can still appear in search results if other pages link to it, because the rule stops the fetch rather than the listing.

Why it matters
robots.txt is the first file most crawlers request, and one of the easiest ways to take a site out of search by accident. A single line left over from a staging environment can block everything.
The distinction that causes the most trouble is between crawling and indexing. Blocking a URL stops it being fetched. It does not reliably stop it being listed, and it actively prevents Google from seeing any instruction on the page that would.
How it works
The file lives at the root, for example https://www.example.com/robots.txt, and is standardised as RFC 9309, the Robots Exclusion Protocol. Google's documentation on how it interprets the file covers the specifics.
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help
Sitemap: https://www.example.com/sitemap.xml
User-agent names which crawler the rules apply to. Disallow lists paths not to request. Allow carves exceptions out of a broader disallow. The Sitemap line is independent of the rules and simply points at the file.
Two properties matter and are frequently misread. The rules are directives for well behaved crawlers and are ignored by crawlers that choose to. And a disallowed URL may still be indexed, typically with no description, when other pages link to it, because Google knows the URL exists without being permitted to look at it.
Worked example
To keep a page out of search results, the instruction has to be on the page, and the crawler has to be allowed to read it:
<meta name="robots" content="noindex" />

Which means the page must not be blocked in robots.txt. Blocking it and adding noindex is a contradiction: the crawler is forbidden from fetching the page, so it never sees the noindex, and the URL can persist in results indefinitely. Google's page removal documentation describes the correct order.
The right tool depends on the goal:
- Keep it out of resultsnoindex, and allow crawling
- Stop wasteful crawlingrobots.txt disallow
- Both, in sequencenoindex first, wait for recrawl, then disallow
- Protect private dataauthentication, not either of these
Common mistakes
- Shipping a staging rule.
Disallow: /deployed to production removes the site from crawling. It is the most expensive one-line mistake in SEO. - Blocking a page to deindex it.It achieves the opposite more often than not.
- Blocking CSS and JavaScript.Google renders pages. Blocking the assets needed to render leaves it judging a broken version.
- Treating it as security.The file is public and lists exactly what you did not want visited. Anything genuinely private needs authentication.
- Blocking a URL that carries a canonical tag.The tag is never read, so the consolidation never happens.
Where it does help
It is the right tool for crawl waste, and one of the first things a technical SEO review checks: infinite filter combinations, internal search result pages, calendars that generate a URL per day. Those consume crawl budget without any chance of ranking, and keeping crawlers out of them is exactly what the file is for.