Robots.txt is a plain-text file at the root of a site that gives crawlers instructions about which paths they should or should not request. It uses a simple syntax of user-agent lines followed by allow and disallow rules. It is important to understand its limits: robots.txt is a request that well-behaved crawlers honour, not an access control, and disallowing a path does not make it private or guarantee it stays out of search results if other pages link to it. It also is not the right tool for hiding a page from search — that is what a noindex directive is for. Misconfiguring it, such as disallowing the whole site, can quietly stop crawling entirely.
Glossary
Robots.txt
A file that tells crawlers which paths they may request.