Robots.txt A file used to keep Web pages from being indexed or to tell which pages you want a search engine to index.