A file that is used to keep Web pages from being indexed or to tell which pages you want a search engine to index. While some spiders can ignore the directions within, having this file correctly formatted is critical to your site's health from the perspective of search engines.