Crawler

Component of a search engine that gathers listings by automatically “crawling” the Web. A search engine’s crawler (also known as a Spider or robot) follows links to Web Pages. It makes copies of those pages and stores them in a search engine’s index.