Crawler

An automated program, also known as a “bot” or “spider”, which visits web pages and indexes their content for search engines.