A text file placed in the root directory of a website that tells search engine crawlers which pages or sections of the site should not be indexed or crawled.
Robots.txt
by Alex Brinkman | August 2, 2024 | 0 comments
A text file placed in the root directory of a website that tells search engine crawlers which pages or sections of the site should not be indexed or crawled.