A plain text file placed in website’s root directory. Allows organizations to give direction to the search engine crawlers and spiders about which website pages should be included in search results, and which should be excluded.
In other words, it’s a way to tell the search engine to ignore specific pages or collections of pages.