What is a robot.txt file? Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on