What is a Robots.txt file? It is a text file that resides in the basis directory of your site and tells Google, Bing along with other search engines like google and yahoo which elements of your website to crawl, index and rank you for, and which components not to index. A WordPress or OpenCart Web-site will have many locations like types and admin