How To robots.txt – Control Search Engine Website Access Using robots.txt
Using robots.txt file and robots.txt disallow and allow rules you can control which search engines should crawl your site’s content. It is is a convention or protocol to prevent cooperating web spiders and other web robots from accessing all or part of a website. The robots.txt file is commonly used to block these spider bots…