Using file and disallow and allow rules you can control which search engines should crawl your site's content. It is is a convention or protocol to prevent cooperating web spiders…
Continue Reading »