ROBOTS tag attribute – META NAME=”robots” and CONTENT values Explained
Meta elements as the name suggests are HTML or XHTML elements present between the <HEAD></HEAD > html tags. The META tags are used to provide metadata about a Web page.The metadata can be used by browsers (display,reload etc) and
Here let us discuss about the meta tag attribute named “robots”.
The robots attribute is supported by many major search engines. This attribute controls whether search engine spiders are allowed to index a page, or not, and whether they should follow links from a page, or not. The robots attribute can contain one or more comma-separate values.The attribute values are discussed below.
The noindex and nofollow values prevents search engines from indexing the page and crawling the page respectively.Other values are recognized by certain search engines and can influence how the engine indexes pages. These values also influence on how those pages appear on the search results
NOINDEX – <META NAME=”ROBOTS” CONTENT=”NOINDEX, FOLLOW”>
The noindex value prevents a page from being indexed
INDEX – <META NAME=”ROBOTS” CONTENT=”INDEX, FOLLOW”>
This directive tells the search engine robots to index the page and to include your page within their search directory.
FOLLOW – <META NAME=”ROBOTS” CONTENT=”INDEX, FOLLOW”>
When you use this tag, you are telling the search engines that you want their robot to follow any links that are found on that page.
NOFOLLOW – <META NAME=”ROBOTS” CONTENT=”INDEX, NOFOLLOW”>
The NOFOLLOW value prevents links of the page from being crawled.the NOFOLLOW directive only applies to links on this page.If the robot might find the same links on some other page without a NOFOLLOW and so still arrives at your undesired page.
NOFOLLOW value is different from that of rel=”nofollow” link attribute.
Read it here:
NOARCHIVE – <META NAME=”ROBOTS” CONTENT=”NOARCHIVE”>
This value instructs a search engine not to store an archived copy of the page.This will prevent your page from being cached.
NOSNIPPETS – <META NAME=”ROBOTS” CONTENT=”NOSNIPPETS”>
The value nosnippets directs the search engine not include a snippet from the page along with the page’s listing in search results.
Some examples of robot usage are as follows:
<meta name="robots" content="index,follow">
<meta name="robots" content="noindex,follow">
<meta name="robots" content="index,nofollow">
<meta name="robots" content="noindex,nofollow">
Even though Meta tags are used to prevent search engines from indexing content of your website.The best and reliable method is the use of the Robots.txt file.