What is the meta robots tag?
The meta robots tag is the tag that we assign to some pages within the <head> section of the HTML. It is used to indicate search engine crawlers how they should behave when they reach to a web page. In this sense, you can ask search engines not to follow the links found on the page (nofollow), not to index (noindex) or not cache (nocache) the page.
The meta robots tag is of great value to optimize the use of your link juice since you’ll be able to decide which pages to transfer authority to or not. Carefully choose the tags to use in each situation since this has a direct impact on your link juice optimization.
Types of meta robots:
You can assign many types of meta robots to a web page. Here you have a list of the most important meta tags and its meaning.
- index= This tag allows search engines to index the page. It comes by default so, if you’re OK with search engines finding and tracking your pages, you don’t need to touch it.
- noindex= It restricts search engines from showing the page on their SERPs (Search Engine Results Pages).
- all= As mentioned above, this tag allows search engines to index the page and follow its links. “All” equals to “index follow.”
- noimageindex= It prohibits search engines from showing an image on their search results. But if the image receives any link, Google will keep indexing it so, in this case, it’s better to assign an X-Robots-Tag HTTP to the header.
- none= Its purpose is to ask search engines not to index nor follow any link on that page: noindex and nofollow. It tells them not to react when they see the page.
- follow= This robots tag invites Google to follow the links on the page, regardless of whether they’re “index” or not.
- nofollow= It asks search engines not to follow any links from the page.
- noarchive= This one prevents search engines from showing cache on the page (the information won’t be stored on the user’s browser for future visits).
- nocache= The same as the previous one, but only for MSN/Live.
- nosnippet= It won’t let the snippets appear on the SERPs, and it also prevents the cache generation.
- noodp= Although it no longer exists, it was used for preventing search engines from using the description.
- noydir= It prevents Yahoo! from using the description on its directory as it would be shown on the search results (it isn’t used either, but you may come across it).
How is the meta robots tag most commonly used?
If you don’t have any meta robots tag on your web page, Google indexes and follows the page by default. It’s like you had the “index, follow” already assigned. Below are some examples of the most commonly used combinations of tags.
- “Index, follow” – “index the web page on SERPs and follow the links”
- “index, nofollow” – “index the page, and not follow the links”
- “noindex, follow” – “do not index the page but follow the links”
- “no index, no nollow, no-cache” – “do not index, do not follow and disable caching for the page.”
When using the meta robots tag, you have to make sure that the guidelines given in the current page have congruence with the meta robots tag specified in the linked pages. For example, if you decide to add “index, follow” to the current page, the outgoing pages should be indexable and never have a “noindex” tag in its meta robots.
How to check the meta robots tag is implemented correctly
Once you’ve assigned the meta robots tag to your pages, it may be difficult to check whether this task has been carried out correctly, especially on large websites. For this purpose, it is recommended to use a web crawler such as FandangoSEO. Once you’ve crawled your site, you’ll be able to see your index and noindex pages as well as if you’ve got pages blocked by robots.txt. By reviewing the lists of URLs with the different tags, you can ensure that you are not hiding relevant pages in search engines.
The use of rel nofollow
When you use the meta robots tag, you want to assign the rule that best suits the majority of the links found on the page. That said, we often come across with some exception of links where you don’t want to apply the general rule. In that case, you can use the rel nofollow. For example, if you have the meta robot tag “index, follow” in a page but you have specific links that you don’t want search engines to follow, insert the rel “nofollow” attribute to them.
Example:
<a rel=”nofollow” href=”http://www.examplepage.com”>anchor text</a>
