A meta robots tag is an HTML element added to a web page’s <head> section that tells search engines and bots how to crawl that page.
Think of it like a piece of code that provides a set of rules and instructions for search engine bots to follow.
The meta robots tag is very similar to the Robots.txt file.
Its primary job is to tell crawlers like:
…and others whether they can crawl your website.
If the bot is programmed to follow the meta robots tag (which most major bots are), it should follow the rules you list.
This is what the Robots meta tag looks like:
<meta name=”robots” content=”index, follow”>
The tag tells search engines two things:
This is a simple example of how the meta robots tag works.
It’s worth mentioning that you can put a number of different rules in your tag to instruct bots that visit your site.
More on that later.
What Will I Learn?
Meta robots tags are important because they give you more control over how search engines and other crawlers interact with your website.
Let me explain:
You probably want most pages on your website to be crawled and indexed by Google. But there might be some sensitive pages that you don’t want Google to display in the organic search results.
This is where the meta robots tag comes in!
By adding a nonidex tag to the pages you don’t want to be indexed, you can keep specific pages out of the SERPs.
This gives you more control over what pages are displayed in the search results.
But that’s just one application…
Let’s say you run a larger site with tons of pages. Using a meta robots tag is great for managing crawl budget.
You can ensure search engines only crawl the important pages on your website. This way, they won’t spend time on pages you care less about.
Think of meta robots tags as your way of talking directly to search engines and other web crawlers. It’s one of the best ways to manage how (and where) your website appears online.
You can add a number of different rules to your meta robots tag to instruct search engines and crawlers in different ways.
Here is a complete list of all the meta robots tag rules and attributes:
That’s a lot of rules, right?
That’s why robot meta tags are great for your site! You can instruct crawlers and search engines on what they can and can’t do with your content.
The easiest way to add a robots meta tag in WordPress is to use a free SEO plugin like RankMath.
Here’s how to do it:
Install RankMath, navigate to the post/page you want to manage and click on the “RankMath” widget.
Click on the “Advanced” tab and check the “No Index” option.
RankMath will automatically add the Noindex rule to your meta robots tag for that page.
Keep in mind that this would STOP search engines like Google from crawling and indexing your page in the SERPs.
Increase Your Search Traffic
In Just 28 Days…