Is the Ahrefsbot giving your site trouble?
Ahrefsbot crawls your website regularly. It acts just the same as Googlebot – coming and going as it pleases.
This isn’t usually a problem but sometimes it can cause problems especially if it’s overloading your server and slowing your site down.
The good news?
You can quickly and easily block Ahrefsbot from visiting your website. You just need to choose the right method that works best for you.
In this article, I’m going to share everything you need to know about Ahrefsbot and the 2 best methods to block it if you need to.
What Will I Learn?
Ahrefsbot is the Ahrefs web crawler that powers the huge link index and database that Ahrefs has. Simply Ahrefsbot is at the heart of what makes Ahrefs such a powerful SEO tool.
Over the years Ahrefsbot has collected and added over 12+ trillion active links to the Ahrefs database.
You can see why Ahrefsbot is an integral part of the Ahrefs tool.
Ahrefsbot crawls as many websites as possible. It collects a range of quality link data on each crawl and sends that data back to the Ahrefs database.
Ahrefs can display accurate data to their users, providing detailed information for marketing and optimising websites.
According to Ahrefs, Ahrefsbot…
That’s a lot of power!
Curious about Ahrefs SEO tools? Check out my complete Ahrefs review.
There are a number of bots used online.
The four most common types are:
All of these bots power different applications and make them more useful for users. These are called ‘good‘ bots.
But there are also ‘bad‘ bots out there. These bots do nasty things like:
They can also use up tons of your server bandwidth and cause a lot of havoc on your site.
While Ahrefsbot is doing good things most of the time – not everything it does is perfect. There is also a potential that it could be used for bad.
Ahrefsbot crawls your website and analyses both:
This data is added to the Ahrefs ever-growing database. But what are the benefits of letting Ahrefs crawl your website?
The Ahrefsbot data offers a huge amount of value to the SEO and marketing community.
The data that it collects allows SEO’s and marketers to understand how to better optimise their websites for large search engines like Google.
Allowing Ahrefsbot to crawl your website ensures that it is always up to date in their database. You get the most accurate data on your own website and you can even compare it to your competitors.
This helps you:
As with anything, there are a few potential downsides to Ahrefsbot that you need to be aware of.
Each of these can cause unwanted issues on your site that may not be worth the risk.
No one likes a slow loading website, right?
Whenever Ahrefsbot crawls your site it loads like it would for any other person which uses up bandwidth.
If Ahrefsbot is crawling your website too regularly, it can take up too much bandwidth which means the site will load slower for other visitors.
We all know that speed is a Google ranking factor and affects user experience negatively.
Ahrefs is very strong on their claim that Ahrefsbot respects robots.txt. They have said multiple times that Ahrefsbot will follow both disallow and allow rules.
Here’s the problem:
There have been a few users that have said this isn’t true. Even after adding robots.txt to their sites, they notice they are receiving traffic from the Ahrefsbot.
Not obeying robots.txt rules is a sign of a bad bot. Red flag!
I hate spammers too. Unfortunately, they do exist and as website owners, they are something we need to be aware of.
There is nothing worse than receiving tons of unwanted spammy referral traffic that not only clogs up your Google analytics report but also affects your website performance.
Spammers can immitate the Ahrefsbot user agent to:
If you notice you are receiving tons of traffic regularly from Ahrefsbot, you might want to review and compare with the official Ahrefs IP Address ranges.
You can block or limit AhrefsBot using your robots.txt file or htaccess file. Both methods should work but take a look at each option below to see which works best for you.
Ahrefs says that Ahrefsbot follows robots.txt rules. You can block Ahrefsbot by adding new rules to your robots.txt file.
You have two options here:
To change the frequency of Ahrefsbot visiting your site, add the following code to your robots.txt file:
User-agent: AhrefsBot
Crawl-Delay: 5
Crawl-Delay value is the time of the delay recorded in seconds. You will need to change this value to delay the number of seconds between each crawl from Ahrefsbot.
Want to block Ahrefsbot completely?
To do that you will need to add a disallow tag to your robots.txt file. Simply copy and paste the code below:
User-agent: AhrefsBot
Disallow: /
That’s it! Ahrefsbot should obey the robots.txt rule you just set up.
This is a very effective method for blocking Ahrefsbot and gives you more control.
Unlike method 1 where you rely on the fact that the Ahrefsbot will obey the rules you set – with method 2, you will be taking complete control at the server level.
You will be adding the range of IP addresses that the Ahrefsbot uses and blocking it based on those IP addresses.
Here is a small example of the code you could add to your .htaccess file to block IP addresses:
Order Allow,Deny
Deny from 51.222.152.133
Deny from 54.36.148.1
Deny from 195.154.122
Allow from all
You want to make sure you block all of the current Ahrefs IP addresses that are listed here. That means you will need to edit the code above to make sure it blocks the current IP’s used by Ahrefs.
Follow this tutorial if you need help with your .htaccess file.
This is a good question.
Using the Ahrefs SEO tools?
Then don’t block the Ahrefsbot because you want to ensure your website is being updated in the Ahrefs index.
Don’t use the Ahrefs bot and it’s causing more harm than good?
Then it might be best to block it.
Ultimately – this question will help you decide on the Ahrefsbot: “Does this benefit my website or does it hurt my website?” and you can only figure that out by looking at yours sites access logs.
Ahrefsbot is the second most powerful SEO bot on the planet.
It has helped Ahrefs amass a huge link index of over 12+ trillion links and powers all of their awesome SEO tools.
But…
Ahrefsbot can in some cases, cause some issues and even worse be used by spammers that refer unwanted traffic to your site.
Use any of the 2 methods to block the Ahrefsbot if you want to and add a layer of protection for your website and don’t forget to check out my full Ahrefs review.
3 Responses
Join My Facebook Group
Subscribe to my YouTube
Connect on Linkedin
Follow My Twitter Page
Nice Matthew, tks for you reply
Hey, I was having trouble with the server since HREFS started crawling my site regularly, then I’ve used this article to block hrefs bot. After two weeks I’d changed the robots I realized the traffic was disappearing. When I double-checked I’ve noticed there is a typo in your article:User-agenCY: AhrefsBotDisallow: /To avoid other people making the same mistake I’m begging you to change the code to:User-agent: AhrefsBotDisallow: /
Good call, so sorry about that! I’ve updated the post just now.