How to BLOCK Bingbot to save crawl budget
One day, our team started getting alerts that our client's website doesn't work. It was slow for customers.
We looked at the activity logs and saw that Bingbot was hammering the site:
Roughly 30% Users, 50% Bing, 15% to Google and 5% Other.
If it was my personal site, the solution would've been simple - just block the Bingbot user agent!
Who uses Bing anyway, right?
Well, people do β About 3%.
Our site was an eCommerce site that maximized SEO. Google was king, but there was still some $$ coming from Bing.
We had to find balance.
How to slow down the Bingbot
According to Microsoft, there are four settings that webmasters can use to control Bingbot's crawl speed:
Crawl-delay setting | Speed |
---|---|
Not set | Normal π |
1 | Slow |
5 | Very slow |
10 | Extremely slow |
We decided to slow down Bingbot's crawling, but still keep it present.
So we simply added this two-liner to the robots.txt
file:
User-agent: bingbot
Crawl-delay: 10
Crawl-delay: 10
directive tells Bingbot to crawl the site "Extremely slowly."
Couple of days later, Bingbot listened:
Roughly 35% Users, 2% Bing, 60% to Google and 2% Other.
Get Google to crawl your site more
Great, Bingbot is down! But isn't the googlebot hammering the site now?
Googlebot is unique amongst crawlers because Google pays attention to load that it's putting on your server:
- If the site responds quickly, the crawl limit goes upβ
- If the site slows down or responds with a lot of HTTP 500 errors, the limit goes down, and Googlebot crawls lessβ
Even though it was hitting the site more now, we didn't have trouble with the HTTP 500 errors that Bingbot was causing.
Control Google's Crawl Budget for your advantage
Slowing down Bingbot tremendously increased our crawl budget with Google.
Crawl budget is the number of pages Google will crawl on your site on any given day.β
More crawling won't necessarily make you rank better on SERP, but it will give you better visibility by making sure crawlers get to your content quicker.
Increased visibility was an advantage for us because we knew we had great content that will rank well.
How to Block Bingbot
If you want to completely block Bingbot from crawling your site, add following lines to your robots.txt
file:
User-agent: bingbot
Disallow: /
- Published on
Related Posts
- C# Tuple: Complete Guide
- C# String Length
- REST API: PATCH vs. PUT
- String Enums in C#: Everything You Need to Know
- 10 Techniques for Estimating Software Projects
- Scrum for Software Developers: Why It's Worth Understanding
- How To Improve Cumulative Layout Shift Score
- How to Update npm Packages In 4 Easy Steps