Back to blog
Block Unwanted Bots by User-Agent

Some bots are helpful, like Googlebot indexing your pages, but others can overload your server, steal your content, or harm SEO.
Now, the Htaccess by BestWebSoft plugin makes it easy to block unwanted bots directly from your WordPress admin panel — no manual .htaccess editing needed.

How the Feature Works

In the plugin settings, you’ll find a new User-Agent Blocking field.
Here you can enter the names of bots you want to block — one per line.
The plugin will automatically add the correct rules to your .htaccess file.

Example:

BrowserMatchNoCase "AhrefsBot" bad_bot

Order Allow,Deny

Allow from all

Deny from env=bad_bot

This tells your server to block any request from a bot whose User-Agent contains “AhrefsBot”.


Adding Bots to Block

  1. Go to Htaccess → Settings in your WordPress dashboard.
  2. Find the Block by User-Agent section.

Enter the bot names. Example:

AhrefsBot

SemrushBot

MJ12bot
  1. Save changes — the plugin updates your .htaccess file automatically.

Example of Settings Page

block by user agent

Why Use User-Agent Blocking

  • Protect server resources — block bots that send too many requests.
  • Prevent content scraping — stop automated tools from copying your site.
  • Improve security — keep suspicious crawlers away.

Conclusion

With the new User-Agent Blocking feature in the Htaccess by BestWebSoft plugin, you can easily stop unwanted bots from crawling your site without touching any code. Simply enter the bot names in the settings, and the plugin will handle the rest.

FAQ: Blocking Bots by User-Agent in WordPress

What is a User-Agent?

A User-Agent is a string sent by a browser or bot to identify itself when visiting a website. It usually contains information about the software, version, and device. For example:
Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)

Why would I want to block a bot by User-Agent?

Some bots can overload your server, waste bandwidth, or scrape your content without permission. Blocking them helps reduce unwanted traffic and improve website performance.

Will blocking a bot affect my SEO?

It depends. You should only block bots you are sure are harmful or unwanted. Blocking search engine crawlers like Googlebot or Bingbot may negatively affect your SEO.

How do I know which bots to block?

You can check your server logs, use analytics tools, or monitor suspicious activity. Common examples of bots people block include:

  • Baiduspider
  • AhrefsBot
  • MJ12bot
  • SemrushBot

Is blocking by User-Agent 100% effective?

No. Some bots can fake (spoof) their User-Agent, so advanced protection may require additional security rules like IP blocking or CAPTCHA.

Can I unblock a bot later?

Yes. You can remove its User-Agent string from the plugin settings at any time, and it will be allowed again immediately.



Like This Article? Subscribe to Our Monthly Newsletter!

Comments are closed.