Security: Restrict a specific Bot

Wappler General Need Help
1

Is there a Wappler way to prevent a specific bot from crawling your site? I have an issue with a bot from jooble.org that is killing my server resources.

I know I can set robots to nofollow but I don’t want to restrict google.

Anyway in Wappler to block Jooble?

1 Like
2

Ended up trying to do this with the .htaccess file.

## Block via User Agent
<IfModule mod_rewrite.c>
	RewriteEngine On
	RewriteCond %{HTTP_USER_AGENT} (JoobleBot) [NC]
	RewriteRule (.*) - [F,L]
</IfModule>

Hopefully that works. Would still like to know if there is a Wappler way. :wink:

3

There’s no a “Wappler way” of adding rules in your htaccess. You just edit it manually, Wappler is not related to this.

1 Like
4

There is a special robots.txt file that you can place in your website root, it tells the bot/crawler which urls it can access and which not.

What is robots.txt? | How a robots.txt file works | Cloudflare

1 Like