bot_verifier
The bot_verifier directive enables or disables the bot verification module for NGINX, allowing or blocking access for requests claiming to be search engine bots.
Description
The bot_verifier directive is part of a NGINX module designed to enhance server security by verifying that incoming requests claiming to be search engine index bots are legitimate. This is crucial since the User-Agent header can easily be spoofed, making it impossible to trust it as an indicator of the request's authenticity. When this directive is set to 'on', the module engages in a validation process that involves checking the request against a cached record of verified bots, stored typically in a Redis database. If the Redis connection is not established or fails, the module will decline the verification process by returning NGX_DECLINED, allowing the request to continue without validation
The verification process entails determining the originating address of the request, checking it against a cache for verification status, and if necessary, conducting a deeper verification. The system can also store new verification results to the cache for future requests. In case a request is found to be legitimate, the server allows it to proceed; conversely, if the verification fails, the request is blocked, returning a HTTP_FORBIDDEN response. This mechanism minimizes performance costs associated with constant real-time verification by caching the results of prior checks, optimizing the system's response times for legitimate requests.
Config Example
location / {
bot_verifier on;
bot_verifier_redis_host localhost;
bot_verifier_redis_port 6379;
bot_verifier_redis_connection_timeout 10;
bot_verifier_redis_read_timeout 10;
bot_verifier_redis_expiry 3600;
bot_verifier_repsheet_enabled on;
}Ensure Redis is correctly setup and reachable; otherwise, validation will fail and requests may be allowed without checking.
Remember to enable the bot_verifier directive in the appropriate context to apply it (http, server, location).
Misconfigured timeouts can lead to blocked requests if Redis is slow or unresponsive.