Check if site allows Bots

Here is a small curl command command to see if a site allows Bots like Bingbot, Amazonbot, Dotbot.

curl -sILA "Bingbot"  https://www.linuxandotherstuff.com

The response should be a 200 code

HTTP/1.1 200 OK
Server: nginx
Date: Fri, 23 Feb 2024 02:17:03 GMT
Content-Type: text/html; charset=UTF-8

Leave a Comment