At the moment AI and their malicious bots are the scourge of the internet. They ignore all the conventions and implicit rules built on trust over many years. Cloudflare has called out Perplexity on this abuse at Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives. You can also read more on the impact of these bots at The crawl before the fall… of referrals: understanding AI’s impact on content providers. Read more

Updated 23 February 2021 Cloudflare is a Content Delivery Network (CDN) service with a free basic offering. To provide the content delivery service they need to manage your DNS and provide a very nice interface to set up DNS. You can use the free DNS service without making use of the rest of their services. If you use Cloudflare for DNS it is adviseable to use their reverse proxy (CDN) service. It really speeds up your site and also provides some protection against attempts to hack your site. If you use a system like Wordpress for your site you will find many articles on the web to assist in setting up Cloudflare correctly e.g. how to stop /wp-admin from being cached. Read more