To ensure your site is protected and performing properly, Cloudflare may crawl or make HTTP requests to your site. More detail about why and when below.
Cloudflare may crawl your site when:
1. Always Online is enabled. The Always Online feature caches a static version of your site if your server goes down. The user agent we use is:
Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online)
We crawl sites based on the Cloudflare pricing plan for the domain.
- Free customers once every 14 days
- Pro customers once every 6 days
- Business and Enterprise customers every other day.
2. Conducting a Speed Test. Speed tests measure performance improvement of your site on Cloudflare. The user agent we use is:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36 PTST/190628.140653
Mozilla/5.0 (compatible; Cloudflare-Traffic-Manager/1.0; +https://www.cloudflare.com/traffic-manager/; pool-id: $poolid)