When does Cloudflare crawl my site?

To ensure your site is protected and performing properly, Cloudflare may crawl your site. More detail about why and when below.


Cloudflare may crawl your site when:

1. Always Online is enabled. The Always Online feature caches a static version of your site if your server goes down. The useragent we use is:

Mozilla/5.0 (compatible; Cloudflare AlwaysOnline/1.0;+http://www.cloudflare.com/always-online

We crawl sites based on the Cloudflare pricing plan for the domain.

  • Free customers once every 7 days
  • Pro customers once every 3 days
  • Business and Enterprise customers daily.

2. Conducting a Speed Test. Speed tests measure performance improvement of your site on Cloudflare. The useragent we use is:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36 PTST/190628.140653
3. Our support engineers use Cloudflare diagnostics to perform error checks and to raise intelligent alerts in the Cloudflare dashboard for your site. The useragent we use is:
Cloudflare-diagnostics
4. When Load Balancing is enabled. We use Load Balancing to perform health checks on your origin server. The useragent we use is:
Mozilla/5.0 (compatible; Cloudflare-Traffic-Manager/1.0; +https://www.cloudflare.com/traffic-manager/; pool-id: $poolid)
Not finding what you need?

95% of questions can be answered using the search tool. This is the quickest way to get a response.

Powered by Zendesk