When does Cloudflare crawl my site?

To ensure your site is protected and performing properly, Cloudflare may crawl or make HTTP requests to your site. More detail about why and when below.

Cloudflare may crawl your site when:

1. Always Online is enabled. The Always Online feature caches a static version of your site if your server goes down. The user agent we use is:

Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online)

We crawl sites based on the Cloudflare pricing plan for the domain.

  • Free customers once every 14 days
  • Pro customers once every 6 days
  • Business and Enterprise customers every other day.

2. Conducting a Speed Test. Speed tests measure performance improvement of your site on Cloudflare. The user agent we use is:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36 PTST/190628.140653
3. Our support engineers use Cloudflare diagnostics to perform error checks and to raise intelligent alerts in the Cloudflare dashboard for your site. The user agent we use is:
4. When Load Balancing is enabled. We use Load Balancing to perform health checks on your origin server. The user agent we use is:
Mozilla/5.0 (compatible; Cloudflare-Traffic-Manager/1.0; +https://www.cloudflare.com/traffic-manager/; pool-id: $poolid)
Not finding what you need?

95% of questions can be answered using the search tool. This is the quickest way to get a response.