Identify, monitor, and mitigate automated requests with Cloudflare Bot Management, a mitigation solution based on machine learning. Bot Management is currently available to customers on the Enterprise plan.
About Cloudflare Bot Management
With Cloudflare Bot Management, you can identify and mitigate scrapers and automated requests. This solution complements other Cloudflare Firewall products, including Web Application Firewall (WAF), Rate Limiting, and IP Reputation Database. Bot Management is able to detect and block bad bots without needing JavaScript injection.
Bot Management takes advantage of machine learning across tens of millions of Internet properties proxied by Cloudflare and assigns a score to every request. Cloudflare's dynamic firewall rules can then match against malicious bot requests whenever the request's score falls below a specific threshold.
Key Cloudflare Bot Management benefits include:
- A large training set that yields more accurate bot identification: Our training set comprises millions of requests made against several million Internet properties.
- Decisions made at the edge reduce latency and result in faster performance: Since such a large number of requests traveling through the Cloudflare network get their score updated, we push the machine learning results to our edge. This way, requests can be evaluated at the edge instead of the origin. This reduces latency and prevents uncached requests from impacting server CPU at the origin.
Many different attack vectors can lead to compromised protection. Denial of service attacks, poor access controls, and SQL injection can aggravate bot-based attacks. To guard against these, security teams can design comprehensive protection with less training and context switching, using Cloudflare's consolidated firewall solution, which includes bot management.
Differences between Bot Management and other Cloudflare bot mitigation tools
Cloudflare Bot Management focuses on explicit bot mitigation as compared to our current WAF and rate limiting solutions. Without Bot Management, blocking a bad bot or crawler requires manually analyzing the traffic and blocking the bot IP address in the Cloudflare Firewall app. Also, the Cloudflare Web Application Firewall (WAF) is available to customers on Pro, Business, or Enterprise plans to help reduce the threat of bad bots and crawlers that exhibit bad behavior.
Currently, Bot Management focuses on mitigating:
- Credential and credit card stuffing,
- content scraping, and
- other types of deception including: spam, registration, marketing, and ad-click fraud.
Bot management detects and blocks bad bots based on the following mitigation methods:
Machine learning (ML)
By applying ML across millions of Internet properties, Cloudflare creates a reliable bot score that can be used to create rules for blocking requests based on the likelihood that they might come from a bot.
Behavioral analysis (BA)
Behavioral analytics detect and block abnormal requests based on HTTP sessions. This approach is user-agent agnostic and identifies potential bots based on actual metrics.
JavaScript injection
If you prefer additional independent protections, you can apply JavaScript inspection to suspicious traffic, but it's not required.
You can combine or use the above methods separately.
The most notable advantages of Bot Management over the standard Cloudflare solution are:
- More reliable bot identification based on a large training set, and
- faster performance with decisions made closer to the client.
Activating Bot Management
Once enabled for your Enterprise domain, activate Bot Management as follows:
- Log in to the Cloudflare dashboard.
- Click the Firewall app.
- Click the Settings tab.
- Scroll down to Bot Management, then toggle the feature to On.
After activating Bot Management, new variables are available to detect automated traffic:
- Bot Threat Score: The score Bot Management generates for the request (1 to 99).
- Verified Bot: A boolean value that is true if the request comes from a good bot (whitelisted by Cloudflare).
- Static Resource: An identifier to match file extensions for many types of static resources.
The variables are associated with the following Cloudflare apps:
- as a field in Firewall Rules, and
- cf.bot_management.score
- cf.bot_management.verified_bot
- cf.bot_management.static_resource
- as part of the request.cf object via Cloudflare Workers
- request.cf.botManagement.score
- request.cf.botManagement.verifiedBot
- request.cf.botManagement.staticResource
Additionally, the Cloudflare bot management _cf_bm cookie is set on your website to help with score improvement.
Frequent Bot Management questions
I enabled Bot Management in Log/Simulate mode. What data should I look for?
We recommend that you study the logged data and analyze if any of the information logged resembles your office IPs, your monitoring service, your favorite Python script, etc. White list them if necessary.
How does machine learning work?
Supervised machine learning takes certain variables (X) like gender and age and predicts another variable (Y) like income.
In Cloudflare Bot Management, the X variables are request features, while the Y variable represents the probability of solving a Captcha based on X values.
We use data from millions of requests and re-train the system on a periodic basis. You can learn about this data from your own request logs such as Cloudflare Logpull and Logpush as well as the Firewall API.
What is the difference between the threat score and bot management score?
The difference is significant:
- Threat score (cf.threat_score) is what Cloudflare uses to determine IP Reputation. It goes from 0 (good) to 100 (bad).
- Bot management score (cf.bot_management.score) is what Cloudflare uses in Bot Management to measure if the request is from a human or a script. The scores range from 1 (bot) to 99 (human). Lower scores indicate the request came from a script, API service, or an automated agent. Higher scores indicate that the request came from a human using a standard desktop or mobile web browser.
These fields are available via Cloudflare Firewall Rules.
What is cf.bot_management.verified_bot?
A request's cf.bot_management.verified_bot value is a boolean indicating whether such request comes from a Cloudflare white listed bot.
Cloudflare has built a white list of good, automated bots, e.g. Google Search Engine, Pingdom, and more.
This white list is large based on reverse DNS verification, meaning that the IPs we whitelist really match the requesting service. In addition to this, Cloudflare uses multiple validation methods including ASN blocks and public lists. If none of these validation types are available for a customer, we use internal Cloudflare data and machine learning to identify legitimate IP addresses from good bots.
I run a good bot and want for it to be added to the white list (cf.bot_management.verified_bot). What should I do?
To be added to the Cloudflare white list, please submit this online application.
Related resources
- Stop the Bots: Practical Lessons in Machine Learning (Cloudflare Blog)
- Cloudflare Bot Fight Mode
- Cloudflare Firewall Rules (Developers Documentation)