Identify, monitor, and mitigate automated requests with Cloudflare Bot Management, a mitigation solution based on machine learning. Bot Management is currently available to customers on the Enterprise plan.

Overview

With Cloudflare Bot Management, you can identify and mitigate scrapers and automated requests. This solution complements other Cloudflare Firewall products, including  Web Application Firewall (WAF), Rate Limiting, and IP Reputation Database. Bot Management is able to detect and block bad bots without needing JavaScript injection.

Bot Management takes advantage of machine learning across tens of millions of Internet properties proxied by Cloudflare and assigns a score to every request. Cloudflare's dynamic firewall rules can then match against malicious bot requests whenever the request's score falls below a specific threshold.

Key Cloudflare Bot Management benefits include:

• A large training set that yields more accurate bot identification: Our training set comprises millions of requests made against several million Internet properties.
• Decisions made at the edge reduce latency and result in faster performance: Since such a large number of requests traveling through the Cloudflare network get their score updated, we push the machine learning results to our edge. This way, requests can be evaluated at the edge instead of the origin. This reduces latency and prevents uncached requests from impacting server CPU at the origin.

Many different attack vectors can lead to compromised protection. Denial of service attacks, poor access controls, and SQL injection can aggravate bot-based attacks. To guard against these, security teams can design comprehensive protection with less training and context switching, using Cloudflare's consolidated firewall solution, which includes bot management.

Bot mitigation with Cloudflare Bot Management

Cloudflare Bot Management focuses on explicit bot mitigation as compared to our current WAF and rate limiting solutions. Without Bot Management, blocking a bad bot or crawler requires manually analyzing the traffic and blocking the bot IP address in the Cloudflare Firewall app. Also, the Cloudflare Web Application Firewall (WAF) is available to customers on Pro, Business, or Enterprise plans to help reduce the threat of bad bots and crawlers that exhibit bad behavior.

Currently, Bot Management focuses on mitigating:

• credential and credit card stuffing: attempts to log into and take-over a user’s account by automatically applying previously stolen account credentials
• content scraping/spam: stealing public information from a website
• marketing ad/click fraud: bots click ads and register in marketing analytics
• inventory hoarding: fraudulent purchases of goods to deprive legitimate customers or resell for a higher price

Bot management detects and blocks bad bots based on the following mitigation methods, heuristics, machine learning (ML) and behavioral analysis (BA).

Heuristics

Cloudflare uses a defined set of heuristics to efficiently identify common automated sources, such as a Python script or Javascript fingerprinting. Before making any other calculations, the incoming request must pass a number of heuristic checks. If any heuristics are triggered, the Bot Score for that request is immediately set to 1, indicating that the request came from a script, API service, or an automated agent.

Machine learning (ML)

By applying ML across millions of Internet properties, Cloudflare creates a reliable bot score that can be used to create rules for blocking requests based on the likelihood that they might come from a bot. This module will only assign Bot Scores between 2 - 99.

Behavioral analysis (BA)

Behavioral analysis detects and blocks abnormal requests based on HTTP sessions. This approach is user-agent agnostic and identifies potential bots based on actual metrics. If behavioral analysis detects abnormalities in traffic for a specific IP address, it assigns a score of 1 to all requests from that IP for a certain period of time.

Configure Bot Management

Once enabled for your Enterprise domain, activate Bot Management as follows:

1. Log in to the Cloudflare dashboard.
2. Click the Firewall app.
3. Click the Settings tab.
4. Scroll down to Bot Management, then toggle the feature to On.

Bot Management variables

After activating Bot Management, new variables are available to detect automated traffic:

• Bot Threat Score: The score Bot Management generates for the request (1 to 99).
• Verified Bot: A boolean value that is true if the request comes from a good bot (allowed by Cloudflare).
• Serves Static Resource: An identifier to match file extensions for many types of static resources.

The variables are associated with the following Cloudflare apps:

• as a field when creating Firewall Rules expression builder

• as part of the request.cf object via Cloudflare Workers
  • request.cf.botManagement.score
  • request.cf.botManagement.verifiedBot
  • request.cf.botManagement.staticResource 

Viewing Bot Management in logs

If you are exporting your raw HTTP request logs using either LogPull or LogPush, you will have access to the following two log fields related to Bot Management:

• BotScore: The bot score assigned to this request. Valid range: [1-99].
• BotScoreSrc: Underlying detection engine or source on where the bot score is calculated. Possible values: [ Not Computed | Heuristics | Machine Learning | Behavioral Analysis | Verified Bot ]

Additionally, the Cloudflare bot management _cf_bm cookie is set on your website to help with score improvement.

Frequent Bot Management questions

I enabled Bot Management in Log/Simulate mode. What data should I look for?

We recommend that you study the logged data and analyze if any of the information logged resembles your office IPs, your monitoring service, your favorite Python script, etc. allow them if necessary.

How does machine learning work?

Supervised machine learning takes certain variables (X) like gender and age and predicts another variable (Y) like income.

In Cloudflare Bot Management, the X variables are request features, while the Y variable represents the probability of solving a Captcha based on X values.

We use data from millions of requests and re-train the system on a periodic basis. You can learn about this data from your own request logs such as Cloudflare Logpull and Logpush as well as the Firewall API.

What is the difference between the threat score and bot management score?

The difference is significant:

• Threat score (cf.threat_score) is what Cloudflare uses to determine IP Reputation. It goes from 0 (good) to 100 (bad).
• Bot management score (cf.bot_management.score) is what Cloudflare uses in Bot Management to measure if the request is from a human or a script. The scores range from 1 (bot) to 99 (human). Lower scores indicate the request came from a script, API service, or an automated agent. Higher scores indicate that the request came from a human using a standard desktop or mobile web browser.

These fields are available via Cloudflare Firewall Rules.

What is cf.bot_management.verified_bot?

A request's cf.bot_management.verified_bot value is a boolean indicating whether such request comes from a Cloudflare allowed bot.

Cloudflare has built an allowlist of good, automated bots, e.g. Google Search Engine, Pingdom, and more.

This allowlist is large based on reverse DNS verification, meaning that the IPs we allow really match the requesting service. In addition to this, Cloudflare uses multiple validation methods including ASN blocks and public lists. If none of these validation types are available for a customer, we use internal Cloudflare data and machine learning to identify legitimate IP addresses from good bots.

I run a good bot and want for it to be added to the allowlist (cf.bot_management.verified_bot). What should I do?

To be added to the Cloudflare allowlist, please submit this online application.

Related resources

• Cloudflare Bot Management: machine learning and more (Cloudflare Blog)
• Stop the Bots: Practical Lessons in Machine Learning (Cloudflare Blog)
• Cloudflare Bot Fight Mode
• Cloudflare Firewall Rules (Developers Documentation)