Description of updates to Cloudflare Logs to be released on the 22nd of February 2021

Overview

On the 22nd of February 2021, we will deploy a number of changes to Cloudflare Logs. These updates will affect both the HTTP Requests and Firewall Events datasets.

If you are ingesting our logs you may need to update your SIEM and other log tooling.

The changes are meant to fix a number of issues and inconsistencies in our log formats as well as improving some of our language to be more accurate and inclusive:

1. OWASP log rules will only be visible in Firewall Events metadata;
2. OWASP anomaly_score will be renamed to score_total and xss_score and sqli_score will no longer be available;
3. drop action will become block;
4. simulate action will become log;
5. whitelist action will become allowlist;
6. sanitycheck source will become validation;

Details on each change

Please find below details on each change:

1. OWASP log rules will only be visible in Firewall Events metadata

Any log action triggered by a rule in the WAF OWASP Ruleset currently appears as an individual action listed FirewallMatchesActions. These events will now only appear in the Metadata field of the Firewall Events dataset and no longer be shown in HTTP Requests dataset.

For example, an HTTP request log line as retrieved from Cloudflare logs that triggered one or more rules in WAF OWASP Ruleset, will be changed from:

{"ClientIP":"xxx.xxx.xxx.xxx","ClientRequestHost":"www.example.com","ClientRequestPath":"/","ClientRequestUserAgent":"curl/7.64.1","__FirewallMatchesActions__":["log","log","log","log","log","log","drop"],"__FirewallMatchesRuleIDs__":["973336","958052","958051","973300","973307","973331","981176"],"__FirewallMatchesSources__":["waf","waf","waf","waf","waf","waf","waf"],"WAFAction":"drop","WAFFlags":"0","WAFMatchedVar":"","WAFProfile":"high","WAFRuleID":"981176","WAFRuleMessage":"Inbound Anomaly Score Exceeded (Total Score: 30, SQLi=0, XSS=30)"}

To:

{"ClientIP":"xxx.xxx.xxx.xxx","ClientRequestHost":"www.example.com","ClientRequestPath":"/","ClientRequestUserAgent":"curl/7.64.1","__FirewallMatchesActions__":["drop"],"__FirewallMatchesRuleIDs__":["981176"],"__FirewallMatchesSources__":["waf"],"WAFAction":"drop","WAFFlags":"0","WAFMatchedVar":"","WAFProfile":"high","WAFRuleID":"981176","WAFRuleMessage":"Inbound Anomaly Score Exceeded (Total Score: 30, SQLi=0, XSS=30)"

The OWASP rule actions are also going to change from log to score. Currently, a request that triggers the OWASP ruleset will appear like so in the Firewall Events log (one line per rule matching):

{"RayName":"5eff3240acbd067e","Source":"waf","RuleId":"973336","Action":"log","EdgeColoName":"LHR","ClientIP":"xxx.xxx.xxx.xxx","ClientCountryName":"gb","UserAgent":"curl/7.64.1","ClientRequestHTTPMethodName":"GET","ClientRequestHTTPHost":"www.example.com"}
{"RayName":"5eff3240acbd067e","Source":"waf","RuleId":"958052","Action":"log","EdgeColoName":"LHR","ClientIP":"xxx.xxx.xxx.xxx","ClientCountryName":"gb","UserAgent":"curl/7.64.1","ClientRequestHTTPMethodName":"GET","ClientRequestHTTPHost":"www.example.com"}
{"RayName":"5eff3240acbd067e","Source":"waf","RuleId":"958051","Action":"log","EdgeColoName":"LHR","ClientIP":"xxx.xxx.xxx.xxx","ClientCountryName":"gb","UserAgent":"curl/7.64.1","ClientRequestHTTPMethodName":"GET","ClientRequestHTTPHost":"www.example.com"}
{"RayName":"5eff3240acbd067e","Source":"waf","RuleId":"973300","Action":"log","EdgeColoName":"LHR","ClientIP":"xxx.xxx.xxx.xxx","ClientCountryName":"gb","UserAgent":"curl/7.64.1","ClientRequestHTTPMethodName":"GET","ClientRequestHTTPHost":"www.example.com"}
{"RayName":"5eff3240acbd067e","Source":"waf","RuleId":"973307","Action":"log","EdgeColoName":"LHR","ClientIP":"xxx.xxx.xxx.xxx","ClientCountryName":"gb","UserAgent":"curl/7.64.1","ClientRequestHTTPMethodName":"GET","ClientRequestHTTPHost":"www.example.com"}
{"RayName":"5eff3240acbd067e","Source":"waf","RuleId":"973331","Action":"log","EdgeColoName":"LHR","ClientIP":"xxx.xxx.xxx.xxx","ClientCountryName":"gb","UserAgent":"curl/7.64.1","ClientRequestHTTPMethodName":"GET","ClientRequestHTTPHost":"www.example.com"}
{"RayName":"5eff3240acbd067e","Source":"waf","RuleId":"981176","Action":"drop","EdgeColoName":"LHR","ClientIP":"xxx.xxx.xxx.xxx","ClientCountryName":"gb","UserAgent":"curl/7.64.1","ClientRequestHTTPMethodName":"GET","ClientRequestHTTPHost":"www.example.com"}

This will change to:

{"RayName":"5eff3240acbd067e","Source":"waf","RuleId":"981176","Action":"drop","EdgeColoName":"LHR","ClientIP":"xxx.xxx.xxx.xxx","ClientRequestHTTPMethodName":"GET","ClientRequestHTTPHost":"www.example.com","__Metadata__":{"score_total":"5","__score_rules__":"[\"973336\",\"958052\",\"958051\",\"973300\",\"973307\",\"973331\"]"}}

In the example above, the OWASP rules that contribute to the score for the given request are no longer represented as single log lines, rather they are shown as rules that contributed to the score in the Metadata field. Note that the rule_message will also no longer be available for each individual rule that contributed to the total score. Other rules that use the log action will still be represented as individual log lines.

This change will affect:

• The FirewallMatchesActions, FirewallMatchesRuleIDs, FirewallMatchesSources fields in HTTP Requests dataset (as shown above)
• In the Firewall Events dataset Individual log lines for rule log events will be collapsed into a single log line for the OWASP ruleset. The information will now be stored in the Metadata field. Individual rule_message fields will no longer be available directly in the logs

2. OWASP anomaly_score will be renamed to score_total and xss_score and sqli_score will no longer be available

The anomaly_score field will be renamed to score_total and the sqli_score and xss_score fields will no longer be available.

3. drop action will become block

For example, an HTTP request log line as retrieved from Cloudflare logs that triggered a Firewall Rule will be changed from:

{"ClientIP":"xxx.xxx.xxx.xxx","ClientRequestHost":"www.example.com","ClientRequestPath":"/","ClientRequestUserAgent":"curl/7.64.1","__FirewallMatchesActions__":["drop"],"FirewallMatchesRuleIDs":["8c1d89f2237e4daca75c4b3ce2d9881b"],"FirewallMatchesSources":["firewallRules"]}

To:

{"ClientIP":"xxx.xxx.xxx.xxx","ClientRequestHost":"www.example.com","ClientRequestPath":"/","ClientRequestUserAgent":"curl/7.64.1","__FirewallMatchesActions__":["block"],"FirewallMatchesRuleIDs":["8c1d89f2237e4daca75c4b3ce2d9881b"],"FirewallMatchesSources":["firewallRules"]}

This change will affect:

• The FirewallMatchesActions, WAFAction, EdgeRateLimitAction fields in HTTP Requests dataset;
• The action field in Firewall Events dataset;

4. simulate action will become log

This change will affect various security products. For example, an HTTP request log line as retrieved from Cloudflare logs that triggered a Firewall Rule will be changed from:

{"ClientIP":"xxx.xxx.xxx.xxx","ClientRequestHost":"www.example.com","ClientRequestPath":"/","ClientRequestUserAgent":"curl/7.64.1","__FirewallMatchesActions__":["simulate"],"FirewallMatchesRuleIDs":["8c1d89f2237e4daca75c4b3ce2d9881b"],"FirewallMatchesSources":["firewallRules"]}

To:

{"ClientIP":"xxx.xxx.xxx.xxx","ClientRequestHost":"www.example.com","ClientRequestPath":"/","ClientRequestUserAgent":"curl/7.64.1","__FirewallMatchesActions__":["log"],"FirewallMatchesRuleIDs":["8c1d89f2237e4daca75c4b3ce2d9881b"],"FirewallMatchesSources":["firewallRules"]}

As for another example, an HTTP request log line as retrieved from Cloudflare logs that triggered a Rate Limiting Rule will be changed from:

{"ClientIP":"xxx.xxx.xxx.xxx","ClientRequestHost":"www.example.com","ClientRequestPath":"/","ClientRequestUserAgent":"curl/7.64.1","__EdgeRateLimitAction__":"simulate","__FirewallMatchesActions__":["simulate"],"FirewallMatchesRuleIDs":["1f6108ad12a94053bca6cea94e77bc8a"],"FirewallMatchesSources":["rateLimit"],"WAFAction":"unknown","WAFFlags":"0","WAFMatchedVar":"","WAFProfile":"unknown","WAFRuleID":"","WAFRuleMessage":""}

To:

{"ClientIP":"xxx.xxx.xxx.xxx","ClientRequestHost":"www.example.com","ClientRequestPath":"/","ClientRequestUserAgent":"curl/7.64.1","__EdgeRateLimitAction__":"log","__FirewallMatchesActions__":["log"],"FirewallMatchesRuleIDs":["1f6108ad12a94053bca6cea94e77bc8a"],"FirewallMatchesSources":["rateLimit"],"WAFAction":"unknown","WAFFlags":"0","WAFMatchedVar":"","WAFProfile":"unknown","WAFRuleID":"","WAFRuleMessage":""}

This change will affect:

• The FirewallMatchesActions and EdgeRateLimitAction fields in HTTP Requests dataset (as shown above)
• The action field in Firewall Events dataset

5. whitelist action will become allowlist

This change will affect:

• The ClientIPClass field in HTTP Requests dataset
• The ClientIPClass field in Firewall Events dataset

6. sanitycheck will become validation

This change will affect:

• The FirewallMatchesSources field in HTTP Requests dataset
• The Source field in Firewall Events dataset

Update your SIEMs

Once these changes are applied, you may need to update any external filters or reports to reflect the new values provided by Cloudflare.