Enterprise Log Share: Logpull REST API

Cloudflare’s Enterprise customers have access to the Enterprise Log Share service, a RESTful API that allows our customers to consume request logs over HTTP. Our REST API provides an easy method for customers to access a domain’s request logs using their existing client API key.

These logs contain data surrounding the connecting client, the request’s path through Cloudflare’s network, and the response from the origin, and are extremely useful for enriching existing logs on an origin server. 

*Note: if you are currently using the old "/logs/requests" endpoint, information about it can be found here. However, please note we will be deprecating it in early 2018. 

Overview

The "logs/received" REST API route exposes data by time received (the time the event was written to disk in our log aggregation system). This is in contrast to our "logs/requests" API documented here, which orders logs by request time. Ordering by log aggregation time instead of log generation time results in lower (faster) log share pipeline latency and deterministic log pulls. Functionally, it is similar to tailing a log file, or reading from rsyslog (albeit in chunks).

This means that if you want to obtain logs for a given time range, you can do so by issuing one call for each consecutive minute (or other time range). Because log lines are batched by time received and made available, there is no "late arriving data." A response for a given minute will never change. You do not have to repeatedly poll a given time range to receive logs as they converge on our aggregation system.

Recommended Access Pattern

Pick a time interval that works (we suggest starting with a minute for most request volumes; try a few intervals, if the interval returns too much data to easily handle, scale down), and then make requests with that fixed interval (not setting "count") for the whole time range in which you are interested. Once you read an entire 200 response for a request, you do not need to hit that url again.

Access

cURL example from burritobot.com

https://api.cloudflare.com/client/v4/zones/<zone_tag>/logs/received?start=<unix|rfc3339>&end=<unix|rfc3339>[&count=<int>][&sample=<float>][&fields=<fields>]

  • start: inclusive, either unix timestamp (which by definition is UTC) or time in rfc3339 (specifies time zone); must be at least 5 minutes in the past.
  • end: exclusive, format same as start.
  • count: return up to that many records; empty or negative means return all records.
  • sample: return only a sample of records; sample=0.1 means return 10% (1 in 10) of records
  • fields: comma separated list of fields to return; when empty default list is returned.
  • headers:
    • X-Auth-Email
    • X-Auth-Key


curl -s -H"X-Auth-Email: [email protected]" -H"X-Auth-Key: banana12345" "https://api.cloudflare.com/client/v4/zones/4f8g90r42275bce56f242596aic6fc5b/logs/received?start=2017-07-18T22:00:00Z&end=2017-07-18T22:01:00Z&count=1&fields=RayID,ClientIP"

 

Expectations

Once a 200 response is received and read completely for a given zone and time range, the following will be true for all subsequent requests:

    • The number and content of returned records will be the same.
    • The order of returned records may be (and is likely to) be different.

Requests will fail when:

    • There are more than 5 requests to the endpoint for a given zone currently in progress.
    • There is more than 1GB of uncompressed data in the response. Because responses are streamed, there is no way to determine response size ahead of time. After 1GB of data is streamed, the response will fail with a terminated connection. This allows users to tune their queries to a range that will not result in failures based on their request volume. We believe this is a better solution than hard-limiting requests to an arbitrary time range (e.g., 1 minute at a time).

Response fields:

    • When set explicitly in the request url, the response fields will never change.
    • When not set explicitly, default fields will be used; default fields may change at any time.

API Parameters In Depth

count

When "?count=" is provided, the response will contain up to count results. Because results are not sorted, you are likely to get different data for repeated requests.

sample

When "?sample=" is provided, a sample of matching records is returned. If "sample=0.1", approximately 10% of records will be returned. Sampling is random: repeated calls will not only return different records, but likely will also vary slightly in number of returned records.

When "?count=" is also specified, count is applied to the number of returned records, not the sampled records. So, with "sample=0.05" and "count=7", when there is a total of 100 records available, approximately 5 will be returned. When there are 1,000 records, 7 will be returned. When there are 10,000 records, 7 will be returned.

fields

If "fields" are not specified, by default a limited set of fields will be returned. This default field set may change at any time. The full list of all available fields can be found here:

https://api.cloudflare.com/client/v4/zones/<zone_tag>/logs/received/fields

Fields are passed to the request as a comma separated list. So, to have "ClientIP" and "RayID", use:

fields=ClientIP,RayID

The order in which fields are specified doesn't matter, and the order of fields in the response is not specified. If you need fields added to the list of available fields, please contact your account manager.

Example of fields available:

{
"CacheCacheStatus": "unknown | miss | expired | updating | stale | hit | ignored | bypass | revalidated",
"CacheResponseBytes": "Number of bytes returned by the cache",
"CacheResponseStatus": "HTTP status code returned by the cache to the edge: all requests (including non-cacheable ones) go through the cache: also see CacheStatus field",
"ClientASN": "Client AS number",
"ClientCountry": "Country of the client IP address",
"ClientDeviceType": "Client device type",
"ClientIP": "IP address of the client",
"ClientIPClass": "Client IP class",
"ClientRequestBytes": "Number of bytes in the client request",
"ClientRequestHost": "Host requested by the client",
"ClientRequestMethod": "HTTP method of client request",
"ClientRequestProtocol": "HTTP protocol of client request",
"ClientRequestReferer": "HTTP request referrer",
"ClientRequestURI": "URI requested by the client",
"ClientRequestUserAgent": "User agent reported by the client",
"ClientSSLCipher": "Client SSL cipher",
"ClientSSLProtocol": "Client SSL protocol",
"ClientSrcPort": "Client source port",
"EdgeColoID": "Cloudflare edge colo id",
"EdgeEndTimestamp": "Unix nanosecond timestamp the edge finished sending response to the client",
"EdgePathingStatus": "Edge pathing status",
"EdgeResponseBytes": "Number of bytes returned by the edge to the client",
"EdgeResponseCompressionRatio": "Edge response compression ratio",
"EdgeResponseStatus": "HTTP status code returned by Cloudflare to the client",
"EdgeStartTimestamp": "Unix nanosecond timestamp the edge received request from the client",
"OriginIP": "IP of the origin server",
"OriginResponseBytes": "Number of bytes returned by the origin server",
"OriginResponseHTTPExpires": "Value of the origin 'expires' header in RFC1123 format",
"OriginResponseHTTPLastModified": "Value of the origin 'last-modified' header in RFC1123 format",
"OriginResponseStatus": "Status returned by the origin server",
"OriginResponseTime": "Number of nanoseconds it took the origin to return the response to edge",
"RayID": "Ray ID of the request",
"WAFAction": "Action taken by the WAF, if triggered",
"WAFRuleID": "ID of the applied WAF rule",
"ZoneID": "Internal zone ID"
}

Response

Data is timestamped by time it was written to disk at our log aggregation point.

Response data is returned in json, 1 json object (1 log message) per line. Sample log message with default fields:

{
"ClientIP": "89.163.242.206",
"ClientRequestHost": "www.theburritobot.com",
"ClientRequestMethod": "GET",
"ClientRequestURI": "/static/img/testimonial-hipster.png",
"EdgeEndTimestamp": 1506702504461999900,
"EdgeResponseBytes": 69045,
"EdgeResponseStatus": 200,
"EdgeStartTimestamp": 1506702504433000200,
"RayID": "3a6050bcbe121a87"
}

Still not finding what you need?

The Cloudflare team is here to help. 95% of questions can be answered using the search tool, but if you can’t find what you need, submit a support request.

Powered by Zendesk