To date, Cloudflare has offered access to raw HTTP logs for all requests passing through our edge network. Historically, HTTP logs have been accessed via a RESTful API endpoint documented here and presented in JSON format. This endpoint is commonly referred to as the “/requests” endpoint and returns a set of ~150 fields per request.
The "/requests" endpoint exposes data aggregated by time of request and can be queried to deliver logs starting from a specific unix timestamp, a set of logs starting from a RayID (an internal unique log identifier), and a single log derived from a RayID.
Cloudflare is planning to sunset access to the "/requests" endpoint at the end of Q1 2018 and require calls to be made to the new “/received” API endpoint, which was released in August 2017 and is documented here. The new endpoint returns data based on the time that Cloudflare’s data processing system received the logs.
There are many benefits to using the new "/received" endpoint, including:
- Reliability: the new endpoint is an order of magnitude more reliable than the previous one; customers should see a negligible rate of errors and timeouts
- Determinism: a request for data for a certain time period will always return the same results. Previously, because the data was organized by the time of the request and data could be delayed in processing, customers could never be sure that they had obtained all the logs for a given period.
- Control over data volume: customers can specify the log fields they'd like to receive, reducing bandwidth and storage costs. By default, only a small set of fields is returned (~10 fields). Previously, the full set of fields was returned (~150 fields) and there was no option to limit it.
- Sampling: ability to request a specified random percentage of logs (e.g., 10%) for a given time period
Typically a Cloudflare customer will use ELS to:
- Continually draw down logs to create visualizations/alerts.
- This will require migrating to the new endpoint. Please see below migration steps to plan your transition.
- Less commonly, a customer will pull down single events based on a RayID
- This functionality will be deprecated with the new endpoint
- At a high level, this is a computationally expensive and inefficient query that can negatively affect service reliability
- We are trading off this lookup functionality for a more reliable and robust service that our customers can count on
To manage the migration process, Cloudflare suggests the following.
First, understand the schema of the new /received endpoint, available from the API:
curl -H “X-Auth-Key: apikey” -H “X-Auth-Email: [email protected]” "https://api.cloudflare.com/client/v4/zones/:zone_id/logs/received/fields"
The schema of all available fields is also linked at the bottom of this document under “received_schema.” The default schema does not include all fields available and is linked under “default_received_schema.” (Please use the API call above to get the latest schema).
Once you are familiar with the new schema, we suggest finding the fields of interest and mapping them to the fields from the /requests endpoint which are important to you. The /requests schema is also included below under "legacy_schema."
You will need to update any dashboards and scripts to use the new /received endpoint and use the fields parameter to request the fields you need for your log analysis.