To date, Cloudflare has offered access to raw HTTP logs for all requests passing through our edge network. Historically, HTTP logs have been accessed via a RESTful API endpoint documented here and presented in JSON format. This endpoint is commonly referred to as the “/requests” endpoint and returns a set of ~130 fields per request.
The "/requests" endpoint exposes data aggregated by time of request and can be queried to deliver logs starting from a specific unix timestamp, a set of logs starting from a RayID (an internal unique log identifier), and a single log derived from a RayID.
Last year, we deprecated the Log Share “/requests” endpoint and scheduled it to be turned off in mid-March of this year. After an additional grace period, this endpoint will be turned off on May 31st. Users will be required to call the new “/received” endpoint, which was released in August 2017 and is documented here. The new endpoint returns data based on the time that Cloudflare’s data processing system received the logs.
There are many benefits to using the new "/received" endpoint, including:
- Reliability: the new endpoint is an order of magnitude more reliable than the previous one; customers should see a negligible rate of errors and timeouts
- Determinism: a request for data for a certain time period will always return the same results. Previously, because the data was organized by the time of the request and data could be delayed in processing, customers could never be sure that they had obtained all the logs for a given period.
- Control over data volume: customers can specify the log fields they'd like to receive, reducing bandwidth and storage costs. By default, only a small set of fields is returned (~10 fields). Previously, the full set of fields was returned (~130 fields) and there was no option to limit it.
- Sampling: ability to request a specified random percentage of logs (e.g., 10%) for a given time period
Typically a Cloudflare customer will use Log Share to:
- Continually draw down logs to create visualizations/alerts.
- This will require migrating to the new endpoint. Please see below migration steps to plan your transition.
- Less commonly, a customer will pull down single events based on a RayID
- This will require using the following new endpoint: https://api.cloudflare.com/client/v4/zones/<zone_id>/logs/rayids/<ray_id>?[&fields=<string>][×tamps=<strings>]
To manage the migration process, Cloudflare suggests the following.
First, understand the schema of the new /received endpoint, available from the API:
curl -H “X-Auth-Key: apikey” -H “X-Auth-Email: [email protected]” "https://api.cloudflare.com/client/v4/zones/<zone_id>/logs/received/fields"
The schema of all available fields is also linked at the bottom of this document under “received_schema.” The default schema does not include all fields available and is linked under “default_received_schema.” (Please use the API call above to get the latest schema).
Once you are familiar with the new schema, we suggest finding the fields of interest and mapping them to the fields from the /requests endpoint which are important to you. The /requests schema is also included below under "legacy_schema."
You will need to update any dashboards and scripts to use the new /received endpoint and use the fields parameter to request the fields you need for your log analysis.