Getting Started with Cloudflare Logpush Beta

Push your HTTPs request logs to Amazon S3 or Google Cloud Storage using Cloudflare Logpush.


Overview

Your can have your HTTP request logs pushed to certain destinations every minute using the Cloudflare Logpush service.

All Enterprise Log Share (ELS) functionality — including selecting fields and sampling — is also available in Logpush.

Currently, you can push your logs to Amazon S3 and Google Cloud Storage (GCS). Cloudflare plans to continue expanding the list of destinations in the near future.

The Cloudflare Logpush beta program is currently available to Enterprise customers by invitation. To join, contact your Customer Success Manager or Solutions Engineer.

Get started

Cloudflare Logpush is currently managed via an API. Through this API, you can execute CRUD operations (Create, Read, Update, and Delete).

With Logpush, you can create a job to upload your HTTP logs every minute. The API allows one job per zone.

Before getting started:

1. Make sure you are familiar with the parameters and timestamp options associated with the Cloudflare Enterprise Log Share (ELS) API. Consult the following resources to learn more:

2. Set up a destination storage and grant Cloudflare write permissions. For specific instructions:

3. Have your Cloudflare API credentials and other information handy. You need:

  • Email address
  • Cloudflare API key
  • Zone ID
  • Destination access details for AWS S3 or Google Cloud Storage

Next, there are two steps to enable log pushing:

  1. Create a job via a HTTP POST request.
  2. Enable the job via a HTTP PUT request.

Below is a tutorial with examples on how to enable Cloudflare Logpush for your zone.


Enable Logpush​ ​in​ AWS S3

Cloudflare uses AWS Identity and Access Management (IAM) to gain access to your S3 bucket. The Cloudflare IAM user needs PutObject permissions for that bucket.


Logs are written into that bucket as gzipped objects using the S3 Access Control List (ACL)
Bucket-owner-full-control permission.

For illustrative purposes, imagine that a customer wants to store logs in the bucket burritobot in the logs directory. The S3 URL would then be s3://burritobot/logs.

To enable Logpush in AWS S3:

1. Create an S3 bucket.

2. Paste the policy below into S3 > Bucket > Permissions > Bucket Policy:

{
  "Id": "Policy1506627184792",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1506627150918",
      "Action": [
        "s3:PutObject"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::burritobot/logs/*",
      "Principal": {
        "AWS": [
          "arn:aws:iam::391854517948:user/cloudflare-logpush"
        ]
      }
    }
  ]
}

3. Make sure no other member has read permissions for the bucket.

By default, Cloudflare pushes the logs once a minute, with 9 fields included. For full details, consult the article Enterprise Log Share: Logpull REST API

Enable Logpush​ ​in​ ​Google​ ​Cloud​ ​Storage​

Cloudflare uses a Google Cloud Storage (GCS) service account to gain access to your bucket. Therefore, the Cloudflare service account needs write permissions for that bucket.

To enable Logpush in GCS:

1. Create a GCS bucket.

2. In Storage > Browser > Bucket > Permissions, add the member logpush@cloudflare-data.iam.gserviceaccount.com with Storage Object Admin permissions.

3. Make sure no other member has read permissions for the bucket.

By default, Cloudflare pushes the logs once a minute, with 9 fields included. For full details, consult the article Enterprise Log Share: Logpull REST API

Endpoint information

CRUD operations:

POST   https://api.cloudflare.com/client/v4/zones/<zone_identifier>/logpush/jobs
GET    https://api.cloudflare.com/client/v4/zones/<zone_identifier>/logpush/jobs/<job>
PUT    https://api.cloudflare.com/client/v4/zones/<zone_identifier>/logpush/jobs/<job>
DELETE https://api.cloudflare.com/client/v4/zones/<zone_identifier>/logpush/jobs/<job>

To list the job associated with your zone:

GET    https://api.cloudflare.com/client/v4/zones/<zone_identifier>/logpush/jobs

To validate a destination:

POST https://api.cloudflare.com/client/v4/zones/<zone_identifier>/logpush/validate/destination

The <job> argument is the numeric job ID. See tutorial below for concrete examples.


Connecting to Cloudflare Logpush

The configiguration API requires credentials like any other Cloudflare API:

curl -s -H "X-Auth-Email: sohei@burritobot.com" -H "X-Auth-Key: xxx" \ 
'https://api.cloudflare.com/client/v4/zones/4e6d50a41172bca54f222576aec3fc2b/logpush/jobs'

API tutorial using cURL

Below are several tasks that you can perform to manage your Cloudflare Logpush service.

1. Create a job

To issue a job creation request:

curl -s -XPOST 
https://api.cloudflare.com/client/v4/zones/4e6d50a41172bca54f222576aec3fc2b/logpush/jobs
-d'{"name":"theburritobot.com",
"destination_conf":"s3://theburritobot/logs?region=us-west-2",
"logpull_options":"fields=RayID,
EdgeStartTimestamp&timestamps=rfc3339"}' | jq .

Parameters:

  • name (optional) - We suggest the zone name as the job name.
  • destination_conf - destination for your logs: 
    • AWS S3: bucket + optional directory + region + optional encryption parameter (if required by your policy); for example: s3://bucket/[dir]?region=<region>[&sse=AES256]
    • Google Cloud: bucket + optional directory; for example: gs://bucket/[dir]
    • To separate logs into daily subdirectories, use the special string {DATE} in the URL path. It will be substituted with the date in YYYYMMDD format; for example 20180523, and subdirectories will be created as appropriate; for example: s3://mybucket/logs/{DATE}?region=us-east-1&sse=AES256
  • logpull_options - Parameters from the Log Share API (Logpull):
  • Typically includes the desired fields and timestamp format
  • All available fields can be obtained using:
    https://api.cloudflare.com/client/v4/zones/<zone_identifier>/logs/received/fields
  • For Google Cloud Big Query usage, set the timestamp format to rfc3339 (&timestamps=rfc3339).

In response, you get the newly created job ID. For example:

Response:
{ "errors": [], "messages": [], "result": { "id": 146, "name": "theburritobot.com", "enabled": false, "logpull_options": "fields=RayID,EdgeStartTimestamp&timestamps=rfc3339", "destination_conf": "s3://theburritobot/logs?region=us-west-2", "last_complete": null, "last_error": null, "error_message": null }, "success": true }

For a detailed description of the job object JSON schema, read the section below.

Response format:

  • The zero (empty, unset) value for fields is null.
  • All fields are always present in the response (even if set to null).
  • Timestamps are formatted to RFC3339; for example: 2018-08-17T14:21:00Z.
  • The API returns data in NDJSON format, whereby each log line is a valid JSON object. Major analysis tools like Google BigQuery and AWS Kinesis require this format.
  • You can use the jq tool to turn the resulting log data into a JSON array with one array element per log line.
<API request code> | jq

2. Enable a job

To start, retrieve information about a specific job, using a job ID:

curl -s -XGET 
https://api.cloudflare.com/client/v4/zones/4e6d50a41172bca54f222576aec3fc2b/logpush/jobs/146
| jq .
Response:
{
"errors": [],
"messages": [],
"result": {
"id": 146,
"name": "theburritobot.com",
"enabled": false,
"logpull_options": "fields=RayID,EdgeStartTimestamp&timestamps=rfc3339",
"destination_conf": "s3://theburritobot/logs?region=us-west-2",
"last_complete": null,
"last_error": null,
"error_message": null
},
"success": true
}

Note that by default a job is not enabled ("enabled": false.).

Next, to enable the job, send an update request:

curl -s -XPUT 
https://api.cloudflare.com/client/v4/zones/4e6d50a41172bca54f222576aec3fc2b/logpush/jobs/146
-d'{"enabled":true}' | jq .
Response:
{
"errors": [],
"messages": [],
"result": {
"id": 146,
"name": "theburritobot.com",
"enabled": true,
"logpull_options": "fields=RayID,EdgeStartTimestamp&timestamps=rfc3339",
"destination_conf": "s3://theburritobot/logs?region=us-west-2",
"last_complete": null,
"last_error": null,
"error_message": null
},
"success": true
}

Once the job is active, you will receive logs about 1 minute later and then every minute after that until you disable the job.

In addition to modifying enabled, you can also update values for logpull_options and destination_conf.

Once a job has been enabled and has started executing, the last_complete field will show the time when the last batch of logs was successfully sent to the destination:

curl -s -XGET 
https://api.cloudflare.com/client/v4/zones/4e6d50a41172bca54f222576aec3fc2b/logpush/jobs/146
| jq .
{"errors": [],
"messages": [],
"result": {
"id": 146,
"name": "theburritobot.com",
"enabled": true,
"logpull_options": "fields=RayID,EdgeStartTimestamp&timestamps=rfc3339",
"destination_conf": "s3://theburritobot/logs?region=us-west-2",
"last_complete": "2018-08-09T21:26:00Z",
"last_error": null,
"error_message": null
},
"success": true
}

3. Delete a job

curl -s -XDELETE 
https://api.cloudflare.com/client/v4/zones/4e6d50a41172bca54f222576aec3fc2b/logpush/jobs/146
| jq .
Response:
{
  "errors": [],
  "messages": [],
  "result": {},
  "success": true
}

To show your job for your zone:

curl -s -XGET 
https://api.cloudflare.com/client/v4/zones/4e6d50a41172bca54f222576aec3fc2b/logpush/jobs
| jq .
Response:
{
  "errors": [],
  "messages": [],
  "result": [
    {
      "id": 17,
      "name": "theburritobot.com-googlecloud",
      "enabled": true,
      "logpull_options": null,
      "destination_conf": "gs://theburritobot/logs",
      "last_complete": "2018-08-09T21:26:00Z",
      "last_error": null,
      "error_message": null
    }
  ],
  "success": true
}

Additional details

Job object schema

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "job",
  "description": "logpush job parameters",
  "type": "object",
  "required": [
    "id",
    "enabled",
    "name",
    "logpull_options",
    "destination_conf",
    "last_complete",
    "last_error",
    "error_message"
  ],
  "properties": {
    "id": {
      "description": "Unique id of the job",
      "type": "integer",
      "minimum": 1
    },
    "enabled": {
      "description": "",
      "type": "boolean"
    },
    "name": {
      "description": "A human readable job name. It doesn’t have to be unique but adding the zone name is recommended.",
      "type": [
        "null",
        "string"
      ],
      "pattern": "^[a-zA-Z0-9\\-\\.]*$",
      "maxLength": 512
    },
    "logpull_options": {
      "description": "A configuration string for the Log Share API. It specifies things like requested fields and timestamp formats. Copy the URL (full URL or just the query string) of your call to the Log Share API here, and Logpush will keep on making this call for you, setting start, and end times appropriately.",
      "type": [
        "null",
        "string"
      ],
      "format": "uri",
      "maxLength": 4096
    },
    "destination_conf": {
      "description": "Uniquely identifies a resource (such as an s3 bucket) where data will be pushed.",
      "type": "string",
      "format": "uri",
      "maxLength": 4096
    },
    "last_complete": {
      "description": "Records the last time for which logs have been successfully pushed. If the last successful push was for logs range 2018-07-23T10:00:00Z to 2018-07-23T10:01:00Z then the value of this field will be 2018-07-23T10:01:00Z. If the job has never run or has just been enabled and hasn't run yet then the field will be empty.",
      "type": [
        "null",
        "string"
      ],
      "format": "date-time"
    },
    "last_error": {
      "description": "Records the last time the job failed. If not null, the job is currently failing. if null, the job has either never failed or has run successfully at least once since last failure. See also the error_message field.",
      "type": [
        "null",
        "string"
      ],
      "format": "date-time"
    },
    "error_message": {
      "description": "If not null, the job is currently failing. Failures are usually repetitive (for example: no permissions to write to destination bucket). Only the last failure is recorded. On successful execution of a job, the error_message and last_error are set to null.",
      "type": [
        "null",
        "string"
      ],
      "format": "date-time"
    }
  }
}

Destinations

Logpush can send data to AWS S3 and GCS. Based on the convention set by the AWS S3 CLI and the Google Cloud Storage CLI, destinations are specified as URIs in the destination_conf job parameter. For GCS, the URI takes the form gs://bucket/[dir] and for S3 s3://bucket/[dir]?region=<region>[&sse=AES256].

Options

Logpush repeatedly calls the Log Share Logpull API on behalf of the user and uploads the results to the destination. Calls to the Log Share API are configured in the logpull_options job parameter. It contains the query string for the API call. For example, if the user uses the following query to get data from the Log Share API:

curl -sv \
-H'X-Auth-Email: mik@cloudflare.com' \
-H'X-Auth-Key: XXX' \ "https://api.cloudflare.com/client/v4/zones/4e6d50a41172bca54f222576aec3fc2b/logs/received?start=2018-08-02T10:00:00Z&end=2018-08-02T10:01:00Z&fields=RayID,EdgeStartTime"

Then "logpull_options": "fields=RayID,EdgeStartTime".

For detailed information on calling the API, see Enterprise Log Share: Logpull REST API.

If left empty, default Log Share API settings are used.

Validation

Destinations can be validated (check if S3 bucket is configured correctly so that Logpush can write files to it):

curl -s -XPOST 
https://api.cloudflare.com/client/v4/zones/4e6d50a41172bca54f222576aec3fc2b/logpush/validate/destination
-d'{"destination_conf":"s3://foo"}' | jq .
Response:
{
"errors": [],
"messages": [],
"result": {
"message": "",
"valid": true
},
"success": true
}

Service Level Agreement

As a beta product, Cloudflare Logpush has no Service Level Agreement (SLA). Cloudflare makes no guarantees on availability or quality of service.

If you experience any unexpected behavior, contact your Cloudflare Solutions Engineer or Customer Success Manager, who will do their best to respond in a timely manner.

Not finding what you need?

95% of questions can be answered using the search tool. This is the quickest way to get a response.

Powered by Zendesk