Cloudflare HTTP Logpush: JSON Regex Parsing, High-Cardinality Edge Filters & ClickHouse Ingestion

1. Cloudflare HTTP Requests Dataset Schema Specification

Cloudflare Logpush streams raw HTTP edge telemetry in newline-delimited JSON (NDJSON) packages. For high-traffic enterprise domains processing billions of requests daily, recording every field without selective filtering can easily produce tens of terabytes of log data per week.

Understanding the core field primitives enables teams to isolate essential diagnostic fields:

2. Edge Filtering: Reducing Ingestion Volume by 70%+

The most effective way to eliminate storage bloat is to configure Logpush Edge Filters inside your Cloudflare Dashboard or via the Cloudflare API before logs leave the network boundary.

Typical high-volume noise sources include favicon requests, static CSS/JS files, and internal synthetic health probes. Here is a production-grade Logpush API filter rule configured via cURL:

curl -s -X POST "https://api.cloudflare.com/client/v4/zones/${CF_ZONE_ID}/logpush/jobs" \
  -H "Authorization: Bearer ${CF_API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "filtered-http-requests-r2",
    "destination_conf": "r2://edge-telemetry-bucket/http_logs?account-id='${CF_ACCOUNT_ID}'",
    "dataset": "http_requests",
    "filter": "{\"where\":{\"and\":[{\"key\":\"ClientRequestURI\",\"operator\":\"!contains\",\"value\":\".css\"},{\"key\":\"ClientRequestURI\",\"operator\":\"!contains\",\"value\":\".jpg\"},{\"key\":\"ClientRequestURI\",\"operator\":\"!contains\",\"value\":\".png\"},{\"key\":\"ClientRequestURI\",\"operator\":\"!contains\",\"value\":\"/healthz\"}]}}",
    "logstream_spec": "edge"
  }'

3. ClickHouse MergeTree Table Optimization & Order By Strategy

ClickHouse achieves exceptional compression and multi-billion-row query speed due to sparse primary indexes and columnar run-length encoding. To maximize pruning efficiency, columns must be positioned in the ORDER BY tuple from lowest cardinality to highest cardinality:

-- Production ClickHouse MergeTree DDL for Cloudflare HTTP Logs
CREATE TABLE IF NOT EXISTS cloudflare_http_logs (
    EdgeStartTimestamp DateTime64(6, 'UTC') CODEC(DoubleDelta, ZSTD(3)),
    ClientRequestHost LowCardinality(String) CODEC(ZSTD(3)),
    ClientRequestMethod LowCardinality(String) CODEC(ZSTD(3)),
    EdgeResponseStatus UInt16 CODEC(T64, ZSTD(3)),
    OriginResponseStatus Nullable(UInt16) CODEC(T64, ZSTD(3)),
    EdgeColoCode LowCardinality(String) CODEC(ZSTD(3)),
    ClientCountry LowCardinality(String) CODEC(ZSTD(3)),
    ClientIP IPv6 CODEC(ZSTD(3)),
    ClientRequestURI String CODEC(ZSTD(6)),
    ClientRequestUserAgent String CODEC(ZSTD(6)),
    EdgeResponseBytes UInt64 CODEC(T64, ZSTD(3)),
    OriginResponseDurationMs UInt32 CODEC(T64, ZSTD(3)),
    RayID FixedString(16) CODEC(ZSTD(3)),
    BotScore UInt8 CODEC(T64, ZSTD(3)),
    WAFAction LowCardinality(String) CODEC(ZSTD(3))
) ENGINE = MergeTree()
PARTITION BY toDate(EdgeStartTimestamp)
ORDER BY (ClientRequestHost, EdgeResponseStatus, toDate(EdgeStartTimestamp), EdgeStartTimestamp, ClientIP)
SETTINGS index_granularity = 8192;

4. Ingestion Engines: ClickHouse S3Queue vs Vector Pipeline

There are two primary paradigms for feeding Cloudflare NDJSON batches from S3/R2 into ClickHouse:

• 1. Native ClickHouse S3Queue Engine: ClickHouse polls S3/R2 directly, tracks processed files via ZooKeeper/Keeper, and ingests them without intermediary infrastructure. Highly resilient and serverless.
• 2. Vector High-Throughput Stream: An external Vector daemon watches S3 object-created notifications via SQS, decompresses GZIP payloads in-memory, applies VRL enrichment (e.g. GeoIP lookups or URL normalization), and sends native bulk HTTP batches to ClickHouse.

5. Python Test Runner: NDJSON Batch Verification & Syntax Validation

Run this verification utility to validate Cloudflare Logpush NDJSON files against the expected schema before committing ingestion configs:

import json
import gzip
from typing import Dict, Any

REQUIRED_FIELDS = {
    "RayID", "ClientIP", "ClientRequestHost", "ClientRequestMethod",
    "ClientRequestURI", "EdgeResponseStatus", "EdgeStartTimestamp"
}

def validate_cloudflare_ndjson_record(record_str: str) -> bool:
    try:
        data: Dict[str, Any] = json.loads(record_str)
        missing = REQUIRED_FIELDS - data.keys()
        if missing:
            print(f"Schema violation: missing fields {missing}")
            return False
        
        # Verify status code integer range
        status = data.get("EdgeResponseStatus")
        if not isinstance(status, int) or not (100 <= status <= 599):
            print(f"Invalid HTTP status: {status}")
            return False
            
        return True
    except json.JSONDecodeError as e:
        print(f"JSON decode failure: {e}")
        return False

if __name__ == "__main__":
    sample = (
        '{"RayID":"8b24ef0182a931c4","ClientIP":"198.51.100.42",'
        '"ClientRequestHost":"api.platform.io","ClientRequestMethod":"GET",'
        '"ClientRequestURI":"/v1/telemetry","EdgeResponseStatus":200,'
        '"EdgeStartTimestamp":1728118462000000000,"EdgeColoCode":"LHR",'
        '"BotScore":95,"WAFAction":"allow"}'
    )
    assert validate_cloudflare_ndjson_record(sample) is True
    print("Cloudflare NDJSON Schema: VALID")

6. Top 4 High-Volume Edge Logging Failure Modes & Remediation

• 1. ClickHouse "Too Many Parts" Write Backpressure: Inserting millions of rows as individual small HTTP requests causes part proliferation and throttles ClickHouse. Remedy: Batch inserts into chunks of at least 50,000 to 100,000 rows or use the Buffer table engine / S3Queue.
• 2. Nanosecond Timestamp Truncation Drift: Cloudflare outputs EdgeStartTimestamp in epoch nanoseconds (19 digits). Casting directly to standard DateTime (seconds) truncates precision and destroys microsecond sorting. Remedy: Store timestamps as DateTime64(6, 'UTC') and divide raw integer nanoseconds by 1,000.
• 3. S3 / R2 API Rate Limiting on Rapid File Creation: When Logpush generates thousands of small 500KB GZIP files every minute across many domains, downstream S3 GET workers hit S3 503 SlowDown errors. Remedy: Configure prefix hash partitioning or enable Cloudflare Logpush batch frequency tuning.
• 4. IPv6 Formatting Incompatibilities: IPv6 addresses containing bracket encodings or leading zeros cause downstream regex parsers to fail. Remedy: Use ClickHouse native IPv6 type which automatically normalizes representation to 16-byte fixed binary arrays.

Frequently Asked Questions