Nginx Access Log Grok Pattern Generator
A deep architectural guide and regex generator for transforming unstructured Nginx combined and custom microservice logs into typed, structured JSON telemetry.
The Nginx combined log format requires the standard Grok pattern %{COMBINEDAPACHELOG} or its explicit equivalent: %{IPORHOST:client_ip} - %{DATA:auth_user} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status_code:int} %{NUMBER:bytes_sent:int} "%{DATA:referrer}" "%{DATA:user_agent}". Adding custom variables like $request_time appends %{NUMBER:request_time:float} for downstream latency profiling in Vector and Elasticsearch.
1. Anatomy of Nginx Access Log Formats
Nginx builds log lines by evaluating variables at request completion. The default combined format captures baseline client interactions, while modern Kubernetes and microservice gateways augment this format with upstream connection and TLS handshakes.
| Nginx Variable | Description | Sample Value | Target Grok Token |
|---|---|---|---|
| $remote_addr | Client IP address or upstream proxy IP | 203.0.113.195 | %{IPORHOST:client_ip} |
| $remote_user | HTTP Basic Auth authenticated username | - or admin_usr | %{DATA:remote_user} |
| $time_local | Common log format localized timestamp | 06/Sep/2026:14:22:18 +0000 | %{HTTPDATE:timestamp} |
| $request | Full HTTP request line (verb, URI, proto) | GET /api/v1/user HTTP/1.1 | %{WORD:verb} %{URIPATHPARAM:request_path} HTTP/%{NUMBER:http_version} |
| $status | Standard HTTP response status code | 200, 404, 502 | %{NUMBER:status_code:int} |
| $body_bytes_sent | Number of payload bytes transmitted to client | 4528 | %{NUMBER:bytes_sent:int} |
| $request_time | Total transaction duration in seconds with ms res | 0.042 | %{NUMBER:duration_seconds:float} |
| $upstream_response_time | Time spent receiving upstream backend response | 0.038 or - | (?:%{NUMBER:upstream_time:float}|-) |
2. Production-Grade Grok Pattern Configurations
Pattern A: Standard Nginx Combined Format
Default Nginx Directive
Matches: log_format combined '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"';
^%{IPORHOST:client_ip} %{DATA:ident} %{DATA:auth_user} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request_path} HTTP/%{NUMBER:http_version}" %{NUMBER:status_code:int} %{NUMBER:bytes_sent:int} "%{DATA:referrer}" "%{DATA:user_agent}"$ Pattern B: Microservice Gateway with Upstream Latency
Includes $request_time & $upstream_response_time
Matches: log_format upstream_perf '$remote_addr - [$time_local] "$request" $status $bytes_sent "$http_referer" "$http_user_agent" $request_time $upstream_response_time';
^%{IPORHOST:client_ip} - \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request_path} HTTP/%{NUMBER:http_version}" %{NUMBER:status_code:int} %{NUMBER:bytes_sent:int} "%{DATA:referrer}" "%{DATA:user_agent}" %{NUMBER:duration_seconds:float} (?:%{NUMBER:upstream_response_time:float}|-)$ 3. Collector Ingestion Configurations
Vector Remap Language (VRL) Transform
Native Rust parsing using parse_grok with automatic type coercion:
# Vector Remap Language (VRL)
.parsed, err = parse_grok(.message, "%{IPORHOST:client_ip} %{DATA:ident} %{DATA:auth_user} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request_path} HTTP/%{NUMBER:http_version}" %{NUMBER:status_code:int} %{NUMBER:bytes_sent:int} "%{DATA:referrer}" "%{DATA:user_agent}"")
if err != null {
log("Failed to parse log: " + err, level: "warn")
} else {
. = merge(., .parsed)
.timestamp = parse_timestamp!(.timestamp, format: "%d/%b/%Y:%T %z")
del(.message)
del(.parsed)
} Fluent Bit Regex Parser (parsers.conf)
Oniguruma regex parser configuration for edge logging:
[PARSER]
Name nginx_combined_custom
Format regex
Regex ^(?<client_ip>[^ ]+) (?<ident>[^ ]+) (?<user>[^ ]+) [(?<time>[^]]+)] "(?<method>S+)(?: +(?<path>[^"]*?)(?: +HTTP/(?<http_version>[0-9.]+))?)?" (?<status_code>[0-9]+) (?<size>[0-9]+) "(?<referer>[^"]*)" "(?<user_agent>[^"]*)"$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
Types status_code:integer size:integer 4. Production Pitfalls & Hardening Rules
1. HTTP 400 Bad Request Truncation
Malicious scanners and clients sending TLS packets to cleartext HTTP ports emit invalid request lines like "-" 400 0 "-" "-". Strict patterns like "%{WORD} %{URIPATH} HTTP/%{NUMBER}" will fail to match. Always wrap request parsing in non-capturing fallback groups or separate invalid lines into a dead-letter queue.
2. Upstream Timeout Hyphens
When upstream backends time out or return 504 Gateway Timeout, $upstream_response_time produces a hyphen - instead of a float. Using a strict float parser fails; always specify (?:%{NUMBER:upstream_time:float}|-).
3. Native JSON Log Recommendation
Where possible, switch Nginx to native structured JSON: log_format json_analytics escape=json '{"time_local":"$time_local","client_ip":"$remote_addr","status":$status}';. Parsing JSON with SIMD in Vector is over 4x faster than Grok regex compilation.