Integrations — Nginx Log Export

Audience: Bank IT, Linux/Nginx administrators, AI agents performing integrationUpdated 2026-07-29

Nginx Log Export

This page documents how to extract LLM bot traffic from Nginx access logs and ship it to Maya, with all data minimization rules applied at source.

When to use this guide: your production stack is Nginx-hosted (origin or reverse proxy) with access logs available on the host or a log-aggregation box. If Nginx sits behind a CDN you can connect natively, prefer the Connect Ready path. If you have a BFF / API gateway that already builds request records, see bff-endpoint.md — it's usually cleaner.

Prerequisites

Requirement	Notes
Nginx 1.11.8+	Required for `escape=json` in `log_format`
A dedicated JSON access log	So the filter parses one record per line, robustly (see below)
Python 3.8+	For the filter script. No third-party packages required.
Outbound HTTPS to `ingest.withmaya.ai` on port 443	For transmission
Maya API key + tenant secret	Provided by Maya; stored as env vars / secret manager, never in source

Recommended: a JSON access log

Rather than parse the default combined format, emit exactly the fields Maya needs as JSON. Add to your http {} block:

Nginx

log_format maya_json escape=json
  '{'
    '"time":"$time_iso8601",'
    '"method":"$request_method",'
    '"uri":"$uri",'
    '"query":"$args",'
    '"status":$status,'
    '"bytes":$body_bytes_sent,'
    '"referer":"$http_referer",'
    '"host":"$host",'
    '"ua":"$http_user_agent",'
    '"ip":"$remote_addr"'
  '}';
 
# A separate log keeps the filter simple and leaves your existing logging untouched.
access_log /var/log/nginx/maya_access.log maya_json;

Reload Nginx (nginx -t && systemctl reload nginx). Note the log still contains all traffic and raw IPs — that is expected. The filter below is what removes non-bot traffic and hashes IPs before anything leaves your network.

Filter script

Save as maya_bot_filter.py. It reads the JSON access log, keeps only allowlisted LLM bots, strips denylisted query keys, HMAC-hashes the client IP, and emits NDJSON.

Python

#!/usr/bin/env python3
"""Maya — Nginx/Apache LLM bot log filter.
 
Reads a JSON access log (one JSON object per line), retains only verified LLM
bot traffic, applies PII strip + IP hashing, produces NDJSON output.
 
Usage:
  MAYA_TENANT_SECRET=... python3 maya_bot_filter.py \
      --log /var/log/nginx/maya_access.log \
      --since 2026-07-28T00:00:00Z \
      --out  /var/maya/export-2026-07-28.ndjson [--dry-run]
"""
import argparse, hashlib, hmac, json, os, sys
from datetime import datetime, timezone
from urllib.parse import parse_qsl, urlencode
 
# 1. Verified LLM bot allowlist (case-insensitive substring match).
BOT_PATTERNS = [
    "GPTBot", "ChatGPT-User", "OAI-SearchBot",
    "ClaudeBot", "Claude-Web", "anthropic-ai",
    "PerplexityBot", "Perplexity-User",
    "Google-Extended", "Googlebot", "Bingbot",
    "Applebot-Extended", "Bytespider", "CCBot",
    "Meta-ExternalAgent", "FacebookBot",
    "DuckAssistBot", "YouBot", "Amazonbot", "Diffbot",
]
 
# 2. Query-string keys to strip.
DENY_KEYS = {"email", "tckn", "phone", "customer_id", "account_id",
             "session", "token", "key", "secret", "auth", "csrf"}
 
 
def is_bot(ua: str) -> bool:
    u = ua.lower()
    return any(p.lower() in u for p in BOT_PATTERNS)
 
 
def sanitize(query: str) -> str:
    # Tolerates a leading '?' so the same script works for Apache's %q.
    query = (query or "").lstrip("?")
    if not query:
        return ""
    kept = [(k, v) for k, v in parse_qsl(query, keep_blank_values=True)
            if k.lower() not in DENY_KEYS]
    return urlencode(kept)
 
 
def hash_ip(ip: str, secret: str):
    if not ip:
        return None
    return hmac.new(secret.encode(), ip.encode(), hashlib.sha256).hexdigest()[:32]
 
 
def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--log", required=True)
    ap.add_argument("--since", required=True, help="ISO-8601, e.g. 2026-07-28T00:00:00Z")
    ap.add_argument("--out", required=True)
    ap.add_argument("--dry-run", action="store_true")
    a = ap.parse_args()
 
    secret = os.environ.get("MAYA_TENANT_SECRET")
    if not secret:
        sys.exit("MAYA_TENANT_SECRET is not set")
 
    since = datetime.fromisoformat(a.since.replace("Z", "+00:00"))
    total = kept = 0
    out = None if a.dry_run else open(a.out, "w", encoding="utf-8")
 
    with open(a.log, encoding="utf-8", errors="replace") as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            total += 1
            try:
                r = json.loads(line)
            except json.JSONDecodeError:
                continue
 
            ua = r.get("ua", "") or ""
            if not is_bot(ua):
                continue
 
            try:
                ts = datetime.fromisoformat(str(r["time"]).replace("Z", "+00:00"))
            except (KeyError, ValueError):
                continue
            ts = ts.astimezone(timezone.utc)
            if ts < since:
                continue
 
            q = sanitize(r.get("query", ""))
            path = r.get("uri", "") or ""
            if q:
                path = f"{path}?{q}"
 
            record = {
                "timestamp":      ts.strftime("%Y-%m-%dT%H:%M:%SZ"),
                "user_agent":     ua,
                "request_method": r.get("method", "") or "",
                "request_path":   path,
                "status_code":    int(r.get("status", 0) or 0),
                "response_bytes": int(r.get("bytes", 0) or 0),
                "referrer":       r.get("referer", "") or "",
                "host":           r.get("host", "") or "",
                "client_ip_hash": hash_ip(r.get("ip", ""), secret),
            }
 
            kept += 1
            if out:
                out.write(json.dumps(record, separators=(",", ":")) + "\n")
 
    if out:
        out.close()
    sys.stderr.write(f"Scanned {total} rows, retained {kept}.\n")
    if a.dry_run:
        sys.stderr.write("Dry run — no file written.\n")
 
 
if __name__ == "__main__":
    main()

Notes on the script

No third-party dependencies — stock Python 3 only, so it runs on a hardened host.
IPs are never written raw. The HMAC step is keyed by a tenant secret held by the brand; Maya does not know this secret. Rotating the secret changes all hashes (pre/post-rotation hashes won't collide).
--dry-run computes counts without writing output — use it for source-side audit.
The output schema is identical to every other stack, so dashboards are unaffected by which stack you export from.

Expected output (sample)

Newline-delimited JSON (.ndjson) — one bot request per line:

JSON

{"timestamp":"2026-07-28T03:14:07Z","user_agent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)","request_method":"GET","request_path":"/products/super-widget","status_code":200,"response_bytes":48329,"referrer":"","host":"www.example.com","client_ip_hash":"a4c6f1b2e7d8c1bb"}
{"timestamp":"2026-07-28T03:14:42Z","user_agent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +https://www.anthropic.com/claudebot)","request_method":"GET","request_path":"/pricing","status_code":200,"response_bytes":62104,"referrer":"","host":"www.example.com","client_ip_hash":"7d9e22aa1f53e2c4"}

Download a full sample (15 rows, ~5 KB): maya-bot-logs.sample.ndjson

Sanity checks on your output

Run these before sending the file to Maya:

Bash

# 1) Every line is valid JSON
while IFS= read -r l; do echo "$l" | python3 -c 'import json,sys; json.loads(sys.stdin.read())' \
  || echo "Invalid: $l"; done < export.ndjson
 
# 2) No disallowed content leaked
grep -iE '(cookie|authorization|set-cookie|session|token|tckn|customer_id|account_id)' export.ndjson
# Expected: nothing.
 
# 3) Every user_agent matches an LLM bot
grep -ivE 'GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-Web|anthropic-ai|PerplexityBot|Perplexity-User|Google-Extended|Googlebot|Bingbot|Applebot-Extended|Bytespider|CCBot|Meta-ExternalAgent|FacebookBot|DuckAssistBot|YouBot|Amazonbot|Diffbot' export.ndjson
# Expected: nothing.

If all three emit nothing, the export is safe to transmit.

Transmission

Bash

curl -sS -X POST 'https://ingest.withmaya.ai/v1/logs' \
  -H "Authorization: Bearer $MAYA_API_KEY" \
  -H 'X-Maya-Tenant: acme-prod' \
  -H 'X-Maya-Schema: 1' \
  -H 'Content-Type: application/x-ndjson' \
  --data-binary @/var/maya/export-2026-07-28.ndjson \
  -w '%{http_code}\n' -o /dev/null

The endpoint returns 202 Accepted on success, with a body containing the batch ID for traceability.

Scheduling

Ship the prior day's (or week's) logs off-peak via cron:

Cron

# /etc/cron.d/maya-log-export  — daily at 03:00, prior day's window
0 3 * * * maya MAYA_TENANT_SECRET=@secret MAYA_API_KEY=@secret \
  /usr/bin/python3 /opt/maya/maya_bot_filter.py \
    --log /var/log/nginx/maya_access.log \
    --since "$(date -u -d 'yesterday 00:00' +\%Y-\%m-\%dT00:00:00Z)" \
    --out /var/maya/export-$(date -u +\%Y-\%m-\%d).ndjson \
  && /opt/maya/upload.sh /var/maya/export-$(date -u +\%Y-\%m-\%d).ndjson

Load secrets from your secret manager (@secret above is a placeholder) — never inline them in crontab.

Validation checklist

Run maya_bot_filter.py --dry-run on 24 hours of logs. Confirm the retained count is reasonable (typically 0.1%–5% of total rows).
Inspect 100 random rows of dry-run output. Confirm: no raw IPs; no cookies/auth/session tokens; no PII in request_path; every user_agent matches the allowlist.
POST a sample batch with X-Maya-Mode: test to verify the schema is accepted without ingesting.
Sign off the script + Nginx config change in your change management system.
Schedule and enable.

Troubleshooting

Symptom	Cause	Fix
Output file is empty	No bot traffic in window, or logging the wrong vhost	Confirm `access_log ... maya_json;` is on the serving `server {}` block; widen `--since`.
Lines skipped silently	Non-JSON lines (BOM, partial writes during rotation)	Filter runs after rotation; point `--log` at the rotated file, not the live one.
`400 Bad Request` from Maya	Schema mismatch (extra/missing field)	Inspect output NDJSON; the record keys must match exactly.
`401 Unauthorized`	API key missing/invalid	Re-check the env var / secret manager binding.
`429 Too Many Requests`	Burst over rate limit	Add exponential backoff in `upload.sh` or split the batch.
Timestamps off by hours	`$time_iso8601` uses server local offset	The script normalizes to UTC; confirm the host clock/timezone is correct.

Migration note

If you later move logging to a BFF or API gateway, retire this filter in favor of the bff-endpoint.md pattern. The schema is identical, so dashboards and downstream analysis are unaffected.

PreviousIIS / Windows Server Next Apache