# [](#nginx-log-export)Nginx Log Export This page documents how to extract LLM bot traffic from Nginx access logs and ship it to Maya, with all data minimization rules applied at source. > **When to use this guide:** your production stack is Nginx-hosted (origin or reverse proxy) with access logs available on the host or a log-aggregation box. If Nginx sits behind a CDN you can connect natively, prefer the [Connect Ready](./overview.md#-connect-ready--edge--cdn-providers) path. If you have a BFF / API gateway that already builds request records, see [`bff-endpoint.md`](./bff-endpoint.md) — it's usually cleaner. ## [](#prerequisites)Prerequisites Requirement Notes Nginx 1.11.8+ Required for `escape=json` in `log_format` A dedicated JSON access log So the filter parses one record per line, robustly (see below) Python 3.8+ For the filter script. No third-party packages required. Outbound HTTPS to `ingest.withmaya.ai` on port 443 For transmission Maya API key + tenant secret Provided by Maya; stored as env vars / secret manager, never in source ## [](#recommended-a-json-access-log)Recommended: a JSON access log Rather than parse the default `combined` format, emit exactly the fields Maya needs as JSON. Add to your `http {}` block: Nginx ``` log_format maya_json escape=json '{' '"time":"$time_iso8601",' '"method":"$request_method",' '"uri":"$uri",' '"query":"$args",' '"status":$status,' '"bytes":$body_bytes_sent,' '"referer":"$http_referer",' '"host":"$host",' '"ua":"$http_user_agent",' '"ip":"$remote_addr"' '}'; # A separate log keeps the filter simple and leaves your existing logging untouched. access_log /var/log/nginx/maya_access.log maya_json; ``` Reload Nginx (`nginx -t && systemctl reload nginx`). Note the log still contains **all** traffic and raw IPs — that is expected. The filter below is what removes non-bot traffic and hashes IPs before anything leaves your network. ## [](#filter-script)Filter script Save as `maya_bot_filter.py`. It reads the JSON access log, keeps only allowlisted LLM bots, strips denylisted query keys, HMAC-hashes the client IP, and emits NDJSON. Python ``` #!/usr/bin/env python3 """Maya — Nginx/Apache LLM bot log filter. Reads a JSON access log (one JSON object per line), retains only verified LLM bot traffic, applies PII strip + IP hashing, produces NDJSON output. Usage: MAYA_TENANT_SECRET=... python3 maya_bot_filter.py \ --log /var/log/nginx/maya_access.log \ --since 2026-07-28T00:00:00Z \ --out /var/maya/export-2026-07-28.ndjson [--dry-run] """ import argparse, hashlib, hmac, json, os, sys from datetime import datetime, timezone from urllib.parse import parse_qsl, urlencode # 1. Verified LLM bot allowlist (case-insensitive substring match). BOT_PATTERNS = [ "GPTBot", "ChatGPT-User", "OAI-SearchBot", "ClaudeBot", "Claude-Web", "anthropic-ai", "PerplexityBot", "Perplexity-User", "Google-Extended", "Googlebot", "Bingbot", "Applebot-Extended", "Bytespider", "CCBot", "Meta-ExternalAgent", "FacebookBot", "DuckAssistBot", "YouBot", "Amazonbot", "Diffbot", ] # 2. Query-string keys to strip. DENY_KEYS = {"email", "tckn", "phone", "customer_id", "account_id", "session", "token", "key", "secret", "auth", "csrf"} def is_bot(ua: str) -> bool: u = ua.lower() return any(p.lower() in u for p in BOT_PATTERNS) def sanitize(query: str) -> str: # Tolerates a leading '?' so the same script works for Apache's %q. query = (query or "").lstrip("?") if not query: return "" kept = [(k, v) for k, v in parse_qsl(query, keep_blank_values=True) if k.lower() not in DENY_KEYS] return urlencode(kept) def hash_ip(ip: str, secret: str): if not ip: return None return hmac.new(secret.encode(), ip.encode(), hashlib.sha256).hexdigest()[:32] def main() -> None: ap = argparse.ArgumentParser() ap.add_argument("--log", required=True) ap.add_argument("--since", required=True, help="ISO-8601, e.g. 2026-07-28T00:00:00Z") ap.add_argument("--out", required=True) ap.add_argument("--dry-run", action="store_true") a = ap.parse_args() secret = os.environ.get("MAYA_TENANT_SECRET") if not secret: sys.exit("MAYA_TENANT_SECRET is not set") since = datetime.fromisoformat(a.since.replace("Z", "+00:00")) total = kept = 0 out = None if a.dry_run else open(a.out, "w", encoding="utf-8") with open(a.log, encoding="utf-8", errors="replace") as f: for line in f: line = line.strip() if not line: continue total += 1 try: r = json.loads(line) except json.JSONDecodeError: continue ua = r.get("ua", "") or "" if not is_bot(ua): continue try: ts = datetime.fromisoformat(str(r["time"]).replace("Z", "+00:00")) except (KeyError, ValueError): continue ts = ts.astimezone(timezone.utc) if ts < since: continue q = sanitize(r.get("query", "")) path = r.get("uri", "") or "" if q: path = f"{path}?{q}" record = { "timestamp": ts.strftime("%Y-%m-%dT%H:%M:%SZ"), "user_agent": ua, "request_method": r.get("method", "") or "", "request_path": path, "status_code": int(r.get("status", 0) or 0), "response_bytes": int(r.get("bytes", 0) or 0), "referrer": r.get("referer", "") or "", "host": r.get("host", "") or "", "client_ip_hash": hash_ip(r.get("ip", ""), secret), } kept += 1 if out: out.write(json.dumps(record, separators=(",", ":")) + "\n") if out: out.close() sys.stderr.write(f"Scanned {total} rows, retained {kept}.\n") if a.dry_run: sys.stderr.write("Dry run — no file written.\n") if __name__ == "__main__": main() ``` ### [](#notes-on-the-script)Notes on the script * No third-party dependencies — stock Python 3 only, so it runs on a hardened host. * IPs are **never** written raw. The HMAC step is keyed by a tenant secret held by the brand; Maya does not know this secret. Rotating the secret changes all hashes (pre/post-rotation hashes won't collide). * `--dry-run` computes counts without writing output — use it for source-side audit. * The output schema is identical to every other stack, so dashboards are unaffected by which stack you export from. ## [](#expected-output-sample)Expected output (sample) Newline-delimited JSON (`.ndjson`) — one bot request per line: JSON ``` {"timestamp":"2026-07-28T03:14:07Z","user_agent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)","request_method":"GET","request_path":"/products/super-widget","status_code":200,"response_bytes":48329,"referrer":"","host":"www.example.com","client_ip_hash":"a4c6f1b2e7d8c1bb"} {"timestamp":"2026-07-28T03:14:42Z","user_agent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +https://www.anthropic.com/claudebot)","request_method":"GET","request_path":"/pricing","status_code":200,"response_bytes":62104,"referrer":"","host":"www.example.com","client_ip_hash":"7d9e22aa1f53e2c4"} ``` > **Download a full sample (15 rows, ~5 KB):** [`maya-bot-logs.sample.ndjson`](/docs/samples/maya-bot-logs.sample.ndjson) ### [](#sanity-checks-on-your-output)Sanity checks on your output Run these before sending the file to Maya: Bash ``` # 1) Every line is valid JSON while IFS= read -r l; do echo "$l" | python3 -c 'import json,sys; json.loads(sys.stdin.read())' \ || echo "Invalid: $l"; done < export.ndjson # 2) No disallowed content leaked grep -iE '(cookie|authorization|set-cookie|session|token|tckn|customer_id|account_id)' export.ndjson # Expected: nothing. # 3) Every user_agent matches an LLM bot grep -ivE 'GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-Web|anthropic-ai|PerplexityBot|Perplexity-User|Google-Extended|Googlebot|Bingbot|Applebot-Extended|Bytespider|CCBot|Meta-ExternalAgent|FacebookBot|DuckAssistBot|YouBot|Amazonbot|Diffbot' export.ndjson # Expected: nothing. ``` If all three emit nothing, the export is safe to transmit. ## [](#transmission)Transmission Bash ``` curl -sS -X POST 'https://ingest.withmaya.ai/v1/logs' \ -H "Authorization: Bearer $MAYA_API_KEY" \ -H 'X-Maya-Tenant: acme-prod' \ -H 'X-Maya-Schema: 1' \ -H 'Content-Type: application/x-ndjson' \ --data-binary @/var/maya/export-2026-07-28.ndjson \ -w '%{http_code}\n' -o /dev/null ``` The endpoint returns `202 Accepted` on success, with a body containing the batch ID for traceability. ## [](#scheduling)Scheduling Ship the prior day's (or week's) logs off-peak via cron: Cron ``` # /etc/cron.d/maya-log-export — daily at 03:00, prior day's window 0 3 * * * maya MAYA_TENANT_SECRET=@secret MAYA_API_KEY=@secret \ /usr/bin/python3 /opt/maya/maya_bot_filter.py \ --log /var/log/nginx/maya_access.log \ --since "$(date -u -d 'yesterday 00:00' +\%Y-\%m-\%dT00:00:00Z)" \ --out /var/maya/export-$(date -u +\%Y-\%m-\%d).ndjson \ && /opt/maya/upload.sh /var/maya/export-$(date -u +\%Y-\%m-\%d).ndjson ``` Load secrets from your secret manager (`@secret` above is a placeholder) — never inline them in crontab. ## [](#validation-checklist)Validation checklist * Run `maya_bot_filter.py --dry-run` on 24 hours of logs. Confirm the retained count is reasonable (typically 0.1%–5% of total rows). * Inspect 100 random rows of dry-run output. Confirm: no raw IPs; no cookies/auth/session tokens; no PII in `request_path`; every `user_agent` matches the allowlist. * POST a sample batch with `X-Maya-Mode: test` to verify the schema is accepted without ingesting. * Sign off the script + Nginx config change in your change management system. * Schedule and enable. ## [](#troubleshooting)Troubleshooting Symptom Cause Fix Output file is empty No bot traffic in window, or logging the wrong vhost Confirm `access_log ... maya_json;` is on the serving `server {}` block; widen `--since`. Lines skipped silently Non-JSON lines (BOM, partial writes during rotation) Filter runs after rotation; point `--log` at the rotated file, not the live one. `400 Bad Request` from Maya Schema mismatch (extra/missing field) Inspect output NDJSON; the record keys must match exactly. `401 Unauthorized` API key missing/invalid Re-check the env var / secret manager binding. `429 Too Many Requests` Burst over rate limit Add exponential backoff in `upload.sh` or split the batch. Timestamps off by hours `$time_iso8601` uses server local offset The script normalizes to UTC; confirm the host clock/timezone is correct. ## [](#migration-note)Migration note If you later move logging to a BFF or API gateway, retire this filter in favor of the [`bff-endpoint.md`](./bff-endpoint.md) pattern. The schema is identical, so dashboards and downstream analysis are unaffected. [PreviousIIS / Windows Server](/docs/integrations/log-export/iis)[Next Apache](/docs/integrations/log-export/apache)