Apache Log Export

This page documents how to extract LLM bot traffic from Apache HTTP Server access logs and ship it to Maya, with all data minimization rules applied at source.

When to use this guide: your production stack is Apache-hosted (origin or reverse proxy) with access logs on the host or a log-aggregation box. If Apache sits behind a CDN you can connect natively, prefer the Connect Ready path. If a BFF / API gateway already builds request records, see bff-endpoint.md.

The filtering, output schema, transmission, and validation are identical to the Nginx guide — only the log-format definition differs. This page gives the Apache-specific config; the filter script is shared.

Prerequisites

Recommended: a JSON access log

Define a LogFormat that emits exactly the fields Maya needs, matching the JSON keys the filter expects. Add to your vhost or httpd.conf:

LogFormat "{ \"time\":\"%{%Y-%m-%dT%H:%M:%S%z}t\", \"method\":\"%m\", \"uri\":\"%U\", \"query\":\"%q\", \"status\":%>s, \"bytes\":%B, \"referer\":\"%{Referer}i\", \"host\":\"%{Host}i\", \"ua\":\"%{User-Agent}i\", \"ip\":\"%a\" }" maya_json
 
# Separate log — leaves your existing logging untouched.
CustomLog "/var/log/apache2/maya_access.log" maya_json

Field notes:

• %U is the URL path without the query string; %q is the query string with a leading ?. The shared filter tolerates the leading ?, so no change is needed.
• %B is bytes sent excluding headers, and emits 0 (not -) for empty responses — safe to parse as an integer.
• %a is the client IP. If Apache is behind a trusted proxy and you want the real client, use %{c}a with mod_remoteip configured; the IP is hashed either way.
• The %{...}t time token emits an offset like +0000; the filter normalizes it to UTC.

Reload Apache (apachectl configtest && systemctl reload apache2). As with Nginx, this log still contains all traffic and raw IPs — the filter is what removes non-bot rows and hashes IPs before anything leaves your network.

Filter script

Use the same maya_bot_filter.py from the Nginx guide — it is format-agnostic because both stacks emit the same JSON keys. Run it against the Apache log:

MAYA_TENANT_SECRET=… python3 maya_bot_filter.py \
  --log  /var/log/apache2/maya_access.log \
  --since 2026-07-28T00:00:00Z \
  --out  /var/maya/export-2026-07-28.ndjson \
  --dry-run

The script keeps only allowlisted LLM bots, strips denylisted query keys, HMAC-hashes the client IP with your tenant secret (Maya never learns it), and writes NDJSON. See the Nginx guide's script notes for the full behavior.

Expected output (sample)

Identical schema to every other stack — one bot request per line:

{"timestamp":"2026-07-28T03:14:07Z","user_agent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)","request_method":"GET","request_path":"/products/super-widget","status_code":200,"response_bytes":48329,"referrer":"","host":"www.example.com","client_ip_hash":"a4c6f1b2e7d8c1bb"}
{"timestamp":"2026-07-28T03:15:11Z","user_agent":"Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)","request_method":"GET","request_path":"/pricing","status_code":200,"response_bytes":62104,"referrer":"https://www.perplexity.ai/","host":"www.example.com","client_ip_hash":"3f8b22cc09111ab7"}

Download a full sample (15 rows, ~5 KB): maya-bot-logs.sample.ndjson

Sanity checks

The same three checks as the Nginx guide: valid JSON per line, no disallowed content, every user_agent on the allowlist.

Transmission

curl -sS -X POST 'https://ingest.withmaya.ai/v1/logs' \
  -H "Authorization: Bearer $MAYA_API_KEY" \
  -H 'X-Maya-Tenant: acme-prod' \
  -H 'X-Maya-Schema: 1' \
  -H 'Content-Type: application/x-ndjson' \
  --data-binary @/var/maya/export-2026-07-28.ndjson \
  -w '%{http_code}\n' -o /dev/null

Returns 202 Accepted on success with a batch ID.

Scheduling

# /etc/cron.d/maya-log-export  — daily at 03:00, prior day's window
0 3 * * * maya MAYA_TENANT_SECRET=@secret MAYA_API_KEY=@secret \
  /usr/bin/python3 /opt/maya/maya_bot_filter.py \
    --log /var/log/apache2/maya_access.log \
    --since "$(date -u -d 'yesterday 00:00' +\%Y-\%m-\%dT00:00:00Z)" \
    --out /var/maya/export-$(date -u +\%Y-\%m-\%d).ndjson \
  && /opt/maya/upload.sh /var/maya/export-$(date -u +\%Y-\%m-\%d).ndjson

Validation checklist

• apachectl configtest passes after adding the LogFormat / CustomLog.
• Run the filter with --dry-run on 24 hours of logs; retained count is 0.1%–5% of total rows.
• Inspect 100 random rows: no raw IPs; no cookies/auth/session; no PII in request_path; every user_agent on the allowlist.
• POST a sample batch with X-Maya-Mode: test to verify schema acceptance without ingesting.
• Sign off the config + script in change management.
• Schedule and enable.

Troubleshooting