BFF / API Gateway Log Export

This page documents the cleanest log-export pattern: emit the Maya record inline from a Backend-for-Frontend, API gateway, or edge-middleware layer, instead of parsing web-server logs after the fact.

When to use this guide: you have a code layer that sees every inbound request — an Express/Fastify BFF, a Next.js middleware, an API gateway with a custom plugin, or a service mesh sidecar. Because you're already in code, you build the minimized record directly and never touch raw log files. If your traffic terminates at plain IIS/Nginx/Apache with no such layer, use those stack guides instead.

Why this is preferred

• No log parsing. You construct the exact schema at request time — no format drift, no rotation edge cases.
• Minimization by construction. The denylisted fields never enter the record, so there's nothing to strip later.
• Same schema. Output is byte-for-byte compatible with every other stack, so dashboards are unaffected.

Two delivery shapes

Both use the identical record schema and the identical allowlist / denylist.

The shared record

Every request that matches the LLM bot allowlist becomes one record:

{
  "timestamp":      "2026-07-28T03:14:07Z",
  "user_agent":     "…GPTBot/1.2…",
  "request_method": "GET",
  "request_path":   "/products/super-widget?ref=home",
  "status_code":    200,
  "response_bytes": 48329,
  "referrer":       "",
  "host":           "www.example.com",
  "client_ip_hash": "a4c6f1b2e7d8c1bb"
}

request_path has denylisted query keys removed; client_ip_hash is HMAC-SHA256 of the client IP keyed by your tenant secret (Maya never learns it). Non-bot requests are dropped entirely.

Push pattern — Express/Node middleware

Drop this in as early middleware. It builds the record, writes non-blockingly to a rotating NDJSON file, and skips everything that isn't an allowlisted bot.

// maya-bot-logger.ts  — framework-agnostic core + Express adapter
import { createHmac } from "node:crypto";
import { appendFile } from "node:fs/promises";
import type { Request, Response, NextFunction } from "express";
 
const BOT_PATTERNS = [
  "gptbot", "chatgpt-user", "oai-searchbot",
  "claudebot", "claude-web", "anthropic-ai",
  "perplexitybot", "perplexity-user",
  "google-extended", "googlebot", "bingbot",
  "applebot-extended", "bytespider", "ccbot",
  "meta-externalagent", "facebookbot",
  "duckassistbot", "youbot", "amazonbot", "diffbot",
];
const DENY_KEYS = new Set([
  "email", "tckn", "phone", "customer_id", "account_id",
  "session", "token", "key", "secret", "auth", "csrf",
]);
 
const TENANT_SECRET = process.env.MAYA_TENANT_SECRET!; // never hard-code
 
const isBot = (ua: string) => {
  const u = ua.toLowerCase();
  return BOT_PATTERNS.some((p) => u.includes(p));
};
 
const hashIp = (ip: string) =>
  ip ? createHmac("sha256", TENANT_SECRET).update(ip).digest("hex").slice(0, 32) : null;
 
function sanitizePath(rawUrl: string): string {
  const [path, query] = rawUrl.split("?");
  if (!query) return path;
  const kept = query
    .split("&")
    .filter((pair) => pair && !DENY_KEYS.has(pair.split("=")[0].toLowerCase()));
  return kept.length ? `${path}?${kept.join("&")}` : path;
}
 
// Trust the left-most X-Forwarded-For hop only if you terminate a trusted proxy.
function clientIp(req: Request): string {
  const xff = (req.headers["x-forwarded-for"] as string | undefined)?.split(",")[0]?.trim();
  return xff || req.socket.remoteAddress || "";
}
 
export function mayaBotLogger(logFile: string) {
  return (req: Request, res: Response, next: NextFunction) => {
    const ua = req.headers["user-agent"] ?? "";
    if (!isBot(ua)) return next();
 
    // Capture response-side fields once the response finishes.
    res.on("finish", () => {
      const record = {
        timestamp: new Date().toISOString().replace(/\.\d+Z$/, "Z"),
        user_agent: ua,
        request_method: req.method,
        request_path: sanitizePath(req.originalUrl),
        status_code: res.statusCode,
        response_bytes: Number(res.getHeader("content-length") ?? 0),
        referrer: (req.headers["referer"] as string) ?? "",
        host: req.headers["host"] ?? "",
        client_ip_hash: hashIp(clientIp(req)),
      };
      // Fire-and-forget; never block the request path.
      appendFile(logFile, JSON.stringify(record) + "\n").catch(() => {});
    });
 
    next();
  };
}

Wire it up before your routes:

import express from "express";
import { mayaBotLogger } from "./maya-bot-logger";
 
const app = express();
app.set("trust proxy", true); // if you terminate a trusted proxy/load balancer
app.use(mayaBotLogger("/var/maya/export-current.ndjson"));
// … your routes …

Then upload the rotated file daily/weekly (same call as every stack):

curl -sS -X POST 'https://ingest.withmaya.ai/v1/logs' \
  -H "Authorization: Bearer $MAYA_API_KEY" \
  -H 'X-Maya-Tenant: acme-prod' \
  -H 'X-Maya-Schema: 1' \
  -H 'Content-Type: application/x-ndjson' \
  --data-binary @/var/maya/export-2026-07-28.ndjson \
  -w '%{http_code}\n' -o /dev/null

Returns 202 Accepted with a batch ID. For high volume, buffer in memory and POST in batches instead of appending per request.

Other runtimes: the four helpers (isBot, sanitizePath, hashIp, clientIp) are ~30 lines of pure logic — port them to Fastify hooks, Next.js middleware, a Kong/Envoy plugin, or a Go/Java gateway filter. The record shape is what matters, not the framework.

Pull pattern — mTLS endpoint

Instead of pushing, expose a read-only endpoint that returns the same NDJSON for a time window, secured by mutual TLS. Maya fetches on a schedule.

Contract:

GET https://logs.acme-prod.example.com/maya/logs?since=2026-07-28T00:00:00Z&until=2026-07-29T00:00:00Z

• Transport: mutual TLS. Maya presents a client certificate you pin; you present a server certificate Maya pins. No bearer token travels the wire.
• Response: 200 OK, Content-Type: application/x-ndjson, body = the minimized records in the window (same schema as above). Empty window → 200 with an empty body.
• Idempotent & bounded: the same since/until must return the same set. Cap the window server-side (e.g. reject ranges > 31 days) and paginate with a next cursor header if needed.
• Source of records: back the endpoint with the push middleware's output, a DB table you populate at request time, or a streaming buffer — as long as each row already conforms to the schema (bots only, sanitized, hashed).

Maya provides the client certificate and the exact scheduling cadence during onboarding.

Validation checklist

• Confirm the middleware runs before auth/session middleware writes anything sensitive into the URL, and that sanitizePath drops every denylisted key.
• Emit 24h to a test file. Inspect 100 rows: no raw IPs; no cookies/auth/session; no PII in request_path; every user_agent on the allowlist.
• Verify client_ip_hash is stable for a fixed IP and changes when MAYA_TENANT_SECRET rotates.
• POST a sample batch with X-Maya-Mode: test (push) or serve a test window over mTLS (pull) and confirm Maya accepts the schema.
• Load-test: confirm the res.on("finish") hook adds no measurable latency to the request path.
• Sign off in change management; enable.

Troubleshooting

Relationship to the other guides

This pattern supersedes the IIS, Nginx, and Apache filters when you have a code layer in the request path. The output schema is identical, so you can migrate from a log-parsing filter to this endpoint with zero change to dashboards or downstream analysis.