# [](#bff--api-gateway-log-export)BFF / API Gateway Log Export

This page documents the cleanest log-export pattern: emit the Maya record **inline** from a Backend-for-Frontend, API gateway, or edge-middleware layer, instead of parsing web-server logs after the fact.

> **When to use this guide:** you have a code layer that sees every inbound request — an Express/Fastify BFF, a Next.js middleware, an API gateway with a custom plugin, or a service mesh sidecar. Because you're already in code, you build the minimized record directly and never touch raw log files. If your traffic terminates at plain IIS/Nginx/Apache with no such layer, use those stack guides instead.

## [](#why-this-is-preferred)Why this is preferred

*   **No log parsing.** You construct the exact schema at request time — no format drift, no rotation edge cases.
*   **Minimization by construction.** The denylisted fields never enter the record, so there's nothing to strip later.
*   **Same schema.** Output is byte-for-byte compatible with every other stack, so dashboards are unaffected.

## [](#two-delivery-shapes)Two delivery shapes

Shape

How it works

Choose when

**Push (batch upload)**

Your service appends minimized records to a daily NDJSON file (or buffer) and POSTs them to Maya.

Default. Simplest.

**Pull (mTLS)**

You expose a read-only, mutually-authenticated endpoint; Maya fetches the window on a schedule.

The brand prefers Maya to pull from a brand-controlled endpoint (no outbound egress rules to manage).

Both use the identical record schema and the identical allowlist / denylist.

## [](#the-shared-record)The shared record

Every request that matches the LLM bot allowlist becomes one record:

JSON

```
{
  "timestamp":      "2026-07-28T03:14:07Z",
  "user_agent":     "…GPTBot/1.2…",
  "request_method": "GET",
  "request_path":   "/products/super-widget?ref=home",
  "status_code":    200,
  "response_bytes": 48329,
  "referrer":       "",
  "host":           "www.example.com",
  "client_ip_hash": "a4c6f1b2e7d8c1bb"
}
```

`request_path` has denylisted query keys removed; `client_ip_hash` is HMAC-SHA256 of the client IP keyed by your tenant secret (Maya never learns it). Non-bot requests are dropped entirely.

## [](#push-pattern--expressnode-middleware)Push pattern — Express/Node middleware

Drop this in as early middleware. It builds the record, writes non-blockingly to a rotating NDJSON file, and skips everything that isn't an allowlisted bot.

TypeScript

```
// maya-bot-logger.ts  — framework-agnostic core + Express adapter
import { createHmac } from "node:crypto";
import { appendFile } from "node:fs/promises";
import type { Request, Response, NextFunction } from "express";
 
const BOT_PATTERNS = [
  "gptbot", "chatgpt-user", "oai-searchbot",
  "claudebot", "claude-web", "anthropic-ai",
  "perplexitybot", "perplexity-user",
  "google-extended", "googlebot", "bingbot",
  "applebot-extended", "bytespider", "ccbot",
  "meta-externalagent", "facebookbot",
  "duckassistbot", "youbot", "amazonbot", "diffbot",
];
const DENY_KEYS = new Set([
  "email", "tckn", "phone", "customer_id", "account_id",
  "session", "token", "key", "secret", "auth", "csrf",
]);
 
const TENANT_SECRET = process.env.MAYA_TENANT_SECRET!; // never hard-code
 
const isBot = (ua: string) => {
  const u = ua.toLowerCase();
  return BOT_PATTERNS.some((p) => u.includes(p));
};
 
const hashIp = (ip: string) =>
  ip ? createHmac("sha256", TENANT_SECRET).update(ip).digest("hex").slice(0, 32) : null;
 
function sanitizePath(rawUrl: string): string {
  const [path, query] = rawUrl.split("?");
  if (!query) return path;
  const kept = query
    .split("&")
    .filter((pair) => pair && !DENY_KEYS.has(pair.split("=")[0].toLowerCase()));
  return kept.length ? `${path}?${kept.join("&")}` : path;
}
 
// Trust the left-most X-Forwarded-For hop only if you terminate a trusted proxy.
function clientIp(req: Request): string {
  const xff = (req.headers["x-forwarded-for"] as string | undefined)?.split(",")[0]?.trim();
  return xff || req.socket.remoteAddress || "";
}
 
export function mayaBotLogger(logFile: string) {
  return (req: Request, res: Response, next: NextFunction) => {
    const ua = req.headers["user-agent"] ?? "";
    if (!isBot(ua)) return next();
 
    // Capture response-side fields once the response finishes.
    res.on("finish", () => {
      const record = {
        timestamp: new Date().toISOString().replace(/\.\d+Z$/, "Z"),
        user_agent: ua,
        request_method: req.method,
        request_path: sanitizePath(req.originalUrl),
        status_code: res.statusCode,
        response_bytes: Number(res.getHeader("content-length") ?? 0),
        referrer: (req.headers["referer"] as string) ?? "",
        host: req.headers["host"] ?? "",
        client_ip_hash: hashIp(clientIp(req)),
      };
      // Fire-and-forget; never block the request path.
      appendFile(logFile, JSON.stringify(record) + "\n").catch(() => {});
    });
 
    next();
  };
}
```

Wire it up **before** your routes:

TypeScript

```
import express from "express";
import { mayaBotLogger } from "./maya-bot-logger";
 
const app = express();
app.set("trust proxy", true); // if you terminate a trusted proxy/load balancer
app.use(mayaBotLogger("/var/maya/export-current.ndjson"));
// … your routes …
```

Then upload the rotated file daily/weekly (same call as every stack):

Bash

```
curl -sS -X POST 'https://ingest.withmaya.ai/v1/logs' \
  -H "Authorization: Bearer $MAYA_API_KEY" \
  -H 'X-Maya-Tenant: acme-prod' \
  -H 'X-Maya-Schema: 1' \
  -H 'Content-Type: application/x-ndjson' \
  --data-binary @/var/maya/export-2026-07-28.ndjson \
  -w '%{http_code}\n' -o /dev/null
```

Returns `202 Accepted` with a batch ID. For high volume, buffer in memory and POST in batches instead of appending per request.

> **Other runtimes:** the four helpers (`isBot`, `sanitizePath`, `hashIp`, `clientIp`) are ~30 lines of pure logic — port them to Fastify hooks, Next.js middleware, a Kong/Envoy plugin, or a Go/Java gateway filter. The record shape is what matters, not the framework.

## [](#pull-pattern--mtls-endpoint)Pull pattern — mTLS endpoint

Instead of pushing, expose a read-only endpoint that returns the same NDJSON for a time window, secured by mutual TLS. Maya fetches on a schedule.

**Contract:**

Plaintext

```
GET https://logs.acme-prod.example.com/maya/logs?since=2026-07-28T00:00:00Z&until=2026-07-29T00:00:00Z
```

*   **Transport:** mutual TLS. Maya presents a client certificate you pin; you present a server certificate Maya pins. No bearer token travels the wire.
*   **Response:** `200 OK`, `Content-Type: application/x-ndjson`, body = the minimized records in the window (same schema as above). Empty window → `200` with an empty body.
*   **Idempotent & bounded:** the same `since`/`until` must return the same set. Cap the window server-side (e.g. reject ranges > 31 days) and paginate with a `next` cursor header if needed.
*   **Source of records:** back the endpoint with the push middleware's output, a DB table you populate at request time, or a streaming buffer — as long as each row already conforms to the schema (bots only, sanitized, hashed).

Maya provides the client certificate and the exact scheduling cadence during onboarding.

## [](#validation-checklist)Validation checklist

*    Confirm the middleware runs **before** auth/session middleware writes anything sensitive into the URL, and that `sanitizePath` drops every denylisted key.
*    Emit 24h to a test file. Inspect 100 rows: no raw IPs; no cookies/auth/session; no PII in `request_path`; every `user_agent` on the allowlist.
*    Verify `client_ip_hash` is stable for a fixed IP and changes when `MAYA_TENANT_SECRET` rotates.
*    POST a sample batch with `X-Maya-Mode: test` (push) or serve a test window over mTLS (pull) and confirm Maya accepts the schema.
*    Load-test: confirm the `res.on("finish")` hook adds no measurable latency to the request path.
*    Sign off in change management; enable.

## [](#troubleshooting)Troubleshooting

Symptom

Cause

Fix

`response_bytes` is `0` for streamed responses

`Content-Length` not set when streaming

Track bytes written on the response stream, or accept `0` (Maya treats it as unknown).

`client_ip_hash` is the proxy IP

`trust proxy` off, or XFF not forwarded

Enable `trust proxy` and ensure the edge sets `X-Forwarded-For`.

Duplicate records

Middleware mounted twice, or ret\[ry\] on the uploader

Mount once; make the uploader idempotent per batch file.

`400 Bad Request` from Maya

Extra field crept into the record

The record must contain exactly the nine schema keys — no `id`, no headers object.

High memory under load

Unbounded in-memory buffer

Flush to disk / POST on a size or time threshold.

## [](#relationship-to-the-other-guides)Relationship to the other guides

This pattern supersedes the [IIS](./iis.md), [Nginx](./nginx.md), and [Apache](./apache.md) filters when you have a code layer in the request path. The output schema is identical, so you can migrate from a log-parsing filter to this endpoint with **zero** change to dashboards or downstream analysis.

[PreviousApache](/docs/integrations/log-export/apache)[Next Overview](/docs/integrations/markdown-rendering/overview)