MayaBook a demo
Maya

Integrations — BFF / API Gateway Log Export

Audience: Bank IT, backend/platform engineers, AI agents performing integrationUpdated 2026-07-29
Claude CodeCodexGitHub Copilot
Set this up with your coding agent

Copy the prompt and paste it into Cursor, Claude Code, Codex, Copilot… — it does the install for you.

This page documents the cleanest log-export pattern: emit the Maya record inline from a Backend-for-Frontend, API gateway, or edge-middleware layer, instead of parsing web-server logs after the fact.

When to use this guide: you have a code layer that sees every inbound request — an Express/Fastify BFF, a Next.js middleware, an API gateway with a custom plugin, or a service mesh sidecar. Because you're already in code, you build the minimized record directly and never touch raw log files. If your traffic terminates at plain IIS/Nginx/Apache with no such layer, use those stack guides instead.

Why this is preferred

  • No log parsing. You construct the exact schema at request time — no format drift, no rotation edge cases.
  • Minimization by construction. The denylisted fields never enter the record, so there's nothing to strip later.
  • Same schema. Output is byte-for-byte compatible with every other stack, so dashboards are unaffected.

Two delivery shapes

ShapeHow it worksChoose when
Push (batch upload)Your service appends minimized records to a daily NDJSON file (or buffer) and POSTs them to Maya.Default. Simplest.
Pull (mTLS)You expose a read-only, mutually-authenticated endpoint; Maya fetches the window on a schedule.The brand prefers Maya to pull from a brand-controlled endpoint (no outbound egress rules to manage).

Both use the identical record schema and the identical allowlist / denylist.

The shared record

Every request that matches the LLM bot allowlist becomes one record:

JSON
{
  "timestamp":      "2026-07-28T03:14:07Z",
  "user_agent":     "…GPTBot/1.2…",
  "request_method": "GET",
  "request_path":   "/products/super-widget?ref=home",
  "status_code":    200,
  "response_bytes": 48329,
  "referrer":       "",
  "host":           "www.example.com",
  "client_ip_hash": "a4c6f1b2e7d8c1bb"
}

request_path has denylisted query keys removed; client_ip_hash is HMAC-SHA256 of the client IP keyed by your tenant secret (Maya never learns it). Non-bot requests are dropped entirely.

Push pattern — Express/Node middleware

Drop this in as early middleware. It builds the record, writes non-blockingly to a rotating NDJSON file, and skips everything that isn't an allowlisted bot.

TypeScript
// maya-bot-logger.ts  — framework-agnostic core + Express adapter
import { createHmac } from "node:crypto";
import { appendFile } from "node:fs/promises";
import type { Request, Response, NextFunction } from "express";
 
const BOT_PATTERNS = [
  "gptbot", "chatgpt-user", "oai-searchbot",
  "claudebot", "claude-web", "anthropic-ai",
  "perplexitybot", "perplexity-user",
  "google-extended", "googlebot", "bingbot",
  "applebot-extended", "bytespider", "ccbot",
  "meta-externalagent", "facebookbot",
  "duckassistbot", "youbot", "amazonbot", "diffbot",
];
const DENY_KEYS = new Set([
  "email", "tckn", "phone", "customer_id", "account_id",
  "session", "token", "key", "secret", "auth", "csrf",
]);
 
const TENANT_SECRET = process.env.MAYA_TENANT_SECRET!; // never hard-code
 
const isBot = (ua: string) => {
  const u = ua.toLowerCase();
  return BOT_PATTERNS.some((p) => u.includes(p));
};
 
const hashIp = (ip: string) =>
  ip ? createHmac("sha256", TENANT_SECRET).update(ip).digest("hex").slice(0, 32) : null;
 
function sanitizePath(rawUrl: string): string {
  const [path, query] = rawUrl.split("?");
  if (!query) return path;
  const kept = query
    .split("&")
    .filter((pair) => pair && !DENY_KEYS.has(pair.split("=")[0].toLowerCase()));
  return kept.length ? `${path}?${kept.join("&")}` : path;
}
 
// Trust the left-most X-Forwarded-For hop only if you terminate a trusted proxy.
function clientIp(req: Request): string {
  const xff = (req.headers["x-forwarded-for"] as string | undefined)?.split(",")[0]?.trim();
  return xff || req.socket.remoteAddress || "";
}
 
export function mayaBotLogger(logFile: string) {
  return (req: Request, res: Response, next: NextFunction) => {
    const ua = req.headers["user-agent"] ?? "";
    if (!isBot(ua)) return next();
 
    // Capture response-side fields once the response finishes.
    res.on("finish", () => {
      const record = {
        timestamp: new Date().toISOString().replace(/\.\d+Z$/, "Z"),
        user_agent: ua,
        request_method: req.method,
        request_path: sanitizePath(req.originalUrl),
        status_code: res.statusCode,
        response_bytes: Number(res.getHeader("content-length") ?? 0),
        referrer: (req.headers["referer"] as string) ?? "",
        host: req.headers["host"] ?? "",
        client_ip_hash: hashIp(clientIp(req)),
      };
      // Fire-and-forget; never block the request path.
      appendFile(logFile, JSON.stringify(record) + "\n").catch(() => {});
    });
 
    next();
  };
}

Wire it up before your routes:

TypeScript
import express from "express";
import { mayaBotLogger } from "./maya-bot-logger";
 
const app = express();
app.set("trust proxy", true); // if you terminate a trusted proxy/load balancer
app.use(mayaBotLogger("/var/maya/export-current.ndjson"));
// … your routes …

Then upload the rotated file daily/weekly (same call as every stack):

Bash
# One request takes at most 5,000 records, so split the export first.
split -l 5000 /var/maya/export-2026-07-28.ndjson /var/maya/part-
for part in /var/maya/part-*; do
  curl -sS -X POST 'https://logs.withmaya.ai/v1/logs' \
    -H "x-maya-signal-project: $MAYA_SIGNAL_PROJECT" \
    -H "x-maya-signal-key: $MAYA_SIGNAL_KEY" \
    -H "x-maya-signal-source: $MAYA_SIGNAL_SOURCE" \
    -H 'Content-Type: application/x-ndjson' \
    --data-binary @"$part"
  echo
done
rm /var/maya/part-*

The endpoint returns 202 with { received, agent_events, forwarded, rejected, missing_url }. A 202 alone doesn't confirm delivery: check that forwarded is above 0, and that rejected and missing_url are 0. Copy the project, key and source values from the Streaming setup in LLM Site Visibility. See the API reference. For high volume, buffer in memory and POST in batches instead of appending per request.

Other runtimes: the four helpers (isBot, sanitizePath, hashIp, clientIp) are ~30 lines of pure logic — port them to Fastify hooks, Next.js middleware, a Kong/Envoy plugin, or a Go/Java gateway filter. The record shape is what matters, not the framework.

Pull pattern — mTLS endpoint

Instead of pushing, expose a read-only endpoint that returns the same NDJSON for a time window, secured by mutual TLS. Maya fetches on a schedule.

Contract:

Plaintext
GET https://logs.acme-prod.example.com/maya/logs?since=2026-07-28T00:00:00Z&until=2026-07-29T00:00:00Z
  • Transport: mutual TLS. Maya presents a client certificate you pin; you present a server certificate Maya pins. No bearer token travels the wire.
  • Response: 200 OK, Content-Type: application/x-ndjson, body = the minimized records in the window (same schema as above). Empty window → 200 with an empty body.
  • Idempotent & bounded: the same since/until must return the same set. Cap the window server-side (e.g. reject ranges > 31 days) and paginate with a next cursor header if needed.
  • Source of records: back the endpoint with the push middleware's output, a DB table you populate at request time, or a streaming buffer — as long as each row already conforms to the schema (bots only, sanitized, hashed).

Maya provides the client certificate and the exact scheduling cadence during onboarding.

Validation checklist

  • Confirm the middleware runs before auth/session middleware writes anything sensitive into the URL, and that sanitizePath drops every denylisted key.
  • Emit 24h to a test file. Inspect 100 rows: no raw IPs; no cookies/auth/session; no PII in request_path; every user_agent on the allowlist.
  • Verify client_ip_hash is stable for a fixed IP and changes when MAYA_TENANT_SECRET rotates.
  • Push: POST a small sample and confirm the response shows forwarded above 0, with rejected and missing_url at 0. The endpoint has no test mode, so sample records appear in your dashboard. Pull: serve a test window over mTLS and confirm Maya accepts the schema.
  • Load-test: confirm the res.on("finish") hook adds no measurable latency to the request path.
  • Sign off in change management; enable.

Troubleshooting

SymptomCauseFix
response_bytes is 0 for streamed responsesContent-Length not set when streamingTrack bytes written on the response stream, or accept 0 (Maya treats it as unknown).
client_ip_hash is the proxy IPtrust proxy off, or XFF not forwardedEnable trust proxy and ensure the edge sets X-Forwarded-For.
Duplicate recordsMiddleware mounted twice, or ret[ry] on the uploaderMount once; make the uploader idempotent per batch file.
missing_url above 0 in the responseRecords have no recognised URL field, so paths show as /Send the path as request_path, path, url or request_uri.
High memory under loadUnbounded in-memory bufferFlush to disk / POST on a size or time threshold.

This pattern supersedes the IIS, Nginx, and Apache filters when you have a code layer in the request path. The output schema is identical, so you can migrate from a log-parsing filter to this endpoint with zero change to dashboards or downstream analysis.