Integrations — BFF / API Gateway Log Export
Copy the prompt and paste it into Cursor, Claude Code, Codex, Copilot… — it does the install for you.
This page documents the cleanest log-export pattern: emit the Maya record inline from a Backend-for-Frontend, API gateway, or edge-middleware layer, instead of parsing web-server logs after the fact.
When to use this guide: you have a code layer that sees every inbound request — an Express/Fastify BFF, a Next.js middleware, an API gateway with a custom plugin, or a service mesh sidecar. Because you're already in code, you build the minimized record directly and never touch raw log files. If your traffic terminates at plain IIS/Nginx/Apache with no such layer, use those stack guides instead.
Why this is preferred
- No log parsing. You construct the exact schema at request time — no format drift, no rotation edge cases.
- Minimization by construction. The denylisted fields never enter the record, so there's nothing to strip later.
- Same schema. Output is byte-for-byte compatible with every other stack, so dashboards are unaffected.
Two delivery shapes
| Shape | How it works | Choose when |
|---|---|---|
| Push (batch upload) | Your service appends minimized records to a daily NDJSON file (or buffer) and POSTs them to Maya. | Default. Simplest. |
| Pull (mTLS) | You expose a read-only, mutually-authenticated endpoint; Maya fetches the window on a schedule. | The brand prefers Maya to pull from a brand-controlled endpoint (no outbound egress rules to manage). |
Both use the identical record schema and the identical allowlist / denylist.
The shared record
Every request that matches the LLM bot allowlist becomes one record:
{
"timestamp": "2026-07-28T03:14:07Z",
"user_agent": "…GPTBot/1.2…",
"request_method": "GET",
"request_path": "/products/super-widget?ref=home",
"status_code": 200,
"response_bytes": 48329,
"referrer": "",
"host": "www.example.com",
"client_ip_hash": "a4c6f1b2e7d8c1bb"
}request_path has denylisted query keys removed; client_ip_hash is HMAC-SHA256 of the client IP keyed by your tenant secret (Maya never learns it). Non-bot requests are dropped entirely.
Push pattern — Express/Node middleware
Drop this in as early middleware. It builds the record, writes non-blockingly to a rotating NDJSON file, and skips everything that isn't an allowlisted bot.
// maya-bot-logger.ts — framework-agnostic core + Express adapter
import { createHmac } from "node:crypto";
import { appendFile } from "node:fs/promises";
import type { Request, Response, NextFunction } from "express";
const BOT_PATTERNS = [
"gptbot", "chatgpt-user", "oai-searchbot",
"claudebot", "claude-web", "anthropic-ai",
"perplexitybot", "perplexity-user",
"google-extended", "googlebot", "bingbot",
"applebot-extended", "bytespider", "ccbot",
"meta-externalagent", "facebookbot",
"duckassistbot", "youbot", "amazonbot", "diffbot",
];
const DENY_KEYS = new Set([
"email", "tckn", "phone", "customer_id", "account_id",
"session", "token", "key", "secret", "auth", "csrf",
]);
const TENANT_SECRET = process.env.MAYA_TENANT_SECRET!; // never hard-code
const isBot = (ua: string) => {
const u = ua.toLowerCase();
return BOT_PATTERNS.some((p) => u.includes(p));
};
const hashIp = (ip: string) =>
ip ? createHmac("sha256", TENANT_SECRET).update(ip).digest("hex").slice(0, 32) : null;
function sanitizePath(rawUrl: string): string {
const [path, query] = rawUrl.split("?");
if (!query) return path;
const kept = query
.split("&")
.filter((pair) => pair && !DENY_KEYS.has(pair.split("=")[0].toLowerCase()));
return kept.length ? `${path}?${kept.join("&")}` : path;
}
// Trust the left-most X-Forwarded-For hop only if you terminate a trusted proxy.
function clientIp(req: Request): string {
const xff = (req.headers["x-forwarded-for"] as string | undefined)?.split(",")[0]?.trim();
return xff || req.socket.remoteAddress || "";
}
export function mayaBotLogger(logFile: string) {
return (req: Request, res: Response, next: NextFunction) => {
const ua = req.headers["user-agent"] ?? "";
if (!isBot(ua)) return next();
// Capture response-side fields once the response finishes.
res.on("finish", () => {
const record = {
timestamp: new Date().toISOString().replace(/\.\d+Z$/, "Z"),
user_agent: ua,
request_method: req.method,
request_path: sanitizePath(req.originalUrl),
status_code: res.statusCode,
response_bytes: Number(res.getHeader("content-length") ?? 0),
referrer: (req.headers["referer"] as string) ?? "",
host: req.headers["host"] ?? "",
client_ip_hash: hashIp(clientIp(req)),
};
// Fire-and-forget; never block the request path.
appendFile(logFile, JSON.stringify(record) + "\n").catch(() => {});
});
next();
};
}Wire it up before your routes:
import express from "express";
import { mayaBotLogger } from "./maya-bot-logger";
const app = express();
app.set("trust proxy", true); // if you terminate a trusted proxy/load balancer
app.use(mayaBotLogger("/var/maya/export-current.ndjson"));
// … your routes …Then upload the rotated file daily/weekly (same call as every stack):
# One request takes at most 5,000 records, so split the export first.
split -l 5000 /var/maya/export-2026-07-28.ndjson /var/maya/part-
for part in /var/maya/part-*; do
curl -sS -X POST 'https://logs.withmaya.ai/v1/logs' \
-H "x-maya-signal-project: $MAYA_SIGNAL_PROJECT" \
-H "x-maya-signal-key: $MAYA_SIGNAL_KEY" \
-H "x-maya-signal-source: $MAYA_SIGNAL_SOURCE" \
-H 'Content-Type: application/x-ndjson' \
--data-binary @"$part"
echo
done
rm /var/maya/part-*The endpoint returns 202 with { received, agent_events, forwarded, rejected, missing_url }. A 202 alone doesn't confirm delivery: check that forwarded is above 0, and that rejected and missing_url are 0. Copy the project, key and source values from the Streaming setup in LLM Site Visibility. See the API reference. For high volume, buffer in memory and POST in batches instead of appending per request.
Other runtimes: the four helpers (
isBot,sanitizePath,hashIp,clientIp) are ~30 lines of pure logic — port them to Fastify hooks, Next.js middleware, a Kong/Envoy plugin, or a Go/Java gateway filter. The record shape is what matters, not the framework.
Pull pattern — mTLS endpoint
Instead of pushing, expose a read-only endpoint that returns the same NDJSON for a time window, secured by mutual TLS. Maya fetches on a schedule.
Contract:
GET https://logs.acme-prod.example.com/maya/logs?since=2026-07-28T00:00:00Z&until=2026-07-29T00:00:00Z- Transport: mutual TLS. Maya presents a client certificate you pin; you present a server certificate Maya pins. No bearer token travels the wire.
- Response:
200 OK,Content-Type: application/x-ndjson, body = the minimized records in the window (same schema as above). Empty window →200with an empty body. - Idempotent & bounded: the same
since/untilmust return the same set. Cap the window server-side (e.g. reject ranges > 31 days) and paginate with anextcursor header if needed. - Source of records: back the endpoint with the push middleware's output, a DB table you populate at request time, or a streaming buffer — as long as each row already conforms to the schema (bots only, sanitized, hashed).
Maya provides the client certificate and the exact scheduling cadence during onboarding.
Validation checklist
- Confirm the middleware runs before auth/session middleware writes anything sensitive into the URL, and that
sanitizePathdrops every denylisted key. - Emit 24h to a test file. Inspect 100 rows: no raw IPs; no cookies/auth/session; no PII in
request_path; everyuser_agenton the allowlist. - Verify
client_ip_hashis stable for a fixed IP and changes whenMAYA_TENANT_SECRETrotates. - Push: POST a small sample and confirm the response shows
forwardedabove 0, withrejectedandmissing_urlat 0. The endpoint has no test mode, so sample records appear in your dashboard. Pull: serve a test window over mTLS and confirm Maya accepts the schema. - Load-test: confirm the
res.on("finish")hook adds no measurable latency to the request path. - Sign off in change management; enable.
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
response_bytes is 0 for streamed responses | Content-Length not set when streaming | Track bytes written on the response stream, or accept 0 (Maya treats it as unknown). |
client_ip_hash is the proxy IP | trust proxy off, or XFF not forwarded | Enable trust proxy and ensure the edge sets X-Forwarded-For. |
| Duplicate records | Middleware mounted twice, or ret[ry] on the uploader | Mount once; make the uploader idempotent per batch file. |
missing_url above 0 in the response | Records have no recognised URL field, so paths show as / | Send the path as request_path, path, url or request_uri. |
| High memory under load | Unbounded in-memory buffer | Flush to disk / POST on a size or time threshold. |
Relationship to the other guides
This pattern supersedes the IIS, Nginx, and Apache filters when you have a code layer in the request path. The output schema is identical, so you can migrate from a log-parsing filter to this endpoint with zero change to dashboards or downstream analysis.