Integrations — Markdown Rendering on SharePoint Client Site

Audience: Backend engineers, Product Owners, AI agents performing integrationUpdated 2026-04-28

Markdown Rendering — SharePoint Client Site (IIS-hosted)

This document describes how to serve a markdown-formatted version of public pages to verified LLM bots — when, and only when, they request it — on an IIS-hosted SharePoint Client Site (Microsoft stack with client-side HTML rendering). This is a common configuration for enterprise sites.

Goal

When a verified LLM bot requests a public page, return high-density markdown instead of full HTML/CSS/JavaScript. When any other client requests the same URL, return the normal SharePoint response. The behavior is invisible to end users.

Why this matters

Metric	Typical HTML response	Markdown response
Bytes per page	200 KB – 2 MB	5 KB – 80 KB
Useful content ratio	5% – 15%	90%+
Crawler retrieval time	high (full page weight)	low
RAG vectorization quality	mediocre (noise)	high (signal-dense)
Crawl budget consumed	high	low

Cloudflare formalized industry-level support for markdown-on-bot-request in late 2025. This is the same idea, implemented at the origin.

Architecture

Plaintext

                    ┌────────────────────────┐
   Request ────────▶│  IIS / Front Edge      │
                    │  (existing)            │
                    └─────────┬──────────────┘
                              │
                              ▼
                    ┌────────────────────────┐
                    │  Maya Header Router    │   ← new component
                    │  (URL Rewrite + ASP.   │
                    │   NET Core middleware) │
                    └─────────┬──────────────┘
                              │
              ┌───────────────┴───────────────┐
              │                               │
              ▼                               ▼
   ┌────────────────────┐          ┌────────────────────┐
   │ User-Agent ∈ allow │          │ Otherwise          │
   │ list and feature   │          │                    │
   │ flag enabled       │          │                    │
   ├────────────────────┤          ├────────────────────┤
   │ → /api/render/     │          │ → SharePoint       │
   │   markdown?path=…  │          │   Client Site      │
   │ → text/markdown    │          │ → text/html        │
   └────────────────────┘          └────────────────────┘

The router is a small ASP.NET Core middleware (or, equivalently, an IIS URL Rewrite rule) that runs in front of SharePoint, never inside it. SharePoint itself is unmodified.

Component A — Header Router

Pattern (ASP.NET Core middleware)

// Program.cs (or Startup.cs)
app.Use(async (context, next) =>
{
    if (!FeatureFlag.IsEnabled("maya.markdown_rendering"))
    {
        await next();
        return;
    }
 
    var ua = context.Request.Headers.UserAgent.ToString();
    if (LlmBotMatcher.IsLlmBot(ua) && context.Request.Method == "GET")
    {
        // Internal redirect to markdown renderer.
        var path = context.Request.Path.Value ?? "/";
        var query = context.Request.QueryString.HasValue
            ? context.Request.QueryString.Value
            : "";
        context.Request.Path = "/api/render/markdown";
        context.Request.QueryString = new QueryString($"?path={Uri.EscapeDataString(path + query)}");
    }
 
    await next();
});

IIS URL Rewrite alternative

If middleware deployment is constrained, IIS URL Rewrite can match on User-Agent and proxy to the markdown endpoint. Reference rule:

XML

<rewrite>
  <rules>
    <rule name="Maya Markdown for LLM bots" stopProcessing="true">
      <match url="^(.*)$" />
      <conditions logicalGrouping="MatchAll">
        <add input="{HTTP_USER_AGENT}" pattern="GPTBot|ClaudeBot|PerplexityBot|OAI-SearchBot|Google-Extended|Bingbot|Applebot-Extended|anthropic-ai|ChatGPT-User|Perplexity-User" />
        <add input="{REQUEST_METHOD}" pattern="^GET$" />
        <add input="{APPL_PHYSICAL_PATH}\featureflags\maya.markdown_rendering" matchType="IsFile" />
      </conditions>
      <action type="Rewrite" url="/api/render/markdown?path={R:1}" appendQueryString="true" />
    </rule>
  </rules>
</rewrite>

The third condition (featureflags\maya.markdown_rendering) is a file-presence check — flipping the file in or out enables/disables the feature without an IIS restart.

Component B — Markdown Renderer

The renderer is a small ASP.NET Core endpoint (or a function on the BFF layer once migrated) that:

Resolves the requested path to its underlying SharePoint resource.
Fetches the rendered HTML internally (server-to-server, no public round-trip).
Converts to clean markdown.
Returns with Content-Type: text/markdown; charset=utf-8 and a short Cache-Control.

Reference handler

[ApiController]
[Route("api/render/markdown")]
public class MarkdownRenderer : ControllerBase
{
    private readonly HttpClient _internalClient;
    private readonly IHtmlToMarkdownConverter _converter;
 
    public MarkdownRenderer(HttpClient internalClient, IHtmlToMarkdownConverter converter)
    {
        _internalClient = internalClient;
        _converter = converter;
    }
 
    [HttpGet]
    public async Task<IActionResult> Render([FromQuery] string path)
    {
        if (string.IsNullOrEmpty(path) || !path.StartsWith("/"))
            return BadRequest();
 
        var html = await _internalClient.GetStringAsync(
            new Uri(new Uri("https://internal-sharepoint/"), path));
 
        var markdown = _converter.Convert(html, options: new()
        {
            StripScripts = true,
            StripStyles = true,
            RetainStructure = true,   // headings, lists, tables, links
            RetainOpenGraph = true,    // emit OG tags as YAML front matter
        });
 
        Response.Headers.CacheControl = "public, max-age=300, s-maxage=900";
        Response.Headers.ContentType = "text/markdown; charset=utf-8";
        return Content(markdown, "text/markdown");
    }
}

Conversion rules

Element	Treatment
`<h1>`–`<h6>`	Preserved as `#`–`######`
`<p>`	Preserved as paragraph
`<a>`	Preserved as `[text](url)`
`<img>`	Preserved as `![alt](url)` (with caption from `figcaption` if present)
`<table>`	Preserved as GitHub-flavored markdown table
`<script>`, `<style>`, `<svg>`	Stripped
`<nav>`, `<footer>`, side widgets	Configurable: keep main `<article>` only
Open Graph / Twitter meta	Emitted as YAML front matter at the top
Inline tracking pixels	Stripped

What the bot sees

For a page like /products/widgets/super-widget, the markdown response begins:

Markdown

---
title: Super Widget — Acme Corp
description: The Super Widget — features, pricing, and how to buy.
canonical: https://www.example.com/products/widgets/super-widget
locale: en_US
---
 
# Super Widget
 
The Super Widget is …
 
## Pricing
 
| Tier | Monthly | Annual |
| ---- | ------- | ------ |
| Starter | $19 | $190 |
| Pro | $49 | $490 |
| Enterprise | Contact us | Contact us |
 
## How to buy
 
1. …

This is what gets vectorized into the LLM's RAG layer.

Operational considerations

Feature flag and rollout

The feature is gated by a flag (maya.markdown_rendering). Recommended rollout:

Internal test — flag on for a single canary path (e.g., /help/markdown-test). Fetch with User-Agent: GPTBot and confirm markdown response.
Single page enable — flag on for one high-value page (e.g., a flagship product page).
Section enable — flag on for an entire site section.
Site-wide — flag on for all GET endpoints.

If anything looks wrong, flag off. Public traffic is never affected because the router only fires for verified bots.

Caching

Markdown responses are cheap to cache. A 5-minute origin cache plus a 15-minute shared cache is a reasonable starting point. Cache by (path, accept-language).

Observability

Emit a per-request metric:

maya_markdown.requests_total{bot, status, cached}
maya_markdown.latency_ms_p95{bot}
maya_markdown.bytes_out_total{bot}

These metrics make ROI obvious within a week of going live.

Security

The markdown endpoint must only render public SharePoint content. The internal HTTP client must not carry authentication for a privileged user.
The endpoint MUST refuse paths that resolve to authenticated areas (/login, /portal, /account, /admin, etc.). Treat these as a denylist.
The endpoint MUST NOT accept paths from external sources other than the router. A simple way: the router sets an internal header (X-Maya-Internal: 1) and the endpoint refuses requests without it.

Decision points for your Product Owner

Question	Recommendation
Implement the router as middleware or URL Rewrite?	Middleware (ASP.NET Core) if available; URL Rewrite as a fallback.
Implement the renderer at edge (IIS) or in BFF?	BFF, once available. Until then, IIS host.
Cache TTL?	5 min origin / 15 min shared, conservatively.
Which sections first?	High-intent product pages and high-traffic content before institutional or "about" pages.
Path denylist scope?	All authenticated areas, all admin/staging hosts, all internal-only assets.

Effort estimate

A typical implementation lands in 15–25 person-hours of engineering, plus QA and feature-flag tooling. Maya provides a paired-debugging session at no cost during rollout.

Backout plan

The feature is fully reversible: flip the flag off. No database changes. No SharePoint configuration changes. No public traffic is affected at any time.

PreviousOverview Next Ingestion API