Serving agents markdown: content negotiation without a second codebase

Your page is already a resource with several possible representations. You have probably shipped two of them by accident: the HTML document and the Open Graph card. Content negotiation is the standard mechanism for letting a client choose between them, and it is the cheapest way to hand an autonomous agent a version of your page it can actually read.

The interesting part is not the header. It is doing this without a second content pipeline that drifts from the first one within a quarter.

Why markdown is a useful second representation

An agent fetching your page wants three things that HTML makes expensive.

Structure that survives parsing. Headings, list items, table rows and code fences are the skeleton of a page. In HTML that skeleton is a tree you have to walk through wrapper elements; in markdown it is the content. An agent that reads ## Pricing and - Free tier: 100 requests/day has the page's outline and one of its facts in the same breath.

Fewer tokens for the same facts. This is token efficiency, and it is a ratio, not a guarantee. <div class="mt-4 flex items-center">, inline SVG, tracking attributes and repeated navigation markup all cost tokens while carrying no meaning. Strip them and the same prose survives with a much smaller payload. We are deliberately not quoting a number here: the saving depends entirely on your markup, and any figure quoted without measuring your own pages is a marketing claim, not a measurement.

No navigation chrome. A markdown twin of a page should contain the page, not the site. Site header, cookie banner, footer link farm and "related articles" are meaningful to a person browsing and are pure noise to an agent that arrived at one URL with one question. The twin is where you get to delete them.

Markdown is registered as text/markdown in RFC 7763, with charset as a required parameter — so text/markdown; charset=utf-8 is the honest type. Not text/plain, which tells the client nothing about the syntax it is about to parse.

How proactive negotiation actually works

HTTP defines two mechanisms. Reactive negotiation waits for the server to answer 300 Multiple Choices or 406 Not Acceptable and lets the client pick; almost nobody implements it. Proactive negotiation, described in RFC 9110 §12, is the one you already rely on: the client states its preferences in request headers, and the server picks a representation from what it has.

The relevant header is Accept, defined in RFC 9110 §12.5.1. It is a comma-separated list of media types, each optionally carrying a quality value:

GET /pricing HTTP/1.1
Host: example.com
Accept: text/markdown, text/html;q=0.9

Quality values are the priority weights described in §12.4.2 and on MDN: ;q= followed by a number between 0 and 1 inclusive, up to three decimal places, absent meaning 1. Ties go to the more specific type. Two details matter when you implement this:

• */* is not an invitation. A browser's default Accept usually ends in */*;q=0.8. If you treat a wildcard as a markdown request, you will serve raw markdown to a browser. Markdown is returned only when text/markdown is named explicitly.
• A client that does not send Accept at all is not asking for markdown. Most command-line clients and a good number of HTTP libraries send no Accept header on a plain GET. The correct default is the HTML document, and the .md URL twin is the escape hatch for a client that cannot negotiate.

The response for the request above looks like this:

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept

# Pricing
...

Note what did not change: the URL. One resource, two representations, chosen per request.

Vary is not optional

Vary is the header that tells caches which request headers influenced the response. Vary: Accept says: this response is one of several, and the choice depended on Accept, so your cache key must include it.

Without Vary, a shared cache — a CDN, a reverse proxy, a corporate middlebox — keys on the URL and method alone. The first requester to warm the entry decides what everybody else gets until it expires. The failure is worse in the markdown direction: an agent warms /pricing, the cache stores the markdown body, and the next visitor gets a raw text file rendered in the browser. Both directions are live bugs, and neither requires an exotic setup.

Two rules follow from RFC 9110 §12.5.5 and MDN's summary of it:

• The markdown response and the HTML response for the same URL must carry the same Vary value, including 304 Not Modified replies. If only one representation declares it, caches have an incentive to get it wrong.
• Vary: User-Agent is not an acceptable substitute. There is no finite set of user agent strings, so every distinct value becomes a separate cache entry and your hit rate collapses. Accept has a small, enumerable set of meaningful values.

One source, two renderers

The reason teams end up with a second codebase is that they write markdown as a separate artifact: someone exports the HTML, pastes it through a converter, fixes the headings by hand, and then the page changes. The fix is to stop treating markdown as content and start treating it as a projection.

Model a page as structured data — a document with blocks — and write two renderers over that array: one to HTML, one to markdown. The same idea generalises to llms.txt, sitemaps and skill packs, but the markdown twin is where it pays for itself first, because the block vocabulary an agent needs is tiny: paragraph, heading, code, list, table, quote.

// content.js — the single source for one page
export const pricing = {
  path: '/pricing',
  title: 'Pricing',
  blocks: [
    { kind: 'lead', text: 'One plan, metered by request.' },
    { kind: 'h2', text: 'Free tier' },
    { kind: 'ul', items: ['100 requests per day', 'No card required'] },
    { kind: 'code', lang: 'bash', code: 'curl -sS $DOMAIN/api/ping' },
  ],
};

// render.js — markdown is a projection, not a file anyone edits
export function toMarkdown(doc) {
  const out = [`# ${doc.title}`];
  for (const b of doc.blocks) {
    if (b.kind === 'lead' || b.kind === 'p') out.push(b.text);
    else if (b.kind === 'h2') out.push(`## ${b.text}`);
    else if (b.kind === 'ul') out.push(b.items.map((i) => `- ${i}`).join('\n'));
    else if (b.kind === 'code') out.push('```' + (b.lang ?? '') + '\n' + b.code + '\n```');
  }
  return out.join('\n\n') + '\n';
}

The server side is then a preference check, not a second routing table:

// server.js — Node/Express; the same shape works in any framework.
import express from 'express';
import { pages } from './content.js';
import { toMarkdown, toHtml as render } from './render.js';

const app = express();

app.get('*', (req, res, next) => {
  const path = req.path.replace(/\/$/, '') || '/';
  const doc = pages.get(path);

  // A .md URL is a stable alternate for clients that cannot negotiate.
  if (path.endsWith('.md')) {
    const md = pages.get(path.slice(0, -3));
    if (!md) return next();
    return res.type('text/markdown; charset=utf-8').set('Vary', 'Accept').send(toMarkdown(md));
  }

  if (!doc) return next();

  // Only an explicit text/markdown — never */* — selects the twin.
  const wantsMarkdown = req.accepts(['text/html', 'text/markdown']) === 'text/markdown';
  if (wantsMarkdown) {
    return res.type('text/markdown; charset=utf-8').set('Vary', 'Accept').send(toMarkdown(doc));
  }

  // The HTML representation declares the same Vary value.
  res.set('Vary', 'Accept').type('text/html; charset=utf-8').send(render(doc));
});

app.use((req, res) => res.status(404).type('text/plain; charset=utf-8').send('Not found'));

Three properties make this wearable. req.accepts() reads the quality values you would otherwise hand-parse, and returns the HTML type when the client sends no Accept or a wildcard. The markdown body is generated from the same object the HTML came from, so a copy edit cannot land in one representation only. And Vary is set on the HTML response too, which is the half of the rule people forget.

If you prefer not to negotiate on the fly, render the twins at build time and serve them as files. Same headers, no per-request work. What you must not do is convert HTML to markdown at request time by stripping tags: it is slower, and it loses exactly the structure you were trying to preserve.

Failure modes worth naming

The last row is the reason negotiation and server rendering are the same conversation: you cannot project content you only ever assemble in the browser.

The honest limits

No mainstream agent is guaranteed to send Accept: text/markdown today. The header is a registered media type and ordinary HTTP, but adoption is not something we can measure across the agents you care about, and we will not pretend otherwise. Treat this as a bet on a direction — that agents will increasingly request a low-markup representation when one is offered — and not as a fix for a measured current loss. If your markdown endpoint never receives a single Accept: text/markdown request this quarter, that is expected, not broken.

That framing has consequences worth accepting up front:

• Do not degrade the HTML path to make the bet. The HTML document is what browsers and most current crawlers get. Negotiation is additive.
• Keep the .md twin. It costs one route and it is the only thing here that works without client cooperation.
• Count the requests yourself. Log the Accept header on HTML routes and count how often text/markdown appears. That number is the only honest evidence for or against the bet, and it is yours, not ours.
• Markdown is lossy in the direction you are not looking. Anchors, absolute URLs baked into inline links, figures with captions and interactive components do not survive a naive projection. Decide per block type what the markdown form is; the methodology page describes how this check is scored and what a partial pass means.

The implementation is small: one branch on Accept, one Vary header on every representation, and one content model with two renderers. The failure mode is large: a second codebase that nobody notices has drifted until an agent reads the wrong price.