# Serving agents markdown: content negotiation without a second codebase

> Serve markdown on a page's own URL with Accept and Vary, so agents get token-efficient text and readers still get HTML from one content source.

- Source: /learn/markdown-content-negotiation
- Updated: 2026-10-08
- Reading time: 9 minutes
- Tags: markdown, content negotiation, HTTP

---
Your page is already a resource with several possible representations. You have probably shipped two of them by accident: the HTML document and the Open Graph card. Content negotiation is the standard mechanism for letting a client choose between them, and it is the cheapest way to hand an autonomous agent a version of your page it can actually read.

The interesting part is not the header. It is doing this without a second content pipeline that drifts from the first one within a quarter.

## Why markdown is a useful second representation

An agent fetching your page wants three things that HTML makes expensive.

**Structure that survives parsing.** Headings, list items, table rows and code fences are the skeleton of a page. In HTML that skeleton is a tree you have to walk through wrapper elements; in markdown it is the content. An agent that reads `## Pricing` and `- Free tier: 100 requests/day` has the page's outline and one of its facts in the same breath.

**Fewer tokens for the same facts.** This is [token efficiency](/checks/token-efficiency), and it is a ratio, not a guarantee. `<div class="mt-4 flex items-center">`, inline SVG, tracking attributes and repeated navigation markup all cost tokens while carrying no meaning. Strip them and the same prose survives with a much smaller payload. We are deliberately not quoting a number here: the saving depends entirely on your markup, and any figure quoted without measuring your own pages is a marketing claim, not a measurement.

**No navigation chrome.** A markdown twin of a page should contain the page, not the site. Site header, cookie banner, footer link farm and "related articles" are meaningful to a person browsing and are pure noise to an agent that arrived at one URL with one question. The twin is where you get to delete them.

Markdown is registered as `text/markdown` in [RFC 7763](https://www.rfc-editor.org/rfc/rfc7763.txt), with `charset` as a required parameter — so `text/markdown; charset=utf-8` is the honest type. Not `text/plain`, which tells the client nothing about the syntax it is about to parse.

## How proactive negotiation actually works

HTTP defines two mechanisms. Reactive negotiation waits for the server to answer `300 Multiple Choices` or `406 Not Acceptable` and lets the client pick; almost nobody implements it. Proactive negotiation, described in [RFC 9110 §12](https://www.rfc-editor.org/rfc/rfc9110.html), is the one you already rely on: the client states its preferences in request headers, and the server picks a representation from what it has.

The relevant header is `Accept`, defined in [RFC 9110 §12.5.1](https://www.rfc-editor.org/rfc/rfc9110.html). It is a comma-separated list of media types, each optionally carrying a quality value:

```http
GET /pricing HTTP/1.1
Host: example.com
Accept: text/markdown, text/html;q=0.9
```

Quality values are the priority weights described in [§12.4.2](https://www.rfc-editor.org/rfc/rfc9110.html) and on [MDN](https://developer.mozilla.org/en-US/docs/Glossary/Quality_values): `;q=` followed by a number between 0 and 1 inclusive, up to three decimal places, absent meaning `1`. Ties go to the more specific type. Two details matter when you implement this:

- **`*/*` is not an invitation.** A browser's default `Accept` usually ends in `*/*;q=0.8`. If you treat a wildcard as a markdown request, you will serve raw markdown to a browser. Markdown is returned only when `text/markdown` is named explicitly.
- **A client that does not send `Accept` at all is not asking for markdown.** Most command-line clients and a good number of HTTP libraries send no `Accept` header on a plain GET. The correct default is the HTML document, and the `.md` URL twin is the escape hatch for a client that cannot negotiate.

The response for the request above looks like this:

```http
HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept

# Pricing
...
```

Note what did *not* change: the URL. One resource, two representations, chosen per request.

## Vary is not optional

`Vary` is the header that tells caches which request headers influenced the response. `Vary: Accept` says: this response is one of several, and the choice depended on `Accept`, so your cache key must include it.

Without `Vary`, a shared cache — a CDN, a reverse proxy, a corporate middlebox — keys on the URL and method alone. The first requester to warm the entry decides what everybody else gets until it expires. The failure is worse in the markdown direction: an agent warms `/pricing`, the cache stores the markdown body, and the next visitor gets a raw text file rendered in the browser. Both directions are live bugs, and neither requires an exotic setup.

Two rules follow from [RFC 9110 §12.5.5](https://www.rfc-editor.org/rfc/rfc9110.html) and [MDN's summary of it](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Vary):

- The markdown response and the HTML response for the same URL must carry the **same** `Vary` value, including `304 Not Modified` replies. If only one representation declares it, caches have an incentive to get it wrong.
- `Vary: User-Agent` is not an acceptable substitute. There is no finite set of user agent strings, so every distinct value becomes a separate cache entry and your hit rate collapses. `Accept` has a small, enumerable set of meaningful values.

## One source, two renderers

The reason teams end up with a second codebase is that they write markdown as a separate artifact: someone exports the HTML, pastes it through a converter, fixes the headings by hand, and then the page changes. The fix is to stop treating markdown as content and start treating it as a projection.

Model a page as structured data — a document with blocks — and write two renderers over that array: one to HTML, one to markdown. The same idea generalises to `llms.txt`, sitemaps and skill packs, but the markdown twin is where it pays for itself first, because the block vocabulary an agent needs is tiny: paragraph, heading, code, list, table, quote.

```js
// content.js — the single source for one page
export const pricing = {
  path: '/pricing',
  title: 'Pricing',
  blocks: [
    { kind: 'lead', text: 'One plan, metered by request.' },
    { kind: 'h2', text: 'Free tier' },
    { kind: 'ul', items: ['100 requests per day', 'No card required'] },
    { kind: 'code', lang: 'bash', code: 'curl -sS $DOMAIN/api/ping' },
  ],
};
```

```js
// render.js — markdown is a projection, not a file anyone edits
export function toMarkdown(doc) {
  const out = [`# ${doc.title}`];
  for (const b of doc.blocks) {
    if (b.kind === 'lead' || b.kind === 'p') out.push(b.text);
    else if (b.kind === 'h2') out.push(`## ${b.text}`);
    else if (b.kind === 'ul') out.push(b.items.map((i) => `- ${i}`).join('\n'));
    else if (b.kind === 'code') out.push('```' + (b.lang ?? '') + '\n' + b.code + '\n```');
  }
  return out.join('\n\n') + '\n';
}
```

The server side is then a preference check, not a second routing table:

```js
// server.js — Node/Express; the same shape works in any framework.
import express from 'express';
import { pages } from './content.js';
import { toMarkdown, toHtml as render } from './render.js';

const app = express();

app.get('*', (req, res, next) => {
  const path = req.path.replace(/\/$/, '') || '/';
  const doc = pages.get(path);

  // A .md URL is a stable alternate for clients that cannot negotiate.
  if (path.endsWith('.md')) {
    const md = pages.get(path.slice(0, -3));
    if (!md) return next();
    return res.type('text/markdown; charset=utf-8').set('Vary', 'Accept').send(toMarkdown(md));
  }

  if (!doc) return next();

  // Only an explicit text/markdown — never */* — selects the twin.
  const wantsMarkdown = req.accepts(['text/html', 'text/markdown']) === 'text/markdown';
  if (wantsMarkdown) {
    return res.type('text/markdown; charset=utf-8').set('Vary', 'Accept').send(toMarkdown(doc));
  }

  // The HTML representation declares the same Vary value.
  res.set('Vary', 'Accept').type('text/html; charset=utf-8').send(render(doc));
});

app.use((req, res) => res.status(404).type('text/plain; charset=utf-8').send('Not found'));
```

Three properties make this wearable. `req.accepts()` reads the quality values you would otherwise hand-parse, and returns the HTML type when the client sends no `Accept` or a wildcard. The markdown body is generated from the same object the HTML came from, so a copy edit cannot land in one representation only. And `Vary` is set on the HTML response too, which is the half of the rule people forget.

If you prefer not to negotiate on the fly, render the twins at build time and serve them as files. Same headers, no per-request work. What you must not do is convert HTML to markdown at request time by stripping tags: it is slower, and it loses exactly the structure you were trying to preserve.

## Failure modes worth naming

| Symptom | What it means | Fix |
|---|---|---|
| `Accept: text/markdown` returns `200` with `Content-Type: text/html` and a `<!doctype html>` body | The handler never looked at `Accept`; a proxy or CDN may also be caching the HTML with no `Vary` | Branch on `Accept` before rendering, and set `Vary: Accept` on every representation. See [markdown-negotiation](/checks/markdown-negotiation). |
| Markdown returned, but `Vary` missing | The cache key ignores `Accept`; whichever representation arrives first is served to everyone | Add `Vary: Accept` to both responses, including `304`s. |
| Markdown body with no `#` or `##` lines | The twin was produced by stripping tags, so heading text was flattened into paragraphs | Render headings from your content model; if headings exist in one representation they exist in both. |
| `text/plain` (or no `charset`) for a markdown body | Clients cannot tell the syntax, and non-ASCII text may decode wrongly | `Content-Type: text/markdown; charset=utf-8`, per RFC 7763. |
| Browser shows raw markdown | A wildcard or missing `Accept` was treated as a markdown request | Return markdown only when the client names `text/markdown`. |
| The `.md` twin and the page disagree | The twin is hand-written and drifted | Delete the hand-written file; generate it from the same source. |
| The twin is an empty shell on a client-rendered route | The page only assembles its content after hydration, so the markdown projection has nothing to project | Render content server-side first. See [spa-fallback](/checks/spa-fallback). |

The last row is the reason negotiation and server rendering are the same conversation: you cannot project content you only ever assemble in the browser.

## The honest limits

No mainstream agent is guaranteed to send `Accept: text/markdown` today. The header is a registered media type and ordinary HTTP, but adoption is not something we can measure across the agents you care about, and we will not pretend otherwise. Treat this as a bet on a direction — that agents will increasingly request a low-markup representation when one is offered — and not as a fix for a measured current loss. If your markdown endpoint never receives a single `Accept: text/markdown` request this quarter, that is expected, not broken.

That framing has consequences worth accepting up front:

- **Do not degrade the HTML path to make the bet.** The HTML document is what browsers and most current crawlers get. Negotiation is additive.
- **Keep the `.md` twin.** It costs one route and it is the only thing here that works without client cooperation.
- **Count the requests yourself.** Log the `Accept` header on HTML routes and count how often `text/markdown` appears. That number is the only honest evidence for or against the bet, and it is yours, not ours.
- **Markdown is lossy in the direction you are not looking.** Anchors, absolute URLs baked into inline links, figures with captions and interactive components do not survive a naive projection. Decide per block type what the markdown form is; the [methodology](/methodology) page describes how this check is scored and what a partial pass means.

The implementation is small: one branch on `Accept`, one `Vary` header on every representation, and one content model with two renderers. The failure mode is large: a second codebase that nobody notices has drifted until an agent reads the wrong price.