Astro build runs out of memory during content sync for large Starlight docs site with heavy KaTeX

[integrable]

Astro Info

Astro                    v7.0.6
Vite                     v8.1.3
Node                     v24.18.0
System                   macOS (arm64)
Package Manager          npm
Output                   static
Adapter                  none
Integrations             @astrojs/starlight (v0.41.3)

If this issue only occurs in one browser, which browser is a problem?

No response

Describe the Bug

I have a large Starlight documentation site with many Markdown/MDX docs containing heavy KaTeX/math content.

The build runs out of memory during Astro’s content sync phase, before page generation starts.

The minimal reproduction repo is https://github.com/integrable/astrov7/

The failure happens after [content] Syncing content and then Node/V8 eventually aborts with:

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

This appears to be a scalability issue in Astro’s content layer for renderable Markdown/MDX entries. My current understanding is that Astro’s glob() content loader eagerly renders Markdown/MDX entries and stores the rendered output in the global content data store. For Starlight docs pages with heavy KaTeX, the rendered HTML is much larger than the source Markdown, so syncing content can consume several GB of heap and eventually crash.

The KaTeX warnings printed before the crash do not seem to be the direct cause; they are warnings, and the process continues until memory is exhausted.

Steps to Reproduce:

git clone https://github.com/integrable/astrov7.git
cd astrov7
npm ci
rm -rf .astro dist node_modules/.astro
npm run build

Actual Result:

The build starts syncing content:

15:52:45 [content] Syncing content

Then several KaTeX warnings are printed, for example:

LaTeX-incompatible input and strict mode is set to 'warn': In LaTeX, \ or \newline does nothing in display mode [newLineInDisplayMode]

After several minutes, Node runs out of memory:

<--- Last few GCs --->

[93872:0x8b540c000]   178849 ms: Scavenge (interleaved) 4093.3 (4108.7) -> 4093.1 (4109.7) MB, pooled: 0 MB, 3.96 / 0.00 ms  (average mu = 0.288, current mu = 0.234) allocation failure; 
[93872:0x8b540c000]   179724 ms: Mark-Compact (reduce) 4094.0 (4109.7) -> 4093.9 (4108.2) MB, pooled: 0 MB, 630.04 / 0.00 ms  (+ 9.1 ms in 14 steps since start of marking, biggest step 5.0 ms, walltime since start of marking 647 ms) (average mu = 0.286, c
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

zsh: abort npm run build

Related issue:

This is related to, but different from, the previous large content-store dev issue:

#17220

That issue was about dev mode returning empty collections when importing a large content data-store virtual module. This report is about build/content sync exhausting memory before page generation because the content store becomes too large when rendered Markdown/KaTeX output is eagerly retained.

What's the expected result?

Astro should be able to build a large static Starlight docs site with heavy Markdown/KaTeX content without exhausting the Node heap during content sync.

More specifically, Astro/Starlight should not need to eagerly render and retain the rendered HTML for every docs page in the global content data store before page generation.

A scalable behavior would be one of:

1. Astro exposes an official deferred-render option for renderable Markdown/MDX content entries, for example glob({ deferRender: true }) or similar.
2. Starlight uses such a deferred-render mode for the docs collection.
3. Astro avoids storing full rendered HTML for every renderable content entry when it is not needed for collection querying.

Workable local patches are as follows:

astro+7.0.6.patch
@astrojs+starlight+0.41.3.patch

Link to Minimal Reproducible Example

https://github.com/integrable/astrov7

Participation

• I am willing to submit a pull request for this issue.

[astrobot-houston]

• Reproduced: Yes — created 200 KaTeX-heavy .md files with the local Astro build; OOM at 256 MB heap during content sync, succeeds at 1024 MB
• Exploration: Yes — View branch
• Unit Test: No — this is a memory/scalability issue; verified via build tests with constrained heap and existing test suite (54 tests passing)
• Priority: P4: important. This causes build failures (OOM crashes) for users with large Markdown collections using heavy rehype plugins like KaTeX. It blocks production builds entirely for affected sites, with no built-in workaround other than increasing Node's heap limit (which has its own ceiling).

The root cause is in packages/astro/src/content/loaders/glob.ts. When the glob() loader syncs .md content entries, it eagerly renders every file via entryType.getRenderFunction and stores the full rendered HTML in the in-memory data store. For KaTeX-heavy content, rehype-katex expands the HTML by ~70x (measured: 820 KB source → 33 MB data store for 200 files). With 1500+ files, this exhausts the default 4 GB Node heap. Notably, .mdx files already avoid this by using deferred rendering (deferredRender: true), and the code even had a // todo: add an explicit way to opt in to deferred rendering comment acknowledging this gap.

I found a potential fix for this issue. The fix adds a deferRender option to the glob() loader. When deferRender: true, .md entries skip eager rendering during content sync and instead use the same deferred rendering path that .mdx already uses — the markdown Vite plugin (astro:markdown) renders each file on demand when the page is actually built. Usage: glob({ pattern: '**/*.md', base: 'src/content/docs', deferRender: true }).

Try this fix

You can test this fix right now without waiting for a release:

npm i https://pkg.pr.new/astro@d57f86b

Then update your content config to opt in:

const docs = defineCollection({
  loader: glob({ pattern: '**/*.md', base: 'src/content/docs', deferRender: true }),
  // ...
});

If this fixes your issue, please leave a comment letting us know (e.g. "confirmed, this fixes it"). We'll then open a pull request to get this merged.

Full Triage Report

Reproduction

• Created a standalone project with 200 KaTeX-heavy Markdown files using the local Astro build
• Each file contains ~13 display-math blocks with QFT formulas processed by remark-math + rehype-katex
• node --max-old-space-size=256 astro build → OOM crash during [content] Syncing content
• node --max-old-space-size=1024 astro build → succeeds (confirming memory is the constraint)

Diagnosis

File: packages/astro/src/content/loaders/glob.ts, function syncData()

The glob loader has three code paths for content entries:

1. Eager render (entryType.getRenderFunction exists): Calls render function, stores full HTML in store.set({ rendered }). Used by .md files.
2. Deferred render ('contentModuleTypes' in entryType): Sets deferredRender: true, renders later via Vite module import. Used by .mdx files.
3. No render (neither): Just stores data.

For .md files, path 1 is always taken. The markdown content entry type (packages/astro/src/vite-plugin-markdown/content-entry-type.ts) defines getRenderFunction which processes the markdown through the full remark/rehype pipeline and returns { html, metadata }. This HTML is stored in the MutableDataStore (packages/astro/src/content/mutable-data-store.ts) for every entry simultaneously.

Memory measurements:

• Source markdown: 820 KB (200 files)
• Data store with rendered HTML: 33 MB
• Expansion ratio: ~70x
• Extrapolated for reporter's 1525 files: ~2.3 GB in-memory data store

The existing // todo: add an explicit way to opt in to deferred rendering comment (present since the Content Layer was introduced in PR #11360) confirms this was a known architectural gap.

Fix

Added deferRender?: boolean option to GlobOptions. When enabled:

• Skips the entryType.getRenderFunction eager-render path
• Falls through to the deferredRender: true path (same as .mdx)
• Rendering happens on demand via the astro:markdown Vite plugin at page build time

The fix is backward-compatible (defaults to false). Verified with:

• 200 KaTeX-heavy files building successfully at 256 MB heap (previously OOM'd)
• 200 KaTeX-heavy files rendering correctly with deferRender: true (KaTeX HTML present in output)
• All 54 content collection tests passing

This report was made by an LLM. The analysis may be wrong, and the potential fix might not work, but is intended as a starting point for exploring the issue.

[integrable]

This almost fixes it: the build succeeded, but the landing page shows a 404 Not Found because it is index.mdx; the .md pages can be viewed properly. The following updated src/content.config.ts completely fixes the issue.

import { defineCollection } from 'astro:content';
import { glob } from 'astro/loaders';
import { docsSchema } from '@astrojs/starlight/schema';

export const collections = {
	docs: defineCollection({
		loader: glob({
			base: './src/content/docs',
			pattern: '**/[^_]*.{markdown,mdown,mkdn,mkd,mdwn,md,mdx}',
			deferRender: true,
		}),
		schema: docsSchema(),
	}),
};

[astrobot-houston]

The fix has been verified! A pull request has been created: #17302