Skip to content

DEVELOPMENT.md

Latest commit

 

History

History
498 lines (356 loc) · 12.5 KB

DEVELOPMENT.md

File metadata and controls

498 lines (356 loc) · 12.5 KB
Raw

Local OCaml documentation & search workflows

This document explains how to set up a development environment in which all installed libraries in the current opam switch and this project’s own documentation are searchable via:

  • a vector + BM25 index of HTML docs generated by odoc (odoc-index / odoc-search);
  • a semantic index of the project’s Markdown docs (md-index / md-search);
  • a type-signature search database built with sherlodoc.

The goal is to give both humans and LLM agents a rich, local knowledge base of the OCaml ecosystem and this repository.

All commands below assume:

  • you are in the project root of this repository; and
  • your shell is already configured to use the correct opam switch (e.g. you started it with opam switch <name> or opam exec -- bash once, but we omit opam exec -- from each command).

0. Prerequisites

You should have, in the active switch:

  • OCaml ≥ 5.1 (see dune-project),

  • dune ≥ 3.18,

  • odoc, odig, and sherlodoc installed:

    opam install dune odoc odig sherlodoc

Environment:

  • OPENAI_API_KEY set – required by:
    • odoc-index and odoc-search,
    • md-index and md-search,
    • ochat index / ochat query.
  • OPAM_SWITCH_PREFIX set – this is done automatically by opam for the active switch and is used to locate the odig caches.

Throughout the rest of this document we assume these are already set.

1. Install the current dune project into the switch

Before indexing, install the project so that:

  • its libraries and docs appear as installed packages in the switch;
  • odig and odoc can see the project’s documentation;
  • the CLI tools in bin/ are built and runnable via dune exec or from your $PATH once installed.

From the project root:

dune build @install
dune install

After this:

  • The umbrella CLI ochat and helper binaries (odoc-index, md-index, odoc-search, md-search, mcp_server, etc.) are installed in the switch.
  • odig will treat the project’s libraries (e.g. ochat) like any other opam package.

2. Generate ODoc HTML for all installed packages (via odig)

odoc-index and sherlodoc both work on top of the HTML/odoc artefacts cached by odig.

2.1 Optional: clear the odig cache

If you want a hard refresh (e.g. after pinning many packages or changing switches):

odig cache clear

This wipes odig’s cache under:

$OPAM_SWITCH_PREFIX/var/cache/odig/

2.2 Regenerate docs for all installed packages

odig odoc

This:

  • compiles .odoc / .odocl documentation files; and

  • produces HTML docs under:

    • "$OPAM_SWITCH_PREFIX/var/cache/odig/html/" – HTML hierarchy
    • "$OPAM_SWITCH_PREFIX/var/cache/odig/odoc/" – .odocl files

These locations are used in later steps.

3. Index type signatures with sherlodoc

sherlodoc provides fuzzy type search over .odocl files. We’ll point it at the odig cache.

3.1 Choose a database location

Decide where the Sherlodoc database should live, e.g. inside the project:

export SHERLODOC_DB="$PWD/.cache/sherlodoc.marshal"
mkdir -p "$(dirname "$SHERLODOC_DB")"

(If you prefer the ancient format on older OCaml, follow the package docs.)

3.2 Index all odoc type information from the current switch

sherlodoc index \
  $(find "$OPAM_SWITCH_PREFIX/var/cache/odig/odoc" -name '*.odocl' | grep -v __)

This:

  • reads all .odocl files generated by odig odoc;
  • ignores internal artefacts containing __ in the path;
  • writes a Sherlodoc database to $SHERLODOC_DB.

3.3 Query by type

You can now run type-based queries such as:

sherlodoc search --print-docstring-html "('a list -> 'a list) -> 'a list -> 'a list"

Sherlodoc will:

  • fuzzy-match the query against the indexed type signatures;
  • print matching symbols and docstrings (HTML).

This database is useful to LLM agents that need to answer “what function has this type?” questions over your entire environment.

4. Build a vector + BM25 index over HTML docs (odoc-index)

odoc-index (implemented in bin/odoc_index.ml and documented in docs-src/bin/odoc_index.doc.md) converts odoc-generated HTML into:

  • dense vectors in a Vector_db corpus;
  • BM25 lexical indices;
  • Markdown snippets.

By default, it writes one subdirectory per opam package, under .odoc_index/.

4.1 Run the indexer

From the project root:

dune exec bin/odoc_index.exe -- \
  --root "$OPAM_SWITCH_PREFIX/var/cache/odig/html/"

Once the binary is installed in the switch, you can also invoke it directly:

odoc-index --root "$OPAM_SWITCH_PREFIX/var/cache/odig/html/"

Optional: change the output directory (defaults to .odoc_index):

odoc-index \
  --root "$OPAM_SWITCH_PREFIX/var/cache/odig/html/" \
  --out  .odoc_index

Resulting layout (simplified):

.odoc_index/
  ├─ core/
  │   ├─ vectors.binio
  │   ├─ bm25.binio
  │   ├─ <id>.md
  │   └─ ...
  ├─ eio/
  ├─ ochat/
  └─ package_index.binio

4.2 Package filter in bin/odoc_index.ml

bin/odoc_index.ml hard-codes a package_filter to avoid reindexing noisy or duplicated docs while always updating key packages:

Odoc_indexer.index_packages
  ~filter:
    (Odoc_indexer.Update
       ( Exclude
           [ "ocaml"
           ; "ocaml_intrinsics_kernel"
           ; "ocaml-compiler-libs"
           ; "ocamlgraph"
           ; "tls"
           ]
       , [ "ochat"; "textmate-language"; "irmin"; "irmin-git"; "irmin-watcher" ] ))
  ~env
  ~root:root_path
  ~output:out_path
  ~net:env#net
  ()

Interpretation:

  • Exclude [...] — skip the listed packages entirely (mostly stdlib / compiler internals / noisy libs).
  • Update (prev, [ pkgs ]) — treat the given pkgs as must-update even if incremental logic would otherwise skip them.

To adjust behaviour:

  • Add newly important packages
    Append them to the second list:

    , [ "ochat"
  ; "textmate-language"
  ; "irmin"
  ; "irmin-git"
  ; "irmin-watcher"
  ; "your-new-package"
  ]

  • Full refresh of all packages (except exclusions)
    Temporarily simplify the filter to a plain Exclude:

    ~filter:(Odoc_indexer.Exclude
           [ "ocaml"
           ; "ocaml_intrinsics_kernel"
           ; "ocaml-compiler-libs"
           ; "ocamlgraph"
           ; "tls"
           ])

    This asks Odoc_indexer to rebuild every non-excluded package.

See docs-src/lib/odoc_indexer.doc.md for full semantics of package_filter (All, Include [pkgs], Exclude [pkgs], Update (prev, pkgs)).

5. Build semantic indexes for Markdown docs (md-index)

This project ships two Markdown indexes (see bin/md_index.ml and docs-src/bin/md_index.doc.md), both rooted at docs-src/lib but with different logical names:

  • docs-src/bin
  • docs-src/lib

Each call to md-index:

  • crawls the --root tree for *.md / *.markdown / *.mdx files;
  • slices them into token-bounded overlapping windows;
  • embeds all unseen snippets with the OpenAI embeddings API;
  • writes a vector DB to .md_index/<name>/.

5.1 Index the project docs

dune exec bin/md_index.exe -- \
  --root docs-src/lib \
  --name "docs-src/bin"
dune exec bin/md_index.exe -- \
  --root docs-src/lib \
  --name "docs-src/lib"

Notes:

  • --root is the directory tree to crawl. In the current setup both logical indexes happen to be built from docs-src/lib; you can point them at separate subtrees if you prefer (e.g. docs-src/bin vs docs-src/lib).
  • --name is just a label; it determines:
    • the subdirectory under .md_index/,
    • how the index is referred to by md-search --index <name>.
  • --out defaults to .md_index. You can override it if you want the vector DB elsewhere.

The on-disk layout for an index called docs-src/lib looks like:

.md_index/
  └─ docs-src/lib/
       ├─ vectors.binio
       ├─ snippets/
       │    ├─ <id>.md
       │    └─ ...
       ├─ meta.json
       └─ ...
md_index_catalog.binio

The global md_index_catalog.binio (in .md_index/) stores centroids and metadata so tools can quickly pick relevant indexes.

6. Optional: smoke-test the indexes

Once all indexes are built you can run quick checks.

6.1 Search odoc snippets

dune exec bin/odoc_search.exe -- \
  --query "query newly indexed opam doc snippets" \
  -k 20

Or, using the installed binary:

odoc-search --query "query newly indexed opam doc snippets" -k 20

Expected behaviour:

  • The tool embeds the query text using OpenAI.

  • It selects a subset of packages (via package_index.binio) if you did not pass --package.

  • It prints ranked Markdown snippets with package and id:

    [1] [eio] 0000abcd-...:
<snippet markdown>

---

See docs-src/bin/odoc_search.doc.md for full CLI details.

6.2 Search Markdown docs

dune exec bin/md_search.exe -- \
  --query "textmate newly indexed markedow docs" \
  -k 10 \
  --index docs-src/lib

Or, after installation:

md-search --query "textmate newly indexed markedow docs" \
          --index docs-src/lib \
          -k 10

This:

  • embeds the query;
  • either searches a specific index (--index docs-src/lib), or if you pass --index all, picks the five closest ones by centroid;
  • prints the best‑matching snippet bodies.

See docs-src/bin/md_search.doc.md for details and limitations.

6.3 Re-check Sherlodoc type search

sherlodoc search --print-docstring-html "sherlodoc type search query"

If everything is wired correctly you should see matching symbols with their docstrings rendered as HTML.

7. How this helps LLM agents

Once the above workflows have been run inside the same workspace as your agent, the following local resources become available:

  • Type-level search (sherlodoc on $SHERLODOC_DB) for any installed package.

  • API‑level semantic search over docs (odoc-search over .odoc_index/).

  • Project‑doc semantic search over Markdown (md-search over .md_index/).

  • Source‑level indexing & search via the umbrella ochat CLI:

    # index OCaml source files under ./lib into ./vector
ochat index -folder-to-index ./lib -vector-db-folder ./vector

# search those snippets
ochat query -vector-db-folder ./vector \
            -query-text "tail-recursive map over a list" \
            -num-results 5

For agent integrations:

  • If your LLM runtime can call shell commands, you can expose:

    • odoc-search, md-search, and sherlodoc search as tools;
    • optionally ochat index / ochat query for source‑level search.

  • This repository also ships an MCP stdio server (bin/mcp_server.ml) that exposes generic tools over JSON‑RPC:

    • echo
    • apply_patch (V4A diff)
    • read_dir
    • get_contents
    • meta_refine
    • webpage_to_markdown
    • plus dynamically loaded .chatmd prompts under ./prompts or $MCP_PROMPTS_DIR.

    You can run it as:

    dune exec bin/mcp_server.exe
# or over HTTP:
dune exec bin/mcp_server.exe -- --http 8080

    While the MCP server does not (yet) expose doc-search as first‑class tools, the combination of:

    • local vector indexes (.odoc_index, .md_index, $SHERLODOC_DB),
    • filesystem access (read_dir, get_contents),
    • and apply_patch

    provides a solid foundation for LLM agents that reason about and edit this OCaml codebase.

8. Summary

To prepare a fresh environment where LLM agents (and humans) can query local OCaml libraries and this repository:

  1. Install the project into the active opam switch:

    dune build @install
dune install

  2. Regenerate docs with odig odoc.

  3. Build type-signature search with sherlodoc index $(find "$OPAM_SWITCH_PREFIX/var/cache/odig/odoc" -name '*.odocl' | grep -v __).

  4. Build HTML-doc search with odoc-index --root "$OPAM_SWITCH_PREFIX/var/cache/odig/html/".

  5. Build Markdown-doc search with md-index --root docs-src/lib --name ….

  6. Smoke-test with odoc-search, md-search, and sherlodoc search.

Once these steps are part of your regular workflow (e.g. after opam upgrade or significant doc changes), contributors and tools will always have up‑to‑date local documentation they can search semantically.