Simplify Extended_ast vs Std_ast

[hhugo]

While working on adding support for mll and mly files. I got very confused with the distinction between Extented_ast and Std_ast.

I started following what was done for Repl_file and Documentation in Std_ast, that is, "no support". But then realized that this mean that we don't check the ast is preserved. (#2557 (comment))

I figured it was more natural to keep the use of Std_ast an implementation detail of the equivalence check for some of the file format in Extended_ast only.

This is an attempt at improving the current situation.

Commits

1. Remove unused functions from Extended_ast and Normalize_extended_ast — drop equal_core_type, equal, Normalize_extended_ast.equal/ast
2. Break Source -> Extended_ast dependency cycle — open Parsetree instead of open Extended_ast
3. Break Normalize_std_ast -> Normalize_extended_ast dependency — make normalize_code a parameter instead of importing it
4. Add paired GADT to Extended_ast — ('a, 'b) paired record, Parse.ast produces both ASTs, update Fmt_ast/tests
5. Add equivalent, dump, dump_normalized, get_std — AST preservation check using embedded std ASTs
6. Absorb Parse_with_comments + simplify Translation_unit — move parsing pipeline into Extended_ast, delete Parse_with_comments, remove std_fg/std_parsed threading
7. Clean up Std_ast — remove Repl_file, Documentation, of_syntax, any_t
8. Document why Std_ast exists — add docstring explaining its role in semantic preservation

Summary

• Make Extended_ast the single entry point for parsing, validation, and AST operations
• Embed both extended and standard ASTs in a ('a, 'b) paired record type, so the standard AST is produced alongside the extended one during parsing
• Absorb Parse_with_comments into Extended_ast — Documentation bypasses the OCaml pipeline entirely (no hash-bang, w50, or token collection)
• Add Extended_ast.equivalent to check AST preservation, dump/dump_normalized for debug output
• Simplify Translation_unit: remove std_fg/std_parsed threading, no longer references Std_ast directly
• Clean up Std_ast: remove unused constructors (Repl_file, Documentation), of_syntax, comment collection
• Break dependency cycles (Source -> Extended_ast, Normalize_std_ast -> Normalize_extended_ast)
• Remove dead code (Extended_ast.equal, equal_core_type, Normalize_extended_ast.equal/ast)

Test plan

• `dune runtest` passes (204/204 tests)
• No behavior change for formatted output
• `printast` tool updated to use new API

[hhugo]

I believe this is a preexisting bug

[hhugo]

We now properly dump ast for Documentation and Repl_file

[hhugo]

This is no longer true, parsing documentation no longer goes through this help function.

[EmileTrotignon]

I know this part of the code was a bit convoluted, thanks for trying to simplify it. Looks a bit hard to review though

[Julow]

I'm not sure to understand what this type provides. Do we want to keep the two AST paired anywhere outside of Translation_unit ?
What are the benefits from moving the code from Parse_with_comments into Extended_ast ?

[hhugo]

Parse_with_comments.parse should only be used for "ocaml code", code parsed by the ocaml parser/lexer. It should not be used for Documentation for instance (it's wrong in the main branch).

We need to dispatch on the syntax to call it only when appropriate, the dispatch is already done in Extended_ast, it feels natural to move it there.

[hhugo]

I'm not sure to understand what this type provides. Do we want to keep the two AST paired anywhere outside of Translation_unit ?

I've been trying to minimize behavior changes in the PR.
We could probably parse the Std_ast only when needed in equivalent, saving the parsing for files already properly formatted. If we go that direction, we would need to have access to the source as string so that we can parse the std_ast.

[Julow]

We do have the source as string in Translation_unit, both before and after formatting. Perhaps Parse_with_comments could be extended with the dispatch ? (and possibly renamed)

[hhugo]

I can possible move back some parsing logic into Parse_with_comments. But I'm not sure it gives much benefit.

[hhugo]

I've added back Parse_with_comments.. Let me know what you think

[Julow]

I'm not sure to see the point of keeping both Parse_with_comments and the paired type. I think I prefer to keep Parse_with_comments because otherwise, we just add a lot of code into Extended_ast that in fact work with something else than the extended ast.

I don't think the equivalent function make much sense. It should be 'a Std_ast.t -> ... -> 'a -> 'a -> ast_check_result and not work with the extended AST at all.
It also can reasonably be string -> string -> .. and we could unify Std_ast.t and Extended_ast.t:

val ext_parse : ('ext, 'std) t -> string -> 'ext
val std_parse : ('ext, 'std) t -> string -> 'std

val equivalent :
     ('ext, 'std) t
  -> ...
  -> string
  -> string
  (* or 'std -> 'std -> *)
  -> ast_check_result

[hhugo]

The equivalent signature you propose forces to parse the same string multiple time when iterating to find a fix point. This is something Ive optimized long time ago in the early days of ocamlformat.

[hhugo]

• equivalent can't take a 'std because not all format have one.
• Extracting comment is very much linked to the implementation of the parser. It seems unnatural to have the logic spread across to files.
• Parse_with_comment and paired are orthogonal.

because otherwise, we just add a lot of code into Extended_ast that in fact work with something else than the extended ast.

The Std_ast is pretty much an implementation detail of the check of some of the file format

[hhugo]

@Julow, think more about this. I wonder if we should inline fields from the "with_comment" record into Extended_ast.t.

• the prefix is only relevent for use_file (and structure maybe)
• comments is not relevent for documentation I think
• Source is a bit weird because it deal with "ocaml" token. which we don't exactly have/want for mll mly files for instance.

The change would mean that an Extended_ast.t contains every thing needed for printing and checking.

[hhugo]

I know this part of the code was a bit convoluted, thanks for trying to simplify it. Looks a bit hard to review though

The PR is now split into more digest commits