Adopt MyST as the default markdown parser in version 5

[offbyone]

The top line proposal is what it says on the tin. I'd like to consider adopting MyST as the default markdown format for Pelican 5. A major version bump is reasonable, as it would change the parsing handling of .md files in a way that would require explicit modification.

For reference, here is the MyST project's documentation

Why should we do this?

MyST is a more metadata-friendly CommonMark markdown format, with support for many of the docutils-derived capabilities that are currently only really available to RST articles in Pelican. While pymarkdown supports extensions as well, MyST provides a more cohesive set of functions out of the box.

How should we do this?

My recommendation is to put it into core with two modes in the 4.x stream: With .myst support by default, and .md support as a configurable option.

Configuration

During the 4.x series, the default MARKDOWN settings dict would be retained as configuration for the current markdown, and a new MYST settings dict would be added.

After 5.0 settings would prefer MYST if present, and use MARKDOWN to configure myst if no MYST block is present. No settings merging would occur, to avoid confusion in upgrade scenarios. If using MARKDOWN for MYST, then LEGACY_MARKDOWN will be used for the older parser.

Content Upgrades

No automated content upgrades would take place.

Retaining the legacy markdown reader

I think we should do so, to provide an off-ramp for iterative migration of existing markdown blogs.

Prior Art

• An existing extension: https://pypi.org/project/pelican-myst-reader/
• A draft PR for direct in-core support: Feature: support for MyST markdown input #3551

[minchinweb]

c.f. (Historical) discussion about moving the default Markdown to CommonMark, (as we are already installing a CommonMark parser with rich) --> #2984

[minchinweb]

Two initial thoughts:

• MyST uses YAML blocks for frontmatter (i.e. keys between two --- lines) while Pelican currently relies on just a blank line after the keys.
• Python-Markdown (our current Markdown library) is explicit in not conforming with the CommonMark specification (under their "Goals"):

This is not a CommonMark implementation; nor is it trying to be! Python-Markdown was developed long before the CommonMark specification was released and has always (mostly) followed the syntax rules and behavior of the original reference implementation. No accommodations have been made to address the changes which CommonMark has suggested. It is recommended that you look elsewhere if you want an implementation which follows the CommonMark specification.

The combination is Pelican has a somewhat distinct flavour of Markdown in practice. Changing the Markdown flavour runs the risk of introducing subtle rendering changes.

---

For what it's worth, Pelican supports ReST out of the box, but requires additional downloads to support Markdown.

[minchinweb]

As for this proposal, on one hand, supporting MyST (or other additional parsers) seems fine. However, it would increase the maintenance burden as well.

I'm also cautious about moving to another "special" Markdown flavour... CommonMark has done a good job of marketing themselves as the "final Markdown", but I still find myself reaching for extensions when using CommonMark. Moving to Markdown would have the same possible issues with rendering changes, but would, in theory, make Pelican posts more likely to be directly useful in other "Markdown" consuming applications (like Obsidian).