RFC: Modernizing Project Documentation with Docusaurus and Automated Sync Workflow

[ryankert01]

Context

Following our recent community call, we discussed the need to modernize our documentation infrastructure. The goal is to make it easier for contributors to write documentation while maintaining a high-quality, searchable website. Based on our conversation, I’m proposing we adopt Docusaurus and implement a specific documentation sync workflow.

The Proposal: Docusaurus

I propose we move our website/documentation framework to Docusaurus.

• Why: It is a React-based static site generator that handles Markdown-to-HTML beautifully. It is developer-friendly, provides excellent SEO, and supports versioning.
• Ease of Use: As discussed, it allows us to focus on content in Markdown without needing to manage complex HTML/CSS layouts manually.

Documentation Workflow: The "Sync" Approach

To keep the repository clean and contributor-friendly, I'd like to recreate a workflow previously used in the gofannon project:

1. Source of Truth: All documentation will live in a top-level /docs directory in the main branch.
2. Build Process: During the site build, a script will copy the contents of /docs into the /website directory.
3. Reference: We can adapt the sync script used here: [sync_docs.py (gofannon)](https://github.com/The-AI-Alliance/gofannon/blob/2eaf7d22a80f4ed8794c02b074ccda137b51ea0b/website/scripts/sync_docs.py).

Architectural Decision Record (ADR)

In the spirit of preserving project history and the "why" behind our technical choices, I also propose we can update our design with an ADR. This ensures that future maintainers understand our reasoning for choosing Docusaurus and this specific sync logic, preventing the "deep-magic" knowledge gaps that can occur during transitions.

Seeking Community Feedback

I’m opening this discussion to the community for feedback:

• Are there any concerns with using a React-based framework like Docusaurus?
• Does anyone have alternative suggestions for the sync workflow?
• If there is general consensus (or "lazy consensus"), I am happy to lead the initial setup and migration.

I look forward to hearing your thoughts!

[rawkintrevo]

I support this. Jekyll was good for what it was when it was (and still is).

React sites are prettier, but if they introduce too much overhead, not worth it. But as everyone and their cat can vibe a web ui now, that maintenance overhead is significantly reduced.

[400Ping]

I think it is a great idea and maybe we can also let some of the new contributors to work on this? like @viiccwen for example? He kind of wants to have more impact in Mahout but there's not much to do left in QDP.

[ryankert01]

I can write a migration plan first to see what's the overhead looks like.

[ryankert01]

I'll probably use an agent to do a POC with the migration plan, so we can review the plan along with some displays. cc @rawkintrevo

[krishna-dave206]

I agree with y'all. This would help us cover both the agendas: "Easy to maintain" and "Prettier"

[ryankert01]

I think it's too late to be part of this release. tho, I get some demos. As I dive in, I would say it need a good planning for docs placement. Today's documents is kinda scattered which add complexity to the migration.

[400Ping]

I am into migrating into a new version of website? I think this can be a post PoC thing and we can involve more new contributors to work on it in GSoC. If there's not enough people to work on it, we can do it in post PoC ourselves. In general I think we can like put it aside and focus on the release. But I think it is a necessary work in the long term maybe?

[ryankert01]

Don't get me wrong. I'm not worried about the engineering power, because I kinda near finish migration locally. I'm just worried about it probably will be a super big PR and that's hard to review.

[400Ping]

hmm I think maybe open it and have more time for people to review it? like having like 4~5 people reviewed and approved?

[rawkintrevo]

For what it's worth- it's docs/website which carries a significantly lower bar for reviewing. If something breaks, oh well just go fix it. It's not a critical dependency or anything. Also, on this PR its going to just be a bunch of boiler plate code anyway. I did gofannon earlier today. Also, don't forget to disable the website build in your initial commit (so the old site remains while we build the new one. (Or do the whole thing on a branch) (edit) you are actually getting a bunch of it in the first shot

[ryankert01]

Great advice! I think disable website build is a great solution!

[magsol]

+1 from me. Love the "single source of truth" approach. The only bit of feedback I'd give is that docusaurus is a BEAST that will do absolutely everything you need it to, and then some. So be prepared for initial setup and config to be a heavier lift than you're expecting, but life after that should be pretty nice.