[bug] [Performance Regression] Command execution time increased from 20s to 7min after v2 migration (1500+ projects)

[sybernatus]

Describe the bug

Significant performance regression observed after migrating from moon v1 to moon v2.0.4. In a large monorepo environment containing 1,500+ projects, the overhead of command initialization (the graph building and loading phase) has increased exponentially.

Total execution time for commands like moon ci has shifted from 10-20 seconds in v1 to 2-7 minutes in v2.0.4, even when the cache is warm or no tasks are actually executed.

Steps to reproduce

1. Use a monorepo with 1,500+ project definitions (moon.yml files).
2. Run a workspace-wide command: moon ci :<target> or a project-specific command: moon run <project>:<target>.
3. Observe the significant delay (several minutes) before the task execution output begins.

Expected behavior

Initialization and project/task graph scanning should remain performant (ideally under 120 seconds) even with 1,500+ projects, matching or improving upon the performance levels observed in version 1.
Or a possibility to skip some consuming optional treatment (but I'm not sure they are optional 😅)

Screenshots

Environment

moon 2.0.4
proto 0.55.4

Additional context

Here is a log trace for moon ci :lint for example. I anonymized some data on the project names & remove some logs as they are the same during all the calculation.
moon-short.log

[milesj]

@sybernatus This seems severe, going to need more information.

• What were your configs in v1? and what are they in v2?
• Were you on the latest v1?
• Any other environment/tooling changes?

Also that log doesn't provide enough information to debug.

[sybernatus]

Yes sorry @milesj you're right, it's not necessarily the upgrade to v2 because I jump a huge gap upgrading moon.

Were you on the latest v1?

I was in moon v1.32.6 so at least 1 year late.

What were your configs in v1? and what are they in v2?

My config didn't change moving to v2 except the breaking changes:

• layer instead of type
• adding windowsShell: bash on tasks so they can be run on bash (Just to confirm you that I have the issue on both linux & windows. I know that git-bash for windows is slow by default so I'm focusing on solving the performance issue on linux here, the os of our CI/CD pipelines)
• I have imported locally the system_toolchain.wasm in the repository and the following .moon/toolchains.yml file

  system:
    plugin: 'file://.moon/plugins/toolchains/system_toolchain.wasm'

Any other environment/tooling changes?

Only the moon/proto upgrade & proto has been moved to offline mode.

---

Just to give you more context because it's not really a complex moon project event if it has 1500+ projects, they are all configured the same and maybe, something will bring to your mind.

Almost all projects are inheriting from the a single task file (.moon/tasks/tag-helm_with-argocd.yml) and this task file have multiple tasks that are launching some bash script or command within the repository.

Here is an example of a project:

moon.yml

Here is the tag defining the tasks:

tag-helm_with-argocd.yml

---

I will try to extract a log that could be better than the previous one. The last one was really huge (1Go+) so I had to clear some logs.

---

EDIT

I've tried the extract a full log for the following command moon ci --log debug --job-total 1 --cache off -c 2 and extract some statistics:

moon-short.log

=== LOG STATS ===
Start time      : 2026-03-20 07:52:03.779]
End time        : 08:02:20.048]
Projects        : 1673
Requested tasks : 60247
Resolved tasks  : 0
Action count    : 1

Build workspace graph : 6m 40s (65%)
Affected tracking : 3m 13s (31%)

Step-by-step timings

Step	Module	Start	End	Duration
Startup / init	moon_app::session	07:52:03.779	07:52:03.985	206ms
Glob scan (locate moon.yml)	workspace_builder	07:52:03.986	07:52:04.584	598ms
Load 1673 project definitions	workspace_builder	07:52:04.584	07:52:05.254	670ms
Toolchain plugin registration	plugin_registry	07:52:05.276	07:52:10.198	4.9s
Version check + workspace hash	launchpad / hash_engine	07:52:10.198	07:52:14.293	4.1s
Build workspace graph (1673 projects)	workspace_builder	07:52:14.294	07:58:54.704	🔴 6m 40.4s
Enforce constraints + git metadata	workspace_builder / exec_command	07:58:54.704	07:58:55.370	666ms
Create project graph	project_graph	07:58:55.370	07:58:56.352	982ms
Create task graph	task_graph	07:58:56.352	07:58:56.512	160ms
Expand 60 247 tasks	task_expander	07:58:56.512	07:59:02.084	5.6s
Detect changed files (git)	queries::changed_files	07:59:02.084	07:59:03.882	1.8s
Build action graph + expand projects	action_graph_builder / project_expander	07:59:03.882	07:59:04.224	342ms
Affected tracking	affected_tracker	07:59:04.224	08:02:17.079	🔴 3m 12.9s
Final resolution (SyncWorkspace)	action_graph	08:02:17.079	08:02:20.048	3.0s

---

EDIT 2:

I just tested the CI with moon v1.41.8 and it really run fast as before. So I can now confirm that the performance issue is on moon v2 only

[milesj]

Interesting stats. Some of these are quite confusing, like "Toolchain plugin registration" taking 5s when you're not actually using toolchains, and "Affected tracking" which simply loops through the graph.

I'll see what I can do there.

As for the workspace graph taking so long, I'm actually working on a daemon for v2.2: #2427 This will run in the background and regenerate the graphs based on file system changes, so that it doesn't happen during moon execution. This should help greatly.

Also, if you're open to emailing me a full log, I can analyze it better. AI can probably help detect some issues.

[Duc-LLR]

Hi,
I am a colleague of @sybernatus, here are the full log of the command moon ci --log debug --job-total 1 --cache off -c 2 I launched :

moon-full-log.zip

If you want logs from other command to troubleshoot, do not hesitate.

[milesj]

This is what Claude had to say. I'll dig into this deeper.

Here's the report:

---

Performance Analysis: moon-anonyme.log

Context

Analysis of a 25MB (128K line) debug log from moon ci --log debug --job-total 1 --cache off -c 2 running moon v2.0.4 on Windows. The monorepo has 1,697 projects but only 2 projects are affected by changes. Total runtime is 13 min 22 sec, of which only 5m 42s is actual task execution. 57% of runtime (7m 40s) is overhead before any task runs.

---

Root Causes of Slowness

1. Content Hashing Black Hole — 234 seconds (29% of total)

Where: Between lines 3446-3447 (14:32:23 → 14:36:17) — a 234-second gap with ZERO log output.

This occurs immediately after all 1,697 projects finish loading, during workspace graph finalization. A git hash-object --stdin-paths call processes 161KB of file paths. The hashing of workspace content for all 1,697 projects appears to happen here with no intermediate logging.

moon source area: moon_workspace::workspace_builder — the gap occurs between "Building project" completing and "Enforcing project constraints" starting. The content hashing / graph building step in between has no debug instrumentation.

2. Affected Tracker — 143 seconds (18% of total)

Where: Between lines 66292-66293 — another silent gap.

The tracker evaluates all 61,111 expanded tasks to determine which are affected by file changes. Result: only 66 tasks across 2 projects are affected (0.1%). The algorithm does not appear to short-circuit on unaffected projects.

moon source area: The affected/changed files tracking system. It should be able to skip entire projects that have no changed files, but instead evaluates every task.

3. Project Loading — 56 seconds (7% of total)

Where: Lines 31-3446 (14:31:24 → 14:32:23)

Loading and renaming 1,697 projects at ~32 projects/sec. Each project triggers config loading + explicit identifier rename. The rate is consistent (no progressive degradation), suggesting this is I/O-bound on the Windows filesystem.

moon source area: moon_workspace::workspace_builder — "Loading projects" and "Project has been configured with an explicit identifier" repeated 1,697 times.

4. Task Expansion — 15 seconds (2% of total)

Where: Lines 3450-66292

All 61,111 tasks (36 task types × ~1,697 projects) are eagerly expanded before any filtering. Lazy expansion of only affected projects would eliminate this entirely.

moon source area: moon_workspace::workspace_builder — "Expanding task" repeated 61,111 times.

---

Task Execution Issues (the remaining 5m 42s)

• Concurrency is limited to 2 (-c 2), and execution is largely serial due to task dependencies
• Slowest tasks are helm operations (45-51s each)
• 28 passed, 21 failed, 18 skipped
• Failures are due to missing Chart.yaml, imageFactory.yaml, and release scripts — likely config/setup issues, not moon bugs

---

Summary

Phase	Duration	%	Optimization Opportunity
Content hashing (silent)	234s	29%	Add logging; consider incremental/cached hashing
Affected tracker	143s	18%	Short-circuit unaffected projects early
Project loading	56s	7%	Parallel config loading; caching
Task expansion	15s	2%	Lazy expansion (only affected projects)
Total overhead	~460s	57%
Task execution	342s	43%	Increase concurrency; fix failing tasks

The two biggest wins would be:

1. Skipping unaffected projects early — if the 1,695 unaffected projects were excluded before content hashing and task expansion, ~390s of overhead could be eliminated
2. Adding instrumentation to the 234s content hashing gap to understand exactly what's happening (this is the single largest time sink and is completely silent in debug logs)

[milesj]

Made some affected tracker changes in v2.1.2

[Duc-LLR]

Perfect, I will upgrade to it tomorrow and let you know how it behave.

[Duc-LLR]

I have tested this morning with the old version 2.0.4 and the newest one 2.1.2.

It doesn't seems to have significant improvement (with the newest version and the same repo state we can even see that there is a 1 minute difference for the overall moon ci command).

Here are the full logs for both command.

moon-performance-regression.zip

[milesj]

If you're running moon ci with no targets then it will effectively run against all tasks. Short-circuiting may not be enough here. But I'll see if there's anything more I can do.

[milesj]

In v2.2, I'm adding an experimental async tracker. Will help here: #2449