Should we switch compilation state holder from Task.dest to a worker?

[lefou]

Currently, compilation state is hold in a specific Task.dest. That means, the only way to safely access that state is by evaluating that task and using it's result. But especially compilation processes have typically more state and interested parties than the single compile task that holds the data.

The most prominent issue is BSP. Due to the way the protocol is designed, BSP clients get access to the compilation target directory, whereas in Mill there isn't really an exposable dir. There is only a final classes dir as a result of the compilation. This leads to a lot on idiosyncrasies in the Mill code. And we still have many issues with that, mostly visible as poor performance.

But we have other state, that isn't directly exposed as a task result, but is nevertheless interesting. E.g. cached compiler messages. For example this question about a possibility to always should compiler messages. Wouldn't it be nice to have a task that simply can show all past but valid compiler messages like warnings and errors without forcing Mill to recompile everything by running mill clean __.compile before?

If we would hold the state in a Worker instead, we could have more than one task that access that state, without exposing everything. Most users of compile are only interested in it's classpath contribution downstream, but not in the incremental state file. But compile itself is sometimes using the incremental information. But other information like cached previous compiler problems aren't accessible at all. Instead, we could have a comiledClasses task only returning the classes and a compilationProblems task to access the compiler problems.

Once we switch over to a worker approach, some currently hard to model issues can be implemented more elegantly or can be solved at all. For example, the open on-demand SemanticDB data generation via a compiler plugin as part of the normal compilation, could be solved much more elegantly and without touching the API surface of the modules.

[lihaoyi]

One potential benefit of keeping the zinc analysis file in Task.dest is remote caching: being in Task.dest marks it as something that can be serialized and shared, whereas workers and their local filesystems generally cannot be. So if we start uploading task outputs to a remote cache, having the zinc file be included would mean people populating their local caches will be able to get started with incremental compilation without first doing an expensive full compile

It's not clear to me what BSP problems you are talking about or how they relate to where the zinc analysis file is kept

[lefou]

Yeah. After more thoughts and some experimenting, I'm not sure anymore, moving state to the worker is what we want. What we want is a more rich return value, containing all compilation results, including reported compile problems, which don't fit into ComilationResult.

What we probably want is some @internal compile task with a feature rich result, and one or more public tasks returning just the interesting parts like classes: Seq[PathRef], zincAnalysisFile: Option[PathRef], compilerProblems: Seq[Problem]. This would also solve issue #2781 (which we closed due to not being fixable via a single polymorphic return value).

It's not clear to me what BSP problems you are talking about

I was referring to "compile problems", the warnings and errors shown to the user when we actually run the compiler. Those are currently not accessible in the cached result and thus cannot be shown later (#6807).