Wasm execution is very slow compared to wazero.

[dzbarsky]

Bazel is using Chicory to execute wasm, and I noticed that a workload that Rust can complete in 5-10ms takes around 20ms on wazero (and 80ms of loading/compilation time) but multiple seconds on Chicory. For comparison, Python can complete the same work in around 150-200ms, so this is 20-30x slower. This was using the interpreter, but even when using the Chicory AOT compiler, the result is still 2x slower than Python.

bazelbuild/bazel#26975 (comment) has more details on the setup/investigation along with a link to a standalone repo with the Rust source code and the compiled wasm

[andreaTP]

Thanks for taking the time to file this issue!
We have an ongoing effort to tackle performance issues, we'd take a closer look soon and report back.

[dzbarsky]

Ok, that's great to hear, will stay tuned.

Also, if the compiled results were able to be serialized to disk and hydrated back later that would also be really nice for Bazel's use case.

[andreaTP]

@dzbarsky again, thanks a lot for the reproducer, extremely useful!

I went ahead and added checks for Chicory in various modes: https://github.com/andreaTP/bazel-wasm-repro/tree/bench-chicory-modes those are some rough results and we have a setup to play with the matrix.

First off, measuring using currentTimeMillis is completely inaccurate, and including the startup time is going to give very bad results.

That said, results are expected:

• interpreter: extremely slow
• runtime compiler: fast execution, but the compilation takes time
• build time compiler: fast execution, fast loading, using native-image startup time becomes comparable

I know nothing about Bazel and the execution modes, so I'd need some help to be able to give guidance.

if the compiled results were able to be serialized to disk

Yes, it's totally doable using the build time compiler API instead of through Maven/Gradle, for example.
Please note that the generated sources are not necessary for execution(they are just IDE's helpers).

[andreaTP]

@dzbarsky @jmillikin

to be clear, I have zero experience with Bazel, I'm looking for guidance here:

• how is Chicory is executed in Bazel? Is always hitting a cold start or there are caching opportunities already built in?
• can I ask for a reproducer project and a couple of lines on how to run/debug have flamegraphs from the Bazel build?

I thought a bit about this use case, and I start to be keen in adding a "cache" functionality for the runtime compiler.
For example, adding a withCache option to this builder.
Looking for feedback about this proposal @chirino

[jmillikin]

The Wasm-related code in Bazel is in https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/bazel/repository/starlark/StarlarkWasmModule.java . There's some limited caching -- a loaded module can be shared between multiple executions within the scope of a single external dependency. Modules are instantiated once per execution, then some setup code happens and a named function is executed.

bazelbuild/bazel#26974 is a feature request to improve caching of loaded Wasm modules from the Bazel side.

StarlarkWasmModule.java is pretty self-contained so I think it would be straightforward to construct a reproducer from it by just deleting the Bazel-specific annotations and integration glue code. If a reproducer doesn't show performance issues then I can try to help narrow down where the slowdown is happening.

[andreaTP]

Thanks @jmillikin for the details, I skimmed through the implementation, I think that something along the lines of #1041 (wherever it goes) makes you able to effectively leverage the runtime compiler.
It should be doable, in user land, today, but I understand a little convenience is necessary to make the process smooth.

This should give you a minimum of a 10X speedup on average loads, paying the compilation time only seldom.
Would it be acceptable? What kind of runtime performance are you looking for?

An alternative would be to use the build-time compiler, does Bazel have a way to load things on the classpath dynamically?

[jmillikin]

I think the main focus is on the execution time. Load/compilation time is less important due to possible caching improvements on the Bazel side.

Does the Chicory build-time compiler generate faster JVM bytecode than its runtime compiler?

bazelbuild/bazel#26975 (comment) reports that @dzbarsky's workload (parsing TOML) is 15-20x slower when comparing Chicory's runtime compiler vs wazero. I think the first step is to establish the cause (interpreter fallback? non-optimal generated bytecode? missing optimizations?), then we'll have a better idea of how much performance improvement can be expected.

[andreaTP]

Does the Chicory build-time compiler generate faster JVM bytecode than its runtime compiler?

The bytecode is the same, there is a slight difference in how is invoked, but is usually negligible in terms of perf.

I think the first step is to establish the cause

From what I can see:

• using the interpreter is the top-most cause
• missing optimizations might come second

You can look up at the results here to settle expectations around the various execution modes.

[andreaTP]

I opened #1044 to show how is possible to cache the compilation and restore it building a MachineFactory.
I understand the implementation is not ideal, but it should have all the necessary building blocks.

From here I need some help, either:

• I need guidance or an example of using caches inside Bazel for this use case(and possibly some hand holding along the process)
• someone volunteer to bring the POC in Bazel

Thoughts?

[fmeum]

What is the compatibility story around precompiled code? Do we have to key it by chicory version?

[dzbarsky]

I opened #1044 to show how is possible to cache the compilation and restore it building a MachineFactory. I understand the implementation is not ideal, but it should have all the necessary building blocks.

From here I need some help, either:

* I need guidance or an example of using caches inside Bazel for this use case(and possibly some hand holding along the process)

* someone volunteer to bring the POC in Bazel

Thoughts?

Hi @andreaTP, I previously tried https://github.com/bazelbuild/bazel/pull/27064/files#diff-1426ba04b1c518e9dc191419ad602563f87c5032ec96d72913e8c2ea19423885 to switch Bazel to using the compiler API instead of the interpreter; I think we can play around with the in-memory caching to start as shown there to see if we can get Chicory into an acceptable performance range. If you want to play around with the bazel integration, the steps would be:

1. Clone the bazel repo, checkout my PR branch, make changes
2. Install a bazelisk for bootstrapping (see https://github.com/bazelbuild/bazelisk?tab=readme-ov-file#installation)
3. bazel build //src:bazel to build the version with the changes
4. Checkout the wasm-repro repo
5. In that repo, /path/to/bazel/checkout/bazel-bin/src/bazel build @wasm-repro//... --profile=/tmp/profile.json
6. Open the profile.json in Chrome's chrome://tracing to see the execution spans, including wasm compile and execution times

[dzbarsky]

* can I ask for a reproducer project and a couple of lines on how to run/debug have flamegraphs from the Bazel build?

In the wasm-repro repo you can run bazel build @wasm-repro//... --profile=/tmp/profile.json and visualize the profile in Chrome's chrome://tracing

[andreaTP]

@fmeum excellent question!
Tl;DR: if it breaks is a bug on us, this is a longer discussion, and there might be slight nuances, but that's where we stand.

[dzbarsky]

@dzbarsky again, thanks a lot for the reproducer, extremely useful!

I went ahead and added checks for Chicory in various modes: https://github.com/andreaTP/bazel-wasm-repro/tree/bench-chicory-modes those are some rough results and we have a setup to play with the matrix.

First off, measuring using currentTimeMillis is completely inaccurate, and including the startup time is going to give very bad results.

That said, results are expected:

* interpreter: extremely slow

* runtime compiler: fast execution, but the compilation takes time

* build time compiler: fast execution, fast loading, using `native-image` startup time becomes comparable

I think it's helpful to take a step back and explain how wasm is used in Bazel, so it's clearer the constraints we want to achieve.

At a high level, Bazel is a build system that allows to schedule and execute a DAG of build actions that turn source code and artifacts into further artifacts. It is common for tools used during the build to also be built from source and serve as inputs to downstream actions.

There exists a pre-build phase known a "repository rules" which serve to bring in external dependencies to the workspace, so they can be used as part of the build. Repository rules are often used for integration with language-specific package managers/repositories (maven, pypi, npm, cargo, etc.). These repository rules often want to execute a tool to parse the lockfile or perform other actions, but since they are running before the build phase, they are unable to bootstrap these tools from source using the standard build rules. That leaves repository rules authors with a few choices:

1. They can compile the tools from source imperatively (such as running cargo build instead of using the bazel rust rules, etc). This can be a source of slowness and non-hermiticity if care is not taken when crafting the build command
2. They can download pre-built tools. This is hermetic but requires building many variants of tools, for all potential OS/arch combinations that downstream users will use
3. They can attempt to reimplement the tools using Bazel's built-in scripting language (which can be annoying and slow)

Wasm execution is intended to provide a 4th alternative to these options - like (2), the ruleset author can use a prebuilt wasm version of the tool, but it only needs to be compiled once.

Essentially, this means that we are trading off runtime performance for all downstream users for ergonomics/convenience of ruleset authors. To me, that means the target needs to be as close to native performance as possible. It's hard to say where each ruleset author will draw the line, but as a strawman, something like 2x slower than native is probably OKish and something 10x slower than native will likely incentivize ruleset authors to prebuild the native tools, and thus not use the wasm functionality.

In other words, if the runtime performance of the wasm tools cannot be made fast enough, there won't be much point in using it :) As the author of rules_rs, the wasm performance was slow enough that I went with option (2) above. The direct binary takes around 5-10ms on the Cargo.lock in the wasm-repro repo, so that should be the target we are aiming for with the wasm runtime. And then the question becomes how close to that we can get. It sounds like the compiler API gets us to a runtime of 400ms or so, so 40-80x slower than native?

[andreaTP]

@dzbarsky first off, thanks a lot for the instructions on how to hack and verify changes in Bazel, that's great!

Also, thank you for the write-up. It provides good context and definitely moves the conversation forward.

I want to point out that if native performance is the only goal, the expectations may be slightly misaligned. Wasm execution is sandboxed and single threaded. For example, a Rust library that optionally uses multiple threads will inevitably show performance results that are not directly comparable to its Wasm counterpart. There are proposals in progress to address this, but nothing that can be applied in practice today.

To put it another way, the expectation should be that Chicory’s speed can reach performance similar to wazero or wasmtime. However, in your reproducer, wazero is still about ten times slower than native.

That said, there are two additional angles that are important in this discussion.

Fallback on unavailable architectures. Wasm can serve as a default fallback when the tool is not compiled for a specific platform. In such cases, slightly degraded performance is acceptable if it allows the full functionality to remain available.

Security. This is the most important aspect. Wasm binaries run in a safe sandbox. They do not have access to Bazel memory in the case of pure Java dependencies, nor to system memory in the case of native execution. Memory isolation helps avoid a number of well known vulnerabilities and supply chain attacks. Since this is a build tool, I believe this is the strongest argument in favor.

In the case of Chicory specifically, code is executed fully within the JVM without the need for native dependencies to be compiled or distributed.