Regression 4.0.10 to 4.0.11: -pthread + -sWASM_WORKERS aborts on trivial allocation/deallocation workload

[slowriot]

There is a regression between 4.0.10 and 4.0.11. The reproduction code below will crash reliably when built with 4.0.11 upwards, but not with 4.0.10.

Code to repoduce in C++:

#include <cstdint>
#include <emscripten.h>
#include <emscripten/wasm_worker.h>

auto main() -> int {
  emscripten_wasm_worker_post_function_v(emscripten_malloc_wasm_worker(1024 * 1024), []{
    for(unsigned i{}; i < 70'000'000; ++i) {
      delete new uint8_t{0};
    }
  });
  emscripten_set_main_loop([]{new uint8_t{0};}, 0, false);
  return EXIT_SUCCESS;
}

Code to repoduce in C:

#include <emscripten.h>
#include <emscripten/wasm_worker.h>
#include <stdlib.h>

static void worker_loop(void) {
  for(unsigned i = 0; i < 70000000u; ++i) {
    free(malloc(1));
  }
}

static void main_loop(void) {
  malloc(1);
}

int main(void) {
  emscripten_wasm_worker_post_function_v(emscripten_malloc_wasm_worker(1024 * 1024), worker_loop);
  emscripten_set_main_loop(main_loop, 0, false);
  return EXIT_SUCCESS;
}

Building and running:

em++ main.cpp -pthread -sWASM_WORKERS -o build/repro.html
emcc main.c -pthread -sWASM_WORKERS -o build/repro-c.html
emrun build/repro.html
emrun build/repro-c.html

The following elements are all needed for this to take place:

• At least one allocation in the main thread (doesn't matter if it's freed or not). Without this allocation, no crash.
• A large (not fixed) number of allocations and frees on the worker thread. Larger allocations seem to reduce the number needed to trip the bug, but no specific limits have been found.

Browser console output

The browser-visible failure is typically in one of the following shapes, changing from run to run:

On the main thread:

Uncaught RuntimeError: Aborted(native code called abort())
abort http://localhost:6931/repro.js:919
__abort_js http://localhost:6931/repro.js:1843
callUserCallback http://localhost:6931/repro.js:1542
runIter http://localhost:6931/repro.js:2190
MainLoop_runner http://localhost:6931/repro.js:2297
repro.js:919:11
abort http://localhost:6931/repro.js:919
__abort_js http://localhost:6931/repro.js:1843
abort http://localhost:6931/repro.wasm:15306
emscripten_builtin_malloc http://localhost:6931/repro.wasm:28994
repro.wasm.operator_new_impl(unsigned long) http://localhost:6931/repro.wasm:37895
repro.wasm.operator new(unsigned long) http://localhost:6931/repro.wasm:37857
repro.wasm.main::$_1::operator()() const http://localhost:6931/repro.wasm:2640
repro.wasm.main::$_1::__invoke() http://localhost:6931/repro.wasm:2398
callUserCallback http://localhost:6931/repro.js:1542
runIter http://localhost:6931/repro.js:2190
MainLoop_runner http://localhost:6931/repro.js:2297

On the worker:

RuntimeError: Aborted(native code called abort()) repro.js:919:11
abort http://localhost:6931/repro.js:919
__abort_js http://localhost:6931/repro.js:1843
abort http://localhost:6931/repro.wasm:15306
emscripten_builtin_malloc http://localhost:6931/repro.wasm:28994
repro.wasm.operator_new_impl(unsigned long) http://localhost:6931/repro.wasm:37895
repro.wasm.operator new(unsigned long) http://localhost:6931/repro.wasm:37857
repro.wasm.main::$_0::operator()() const http://localhost:6931/repro.wasm:2513
repro.wasm.main::$_0::__invoke() http://localhost:6931/repro.wasm:2323
_wasmWorkerRunPostMessage http://localhost:6931/repro.js:1568
callUserCallback http://localhost:6931/repro.js:1542
_wasmWorkerRunPostMessage http://localhost:6931/repro.js:1568
(Async: EventListener.handleEvent)
_wasmWorkerInitializeRuntime http://localhost:6931/repro.js:1616
initRuntime http://localhost:6931/repro.js:787
run http://localhost:6931/repro.js:3063
removeRunDependency http://localhost:6931/repro.js:889
receiveInstance http://localhost:6931/repro.js:1061
wasmModuleReceived http://localhost:6931/repro.js:1107
startWasmWorker http://localhost:6931/repro.js:643
onmessage http://localhost:6931/repro.js:680
(Async: EventHandlerNonNull)
http://localhost:6931/repro.js:673

Note that the above programs will periodically produce EITHER or BOTH of these, varying randomly from run to run.

One important point is that the abort site is not stable. Depending on the exact repro shape and timing, the failure can surface
during either allocation or deallocation. In the larger original program this showed up variously on malloc, free, operator new,
operator delete, and other ordinary STL/string/container allocation sites. In other words, it does not look like one specific call
is “the bug”; it looks more like some allocator/runtime corruption or invalid state is reached first, and the crash is only detected
later at a normal heap operation.

Other observed characteristics:

• the crash is not instantaneous; it happens after a little runtime/allocator churn
• it requires -pthread even though no pthread features are used in the example
• it does not require any real application logic, WebGPU, GUI, or shared data structures
• increasing allocation size reduces the amount of iteration needed before the abort, which suggests the issue scales with allocator churn / bytes processed rather than with one specific code path

So the failure pattern is: ordinary heap traffic on the main thread plus repeated heap traffic on a wasm worker, under -pthread + -sWASM_WORKERS, eventually leads to a browser abort in 4.0.11, even though the same code remains stable on 4.0.10.

Impact and discovery

This is currently impacting real-life applications using WebGPU - after Emscripten's WebGPU internal support is dropped in favour of --use-port=emdawnwebgpu, a -pthread requirement has been added when using with WASM_WORKERS (because emdawnwebgpu requires mutex features). So unfortunately it's not just an obscure corner case. Code that worked on 4.0.10 now crashes on 4.0.11 upward.

[sbc100]

Thanks for the report. I've been working recently on issues related to Pthead + WASM_WORKERS so I'm curious to get the bottom of this.

[sbc100]

I'm curious if you were able to reproduce this issue with the version latest 5.0.4 release? I assume so?

How about under node? Does node build/repro.js ever crash for you? (I tried this just now to no avail, but will try the web version when i get back tomorrow).

[slowriot]

@sbc100 Yes, this reproduces on every version I tested, a random selection between 4.0.11 up to and including 5.0.4. I'm away from the workstation so unable to test node, but can try that tomorrow.

[slowriot]

I should add, in terms of browsers, my testing so far has been on Firefox Nightly and a recent Chromium build, both on linux.

[slowriot]

Another finding: it seems that this does not occur in release / NDEBUG builds. Specifically, setting -sASSERTIONS=0 is sufficient to prevent the bug. So it's probably specific to the debug variants libc-mt-debug.a / libdlmalloc-mt-debug.a.

[sbc100]

I can reproduce the .cpp crash on node, but no the .c version. Are you sure the above .c version has the same issue?

[sbc100]

It looks like the problem is that when we build with -pthread + -sWASM_WORKERS the version of dlmalloc you get is libdlmalloc-mt-debug.a. This version of the library is built with pthread support and it will use pthread_xxx APIs internally. However, these obviously do not work correctly when called from a wasm worker.

IIUC this has always been an issue ever since its been possible to link with both pthreads and wasm workers in the same build. I'm not sure why it didn't also crash for you in 4.0.10 TBH.

I've been work on various solution to this issue recently. See the discussion here: #26568 (comment)

[sbc100]

I can think of a couple of different options for now to proceed from here:

1. Make it so that the pthread-API can be used form wasm workers in the hybrid mode.
2. Force the use of the atomics-only version malloc in the hybrid mode (and add some kind of assertions to the pthread function so that will trap if called from wasm workers).

[sbc100]

Using the cpp version above (I still cannot get the C version to crash) I was able to bisect to #24607. It makes sense that that commit would cause this failure, but its not really the cause of the bug. The bug is that pthread_mutex API should be used from wasm workers, and it just happened to be silently (and wrongly) working prior to #24607.

Basically the pthread_mutex_lock calls that dlmalloc was making from the wasm worker were always invalid and not doing their job correctly, but #24607 just made this more clear.

[sbc100]

The root cause here is that libdlmalloc-mt-debug.a calls pthread APIs

[slowriot]

@sbc100 It looks like in 4.0.11, PTHREAD_MUTEX_NORMAL changed in debug system-library builds so it no longer takes the old fast CAS path. The comment says why: “always take the slow path in debug builds so we can trap rather than deadlock” in emsdk-4.0.11/upstream/emscripten/system/lib/libc/musl/src/thread/pthread_mutex_lock.c:5 and again in emsdk-4.0.11/upstream/emscripten/system/lib/libc/musl/src/thread/pthread_mutex_timedlock.c:64. - this appears to be the breaking change.

In 4.0.10, normal mutexes always used the fast a_cas(..., 0, EBUSY) path in emsdk-4.0.10/upstream/emscripten/system/lib/libc/musl/src/thread/pthread_mutex_lock.c:3 and emsdk-4.0.10/upstream/emscripten/system/lib/libc/musl/src/thread/pthread_mutex_timedlock.c:62. In 4.0.11, that fast path is only used when NDEBUG is set, because of the #if !defined(__EMSCRIPTEN__) || defined(NDEBUG) guard in those files.

Regarding node repros, node build/repro.js gives:

Aborted(Assertion failed: at || m->_m_count != __pthread_self()->tid && "pthread mutex deadlock detected", at: /emsdk/emscripten/system/lib/libc/musl/src/thread/pthread_mutex_timedlock.c,91,__pthread_mutex_timedlock)

node:internal/event_target:1122
process.nextTick(() => { throw err; });
^
Error [RuntimeError]: Aborted(Assertion failed: at || m->_m_count != __pthread_self()->tid && "pthread mutex deadlock detected", at: /emsdk/emscripten/system/lib/libc/musl/src/thread/pthread_mutex_timedlock.c,91,__pthread_mutex_timedlock)
at abort (/home/slowriot/code/webgpu-demo3/build/repro.js:919:11)
at ___assert_fail (/home/slowriot/code/webgpu-demo3/build/repro.js:1830:7)
at repro.wasm.__pthread_mutex_timedlock (wasm://wasm/repro.wasm-001567fe:wasm-function[183]:0x3b7e)
at repro.wasm.__pthread_mutex_lock (wasm://wasm/repro.wasm-001567fe:wasm-function[187]:0x3bc1)
at repro.wasm.emscripten_builtin_free (wasm://wasm/repro.wasm-001567fe:wasm-function[235]:0x7a34)
at repro.wasm.operator delete(void*) (wasm://wasm/repro.wasm-001567fe:wasm-function[257]:0x942b)
at repro.wasm.operator delete(void*, unsigned long) (wasm://wasm/repro.wasm-001567fe:wasm-function[258]:0x9433)
at repro.wasm.main::$_0::operator()() const (wasm://wasm/repro.wasm-001567fe:wasm-function[29]:0x9c6)
at repro.wasm.main::$_0::__invoke() (wasm://wasm/repro.wasm-001567fe:wasm-function[27]:0x90e)
at /home/slowriot/code/webgpu-demo3/build/repro.js:1568:69

Node.js v22.22.0

And node build/repro-c.js gives an identical result:

Aborted(Assertion failed: at || m->_m_count != __pthread_self()->tid && "pthread mutex deadlock detected", at: /emsdk/emscripten/system/lib/libc/musl/src/thread/pthread_mutex_timedlock.c,91,__pthread_mutex_timedlock)

node:internal/event_target:1122
process.nextTick(() => { throw err; });
^
Error [RuntimeError]: Aborted(Assertion failed: at || m->_m_count != __pthread_self()->tid && "pthread mutex deadlock detected", at: /emsdk/emscripten/system/lib/libc/musl/src/thread/pthread_mutex_timedlock.c,91,__pthread_mutex_timedlock)
at abort (/home/slowriot/code/webgpu-demo3/build/repro-c.js:919:11)
at ___assert_fail (/home/slowriot/code/webgpu-demo3/build/repro-c.js:1830:7)
at repro-c.wasm.__pthread_mutex_timedlock (wasm://wasm/repro-c.wasm-0011e162:wasm-function[179]:0x3a1e)
at repro-c.wasm.__pthread_mutex_lock (wasm://wasm/repro-c.wasm-0011e162:wasm-function[183]:0x3a61)
at repro-c.wasm.emscripten_builtin_malloc (wasm://wasm/repro-c.wasm-0011e162:wasm-function[207]:0x3f72)
at repro-c.wasm.worker_loop (wasm://wasm/repro-c.wasm-0011e162:wasm-function[25]:0x889)
at /home/slowriot/code/webgpu-demo3/build/repro-c.js:1568:69
at callUserCallback (/home/slowriot/code/webgpu-demo3/build/repro-c.js:1542:9)
at _wasmWorkerRunPostMessage (/home/slowriot/code/webgpu-demo3/build/repro-c.js:1568:19)
at MessagePort.f (/home/slowriot/code/webgpu-demo3/build/repro-c.js:661:20)

Node.js v22.22.0

You may need to increase the loop count or just make the loop infinite using for(;;) to get it to trigger on your machine - the loop count was just an arbitrary value I chose which seemed to reproduce reliably on my workstation.