Core ML segfault when running pybindings

[GregoryComer]

🐛 Describe the bug

Running a CoreML-delegated model using pybindings crashes with a segfault. This started within the last few weeks, and seems to be consistent. I can't reproduce the fault on XNNPACK or Vulkan, though from the initial bisect (see below), this might just be due to chance.

Repro:

import torch

from executorch.backends.apple.coreml.partition import CoreMLPartitioner
from executorch.exir import to_edge_transform_and_lower, to_edge
from executorch.extension.pybindings.portable_lib import _load_for_executorch_from_buffer

class Model(torch.nn.Module):
    def __init__(self):
        super().__init__()
    def forward(self, x):
        return x * 2

model = Model()
inputs = (
    torch.randn(1, 2024),
)

eager_outputs = model(*inputs)

ep = torch.export.export(model, inputs)
lowered = to_edge_transform_and_lower(
    ep,
    partitioner=[CoreMLPartitioner()],
).to_executorch()

print(ep)

et_model = _load_for_executorch_from_buffer(lowered.buffer)
et_outputs = et_model([*inputs])[0]

et_outputs - eager_outputs

Note that it repros when running the script standalone, but not from a notebook. From initial investigation, it appears to be faulting on cleanup.

From @metascroy:
"I bisected the error to #11391."

Versions

M1 Pro, reproduced on cd90531. coremltools version 8.3.0.

cc @larryliu0820 @JacobSzwejbka @lucylq @kimishpatel @YifanShenSZ @cymbalrush @metascroy

[metascroy]

@cccclai I bisected the issue to #11391. Can you have a look? @GregoryComer's script segfaults on cleanup of the backend delegate. Here is the stacktrace: P1866017750

There might be broader issues with ET on macOS because we also see segfaults on mac runners on XNNPACK models too when trying to bump the ET pin in optimum-executorch: huggingface/optimum-executorch#103

cc @guangy10

[GregoryComer]

ASAN capture:

=================================================================
==46186==ERROR: AddressSanitizer: global-buffer-overflow on address 0x00014cee24c8 at pc 0x00014cc34fe4 bp 0x00016b4d2690 sp 0x00016b4d2688
READ of size 8 at 0x00014cee24c8 thread T0
    #0 0x00014cc34fe0 in executorch::runtime::Method::~Method()+0x378 (_portable_lib.cpython-310-darwin.so:arm64+0x27d4fe0)
    #1 0x00014cc45b40  (_portable_lib.cpython-310-darwin.so:arm64+0x27e5b40)
    #2 0x00014cc61000  (_portable_lib.cpython-310-darwin.so:arm64+0x2801000)
    #3 0x000109600e7c in pybind11::detail::clear_instance(_object*)+0x17c (libtorch_python.dylib:arm64+0x14e7c)
    #4 0x000109600b10 in pybind11_object_dealloc+0x28 (libtorch_python.dylib:arm64+0x14b10)
    #5 0x0001049d7080 in PyDict_SetItem+0x74c (python3.10:arm64+0x1000ab080)
    #6 0x0001049f021c in _PyModule_ClearDict+0x228 (python3.10:arm64+0x1000c421c)
    #7 0x000104b102f0 in finalize_modules_clear_weaklist+0x160 (python3.10:arm64+0x1001e42f0)
    #8 0x000104b0ef2c in finalize_modules+0x70 (python3.10:arm64+0x1001e2f2c)
    #9 0x000104b0ed48 in Py_FinalizeEx+0x74 (python3.10:arm64+0x1001e2d48)
    #10 0x000104b36994 in Py_RunMain+0x28 (python3.10:arm64+0x10020a994)
    #11 0x00010493264c in main+0x34 (python3.10:arm64+0x10000664c)
    #12 0x00018b2deb94 in start+0x17b8 (dyld:arm64e+0xfffffffffff3ab94)

0x00014cee24c8 is located 8 bytes after global variable 'vtable for executorch::backends::coreml::CoreMLBackendDelegate' defined in '/Users/gjcomer/src/executorch/backends/apple/coreml/runtime/delegate/coreml_backend_delegate.mm' (0x00014cee2480) of size 64
SUMMARY: AddressSanitizer: global-buffer-overflow (_portable_lib.cpython-310-darwin.so:arm64+0x27d4fe0) in executorch::runtime::Method::~Method()+0x378
Shadow bytes around the buggy address:
  0x00014cee2200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x00014cee2280: 00 00 00 00 00 f9 f9 f9 f9 f9 f9 f9 00 00 00 00
  0x00014cee2300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x00014cee2380: 00 00 00 00 00 f9 f9 f9 f9 f9 f9 f9 00 00 00 00
  0x00014cee2400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x00014cee2480: 00 00 00 00 00 00 00 00 f9[f9]f9 f9 00 00 00 00
  0x00014cee2500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x00014cee2580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x00014cee2600: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x00014cee2680: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x00014cee2700: 00 00 00 00 00 f9 f9 f9 f9 f9 f9 f9 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==46186==ABORTING

[cccclai]

I think the backend option isn't likely the case, because it's not used and probably stripped during compilation. I have an internal diff "Use exension module for pytbind" and maybe can try that to see if CoreML segfault is resolved. The module definition inside pybindings is quite outdated and the one inside the extension is more reliable

[GregoryComer]

I dug around a bit more. From the instructions surrounding the faulting address, it looks like it's failing to call destroy on the backend here:

executorch/runtime/executor/method.cpp

Line 124 in 1fdebef

backend_->destroy(handle_);

From looking at the instructions in lldb, it's faulting on making an indirect call to a null address (reading the vtable and calling). From ASAN, the read starts 8 bytes after the end of the vtable for CoreMLBackendDelegate. If I'm following correctly, that lines up with 2 missing entries (1 missing would read 0 bytes after the end).

That lines up with the idea that somehow, the Core ML backend class is missing implementations + vtable entries for the two new backend option functions. They are also declared after all of the other functions except destroy, which would explain why it only shows up when calling destroy (the vtable entries are in declaration order).

What I'm not clear on is why CoreML's backend class would be missing those. Core ML is built as a shared lib. Maybe they are getting optimized out in the CoreML lib build but not the core? It seems like the base class (BackendInterface) should have public visibility and thus this should not happen... so I'm not sure.

[guangy10]

Is this issue a release blocker? If so we should add it to the tracker

[metascroy]

Yeah, I think the two new functions get_option and set_option are the issue. I tried commenting those out and the segfault goes away. cc @cccclai

[metascroy]

Is this issue a release blocker? If so we should add it to the tracker

Yes, I think this is a release blocker if it affects 0.7.

[cccclai]

Okay let me try to repro

[metascroy]

Okay let me try to repro

@cccclai are you taking a look at this?