Skip to content

Add Python runtime end-to-end tests for XNNPACK on Linux#18703

Open
Lidang-Jiang wants to merge 3 commits intopytorch:mainfrom
Lidang-Jiang:feat/python-e2e-xnnpack
Open

Add Python runtime end-to-end tests for XNNPACK on Linux#18703
Lidang-Jiang wants to merge 3 commits intopytorch:mainfrom
Lidang-Jiang:feat/python-e2e-xnnpack

Conversation

@Lidang-Jiang
Copy link
Copy Markdown
Contributor

@Lidang-Jiang Lidang-Jiang commented Apr 6, 2026

Summary

Add Python runtime end-to-end tests for the XNNPACK backend on Linux — the first batch for the testing gap described in #11225.

This update keeps the Linux-only coverage under runtime/test/test_runtime_xnnpack.py so the standard unittest jobs collect it cleanly. Each test exports a small model with XNNPACK delegation, saves it as .pte, loads it via executorch.runtime.Runtime, and compares the output against PyTorch eager mode.

Models tested:

  • test_add — element-wise addition
  • test_linear — single Linear layer (fp32)
  • test_conv2d_reluConv2d + ReLU
  • test_mlp — multi-layer perceptron (Linear -> ReLU -> Linear)
  • test_depthwise_separable_conv — depthwise + pointwise conv (MobileNet building block)
  • test_classifier_headAdaptiveAvgPool2d + Flatten + Linear
  • test_conv_bnConv2d + BatchNorm2d + ReLU (skipped: FuseBatchNormPass crashes; TODO to re-enable)

Part of #11225 (Linux XNNPACK)

Before
$ pull / unittest-editable / linux / linux-job

==================================== ERRORS ====================================
_________ ERROR collecting test/end2end/test_python_runtime_xnnpack.py _________
ImportError while importing test module '/pytorch/executorch/test/end2end/test_python_runtime_xnnpack.py'.
Traceback:
E   ModuleNotFoundError: No module named 'test.end2end'

$ lintrunner --skip MYPY test/end2end/test_python_runtime_xnnpack.py

Warning (UFMT) format
- from executorch.backends.xnnpack.partition.xnnpack_partitioner import (
-     XnnpackPartitioner,
- )
+ from executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner
After
$ lintrunner --skip MYPY runtime/test/test_runtime_xnnpack.py
ok No lint issues.

$ python -m py_compile runtime/test/test_runtime_xnnpack.py

Test plan

  • Moved the Linux-only XNNPACK runtime tests to runtime/test/test_runtime_xnnpack.py
  • Added runtime/test/TARGETS coverage for the new Python runtime test
  • lintrunner --skip MYPY runtime/test/test_runtime_xnnpack.py
  • python -m py_compile runtime/test/test_runtime_xnnpack.py
  • test_conv_bn remains skipped with TODO — FuseBatchNormPass still has a known bug on Sequential(Conv2d, BN) export
  • Full python -m pytest runtime/test/test_runtime_xnnpack.py -v
    Local blocker: this workstation does not currently have a fully provisioned ExecuTorch runtime test environment; upstream CI remains the source of truth for this suite.

cc @GregoryComer @digantdesai @cbilgin @jathu @larryliu0820

This PR was authored with the help of Claude.

@pytorch-bot
Copy link
Copy Markdown

pytorch-bot bot commented Apr 6, 2026

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18703

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit 7e51933 with merge base ac68932 (image):

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 6, 2026
@Lidang-Jiang
Copy link
Copy Markdown
Contributor Author

@pytorchbot label "release notes: xnnpack"

@pytorch-bot pytorch-bot bot added the release notes: xnnpack Changes to the XNNPack backend delegate label Apr 6, 2026
@nil-is-all nil-is-all added the module: xnnpack Issues related to xnnpack delegation and the code under backends/xnnpack/ label Apr 7, 2026
Export small models (add, linear, conv2d+relu, MLP, depthwise separable
conv, classifier head) with XNNPACK delegation and verify correctness
through the Python runtime (executorch.runtime.Runtime).

This is the Linux XNNPACK portion of the Python runtime test gap
described in the issue.  Each test exports a model to .pte, loads it
via Runtime, and asserts that the output matches PyTorch eager mode.

Part of pytorch#11225

Signed-off-by: Lidang-Jiang <lidangjiang@gmail.com>
@Lidang-Jiang Lidang-Jiang force-pushed the feat/python-e2e-xnnpack branch from d27e33b to 931add3 Compare April 8, 2026 02:37
@GregoryComer
Copy link
Copy Markdown
Member

@Lidang-Jiang Can you resolve the lint - https://github.com/pytorch/executorch/actions/runs/24114688479/job/71164744422?pr=18703? I can go ahead and merge once CI is green. I ended up closing the original issue, as it's old, but adding more test coverage here doesn't hurt. Thanks!

- resolve the standard unittest collection failure
- fix the UFMT import formatting in the XNNPACK runtime test

Signed-off-by: Lidang-Jiang <lidangjiang@gmail.com>
@Lidang-Jiang
Copy link
Copy Markdown
Contributor Author

@GregoryComer Fixed in e1b05d5. I moved the Linux-only XNNPACK Python runtime tests into runtime/test so they collect cleanly in the standard unittest jobs, and I fixed the UFMT lint issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. module: xnnpack Issues related to xnnpack delegation and the code under backends/xnnpack/ release notes: xnnpack Changes to the XNNPack backend delegate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants