Add per-layer MLP type support for executorch export (#18856)#18856
Open
navsud wants to merge 1 commit intopytorch:mainfrom
Open
Add per-layer MLP type support for executorch export (#18856)#18856navsud wants to merge 1 commit intopytorch:mainfrom
navsud wants to merge 1 commit intopytorch:mainfrom
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18856
Note: Links to docs will display an error until the docs builds have been completed. ❌ 3 New Failures, 2 Unrelated FailuresAs of commit e2225f8 with merge base eaef2ed ( NEW FAILURES - The following jobs have failed:
FLAKY - The following jobs failed but were likely due to flakiness present on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
bot
added
the
CLA Signed
Contributor
|
@navsud has exported this pull request. If you are a Meta employee, you can view the originating Diff in D100682545. |
navsud
changed the title
meta-codesync
bot
changed the title
navsud
added a commit
to navsud/executorch
that referenced
this pull request
Apr 13, 2026
Summary: Add per-layer MLP type support to the ExecuTorch export path. This allows hybrid models to configure FFN blocks per layer (e.g. skip FFN on specified layers), reducing model size and inference latency. The per-layer config uses an mlp_type list in ModelArgs, where each layer can be set to "default" (standard FFN) or "skip" (no FFN block). This is extensible to future MLP types. - Add mlp_type field to ModelArgs (model_args.py) — optional list of strings, one per layer - Update TransformerBlock.__init__ to accept mlp_type string and skip FFN/ffn_norm creation when mlp_type == "skip" (llama_transformer.py) - Update TransformerBlock.from_type() to read mlp_type from ModelArgs per layer - Update TransformerBlock.forward() to pass through attention output directly when mlp_type == "skip" Reviewed By: ifed-ucsd Differential Revision: D100682545
navsud
force-pushed
the
export-D100682545
branch
from
a341ca0 to
04faf26
Compare
navsud
added a commit
to navsud/executorch
that referenced
this pull request
Apr 13, 2026
Summary: Pull Request resolved: pytorch#18856 Add per-layer MLP type support to the ExecuTorch export path. This allows hybrid models to configure FFN blocks per layer (e.g. skip FFN on specified layers), reducing model size and inference latency. The per-layer config uses an mlp_type list in ModelArgs, where each layer can be set to "default" (standard FFN) or "skip" (no FFN block). This is extensible to future MLP types. - Add mlp_type field to ModelArgs (model_args.py) — optional list of strings, one per layer - Update TransformerBlock.__init__ to accept mlp_type string and skip FFN/ffn_norm creation when mlp_type == "skip" (llama_transformer.py) - Update TransformerBlock.from_type() to read mlp_type from ModelArgs per layer - Update TransformerBlock.forward() to pass through attention output directly when mlp_type == "skip" Reviewed By: ifed-ucsd Differential Revision: D100682545
navsud
force-pushed
the
export-D100682545
branch
2 times, most recently
from
78d8419 to
9ce06a5
Compare
navsud
added a commit
to navsud/executorch
that referenced
this pull request
Apr 15, 2026
Summary: Add per-layer MLP type support to the ExecuTorch export path. This allows hybrid models to configure FFN blocks per layer (e.g. skip FFN on specified layers), reducing model size and inference latency. The per-layer config uses an mlp_type list in ModelArgs, where each layer can be set to "default" (standard FFN) or "skip" (no FFN block). This is extensible to future MLP types. - Add mlp_type field to ModelArgs (model_args.py) — optional list of strings, one per layer - Update TransformerBlock.__init__ to accept mlp_type string and skip FFN/ffn_norm creation when mlp_type == "skip" (llama_transformer.py) - Update TransformerBlock.from_type() to read mlp_type from ModelArgs per layer - Update TransformerBlock.forward() to pass through attention output directly when mlp_type == "skip" Reviewed By: ifed-ucsd Differential Revision: D100682545
Summary: Add per-layer MLP type support to the ExecuTorch export path. This allows hybrid models to configure FFN blocks per layer (e.g. skip FFN on specified layers), reducing model size and inference latency. The per-layer config uses an mlp_type list in ModelArgs, where each layer can be set to "default" (standard FFN) or "skip" (no FFN block). This is extensible to future MLP types. - Add mlp_type field to ModelArgs (model_args.py) — optional list of strings, one per layer - Update TransformerBlock.__init__ to accept mlp_type string and skip FFN/ffn_norm creation when mlp_type == "skip" (llama_transformer.py) - Update TransformerBlock.from_type() to read mlp_type from ModelArgs per layer - Update TransformerBlock.forward() to pass through attention output directly when mlp_type == "skip" Reviewed By: ifed-ucsd Differential Revision: D100682545
navsud
force-pushed
the
export-D100682545
branch
from
9ce06a5 to
e2225f8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Add per-layer MLP type support to the ExecuTorch export path. This allows hybrid models to configure FFN blocks per layer (e.g. skip FFN on specified layers), reducing model size and inference latency.
The per-layer config uses an mlp_type list in ModelArgs, where each layer can be set to "default" (standard FFN) or "skip" (no FFN block). This is extensible to future MLP types.
Reviewed By: ifed-ucsd
Differential Revision: D100682545