Track nested memory used by `xcm::DoubleEncoded`: approach #1 by serban300 · Pull Request #10747 · paritytech/polkadot-sdk

serban300 · 2026-01-08T08:30:00Z

Resolves #8675

The PR introduces a global variable that limits the amount of heap memory available for decoding structures within nested contexts.

bkchr

I find the current approach hard to follow and it also doesn't forward the left over limit from decoding the extrinsic.

E.g. when executing pallet_xcm::execute it should only leave the rest from the budget for Transact and not give it the full budget again. Also the budgeting right now maybe shares too much. E.g. when I want to load two XCM messages from the storage, they will both use the same limit. This is probably not wanted. We should provide some better "tools" around the bounded limit.

struct DecodeWithLimit {
       limit: usize,
       decoded: T,
}

impl Decode using DecodeWithLimitThingy {}

fn DecodeWithLimit {
     pub use(use: Impl(&mut T) -> R) -> R {
              mem::run_with(self.limit, || use(&mut self.decoded))
     }
}

Maybe something like this? Then we could use this to load XCM messages from storage?

bkchr · 2026-01-29T20:51:48Z

+//!
+//! There are also cases where there can be multiple nested double-encoded layers.
+
+use crate::generic::DEFAULT_CALL_SIZE_LIMIT;


If we want that to be generic, we should pass the limit as an input value and not use this constant.

bkchr · 2026-01-29T21:04:18Z

-								"Dispatch failed"
-							);
+			Transact { origin_kind, call, .. } => {
+				sp_runtime::nested_mem::using_limiter_once(|| {


The problem here is that we will run here without knowing how much the outer extrinsic already used for decoding.

We can find out the remaining mem limit by calling sp_runtime::nested_mem::get_current_limit()

bkchr · 2026-01-29T21:28:31Z

-			signature_check,
-			select,
-		);
+		nested_mem::using_limiter_once(|| {


The scope is way too broad here. We only need this for extrinsic applying.

We also need it for on_idle(). Because the XCMs received from other chains are being executed in the on_idle() hook of the pallet-message-queue.

I wasn't sure if we needed it for other hooks as well and I thought that it would be safer to execute the entire block inside using_limiter_once().

Anyway, trying to find a better approach overall.

serban300

E.g. when executing pallet_xcm::execute it should only leave the rest from the budget for Transact and not give it the full budget again.

You mean that when we execute and extrinsic, we should subtract the mem used for decoding the extrinsic from the budget first, and then use only the rest for Transact ? The problem is that this can lead to inconsistencies. For example:

dry running an XCM can succeed, because it's not ran as part of an extrinsic, but if we run it via pallet_xcm::execute() it can fail, because of the extra extrinsic size.
an XCM can succeed when running it in on_idle and fail if we run it via pallet_xcm::execute()
an XCM can succeed in an extrinsic and fail in a different one

Also the budgeting right now maybe shares too much. E.g. when I want to load two XCM messages from the storage, they will both use the same limit. This is probably not wanted.

When simply loading 2 XCMs from the storage, the nested mem budget is not used. Only when we execute them and we decode the Transact instructions we end up using the nested mem budget. But:

normally the first XCM should free the memory after it is executed, because try_using_decoded() consumes the object. So normally we don't need to use a lower limit for the second XCM. There is a problem when using ensure_decoded(), because this doesn't consume the object. But if the entire block is executed inside nested_mem::using_limiter_once, this should also be covered.
I think it would be problematic if the successful execution of an XCM would depend on the memory used by the other XCMs that were executed before it. It would lead to inconsistencies depending on the order in which the XCMs are run

I find the current approach hard to follow and it also doesn't forward the left over limit from decoding the extrinsic.

I agree this approach with global variables is hard to follow and has some limitations. Maybe we can use a storage value instead. Maybe this could make it more readable and more flexible. I need to experiment a bit. Will try to change the approach.

serban300 · 2026-02-13T13:29:55Z

-			signature_check,
-			select,
-		);
+		nested_mem::using_limiter_once(|| {


We also need it for on_idle(). Because the XCMs received from other chains are being executed in the on_idle() hook of the pallet-message-queue.

I wasn't sure if we needed it for other hooks as well and I thought that it would be safer to execute the entire block inside using_limiter_once().

Anyway, trying to find a better approach overall.

serban300 · 2026-02-16T13:46:33Z

-								"Dispatch failed"
-							);
+			Transact { origin_kind, call, .. } => {
+				sp_runtime::nested_mem::using_limiter_once(|| {


We can find out the remaining mem limit by calling sp_runtime::nested_mem::get_current_limit()

serban300 · 2026-02-19T14:20:08Z

Another option would be to try to recursively decode the entire XCM, together with the double encoded calls. When we do this as part of an extrinsic, it will be limited by the extrinsic heap limit. As part of the XCMP/UMP/DMP queue, we can use a custom limit.

Will experiment with this as well.

serban300 · 2026-02-23T11:21:34Z

Closing in favor of #11147

…tech#11147) Resolves paritytech#8675 Follow-up for paritytech#10747 (approach # 1) Different approach for tracking the nested memory used by `xcm::DoubleEncoded` The idea is to decode the entire `xcm::DoubleEncoded<Call>` when possible. This can be done when the `Call` is the `RuntimeCall` of our local Runtime. When the `Call` is `()` it means that it's a remote `RuntimeCall`, and it will be decoded on the destination chain. This covers the XCMP/UMP/DMP pathways and the `pallet_xcm::execute()` extrinsic or any other extrinsic that accepts XCMs. For extrinsics we have the heap memory limit used for decoding the extrinsic (16 MB) and for XCMP/UMP/DMP we call `decode_with_constraints()` which uses a hardcoded 10MB limit now. We can adjust it or make it configurable. I hope I'm not missing any use cases. Known issues (to be fixed in future PRs): - dry-running: The methods defined in `RecordXcm` return `VersionedXcm<()>` directly. We should add the `RuntimeCall` as a generic type param (`RecordXcm<Call>`) and modify all the methods to return `VersionedXcm<Call>` - `DoubleEncoded::transmute_encoded()` is used in some mocks. Not sure if it's needed outside of mocks. Maybe we can remove it with some cleanup. --------- Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>

serban300 self-assigned this Jan 8, 2026

serban300 requested review from a team as code owners January 8, 2026 08:30

serban300 added the T6-XCM This PR/Issue is related to XCM. label Jan 8, 2026

paritytech-review-bot Bot requested a review from a team January 8, 2026 08:30

serban300 force-pushed the limit-memory-for-nested-extrinsics branch 2 times, most recently from 2bf2683 to 351c175 Compare January 8, 2026 09:20

serban300 force-pushed the limit-memory-for-nested-extrinsics branch 2 times, most recently from b4ba120 to 59bccbe Compare January 19, 2026 07:02

serban300 added 3 commits January 29, 2026 13:40

MAX_CALL_SIZE -> CALL_SIZE_LIMIT

bbefef3

DoubleEncoded: simplifications

1b5e463

Track nested memory used by xcm::DoubleEncoded

ea4fbeb

serban300 force-pushed the limit-memory-for-nested-extrinsics branch from 59bccbe to ea4fbeb Compare January 29, 2026 11:46

serban300 changed the title ~~[WIP] Track nested memory used by xcm::DoubleEncoded~~ Track nested memory used by xcm::DoubleEncoded Jan 29, 2026

bkchr reviewed Jan 29, 2026

View reviewed changes

serban300 commented Feb 16, 2026

View reviewed changes

serban300 mentioned this pull request Feb 23, 2026

Track nested memory used by xcm::DoubleEncoded: approach #2 #11147

Merged

serban300 changed the title ~~Track nested memory used by xcm::DoubleEncoded~~ Approach 1: Track nested memory used by xcm::DoubleEncoded Feb 23, 2026

serban300 mentioned this pull request Feb 23, 2026

[DecodeWithMemTracking] Research keeping track of all the nested double encoded structs #8675

Closed

serban300 changed the title ~~Approach 1: Track nested memory used by xcm::DoubleEncoded~~ Track nested memory used by xcm::DoubleEncoded: approach #1 Feb 23, 2026

serban300 closed this Feb 23, 2026

Navigation Menu

Uh oh!

Track nested memory used by `xcm::DoubleEncoded`: approach #1 - #10747

Track nested memory used by `xcm::DoubleEncoded`: approach #1#10747
serban300 wants to merge 3 commits into
paritytech:masterfrom
serban300:limit-memory-for-nested-extrinsics

serban300 commented Jan 8, 2026 •

edited

Loading

Uh oh!

bkchr left a comment

Uh oh!

bkchr Jan 29, 2026

Uh oh!

bkchr Jan 29, 2026

Uh oh!

serban300 Feb 16, 2026

Uh oh!

bkchr Jan 29, 2026

Uh oh!

serban300 Feb 13, 2026

Uh oh!

serban300 left a comment

Uh oh!

serban300 Feb 13, 2026

Uh oh!

serban300 Feb 16, 2026

Uh oh!

serban300 commented Feb 19, 2026 •

edited

Loading

Uh oh!

serban300 commented Feb 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

serban300 commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bkchr left a comment

Choose a reason for hiding this comment

Uh oh!

bkchr Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

bkchr Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

serban300 Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

bkchr Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

serban300 Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

serban300 left a comment

Choose a reason for hiding this comment

Uh oh!

serban300 Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

serban300 Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

serban300 commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serban300 commented Feb 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

serban300 commented Jan 8, 2026 •

edited

Loading

serban300 commented Feb 19, 2026 •

edited

Loading