1877 persisted da in blockasm by bewakes · Pull Request #1474 · alpenlabs/alpen

bewakes · 2026-03-09T10:01:41Z

Description

This is a draft implementation of one of the DA data generation designs proposed in https://www.notion.so/DA-data-generation-Design-319901ba000f80b5b2e4fdad48416bd1.

The following things are done in this PR:

Update BlockAssembly context trait to include fetch_accumulated_da and store_accumulated_da methods.
Wrap WriteTrackingState in block assembly by DaAccumulatingState.
Derive Codec for EpochDaAccumulator which has been cascaded into subsequent internal types. This Codec derivation is for storing the accumulator in the database and use it for subsequent block processing.

Todo

To make the intended functionality complete, the following need to be done:

Storage definition for accumulator.
Codec/serialization tests.
Update block assembly tests.
Implement the new methods in the block assembly context.
Spawn a thin cleanup service to prune accumulated DA data after an epoch is finalized.

Type of Change

Bug fix (non-breaking change which fixes an issue)
New feature/Enhancement (non-breaking change which adds functionality or enhances an existing one)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update
Refactor
New or updated tests
Dependency Update

Notes to Reviewers

Is this PR addressing any specification, design doc or external reference document?

Yes
No

If yes, please add relevant links:

Checklist

I have performed a self-review of my code.
I have commented my code where necessary.
I have updated the documentation if needed.
My changes do not introduce new warnings.
I have added (where necessary) tests that prove my changes are effective or that my feature works.
New and existing tests pass with my changes.
I have disclosed my use of AI in the body of this PR.

Related Issues

Add AccumulatedDaData type to hold state diffs and logs accumulated during block processing within an epoch. This structure will be used to track DA data incrementally as blocks are assembled.

…AnchorContext Add fetch_accumulated_da and store_accumulated_da methods to the BlockAssemblyAnchorContext trait. These will be used to retrieve and persist accumulated DA data for blocks during assembly. Implementations are stubbed for now and will be connected to actual storage in a future commit.

…ruction Add basic DA accumulation logic to construct_block that: - Fetches accumulated DA from parent block - Detects epoch boundaries and resets accumulation - Accumulates logs from the current block - Stores accumulated DA data for the newly constructed block Note: State diff accumulation is left as TODO for future implementation once we figure out how to maintain DaAccumulatingState across blocks.

github-actions · 2026-03-09T10:34:03Z

Commit: cfa3667

SP1 Execution Results

program	cycles	success
EVM EE STF	1,320,664	✅
CL STF	905,862	✅
Checkpoint	2,249	✅
Checkpoint New	885,216	✅

delbonis

Also, is "accumulator" really a good name here? It seems like "builder" is reasonably accurate and it matches how the other DA types are named.

delbonis · 2026-03-09T16:28:51Z

crates/ol/state-support-types/src/da_accumulating_layer.rs

 /// Minimal tracking data for a newly created account.
 #[derive(Clone, Debug)]
-struct NewAccountRecord {
+pub struct NewAccountRecord {
    serial: AccountSerial,
    account_id: AccountId,
 }

+impl Codec for NewAccountRecord {
+    fn encode(&self, enc: &mut impl Encoder) -> Result<(), CodecError> {
+        self.serial.encode(enc)?;
+        self.account_id.encode(enc)?;
+        Ok(())
+    }
+
+    fn decode(dec: &mut impl Decoder) -> Result<Self, CodecError> {
+        let serial = AccountSerial::decode(dec)?;
+        let account_id = AccountId::decode(dec)?;
+        Ok(Self { serial, account_id })
+    }
+}


"serial" is redundant.

delbonis · 2026-03-09T16:29:26Z

crates/ol/state-support-types/src/da_accumulating_layer.rs

+        // For DaCounterBuilder, recreate it preserving both original and new values
+        let slot_builder = self.slot_builder.as_ref().map(|builder| {
+            let mut new_builder =
+                DaCounterBuilder::<CtrU64ByU16>::from_source(*builder.original_value());
+            // Set the new value to match the current state
+            let _ = new_builder.set(*builder.new_value());
+            new_builder
+        });


Surely we can clone it directly?

No, DaCounterBuilder cannot be cloned as it doesn't implement Clone ( requires Clone for type Base).

I don't think I had a reason specifically for excluding that, I would think it's safe to add it.

delbonis · 2026-03-09T16:30:28Z

crates/ol/block-assembly/src/da_accumulation.rs

+        // For now, we'll skip encoding logs since OLLog might not have Codec
+        // In a real implementation, you'd need to ensure OLLog implements Codec
+        // or serialize them differently


Wrap it in the CodecSsz shim in the type so this can just use the codec derive macro. It's #[repr(transparent)] so it can be safely mem::transmuted.

delbonis · 2026-03-09T16:32:34Z

crates/ol/block-assembly/src/da_accumulation.rs

+/// Accumulated DA data for a block within an epoch.
+///
+/// Contains both the epoch accumulator state and the logs
+/// generated up to this point in the epoch.
+#[derive(Debug)]
+pub struct AccumulatedDaData {
+    /// The epoch this accumulation belongs to.
+    epoch: Epoch,
+
+    /// The epoch DA accumulator that tracks state changes.
+    /// This is serialized and persisted between blocks.
+    accumulator: EpochDaAccumulator,
+
+    /// All logs emitted in the epoch up to and including this block.
+    logs: Vec<OLLog>,
+}


This naming is misleading since it includes logs, which are not part of the DA diff data that EpochDaAccumulator contains.

delbonis · 2026-03-09T16:32:43Z

crates/ol/state-support-types/src/lib.rs


 mod batch_diff_layer;
-mod da_accumulating_layer;
+pub mod da_accumulating_layer;


This doesn't need to be pub.

delbonis · 2026-03-09T16:34:44Z

crates/ol/state-support-types/src/da_accumulating_layer.rs

+    /// Takes ownership of the accumulator, replacing it with a default one.
+    /// This is useful for extracting the accumulator for serialization.
+    pub fn take_accumulator(&mut self) -> EpochDaAccumulator {
+        mem::take(&mut self.epoch_acc)
+    }


This is a C++-ish pattern. This should be into_accumulator or have the type defined around a &mut EpochDaAccumulator in the first place so that the consumer can manage it without keeping the layer around across STF invocations or needing to juggle around the EpochDaAccumulator around.

storopoli

From my understanding, the design this PR is using is combination 1 from the Notion page: accumulate in block assembly, persist by block, and do not enforce DA limits. Is this the intended design?

storopoli · 2026-03-20T14:39:46Z

crates/ol/block-assembly/src/block_assembly.rs

+    } else {
+        ctx.fetch_accumulated_da(parent_commitment)
+            .await?
+            .ok_or(BlockAssemblyError::Other(


This turns the TODO stub in BlockAssemblyContext into a hard runtime failure for every non-terminal child block. fetch_accumulated_da() still returns Ok(None) and store_accumulated_da() is still a no-op in the default context

alpen/crates/ol/block-assembly/src/context.rs

Lines 297 to 312 in c552ce9

async fn fetch_accumulated_da(

&self,

_commitment: OLBlockCommitment,

) -> BlockAssemblyResult<Option<AccumulatedDaData>> {

// TODO: Implement actual storage retrieval once storage layer is ready

// For now, return None to indicate no DA data exists

Ok(None)

}

async fn store_accumulated_da(

&self,

_commitment: OLBlockCommitment,

_da_data: AccumulatedDaData,

) -> BlockAssemblyResult<()> {

// TODO: Implement actual storage once storage layer is ready

Ok(())

so the second block in an epoch now errors with Other(Cannot find expected da data) instead of continuing to build. This path should stay non-fatal or be guarded behind a real persistence implementation.

storopoli · 2026-03-20T14:39:46Z

crates/ol/block-assembly/src/da_accumulation.rs

+    fn encode(&self, enc: &mut impl Encoder) -> Result<(), CodecError> {
+        self.epoch.encode(enc)?;
+        self.accumulator.encode(enc)?;
+        // For now, we'll skip encoding logs since OLLog might not have Codec


This persisted format is lossy: encode() drops logs, and decode() reconstructs them as empty. But block assembly explicitly carries the prior epoch logs forward via into_parts() and appends the current block logs before re-storing

alpen/crates/ol/block-assembly/src/block_assembly.rs

Lines 277 to 317 in c552ce9

let (accumulator, mut accum_logs) = accumulated_da.into_parts();

// Phase 2: Process each transaction against accumulated state using AccumulatorProofGenerator.

let ProcessTransactionsOutput {

successful_txs,

failed_txs,

accumulated_batch,

accumulated_da: final_accumulator,

} = process_transactions(

ctx,

&block_context,

&output_buffer,

parent_state.as_ref(),

accumulated_batch,

mempool_txs,

accumulator,

);

// Phase 3: Detect terminal blocks and fetch L1 manifests if needed.

debug!(%block_slot, "Calling should seal_epoch");

let manifest_container = if epoch_sealing_policy.should_seal_epoch(block_slot) {

debug!(%block_slot, "Calling should seal_epoch returned true");

fetch_asm_manifests_for_terminal_block(ctx, parent_state.as_ref()).await?

} else {

debug!(%block_slot, "Calling should seal_epoch returned false");

None

};

// Phase 4: Finalize block construction.

let (template, post_state) = build_block_template(

config,

&block_context,

&parent_state,

accumulated_batch,

output_buffer.clone(),

successful_txs,

manifest_container,

)?;

// Append this block's logs to the accumulated logs

accum_logs.extend_from_slice(&output_buffer.into_logs());

let accumulated_da = AccumulatedDaData::new(block_epoch, final_accumulator, accum_logs);

Once fetch/store_accumulated_da() is backed by real storage, reloading any accumulated value will silently discard the earlier withdrawal and log history for the epoch.

bewakes added 4 commits March 8, 2026 10:56

feat(ol-block-assembly): define AccumulatedDaData structure

c7e8589

Add AccumulatedDaData type to hold state diffs and logs accumulated during block processing within an epoch. This structure will be used to track DA data incrementally as blocks are assembled.

Implement Codec for EpochDaAccumulator

c552ce9

delbonis requested changes Mar 9, 2026

View reviewed changes

storopoli requested changes Mar 20, 2026

View reviewed changes

	async fn fetch_accumulated_da(
	&self,
	_commitment: OLBlockCommitment,
	) -> BlockAssemblyResult<Option<AccumulatedDaData>> {
	// TODO: Implement actual storage retrieval once storage layer is ready
	// For now, return None to indicate no DA data exists
	Ok(None)
	}

	async fn store_accumulated_da(
	&self,
	_commitment: OLBlockCommitment,
	_da_data: AccumulatedDaData,
	) -> BlockAssemblyResult<()> {
	// TODO: Implement actual storage once storage layer is ready
	Ok(())

	let (accumulator, mut accum_logs) = accumulated_da.into_parts();
	// Phase 2: Process each transaction against accumulated state using AccumulatorProofGenerator.
	let ProcessTransactionsOutput {
	successful_txs,
	failed_txs,
	accumulated_batch,
	accumulated_da: final_accumulator,
	} = process_transactions(
	ctx,
	&block_context,
	&output_buffer,
	parent_state.as_ref(),
	accumulated_batch,
	mempool_txs,
	accumulator,
	);

	// Phase 3: Detect terminal blocks and fetch L1 manifests if needed.
	debug!(%block_slot, "Calling should seal_epoch");
	let manifest_container = if epoch_sealing_policy.should_seal_epoch(block_slot) {
	debug!(%block_slot, "Calling should seal_epoch returned true");
	fetch_asm_manifests_for_terminal_block(ctx, parent_state.as_ref()).await?
	} else {
	debug!(%block_slot, "Calling should seal_epoch returned false");
	None
	};

	// Phase 4: Finalize block construction.
	let (template, post_state) = build_block_template(
	config,
	&block_context,
	&parent_state,
	accumulated_batch,
	output_buffer.clone(),
	successful_txs,
	manifest_container,
	)?;

	// Append this block's logs to the accumulated logs
	accum_logs.extend_from_slice(&output_buffer.into_logs());
	let accumulated_da = AccumulatedDaData::new(block_epoch, final_accumulator, accum_logs);

Conversation

bewakes commented Mar 9, 2026

Description

Todo

Type of Change

Notes to Reviewers

Checklist

Related Issues

Uh oh!

github-actions bot commented Mar 9, 2026

Uh oh!

delbonis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

storopoli left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants