Skip to content

Commit c0a6d88

Browse files
consensus: log waiting-pool reward computation per cycle (diagnostic only)
Follow-up to the accounts-fingerprint / APPLY_REVERTED diagnostics. Bisecting the retained fingerprint logs found the true origin of the 2026-07-26 bal divergence at cycle ~89253580-81 (not 89254196, already ~700 cycles downstream), and that asia's own SELF_PRODUCED_MISMATCH detection fires at that exact cycle -- immediately followed by a recurring cluster of refused self-produced applies and skipped peer-synced LSUs (prev_root mismatch) over the following ~100 cycles, matching where the measured lag jumped from ~10 to ~28 cycles-equivalent. asia's root_counter fingerprint matches eu's exactly throughout, ruling out a wrong-cycle bug -- this is isolated to reward accumulation specifically. distribute_waiting_pool_rewards_and_fee_credits now logs, on every cycle: computed waiting_pool, eligible-worker count, per-worker share and remainder, and short fingerprints of the eligible-waiting and producer pubkey lists (same 4-byte SHA-256 convention as the accounts fingerprint). Also logs the two early-return cases explicitly (zero pool, and nonzero pool with zero eligible workers -- a reward silently forfeited that cycle). This lets a future occurrence show directly whether the computed payout itself differs between nodes, or whether it matches and the divergence is in the starting balance instead. No behavior change, full test suite passes (96/96).
1 parent 16eacb1 commit c0a6d88

1 file changed

Lines changed: 31 additions & 0 deletions

File tree

crates/catalyst-cli/src/node.rs

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2506,6 +2506,18 @@ async fn waiting_worker_is_eligible(
25062506
true
25072507
}
25082508

2509+
/// Short (4-byte) SHA-256 fingerprint of a sorted list of pubkeys, for compact cross-node log
2510+
/// comparison (see `log_accounts_fingerprint_breakdown` for the same convention).
2511+
fn pubkey_list_fingerprint(pks: &[[u8; 32]]) -> String {
2512+
use sha2::{Digest, Sha256};
2513+
let mut h = Sha256::new();
2514+
for pk in pks {
2515+
h.update(pk);
2516+
}
2517+
let full: [u8; 32] = h.finalize().into();
2518+
hex_encode(&full[..4])
2519+
}
2520+
25092521
async fn distribute_waiting_pool_rewards_and_fee_credits(
25102522
store: &StorageManager,
25112523
lsu: &catalyst_consensus::types::LedgerStateUpdate,
@@ -2525,12 +2537,17 @@ async fn distribute_waiting_pool_rewards_and_fee_credits(
25252537
let paid_to_producers: u64 = lsu.compensation_entries.iter().map(|e| e.amount).sum();
25262538
let waiting_pool = waiting_pool_from_formula.min(total_reward_pool.saturating_sub(paid_to_producers));
25272539
if waiting_pool == 0 && !TOKENOMICS_FEE_CREDITS_ENABLED {
2540+
info!(
2541+
"Cycle {} waiting_pool_reward: waiting_pool=0 total_fees={} paid_to_producers={} fee_credits_enabled=false -- nothing to distribute",
2542+
lsu.cycle_number, total_fees, paid_to_producers
2543+
);
25282544
return;
25292545
}
25302546

25312547
let mut workers = load_workers_from_state(store);
25322548
workers.sort();
25332549
workers.dedup();
2550+
let producers_hash = pubkey_list_fingerprint(&producers.iter().copied().collect::<Vec<_>>());
25342551
let mut waiting: Vec<[u8; 32]> = Vec::new();
25352552
for pk in workers {
25362553
if producers.contains(&pk) {
@@ -2541,12 +2558,26 @@ async fn distribute_waiting_pool_rewards_and_fee_credits(
25412558
}
25422559
}
25432560
if waiting.is_empty() {
2561+
// Diagnostic (2026-07-26 asia bal-lag investigation): if `waiting_pool` above is
2562+
// nonzero but no worker is eligible, that reward is silently forfeited this cycle
2563+
// (never distributed, never carried forward) -- a plausible source of a permanent,
2564+
// one-node-only balance shortfall if eligibility computation ever disagrees between
2565+
// nodes despite an otherwise-identical LSU and identical wfs:/wlr: state.
2566+
info!(
2567+
"Cycle {} waiting_pool_reward: waiting_pool={} but n_eligible=0 (forfeited, not carried forward) producers_hash={}",
2568+
lsu.cycle_number, waiting_pool, producers_hash
2569+
);
25442570
return;
25452571
}
25462572

25472573
let n = waiting.len() as u64;
25482574
let per = waiting_pool / n;
25492575
let mut rem = waiting_pool % n;
2576+
info!(
2577+
"Cycle {} waiting_pool_reward: waiting_pool={} n_eligible={} per={} rem={} waiting_hash={} producers_hash={}",
2578+
lsu.cycle_number, waiting_pool, n, per, rem,
2579+
pubkey_list_fingerprint(&waiting), producers_hash
2580+
);
25502581
for pk in waiting {
25512582
if per > 0 || rem > 0 {
25522583
let bonus = if rem > 0 {

0 commit comments

Comments
 (0)