rhdl_dsig_scalable_core/

chain_step.rs

//! A Haraka lane specialised to exactly one DSig chain step.
//!
//! # The problem
//!
//! A DSig key is 68 W-OTS+ chains (`w = 4`). Making a key walks every chain
//! from its secret node three steps to its public end, and every step is one
//! Haraka-512 hash of a 64-byte *chain block*. That is 204 chain steps per
//! signature. The signer wants **one chain step per clock per lane** at
//! 205 MHz, so the chain-step lane is replicated many times, and its size
//! decides how many signers fit on the chip.
//!
//! Most of a lane's area is AES S-boxes. A plain Haraka-512 has 640 of them
//! (10 AES rounds × 64 bytes). This module asks: *how many of those 640 does a
//! DSig chain step really need?* The answer is 494, and this file is the
//! hardware that uses only those.
//!
//! # The reasoning, step by step
//!
//! 1. The chain block is mostly fixed. Only the 16-byte salted nonce and the
//!    18-byte node change; bytes 34..64 are always zero.
//! 2. An S-box whose input is a known constant has a known constant output.
//!    It needs no hardware at all.
//! 3. Haraka's AES lanes do not talk to each other until the first word mix,
//!    after the *second* AES round. So the constants survive for two rounds.
//! 4. DSig keeps only 18 of the 32 digest bytes. Any S-box that cannot reach
//!    those 18 bytes is wasted work.
//!
//! The decision: compute, at compile time, which bytes are constant (going
//! forwards) and which bytes are needed (going backwards), then build a
//! pipeline in which the constant bytes are literal values. Synthesis does
//! the rest: it deletes logic that drives nothing and folds logic whose
//! inputs are constants.
//!
//! # How the constants fold away
//!
//! ```text
//!   live = varies per chain step     · = compile-time constant
//!
//!   AES lane:          0                 1                 2               3
//!   block bytes:     0..16            16..32            32..48          48..64
//!   round 0 in:   live×16           live×16        live×2 (32,33)    ·×16
//!                 (salted nonce)    (node 0..16)   ·×14              (all zero)
//!                    │                 │                 │               │
//!   rounds 0-1:   all S-boxes       all S-boxes      only the S-boxes   every S-box
//!                 live              live             fed by bytes 32,   constant:
//!                                                    33 and what they   32 boxes fold
//!                                                    spread to
//!                    └─────────────────┴─── word MIX after round 1 ──┴───────────────┘
//!   round 2:      the mix has moved lane 3's still-constant words into other
//!                 lanes; S-boxes on those 16 bytes still fold
//!   rounds 3-6:   every byte varies and every byte is needed
//!   rounds 7-8:   working backwards from the output, one lane's worth of
//!                 bytes (16) can no longer reach the 18 kept bytes
//!   round 9:      only the columns that feed the 18 kept bytes matter:
//!                 5 of 16 columns × 4 bytes = 20 of 64 S-boxes
//!                    │
//!   output:       18 bytes = mid_haraka512(block)
//!
//!   live S-boxes per AES round (constant ones folded, unobserved ones deleted):
//!
//!     round:   0    1    2    3    4    5    6    7    8    9    total
//!     live:   34   40   48   64   64   64   64   48   48   20  =  494   (of 640)
//! ```
//!
//! The per-round detail is not written down by hand in the code:
//! [`LIVE_SBOX`] records it, [`live_sbox_evaluations`] counts it, and a test
//! pins the total of 494. The per-round row above is that same analysis
//! re-run by hand for this explanation; only the total is checked by a test.
//!
//! # Why a second Haraka pipeline?
//!
//! [`Haraka512Pipeline`](crate::haraka::Haraka512Pipeline) hashes *any* 64-byte
//! block and returns the full 32-byte digest. DSig never asks for that. Every
//! chain step hashes a block with a fixed shape
//! ([`dsig_chain_block`](crate::haraka::dsig_chain_block)) and keeps only 18
//! bytes of the answer:
//!
//! ```text
//!  byte  0      16        34                                   63
//!        ├──────┼─────────┼────────────────────────────────────┤
//!  block │nonce │ secret  │        thirty zero bytes           │
//!        └──────┴─────────┴────────────────────────────────────┘
//!         AES lane 0 = bytes 0..16      AES lane 2 = bytes 32..48
//!         AES lane 1 = bytes 16..32     AES lane 3 = bytes 48..64 (all zero)
//! ```
//!
//! Haraka runs four independent AES lanes and only mixes them after every
//! *second* AES round. So for the first two rounds AES lane 3 sees nothing but
//! zeros: its 32 S-box evaluations always produce the same bytes. Lane 2 only
//! has two live bytes, so most of its first-round S-boxes are also constant.
//! At the far end, the 18 kept output bytes depend on only five of the sixteen
//! 32-bit columns, so most of the final round is never observed.
//!
//! This module turns those facts into hardware savings:
//!
//! 1. A `const fn` ([`KNOWN`]) symbolically runs the pipeline over "known
//!    constant" / "live" bytes and records, for every stage, which state bytes
//!    are compile-time constants and what their values are.
//! 2. The RHDL kernel pins those bytes to literals before each stage. Synthesis
//!    then folds their S-boxes into constants and deletes the registers that
//!    would have carried them.
//! 3. The lane outputs only the 18 DSig bytes, so synthesis also deletes every
//!    S-box whose result can never reach an output.
//!
//! Counted by [`live_sbox_evaluations`], one chain step needs **494** S-box
//! evaluations instead of Haraka's 640, about 23% less of the dominant area.
//! The count is a structural planning number. Only an implemented-design report
//! measures LUTs (see the validation ladder in the [crate documentation](crate)).
//!
//! # Where this sits
//!
//! ```text
//!  ChainEngine ── three steps in series (depth 0 → 1 → 2) ──▶ four cache nodes
//!       │
//!       └── each step is one ChainStepLane: 20 registered stages, II = 1
//! ```
//!
//! "ChainEngine" is [`SliceChainEngine`](crate::slice_chain::SliceChainEngine)
//! in [`crate::slice_chain`]. "II = 1" (initiation interval of one) means a
//! new token can enter every clock. Which chain hash is compiled in, and so
//! whether this lane is used at all, is decided in [`crate::chain_lane`].
//!
//! The neighbours this file leans on:
//!
//! | Module | What it gives this file |
//! |---|---|
//! | [`crate::haraka`] | round keys, word mix, truncation, `mix_key512_kernel`, the [`dsig_chain`](crate::haraka::dsig_chain) oracle |
//! | [`crate::aes_table`] | the S-box stages ([`SboxStages`]), the UltraRAM loader, the T-table ROMs |
//! | [`crate::sbox_placement`] | which of the 494 live S-boxes ([`LIVE_SBOX`]) go to UltraRAM or block RAM |
//!
//! # Reading order
//!
//! 1. [`KnownBytes`], [`KNOWN`]: the forward constant analysis.
//! 2. [`live_sbox_evaluations`], [`LIVE_SBOX`]: the backward "is it needed" analysis.
//! 3. [`ChainStepInput`], [`ChainStepOutput`]: the lane's ports.
//! 4. [`ChainStepLane`], [`chain_step_kernel`]: the all-S-box lane.
//! 5. [`TableChainStepLane`], [`table_chain_step_kernel`]: the same lane with
//!    some rounds done by T-table lookups instead.
//!
//! The lane has the same timing contract as the general pipeline: it accepts a
//! token every cycle, has no backpressure, and returns it after exactly
//! [`CHAIN_STEP_LATENCY`] registered transfers with its tag unchanged. Reset
//! clears only the valid bits, so all in-flight tokens are discarded.
//! Compare it bit-for-bit with [`dsig_chain`](crate::haraka::dsig_chain) using
//! `tests/chain_step.rs`.

use crate::aes_table::{DualTableRom, SboxLoader, SboxStages, SboxStagesIn, TableRead, TableWords};
use crate::haraka::{
    AES_ROUNDS, DIGEST_BYTES, MIX_WORDS, ROUND_CONSTANTS, ROUND_KEYS_RHDL, aes_sbox,
    mix_key512_kernel, mix512_kernel,
};
use rhdl::prelude::*;
use rhdl_fpga::core::dff::DFF;

/// Registered stages per chain step (two per AES round, as in Haraka).
///
/// 20 clock cycles from input to output, the same as
/// [`HARAKA_PIPELINE_LATENCY`](crate::haraka::HARAKA_PIPELINE_LATENCY).
pub const CHAIN_STEP_LATENCY: usize = 2 * AES_ROUNDS;
/// DSig keeps this many leading bytes of the standard 32-byte digest.
///
/// 18 bytes = 144 bits = one chain node.
pub const STEP_OUTPUT_BYTES: usize = 18;
/// First block byte that is always zero in a DSig chain block.
///
/// 16 nonce bytes + 18 node bytes = 34; bytes 34..64 are the zero tail.
pub const ZERO_TAIL_START: usize = 34;

/// Compile-time knowledge about the state entering one stage.
///
/// `mask[b]` is true when byte `b` is the same constant for every chain step;
/// `value[b]` then holds that constant. Live bytes have `mask[b] == false`.
///
/// Think of it as a 64-byte state where each byte is either a number or a
/// question mark. Only `const fn`s use it; it never becomes hardware.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct KnownBytes {
    /// True for bytes that never vary.
    pub mask: [bool; 64],
    /// The constant value of each masked byte (zero for live bytes).
    pub value: [u8; 64],
}

/// Multiply a byte by 2 in GF(2^8), for the compile-time MixColumns below.
///
/// Same as [`aes_xtime_kernel`](crate::haraka::aes_xtime_kernel), but a
/// plain `const fn` on `u8` so it can run during compilation.
const fn xtime(value: u8) -> u8 {
    (value << 1) ^ if value & 0x80 == 0 { 0 } else { 0x1b }
}

/// Symbolic `SubBytes ∘ ShiftRows` for all four AES lanes.
///
/// An output byte is known exactly when its ShiftRows source byte is known,
/// and then its value is `S(source)`.
const fn known_sub_shift(input: KnownBytes) -> KnownBytes {
    // Start with "nothing known"; known bytes are filled in below.
    let mut out = KnownBytes {
        mask: [false; 64],
        value: [0; 64],
    };
    let mut lane = 0;
    while lane < 4 {
        let mut column = 0;
        while column < 4 {
            let mut row = 0;
            while row < 4 {
                // The input byte that ShiftRows moves to this output position.
                let source = 16 * lane + 4 * ((column + row) & 3) + row;
                // This output position: lane, column, row.
                let target = 16 * lane + 4 * column + row;
                if input.mask[source] {
                    out.mask[target] = true;
                    out.value[target] = aes_sbox(input.value[source]);
                }
                row += 1;
            }
            column += 1;
        }
        lane += 1;
    }
    out
}

/// Symbolic `MixColumns ∘ AddRoundKey`, then Haraka's word mix when `mix`.
///
/// A column comes out known only if all four of its input bytes are known
/// (one live byte makes the whole column live). The word mix then just moves
/// the knowledge around. `round` is the AES round number (0 to 9), used to
/// pick the round key.
const fn known_mix_key(input: KnownBytes, round: usize, mix: bool) -> KnownBytes {
    // Start with "nothing known".
    let mut out = KnownBytes {
        mask: [false; 64],
        value: [0; 64],
    };
    let mut lane = 0;
    while lane < 4 {
        // This AES lane's 16-byte round key for this round.
        let key = ROUND_CONSTANTS[4 * round + lane].to_le_bytes();
        let mut column = 0;
        while column < 4 {
            // Index of this column's first byte in the 64-byte state.
            let base = 16 * lane + 4 * column;
            // A MixColumns output byte depends on all four bytes of its column.
            if input.mask[base]
                && input.mask[base + 1]
                && input.mask[base + 2]
                && input.mask[base + 3]
            {
                // The column's four known byte values.
                let s = [
                    input.value[base],
                    input.value[base + 1],
                    input.value[base + 2],
                    input.value[base + 3],
                ];
                // XOR of the column, as in aes_mix_key_kernel.
                let total = s[0] ^ s[1] ^ s[2] ^ s[3];
                let mut row = 0;
                while row < 4 {
                    // Same formula as the hardware, evaluated now.
                    out.mask[base + row] = true;
                    out.value[base + row] =
                        s[row] ^ total ^ xtime(s[row] ^ s[(row + 1) & 3]) ^ key[4 * column + row];
                    row += 1;
                }
            }
            column += 1;
        }
        lane += 1;
    }
    if mix {
        // Snapshot, then permute both mask and value by MIX_WORDS.
        let before = out;
        let mut word = 0;
        while word < 16 {
            let mut byte = 0;
            while byte < 4 {
                out.mask[4 * word + byte] = before.mask[4 * MIX_WORDS[word] + byte];
                out.value[4 * word + byte] = before.value[4 * MIX_WORDS[word] + byte];
                byte += 1;
            }
            word += 1;
        }
    }
    out
}

/// Stage `s` is a SubBytes/ShiftRows half-round when `s` is even.
///
/// Odd stages are MixColumns/AddRoundKey. Stage `s` belongs to AES round `s / 2`.
const fn is_sub_stage(stage: usize) -> bool {
    stage & 1 == 0
}

/// Haraka mixes the 32-bit words after every second AES round.
///
/// True for stages 3, 7, 11, 15 and 19: the MixColumns stage of AES rounds
/// 1, 3, 5, 7 and 9.
const fn mixes_after(stage: usize) -> bool {
    stage & 3 == 3
}

/// Run the forward constant analysis over all 20 stages and build [`KNOWN`].
///
/// Entry 0 is the chain block: bytes `0..34` live, bytes `34..64` known zero.
/// Entry `s + 1` is entry `s` pushed through stage `s` symbolically.
const fn known_table() -> [KnownBytes; CHAIN_STEP_LATENCY + 1] {
    // The chain block: everything live except the zero tail (value 0).
    let mut input = KnownBytes {
        mask: [false; 64],
        value: [0; 64],
    };
    let mut byte = ZERO_TAIL_START;
    while byte < 64 {
        input.mask[byte] = true;
        byte += 1;
    }
    // table[s] = knowledge entering stage s; filled in order.
    let mut table = [input; CHAIN_STEP_LATENCY + 1];
    let mut stage = 0;
    while stage < CHAIN_STEP_LATENCY {
        table[stage + 1] = if is_sub_stage(stage) {
            known_sub_shift(table[stage])
        } else {
            known_mix_key(table[stage], stage >> 1, mixes_after(stage))
        };
        stage += 1;
    }
    table
}

/// `KNOWN[s]` describes the state *entering* stage `s`; `KNOWN[20]` is the
/// final state before feed-forward. Index 0 is the DSig chain block itself.
pub const KNOWN: [KnownBytes; CHAIN_STEP_LATENCY + 1] = known_table();

/// Convert every [`KNOWN`] value from `u8` to RHDL's `b8`, so kernels can use it.
const fn rhdl_known_values() -> [[b8; 64]; CHAIN_STEP_LATENCY + 1] {
    // out[stage][byte], same shape as KNOWN.
    let mut out = [[b8(0); 64]; CHAIN_STEP_LATENCY + 1];
    let mut stage = 0;
    while stage <= CHAIN_STEP_LATENCY {
        let mut byte = 0;
        while byte < 64 {
            out[stage][byte] = b8(KNOWN[stage].value[byte] as u128);
            byte += 1;
        }
        stage += 1;
    }
    out
}

/// Copy every [`KNOWN`] mask into a plain array that kernels can index.
const fn known_masks() -> [[bool; 64]; CHAIN_STEP_LATENCY + 1] {
    // out[stage][byte], same shape as KNOWN.
    let mut out = [[false; 64]; CHAIN_STEP_LATENCY + 1];
    let mut stage = 0;
    while stage <= CHAIN_STEP_LATENCY {
        out[stage] = KNOWN[stage].mask;
        stage += 1;
    }
    out
}

/// RHDL-typed copy of the masks in [`KNOWN`], indexed `[stage][byte]`.
///
/// Why a separate copy: [`KnownBytes`] holds plain `u8` and is not an RHDL
/// `Digital` type, so a kernel cannot use it directly.
pub const KNOWN_MASK: [[bool; 64]; CHAIN_STEP_LATENCY + 1] = known_masks();
/// RHDL-typed copy of the constant values in [`KNOWN`].
///
/// Indexed `[stage][byte]`; meaningful only where [`KNOWN_MASK`] is true.
pub const KNOWN_VALUE: [[b8; 64]; CHAIN_STEP_LATENCY + 1] = rhdl_known_values();

/// Bytes of the final AES state that reach the 18 kept digest bytes.
///
/// The starting point of the backward analysis: these 18 positions (from
/// [`DIGEST_BYTES`]) are "needed"; nothing else in the final state is.
const fn observed_final_bytes() -> [bool; 64] {
    // observed[b]: final-state byte b reaches the output.
    let mut observed = [false; 64];
    let mut byte = 0;
    while byte < STEP_OUTPUT_BYTES {
        observed[DIGEST_BYTES[byte]] = true;
        byte += 1;
    }
    observed
}

/// Count the S-box evaluations that are neither constant nor unobserved.
///
/// Works backwards from the 18 kept bytes: a MixColumns output needs its whole
/// column, a SubBytes output needs its ShiftRows source, and the word mix is
/// a permutation. Works forwards with [`KNOWN`] to drop constant S-boxes. The
/// result is a structural planning count, not a synthesis measurement.
#[must_use]
pub const fn live_sbox_evaluations() -> usize {
    // needed[s][b]: byte b of the state entering stage s is used downstream.
    let mut needed = [[false; 64]; CHAIN_STEP_LATENCY + 1];
    needed[CHAIN_STEP_LATENCY] = observed_final_bytes();
    // Live S-boxes found so far.
    let mut count = 0;
    // Walk the stages from last (19) to first (0).
    let mut stage = CHAIN_STEP_LATENCY;
    while stage > 0 {
        stage -= 1;
        // What the stage's output must provide.
        let after = needed[stage + 1];
        // What the stage's input must provide; worked out below.
        let mut before = [false; 64];
        if is_sub_stage(stage) {
            // SubBytes/ShiftRows: each needed output byte needs exactly one
            // input byte, its ShiftRows source.
            let mut lane = 0;
            while lane < 4 {
                let mut column = 0;
                while column < 4 {
                    let mut row = 0;
                    while row < 4 {
                        let source = 16 * lane + 4 * ((column + row) & 3) + row;
                        let target = 16 * lane + 4 * column + row;
                        if after[target] {
                            // Needed downstream, so its source is needed too.
                            before[source] = true;
                            // A needed S-box is live unless its input is a
                            // compile-time constant.
                            if !KNOWN[stage].mask[source] {
                                count += 1;
                            }
                        }
                        row += 1;
                    }
                    column += 1;
                }
                lane += 1;
            }
        } else {
            // Undo the optional word mix, then widen to whole columns.
            let mut unmixed = after;
            if mixes_after(stage) {
                // The mix is a permutation: map each needed output byte
                // back to the input byte it came from.
                unmixed = [false; 64];
                let mut word = 0;
                while word < 16 {
                    let mut byte = 0;
                    while byte < 4 {
                        if after[4 * word + byte] {
                            unmixed[4 * MIX_WORDS[word] + byte] = true;
                        }
                        byte += 1;
                    }
                    word += 1;
                }
            }
            // MixColumns: one needed byte makes its whole 4-byte column
            // needed (16 columns across the four AES lanes).
            let mut column = 0;
            while column < 16 {
                let base = 4 * column;
                if unmixed[base] || unmixed[base + 1] || unmixed[base + 2] || unmixed[base + 3] {
                    before[base] = true;
                    before[base + 1] = true;
                    before[base + 2] = true;
                    before[base + 3] = true;
                }
                column += 1;
            }
        }
        // Record and continue with the stage before.
        needed[stage] = before;
    }
    count
}

/// Which S-boxes are live: `LIVE_SBOX[round][k]` is true when output byte `k`
/// of round `round`'s SubBytes stage is both used downstream and not constant.
///
/// Same analysis as [`live_sbox_evaluations`], recorded per position so that
/// S-boxes can be placed one by one (see [`sbox_placement`](crate::sbox_placement)).
///
/// Works backwards from the 18 kept bytes: a MixColumns output needs its whole
/// column, a SubBytes output needs its ShiftRows source, and the word mix is
/// a permutation. Works forwards with [`KNOWN`] to drop constant S-boxes. The
/// result is a structural planning count, not a synthesis measurement.
const fn live_sboxes() -> [[bool; 64]; AES_ROUNDS] {
    // live[round][k]: the result, filled in as live S-boxes are found.
    let mut live = [[false; 64]; AES_ROUNDS];
    // needed[s][b]: byte b of the state entering stage s is used downstream.
    let mut needed = [[false; 64]; CHAIN_STEP_LATENCY + 1];
    needed[CHAIN_STEP_LATENCY] = observed_final_bytes();
    // Walk the stages from last (19) to first (0).
    let mut stage = CHAIN_STEP_LATENCY;
    while stage > 0 {
        stage -= 1;
        // What the stage's output must provide.
        let after = needed[stage + 1];
        // What the stage's input must provide; worked out below.
        let mut before = [false; 64];
        if is_sub_stage(stage) {
            // SubBytes/ShiftRows: each needed output byte needs exactly one
            // input byte, its ShiftRows source.
            let mut lane = 0;
            while lane < 4 {
                let mut column = 0;
                while column < 4 {
                    let mut row = 0;
                    while row < 4 {
                        let source = 16 * lane + 4 * ((column + row) & 3) + row;
                        let target = 16 * lane + 4 * column + row;
                        if after[target] {
                            // Needed downstream, so its source is needed too.
                            before[source] = true;
                            // A needed S-box is live unless its input is a
                            // compile-time constant.
                            if !KNOWN[stage].mask[source] {
                                live[stage / 2][target] = true;
                            }
                        }
                        row += 1;
                    }
                    column += 1;
                }
                lane += 1;
            }
        } else {
            // Undo the optional word mix, then widen to whole columns.
            let mut unmixed = after;
            if mixes_after(stage) {
                // The mix is a permutation: map each needed output byte
                // back to the input byte it came from.
                unmixed = [false; 64];
                let mut word = 0;
                while word < 16 {
                    let mut byte = 0;
                    while byte < 4 {
                        if after[4 * word + byte] {
                            unmixed[4 * MIX_WORDS[word] + byte] = true;
                        }
                        byte += 1;
                    }
                    word += 1;
                }
            }
            // MixColumns: one needed byte makes its whole 4-byte column
            // needed (16 columns across the four AES lanes).
            let mut column = 0;
            while column < 16 {
                let base = 4 * column;
                if unmixed[base] || unmixed[base + 1] || unmixed[base + 2] || unmixed[base + 3] {
                    before[base] = true;
                    before[base + 1] = true;
                    before[base + 2] = true;
                    before[base + 3] = true;
                }
                column += 1;
            }
        }
        // Record and continue with the stage before.
        needed[stage] = before;
    }
    live
}

/// Per-position live S-boxes (computed by the private `live_sboxes`).
///
/// `LIVE_SBOX[round][k]` for `round` in `0..10`, `k` in `0..64` (output byte
/// `16·lane + 4·column + row`). Exactly 494 entries are true.
/// [`crate::sbox_placement`] walks it to decide where each live box lives.
pub const LIVE_SBOX: [[bool; 64]; AES_ROUNDS] = live_sboxes();

/// One chain-step request; an asserted `valid` is always accepted.
///
/// Only the two live parts of the chain block travel on the port. The zero
/// tail is rebuilt inside the lane as literal zeros, which is what lets
/// synthesis fold it.
#[derive(Clone, Copy, Debug, PartialEq, Digital)]
pub struct ChainStepInput {
    /// Nonce with the chain salt already added to its low 32 bits.
    ///
    /// Build it with [`dsig_chain_block`](crate::haraka::dsig_chain_block):
    /// the first 16 bytes of that block are exactly this field.
    pub salted_nonce: [b8; 16],
    /// The 18-byte chain node being advanced.
    pub secret: [b8; 18],
    /// Opaque caller identifier, returned unchanged.
    ///
    /// 32 bits. The lane never looks at it; the chain engine uses it to know
    /// which key, chain and depth a result belongs to.
    pub tag: b32,
    /// Launch this token; false inserts a bubble.
    pub valid: bool,
}

/// The idle input: all zeros with `valid` false.
impl Default for ChainStepInput {
    /// A bubble: nothing is launched.
    fn default() -> Self {
        Self {
            salted_nonce: [b8(0); 16],
            secret: [b8(0); 18],
            tag: b32(0),
            valid: false,
        }
    }
}

/// The next chain node; all fields except `valid` are meaningless when invalid.
///
/// Appears exactly [`CHAIN_STEP_LATENCY`] clocks after the matching input.
#[derive(Clone, Copy, Debug, PartialEq, Digital)]
pub struct ChainStepOutput {
    /// `mid_haraka512` of the chain block: the next 18-byte node.
    pub node: [b8; STEP_OUTPUT_BYTES],
    /// Caller tag aligned with `node`.
    pub tag: b32,
    /// A result is present and was not invalidated by reset.
    pub valid: bool,
    /// Every UltraRAM S-box of the lane is loaded; until then outputs are
    /// meaningless and the factory starts no keys.
    pub ready: bool,
}

/// Builds the zero-tailed DSig chain block from its two live fields.
///
/// Inputs: the 16-byte salted nonce and the 18-byte node. Output: the
/// 64-byte block of [`dsig_chain_block`](crate::haraka::dsig_chain_block).
/// Pure wiring: no gates. The other chain hashes in [`crate::chain_lane`]
/// reuse it.
#[allow(clippy::needless_range_loop)] // Static byte wiring.
#[kernel]
pub fn chain_step_block_kernel(salted_nonce: [b8; 16], secret: [b8; 18]) -> [b8; 64] {
    // Literal zeros: bytes 34..64 are never overwritten, so synthesis sees
    // constants there.
    let mut block = [b8(0); 64];
    // Bytes 0..16: the salted nonce.
    for byte in 0..16 {
        block[byte] = salted_nonce[byte];
    }
    // Bytes 16..34: the node.
    for byte in 0..18 {
        block[16 + byte] = secret[byte];
    }
    block
}

/// Replace the bytes `mask` marks as compile-time constants by `value`.
///
/// Call it with a row of [`KNOWN_MASK`]/[`KNOWN_VALUE`]: the selects are then
/// constants, the register feeding a pinned byte has no loads, and synthesis
/// deletes it together with the logic that computed it.
///
/// Why this is needed at all: to the tools, a register's output is not a
/// constant. Its power-up value is "don't care", so they cannot assume it
/// always equals the constant computed into it. Pinning states the value
/// directly, so the S-boxes behind it fold to constants. Combinational: 64
/// selects whose controls are all constants.
#[allow(clippy::needless_range_loop)] // Static byte wiring.
#[kernel]
pub fn pin_known_kernel(state: [b8; 64], mask: [bool; 64], value: [b8; 64]) -> [b8; 64] {
    // Start from the register values; constant bytes are replaced below.
    let mut pinned = state;
    for byte in 0..64 {
        // `mask` is a compile-time constant at every call site, so this `if`
        // becomes a plain wire (live byte) or a literal (constant byte).
        if mask[byte] {
            pinned[byte] = value[byte];
        }
    }
    pinned
}

/// Final feed-forward: the 18 kept digest bytes of a finished state.
///
/// Inputs: the 64-byte state after the last stage and the 18 saved input
/// bytes. Output: the next 18-byte node, `state[DIGEST_BYTES[i]] ⊕ feed[i]`.
/// Combinational: one XOR per output bit (18 × 8). Because only 18 bytes
/// are read, the S-boxes that feed only the other 46 bytes have no load and
/// are deleted.
#[allow(clippy::needless_range_loop)] // Static byte wiring.
#[kernel]
pub fn chain_step_digest_kernel(
    state: [b8; 64],
    feed_forward: [b8; STEP_OUTPUT_BYTES],
) -> [b8; STEP_OUTPUT_BYTES] {
    // Pin whatever is still constant in the final state.
    let last = pin_known_kernel(
        state,
        KNOWN_MASK[CHAIN_STEP_LATENCY],
        KNOWN_VALUE[CHAIN_STEP_LATENCY],
    );
    // The output node, byte by byte: truncate, then XOR the saved input.
    let mut node = [b8(0); STEP_OUTPUT_BYTES];
    for byte in 0..STEP_OUTPUT_BYTES {
        node[byte] = last[DIGEST_BYTES[byte]] ^ feed_forward[byte];
    }
    node
}

/// The 18 input bytes that the feed-forward XORs back into the digest.
///
/// Input: the 64-byte chain block. Output: `block[DIGEST_BYTES[i]]` for
/// `i` in `0..18`. Pure wiring. The lane saves these 18 bytes (not all 64)
/// and carries them down the pipeline next to the state, 20 stages deep.
#[allow(clippy::needless_range_loop)] // Static byte wiring.
#[kernel]
pub fn chain_step_feed_kernel(block: [b8; 64]) -> [b8; STEP_OUTPUT_BYTES] {
    // The saved bytes, in digest order.
    let mut feed = [b8(0); STEP_OUTPUT_BYTES];
    for byte in 0..STEP_OUTPUT_BYTES {
        feed[byte] = block[DIGEST_BYTES[byte]];
    }
    feed
}

/// Twenty-stage, II = 1 DSig chain step with compile-time constant folding.
///
/// Each AES round `r` is two registered half-rounds:
///
/// ```text
///  mixed[r-1] ─pin─▶ SboxStage subs[r] ─pin─▶ MixColumns+key(+word mix) ─▶ mixed[r]
///                  (S-box table, reg)                                     (reg)
/// ```
///
/// [`SboxStages`] keeps the generated netlist compact. The `pin` steps apply
/// [`KNOWN`]: registers that only ever hold constants, or that never reach an
/// output, have no loads, and synthesis removes them.
///
/// The whole lane, stage by stage (`S` = S-box stage register, `M` = mixed
/// register; both are 64 bytes wide):
///
/// ```text
///   block ─▶ S0 ─▶ M0 ─▶ S1 ─▶ M1 ─▶ S2 ─▶ ... ─▶ S9 ─▶ M9 ─▶ digest ─▶ node
///   stage:   0     1     2     3     4            18    19
///                              +MIX                     +MIX
///   side band:  feed_forward (18 B), tag (32 bit), valid — 20 registers each
/// ```
///
/// Timing contract: one token per clock in, the result exactly 20 clocks
/// later, no backpressure. The S-box register `S` lives inside
/// [`SboxStages`] (a ROM's registered read); the `M` registers are `mixed`.
#[derive(Clone, Debug, Synchronous, SynchronousDQ)]
pub struct ChainStepLane {
    /// The ten S-box stages (SubBytes + ShiftRows), one per AES round.
    /// Their ROM output registers are the even pipeline stages.
    subs: SboxStages,
    /// Fills any UltraRAM S-boxes after reset and drives `ready`.
    loader: SboxLoader,
    /// The odd pipeline stages: the 64-byte state after each AES round's
    /// MixColumns, AddRoundKey and (every second round) word mix.
    mixed: [DFF<[b8; 64]>; AES_ROUNDS],
    /// The 18 input bytes needed by the final XOR, delayed 20 stages.
    feed_forward: [DFF<[b8; STEP_OUTPUT_BYTES]>; CHAIN_STEP_LATENCY],
    /// The caller's tag, delayed 20 stages.
    tag: [DFF<b32>; CHAIN_STEP_LATENCY],
    /// "This stage holds a real token", delayed 20 stages. The only
    /// registers here that reset (to false).
    valid: [DFF<bool>; CHAIN_STEP_LATENCY],
}

/// Build a lane with S-boxes placed as [`crate::sbox_placement`] decided.
impl Default for ChainStepLane {
    /// Data registers start as "don't care"; every valid bit starts false.
    fn default() -> Self {
        Self {
            subs: SboxStages::default(),
            loader: SboxLoader::default(),
            mixed: core::array::from_fn(|_| DFF::resetless()),
            feed_forward: core::array::from_fn(|_| DFF::resetless()),
            tag: core::array::from_fn(|_| DFF::resetless()),
            valid: core::array::from_fn(|_| DFF::new(false)),
        }
    }
}

/// Ports of [`ChainStepLane`]: one chain step in, one node out per clock.
impl SynchronousIO for ChainStepLane {
    /// Salted nonce, node, tag and valid.
    type I = ChainStepInput;
    /// Next node, tag, valid and ready, 20 clocks later.
    type O = ChainStepOutput;
    /// The stage wiring below.
    type Kernel = chain_step_kernel;
}

/// Stage wiring for [`ChainStepLane`].
///
/// Inputs: `clock_reset`, the new request `input`, and `q` (every child's
/// current output). Outputs: the [`ChainStepOutput`] leaving stage 19 and
/// `d` (every child's next input). Per AES round `r`:
///
/// - S-box stage `r` reads `mixed[r - 1]` (or the new block for `r = 0`),
///   with the constant bytes pinned;
/// - `mixed[r]` reads S-box stage `r`'s output, pinned again, through
///   MixColumns + AddRoundKey (+ word mix when `r` is odd).
///
/// Every loop bound and index is a constant, so RHDL unrolls all of it into
/// fixed wiring.
#[allow(clippy::needless_range_loop)] // Compile-time pipeline elaboration.
#[kernel]
pub fn chain_step_kernel(
    clock_reset: ClockReset,
    input: ChainStepInput,
    q: ChainStepLaneQ,
) -> (ChainStepOutput, ChainStepLaneD) {
    // True while reset is asserted; hides the output during reset.
    let resetting = clock_reset.reset.any();
    // The 64-byte chain block, with its zero tail as literals.
    let block = chain_step_block_kernel(input.salted_nonce, input.secret);
    // Defaults for every child: hold the registers, give every S-box stage
    // the block (only round 0 keeps it; rounds 1 to 9 are overwritten
    // below), pass the loader's write to every UltraRAM box.
    let mut d = ChainStepLaneD {
        subs: SboxStagesIn {
            bytes: [block; AES_ROUNDS],
            load: q.loader.load,
        },
        // The loader has no input.
        loader: (),
        mixed: q.mixed,
        feed_forward: q.feed_forward,
        tag: q.tag,
        valid: q.valid,
    };
    // The block's zero tail is literal, so round 0's S-boxes fold by themselves.
    // Even stages of rounds 1 to 9: the previous round's state, pinned.
    for round in 1..AES_ROUNDS {
        d.subs.bytes[round] = pin_known_kernel(
            q.mixed[round - 1],
            KNOWN_MASK[2 * round],
            KNOWN_VALUE[2 * round],
        );
    }
    // Odd stages: MixColumns + key on each S-box stage's output.
    for round in 0..AES_ROUNDS {
        // The S-box output with its still-constant bytes pinned
        // (KNOWN index 2r + 1 = the state entering stage 2r + 1).
        let source = pin_known_kernel(
            q.subs[round],
            KNOWN_MASK[2 * round + 1],
            KNOWN_VALUE[2 * round + 1],
        );
        // Word mix after odd AES rounds (1, 3, 5, 7, 9).
        d.mixed[round] = mix_key512_kernel(source, ROUND_KEYS_RHDL[round], round & 1 == 1);
    }
    // Side band: save the feed-forward bytes, tag and valid at stage 0 ...
    d.feed_forward[0] = chain_step_feed_kernel(block);
    d.tag[0] = input.tag;
    d.valid[0] = input.valid;
    // ... and shift them one stage along every clock.
    for stage in 1..CHAIN_STEP_LATENCY {
        d.feed_forward[stage] = q.feed_forward[stage - 1];
        d.tag[stage] = q.tag[stage - 1];
        d.valid[stage] = q.valid[stage - 1];
    }
    // Output: the finished node from the last mixed register and the
    // matching feed-forward bytes. `valid` is forced low during reset;
    // `ready` comes straight from the loader.
    (
        ChainStepOutput {
            node: chain_step_digest_kernel(
                q.mixed[AES_ROUNDS - 1],
                q.feed_forward[CHAIN_STEP_LATENCY - 1],
            ),
            tag: q.tag[CHAIN_STEP_LATENCY - 1],
            valid: q.valid[CHAIN_STEP_LATENCY - 1] && !resetting,
            ready: q.loader.done,
        },
        d,
    )
}

/// AES rounds that may use block-RAM T-tables: every byte is live and used.
///
/// The first such round is AES round 3 (0-based). Rounds 0-2 still have
/// constant bytes, which a table cannot fold, and round 9 needs only 20 of
/// its 64 S-boxes. (Rounds 7 and 8 also have 16 unobserved S-boxes each;
/// there the table ROMs whose words are never used have no loads and are
/// removed the same way, so nothing is wasted.)
pub const TABLE_ROUND_FIRST: usize = 3;
/// Number of table-capable rounds (rounds 3 through 8).
pub const TABLE_ROUND_SLOTS: usize = 6;
/// Dual-port ROMs per table round: 64 S-boxes, two per ROM.
///
/// Each [`DualTableRom`] is one true dual-port `T0` ROM (one BRAM18) serving
/// two state bytes, so a table round needs 32 of them for its 64 bytes.
pub const ROMS_PER_TABLE_ROUND: usize = 32;

/// Whether `mask` is a legal [`TableChainStepLane`] table selection.
///
/// Legal means: only bits 3..=8 may be set.
#[must_use]
pub const fn valid_table_mask(mask: usize) -> bool {
    // Six ones shifted up to bits 3..=8: binary 1_1111_1000.
    let allowed = ((1 << TABLE_ROUND_SLOTS) - 1) << TABLE_ROUND_FIRST;
    mask & !allowed == 0
}

/// Present ShiftRows-ordered state bytes to a round's 32 dual-port ROMs.
///
/// Input: the 64-byte state entering the round. Output: 32 address pairs.
/// Table `port = 16·lane + 4·column + row` reads the byte ShiftRows would put
/// at that position, so ShiftRows costs nothing here either. Port `p` is
/// side `a` of ROM `p / 2` when `p` is even and side `b` when odd. Pure wiring.
#[allow(clippy::needless_range_loop)] // Static byte wiring.
#[kernel]
pub fn table_addresses_kernel(state: [b8; 64]) -> [TableRead; ROMS_PER_TABLE_ROUND] {
    // The 32 address pairs; every port is assigned below.
    let mut reads = [TableRead::default(); ROMS_PER_TABLE_ROUND];
    for lane in 0..4 {
        for column in 0..4 {
            for row in 0..4 {
                // ShiftRows source byte for this position.
                let byte = state[16 * lane + 4 * ((column + row) & 3) + row];
                // Which of the 64 table ports serves this position.
                let port = 16 * lane + 4 * column + row;
                // Even ports use side a, odd ports side b; `>> 1` is `/ 2`.
                if port & 1 == 0 {
                    reads[port >> 1].a = byte;
                } else {
                    reads[port >> 1].b = byte;
                }
            }
        }
    }
    reads
}

/// Combine one round's 64 table words: rotated XOR per column plus the key.
///
/// Row `out` of `rotl(T0[x], 8·row)` is byte `(out − row) mod 4` of `T0[x]`.
///
/// Inputs: the 64 table words read last cycle, the round's 64-byte key, and
/// `mix` (apply the word mix after this round). Output: the 64-byte state
/// after the whole AES round. Combinational: each output byte is a 5-input
/// XOR (four table bytes and one key byte). The "rotation" is only a choice
/// of which byte of each word to wire in; it costs nothing.
///
/// ```text
///   column c of lane l, output row o:
///     out[o] = key[o] ⊕ T0[x0].byte(o) ⊕ T0[x1].byte(o−1) ⊕ T0[x2].byte(o−2) ⊕ T0[x3].byte(o−3)
///                                                   (row offsets taken mod 4)
/// ```
#[allow(clippy::needless_range_loop)] // Static byte wiring.
#[kernel]
pub fn table_combine_kernel(
    words: [TableWords; ROMS_PER_TABLE_ROUND],
    key: [b8; 64],
    mix: bool,
) -> [b8; 64] {
    // The round's output state; every byte is assigned below.
    let mut mixed = [b8(0); 64];
    for lane in 0..4 {
        for column in 0..4 {
            // Index of this column's first byte (and first table port).
            let base = 16 * lane + 4 * column;
            for out_row in 0..4 {
                // Accumulator for this output byte, starting with the key.
                let mut value = key[base + out_row];
                // Add the contribution of each of the column's four bytes.
                for row in 0..4 {
                    // The table port that looked up input row `row`.
                    let port = base + row;
                    // Its 32-bit word, from side a or b of the ROM pair.
                    let word = if port & 1 == 0 {
                        words[port >> 1].a
                    } else {
                        words[port >> 1].b
                    };
                    // Which byte of the word lands in `out_row`:
                    // (out_row − row) mod 4, written without a subtraction
                    // that could go negative.
                    let pick = (out_row + 4 - row) & 3;
                    // Select that byte (all constants, so plain wiring).
                    let byte: b8 = if pick == 0 {
                        word.resize()
                    } else if pick == 1 {
                        (word >> 8).resize()
                    } else if pick == 2 {
                        (word >> 16).resize()
                    } else {
                        (word >> 24).resize()
                    };
                    value = value ^ byte;
                }
                mixed[base + out_row] = value;
            }
        }
    }
    // Word mix after odd AES rounds, as in the LUT path.
    if mix { mix512_kernel(mixed) } else { mixed }
}

/// [`ChainStepLane`] with selected AES rounds moved from LUTs into T-table ROMs.
///
/// Bit `r` of `TABLE_MASK` moves AES round `r` (0-based) into
/// [`DualTableRom`]s. Only rounds 3–8 are allowed: there [`KNOWN`] proves
/// nothing constant and every output byte is needed, so a table never replaces
/// logic synthesis would have deleted anyway. Each table round uses 32 ROMs,
/// normally 32 BRAM18s.
///
/// A precision note: by the same backward analysis as [`LIVE_SBOX`], rounds
/// 7 and 8 each have 16 positions that cannot reach the output (see the
/// per-round table in the module docs). In those rounds the ROM pairs for
/// the unobserved positions have no loads and are removed like the S-boxes
/// they replace, so the table still never pays for deleted work; it just
/// uses fewer ROMs than 32.
///
/// ```text
///  LUT round:    mixed[r-1] ─▶ SboxStage (reg) ─▶ MixColumns+key (LUT) ─▶ mixed[r]
///  table round:  mixed[r-1] ─▶ 32 × ROM  (reg) ─▶ 4-way XOR+key (LUT) ─▶ mixed[r]
/// ```
///
/// Latency, II and tag behaviour equal [`ChainStepLane`]; the tests check
/// several masks against the same reference.
///
/// Why offer this at all: block RAM is a separate budget from LUTs. Moving a
/// round into tables trades LUTs (the binding limit) for block RAMs that
/// would otherwise sit unused. `TABLE_MASK = 0` is exactly [`ChainStepLane`].
/// This is the lane type the Haraka profile instantiates (see
/// [`crate::chain_lane`]).
///
/// Every round keeps both a LUT path and, if it is a table slot, a table
/// path in the source. The constant `TABLE_MASK` picks one per round, and
/// synthesis deletes the other: unused ROMs have no loads, and an S-box stage
/// fed with zeros folds to constants.
#[derive(Clone, Debug, Synchronous, SynchronousDQ)]
pub struct TableChainStepLane<const TABLE_MASK: usize> {
    /// The ten S-box stages, as in [`ChainStepLane`]. A table round's stage
    /// is fed constants and so disappears.
    subs: SboxStages,
    /// Fills any UltraRAM S-boxes after reset and drives `ready`.
    loader: SboxLoader,
    /// `roms[slot]` holds the 32 ROM pairs of AES round
    /// `TABLE_ROUND_FIRST + slot`; unused when that round's mask bit is 0.
    roms: [[DualTableRom; ROMS_PER_TABLE_ROUND]; TABLE_ROUND_SLOTS],
    /// The state after each AES round, as in [`ChainStepLane`].
    mixed: [DFF<[b8; 64]>; AES_ROUNDS],
    /// The 18 input bytes needed by the final XOR, delayed 20 stages.
    feed_forward: [DFF<[b8; STEP_OUTPUT_BYTES]>; CHAIN_STEP_LATENCY],
    /// The caller's tag, delayed 20 stages.
    tag: [DFF<b32>; CHAIN_STEP_LATENCY],
    /// "This stage holds a real token", delayed 20 stages; resets to false.
    valid: [DFF<bool>; CHAIN_STEP_LATENCY],
}

/// Build a lane, rejecting masks outside rounds 3..=8.
impl<const TABLE_MASK: usize> Default for TableChainStepLane<TABLE_MASK> {
    /// Data registers start as "don't care"; every valid bit starts false.
    ///
    /// # Panics
    /// Panics if `TABLE_MASK` sets a bit outside rounds 3..=8.
    fn default() -> Self {
        assert!(
            valid_table_mask(TABLE_MASK),
            "table rounds must lie within rounds 3..=8"
        );
        Self {
            subs: SboxStages::default(),
            loader: SboxLoader::default(),
            roms: core::array::from_fn(|_| core::array::from_fn(|_| DualTableRom::default())),
            mixed: core::array::from_fn(|_| DFF::resetless()),
            feed_forward: core::array::from_fn(|_| DFF::resetless()),
            tag: core::array::from_fn(|_| DFF::resetless()),
            valid: core::array::from_fn(|_| DFF::new(false)),
        }
    }
}

/// Same ports as [`ChainStepLane`].
impl<const TABLE_MASK: usize> SynchronousIO for TableChainStepLane<TABLE_MASK> {
    /// Salted nonce, node, tag and valid.
    type I = ChainStepInput;
    /// Next node, tag, valid and ready, 20 clocks later.
    type O = ChainStepOutput;
    /// The stage wiring below.
    type Kernel = table_chain_step_kernel<TABLE_MASK>;
}

/// Stage wiring for [`TableChainStepLane`].
///
/// Inputs and outputs as [`chain_step_kernel`]. First it wires every round
/// the LUT way, exactly like [`chain_step_kernel`]. Then, for each table
/// round selected by `TABLE_MASK`, it overrides that round: the S-box stage
/// gets zeros, the ROMs get the (pinned) state, and `mixed[r]` takes the
/// table result. Both paths have one register (S-box ROM or table ROM)
/// followed by one `mixed` register, so latency stays 20.
#[allow(clippy::needless_range_loop)] // Compile-time pipeline elaboration.
#[kernel]
pub fn table_chain_step_kernel<const TABLE_MASK: usize>(
    clock_reset: ClockReset,
    input: ChainStepInput,
    q: TableChainStepLaneQ<TABLE_MASK>,
) -> (ChainStepOutput, TableChainStepLaneD<TABLE_MASK>) {
    // True while reset is asserted; hides the output during reset.
    let resetting = clock_reset.reset.any();
    // The 64-byte chain block, with its zero tail as literals.
    let block = chain_step_block_kernel(input.salted_nonce, input.secret);
    // Defaults: as in chain_step_kernel, plus every table ROM reading
    // address 0 (overridden for selected table rounds).
    let mut d = TableChainStepLaneD::<TABLE_MASK> {
        subs: SboxStagesIn {
            bytes: [block; AES_ROUNDS],
            load: q.loader.load,
        },
        // The loader has no input.
        loader: (),
        roms: [[TableRead::default(); ROMS_PER_TABLE_ROUND]; TABLE_ROUND_SLOTS],
        mixed: q.mixed,
        feed_forward: q.feed_forward,
        tag: q.tag,
        valid: q.valid,
    };
    // LUT path for every round; table rounds are overridden below.
    for round in 1..AES_ROUNDS {
        d.subs.bytes[round] = pin_known_kernel(
            q.mixed[round - 1],
            KNOWN_MASK[2 * round],
            KNOWN_VALUE[2 * round],
        );
    }
    // Odd stages: MixColumns + key on each S-box stage's output.
    for round in 0..AES_ROUNDS {
        // The S-box output with its still-constant bytes pinned
        // (KNOWN index 2r + 1 = the state entering stage 2r + 1).
        let source = pin_known_kernel(
            q.subs[round],
            KNOWN_MASK[2 * round + 1],
            KNOWN_VALUE[2 * round + 1],
        );
        // Word mix after odd AES rounds (1, 3, 5, 7, 9).
        d.mixed[round] = mix_key512_kernel(source, ROUND_KEYS_RHDL[round], round & 1 == 1);
    }
    // Every index below stays in range for every mask, so RHDL can unroll both
    // arms of the constant `if`.
    for slot in 0..TABLE_ROUND_SLOTS {
        // The AES round this slot can replace (3..=8).
        let round = TABLE_ROUND_FIRST + slot;
        // Compile-time test: is this round a table round in this build?
        if (TABLE_MASK >> round) & 1 == 1 {
            // The table round's S-box stage is unused: feed it a constant.
            d.subs.bytes[round] = [b8(0); 64];
            // The ROMs read the state the S-box stage would have read.
            d.roms[slot] = table_addresses_kernel(pin_known_kernel(
                q.mixed[round - 1],
                KNOWN_MASK[2 * round],
                KNOWN_VALUE[2 * round],
            ));
            // One cycle later, combine the table words into the round result.
            d.mixed[round] =
                table_combine_kernel(q.roms[slot], ROUND_KEYS_RHDL[round], round & 1 == 1);
        }
    }
    // Side band: save the feed-forward bytes, tag and valid at stage 0 ...
    d.feed_forward[0] = chain_step_feed_kernel(block);
    d.tag[0] = input.tag;
    d.valid[0] = input.valid;
    // ... and shift them one stage along every clock.
    for stage in 1..CHAIN_STEP_LATENCY {
        d.feed_forward[stage] = q.feed_forward[stage - 1];
        d.tag[stage] = q.tag[stage - 1];
        d.valid[stage] = q.valid[stage - 1];
    }
    // Output: the finished node from the last mixed register and the
    // matching feed-forward bytes. `valid` is forced low during reset;
    // `ready` comes straight from the loader.
    (
        ChainStepOutput {
            node: chain_step_digest_kernel(
                q.mixed[AES_ROUNDS - 1],
                q.feed_forward[CHAIN_STEP_LATENCY - 1],
            ),
            tag: q.tag[CHAIN_STEP_LATENCY - 1],
            valid: q.valid[CHAIN_STEP_LATENCY - 1] && !resetting,
            ready: q.loader.done,
        },
        d,
    )
}