rhdl_dsig_scalable_core/

blake3_core.rs

//! BLAKE3 compression and a fully pipelined Rust/RHDL implementation.
//!
//! # Why this module exists
//!
//! Haraka does the chain steps, but everything else a DSig key needs is
//! BLAKE3, in every profile: deriving the one-time seed, expanding it into
//! the 68 chain secrets and the nonces (BLAKE3's extendable output, "XOF"),
//! committing to the 68 chain ends, building the Merkle tree, and hashing the
//! message. That is about 47 BLAKE3 compressions per signature
//! ([`sustained`](crate::sustained)).
//!
//! BLAKE3 hashes any input by cutting it into 64-byte blocks and running a
//! *compression function* on each one. A tree-hashing controller decides the
//! order, the counters and the flags; the compression function does the
//! actual mixing. This module is only the compression function. The callers
//! that drive it:
//!
//! | Caller | What it hashes |
//! |---|---|
//! | [`crate::slice_factory`] | seeds, XOF blocks, chain-end commitment, subtree, message digest, on a fixed 68-cycle timetable ([`crate::slice_schedule`]) |
//! | [`crate::cert_unit`] | the Merkle levels shared by all slices |
//! | [`crate::chain_lane`] | the chain step itself in the `DSIG_CHAIN=blake3` profile |
//! | [`crate::slice_model`] | the same work in plain Rust, using [`compress`] |
//!
//! Because one BLAKE3 lane per slice serves all of that work, the lane must
//! accept a new block **every clock**. So this file builds a fully pipelined
//! compression: 28 register stages, one block in and one block out per clock.
//!
//! # What one compression does
//!
//! The state is 16 words of 32 bits, viewed as a 4 × 4 matrix. It starts as
//! the 8-word chaining value, 4 words of [`IV`], the 64-bit counter, the block
//! length and the flags. Seven rounds then mix in the 16 message words:
//!
//! ```text
//!   state (16 × 32-bit words)          one round = 8 G functions
//!   ┌─────┬─────┬─────┬─────┐
//!   │ v0  │ v1  │ v2  │ v3  │          column step:   G(v0,v4,v8,v12)  G(v1,v5,v9,v13)
//!   ├─────┼─────┼─────┼─────┤                         G(v2,v6,v10,v14) G(v3,v7,v11,v15)
//!   │ v4  │ v5  │ v6  │ v7  │
//!   ├─────┼─────┼─────┼─────┤          diagonal step: G(v0,v5,v10,v15) G(v1,v6,v11,v12)
//!   │ v8  │ v9  │ v10 │ v11 │                         G(v2,v7,v8,v13)  G(v3,v4,v9,v14)
//!   ├─────┼─────┼─────┼─────┤
//!   │ v12 │ v13 │ v14 │ v15 │          then permute the 16 message words
//!   └─────┴─────┴─────┴─────┘          (MESSAGE_PERMUTATION) for the next round
//!
//!   G(a, b, c, d) with message words x, y:
//!     first half:   a += b + x;  d = (d ⊕ a) ⋙ 16;  c += d;  b = (b ⊕ c) ⋙ 12
//!     second half:  a += b + y;  d = (d ⊕ a) ⋙ 8;   c += d;  b = (b ⊕ c) ⋙ 7
//! ```
//!
//! (`⋙` is a rotate right; `+` wraps at 2^32.) After seven rounds a
//! *feed-forward* XORs the two halves of the state together and the original
//! chaining value into the upper half, giving 16 output words.
//!
//! # From the picture to a pipeline
//!
//! The four G functions of a step are independent, so they run side by side.
//! Each G is cut in half, so one round is four register stages:
//!
//! ```text
//!   CompressionInput ─▶ initialize ─▶ ┌ phase 0: column G, first half   ┐ ─reg─▶
//!                                     │ phase 1: column G, second half  │ ─reg─▶  × 7 rounds
//!                                     │ phase 2: diagonal G, first half │ ─reg─▶  = 28 stages
//!                                     └ phase 3: diagonal G, second half┘ ─reg─▶
//!                                       + message permutation
//!                                                          ─▶ feed_forward ─▶ CompressionOutput
//! ```
//!
//! Why half a G per stage: each half has two additions in series. A whole G
//! per stage would put four in series between two registers and lower the
//! clock the stage can reach. The out-of-context measurement of one BLAKE3
//! lane (~325 MHz alone, see [`sustained`](crate::sustained)) is the evidence
//! that the 205 MHz target has margin; nothing in this file measures it.
//!
//! # Reading order
//!
//! 1. Constants: [`ROUNDS`], [`PIPELINE_STAGES`], the flags, [`IV`].
//! 2. The software model: [`compress`], [`salted_digest_16`].
//! 3. Ports: [`CompressionInput`], [`CompressionOutput`].
//! 4. Kernels: [`g_first_kernel`], [`g_second_kernel`],
//!    [`quarter_stage_kernel`], [`feed_forward_kernel`].
//! 5. The pipeline: [`RoundPipeline`], [`round_pipeline_kernel`],
//!    [`CompressionPipeline`], [`InlineCompressionPipeline`].
//!
//! # Summary of the contract
//!
//! [`CompressionPipeline`](crate::blake3_core::CompressionPipeline) implements all seven BLAKE3 rounds, with four
//! register transfers per round. Each stage executes one half of four parallel
//! G functions. Inputs are accepted every cycle; a corresponding output is
//! visible after exactly [`PIPELINE_STAGES`](crate::blake3_core::PIPELINE_STAGES) registered transfers. There is no
//! backpressure. The caller must provision downstream capacity before sending
//! a valid token.
//!
//! Message and chaining words use BLAKE3's **little-endian** encoding. This is a
//! compression primitive, not a general-purpose tree-hashing controller. Its
//! caller owns block padding, chunk counters, flag selection, and chaining.
//! [`salted_digest_16`](crate::blake3_core::salted_digest_16) is the complete fixed-size online message operation:
//! the first 64 XOF bytes of `BLAKE3(pk_hash || nonce || message)`, with a 32-byte
//! public-key hash, 16-byte nonce, and exactly 16 message bytes.
//!
//! The round state, original chaining value, message schedule, and tag reside
//! in resetless registers. Only the valid bits reset. Data is unspecified when
//! `valid` is false, and reset discards every in-flight token. No resource,
//! routed-frequency, or board-throughput claim follows from this architecture.

use rhdl::{core::circuit::descriptor::SyncKind, prelude::*};
use rhdl_fpga::core::dff::DFF;

/// Number of BLAKE3 compression rounds.
///
/// Fixed by the BLAKE3 specification.
pub const ROUNDS: usize = 7;
/// Number of registered G-half stages in a complete compression.
///
/// 4 stages per round × 7 rounds = 28: the lane latency in clock cycles.
/// The factory's timetable and the BLAKE3 chain lane read this value.
pub const PIPELINE_STAGES: usize = 4 * ROUNDS;
/// Number of input bytes in a BLAKE3 compression block.
///
/// 64 bytes = 16 little-endian 32-bit message words.
pub const BLOCK_BYTES: usize = 64;
/// Number of bytes returned by one BLAKE3 XOF output block.
///
/// All 16 output words; a plain 32-byte hash uses only the first half.
pub const OUTPUT_BYTES: usize = 64;
/// Marks the first block of a chunk.
///
/// The flags below are single bits ORed together into the 32-bit `flags`
/// word. They tell the compression where this block sits in BLAKE3's tree,
/// so that otherwise identical blocks in different roles hash differently.
/// A *chunk* is up to 1 KiB of input (16 blocks).
pub const CHUNK_START: u32 = 1;
/// Marks the final block of a chunk.
pub const CHUNK_END: u32 = 2;
/// Marks a tree-parent block made from two child chaining values.
pub const PARENT: u32 = 4;
/// Selects root/XOF output rather than an intermediate chaining value.
pub const ROOT: u32 = 8;
/// Selects BLAKE3's keyed-hash mode.
pub const KEYED_HASH: u32 = 16;
/// Selects the context-hashing step of BLAKE3 key derivation.
pub const DERIVE_KEY_CONTEXT: u32 = 32;
/// Selects the material-hashing step of BLAKE3 key derivation.
pub const DERIVE_KEY_MATERIAL: u32 = 64;
/// Flags for one complete, unkeyed, single-block root hash.
///
/// `1 | 2 | 8 = 11`: the block is the first and last of the only chunk, and
/// it is the root, so its output is the final hash.
pub const SINGLE_BLOCK_ROOT: u32 = CHUNK_START | CHUNK_END | ROOT;

/// BLAKE3 initialization vector, as specified by the BLAKE3 team.
///
/// Eight 32-bit words (the same as SHA-256's). They are the chaining value
/// of an unkeyed hash's first block, and words 8..12 of every compression's
/// starting state.
pub const IV: [u32; 8] = [
    0x6a09_e667,
    0xbb67_ae85,
    0x3c6e_f372,
    0xa54f_f53a,
    0x510e_527f,
    0x9b05_688c,
    0x1f83_d9ab,
    0x5be0_cd19,
];

/// Word permutation applied after each complete round.
///
/// After round `r`, message word `i` of round `r + 1` is word
/// `MESSAGE_PERMUTATION[i]` of round `r`. The hardware applies it as wiring
/// in phase 3 of [`quarter_stage_kernel`].
pub const MESSAGE_PERMUTATION: [usize; 16] = [2, 6, 3, 10, 7, 0, 4, 13, 1, 11, 12, 5, 9, 14, 15, 8];

/// Official seven-round message schedule, used by the independent Rust model.
///
/// Row `r` lists which original message word each G input uses in round
/// `r`. It is [`MESSAGE_PERMUTATION`] applied `r` times, written out so the
/// software model does not share the hardware's way of computing it.
pub const MESSAGE_SCHEDULE: [[usize; 16]; ROUNDS] = [
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15],
    [2, 6, 3, 10, 7, 0, 4, 13, 1, 11, 12, 5, 9, 14, 15, 8],
    [3, 4, 10, 12, 13, 2, 7, 14, 6, 5, 9, 0, 11, 15, 8, 1],
    [10, 7, 12, 9, 14, 3, 13, 15, 4, 0, 11, 2, 5, 8, 1, 6],
    [12, 13, 9, 11, 15, 10, 14, 8, 7, 2, 5, 3, 0, 1, 6, 4],
    [9, 14, 11, 5, 8, 12, 15, 1, 13, 3, 0, 10, 2, 6, 4, 7],
    [11, 15, 5, 0, 1, 9, 8, 6, 14, 10, 2, 12, 3, 4, 7, 13],
];

/// The phase (0..4) of every pipeline stage: stage `s` has phase `s mod 4`.
///
/// Written out as a constant table because RHDL kernels avoid `%`; the
/// pipeline kernel indexes it with the constant stage number, so each stage
/// gets a fixed phase and only that phase's logic survives synthesis.
/// 2-bit values.
const STAGE_PHASES: [b2; PIPELINE_STAGES] = [
    b2(0),
    b2(1),
    b2(2),
    b2(3),
    b2(0),
    b2(1),
    b2(2),
    b2(3),
    b2(0),
    b2(1),
    b2(2),
    b2(3),
    b2(0),
    b2(1),
    b2(2),
    b2(3),
    b2(0),
    b2(1),
    b2(2),
    b2(3),
    b2(0),
    b2(1),
    b2(2),
    b2(3),
    b2(0),
    b2(1),
    b2(2),
    b2(3),
];

/// Performs one software-model G function using wrapping 32-bit arithmetic.
///
/// `indices` names the four state words `(a, b, c, d)` this G mixes; `x`
/// and `y` are its two message words. Returns the whole updated state.
#[allow(clippy::many_single_char_names)]
const fn g(mut words: [u32; 16], indices: [usize; 4], x: u32, y: u32) -> [u32; 16] {
    // Positions of the four words in the 16-word state.
    let [a, b, c, d] = indices;
    // First half (rotations 16 and 12), using message word x.
    words[a] = words[a].wrapping_add(words[b]).wrapping_add(x);
    words[d] = (words[d] ^ words[a]).rotate_right(16);
    words[c] = words[c].wrapping_add(words[d]);
    words[b] = (words[b] ^ words[c]).rotate_right(12);
    // Second half (rotations 8 and 7), using message word y.
    words[a] = words[a].wrapping_add(words[b]).wrapping_add(y);
    words[d] = (words[d] ^ words[a]).rotate_right(8);
    words[c] = words[c].wrapping_add(words[d]);
    words[b] = (words[b] ^ words[c]).rotate_right(7);
    words
}

/// Compresses a single BLAKE3 block, returning all sixteen feed-forward words.
///
/// The first eight words are the next chaining value when `ROOT` is absent;
/// all sixteen words are one output block when `ROOT` is present. For root
/// output, `counter` is the XOF output-block number, not the input chunk number.
/// This low-level function does not validate caller-owned lengths or flags.
///
/// Inputs: `cv` (8 words: the chaining value, or [`IV`] for a first block),
/// `block` (16 message words), `counter` (64-bit chunk or output-block
/// number), `block_len` (bytes used, 0..=64) and `flags`. This is the
/// reference the hardware pipeline is tested against.
#[must_use]
pub const fn compress(
    cv: [u32; 8],
    block: [u32; 16],
    counter: u64,
    block_len: u32,
    flags: u32,
) -> [u32; 16] {
    // The counter split into bytes, to form its low and high 32-bit words.
    let counter_bytes = counter.to_le_bytes();
    // The starting state: cv, four IV words, counter low/high, length, flags.
    let mut words = [
        cv[0],
        cv[1],
        cv[2],
        cv[3],
        cv[4],
        cv[5],
        cv[6],
        cv[7],
        IV[0],
        IV[1],
        IV[2],
        IV[3],
        u32::from_le_bytes([
            counter_bytes[0],
            counter_bytes[1],
            counter_bytes[2],
            counter_bytes[3],
        ]),
        u32::from_le_bytes([
            counter_bytes[4],
            counter_bytes[5],
            counter_bytes[6],
            counter_bytes[7],
        ]),
        block_len,
        flags,
    ];
    let mut round = 0;
    while round < ROUNDS {
        // This round's message word order.
        let schedule = MESSAGE_SCHEDULE[round];
        // Column step: four G functions down the four columns.
        words = g(words, [0, 4, 8, 12], block[schedule[0]], block[schedule[1]]);
        words = g(words, [1, 5, 9, 13], block[schedule[2]], block[schedule[3]]);
        words = g(
            words,
            [2, 6, 10, 14],
            block[schedule[4]],
            block[schedule[5]],
        );
        words = g(
            words,
            [3, 7, 11, 15],
            block[schedule[6]],
            block[schedule[7]],
        );
        // Diagonal step: four G functions along the diagonals.
        words = g(
            words,
            [0, 5, 10, 15],
            block[schedule[8]],
            block[schedule[9]],
        );
        words = g(
            words,
            [1, 6, 11, 12],
            block[schedule[10]],
            block[schedule[11]],
        );
        words = g(
            words,
            [2, 7, 8, 13],
            block[schedule[12]],
            block[schedule[13]],
        );
        words = g(
            words,
            [3, 4, 9, 14],
            block[schedule[14]],
            block[schedule[15]],
        );
        round += 1;
    }
    // Feed-forward: low half ^= high half; high half ^= original cv.
    let mut index = 0;
    while index < 8 {
        words[index] ^= words[index + 8];
        words[index + 8] ^= cv[index];
        index += 1;
    }
    words
}

/// Decodes a complete 64-byte block as sixteen little-endian words.
#[must_use]
pub const fn decode_block(bytes: [u8; BLOCK_BYTES]) -> [u32; 16] {
    // The 16 decoded words.
    let mut words = [0; 16];
    let mut index = 0;
    while index < 16 {
        // First byte of word `index`.
        let offset = index * 4;
        words[index] = u32::from_le_bytes([
            bytes[offset],
            bytes[offset + 1],
            bytes[offset + 2],
            bytes[offset + 3],
        ]);
        index += 1;
    }
    words
}

/// Encodes sixteen compression-output words as 64 little-endian XOF bytes.
#[must_use]
pub const fn encode_words(words: [u32; 16]) -> [u8; OUTPUT_BYTES] {
    // The 64 output bytes.
    let mut bytes = [0; OUTPUT_BYTES];
    let mut index = 0;
    while index < 16 {
        // Word `index` as four little-endian bytes.
        let encoded = words[index].to_le_bytes();
        // Where those bytes go.
        let offset = index * 4;
        bytes[offset] = encoded[0];
        bytes[offset + 1] = encoded[1];
        bytes[offset + 2] = encoded[2];
        bytes[offset + 3] = encoded[3];
        index += 1;
    }
    bytes
}

/// Constructs the exact single block `pk_hash || nonce || message`.
///
/// ```text
///   byte 0           32         48          64
///        ├───────────┼──────────┼───────────┤
///        │ pk_hash   │ nonce    │ message   │   exactly one full block,
///        │ 32 bytes  │ 16 bytes │ 16 bytes  │   no padding needed
///        └───────────┴──────────┴───────────┘
/// ```
#[must_use]
pub const fn salted_block_16(
    pk_hash: [u8; 32],
    nonce: [u8; 16],
    message: [u8; 16],
) -> [u8; BLOCK_BYTES] {
    let mut block = [0; BLOCK_BYTES];
    // Bytes 0..32: the public-key hash.
    let mut index = 0;
    while index < 32 {
        block[index] = pk_hash[index];
        index += 1;
    }
    // Bytes 32..48: the nonce; bytes 48..64: the message.
    index = 0;
    while index < 16 {
        block[index + 32] = nonce[index];
        block[index + 48] = message[index];
        index += 1;
    }
    block
}

/// Returns 64 BLAKE3 XOF bytes for the fixed 16-byte online-message profile.
///
/// The input is exactly one full block, so flags are `CHUNK_START | CHUNK_END |
/// ROOT` (11), the block length is 64, and the output-block counter is zero.
///
/// The software reference for [`salted_input_kernel`] followed by the
/// pipeline.
#[must_use]
pub const fn salted_digest_16(
    pk_hash: [u8; 32],
    nonce: [u8; 16],
    message: [u8; 16],
) -> [u8; OUTPUT_BYTES] {
    encode_words(compress(
        IV,
        decode_block(salted_block_16(pk_hash, nonce, message)),
        0,
        64,
        SINGLE_BLOCK_ROOT,
    ))
}

/// One compression request; all fields except `valid` are ignored in bubbles.
///
/// Everything [`compress`] takes, as RHDL bit vectors, plus a tag and a
/// valid bit. 8 + 16 + 5 words of 32 bits and one bit of valid.
#[derive(Clone, Copy, Debug, Default, Digital, Eq, PartialEq)]
pub struct CompressionInput {
    /// Original chaining value, or IV/key words for a first block.
    pub cv: [b32; 8],
    /// Zero-padded message block decoded as little-endian words.
    pub block: [b32; 16],
    /// Low 32 bits of the chunk counter, or root XOF output-block counter.
    pub counter_low: b32,
    /// High 32 bits of the chunk counter, or root XOF output-block counter.
    pub counter_high: b32,
    /// Number of meaningful bytes in the input block, in `0..=64`.
    pub block_len: b32,
    /// Caller-supplied BLAKE3 compression flags.
    pub flags: b32,
    /// Opaque caller tag, preserved without interpretation.
    pub tag: b32,
    /// Marks a real input token; false inserts a pipeline bubble.
    pub valid: bool,
}

/// Host-side helpers for building requests in tests and models.
impl CompressionInput {
    /// Encodes a valid request from ordinary Rust integer words.
    ///
    /// The 64-bit `counter` is split into `counter_low` and `counter_high`.
    #[must_use]
    pub fn new(
        cv: [u32; 8],
        block: [u32; 16],
        counter: u64,
        block_len: u32,
        flags: u32,
        tag: u32,
    ) -> Self {
        Self {
            cv: cv.map(|word| b32(u128::from(word))),
            block: block.map(|word| b32(u128::from(word))),
            counter_low: b32(u128::from(counter & 0xffff_ffff)),
            counter_high: b32(u128::from(counter >> 32)),
            block_len: b32(u128::from(block_len)),
            flags: b32(u128::from(flags)),
            tag: b32(u128::from(tag)),
            valid: true,
        }
    }

    /// Prepares a valid fixed-size salted-message request.
    ///
    /// The same request [`salted_input_kernel`] builds in hardware.
    #[must_use]
    pub fn salted_16(pk_hash: [u8; 32], nonce: [u8; 16], message: [u8; 16], tag: u32) -> Self {
        Self::new(
            IV,
            decode_block(salted_block_16(pk_hash, nonce, message)),
            0,
            64,
            SINGLE_BLOCK_ROOT,
            tag,
        )
    }
}

/// Compression result; `words` and `tag` are meaningful only when valid.
///
/// Appears exactly [`PIPELINE_STAGES`] (28) clocks after its input.
#[derive(Clone, Copy, Debug, Default, Digital, Eq, PartialEq)]
pub struct CompressionOutput {
    /// Sixteen output words after both BLAKE3 XOR feed-forward operations.
    pub words: [b32; 16],
    /// Unmodified tag belonging to this result.
    pub tag: b32,
    /// True only for an input token that survived reset and all stages.
    pub valid: bool,
}

/// Host-side helpers for reading results.
impl CompressionOutput {
    /// Converts the datapath words into 64 little-endian bytes.
    ///
    /// Callers must check `valid` before using the returned data.
    #[must_use]
    pub fn to_bytes(self) -> [u8; OUTPUT_BYTES] {
        encode_words(self.words.map(bits_to_word))
    }
}

/// Convert an RHDL 32-bit value to a Rust `u32` (host side only).
fn bits_to_word(value: b32) -> u32 {
    // `raw()` is a u128; keep its four low bytes.
    let bytes = value.raw().to_le_bytes();
    u32::from_le_bytes([bytes[0], bytes[1], bytes[2], bytes[3]])
}

/// Resetless token carried between the quarter-round stages.
///
/// 40 words of 32 bits (1,280 bits) before synthesis optimizes it. Each of
/// the 28 stages holds one of these.
#[derive(Clone, Copy, Debug, Default, Digital, Eq, PartialEq)]
pub struct RoundState {
    /// Sixteen working-state words.
    pub words: [b32; 16],
    /// Message schedule, permuted after each complete round.
    pub message: [b32; 16],
    /// Original chaining value needed for the second eight output words.
    pub cv: [b32; 8],
}

/// Four words manipulated by one BLAKE3 G function.
///
/// One column (or diagonal) of the 4 × 4 state. Exists so the two G-half
/// kernels can take and return the four words as one value.
#[derive(Clone, Copy, Debug, Default, Digital, Eq, PartialEq)]
pub struct GState {
    /// First word, receiving the message-word additions.
    pub a: b32,
    /// Second word, rotated by 12 or 7 bits.
    pub b: b32,
    /// Third word, receiving additions from the fourth word.
    pub c: b32,
    /// Fourth word, rotated by 16 or 8 bits.
    pub d: b32,
}

/// Synthesizable first half of G: rotations by 16 and 12 bits.
///
/// Inputs: the four words and one 32-bit message word. Output: the four
/// updated words. Combinational: a 3-input add, then an XOR, then an add,
/// then an XOR. Rotations are pure wiring.
#[kernel]
pub fn g_first_kernel(input: GState, message: b32) -> GState {
    // a = a + b + m (wraps at 2^32).
    let a = input.a + input.b + message;
    // d = (d ⊕ a) rotated right by 16.
    let mixed_d = input.d ^ a;
    let d = (mixed_d >> 16) | (mixed_d << 16);
    // c = c + d.
    let c = input.c + d;
    // b = (b ⊕ c) rotated right by 12 (= right 12 OR left 20).
    let mixed_b = input.b ^ c;
    let b = (mixed_b >> 12) | (mixed_b << 20);
    GState { a, b, c, d }
}

/// Synthesizable second half of G: rotations by 8 and 7 bits.
///
/// Same shape as [`g_first_kernel`] with the second message word and the
/// other two rotation amounts.
#[kernel]
pub fn g_second_kernel(input: GState, message: b32) -> GState {
    // a = a + b + m.
    let a = input.a + input.b + message;
    // d = (d ⊕ a) rotated right by 8.
    let mixed_d = input.d ^ a;
    let d = (mixed_d >> 8) | (mixed_d << 24);
    // c = c + d.
    let c = input.c + d;
    // b = (b ⊕ c) rotated right by 7.
    let mixed_b = input.b ^ c;
    let b = (mixed_b >> 7) | (mixed_b << 25);
    GState { a, b, c, d }
}

/// Constructs the round-zero working state without modifying input words.
///
/// Pure wiring plus four constant words ([`IV`] words 0..4, written out as
/// literals). Combinational; feeds stage 0 directly.
#[kernel]
pub fn initialize_kernel(input: CompressionInput) -> RoundState {
    // Words 0..8: cv; 8..12: IV; 12, 13: counter; 14: length; 15: flags.
    RoundState {
        words: [
            input.cv[0],
            input.cv[1],
            input.cv[2],
            input.cv[3],
            input.cv[4],
            input.cv[5],
            input.cv[6],
            input.cv[7],
            b32(0x6a09_e667),
            b32(0xbb67_ae85),
            b32(0x3c6e_f372),
            b32(0xa54f_f53a),
            input.counter_low,
            input.counter_high,
            input.block_len,
            input.flags,
        ],
        message: input.block,
        cv: input.cv,
    }
}

/// Executes one quarter-round with compile-time phase selection in the pipeline.
///
/// Phases 0/1 are first/second halves of the four column G functions. Phases
/// 2/3 are first/second halves of the diagonal G functions. Phase 3 also
/// rewires the message schedule for the next round. All state and schedule
/// selections use constant indices; no data-dependent indexing is needed.
///
/// Inputs: the stage's incoming [`RoundState`] and its 2-bit `phase`.
/// Output: the state after this half-step. Combinational; the pipeline
/// registers the result. In the pipeline `phase` is always a constant, so
/// three of the four branches are removed by synthesis and each stage holds
/// only four G halves.
///
/// | Phase | G inputs `(a, b, c, d)` | Message words |
/// |---|---|---|
/// | 0 | columns, first half | 0, 2, 4, 6 |
/// | 1 | columns, second half | 1, 3, 5, 7 |
/// | 2 | diagonals, first half | 8, 10, 12, 14 |
/// | 3 | diagonals, second half (+ permute message) | 9, 11, 13, 15 |
#[allow(clippy::many_single_char_names)]
#[kernel]
pub fn quarter_stage_kernel(input: RoundState, phase: b2) -> RoundState {
    // Short names: `w` the 16 state words, `m` the 16 message words.
    let w = input.words;
    let m = input.message;
    // Copy everything (message, cv); the branches overwrite what changes.
    let mut output = input;
    // `p`, `q`, `r`, `s` below are the four G functions of this half-step.
    if phase == b2(0) {
        // Columns, first half.
        let p = g_first_kernel(
            GState {
                a: w[0],
                b: w[4],
                c: w[8],
                d: w[12],
            },
            m[0],
        );
        let q = g_first_kernel(
            GState {
                a: w[1],
                b: w[5],
                c: w[9],
                d: w[13],
            },
            m[2],
        );
        let r = g_first_kernel(
            GState {
                a: w[2],
                b: w[6],
                c: w[10],
                d: w[14],
            },
            m[4],
        );
        let s = g_first_kernel(
            GState {
                a: w[3],
                b: w[7],
                c: w[11],
                d: w[15],
            },
            m[6],
        );
        // Put each column's words back in their column.
        output.words = [
            p.a, q.a, r.a, s.a, p.b, q.b, r.b, s.b, p.c, q.c, r.c, s.c, p.d, q.d, r.d, s.d,
        ];
    } else if phase == b2(1) {
        // Columns, second half.
        let p = g_second_kernel(
            GState {
                a: w[0],
                b: w[4],
                c: w[8],
                d: w[12],
            },
            m[1],
        );
        let q = g_second_kernel(
            GState {
                a: w[1],
                b: w[5],
                c: w[9],
                d: w[13],
            },
            m[3],
        );
        let r = g_second_kernel(
            GState {
                a: w[2],
                b: w[6],
                c: w[10],
                d: w[14],
            },
            m[5],
        );
        let s = g_second_kernel(
            GState {
                a: w[3],
                b: w[7],
                c: w[11],
                d: w[15],
            },
            m[7],
        );
        output.words = [
            p.a, q.a, r.a, s.a, p.b, q.b, r.b, s.b, p.c, q.c, r.c, s.c, p.d, q.d, r.d, s.d,
        ];
    } else if phase == b2(2) {
        // Diagonals, first half: p = (0,5,10,15), q = (1,6,11,12),
        // r = (2,7,8,13), s = (3,4,9,14).
        let p = g_first_kernel(
            GState {
                a: w[0],
                b: w[5],
                c: w[10],
                d: w[15],
            },
            m[8],
        );
        let q = g_first_kernel(
            GState {
                a: w[1],
                b: w[6],
                c: w[11],
                d: w[12],
            },
            m[10],
        );
        let r = g_first_kernel(
            GState {
                a: w[2],
                b: w[7],
                c: w[8],
                d: w[13],
            },
            m[12],
        );
        let s = g_first_kernel(
            GState {
                a: w[3],
                b: w[4],
                c: w[9],
                d: w[14],
            },
            m[14],
        );
        // Put each diagonal's words back where they came from (e.g. p.b
        // goes to word 5, s.b to word 4).
        output.words = [
            p.a, q.a, r.a, s.a, s.b, p.b, q.b, r.b, r.c, s.c, p.c, q.c, q.d, r.d, s.d, p.d,
        ];
    } else {
        // Phase 3: diagonals, second half.
        let p = g_second_kernel(
            GState {
                a: w[0],
                b: w[5],
                c: w[10],
                d: w[15],
            },
            m[9],
        );
        let q = g_second_kernel(
            GState {
                a: w[1],
                b: w[6],
                c: w[11],
                d: w[12],
            },
            m[11],
        );
        let r = g_second_kernel(
            GState {
                a: w[2],
                b: w[7],
                c: w[8],
                d: w[13],
            },
            m[13],
        );
        let s = g_second_kernel(
            GState {
                a: w[3],
                b: w[4],
                c: w[9],
                d: w[14],
            },
            m[15],
        );
        // Diagonal words back to their positions, as in phase 2.
        output.words = [
            p.a, q.a, r.a, s.a, s.b, p.b, q.b, r.b, r.c, s.c, p.c, q.c, q.d, r.d, s.d, p.d,
        ];
        // End of the round: permute the message words (MESSAGE_PERMUTATION)
        // so the next round sees them in its order. Wiring only.
        output.message = [
            m[2], m[6], m[3], m[10], m[7], m[0], m[4], m[13], m[1], m[11], m[12], m[5], m[9],
            m[14], m[15], m[8],
        ];
    }
    output
}

/// Applies BLAKE3's final XORs, without any additional registered transfer.
///
/// Input: the state leaving the last stage. Output: the 16 result words,
/// `out[i] = v[i] ⊕ v[i+8]` and `out[i+8] = v[i+8] ⊕ cv[i]` for `i < 8`.
/// Combinational, on the output side of the last register.
#[allow(clippy::needless_range_loop)]
#[kernel]
pub fn feed_forward_kernel(state: RoundState) -> [b32; 16] {
    // Placeholder value; every word is overwritten.
    let mut words = state.words;
    for index in 0..8 {
        words[index] = state.words[index] ^ state.words[index + 8];
        words[index + 8] = state.words[index + 8] ^ state.cv[index];
    }
    words
}

/// Builds the fixed 16-byte-message input directly from RHDL word arrays.
///
/// Every word is little-endian. This adapter is wiring and constants only;
/// unlike a software prehash, it preserves the complete defined message.
///
/// Inputs: the 8-word public-key hash, 4-word nonce, 4-word message, a tag
/// and a valid bit. Output: a [`CompressionInput`] equal to
/// [`CompressionInput::salted_16`].
#[kernel]
pub fn salted_input_kernel(
    pk_hash: [b32; 8],
    nonce: [b32; 4],
    message: [b32; 4],
    tag: b32,
    valid: bool,
) -> CompressionInput {
    CompressionInput {
        // cv = IV (an unkeyed hash's first block), written as literals.
        cv: [
            b32(0x6a09_e667),
            b32(0xbb67_ae85),
            b32(0x3c6e_f372),
            b32(0xa54f_f53a),
            b32(0x510e_527f),
            b32(0x9b05_688c),
            b32(0x1f83_d9ab),
            b32(0x5be0_cd19),
        ],
        // block = pk_hash || nonce || message (see salted_block_16).
        block: [
            pk_hash[0], pk_hash[1], pk_hash[2], pk_hash[3], pk_hash[4], pk_hash[5], pk_hash[6],
            pk_hash[7], nonce[0], nonce[1], nonce[2], nonce[3], message[0], message[1], message[2],
            message[3],
        ],
        // Counter 0, a full 64-byte block, flags SINGLE_BLOCK_ROOT (11).
        counter_low: b32(0),
        counter_high: b32(0),
        block_len: b32(64),
        flags: b32(11),
        tag,
        valid,
    }
}

/// Registered prefix of the BLAKE3 compression pipeline.
///
/// Only `STAGES = 28` produces complete BLAKE3 output. Smaller specializations
/// exist for bounded lowering and RTL-equivalence tests; they return XORs of
/// incomplete working states and must **not** be used as a hash. The complete
/// specialization is [`CompressionPipeline`].
///
/// The initiation interval is one cycle, with `STAGES` registered transfers
/// of latency. Data and tags do not reset. Synchronous reset clears all valid
/// bits and therefore flushes every in-flight token.
///
/// ```text
///   stage 0        stage 1    stage 2    stage 3          stage 27
///   init+phase 0 ─▶ phase 1 ─▶ phase 2 ─▶ phase 3 ─▶ ... ─▶ phase 3 ─▶ feed_forward ─▶ output
///   (reg)           (reg)      (reg)      (reg)            (reg)      (no register)
///
///   tag, valid: copied one stage per clock beside the data
/// ```
///
/// `#[rhdl(dq_no_prefix)]` names the generated input/output bundles plain
/// `D` and `Q` (instead of `RoundPipelineD`/`RoundPipelineQ`); the kernel
/// below uses those names.
#[derive(Clone, Debug, Synchronous, SynchronousDQ)]
#[rhdl(dq_no_prefix)]
pub struct RoundPipeline<const STAGES: usize>
where
    rhdl::bits::W<STAGES>: BitWidth,
{
    /// The working state, message schedule and cv after each stage.
    data: [DFF<RoundState>; STAGES],
    /// The caller's tag, delayed to match the data.
    tag: [DFF<b32>; STAGES],
    /// "This stage holds a real token." The only registers that reset.
    valid: [DFF<bool>; STAGES],
}

/// Build a pipeline of `STAGES` stages, checking that `STAGES` is legal.
impl<const STAGES: usize> Default for RoundPipeline<STAGES>
where
    rhdl::bits::W<STAGES>: BitWidth,
{
    /// Data registers start as "don't care"; every valid bit starts false.
    ///
    /// # Panics
    /// Panics if `STAGES` is 0 or more than 28.
    fn default() -> Self {
        assert!(
            STAGES > 0 && STAGES <= PIPELINE_STAGES,
            "invalid BLAKE3 stage count"
        );
        Self {
            data: core::array::from_fn(|_| DFF::resetless()),
            tag: core::array::from_fn(|_| DFF::resetless()),
            valid: core::array::from_fn(|_| DFF::new(false)),
        }
    }
}

/// Ports of [`RoundPipeline`]: one request in, one result out per clock.
impl<const STAGES: usize> SynchronousIO for RoundPipeline<STAGES>
where
    rhdl::bits::W<STAGES>: BitWidth,
{
    /// A compression request.
    type I = CompressionInput;
    /// Its result, `STAGES` clocks later.
    type O = CompressionOutput;
    /// The stage wiring below.
    type Kernel = round_pipeline_kernel<STAGES>;
}

/// Connects fixed-phase stages and aligns resettable validity with data/tag.
///
/// Inputs: `clock_reset`, the new request `input`, and `q` (every register's
/// current value). Outputs: the result leaving the last stage and `d` (every
/// register's next value). Timing: a request entering at clock `t` leaves at
/// clock `t + STAGES`, and a new one may enter every clock.
#[allow(clippy::needless_range_loop)]
#[kernel]
pub fn round_pipeline_kernel<const STAGES: usize>(
    clock_reset: ClockReset,
    input: CompressionInput,
    q: Q<STAGES>,
) -> (CompressionOutput, D<STAGES>)
where
    rhdl::bits::W<STAGES>: BitWidth,
{
    // True while reset is asserted; hides the output during reset.
    let resetting = clock_reset.reset.any();
    // Start with "every register keeps its value"; each stage is set below.
    let mut d = D::<STAGES> {
        data: q.data,
        tag: q.tag,
        valid: q.valid,
    };
    // Stage 0: build the starting state and run phase 0 on it.
    d.data[0] = quarter_stage_kernel(initialize_kernel(input), b2(0));
    d.tag[0] = input.tag;
    d.valid[0] = input.valid;
    // Stages 1..STAGES: each runs its fixed phase on the previous stage.
    for stage in 1..STAGES {
        d.data[stage] = quarter_stage_kernel(q.data[stage - 1], STAGE_PHASES[stage]);
        d.tag[stage] = q.tag[stage - 1];
        d.valid[stage] = q.valid[stage - 1];
    }
    // Output: feed-forward on the last register (combinational), with
    // `valid` forced low during reset.
    (
        CompressionOutput {
            words: feed_forward_kernel(q.data[STAGES - 1]),
            tag: q.tag[STAGES - 1],
            valid: q.valid[STAGES - 1] && !resetting,
        },
        d,
    )
}

/// All seven BLAKE3 rounds, 28-cycle latency, and one input per clock.
///
/// The one specialization of [`RoundPipeline`] that computes real BLAKE3.
pub type CompressionPipeline = RoundPipeline<PIPELINE_STAGES>;

/// Copyable, zero-sized adapter for embedding the complete pipeline in a parent.
///
/// Pinned RHDL derives conservatively require injected child types to be
/// `Copy`. This adapter delegates state, native simulation, and elaboration to
/// the exact [`CompressionPipeline`]; it adds no datapath or authored HDL.
///
/// Why it is safe to build a fresh `CompressionPipeline` in every call:
/// every fresh one is identical (same stages, same reset values). The
/// register values themselves live in the simulation state `S`, which the
/// caller owns and passes in.
#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
pub struct InlineCompressionPipeline;

/// No RHDL-visible children; everything is delegated.
impl SynchronousDQ for InlineCompressionPipeline {
    /// Nothing to drive.
    type D = ();
    /// Nothing to read.
    type Q = ();
}

/// Same ports as [`CompressionPipeline`].
impl SynchronousIO for InlineCompressionPipeline {
    /// A compression request.
    type I = CompressionInput;
    /// Its result, 28 clocks later.
    type O = CompressionOutput;
    /// No kernel of its own; see the hand-written [`Synchronous`] impl.
    type Kernel = NoSynchronousKernel<ClockReset, CompressionInput, (), (CompressionOutput, ())>;
}

/// Every method forwards to a [`CompressionPipeline`].
impl Synchronous for InlineCompressionPipeline {
    /// The full pipeline's simulation state.
    type S = <CompressionPipeline as Synchronous>::S;

    /// The full pipeline's power-up state.
    fn init(&self) -> Self::S {
        CompressionPipeline::default().init()
    }

    /// One simulation step of the full pipeline.
    fn sim(&self, clock_reset: ClockReset, input: Self::I, state: &mut Self::S) -> Self::O {
        CompressionPipeline::default().sim(clock_reset, input, state)
    }

    /// The full pipeline's hardware description, so the Verilog is identical.
    fn descriptor(&self, scoped_name: ScopedName) -> Result<Descriptor<SyncKind>, RHDLError> {
        CompressionPipeline::default().descriptor(scoped_name)
    }
}

/// Emits a checked BLAKE3 pipeline or small test prefix as Verilog.
///
/// # Errors
///
/// Returns RHDL lowering/rendering errors or an error from its external syntax
/// checker. A successful string is not a synthesis, timing, or hardware result.
pub fn round_pipeline_verilog<const STAGES: usize>() -> Result<String, RHDLError>
where
    rhdl::bits::W<STAGES>: BitWidth,
{
    // Build, describe under a fixed module name, lower to HDL, print.
    Ok(RoundPipeline::<STAGES>::default()
        .descriptor("blake3_compression_pipeline".into())?
        .hdl()?
        .modules
        .pretty())
}

/// Emits the complete checked 28-stage BLAKE3 compression hierarchy.
///
/// # Errors
///
/// Propagates lowering, rendering, and external syntax-check errors.
pub fn compression_pipeline_verilog() -> Result<String, RHDLError> {
    round_pipeline_verilog::<PIPELINE_STAGES>()
}

/// Renders the complete pipeline without invoking RHDL's external syntax check.
///
/// This is only generation evidence. Before FPGA use, an independent HDL
/// parser, equivalence simulation, and synthesis must accept the exact bytes.
///
/// # Errors
///
/// Propagates lowering and rendering errors.
pub fn compression_pipeline_unchecked_verilog() -> Result<String, RHDLError> {
    // As round_pipeline_verilog, but `hdl_unchecked` skips the syntax check.
    Ok(CompressionPipeline::default()
        .descriptor("blake3_compression_pipeline".into())?
        .hdl_unchecked()?
        .modules
        .pretty())
}