rhdl_dsig_scalable_core/

aes_table.rs

//! AES T-table ROM: one BRAM port replaces one S-box and its MixColumns share.
//!
//! # The story so far
//!
//! A DSig signature needs 204 Haraka chain steps, and each folded chain step
//! needs 494 AES S-box evaluations (see [`crate::chain_step`]). An *S-box*
//! ("substitution box") is the one non-linear part of AES: a fixed table that
//! maps every byte value `0..256` to another byte. Everything else in an AES
//! round is XOR and wiring. So the S-boxes are where the chip area goes, and
//! *where each S-box physically lives* is the biggest area decision in the
//! signer.
//!
//! This module owns that physical side. It builds every S-box and every
//! T-table as a read-only memory (ROM) out of RHDL parts, and leaves the
//! choice of memory type to [`crate::sbox_placement`]. Its neighbours:
//!
//! ```text
//!   crate::haraka          the math: what an AES round computes (reference + kernels)
//!        │
//!        ▼
//!   crate::aes_table       this file: the S-box / T-table ROMs, registered
//!        │                 (SboxStage, SboxStages, SboxLoader, DualTableRom)
//!        ▼
//!   crate::chain_step      the lane: 20 stages that use these ROMs every cycle
//!        ▲
//!   crate::sbox_placement  which of the 494 live boxes are UltraRAM / BRAM / LUT
//! ```
//!
//! # One AES round, and which part this file builds
//!
//! An AES round (Intel's `AESENC` instruction) runs on a 16-byte state, laid out
//! as four columns of four bytes. It has four steps:
//!
//! ```text
//!   state ──▶ SubBytes ──▶ ShiftRows ──▶ MixColumns ──▶ AddRoundKey ──▶ state'
//!             (16 S-box    (row r moves   (each column   (XOR a 16-byte
//!              lookups)     left r places) times a fixed  round constant)
//!                                          4×4 matrix)
//!             └─── non-linear ───┘└───────────── linear (XOR only) ─────────┘
//!
//!   In this crate:  SubBytes+ShiftRows  = SboxStage   (one register stage)
//!                   MixColumns+AddKey   = mix_key512_kernel in crate::haraka
//!                                         (the next register stage)
//! ```
//!
//! ShiftRows is only a re-ordering of bytes, so it costs no logic: it just
//! decides *which* input byte each S-box reads. That is why [`SboxStage`]
//! does SubBytes and ShiftRows together.
//!
//! # The idea in one picture
//!
//! An AES round turns each four-byte column `(s0, s1, s2, s3)` into
//!
//! ```text
//!   out = 2·S(s0) ⊕ 3·S(s1) ⊕ S(s2) ⊕ S(s3)       (row 0, and rotations)
//! ```
//!
//! Because MixColumns is linear, each input byte's contribution to the whole
//! output column depends on that byte alone. Precompute it once:
//!
//! ```text
//!   T0[x] = bytes (2·S(x), S(x), S(x), 3·S(x))      — a 256 × 32-bit table
//!   out   = T0[s0] ⊕ rotl8(T0[s1]) ⊕ rotl16(T0[s2]) ⊕ rotl24(T0[s3]) ⊕ key
//! ```
//!
//! In LUT logic an S-box costs about 32 LUTs. As a table it costs one read
//! port of a block RAM, and the only LUTs left are one 5-input XOR per output
//! bit. See [`sustained`](crate::sustained) for the budget.
//!
//! # Built only from RHDL parts
//!
//! Every table here is an [`rhdl_fpga`] [`SyncBRAM`] that is filled at
//! configuration time and never written, i.e. a ROM with a registered read:
//!
//! | Item | Made of | Vivado maps it to |
//! |---|---|---|
//! | [`SboxStage`] | 32 × [`SboxPair`] (two S-box lookups each, [`SboxStageRoms`]) | LUT ROMs (≈32 LUTs per lookup), or one BRAM18 / URAM per pair |
//! | [`DualTableRom`] | one dual-port `256 × 32` ROM holding [`T0`] | one BRAM18, two reads |
//!
//! The ROM registers its output like any synchronous BRAM read. In a pipeline
//! that register *is* the stage register, so a table round has the same
//! latency as a LUT round (see [`chain_step`](crate::chain_step)).
//!
//! This is the textbook "T-table" software AES trick moved to FPGA block RAM.
//! In software it leaks timing through caches. A dedicated ROM port has fixed
//! single-cycle latency regardless of address, so that concern does not apply.
//!
//! # Three places an S-box can live
//!
//! The same 256 × 8 ROM can be built three ways on the U280. The numbers are
//! the ones in [`crate::sbox_placement`] and [`sustained`](crate::sustained):
//!
//! | Place | Cost per S-box | Filled how |
//! |---|---|---|
//! | LUT ROM (the default) | ~32 CLB LUTs | constant logic, nothing to fill |
//! | Block RAM | ½ BRAM18 (an [`SboxPair`] serves two boxes) | preloaded by the bitstream |
//! | UltraRAM | ½ URAM (an [`SboxPair`] serves two boxes) | **cannot be preloaded**: [`SboxLoader`] writes it after reset |
//!
//! UltraRAM is the odd one out. Because the bitstream cannot fill it, every
//! UltraRAM S-box needs a write port, and the lane needs a small loader that
//! runs after every reset. Until the loader is done the lane reports
//! "not ready" ([`ChainStepOutput::ready`](crate::chain_step::ChainStepOutput::ready)),
//! and nothing upstream starts a key. That keeps the one-time-key rule safe:
//! no key is ever spent on a lane whose tables are still empty.
//!
//! # Reading order
//!
//! 1. [`T0`] and [`DualTableRom`]: the T-table trick (used by
//!    [`TableChainStepLane`](crate::chain_step::TableChainStepLane)).
//! 2. [`SboxPair`] and [`SboxStageRoms`]: 32 dual-port S-box pairs for one
//!    round, wired through ShiftRows.
//! 3. [`SboxStage`]: the same hardware, plus a fast simulation model.
//! 4. [`SboxStages`]: ten of them, one per AES round of a Haraka-512 lane.
//! 5. [`SboxLoader`]: fills the UltraRAM boxes after reset.

use crate::haraka::aes_sbox;
use rhdl::{
    core::{ScopedName, circuit::descriptor::SyncKind},
    prelude::*,
};
use crate::haraka::AES_ROUNDS;
use crate::sbox_placement::{BLOCK_MASK, LANE_NEEDS_LOAD, ULTRA_MASK};
use rhdl_fpga::core::{
    dff::DFF,
    ram::synchronous::{In as RamIn, SyncBRAM, Write as RamWrite},
};

/// Multiply a byte by 2 in the AES field GF(2^8) (compile-time helper).
///
/// "GF(2^8)" is the AES byte arithmetic: addition is XOR, and multiplying by
/// 2 is a left shift that, if the top bit falls off, XORs in `0x1b` (the AES
/// reduction polynomial `x^8 + x^4 + x^3 + x + 1`). MixColumns only ever
/// multiplies by 1, 2 and 3, and `3·s = 2·s ⊕ s`, so this is all [`T0`] needs.
const fn xtime(value: u8) -> u8 {
    (value << 1) ^ if value & 0x80 == 0 { 0 } else { 0x1b }
}

/// Build [`T0`] at compile time: for every byte `x`, pack the four MixColumns
/// multiples of `S(x)` into one 32-bit word.
const fn t0_table() -> [u32; 256] {
    // The finished 256-entry table, one 32-bit word per possible input byte.
    let mut table = [0; 256];
    // The input byte being tabulated (a `usize` so it can index `table`).
    let mut x = 0;
    while x < 256 {
        // `S(x)`: the AES S-box output for this input byte.
        let s = aes_sbox(x as u8);
        // `2·S(x)` in GF(2^8).
        let two = xtime(s);
        // `3·S(x) = 2·S(x) ⊕ S(x)`.
        let three = two ^ s;
        // Little-endian rows: row 0 in bits 0..8, matching the column layout.
        table[x] = two as u32 | (s as u32) << 8 | (s as u32) << 16 | (three as u32) << 24;
        x += 1;
    }
    table
}

/// `T0[x] = (2·S(x), S(x), S(x), 3·S(x))`, row 0 in the low byte.
///
/// Row `j` of a column uses `T0[x].rotate_left(8 * j)`.
///
/// 256 entries × 32 bits = 1 KiB. It is the contents of every
/// [`DualTableRom`] port. Why it exists: one lookup returns an S-box result
/// already multiplied by every MixColumns coefficient it will ever need, so a
/// table round needs no separate S-box and almost no MixColumns logic.
pub const T0: [u32; 256] = t0_table();

/// Two independent byte addresses presented in the same cycle.
///
/// One [`DualTableRom`] serves two of a round's 64 state bytes, so a table
/// round has 32 of these (see
/// [`table_addresses_kernel`](crate::chain_step::table_addresses_kernel)).
#[derive(Clone, Copy, Debug, PartialEq, Digital, Default)]
pub struct TableRead {
    /// Port A address: an 8-bit state byte, used as the `T0` index.
    pub a: b8,
    /// Port B address: an 8-bit state byte, used as the `T0` index.
    pub b: b8,
}

/// `T0` entries for the addresses presented one cycle earlier.
///
/// The one-cycle delay is the BRAM's registered read. In the pipeline that
/// register doubles as the stage register, so it costs no extra latency.
#[derive(Clone, Copy, Debug, PartialEq, Digital, Default)]
pub struct TableWords {
    /// `T0[a]` from the previous cycle (32 bits: four MixColumns multiples).
    pub a: b32,
    /// `T0[b]` from the previous cycle (32 bits: four MixColumns multiples).
    pub b: b32,
}

/// A read-only `SyncBRAM` input: read `address`, never write.
///
/// RHDL's `SyncBRAM` always has a write port. A ROM is simply a RAM whose
/// write enable is tied to `false`. Synthesis sees the constant and removes
/// the write logic. Input: an 8-bit address. Output: the RAM input bundle
/// for a 256 × 32-bit memory. Purely combinational; the read result appears
/// on the RAM's output one clock later.
#[kernel]
pub fn rom_read8_kernel(address: b8) -> RamIn<b32, 8> {
    RamIn::<b32, 8> {
        read_addr: address,
        // Write port tied off: address 0, value 0, never enabled.
        write: RamWrite::<b32, 8> {
            addr: b8(0),
            value: b32(0),
            enable: false,
        },
    }
}

/// Like [`rom_read8_kernel`] for a byte-wide ROM.
///
/// Input: the 8-bit byte to substitute. Output: the RAM input bundle for a
/// 256 × 8-bit S-box memory, with the write port disabled. An UltraRAM box
/// later overwrites the `write` field with the loader's write (see
/// [`sbox_stage_kernel`]); every other box keeps it disabled.
#[kernel]
pub fn sbox_read_kernel(address: b8) -> RamIn<b8, 8> {
    RamIn::<b8, 8> {
        read_addr: address,
        // Write port tied off; only UltraRAM boxes replace this.
        write: RamWrite::<b8, 8> {
            addr: b8(0),
            value: b8(0),
            enable: false,
        },
    }
}

/// A fresh AES S-box ROM (contents preloaded; LUTs unless styled).
///
/// A 256 × 8-bit `SyncBRAM` holding `S(x)` at address `x`. Without a
/// `RamStyle`, Vivado turns a memory this small into LUT logic (~32 LUTs).
fn sbox_rom() -> SyncBRAM<b8, 8> {
    // Pair every address `x` with the S-box output `S(x)`.
    SyncBRAM::new((0..256_u128).map(|x| (b8(x), b8(u128::from(aes_sbox(x as u8))))))
}

/// Two registered `T0` lookups per cycle from **one** true dual-port ROM.
///
/// # Why this is a hand-written RHDL primitive
///
/// A block RAM tile on the U280 (a BRAM18) has two independent read ports.
/// RHDL's own `SyncBRAM` exposes only one of them, so the first version of
/// this type was two `SyncBRAM`s holding the same table: one BRAM18 *per
/// lookup*. Build v4 (four slices, T-tables in rounds 3–8) needed 4,033
/// BRAM18 sites in the kernel region, which has 3,552, and failed placement.
/// Serving both lookups from one memory halves that: a table round needs 32
/// BRAM18s instead of 64.
///
/// RHDL builds its memories the same way this type does: `SyncBRAM` is a
/// Rust simulation model plus a Verilog template, registered with RHDL as a
/// netlist primitive. `DualTableRom` follows that recipe exactly (compare
/// `rhdl_fpga::core::ram::synchronous`), with a second read port and no write
/// port. Nothing about it is opaque: the simulation model below is ten lines,
/// the Verilog is printed in full by [`DualTableRom::hdl`], and the tests in
/// `tests/chain_step.rs` check every table round against the AES reference.
///
/// # Timing
///
/// Addresses presented in cycle `t` give their words in cycle `t + 1`, on
/// both ports, exactly like the two-ROM version it replaces. The output
/// before the first clock edge is unspecified; ownership lives in the
/// enclosing pipeline's valid bits. Reset is ignored: a ROM has no state.
///
/// ```text
///            ┌──────────── one BRAM18 (ram_style = "block") ────────────┐
///  a ──────▶ │ port A: o_a <= T0[a]       256 × 32 T0, preloaded        │ ──▶ o.a
///  b ──────▶ │ port B: o_b <= T0[b]       by the bitstream              │ ──▶ o.b
///            └──────────────────────────────────────────────────────────┘
/// ```
#[derive(Clone, Debug)]
pub struct DualTableRom {
    /// The 256 table words, `contents[x] = T0[x]`.
    contents: [u32; 256],
}

/// The ROM holds [`T0`].
impl Default for DualTableRom {
    fn default() -> Self {
        Self { contents: T0 }
    }
}

/// Ports of [`DualTableRom`]: two addresses in, two table words out.
impl SynchronousIO for DualTableRom {
    /// Two 8-bit addresses per cycle.
    type I = TableRead;
    /// The two 32-bit `T0` words, one cycle later.
    type O = TableWords;
    /// No RHDL kernel: the behaviour is the hand-written `sim` and `hdl`.
    type Kernel = NoSynchronousKernel<ClockReset, TableRead, (), (TableWords, ())>;
}

/// A primitive has no child circuits, so no `D`/`Q` wiring.
impl SynchronousDQ for DualTableRom {
    type D = ();
    type Q = ();
}

/// Native-simulation state of [`DualTableRom`]: the two output registers.
#[derive(PartialEq, Debug, Digital, Clone, Copy)]
#[doc(hidden)]
pub struct DualTableRomState {
    /// The clock level seen on the previous `sim` call, to detect a rising edge.
    clock: Clock,
    /// What both ports show: the words read at the last rising edge.
    current: TableWords,
    /// What both ports will capture at the next rising edge.
    next: TableWords,
}

/// Behaviour (simulation) and structure (Verilog) of the dual-port ROM.
impl Synchronous for DualTableRom {
    /// Two output registers, modelled by [`DualTableRomState`].
    type S = DualTableRomState;

    /// Power-up output is "don't care", like the RAM's output register.
    fn init(&self) -> Self::S {
        Self::S::dont_care()
    }

    /// One half-clock step: look both addresses up while the clock is low,
    /// capture on the rising edge, always show the captured words.
    fn sim(&self, clock_reset: ClockReset, input: Self::I, state: &mut Self::S) -> Self::O {
        // The clock level of this call.
        let clock = clock_reset.clock;
        if !clock.raw() {
            // The words each port will register at the next edge.
            let word = |address: b8| b32(u128::from(self.contents[address.raw() as usize]));
            state.next = TableWords {
                a: word(input.a),
                b: word(input.b),
            };
        }
        // Rising edge (was low, now high): both output registers capture.
        if clock.raw() && !state.clock.raw() {
            state.current = state.next;
        }
        state.clock = clock;
        state.current
    }

    /// Register the Verilog below with RHDL as a primitive, the same way
    /// `SyncBRAM` does.
    fn descriptor(&self, scoped_name: ScopedName) -> Result<Descriptor<SyncKind>, RHDLError> {
        let name = scoped_name.to_string();
        Descriptor::<SyncKind> {
            name: scoped_name,
            input_kind: <<Self as SynchronousIO>::I as Digital>::static_kind(),
            output_kind: <<Self as SynchronousIO>::O as Digital>::static_kind(),
            d_kind: Kind::Empty,
            q_kind: Kind::Empty,
            kernel: None,
            netlist: None,
            hdl: Some(self.hdl(&name)?),
            _phantom: std::marker::PhantomData,
        }
        .with_embedded_netlist_black_box_unchecked()
    }
}

impl DualTableRom {
    /// The Verilog module: one preloaded `256 × 32` array, `ram_style =
    /// "block"`, read on both ports at every rising edge.
    ///
    /// Input `i` packs [`TableRead`] and output `o` packs [`TableWords`]; the
    /// bit ranges of their fields come from RHDL's own layout (`bit_range`),
    /// so the module always matches the Rust types. Two reads of one array in
    /// one clocked block is the form Vivado maps to a single true dual-port
    /// block RAM (UG901, "Dual-Port Block RAM").
    fn hdl(&self, name: &str) -> Result<HDLDescriptor, RHDLError> {
        use quote::{format_ident, quote};
        let module = format_ident!("{}", name);
        let i_kind = TableRead::static_kind();
        let o_kind = TableWords::static_kind();
        let i = TableRead::dont_care();
        let o = TableWords::dont_care();
        let input_bits: vlog::BitRange = (0..TableRead::BITS).into();
        let output_bits: vlog::BitRange = (0..TableWords::BITS).into();
        let a_in: vlog::BitRange = bit_range(i_kind, &path!(i.a))?.0.into();
        let b_in: vlog::BitRange = bit_range(i_kind, &path!(i.b))?.0.into();
        let a_out: vlog::BitRange = bit_range(o_kind, &path!(o.a))?.0.into();
        let b_out: vlog::BitRange = bit_range(o_kind, &path!(o.b))?.0.into();
        let cr: ClockReset = ClockReset::dont_care();
        let clock_index: vlog::BitRange = bit_range(ClockReset::static_kind(), &path!(cr.clock))?.0.into();
        // `mem[x] = T0[x];` for every entry: the bitstream's preload.
        let initial_values = self.contents.iter().enumerate().map(|(x, word)| {
            let value: vlog::LitVerilog = b32(u128::from(*word)).typed_bits().into();
            let x = syn::Index::from(x);
            quote! { mem[#x] = #value; }
        });
        let module: vlog::ModuleDef = syn::parse_quote! {
            module #module(
                input wire [1:0] clock_reset,
                input wire [#input_bits] i,
                output wire [#output_bits] o
            );
                wire [0:0] clock;
                wire [7:0] addr_a;
                wire [7:0] addr_b;
                reg [31:0] word_a;
                reg [31:0] word_b;
                (* ram_style = "block" *) reg [31:0] mem[255:0];
                initial begin
                    #(#initial_values)*
                end
                assign clock = clock_reset[#clock_index];
                assign addr_a = i[#a_in];
                assign addr_b = i[#b_in];
                always @(posedge clock) begin
                    word_a <= mem[addr_a];
                    word_b <= mem[addr_b];
                end
                assign o[#a_out] = word_a;
                assign o[#b_out] = word_b;
            endmodule
        };
        Ok(HDLDescriptor {
            name: name.to_owned(),
            modules: module.into(),
        })
    }
}

/// One write into an UltraRAM S-box: entry `addr` becomes `value`.
///
/// Only UltraRAM boxes ever see this. [`SboxLoader`] produces one per cycle
/// after reset, and every UltraRAM box of the lane receives the same write,
/// so all of them fill in parallel.
#[derive(Clone, Copy, Debug, PartialEq, Digital, Default)]
pub struct RomLoad {
    /// Write this cycle.
    pub enable: bool,
    /// Table entry (8-bit S-box input `0..256`).
    pub addr: b8,
    /// `S(addr)` (8-bit S-box output).
    pub value: b8,
}

/// Which memory an [`SboxPair`] is built from.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum PairStyle {
    /// Preloaded ROM that Vivado builds from LUTs (≈32 LUTs per lookup).
    Lut,
    /// Preloaded ROM in one true dual-port BRAM18: two lookups per tile.
    Block,
    /// UltraRAM, empty at power-up and filled by the lane's [`SboxLoader`]:
    /// two lookups per URAM.
    Ultra,
}

/// The two addresses of an [`SboxPair`], and the loader's write.
#[derive(Clone, Copy, Debug, PartialEq, Digital)]
pub struct PairRead {
    /// Port A address (the state byte of the pair's even position).
    pub a: b8,
    /// Port B address (the state byte of the pair's odd position).
    pub b: b8,
    /// Table load; only an UltraRAM pair uses it.
    pub load: RomLoad,
}

/// `S(a)` and `S(b)` for the addresses presented one cycle earlier.
#[derive(Clone, Copy, Debug, PartialEq, Digital)]
pub struct PairBytes {
    /// Port A output.
    pub a: b8,
    /// Port B output.
    pub b: b8,
}

/// Two registered S-box lookups per cycle from **one** dual-port memory.
///
/// # Why pairs
///
/// A block RAM tile (BRAM18) and an UltraRAM each have two ports, but RHDL's
/// `SyncBRAM` uses only one. Built from `SyncBRAM`s, every S-box placed in
/// RAM cost a whole BRAM18 or URAM, so four slices could not afford to move
/// their leftover S-boxes (rounds 0–2 and 9 of a T-table lane) out of LUTs:
/// build v5 still spent 17,040 LUT-RAM per slice on them. A pair serves two
/// S-boxes of the same round from one memory, halving the RAM per box, the
/// same step [`DualTableRom`] took for T-tables.
///
/// ```text
///            ┌──────── one memory (style: LUT, block or ultra) ────────┐
///  a ──────▶ │ port A: o_a <= S[a]   (UltraRAM: or write the loader's   │ ──▶ o.a
///            │                        entry while loading)               │
///  b ──────▶ │ port B: o_b <= S[b]                                      │ ──▶ o.b
///            └──────────────────────────────────────────────────────────┘
/// ```
///
/// # Timing
///
/// Addresses in cycle `t` give their bytes in cycle `t + 1` on both ports,
/// like a `SyncBRAM` ROM. Reset is ignored: ownership lives in the lane's
/// valid bits, and an UltraRAM pair's lane reports "not ready" until its
/// loader has written all 256 entries. While loading, port A does the write
/// (its read output is meaningless then), so the memory never needs more
/// than the two ports UltraRAM has.
///
/// Like [`DualTableRom`] this is a hand-specified RHDL primitive: a Rust
/// simulation model plus the Verilog printed by [`SboxPair::hdl`], checked
/// against each other in Icarus by the tests below.
#[derive(Clone, Debug)]
pub struct SboxPair {
    /// Which memory the pair is built from.
    style: PairStyle,
}

impl SboxPair {
    /// A pair built from `style` memory.
    #[must_use]
    pub fn new(style: PairStyle) -> Self {
        Self { style }
    }
}

/// Ports of [`SboxPair`].
impl SynchronousIO for SboxPair {
    /// Two addresses and the loader's write.
    type I = PairRead;
    /// The two substituted bytes, one cycle later.
    type O = PairBytes;
    /// No RHDL kernel: the behaviour is the hand-written `sim` and `hdl`.
    type Kernel = NoSynchronousKernel<ClockReset, PairRead, (), (PairBytes, ())>;
}

/// A primitive has no child circuits, so no `D`/`Q` wiring.
impl SynchronousDQ for SboxPair {
    type D = ();
    type Q = ();
}

/// Native-simulation state of [`SboxPair`].
#[derive(PartialEq, Debug, Digital, Clone, Copy)]
#[doc(hidden)]
pub struct SboxPairState {
    /// The clock level seen on the previous `sim` call, to detect a rising edge.
    clock: Clock,
    /// The 256 entries: the S-box for a ROM, written by the loader for UltraRAM.
    mem: [b8; 256],
    /// What both ports show: the bytes read at the last rising edge.
    current: PairBytes,
    /// What both ports will capture at the next rising edge.
    next: PairBytes,
    /// The write an UltraRAM pair applies at the next rising edge.
    write: RomLoad,
}

/// Behaviour (simulation) and structure (Verilog) of the pair.
impl Synchronous for SboxPair {
    /// Memory contents and the two output registers.
    type S = SboxPairState;

    /// A ROM starts holding the S-box; an UltraRAM starts empty (zeros
    /// here, unknown on the chip). Outputs start as "don't care".
    fn init(&self) -> Self::S {
        let mut state = Self::S::dont_care();
        if self.style != PairStyle::Ultra {
            state.mem = core::array::from_fn(|x| b8(u128::from(aes_sbox(x as u8))));
        }
        state
    }

    /// One half-clock step: read both addresses while the clock is low,
    /// capture (and, for UltraRAM, write) on the rising edge.
    fn sim(&self, clock_reset: ClockReset, input: Self::I, state: &mut Self::S) -> Self::O {
        let clock = clock_reset.clock;
        if !clock.raw() {
            // Port A of an UltraRAM pair takes the loader's address while it
            // writes (read-first, like the Verilog).
            let port_a = if self.style == PairStyle::Ultra && input.load.enable {
                input.load.addr
            } else {
                input.a
            };
            state.next = PairBytes {
                a: state.mem[port_a.raw() as usize],
                b: state.mem[input.b.raw() as usize],
            };
            state.write = input.load;
        }
        if clock.raw() && !state.clock.raw() {
            state.current = state.next;
            if self.style == PairStyle::Ultra && state.write.enable {
                state.mem[state.write.addr.raw() as usize] = state.write.value;
            }
        }
        state.clock = clock;
        state.current
    }

    /// Register the Verilog below with RHDL as a primitive.
    fn descriptor(&self, scoped_name: ScopedName) -> Result<Descriptor<SyncKind>, RHDLError> {
        let name = scoped_name.to_string();
        Descriptor::<SyncKind> {
            name: scoped_name,
            input_kind: <<Self as SynchronousIO>::I as Digital>::static_kind(),
            output_kind: <<Self as SynchronousIO>::O as Digital>::static_kind(),
            d_kind: Kind::Empty,
            q_kind: Kind::Empty,
            kernel: None,
            netlist: None,
            hdl: Some(self.hdl(&name)?),
            _phantom: std::marker::PhantomData,
        }
        .with_embedded_netlist_black_box_unchecked()
    }
}

impl SboxPair {
    /// The Verilog module: one `256 × 8` array read on two ports.
    ///
    /// LUT and block pairs are preloaded ROMs: block adds `ram_style =
    /// "block"`, LUT adds `rom_style = "distributed"`. The LUT style must
    /// say so. Without an attribute, Vivado maps a `256 × 8` ROM read on two
    /// registered ports to a BRAM18, as it did in build v8 (2026-10-16):
    /// 142 BRAM18 per slice instead of LUTs, every SLR's block RAM 91 %
    /// full, and slices spread over all three SLRs. A single-port
    /// `SyncBRAM` ROM (before [`SboxPair`]) stayed in LUTs. An UltraRAM pair has `ram_style = "ultra"`, no preload, and a
    /// write on port A: port A's address is the loader's while it writes.
    /// Bit positions come from RHDL's layout (`bit_range`).
    fn hdl(&self, name: &str) -> Result<HDLDescriptor, RHDLError> {
        use quote::{format_ident, quote};
        let module = format_ident!("{}", name);
        let i_kind = PairRead::static_kind();
        let o_kind = PairBytes::static_kind();
        let i = PairRead::dont_care();
        let o = PairBytes::dont_care();
        let input_bits: vlog::BitRange = (0..PairRead::BITS).into();
        let output_bits: vlog::BitRange = (0..PairBytes::BITS).into();
        let a_in: vlog::BitRange = bit_range(i_kind.clone(), &path!(i.a))?.0.into();
        let b_in: vlog::BitRange = bit_range(i_kind.clone(), &path!(i.b))?.0.into();
        let we_in: vlog::BitRange = bit_range(i_kind.clone(), &path!(i.load.enable))?.0.into();
        let wa_in: vlog::BitRange = bit_range(i_kind.clone(), &path!(i.load.addr))?.0.into();
        let wd_in: vlog::BitRange = bit_range(i_kind, &path!(i.load.value))?.0.into();
        let a_out: vlog::BitRange = bit_range(o_kind.clone(), &path!(o.a))?.0.into();
        let b_out: vlog::BitRange = bit_range(o_kind, &path!(o.b))?.0.into();
        let cr: ClockReset = ClockReset::dont_care();
        let clock_index: vlog::BitRange = bit_range(ClockReset::static_kind(), &path!(cr.clock))?.0.into();
        let initial_values = (0..256_u128).map(|x| {
            let value: vlog::LitVerilog = b8(u128::from(aes_sbox(x as u8))).typed_bits().into();
            let x = syn::Index::from(x as usize);
            quote! { mem[#x] = #value; }
        });
        let module: vlog::ModuleDef = match self.style {
            PairStyle::Ultra => syn::parse_quote! {
                module #module(
                    input wire [1:0] clock_reset,
                    input wire [#input_bits] i,
                    output wire [#output_bits] o
                );
                    wire [0:0] clock;
                    wire [0:0] write_enable;
                    wire [7:0] port_a;
                    wire [7:0] addr_b;
                    reg [7:0] byte_a;
                    reg [7:0] byte_b;
                    (* ram_style = "ultra" *) reg [7:0] mem[255:0];
                    assign clock = clock_reset[#clock_index];
                    assign write_enable = i[#we_in];
                    assign port_a = write_enable ? i[#wa_in] : i[#a_in];
                    assign addr_b = i[#b_in];
                    always @(posedge clock) begin
                        if (write_enable) begin
                            mem[port_a] <= i[#wd_in];
                        end
                        byte_a <= mem[port_a];
                    end
                    always @(posedge clock) begin
                        byte_b <= mem[addr_b];
                    end
                    assign o[#a_out] = byte_a;
                    assign o[#b_out] = byte_b;
                endmodule
            },
            style => {
                let memory = if style == PairStyle::Block {
                    quote! { (* ram_style = "block" *) reg [7:0] mem[255:0]; }
                } else {
                    quote! { (* rom_style = "distributed" *) reg [7:0] mem[255:0]; }
                };
                syn::parse_quote! {
                    module #module(
                        input wire [1:0] clock_reset,
                        input wire [#input_bits] i,
                        output wire [#output_bits] o
                    );
                        wire [0:0] clock;
                        wire [7:0] addr_a;
                        wire [7:0] addr_b;
                        reg [7:0] byte_a;
                        reg [7:0] byte_b;
                        #memory
                        initial begin
                            #(#initial_values)*
                        end
                        assign clock = clock_reset[#clock_index];
                        assign addr_a = i[#a_in];
                        assign addr_b = i[#b_in];
                        always @(posedge clock) begin
                            byte_a <= mem[addr_a];
                            byte_b <= mem[addr_b];
                        end
                        assign o[#a_out] = byte_a;
                        assign o[#b_out] = byte_b;
                    endmodule
                }
            }
        };
        Ok(HDLDescriptor {
            name: name.to_owned(),
            modules: module.into(),
        })
    }
}

/// Input of one S-box stage: the 64 state bytes, and the loader's write.
#[derive(Clone, Copy, Debug, PartialEq, Digital)]
pub struct SboxStageIn {
    /// State bytes entering SubBytes: the full 64-byte (512-bit) Haraka
    /// state, i.e. four 16-byte AES lanes side by side.
    pub bytes: [b8; 64],
    /// Table load, used only by UltraRAM S-boxes.
    pub load: RomLoad,
}

/// The hardware of [`SboxStage`]: 64 S-box ROMs (`SyncBRAM<b8, 8>`), one per
/// output byte, addressed through ShiftRows.
///
/// Bit `k` of `ULTRA` makes box `k` an UltraRAM (written by the lane's
/// [`SboxLoader`]); the constructor's `block` mask makes box `k` a block-RAM
/// ROM; every other box is a LUT ROM. See
/// [`sbox_placement`](crate::sbox_placement).
///
/// Output byte `16·lane + 4·column + row` is `S(input[16·lane + 4·((column +
/// row) mod 4) + row])`, exactly
/// [`sub_shift512_kernel`](crate::haraka::sub_shift512_kernel) one cycle
/// later. ROMs whose output the parent never reads are removed by synthesis.
///
/// ```text
///   state byte  ──ShiftRows wiring──▶ ROM k (256 × 8) ──reg──▶ output byte k
///   (64 of them)                      LUT, BRAM18 or URAM       (64 of them)
///                                     chosen by ULTRA / block
/// ```
///
/// Why `ULTRA` is a type parameter but `block` is a constructor argument:
/// UltraRAM boxes need a write port wired in the *kernel*, and RHDL kernels
/// can only branch on compile-time constants. Block RAM is just a synthesis
/// style hint on an otherwise identical ROM, so a run-time value is enough.
#[derive(Clone, Debug, Synchronous, SynchronousDQ)]
pub struct SboxStageRoms<const ULTRA: usize> {
    /// The 32 S-box pairs: pair `p` serves output bytes `2p` (port A) and
    /// `2p + 1` (port B), output byte `k` being `16·lane + 4·column + row`.
    pairs: [SboxPair; 32],
}

impl<const ULTRA: usize> SboxStageRoms<ULTRA> {
    /// A stage whose boxes in `block` are block-RAM ROMs.
    ///
    /// `block` is a 64-bit mask, one bit per output byte, normally a row of
    /// [`BLOCK_MASK`]. Bit `k` set in `ULTRA` wins over `block`.
    #[must_use]
    pub fn new(block: usize) -> Self {
        Self {
            // Pair `p` (bits `2p`, `2p + 1` of the masks): UltraRAM if either
            // box is UltraRAM, else block RAM if either is, else LUTs.
            // `sbox_placement` hands out whole pairs, so both bits agree.
            pairs: core::array::from_fn(|p| {
                let bits = 0b11 << (2 * p);
                if ULTRA & bits != 0 {
                    // UltraRAM: starts empty; the SboxLoader fills it.
                    SboxPair::new(PairStyle::Ultra)
                } else if block & bits != 0 {
                    // Block RAM: preloaded, one BRAM18 for both boxes.
                    SboxPair::new(PairStyle::Block)
                } else {
                    // Preloaded ROM that Vivado builds from LUTs.
                    SboxPair::new(PairStyle::Lut)
                }
            }),
        }
    }
}

/// Ports of [`SboxStageRoms`]: 64 bytes (plus a load) in, 64 bytes out.
impl<const ULTRA: usize> SynchronousIO for SboxStageRoms<ULTRA> {
    /// The 64 state bytes and the loader's write.
    type I = SboxStageIn;
    /// `ShiftRows(SubBytes(state))`, one cycle later.
    type O = [b8; 64];
    /// The wiring kernel below.
    type Kernel = sbox_stage_kernel<ULTRA>;
}

/// ShiftRows decides which input byte each ROM reads; only UltraRAM boxes
/// see the loader's writes.
///
/// Inputs: `input.bytes` (the 64-byte state), `input.load` (the loader's
/// write) and `q` (the 64 ROM outputs). Outputs: the 64 substituted bytes
/// read last cycle, and `d` (each ROM's next address and write). Latency:
/// one clock. Reset is not used.
#[allow(clippy::needless_range_loop)]
#[kernel]
pub fn sbox_stage_kernel<const ULTRA: usize>(
    _clock_reset: ClockReset,
    input: SboxStageIn,
    q: SboxStageRomsQ<ULTRA>,
) -> ([b8; 64], SboxStageRomsD<ULTRA>) {
    // The state byte each output position reads (ShiftRows): output
    // (column, row) of an AES lane reads input column (column + row) mod 4
    // of the same row. `& 3` is `mod 4` without a divider.
    let mut addresses = [b8(0); 64];
    for lane in 0..4 {
        for column in 0..4 {
            for row in 0..4 {
                addresses[16 * lane + 4 * column + row] = input.bytes[16 * lane + 4 * ((column + row) & 3) + row];
            }
        }
    }
    // Pair `p` reads positions `2p` (port A) and `2p + 1` (port B). Every
    // pair sees the loader's write; only UltraRAM pairs have a write port
    // (`SboxPair` ignores it otherwise), so `ULTRA` needs no branch here.
    let mut pairs = [PairRead {
        a: b8(0),
        b: b8(0),
        load: input.load,
    }; 32];
    for p in 0..32 {
        pairs[p] = PairRead {
            a: addresses[2 * p],
            b: addresses[2 * p + 1],
            load: input.load,
        };
    }
    // Output: the bytes the pairs registered on the last edge.
    let mut out = [b8(0); 64];
    for p in 0..32 {
        out[2 * p] = q.pairs[p].a;
        out[2 * p + 1] = q.pairs[p].b;
    }
    (out, SboxStageRomsD::<ULTRA> { pairs })
}

/// Registered `SubBytes ∘ ShiftRows` for all four Haraka AES lanes.
///
/// **Hardware** is exactly [`SboxStageRoms`]: [`Synchronous::descriptor`]
/// returns its descriptor, so the generated Verilog is the 64 RHDL ROMs.
///
/// **Native simulation** uses a faster but cycle-identical model: one register
/// loaded with [`sub_shift512_kernel`](crate::haraka::sub_shift512_kernel).
/// Simulating 64 separate memories per stage made whole-slice tests ~90×
/// slower. The test `fast_model_matches_the_rom_hardware_cycle_for_cycle`
/// pins the two together. (The fast model treats UltraRAM boxes as already
/// loaded; the lane's `ready` output keeps them unused until they are.)
#[derive(Clone, Debug)]
pub struct SboxStage<const ULTRA: usize> {
    /// The real hardware. Used for Verilog generation; native simulation
    /// uses the fast model in [`Synchronous::sim`] instead.
    roms: SboxStageRoms<ULTRA>,
}

impl<const ULTRA: usize> SboxStage<ULTRA> {
    /// A stage whose boxes in `block` are block-RAM ROMs.
    ///
    /// `block` is the round's 64-bit block-RAM mask (a row of [`BLOCK_MASK`]).
    #[must_use]
    pub fn new(block: usize) -> Self {
        Self {
            roms: SboxStageRoms::new(block),
        }
    }
}

/// Same ports as [`SboxStageRoms`].
impl<const ULTRA: usize> SynchronousIO for SboxStage<ULTRA> {
    /// The 64 state bytes and the loader's write.
    type I = SboxStageIn;
    /// `ShiftRows(SubBytes(state))`, one cycle later.
    type O = [b8; 64];
    /// No kernel of its own: [`Synchronous::sim`] and
    /// [`Synchronous::descriptor`] are written by hand below.
    type Kernel = NoSynchronousKernel<ClockReset, SboxStageIn, (), ([b8; 64], ())>;
}

/// No RHDL-visible children of its own; the hardware comes from `roms`.
impl<const ULTRA: usize> SynchronousDQ for SboxStage<ULTRA> {
    /// Nothing to drive.
    type D = ();
    /// Nothing to read.
    type Q = ();
}

/// Native-simulation state of [`SboxStage`] (a single register).
#[derive(PartialEq, Debug, Digital, Clone, Copy)]
#[doc(hidden)]
pub struct SboxStageState {
    /// The clock level seen on the previous `sim` call, to detect a rising edge.
    clock: Clock,
    /// The register's output: the 64 bytes captured at the last rising edge.
    current: [b8; 64],
    /// The value the register will capture at the next rising edge.
    next: [b8; 64],
}

/// Hand-written behaviour: a fast model for simulation, the real ROMs for HDL.
impl<const ULTRA: usize> Synchronous for SboxStage<ULTRA> {
    /// One 64-byte register, modelled by [`SboxStageState`].
    type S = SboxStageState;

    /// Power-up state is "don't care", just like an un-reset register.
    fn init(&self) -> Self::S {
        Self::S::dont_care()
    }

    /// One half-clock step of the fast model: compute while the clock is low,
    /// capture on the rising edge, always show the captured value.
    fn sim(&self, clock_reset: ClockReset, input: Self::I, state: &mut Self::S) -> Self::O {
        // The clock level of this call.
        let clock = clock_reset.clock;
        // Clock low: work out what the ROMs would return for this input.
        if !clock.raw() {
            state.next = crate::haraka::sub_shift512_kernel(input.bytes);
        }
        // Rising edge (was low, now high): the register captures it.
        if clock.raw() && !state.clock.raw() {
            state.current = state.next;
        }
        // Remember this level for edge detection next call.
        state.clock = clock;
        state.current
    }

    /// Hand RHDL the descriptor of the real 64-ROM hardware, so the
    /// generated Verilog is [`SboxStageRoms`], not the fast model.
    fn descriptor(&self, scoped_name: ScopedName) -> Result<Descriptor<SyncKind>, RHDLError> {
        self.roms.descriptor(scoped_name)
    }
}

/// The ten S-box stages of one chain-step lane, each placed by
/// [`sbox_placement`](crate::sbox_placement).
///
/// One struct field per AES round (not an array), because each round's
/// UltraRAM mask is a different compile-time constant and so a different type.
///
/// Ten stages because Haraka-512 has 5 rounds of 2 AES rounds each
/// ([`AES_ROUNDS`]). Each stage serves all four 16-byte AES lanes at once,
/// so the struct holds 10 × 64 = 640 S-box ROMs before folding; synthesis
/// deletes the ones that are constant or unobserved, leaving the 494 live
/// ones counted in [`crate::chain_step`].
#[derive(Clone, Debug, Synchronous, SynchronousDQ)]
pub struct SboxStages {
    /// AES round 0's SubBytes/ShiftRows.
    s0: SboxStage<{ ULTRA_MASK[0] }>,
    /// AES round 1's SubBytes/ShiftRows.
    s1: SboxStage<{ ULTRA_MASK[1] }>,
    /// AES round 2's SubBytes/ShiftRows.
    s2: SboxStage<{ ULTRA_MASK[2] }>,
    /// AES round 3's SubBytes/ShiftRows.
    s3: SboxStage<{ ULTRA_MASK[3] }>,
    /// AES round 4's SubBytes/ShiftRows.
    s4: SboxStage<{ ULTRA_MASK[4] }>,
    /// AES round 5's SubBytes/ShiftRows.
    s5: SboxStage<{ ULTRA_MASK[5] }>,
    /// AES round 6's SubBytes/ShiftRows.
    s6: SboxStage<{ ULTRA_MASK[6] }>,
    /// AES round 7's SubBytes/ShiftRows.
    s7: SboxStage<{ ULTRA_MASK[7] }>,
    /// AES round 8's SubBytes/ShiftRows.
    s8: SboxStage<{ ULTRA_MASK[8] }>,
    /// AES round 9's SubBytes/ShiftRows.
    s9: SboxStage<{ ULTRA_MASK[9] }>,
}

/// Each round gets its own block-RAM mask from [`BLOCK_MASK`].
impl Default for SboxStages {
    /// Ten stages, placed as [`crate::sbox_placement`] decided.
    fn default() -> Self {
        Self {
            s0: SboxStage::new(BLOCK_MASK[0]),
            s1: SboxStage::new(BLOCK_MASK[1]),
            s2: SboxStage::new(BLOCK_MASK[2]),
            s3: SboxStage::new(BLOCK_MASK[3]),
            s4: SboxStage::new(BLOCK_MASK[4]),
            s5: SboxStage::new(BLOCK_MASK[5]),
            s6: SboxStage::new(BLOCK_MASK[6]),
            s7: SboxStage::new(BLOCK_MASK[7]),
            s8: SboxStage::new(BLOCK_MASK[8]),
            s9: SboxStage::new(BLOCK_MASK[9]),
        }
    }
}

/// Input of [`SboxStages`]: every round's bytes plus the shared table load.
#[derive(Clone, Copy, Debug, PartialEq, Digital)]
pub struct SboxStagesIn {
    /// State bytes entering each round's SubBytes: ten 64-byte states, one
    /// per AES round, all in flight at once for ten different tokens.
    pub bytes: [[b8; 64]; AES_ROUNDS],
    /// Table load from the lane's [`SboxLoader`].
    pub load: RomLoad,
}

/// Ports of [`SboxStages`]: ten states in, ten substituted states out.
impl SynchronousIO for SboxStages {
    /// Every round's input bytes plus the shared load.
    type I = SboxStagesIn;
    /// Every round's `ShiftRows(SubBytes(·))`, one cycle later.
    type O = [[b8; 64]; AES_ROUNDS];
    /// The fan-out kernel below.
    type Kernel = sbox_stages_kernel;
}

/// Hand each round its bytes; broadcast the table load.
///
/// Pure wiring, no logic. Output `[r]` is round `r`'s stage output (one
/// clock after its input). Every stage receives the same `load`, because the
/// loader writes the same entry into every UltraRAM box of the lane.
#[kernel]
pub fn sbox_stages_kernel(
    _clock_reset: ClockReset,
    input: SboxStagesIn,
    q: SboxStagesQ,
) -> ([[b8; 64]; AES_ROUNDS], SboxStagesD) {
    // Round r's bytes go to stage s<r>; the load goes everywhere.
    let d = SboxStagesD {
        s0: SboxStageIn { bytes: input.bytes[0], load: input.load },
        s1: SboxStageIn { bytes: input.bytes[1], load: input.load },
        s2: SboxStageIn { bytes: input.bytes[2], load: input.load },
        s3: SboxStageIn { bytes: input.bytes[3], load: input.load },
        s4: SboxStageIn { bytes: input.bytes[4], load: input.load },
        s5: SboxStageIn { bytes: input.bytes[5], load: input.load },
        s6: SboxStageIn { bytes: input.bytes[6], load: input.load },
        s7: SboxStageIn { bytes: input.bytes[7], load: input.load },
        s8: SboxStageIn { bytes: input.bytes[8], load: input.load },
        s9: SboxStageIn { bytes: input.bytes[9], load: input.load },
    };
    // Collect the ten stage outputs back into one array, in round order.
    (
        [q.s0, q.s1, q.s2, q.s3, q.s4, q.s5, q.s6, q.s7, q.s8, q.s9],
        d,
    )
}

/// What the loader presents each cycle.
///
/// `load` goes to every UltraRAM S-box of the lane; `done` becomes the lane's
/// [`ready`](crate::chain_step::ChainStepOutput::ready) bit.
#[derive(Clone, Copy, Debug, PartialEq, Digital, Default)]
pub struct LoaderOut {
    /// The write for every UltraRAM S-box of the lane.
    pub load: RomLoad,
    /// All 256 entries are written (or the lane has no UltraRAM S-boxes).
    pub done: bool,
}

/// Fills a lane's UltraRAM S-boxes after reset: entry `i` gets `S(i)` for
/// `i = 0..256`, all boxes in parallel, from one LUT S-box ROM.
///
/// UltraRAM cannot be preloaded by the bitstream, so this runs on every
/// reset (about 258 cycles). Until `done`, the lane reports not ready and the
/// factory starts no keys. A lane with no UltraRAM S-boxes is ready at once.
///
/// Timeline after reset (one row per clock):
///
/// ```text
///   cycle   next   ROM read   pending (addr)   write presented   done
///     0       0     S(0)       -                 -                 no
///     1       1     S(1)       0                 S(0) → entry 0    no
///     2       2     S(2)       1                 S(1) → entry 1    no
///    ...
///   256     256     -          255               S(255) → 255      no
///   257     256     -          -                 -                 yes
/// ```
///
/// Why it is safe to use the LUT ROM as the source: it is preloaded by the
/// bitstream, so it is correct from the first cycle.
#[derive(Clone, Debug, Synchronous, SynchronousDQ)]
pub struct SboxLoader {
    /// A preloaded LUT S-box ROM: the source of the values to copy.
    rom: SyncBRAM<b8, 8>,
    /// The next entry to read, `0..=256` (9 bits so that 256 means "finished").
    /// Resets to 0, so the load restarts after every reset.
    next: DFF<b9>,
    /// The write waiting for the ROM's registered read: `enable` and `addr`
    /// are known one cycle before `value` arrives from `rom`.
    pending: DFF<RomLoad>,
}

/// Start at entry 0 with nothing pending.
impl Default for SboxLoader {
    /// A loader that will fill all 256 entries after reset.
    fn default() -> Self {
        Self {
            rom: sbox_rom(),
            next: DFF::new(b9(0)),
            pending: DFF::new(RomLoad::default()),
        }
    }
}

/// Ports of [`SboxLoader`]: no input, one write plus `done` out.
impl SynchronousIO for SboxLoader {
    /// No input: the loader runs by itself after reset.
    type I = ();
    /// This cycle's write and the `done` flag.
    type O = LoaderOut;
    /// The counter kernel below.
    type Kernel = sbox_loader_kernel;
}

/// Read `S(i)` one cycle, write it the next.
///
/// Inputs: only `q` (the counter, the pending write and the ROM output).
/// Outputs: the [`LoaderOut`] for this cycle and `d` (the next counter,
/// the next pending write and the next ROM address). A two-stage pipeline:
/// stage 1 reads the ROM, stage 2 presents the write.
#[kernel]
pub fn sbox_loader_kernel(_clock_reset: ClockReset, _input: (), q: SboxLoaderQ) -> (LoaderOut, SboxLoaderD) {
    // Still entries left to read?
    let issuing = q.next < b9(256);
    // The low 8 bits of the counter are the table entry to read now.
    let address: b8 = q.next.resize();
    let d = SboxLoaderD {
        // Stage 1: read S(address); the value appears on `q.rom` next cycle.
        rom: sbox_read_kernel(address),
        // Count up to 256, then hold there.
        next: if issuing { q.next + b9(1) } else { q.next },
        // Remember which entry that read was for. `value` is a placeholder;
        // the real value comes from the ROM output next cycle.
        pending: RomLoad {
            enable: issuing,
            addr: address,
            value: b8(0),
        },
    };
    // Stage 2: the write for last cycle's read, with the ROM's value.
    let load = RomLoad {
        enable: q.pending.enable,
        addr: q.pending.addr,
        value: q.rom,
    };
    // Done when no read is left and the last write has been presented, or
    // at once when this build has no UltraRAM S-boxes (`LANE_NEEDS_LOAD`
    // is a compile-time constant from crate::sbox_placement).
    let done = !LANE_NEEDS_LOAD || (!issuing && !q.pending.enable);
    (LoaderOut { load, done }, d)
}

/// Unit tests: each ROM agrees with the reference AES in [`crate::haraka`].
#[cfg(test)]
mod tests {
    use super::*;
    use crate::haraka::{aes_round, sub_shift512_kernel};

    /// The T-table formula `T0[s0] ⊕ rotl8(T0[s1]) ⊕ …` gives exactly the
    /// SubBytes+ShiftRows+MixColumns columns of the reference `aes_round`
    /// (with a zero key, so AddRoundKey drops out).
    #[test]
    fn t_table_columns_equal_the_reference_aes_round() {
        // Build one column's output purely from T0 and compare with aes_round.
        for seed in 0..64_u32 {
            // A deterministic pseudo-random 16-byte AES state.
            let state: [u8; 16] = core::array::from_fn(|i| (seed * 37 + i as u32 * 91) as u8);
            // Zero key, so the reference output is pure Sub+Shift+Mix.
            let key = [0_u8; 16];
            let reference = aes_round(&state, &key);
            for column in 0..4 {
                // The column's four output bytes, accumulated as one word.
                let mut word = 0_u32;
                for row in 0..4 {
                    // ShiftRows source byte for this (column, row).
                    let source = state[4 * ((column + row) % 4) + row];
                    word ^= T0[usize::from(source)].rotate_left(8 * row as u32);
                }
                assert_eq!(word.to_le_bytes(), reference[4 * column..4 * column + 4]);
            }
        }
    }

    /// Run one full clock cycle (low half, then rising edge) with reset off,
    /// returning the output seen just after the edge.
    fn clocked<C: Synchronous>(circuit: &C, input: C::I, state: &mut C::S) -> C::O {
        circuit.sim(clock_reset(clock(false), reset(false)), input, state);
        circuit.sim(clock_reset(clock(true), reset(false)), input, state)
    }

    /// A stage input with the given bytes and no loader write.
    fn bytes_in(bytes: [b8; 64]) -> SboxStageIn {
        SboxStageIn {
            bytes,
            load: RomLoad::default(),
        }
    }

    /// The fast model returns `sub_shift512_kernel(input)` after one clock.
    #[test]
    fn sbox_stage_is_sub_shift_one_cycle_later() {
        // All-LUT stage: no UltraRAM, no block RAM.
        let stage = SboxStage::<0>::new(0);
        let mut state = stage.init();
        for seed in 0..8_u128 {
            // A deterministic pseudo-random 64-byte state.
            let input: [b8; 64] = core::array::from_fn(|i| b8((seed * 53 + i as u128 * 29) & 0xff));
            assert_eq!(
                clocked(&stage, bytes_in(input), &mut state),
                sub_shift512_kernel(input)
            );
        }
    }

    /// Half the boxes in UltraRAM, a quarter in block RAM, the rest in LUTs,
    /// in whole pairs (bits `2p`, `2p + 1` agree, as `sbox_placement` makes them).
    const MIXED_ULTRA: usize = 0x3333_3333_3333_3333;
    /// The block-RAM quarter of that mix (bits disjoint from [`MIXED_ULTRA`]).
    const MIXED_BLOCK: usize = 0x0C0C_0C0C_0C0C_0C0C;

    /// Empty UltraRAM boxes become correct S-boxes once the loader has run,
    /// and the loader writes each of the 256 entries exactly once.
    #[test]
    fn ultraram_boxes_work_once_the_loader_has_filled_them() {
        let loader = SboxLoader::default();
        // Real ROM hardware (not the fast model), with all three placements.
        let roms = SboxStageRoms::<MIXED_ULTRA>::new(MIXED_BLOCK);
        // Simulation state of the loader (`ls`) and of the ROMs (`rs`).
        let (mut ls, mut rs) = (loader.init(), roms.init());
        // Placeholder state bytes while loading; their outputs are ignored.
        let idle = [b8(0); 64];
        // Reset, then let the loader run to completion, feeding the stage.
        let cr = clock_reset(clock(false), reset(true));
        loader.sim(cr, (), &mut ls);
        loader.sim(clock_reset(clock(true), reset(true)), (), &mut ls);
        // Number of enabled writes seen; must end at exactly 256.
        let mut writes = 0;
        // 300 cycles is comfortably more than the ~258 the load takes.
        for _ in 0..300 {
            let out = clocked(&loader, (), &mut ls);
            if out.load.enable {
                assert_eq!(out.load.value, b8(u128::from(aes_sbox(out.load.addr.raw() as u8))));
                writes += 1;
            }
            clocked(&roms, SboxStageIn { bytes: idle, load: out.load }, &mut rs);
        }
        assert_eq!(writes, 256, "every table entry written exactly once");
        // Now every box, whatever its placement, must behave as an S-box.
        for seed in 0..8_u128 {
            let input: [b8; 64] = core::array::from_fn(|i| b8((seed * 71 + i as u128 * 13) & 0xff));
            assert_eq!(clocked(&roms, bytes_in(input), &mut rs), sub_shift512_kernel(input));
        }
    }

    /// Both ports of [`DualTableRom`] return `T0` one cycle after the address,
    /// for every address, with the two ports reading different entries.
    #[test]
    fn dual_table_rom_reads_t0_on_both_ports() {
        let rom = DualTableRom::default();
        let mut state = rom.init();
        // Port A walks up while port B walks down.
        for x in 0..256_u128 {
            let words = clocked(
                &rom,
                TableRead {
                    a: b8(x),
                    b: b8(255 - x),
                },
                &mut state,
            );
            assert_eq!(words.a, b32(u128::from(T0[x as usize])));
            assert_eq!(words.b, b32(u128::from(T0[255 - x as usize])));
        }
    }

    /// The Verilog of a [`DualTableRom`] is one block-RAM array read on two
    /// ports in one clocked block: the shape Vivado maps to a single true
    /// dual-port BRAM18 (build v4 used two single-port ROMs instead).
    #[test]
    fn dual_table_rom_is_one_array_with_two_registered_reads() -> Result<(), RHDLError> {
        let hdl = DualTableRom::default().descriptor("t_rom".into())?.hdl()?.modules.pretty();
        assert_eq!(hdl.matches("reg [31:0] mem[255:0]").count(), 1, "{hdl}");
        assert_eq!(hdl.matches("ram_style = \"block\"").count(), 1);
        assert_eq!(hdl.matches("<= mem[").count(), 2, "two read ports");
        assert!(!hdl.contains("mem[addr] <="), "no write port");
        Ok(())
    }

    /// The generated Verilog of [`DualTableRom`], run in Icarus, matches its
    /// Rust model on every cycle of a stream of address pairs (both ports,
    /// equal and different addresses, reset in the middle).
    #[test]
    fn dual_table_rom_rtl_matches_the_model() -> Result<(), RHDLError> {
        let rom = DualTableRom::default();
        // 64 address pairs covering low/high entries and a = b.
        let reads = (0..64_u128).map(|n| {
            rhdl::core::sim::ResetOrData::Data(TableRead {
                a: b8((n * 37 + 5) & 0xff),
                b: b8(if n % 8 == 0 { (n * 37 + 5) & 0xff } else { (255 + 1024 - n * 11) & 0xff }),
            })
        });
        let stream = std::iter::once(rhdl::core::sim::ResetOrData::Reset)
            .chain(reads)
            .chain(std::iter::once(rhdl::core::sim::ResetOrData::Reset))
            .chain((0..8_u128).map(|n| rhdl::core::sim::ResetOrData::Data(TableRead { a: b8(n), b: b8(255 - n) })))
            .clock_pos_edge(10);
        let bench = rom.run(stream).collect::<SynchronousTestBench<_, _>>();
        // Skip the first samples: the output register starts as "don't care".
        bench.rtl(&rom, &TestBenchOptions::default().skip(2))?.run_iverilog()?;
        Ok(())
    }

    /// The generated Verilog of a block-RAM and an UltraRAM [`SboxPair`], run
    /// in Icarus, matches the Rust model on every cycle: the UltraRAM pair
    /// is loaded first through port A, then both pairs read on both ports.
    #[test]
    fn sbox_pair_rtl_matches_the_model() -> Result<(), RHDLError> {
        let idle = RomLoad::default();
        // 256 load cycles (port B reads meanwhile), then 64 read pairs.
        let load = (0..256_u128).map(|x| PairRead {
            a: b8(0),
            b: b8(x),
            load: RomLoad { enable: true, addr: b8(x), value: b8(u128::from(aes_sbox(x as u8))) },
        });
        let reads = (0..64_u128).map(|n| PairRead {
            a: b8((n * 37 + 5) & 0xff),
            b: b8((255 + 1024 - n * 11) & 0xff),
            load: idle,
        });
        let inputs: Vec<PairRead> = load.chain(reads).collect();
        for style in [PairStyle::Block, PairStyle::Ultra, PairStyle::Lut] {
            let pair = SboxPair::new(style);
            let stream = std::iter::once(rhdl::core::sim::ResetOrData::Reset)
                .chain(inputs.iter().copied().map(rhdl::core::sim::ResetOrData::Data))
                .clock_pos_edge(10);
            let bench = pair.run(stream).collect::<SynchronousTestBench<_, _>>();
            // An UltraRAM reads unknown values until written, so for it the
            // comparison starts after the load (the bench has four samples
            // per clock cycle; 262 cycles covers reset and 256 writes).
            let skip = if style == PairStyle::Ultra { 4 * 262 } else { 2 };
            bench.rtl(&pair, &TestBenchOptions::default().skip(skip))?.run_iverilog()?;
        }
        Ok(())
    }

    /// An UltraRAM pair holds the S-box once the loader has written it.
    #[test]
    fn ultra_pair_reads_what_the_loader_wrote() {
        let pair = SboxPair::new(PairStyle::Ultra);
        let mut state = pair.init();
        for x in 0..256_u128 {
            let load = RomLoad { enable: true, addr: b8(x), value: b8(u128::from(aes_sbox(x as u8))) };
            clocked(&pair, PairRead { a: b8(0), b: b8(0), load }, &mut state);
        }
        // The output after a clock edge is what that edge registered.
        let out = clocked(&pair, PairRead { a: b8(5), b: b8(255), load: RomLoad::default() }, &mut state);
        assert_eq!((out.a, out.b), (b8(u128::from(aes_sbox(5))), b8(u128::from(aes_sbox(255)))));
    }

    /// Both ports of a ROM pair return `S(x)` one cycle after the address.
    #[test]
    fn sbox_pair_reads_the_sbox_on_both_ports() {
        let pair = SboxPair::new(PairStyle::Block);
        let mut state = pair.init();
        for x in 0..256_u128 {
            let out = clocked(&pair, PairRead { a: b8(x), b: b8(255 - x), load: RomLoad::default() }, &mut state);
            assert_eq!(out.a, b8(u128::from(aes_sbox(x as u8))));
            assert_eq!(out.b, b8(u128::from(aes_sbox(255 - x as u8))));
        }
    }

    /// The fast simulation model of [`SboxStage`] and the real
    /// [`SboxStageRoms`] give the same output on every half-cycle. This is
    /// what makes it safe to simulate with the fast model.
    #[test]
    fn fast_model_matches_the_rom_hardware_cycle_for_cycle() {
        // No UltraRAM here: the fast model assumes UltraRAM is already loaded.
        let fast = SboxStage::<0>::new(MIXED_BLOCK);
        let roms = SboxStageRoms::<0>::new(MIXED_BLOCK);
        // Simulation states: `fs` for the fast model, `rs` for the ROMs.
        let (mut fs, mut rs) = (fast.init(), roms.init());
        for seed in 0..32_u128 {
            let input: [b8; 64] = core::array::from_fn(|i| b8((seed * 97 + i as u128 * 41) & 0xff));
            // Compare at both clock levels, not just after the edge.
            for phase in [false, true] {
                let cr = clock_reset(clock(phase), reset(false));
                let (a, b) = (
                    fast.sim(cr, bytes_in(input), &mut fs),
                    roms.sim(cr, bytes_in(input), &mut rs),
                );
                // Skip the first cycle: both start as "don't care".
                if seed > 0 {
                    assert_eq!(a, b, "seed {seed} phase {phase}");
                }
            }
        }
    }

    /// The generated Verilog of an [`SboxStage`] is 32 pair memories, with the
    /// requested UltraRAM (16 pairs = half) and block-RAM (8 pairs = a
    /// quarter) styles; the other 8 are marked for LUTs.
    #[test]
    fn stage_lowers_to_rhdl_roms_with_the_placed_styles() -> Result<(), RHDLError> {
        // Lower the stage to Verilog text.
        let hdl = SboxStage::<MIXED_ULTRA>::new(MIXED_BLOCK)
            .descriptor("s_stage".into())?
            .hdl()?
            .modules
            .pretty();
        assert_eq!(hdl.matches("reg [7:0] mem[255:0]").count(), 32, "one memory per pair of bytes");
        assert_eq!(hdl.matches("ram_style = \"ultra\"").count(), 16);
        assert_eq!(hdl.matches("ram_style = \"block\"").count(), 8);
        assert_eq!(hdl.matches("rom_style = \"distributed\"").count(), 8);
        Ok(())
    }
}