rhdl_ubft/

lib.rs

//! # rhdl_ubft: ubft_new's consensus kernel, rewritten in RHDL
//!
//! # Building for Wolverine's U280
//!
//! The full consensus implementation does not yet have an established,
//! accepted end-to-end U280 image flow. A successful Cargo build or RTL export
//! is only a prerequisite; it does not establish routed timing or card behavior.
//! Do not substitute `crates/dsig/tools/wolverine-ubft.sh`: that script builds
//! the isolated DSig signature-service probe, not this consensus kernel.
//!
//! As recorded in `PROJECT_COORDINATION.md` on October 24, 2026, production
//! export for source `d806f5bd` passed, but the subsequent child synthesis timed
//! out without an accepted checkpoint. A later receiver fixture hit an Icarus
//! parser input-buffer overflow. The ledger also recorded less than the 30 GiB
//! free disk required for the planned full link. These are dated observations,
//! not a live Wolverine inventory or a claim that the current source was built.
//!
//! Continue from the pinned source, configuration and evidence under
//! `evidence/rhdl-ubft/card-bringup-20261024/`: resolve the lowering/parser and
//! child-synthesis gates, confirm disk capacity, then package/link with
//! Vivado/Vitis 2023.2 and the installed U280 platform. Require routed acceptance
//! and host verification before programming or claiming performance. Preserve
//! the existing evidence and match the host configuration to the exported kernel.
//!
//! Run commands from the workspace root. Before compilation or RTL generation,
//! record ownership in `PROJECT_COORDINATION.md` and
//! `/home/sahasta/Development/FPGA_COORDINATION.md`, and use the applicable
//! nonblocking `local-heavy.lock`; `crates/ed25519/tools/bounded.sh` wraps local
//! Cargo commands with that lock and resource limits. Reserve Wolverine builds
//! in both ledgers too, then use `crates/dsig/tools/with-u280-lock.sh` to hold
//! `u280.lock`. Do not bypass a busy lock.
//!
//! [ubft_new](https://github.com/YiJiang0908/ubft_new) (commit `a7537bd`)
//! runs Byzantine fault-tolerant consensus on an Alveo U280. Each card is
//! one replica. Replicas talk by RDMA-writing into each other's HBM over a
//! RoCE stack. A key-value store sends every PUT through consensus before
//! it touches its hash table. ubft_new's kernels are Vitis HLS C++.
//! rhdl_ubft is its consensus kernel (`ubft`) in RHDL: the same ports, the
//! related HBM layout and protocol, with targeted repairs and mechanics rebuilt
//! for hardware. Certified Byzantine recovery and full integration remain open.
//!
//! **In numbers.** Not measured yet. The comparison against ubft_new's own
//! RTL (HLS cosimulation, Vivado route and the card) is the next chapter of
//! this page to be written. Until then this page states no speedup.
//!
//! # Contents
//!
//! 1. [What ubft_new does](#chapter-1-what-ubft_new-does)
//! 2. [Why it is slow, and where it is wrong](#chapter-2-why-it-is-slow-and-where-it-is-wrong)
//! 3. [The rewrite](#chapter-3-the-rewrite)
//! 4. [Choosing a build](#chapter-4-choosing-a-build)
//! 5. [What is proven](#chapter-5-what-is-proven)
//! 6. [Where to go next](#where-to-go-next)
//!
//! # Chapter 1: What ubft_new does
//!
//! A slot is one consensus decision: sequence number `k`, value `m`. The
//! leader proposes, and every replica votes by writing a word into the
//! same mailbox address of every peer's HBM ([`layout`]). Replicas poll
//! their own HBM to count votes.
//!
//! - **Fast path:** prepare, then two rounds of votes (`will_cert`,
//!   `will_commit`). Each round needs a quorum of `2f + 1`, where uBFT
//!   tolerates `f = (n − 1) / 2` faults among `n` replicas.
//! - **Slow path**, after a timeout: every replica signs the slot's summary
//!   with Ed25519, sends its 64-byte share to all, and verifies shares
//!   until it holds `2f + 1`. Then one round of commit votes.
//! - **View change**, after another timeout: the leader rotates, and the
//!   new leader re-proposes the highest-numbered prepare anyone saw.
//!
//! # Chapter 2: Why it is slow, and where it is wrong
//!
//! **Slow.** The HLS kernel is one loop that does one state's work per
//! iteration, and every mailbox word it reads is its own HBM transaction.
//! Its Ed25519 calls are inline: a signature takes 3.07M cycles and a
//! verification 10.4M (C/RTL cosimulation, measured by DSig), and the loop
//! stops for each. That signer is also the routed design's worst path. The
//! kernel clock misses 300 MHz by 2.866 ns, and the linker selects about
//! 161 MHz. That is a timing-derived setting, not measured runtime frequency.
//!
//! **Wrong.** Reading the code, and running it as a C-simulated cluster,
//! shows a protocol that cannot reach a decision as written:
//!
//! - a fast-path vote is sent to one queue pair, then the state moves on;
//! - the leader never votes, so `2f + 1` out of `2f + 1` is unreachable;
//! - the slow path never broadcasts the flag that tells peers its share is
//!   there;
//! - sends go to queue pairs `id·N + j`, but the RoCE stack programs `N −
//!   1` connections at `(N − 1)·id + rank`;
//! - `k = slot + 1` repeats, and nothing clears old votes;
//! - shares go out as eight one-word packets;
//! - the signed summary XORs its fields, and the library hashes only one
//!   byte of it.
//!
//! [`protocol`] lists each fix next to the line it changes.
//!
//! # Chapter 3: The rewrite
//!
//! The kernel is small engines around one controller ([`replica`]):
//!
//! | Engine | Job | Replaces |
//! |---|---|---|
//! | [`control`] | ubft_new's register map, plus counters the host reads after a run | HLS `s_axilite` |
//! | [`fetch`] | one burst per poll, any alignment | one HBM read per word |
//! | [`broadcast`] | a write to every peer, one packet each, nothing dropped | `rdma_write*` loops |
//! | [`writer`] | the replica's own copy of what it sent | (missing) |
//! | [`share`] | a peer's share from HBM, realigned, into the verifier | 16 word reads |
//! | [`setup`] + [`crypto`] | keys into a signature service on streams | inline xf Ed25519 |
//! | [`protocol`] | the four machines | `ubft_machines.cpp` |
//!
//! # Chapter 4: Choosing a build
//!
//! | Choice | How | Values |
//! |---|---|---|
//! | Signature scheme | Cargo feature, then `RHDL_UBFT_SCHEME` if both are on | `ed25519` (default), `dsig` |
//! | Most replicas | `RHDL_UBFT_NODES` | 3..=16 (ubft_new: 12) |
//! | Window | `RHDL_UBFT_WINDOW` | 1..=8 (ubft_new: 3) |
//!
//! The run's cluster size is `node_num`, at most the build's. The Tuner writes supported values to `build-config.toml`; apply them with
//! `apply_ubft_plan`. Saving TOML alone does not change a build.
//!
//! # Chapter 5: What is proven
//!
//! Nothing on hardware yet. In cycle simulation (feature `sim`: three real
//! RHDL replicas, models of HBM, RoCE and the host, and a stand-in
//! signature scheme that is **not secure**):
//!
//! | Run | Result |
//! |---|---|
//! | healthy cluster, 5 proposals | all decide on the fast path, in order, and agree; every write one framed packet to a programmed queue pair |
//! | one replica's fast votes dropped | all decide on the slow path (2 shares verified each) |
//! | equivocating leader | no conflicting decisions, but **nothing decided**: views drift apart and an all-`n` view change never completes (a known failure, an ignored test) |
//!
//! The replica lowers to Verilog (`export_rhdl_ubft`). Its view change is a
//! value-binding prototype, not uBFT's certified recovery; the plan to bring
//! that back is docs/rhdl_ubft-recovery-design.md.
//!
//! # Where to go next
//!
//! | You want to... | Read |
//! |---|---|
//! | see the protocol and its fixes | [`protocol`] |
//! | see what crosses the wire | [`layout`], [`wire`] |
//! | add a signature scheme | [`ports::CryptoIn`], [`ports::CryptoScheme`], [`crypto`] |
//! | plug into the Vitis link | [`ports`] |

#![warn(missing_docs)]

pub mod broadcast;
pub mod consensus;
pub mod control;
/// The one-card cluster: replicas, the on-chip fabric, the service (DSig).
pub mod card;
pub mod cert;
pub mod checkpoint;
pub mod crypto;
pub mod ctb;
pub mod fetch;
// Atlas's key-value kernels (docs pending: docs/rhdl_ubft-kv-journey.md).
#[allow(missing_docs)]
pub mod kv;
pub mod layout;
pub mod ports;
pub mod protocol;
pub mod register;
pub mod register_reader;
#[cfg(feature = "replicated-register")]
pub mod register_writer;
#[cfg(feature = "replicated-register")]
pub mod replicated_register;
pub mod replica;
pub mod rpc;
pub mod setup;
pub mod share;
pub mod view_change;
#[cfg(feature = "sim")]
pub mod sim;
pub mod wire;
pub mod writer;

/// Build variables (`build.rs`).
pub mod build_config {
    include!(concat!(env!("OUT_DIR"), "/build_config.rs"));
}

#[cfg(feature = "ed25519")]
pub use replica::RhdlUbftEd25519;

/// The replica selected at build time, including its actual service and shape.
#[cfg(ubft_scheme = "ed25519")]
pub type SelectedReplica = RhdlUbftEd25519<{ build_config::BUILD_NODES }, { build_config::BUILD_WINDOW }>;
/// DSig selection preserves its named W4/Haraka/Blake3 profile.
#[cfg(ubft_scheme = "dsig")]
pub type SelectedReplica = crypto::dsig::RhdlUbftDsig<{ build_config::BUILD_NODES }, { build_config::BUILD_WINDOW }>;