Module relay 

Relay stations and delay lines: registers that let slice traffic cross dies.

The problem: handshakes that span three SLRs

U280 is three dies (SLRs) stacked on an interposer. The HBM ports, the AXI-Lite control port and therefore the engine’s shared blocks (control registers, message reader and HBM writers) belong at the bottom of SLR0, the certificate unit sits in SLR1, and build v9 (2026-10-17) pinned each slice wholly into one SLR (fpga/u280/engine/floorplan.tcl) so that no slice straddles a die.

That worked: every slice landed inside its SLR. But every signal between a slice and a shared block was built for neighbours, with logic on both ends of the wire. The slices in SLR2 were two die crossings away, and v9 routed at WNS -4.715 ns (TNS -81,716). Of its 5,000 worst paths:

paths	link
2,809	writer → HBM: the writer’s wvalid is combinational from the slice’s beat, and beat_ready from HBM’s wready goes straight back into the signer
875	control registers → the slices’ BLAKE3 lanes (run configuration)
~180	certificate unit ↔ slices (certificates in, roots out)
~30	slice status counters → the register-read mux

The worst one ran from a writer counter in SLR1 through the writer’s logic, into slice 3’s signer in SLR2, and back down to the HBM interconnect in SLR0: 9.6 ns for a 4.878 ns clock, 93 % of it wire.

The decision: registers on both ends of every long wire

A long wire is cheap if a flip-flop drives it and at most one LUT sits in front of the flip-flop it reaches. So every slice-boundary signal goes through registers in the engine (crate::engine), by kind:

• Full-rate handshakes (frame beats out) go through a RelayPipe: PIPE_STAGES Carloni relay stations (Relay). A handshake cannot simply be delayed, because the receiver’s “taken” must reach the sender in the same clock.
• Slower handshakes (messages and certificates in) go through a ToggleLink: plain registers plus a one-bit toggle protocol. It moves one item per 6 clocks but costs almost no LUTs (relays cost 21,031 LUTs on these two paths in build v12). A signer takes at most one message per message-hash round trip and one certificate per 128 signatures.
• Everything else (run configuration and run, subtree-root pulses, the roots_popped credit, status counters and the fault bit) goes through a Delay: DELAY_STAGES plain registers. These signals tolerate latency: configuration is static during a run, a root is a one-clock pulse that arrives two clocks later, a late credit only makes the slice more cautious, and the host reads status long after it changes.

Relay stations

A relay station (L. P. Carloni, “From Latency-Insensitive Design to Communication-Based System-Level Design”, Proc. IEEE 2015, figure 4) is a skid buffer: two data registers and a two-state controller inserted into a valid/ready link. Its outputs in both directions come from flip-flops:

• void_out and the data come from the main register;
• stop_out (“not ready”) is just “the controller is in Stall”.

It still moves one item per clock: when the far side stops, the item already in flight is caught in aux instead of being lost. Each station adds one clock of latency.

  SLR2: signer ─▶ [relay 0] ─┐
  SLR1:                      └▶ [relay 1] ─┐
  SLR0:                                    └▶ [relay 2] ─▶ writer ─▶ HBM
           ◀── stop (registered) ── each station ── stop ◀── !beat_ready

RHDL ships the same station as rhdl_fpga::lid::carloni::Carloni. This one differs in one way: its two data registers are resetless ([DFF::resetless]), so the wide payloads (a certificate is 1,345 bits) add no loads to the engine’s reset tree (build v8’s worst paths started at the reset). Only void and the state reset.

Protocol mapping

handshake	relay
valid	!void
the receiver took the item this clock (beat_ready, message_taken, certificate_taken)	!stop

The receivers’ answers are all “taken this clock”, so they are low when nothing is offered. A station with an empty output ignores stop, so that is harmless. The sender sees “taken” as valid && !stop_out of the first station.

Structs

Delay

DELAY_STAGES plain registers: the output is the input from DELAY_STAGES clocks ago. Resets to T::dont_care() (all zeros), so a delayed valid, run or fault bit is low after reset.

PipeIn

Inputs of a RelayPipe.

PipeOut

Outputs of a RelayPipe; both come from registers.

Relay

One Carloni relay station carrying a T.

RelayIn

Inputs of one relay station (Carloni’s names).

RelayOut

Outputs of one relay station; every field comes from a register.

RelayPipe

PIPE_STAGES relay stations in a row: sender side first.

ToggleLink

A valid/ready link for rare, wide items, built from plain registers.

Constants

DELAY_STAGES

Registers in each Delay.

PIPE_STAGES

Relay stations in each RelayPipe.

TOGGLE_STAGES

Forward data registers in a ToggleLink after its held register.

Functions

delay_kernel

Shift the input one register along per clock.

relay_kernel

Carloni’s controller (figure 4), with the state as one bit.

relay_pipe_kernel

Wire the stations: data and void flow toward the receiver, stop flows back.

toggle_link_kernel

Load, shift and compare (see ToggleLink).