Module fabric 

The on-chip fabric of the one-card cluster: what RoCE would carry between replicas, as an RHDL experiment (D2).

What it is, and what it is not

Each replica sends RDMA writes as a meta beat (opcode, QPN, addresses, length) and data beats, exactly as it would to the RoCE stack. The fabric turns each write into AXI writes into the destination replica’s HBM region. It reproduces the parts of RoCE the source audit established (Atlas, docs/rhdl_ubft-roce.md) and nothing more:

• The host-programmed connection table. Replica i’s local QPN (N−1)·i + r (r < N − 1) reaches peer j = r if r < i, else r + 1. Any other QPN has no entry on replica i: it is not delivered and counts as unmapped. A QPN that has an entry is delivered to that entry’s peer even when the sender meant another. Upstream’s kernel indexes QPNs as id·N + j; with upstream_qpns set (the replay of upstream’s sends) the fabric counts every write whose actual peer differs from that meaning as misrouted.
• No loopback. No QPN reaches the sender itself.
• Packet boundaries. A write takes data beats up to the first tlast. Beats past len bytes before tlast are not written and count as oversize; a tlast before len bytes ends the write short (short_packet); len = 0 writes nothing (zero_length); a write past the destination region, or whose HBM address would wrap 64 bits, counts as out_of_bounds and is not written. A runtime n outside 2..=16 or an id ≥ n makes every descriptor bad_config (nothing is delivered). The final beat’s strobe covers only the bytes left of len.
• Physical lanes. AXI write lanes belong to the 64-byte-aligned address, so the payload stream is re-aligned onto lines: a write at raddr starts raddr mod 64 bytes into its first line, each line takes the previous beat’s tail and the current beat’s head (realign), and one extra line carries the last beat’s tail when the shift pushes it past. Shifts are in 4-byte steps (rhdl_ubft addresses words), so a raddr that is not a multiple of 4 is refused (misaligned), and a region base must be 64-byte aligned (else bad_config).

Receive authorization

A mapped write is then checked as a whole, before any AXI write, by Atlas’s authorize_write: the trusted ingress (this link’s id), the destination, the requested interval and W trusted WriteWindows. A rejected descriptor is drained without writing and counted once, by the guard’s reason: membership (bad_config), empty (zero_length), outside the region or wrapping (out_of_bounds), or no window (unauthorized). The windows are configuration and change only while the fabric is drained.

What the audit left unknown is a labelled assumption of this model, not a claim about RoCE: the source is the fabric port the write came in on (never a payload field); one write at a time per source, and the next starts only when every byte of the previous one is acknowledged by HBM (B responses), which keeps each source’s writes in order and makes a later flag write land after its payload. That B response is this model’s commit point, not a statement about when real RDMA data is visible. A fixed latency (cycles before a write starts) stands in for the network.

Structs

FabricLink

One source link (see the module docs).

LinkCounters

Error and progress counters of one source link.

LinkIn

One source link’s inputs.

LinkOut

One source link’s outputs.

Functions

fabric_link_kernel

The link’s logic.

realign

One 64-byte line of a re-aligned write: the payload stream shifted up by sw 4-byte words, from the previous beat’s tail and the current beat’s head (sw = 0: the current beat as it is).

realign_keep

The byte enables of a re-aligned line, shifted like realign.

strobe

The strobe of a beat with left bytes still to write.