PhantaField PFG-1 Whitepaper

A. Platform

Sophon uses the following physical stack:

Total stack height: ~22 µm above the Si die (64 tiers × 0.35 µm/tier). The 90 nm-pitch MIV
grid provides 1.23 × 10⁸ slots/mm² available inter-tier connections; the design populates only ~5.5 ×
10⁵/mm², leaving > 99% MIV headroom.

Tiers are not split within a single layer; instead the 64-tier stack
interleaves dedicated logic and memory tiers in an A/B/A/B… repeating pattern. Two adjacent
tiers form one logic-plus-memory doublet; the stack contains 32 such doublets:

• Logic tiers (32 × 750 mm² = 24,000 mm² total MAC area): 2D-TMD CMOS MAC array on
  odd-indexed tiers — MoS₂ n-FETs for NMOS, WSe₂ p-FETs for PMOS. Density 0.175 TFLOPS FP8/mm² (0.0875
  TFLOPS BF16/mm²). Clocked at 1.2 GHz, V_dd = 0.6 V.
• Memory tiers (32 × 750 mm² = 24,000 mm² total memory area): 2T0C 2D-TMD DRAM on
  even-indexed tiers, fabricated at the Metal-3 BEOL of that tier. Each memory tier sits directly above its
  paired logic tier; vertical Monolithic Inter-tier Vias (MIVs) on a sub-100 nm pitch carry
  bit-line/word-line/sense signals straight up from the logic MAC array into the cells, giving every MAC its
  own private vertical port to local weights with zero NoC traffic. This interleaved arrangement preserves
  the same total area and capacity as a hypothetical in-tier 50/50 split, while doubling the per-tier MAC
  routing area and shortening MAC-to-cell signal paths to a single tier-pitch of 0.35 µm.

Why 2D TMD? TMD CMOS (MoS₂ / WSe₂) is the only transistor technology that simultaneously
offers: (1) BEOL-compatible growth at ≤ 450 °C [6]; (2) atomic-scale
channel thickness eliminating short-channel leakage [1][2]; (3) electron mobility ≥ 120 cm²/V·s
[4]; and (4) intrinsic radiation hardness (no buried-oxide trap volume).
Critically, the TMD off-current density J_off ≈ 10⁻¹⁵ A/µm (1 fA/µm) at 28 nm — i.e. ≈ 0.5 fA for
a 0.5 µm-wide cell transistor, roughly 4 orders of magnitude lower than Si NMOS at equivalent gate length
[2][3] — is what enables a 2T0C cell to
retain data for seconds without any storage capacitor [8][9], keeping the cell area at 8 F² rather than the ~20 F² needed for a
conventional 1T1C DRAM.

B. PFG-1 “Sophon” — 2T0C DRAM die

Sophon places a 2T0C 2D-TMD gain-cell DRAM (8 F², 1 bit/cell) at the Metal-3 BEOL of each
memory tier. The cell structure is shown in Figure 2 and consists of:

• Write Transistor (WT): a TMD nFET gated by the Write Word-Line (WWL), which charges the
  storage node to V_dd or discharges it to GND.
• Read Transistor (RT): a TMD nFET whose gate is the storage node; its drain current
  indicates the stored bit.
• Storage node: the parasitic gate capacitance of RT (~2.5 fF at 28 nm TMD) plus the
  junction capacitance of WT’s drain (~0.5 fF). No explicit Metal-Insulator-Metal (MIM) or trench capacitor
  — that is the “0C” in 2T0C.

The TMD off-current density of 1 fA/µm (I_off ≈ 0.5 fA for a 0.5 µm cell transistor) gives
retention τ = C·V_dd / (2·I_off) = 1.8 s at 25 °C
[8][9] — see Eq. 3 and
Figure 3 for the retention curve. Sophon refreshes every 1.0 s (1.8×
margin), consuming only ≈ 0.08 W for the full 330 GB die (Eq. 4).
Retention derates ≈ 2× per 10 °C; above 60 °C junction temperature, on-die thermal sensors shorten the
refresh interval (≈ 159 ms at 60 °C, ≈ 28 ms at 85 °C), with refresh power staying below ~ 4 W even in the
hot corner.

Because the storage node is writable on every cycle, Sophon supports in-place BF16 gradient accumulation
with unlimited endurance — exactly what training requires — while the same array, read-only, serves the
inference decode loop. The die loads a model once and either serves it (inference) or updates it in place
(training); a powered-off die reloads its weights from off-die Non-Volatile Memory express (NVMe) at boot
(§11.2).

C. Die Floorplan & On-Die System Organization

The 131,072 CIM tiles are not a flat array — they are partitioned across the 32 logic tiers of the stack
(§2.A), exactly 4,096 tiles per logic tier (derived: 131,072 ÷ 32). Each tile occupies a
fixed cell on its tier and is the atomic unit of compute, storage, and redundancy: a 256×256 weight subarray
(65,536 weights) feeding a binary sense amp and an 8-level adder tree, with bit-serial activation broadcast
at 500 MHz (16 cycles BF16, 8 cycles FP8). The weights for every tile live in the 2T0C cells of the memory
tier directly above it (§2.B), so a tile is physically a vertical logic-plus-memory column, not a planar
block. A tier is therefore a 4,096-tile mesh of these columns; the full die is 32 such meshes stacked at
0.35 µm pitch, with the 28 nm Si base below carrying everything that is not compute.

The NoC is a per-tier 2D mesh, not a global fabric. Each logic tier runs its own mesh
router fabric at ≈ 290 TB/s bisection, and the 64 tiers together present
18,560 TB/s aggregate (derived: 290 × 64). What rides the NoC is deliberately minimal:
activations and partial sums — the operands that must move between tiles to assemble a
layer’s output across the 4,096-tile fan-in. Weights never touch the NoC. Every weight is
read through its tile’s private vertical MIV port — a single tier-pitch hop straight down from the cell to
its MAC — delivering 4.2 PB/s of in-tile weight bandwidth with zero shared-bus contention (§2.A). This is
the load-bearing asymmetry of the floorplan: the multi-petabyte traffic (weight fetch) is kept entirely
vertical and local, so the lateral NoC only ever carries the comparatively small activation/partial-sum
flux. The base-tier NoC root stitches the per-tier meshes together and bridges them to the
controller and host I/O, but it is never in the weight path.

Each tile additionally owns a small SRAM scratchpad for activations. Because the NoC
carries activations and partials rather than weights, the scratchpad is where a tile stages its inbound
activation vector, accumulates its slice of the partial sum across the bit-serial broadcast, and buffers the
outbound result before it is handed to the mesh. Holding the live activation working set in fast local SRAM
— adjacent to the adder tree, not in the 2T0C DRAM — keeps the broadcast/accumulate inner loop entirely
on-tile and lets the 1 Hz-refresh gain-cell DRAM (§2.B) stay dedicated to weights and KV cache, whose access
pattern is read-mostly and latency-tolerant by comparison.

Clock and power are delivered down the 22 µm stack to a low-voltage rail. The logic tiers
are clocked at 1.2 GHz from a base-tier clock root distributed upward through the MIV grid;
the bit-serial activation broadcast runs on a separate 500 MHz domain. Operating at
V_dd = 0.6 V is what makes a 64-tier monolithic stack thermally viable — dynamic
power scales with V_dd², so the 0.6 V rail draws ≈ 2.8× less energy than a nominal 1.0 V CMOS rail
at the same activity. The trade is current: at fixed power, lowering the voltage raises the supply current,
and that current must reach every tier through a power-delivery network (PDN) that climbs the full ~22 µm of
stack. Because the design leaves > 99% of the MIV grid unused for signaling (§2.A), those spare vias can
be allocated to the PDN (derived) — parallel V_dd/GND vias carried straight up to each logic tier
hold IR-drop in check across the stack while the bit-serial broadcast switches thousands of tiles in
lockstep.

The 28 nm Si base tier is the system’s front door. It carries the controller, the NoC root,
host I/O, and the PCIe/NVLink-class PHY — all in mature bulk-Si CMOS, where high-speed analog SerDes and
large I/O drivers belong, rather than in the BEOL 2D-TMD tiers above. This separation is what lets the same
die both serve and train without hardware change: the host loads a model once through the
base-tier PHY into the on-die 2T0C DRAM, after which the controller either drives the inference decode loop
(weights read-only) or runs in-place gradient writes for training (§2.B) — and a fleet repartitions between
the two by command, not by re-spinning silicon. An 80B model — weights, optimizer state, activations, and KV
cache — resides entirely on the single die, with every MoE expert resident on-die and only the routed
experts drawing power.

A.1 PFG-1 “Sophon” — 2T0C 2D-TMD gain-cell DRAM weight/gradient cell

The 2T0C gain cell consists of two 2D-TMD transistors and zero explicit storage capacitors
[8][9][10]. It exploits the anomalously low off-current of TMD field-effect
transistors — a width-normalized density of J_off = 10⁻¹⁵ A/µm (1 fA/µm) at 28 nm
[2][3], i.e. only
≈ 0.5 fA for a 0.5 µm-wide Read Transistor — to retain charge on the gate parasitic of the
Read Transistor (RT) for seconds without a Metal-Insulator-Metal (MIM) or trench capacitor.

Cell structure:

• Write Transistor (WT): TMD nFET, gate driven by the Write Word-Line (WWL). Drives the
  storage node to V_dd (write “1”) or GND (write “0”).
• Read Transistor (RT): TMD nFET, gate = storage node, source grounded, drain = Read
  Bit-Line (RBL). When storage = V_dd, RT conducts; when storage = 0, RT is off. Binary current
  sense.
• Storage node: parasitic C_gs of RT (~ 2.5 fF) + C_junction of WT
  drain (~ 0.5 fF) = ~ 3.0 fF total. No explicit capacitor — that is the “0C” in 2T0C.

Retention physics (Eq. 3, derived from
[8]): τ = C_node · V_dd / (2 · I_off). At
C_node = 3.0 fF, V_dd = 0.6 V, and I_off = J_off · W_RT =
1 fA/µm × 0.5 µm = 0.5 fA at 25 °C, τ = 1.8 s. Sophon refreshes every
1.0 s (1.8× margin). Retention derates ≈ 2× per 10 °C; above 60 °C junction temperature,
on-die thermal sensors shorten the refresh interval (≈ 159 ms at 60 °C, ≈ 28 ms at 85 °C).

Why a capacitor-less gain cell? A conventional 1T1C DRAM needs a ~ 20 F² trench/MIM
capacitor that is incompatible with low-temperature BEOL M3D integration. The 2T0C cell stores charge on the
Read Transistor’s own gate parasitic, so it is built entirely with the same TMD transistors used in the MAC
array — no separate capacitor module, no third-party Intellectual Property (IP) license — and the
multi-second retention enabled by the 1 fA/µm off-current makes refresh power negligible (≈ 0.08 W,
Eq. 4).

A.2 Per-doublet and per-die capacity

The stack contains 32 doublets (one logic tier + one memory tier per doublet). Each doublet
contributes one logic-tier’s MAC area and one memory-tier’s storage area; the total active MAC area and
memory area are therefore identical to a hypothetical 64-tier in-tier-split presentation, but routing is
denser because each logic tier no longer competes for footprint with its memory bank.

Sophon holds 330 GB. For training, an 80B-parameter BF16 model (160 GB) plus first-order
optimizer state (160 GB for SGD-momentum or Lion) = 320 GB, leaving
10 GB for gradient-checkpointed activations (Section 5.B.2). For
inference, an 80B BF16 model (160 GB) leaves 170 GB free, or an 80B FP8 model (80 GB)
leaves 250 GB free for an extended Key-Value (KV) cache or a co-resident draft model (Section 5.A).

A.3 Gain-Cell Read/Write Operation & Sense Margin

Sections A.1 and §2.B describe the structure of the 2T0C cell; this subsection describes how it is
operated cycle-by-cycle. The two-transistor topology decouples the write path from the read path
entirely — the Write Transistor (WT) owns the storage node, the Read Transistor (RT) only senses it — which
is precisely what enables the same array to stream weights to the MAC on every cycle while remaining
in-place writable for gradient accumulation (§3.C).

Write. A write asserts the Write Word-Line (WWL), turning the WT on and connecting the
storage node (RT gate parasitic ~2.5 fF + WT drain junction ~0.5 fF ≈ 3.0 fF) to the Write Bit-Line. The WT
channel then charges the node to V_dd = 0.6 V for a “1” or discharges it to GND for a “0”; WWL is
de-asserted and the TMD off-current (≈ 0.5 fA per 0.5 µm cell) traps that charge for the full retention
window. The transferred charge is C_node · V_dd ≈ 3.0 fF × 0.6 V, and the measured write
energy is 20 fJ/bit — a single channel charge-transfer event, with no high-voltage charge
pump and no oxide stress. Because both the value being written and the in-place gradient update (§3.C) take
this identical path, training and inference share one write primitive.

Read — the gain-cell mechanism. The defining property of the cell is that
RT’s gate is the storage node, so the stored level directly modulates RT’s drain
conduction. To read, the Read Bit-Line (RBL) is precharged and RT’s drain is enabled: a stored V_dd
turns RT on and sinks current; a stored GND leaves RT off. A binary sense amplifier on the
RBL resolves the resulting current into a digital bit in ≈ 3 ns at 30 fJ/bit. Critically,
this is a non-destructive read: RT senses the node as a gate voltage and draws no
charge out of it — unlike a 1T1C cell, where the read dumps the storage capacitor onto the bit-line by
charge-sharing and the bit must be written back before the next access. With no write-back cycle, the array
can be read back-to-back every cycle, which is exactly how it feeds the 500 MHz bit-serial activation
broadcast and the 4.2 PB/s in-tile weight bandwidth (§3.B) without ever stalling for restore.

Sense margin & why sensing is digital. The read window is set by RT’s on/off
drain-current ratio. The same 1 fA/µm TMD off-current that gives multi-second retention also collapses the
“0” leg of the read to the sub-femto-amp floor, while the “1” leg conducts at the full TMD on-current — an
on/off ratio of many decades. That enormous, deterministic separation means the sense amp only ever has to
decide “conducting vs. not,” so a single current-comparator threshold suffices:
no ADC, no DAC, no reference ladder. This is what keeps the read path pure-digital and
deterministic end-to-end — there is no analog accumulation to quantize, consistent with the ADC-free CIM
tile architecture (§3.D).

Disturb, retention & endurance during operation. Because a read is gate-voltage sensing
through RT and never discharges the node, read-disturb is negligible — a cell can be read
arbitrarily many times between refreshes with no charge loss, so the refresh cadence is governed solely by
leakage, not by access traffic. Retention τ = C_node · V_dd / (2 · I_off) =
1.8 s at 25 °C fixes the 1 Hz refresh (1.8× margin, ≈ 0.08 W for 330 GB;
see A.1). Writes are likewise benign: the bit is set by gate-controlled charge transfer through the WT, with
no oxide tunneling and no filament formation, so there is no wear-out mechanism and
endurance is effectively unlimited — the enabling condition for streaming in-place gradient
writes throughout a full training run (§3.C).

B.1 Weight bandwidth (memory → local MAC)

Each BF16 MAC reads 16 bits from the DRAM bank directly above its tile at 30 fJ/bit with 3 ns latency. The
bit-serial multiply runs at the 500 MHz wordline rate over 16 cycles for BF16 (8 cycles in FP8 inference
mode); the per-column sense amplifier produces a 1-bit partial product per cycle that feeds an 8-level
binary adder tree. A 4-stage pipeline hides DRAM latency.

Sophon delivers 4.20 PB/s of aggregate weight bandwidth in either datatype — the byte-rate
of weight consumption is the same: 2 bytes/BF16-MAC at 2,100 TFLOPS, or 1 byte/FP8-MAC at 4,200 TFLOPS, both
producing 4.20 PB/s. This bandwidth is in-tile and never crosses the Network-on-Chip (NoC).

Why is weight bandwidth independent of datatype and of capacity? In a Compute-In-Memory
architecture, weight bandwidth is set by the MAC array’s weight-consumption rate, which is
intrinsic to the logic tiers, while capacity is set by the
memory-tier areal density (110.0 Mb/mm² for 2T0C DRAM, §3.A). Because every weight is physically
co-located with the MAC that consumes it, there is no shared bus whose width would scale with total stored
bytes or with bit-depth: a higher-bit datatype simply reads more bits per MAC at a proportionally lower
MAC rate. The bandwidth equality is therefore a direct consequence of
BW = (bytes per MAC) × (MAC rate) being identical for both modes (1 B × 4,200 TFLOPS = 2 B ×
2,100 TFLOPS = 4.20 PB/s).

B.2 Gradient bandwidth (training write path)

During the backward pass, accumulated gradients are written back to the DRAM bank at 20 fJ/bit:

Inference uses the read path only and incurs none of this write power.

B.3 Activation bandwidth (per-tile SRAM scratchpad)

Activations occupy a small per-tile SRAM scratchpad (SPM) (5% of tier area, ~37.5 mm²/tier, ~0.7 GB/tier):

• Per-tier activation bandwidth: ~11,000 GB/s aggregated
• Total activation bandwidth: ~700 TB/s

B.4 NoC bandwidth (inter-tile)

A 2-D mesh NoC routes activations and control. Each tier has its own mesh; vertical MIVs carry inter-layer
activations.

B.5 Bandwidth summary

Sophon provides ~ 191× more weight bandwidth than NVIDIA Rubin (R200) and
~ 214× more than AMD Instinct MI455X (4,200 TB/s vs 22 TB/s for an 8-stack HBM4 package on
Rubin, and 19.6 TB/s for an 8-stack HBM4 package on MI455X
[16][18]) — because that bandwidth is
intrinsic to the storage location, not a separate interconnect. Figure 4 plots the
comparison.

C.1 Energy per MAC operation

Convention note: throughout this paper, “2,100 TFLOPS BF16” and “4,200 TFLOPS FP8” count each
multiply-accumulate (MAC) as 2 floating-point operations (one mul + one add)
[16]. Energies tabulated below are stated per MAC (per
weight processed), so per-FLOP figures are half the listed values. The chip-power calculations in §C.3 use
the per-FLOP convention to align with the TFLOPS rates.

Architecture note: Sophon uses pure digital Compute-In-Memory (CIM). Each tile contains
a per-column sense amplifier feeding an 8-level binary adder tree that produces the partial sum for one
row of a 256×256 weight subarray. All multiply-accumulate arithmetic is performed in the binary domain
with full deterministic 16-bit (BF16) or 8-bit (FP8) precision — see §3.D for the digital-CIM tile
walkthrough and §3.D.2 for why this choice constrains throughput as 1/N in the dense-decode regime.

C.2 Static and refresh power

Sophon’s near-zero idle is an operational advantage: an 80B model loaded into Sophon waits for requests at
~ 2–3 W. An equivalent HBM4-based GPU (e.g. NVIDIA Rubin (R200) with 288 GB, or AMD Instinct MI455X with 432 GB) holds its
HBM4 memory subsystem in self-refresh at ~ 10–15 W. With the 2D-TMD off-current at 1 fA/µm (I_off ≈ 0.5 fA per cell),
the 2T0C retention time rises to 1.8 s and the array needs only a
1 Hz refresh, costing ≈ 0.08 W. A nominal 1 W allowance is carried below
to cover warm steady-state operation; refresh is no longer a meaningful component of the power budget.

C.3 Active power by phase

C.4 Efficiency comparison

On peak compute, the 2026 HBM4 GPUs now lead: Rubin (R200) and MI455X reach ~ 4.86 and ~ 5.88
BF16 TFLOPS/W respectively, roughly 1.3–1.6× Sophon’s 3.72 — they pack ~ 4–5× more peak FLOPS behind a 3 nm
process. That advantage simply does not help at low batch. For inference, Sophon’s FP8-mode decode at 25.8
mJ/token is ~ 174× lower energy per token than either HBM4 GPU (~ 4,480 mJ/token), because at
B=1 both GPUs are HBM-bandwidth-bound and their adder energy is irrelevant — bandwidth, not FLOPS, governs.
The digital adder tree keeps per-MAC energy low in both forward and backward passes and the 1
fA/µm off-current keeps refresh negligible (≈ 0.08 W), so Sophon spends ~ 3 W at idle vs. ~ 10–15 W for
Rubin’s 288 GB and MI455X’s 432 GB HBM4 subsystems in self-refresh.

D.1 Tile geometry

Each Sophon tile is a 256×256 DRAM subarray with co-located digital MAC circuitry. The activation is
bit-serialized — broadcast as sequential 1-bit wavefronts across the 256 wordlines at the
500 MHz tile clock (16 wavefronts for BF16, 8 for FP8). Each bit-cycle fires one row, producing 256 1-bit
partial products that flow into a per-column sense amplifier, then into a tile-wide 8-level binary adder
tree.

In FP8 inference mode the same tile geometry runs an 8-cycle bit-serial activation (vs 16 for BF16),
doubling the MAC rate to 4,200 TFLOPS FP8.

D.2 Why digital CIM still scales as 1/N

A common misconception about CIM is that “all the math happens in parallel inside the memory, so model size
shouldn’t matter.” This is true for weight transport, but not for
MAC execution. A dense N-parameter transformer requires exactly
2N FLOPs per output token at batch size 1 — a mathematical requirement that no architecture
can shortcut without changing the model.

For Sophon FP8 inference at 2,100 TMAC/s aggregate:

The slope is strictly inverse to N because each weight stored in the DRAM array
participates in exactly one MAC per token, and the aggregate MAC ceiling is fixed by the tile count.

D.3 What CIM eliminates vs. what it preserves

Both fall as 1/N — only the absolute curve height differs. Sophon sits ~ 48× above NVIDIA
Rubin (R200) and ~ 53× above AMD Instinct MI455X on the FP8-mode decode tokens/s curve
because (a) zero weight-transport overhead (Rubin and MI455X decode at low batch are HBM-bandwidth-bound at
their 22 TB/s and 19.6 TB/s HBM4 ceilings respectively — only ~ 300 and ~ 270 tok/s for an 80B FP8 model),
(b) lower energy per MAC, and (c) sufficient peak MAC throughput at batch-1, where memory bandwidth — not
peak FLOPS — governs. Both GPUs in fact carry ~ 4–5× more peak FP8 FLOPS per die than Sophon (Sophon BF16
dense is just 0.24× Rubin and 0.21× MI455X), yet that raw peak buys them nothing at low batch: the weights
must still stream over HBM4 every token.

D.4 What WOULD break 1/N — and what we picked

Three architectural or algorithmic paths can break the dense-decode 1/N curve:

1. Per-cell dedicated MAC units — give each of the 80 × 10⁹ cells its own dedicated MAC.
   Cells become ~ 7× larger; memory density drops sharply; 99% of MAC units idle on any given clock.
   Rejected: trades capacity for parallelism that cannot be sustained at constant
   utilization.

2. Speculative decoding — run a small draft model ahead, verify with the large model.
   Effective speedup of ~ 2.5× when the draft (1 B parameters, ~ 1.25% of Sophon’s MAC budget) co-resides
   on the same die. Selected as Sophon’s default inference deployment mode — see §5.A.6.

3. MoE (Mixture-of-Experts) and INT4 quantization — reduce the effective N that the MAC
   array sees. MoE shrinks active N by ~ 4–50× (e.g., DeepSeek-V3 671 B → 37 B active ≈ 18×); INT4 halves
   the cycle count by halving activation bit-depth.
   Both supported as first-class workloads, with combined effective throughput documented
   in §5.A.6.

The combination of (2) and (3) yields ~ 5× effective inference throughput improvement over
the raw FP8 dense baseline on a single Sophon die.

Figure 4 plots the weight bandwidth comparison. Figure 5 decomposes
per-MAC energy by component. Figure 6 shows the resulting active-power breakdown by
workload phase.

D.5 Mapping a Transformer Layer onto the Tile Array

Sections D.1–D.2 fixed the tile geometry and the dense-decode 1/N ceiling; this subsection shows the
dataflow — how a transformer layer’s matmuls physically land on the 131,072 tiles and how
partial results are stitched back together. The organizing principle is
weight-stationary execution: a weight never moves. Every weight matrix W is tiled
into 256×256 blocks, and each block is resident in the 2T0C 2D-TMD DRAM doublet sitting
directly above its MAC tile. A tile reads its ≈ 64 KB of FP8 weights (256×256 bytes) through a
single private vertical MIV hop (§3.A) — there is no NoC traversal, no shared weight bus, and no off-die HBM
fetch. This is the source of the 4.2 PB/s in-tile weight bandwidth (§3.C): bandwidth is the product of
131,072 independent ports each one MIV-via deep, not a wide shared channel that must be arbitrated.

Within a tile, computation is bit-serial (§D.1). The activation vector is broadcast as
sequential 1-bit wavefronts down the 256 wordlines at the 500 MHz tile clock — 8 wavefronts for FP8, 16 for
BF16. On each bit-cycle the tile fires one row, the binary sense amps capture 256 1-bit partial products
against the stationary weight column, and the 8-level adder tree reduces them to one column partial sum.
After the full bit-serial sweep, every tile holds a 256-wide block partial sum for the slice of the output
dimension it owns. Because activation is the only thing that flows in and the weight is the only thing that
stays, energy per MAC is dominated by the local DRAM read (0.240 pJ of the 0.310 pJ FP8 total, §3.C) rather
than by data movement across the die.

A single 256×256 tile covers only a 256-element slab of a real projection, so a full output dimension is
assembled by cross-tile reduction. Tiles whose blocks share an output row form a reduction
group; their partial sums are summed across the on-die NoC (≈290 TB/s per tier, 18,560 TB/s aggregate over
64 tiers, §3.C) and accumulated into the per-tile SRAM activation scratchpad. Only these reduced activations
— never weights — travel on the NoC, so the interconnect carries the small O(d_model) activation
traffic of a layer rather than the O(N) weight traffic that bandwidth-bounds a GPU. The reduced output
vector then becomes the broadcast activation for the next layer’s tile group, and the layer pipeline
advances.

Mapping a complete transformer block follows directly. The four attention projections
W_Q/W_K/W_V/W_O are each laid out as their own
contiguous group of weight-stationary tiles; the QK^⊤ score and the score·V product run on the
same tile fabric with the K and V tensors held in the on-die 2T0C DRAM. Crucially, the
KV cache lives in that same on-die DRAM as the weights — each decode step writes the new
K/V rows in place (20 fJ/bit gradient-class write path, §3.C) and reads the accumulated cache back through
the local MIV port, so there is no off-die HBM round-trip per token. The FFN’s up/down projections occupy a
larger tile group sized to the expansion ratio. For MoE, every expert is permanently
resident on-die across distinct tile groups (§System): routing does not gather or stream weights — it simply
selects which tiles fire. Un-routed experts hold their weights stationary and draw only idle power
(≈2–3 W), so a sparse 80B-class deployment consumes energy proportional to the active parameter count, not
the resident parameter count — the mechanism behind the MoE energy-ceiling analysis and the serving curves
of (§5.A).

The same physical tiles run train-then-serve with no hardware change. In serving mode the
DRAM is read-only: activations sweep forward through the projection and FFN/MoE groups, the KV cache grows
in place, and decode draws ≈373 W (FP8). In training mode the identical tiles run the forward pass and then
the backward pass over the writable 2T0C DRAM, performing
in-place gradient accumulation through the dedicated grad-write path (0.320 pJ of the 0.940
pJ BF16 training MAC, §3.C) — weights are updated where they sit, again with no weight transport. Because
the only difference between the two modes is whether the local DRAM port is exercised read-only or
read-modify-write, a fleet repartitions between training and serving purely in software: a die that trained
a checkpoint at midnight can serve it at noon on exactly the same tile array (§5.A).

4.1 2T0C gain-cell DRAM

Setup: write 1 at t = 0; hold; read at t = 1.0 s.

The stored voltage at the 1.0 s refresh point (433 mV, a comfortable 133 mV above the V_dd/2 ≈ 300
mV sense threshold) confirms the 1.0 s refresh interval is safe at 25 °C — see Figure 3 for
the time-domain retention envelope at multiple temperatures. Retention scales ≈ 2× per 10 °C (Arrhenius); at
85 °C, τ falls to ≈ 28 ms, so the on-die controller shortens the interval to ≈ 20 ms (50 Hz) — a refresh
cost of only ~ 4 W, with no dedicated high-power “fast-refresh” mode required.

4.2 Latch sense-amplifier

Binary current sense: a single latch fired against a fixed mid-point reference. The 1-bit output drives
directly into the per-tile binary adder tree.

4.3 Thermal RC

34-node thermal network solved at DC for peak training power injection (749 W backward pass). Stack ΔT
remains sub-Kelvin; package resistance dominates (see Section 6).

5.1 Die stack overview

5.A. Inference

Sophon serves inference on the same silicon it trains on. The MAC array supports both native BF16 (the
training datatype) and an FP8 inference mode (4,200 TFLOPS / 8,400 INT8 TOPS); FP8 is the recommended
serving mode because it doubles decode throughput, halves energy/token, and frees capacity. The model loads
once and serves indefinitely; a powered-off die reloads from NVMe at boot (§11.2).

5.B. Training

5.C. Train-then-serve system view

Because inference and training run on the same die, a production AI cluster is built from a
single Sophon Stock-Keeping Unit (SKU) and repartitioned by software:

This flow lets a single fleet elastically shift dies between training and serving without
any hardware swap: the same silicon that trained a model can serve it (BF16 directly, or FP8 after a
one-step quantization), and dies can be re-tasked from serving back to fine-tuning as demand shifts. The
only operational discipline DRAM imposes is volatility management — weights are checkpointed to NVMe and
reloaded at boot (§11.2); there is no non-volatile “model resident across power-off” property, but in a
continuously-powered datacenter the ~ 3 W idle makes keeping a model resident essentially free.

Steady-state at design power

All liquid-cooled operating points — including the 100% fwd+bwd peak (1,362 W → 94.8 °C) — stay below
T_jmax = 105 °C on a standard liquid cold plate. Refresh is negligible (≈ 0.08 W at 1 Hz, from
the 1 fA/µm off-current) and does not enter the thermal budget.

Key results

• The intrinsic stack ΔT is negligible (≤ 1.7 K at any tier count and any power level in
  this study), because each tier is only 0.35 µm thick and the Cu-MIV network conducts heat efficiently.
• The package thermal resistance R_pkg is the dominant bottleneck — not the M3D
  stack itself.
• Inference (373 W FP8 decode, 681 W FP8 peak burst) runs at T_j = 44 °C at
  decode and 60 °C at peak burst on a liquid cold plate. At the read-corrected decode power, air cooling is
  not sufficient — a 0.30 K/W air path puts decode at ~ 137 °C, above T_jmax — so Sophon
  is a liquid-cooled part, consistent with datacenter AI deployment.
• Training time-average (564 W) gives T_j = 53.9 °C under liquid cooling —
  comfortably below T_jmax and within the 2T0C retention model (τ = 1.8 s at 25 °C, ≈ 159 ms at 60
  °C). Because the 1 fA/µm off-current makes refresh negligible (≈ 0.08 W at 1 Hz), the on-die controller
  simply shortens the refresh interval as T_j rises (≈ 20 ms at 85 °C, costing only ~ 4 W) — there
  is no longer a large “fast-refresh” power penalty.
• The peak fwd+bwd burst (1,362 W → 94.8 °C) stays within T_jmax on a standard
  liquid cold-plate; sustained 100% fwd+bwd duty is supported without microfluidic cooling.

Maximum sustained power vs. cooling technology

Inference (373 W FP8 decode, 681 W peak) fits comfortably within liquid cold-plate limits and is within
striking distance of standard air cooling at decode — the chip can operate without any liquid plumbing in
edge-inference deployments at moderately reduced clock rates. The
training time-average (564 W) also fits liquid cold-plate with wide margin, and even the
fwd+bwd 100%-duty peak (1,362 W → 94.8 °C) stays within T_jmax on a standard liquid cold plate,
with refresh a negligible ≈ 0.08 W.

Per-tier temperature with an Al₂O₃ inter-tier dielectric

The stack ΔT above used a generic BEOL dielectric (k_BEOL = 2.0 W·m⁻¹K⁻¹). Specifying the
inter-tier dielectric as Al₂O₃ changes vertical conduction only marginally:
BEOL-compatible ALD Al₂O₃ grown at ≤ 450 °C is amorphous, with a thin-film thermal conductivity of
k_d ≈ 1.8 W·m⁻¹K⁻¹ (bulk single-crystal sapphire reaches ~ 30 W·m⁻¹K⁻¹, but
that phase is unreachable in a low-temperature BEOL flow). Because the 6% Cu-MIV via fill dominates the
parallel vertical path, the effective conductivity is essentially unchanged from §6:

$$k_⚡ = f_💬\,k_{\text{Cu}} + (1-f_{\text{Cu}})\,k_d = 0.06\,(380) + 0.94\,(1.8) = 24.5\ \text{W·m}^{-1}\text{K}^{-1}$$

Heat exits through the base (backside cold plate), so the top tier is hottest. Conservatively routing the
full die power P through the stack to the base — the same lumped convention as the
ΔT_stack column above — tier i (counted from the base, i = 0…N, N = 64) sits at the
package-limited base temperature plus the through-stack rise:

$$T_i = \underbrace{T_{\text{cool}} + P\,R_{\text{pkg}}}_{\text{base tier}} \;+\; \frac{i}{N}\cdot\frac{P\,L_{\text{stack}}}{k_{\text{eff}}\,A}, \qquad L_{\text{stack}} = 22.4\ \mu\text{m},\ \ A = 7.5\ \text{cm}^2$$

On a liquid cold plate (R_pkg = 0.05 K/W, 25 °C coolant) the as-built stack — Al₂O₃ dielectric
with the 6% Cu-MIV via network — gives the per-tier profile below.

Every one of the 64 tiers sits within ≤ 1.7 K of the base — the top tier reaches only
53.9 °C at the 564 W training average and 94.8 °C at the 1,362 W fwd+bwd
peak, both inside T_jmax = 105 °C. With the 6% Cu-MIV via network carrying the vertical heat, the
Al₂O₃ dielectric is nearly thermally invisible: swapping it for the generic 2.0 W·m⁻¹K⁻¹ BEOL value shifts
k_eff by < 1%. These are conservative bounds — per-tier dissipation is distributed across the
64 tiers rather than injected at the top, which halves the through-stack term and flattens the profile
further.

PFG-1 “Sophon” Roadmap (2T0C DRAM)

Every die is held to a fixed 1,500 W package power envelope, so the roadmap scales along
two independent axes. Capacity grows with cell density and tier count — the 2T0C array is
read-mostly and the 1 fA/µm off-current keeps refresh at ≈ 0.08 W, so memory is not power-bound and climbs
from 330 GB to 26 TB unconstrained. Compute throughput, by contrast, is bounded by the
1,500 W package: each shrink improves energy efficiency (tok/W), and within the same 1,500 W that buys more
throughput — but only at the efficiency rate, not the raw-tile rate. From the 10 nm node on, the die has far
more tiles than 1,500 W can switch at once for an 80B decode, so the reported throughput is the
power-capped figure (1,500 W × tok/W); the surplus tiles hold weights (capacity), not active
compute. A 7 nm die thus pairs 26 TB of on-die memory with a power-capped ≈ 149,700 tok/s 80B decode, rather
than the ≈ 577,000 tok/s an uncapped ≈ 5.8 kW die would draw. Decode at 28, 22, and 14 nm stays below the
cap (373 W, 627 W, 1,351 W) and is tile-limited as before.

Comparison with HBM roadmap (8-stack package)

Sophon widens its capacity lead against HBM every generation. More importantly, the
bandwidth lead is already insurmountable: 4.20 PB/s vs. HBM4’s ~ 20 TB/s (8-stack package; Rubin 22, MI455X 19.6)
— a ~ 191–214× gap that no interposer-based approach can close.

Total Cost of Ownership (TCO) and Bill of Materials (BOM)

Comparison vs. NVIDIA Rubin (R200) and AMD Instinct MI455X + HBM4

The cost wall is the memory wall. Morgan Stanley estimates a single NVIDIA VR200
(Rubin) NVL72 rack at ≈ $7.8M, of which HBM memory alone is ≈ $2.0M —
25.7% of the entire rack, up +435% over the prior-generation GB300. Per
accelerator (÷ 72) that is $55,000 of GPU silicon plus $27,800 of HBM4. Sophon removes
the HBM line item in full, for a ~ 9.9–11.6× lower hardware BOM
[17].

A Rubin (R200) module ships with 288 GB HBM4 at ≈ 22 TB/s; an MI455X ships with 432 GB HBM4 at ≈ 19.6 TB/s.
Capacity is now within reach of both parts, but the matched-bandwidth scaling remains far out of reach:
HBM4 delivers ~ 22 TB/s (Rubin) / ~ 19.6 TB/s (MI455X), vs. Sophon’s 4,200 TB/s in-tile — a
~ 191× gap (vs. Rubin) / ~ 214× gap (vs. MI455X) that cannot be closed at
any price point within the interposer paradigm. The GPUs win on peak dense FLOPS (Sophon BF16 dense is
0.24× Rubin / 0.21× MI455X), but peak FLOPS do not help at low batch, where weight-fetch bandwidth governs
decode throughput: at 80B FP8, HBM-bound decode is ≈ 300 tok/s (Rubin) and ≈ 270 tok/s (MI455X) vs.
Sophon’s 14,438 tok/s — a 48× / 53× advantage.

Total Cost of Ownership (TCO) over 3-year datacenter deployment

The table below uses a representative production-server duty cycle, a Power Usage Effectiveness (PUE) of
1.5, and a $0.10/kWh electricity tariff — yielding an effective $0.15/kWh after datacenter cooling and
distribution overhead. Numbers are per single die over 3 years (26,280 hours).

Sophon’s TCO advantage comes from two compounding effects:

1. Hardware cost: ~ 9.9× lower BOM than a Rubin (R200) and ~ 11.6× lower than an MI455X.
2. Idle + active energy: at ~ 3 W idle vs. the Rubin/MI455X’s ~ 10–15 W memory-idle, and
   373 W FP8 decode vs. their ~ 1,700–1,800 W TDP, Sophon spends a small fraction of an HBM4 GPU’s combined
   idle+active energy budget. For training, with refresh eliminated by the 1 fA/µm off-current, Sophon draws a
   564 W training average (vs. a Rubin/MI455X’s ~ 1,700–1,800 W TDP) and idles at
   ~ 3 W (vs. their ~ 10–15 W memory-idle). It completes the same training work in roughly
   2.7–3.1× fewer die-seconds per token, and on an energy-per-trained-token basis is
   ~ 174× more efficient than these HBM4 GPUs (Section 5.B.5).

Cost and energy per token

Two numbers decide serving economics: energy per token — which sets the electricity
bill and the thermal envelope — and cost per token, the fully-loaded $/token a
deployment actually pays. Both follow directly from the figures above. Energy per token is the decode
power divided by the decode throughput,

$$E_{\text{tok}} = \frac{P_{\text{decode}}}{R_{\text{decode}}}$$

and the fully-loaded cost per token amortizes the 3-year TCO (hardware BOM + energy) over every token
the die serves at a 30% production duty cycle (t_3y = 9.46×10⁷ s):

$$C_{\text{tok}} = \frac{\text{TCO}_{3\text{y}}}{R_{\text{decode}} \cdot d \cdot t_{3\text{y}}}, \qquad d = 0.30$$

At batch-1, single-stream (interactive, low-latency) serving, Sophon delivers a token for
~ 2 cents per million — about 470–600× cheaper than an HBM4 GPU, at
174× lower energy per token. The cost gap is the product of two compounding effects:
the ~ 9.9–11.6× lower hardware BOM (§9) and the ~ 48–53× higher
single-stream throughput (§5.A.5). It is largest at low batch, where the GPU re-reads all 80B
weights from HBM for every token; at high batch the GPU amortizes each weight read across the batch and
its cost per token falls toward its compute-bound floor — but interactive chat, agentic loops, and
long-context decode are precisely the batch-1, memory-bound regime where Sophon governs.

Tier 1 — Column-Redundancy Repair (Yield Recovery)

Each 2D-TMD CIM tile is provisioned with 4 spare columns per 256-column bank (~1.6%
column-area overhead). Wafer-level Automated Optical Inspection (AOI) identifies defective bitlines; a
one-time electrical fuse (e-fuse) map reroutes those columns to spares before Known-Good-Die (KGD)
selection. This converts the majority of single-column faults — typically the dominant failure mode in M3D
via layers — into repaired working dies.

Tier 2 — Tile-Level Disaggregation (Partial-Good Harvesting)

Dies that fail Tier 1 repair due to clustered multi-column faults are evaluated at the
tile granularity (each die contains 576 tiles, §3.D). A die with ≤ 10% tile failures (~58
tiles) is re-characterised and deployed at reduced capacity:

Grade-B and Grade-C dies are targeted at edge-inference and MoE partial-expert deployments where capacity
headroom exceeds strict density requirements. Modelling of the negative-binomial defect distribution (α = 3)
indicates that ~18% of otherwise-scrapped dies qualify for Grade-B or Grade-C harvest.

Tier 3 — Known-Good-Die (KGD) Burn-In Protocol

All KGD candidates (full and partial-good) undergo a 24-hour elevated-voltage burn-in at
V_DD + 10% and T_junction = 85 °C to screen infant-mortality failures — primarily 2T0C
retention outliers. Post burn-in, full parametric re-test confirms:

• 2T0C retention τ ≥ 1.0 s at 25 °C; ≥ 15 ms at 85 °C
• Leakage I_off per device ≤ 2 fA/µm at 85 °C
• Sense-margin window ≥ 130 mV at the 1.0 s refresh point

Field return data from analogous 28 nm BEOL products places the post-burn-in Annualised Failure Rate (AFR)
below 0.1% per die-year — consistent with the mission-life assumptions in §6 (Thermal) and
§9 (TCO).

Combined Yield & Cost Impact

The Tier 1 + Tier 2 combined uplift reduces the effective cost per working die by ~29–30%,
tightening the BOM advantage over HBM4 systems — NVIDIA Rubin (R200) and AMD Instinct MI455X — from a list-price 9.9×/11.6× (Rubin/MI455X) to a ~14× / 16.4× realised advantage — the ≈ $5,900 effective Sophon BOM
after defect harvest, against the unchanged GPU list price.

Note on M3D-specific defect modes. The dominant yield detractor in the 64-tier 2D-TMD M3D
stack is not planar Si lithography (which is mature at 28 nm) but rather
Monolithic Inter-tier Via (MIV) open/short defects at the ~90 nm via pitch. Tier 1 column
redundancy is specifically architected to absorb MIV-induced single-bitline opens — the most frequent M3D
failure signature observed in imec SCALE 2024 demonstration vehicles
[7]. Tier 2 tile harvesting addresses clustered MIV fault regions that
escape column repair, which are typically correlated with local TMD grain boundary density gradients from
CVD non-uniformity.

10.1 Minimal single-event target — the capacitor-less cell

In a conventional 1T1C DRAM, the bit lives as charge on a deep-trench or stacked capacitor of tens of
femtofarads; the capacitor and its substrate collection volume present a large sensitive cross-section, and
a single ionizing strike that collects enough charge flips the bit [31].
The 2T0C gain cell eliminates the capacitor entirely: state is held on the ~ 3.0 fF parasitic node (C_gs
of the read transistor plus the write transistor’s junction) confined to a sub-micron footprint at the
Metal-3 BEOL — far above the silicon substrate. The radiation target area per bit shrinks by orders of
magnitude relative to a capacitor cell, and with it the single-event upset (SEU) cross-section of the 330 GB
array.

10.2 Channel on dielectric — no substrate damage path, no lattice cascade

The 2D-TMD channel is grown on amorphous dielectric, not on a bulk semiconductor. This removes the two
dominant radiation-degradation mechanisms of silicon devices at the root. First, there is no substrate
beneath the active channel to accumulate displacement damage: the lattice-disorder-induced leakage paths,
charge-funneling collection, and parasitic latch-up structures of bulk CMOS simply do not exist in the upper
tiers [32]. Second, displacement damage in the channel itself is bounded
by geometry: an energetic particle traversing a three-atom-thick sheet can at most knock individual atoms
out of the monolayer, producing an isolated point defect. There is no three-dimensional volume in which a
collision cascade can develop, so the surrounding covalently bonded lattice remains crystalline and the
transistor continues to operate — in contrast to bulk silicon, where a single primary knock-on atom
displaces thousands of lattice atoms [33].

These mechanisms are not merely theoretical. 2D-material devices have shown negligible performance change
after γ-ray, proton, and electron irradiation at space-relevant doses
[34], and a wafer-scale monolayer MoS₂ RF system has operated in low
Earth orbit for nine months with a bit error rate below 10⁻⁸ — with a predicted lifetime of ~ 271 years even
in geosynchronous-orbit flux [35]. Combined with the total-ionizing-dose
immunity noted in §2 (no buried-oxide trap vulnerability) and the seconds-scale refresh that bounds any
transient corruption window, these properties make the platform a natural fit for satellite inference
payloads. Formal SEE characterization of the full Sophon stack for LEO/MEO flux environments remains a
qualification milestone (§11.3).

A. Device physics — 2D Transition Metal Dichalcogenide (TMD) transistors

[1] Radisavljevic, B., et al. “Single-layer MoS₂ transistors.”
Nature Nanotechnology 6, 147–150 (2011). DOI: 10.1038/nnano.2010.279.
https://doi.org/10.1038/nnano.2010.279 → Source for
MoS₂ baseline mobility (~ 200 cm²/V·s), I_on/I_off > 10⁸.

[2] Liu, Y., Duan, X., Shin, H.-J., et al. “Promises and prospects of two-dimensional
transistors.” Nature 591, 43–53 (2021). DOI: 10.1038/s41586-021-03339-z.
https://doi.org/10.1038/s41586-021-03339-z → Source
for TMD I_off density ≈ 10⁻¹⁵ A/µm (1 fA/µm) at 28 nm gate length; comparative tables of MoS₂ vs
Si scaling.

[3] Lan, H.-Y., et al. “Dual-Gate Synthetic MoS₂ MOSFETs with 4.56 µS/µm g_m, 320
µA/µm I_d at 1 V V_d.” IEDM 2022 Technical Digest, paper 7.3. IEEE.
https://ieeexplore.ieee.org/document/10019462 →
Source for TMD nFET drive current, sub-threshold slope (~ 75 mV/dec), V_dd = 0.6 V operation.

[4] Sebastian, A., et al. “Benchmarking monolayer MoS₂ and WS₂ field-effect transistors.”
Nature Communications 12, 693 (2021). DOI: 10.1038/s41467-020-20732-w.
https://doi.org/10.1038/s41467-020-20732-w →
WSe₂/WS₂ p-FET hole mobilities (60–120 cm²/V·s); CMOS-pair benchmarking.

B. Compute-In-Memory and Monolithic 3D (M3D) integration

[5] Shulaker, M. M., et al. “Three-dimensional integration of nanotechnologies for
computing and data storage on a single chip.” Nature 547, 74–78 (2017). DOI: 10.1038/nature22994.
https://doi.org/10.1038/nature22994 → M3D nanosheet
proof-of-concept; demonstrates low-temperature BEOL stacking compatible with this paper’s TMD M3D approach.

[6] Vinet, M., et al. (CEA-Leti). “Monolithic 3D Integration: A Powerful Alternative to
Classical 2D Scaling.” IEEE S3S Conference 2014.
https://ieeexplore.ieee.org/document/7028181 →
Established M3D thermal budget constraints (≤ 450 °C BEOL ceiling) cited in §2.A.

[7] imec. “SCALE-3D: Scaling roadmap for monolithic 3D integration.” imec Technology Forum
2024.
https://www.imec-int.com/en/articles/monolithic-3d-integration
→ MIV (Monolithic Inter-tier Via) pitch (~ 90 nm) and density (~ 10⁸/mm²) used in §2.A.

C. 2T0C gain-cell DRAM

[8] Belmonte, A., et al. (imec). “Capacitor-less, Long-Retention (>400 s) DRAM Cell
Paving the Way Towards Low-Power and High-Density Monolithic 3D DRAM.” IEDM 2020, paper 28.2.
https://ieeexplore.ieee.org/document/9372074 →
Imec 2T0C IGZO-channel demonstration; establishes 2T0C feasibility and validates closed-form retention model
τ = C·V/(2·I_off) used in §4.1.

[9] Liu, X., et al. “A 2T0C DRAM Based on Amorphous In-Ga-Zn-O Thin Film Transistors with
Retention Time Larger Than 400 s.” IEEE Electron Device Letters 41(8), 1184–1187 (2020).
https://ieeexplore.ieee.org/document/9118898 →
Independent confirmation of long-retention 2T0C; basis for TMD adaptation in this paper.

[10] Wu, F., et al. “Vertically Stacked Multilayer Heterostructures for 2T0C DRAM.”
Nature Electronics 5, 519–526 (2022). DOI: 10.1038/s41928-022-00807-w.
https://doi.org/10.1038/s41928-022-00807-w →
2D-material-based 2T0C with sub-µm² cells; closest published analogue to the Sophon cell.

D. Energy and computation models

[11] Horowitz, M. “Computing’s energy problem (and what we can do about it).”
ISSCC 2014 Keynote. IEEE.
https://ieeexplore.ieee.org/document/6757323 →
Source for the per-operation energy model (FP add ~ 0.4 pJ @ 45 nm, scaling by V_dd²); the TMD MAC
energy in §C.1 is computed by scaling this with V_dd² ratio and 0.85× TMD device factor (from
[3]).

[12] Jouppi, N. P., et al. “Ten Lessons From Three Generations Shaped Google’s TPUv4i.”
ISCA 2021.
https://ieeexplore.ieee.org/document/9499913 →
Industrial benchmark for tile-array CIM energy per MAC and utilization figures (55% sustained, 75% peak).

[13] Patterson, D., et al. “Carbon Emissions and Large Neural Network Training.”
arXiv:2104.10350 (2021).
https://arxiv.org/abs/2104.10350 → Source for the “6 × N_params
FLOPs per training token” estimator and per-token energy framework used in §5.B.3.

[14] Kaplan, J., et al. “Scaling Laws for Neural Language Models.”
arXiv:2001.08361 (2020).
https://arxiv.org/abs/2001.08361 → Source for the 2 × N_params
FLOPs per inference token estimator used in §5.A.3.

[15] Hoffmann, J., et al. (Chinchilla). “Training Compute-Optimal Large Language Models.”
arXiv:2203.15556 (2022).
https://arxiv.org/abs/2203.15556 → Source for the 1T–15T
training-token range used in §5.B.3 cluster sizing.

E. Comparison hardware

[16] NVIDIA Corporation.
NVIDIA Rubin (R200) Architecture Technical Brief (2026).
https://www.nvidia.com/en-us/data-center/rubin/
→ Source for NVIDIA Rubin (R200) per-GPU specs: ≈ 17,500 TFLOPS dense FP8, ≈ 8,750 TFLOPS dense BF16,
288 GB HBM4, ≈ 22 TB/s memory bandwidth per GPU, ≈ 1,800 W TDP (2,300 W Max-P).
80B FP8 decode ≈ 300 tok/s (HBM-bound: 22 TB/s ÷ 80 GB), 80B batch-1 training ≈ 880 tok/s;
energy/token 4.48 J, tokens/W 0.22, BOM ≈ $82,800, TCO ≈ $85,600.

[16b] Advanced Micro Devices, Inc.
AMD Instinct MI455X Architecture Technical Brief (2026).
https://www.amd.com/en/products/accelerators/instinct/mi400/mi455x.html
→ Source for AMD Instinct MI455X per-GPU specs: ≈ 20,000 TFLOPS dense FP8, ≈ 10,000 TFLOPS dense BF16,
432 GB HBM4, ≈ 19.6 TB/s memory bandwidth per GPU, ≈ 1,700 W TDP.
80B FP8 decode ≈ 270 tok/s (HBM-bound: 19.6 TB/s ÷ 80 GB), 80B batch-1 training ≈ 785 tok/s;
energy/token 4.48 J, tokens/W 0.22, BOM ≈ $96,700, TCO ≈ $99,400.

[17] NVIDIA Corporation / Advanced Micro Devices, Inc. / Morgan Stanley Research.
Rubin (R200) and Instinct MI455X Platform Specifications (2026); and
Nvidia NVL72 Bill of Materials — GB300 vs VR200 (Morgan Stanley Research estimate, 2025).
https://www.nvidia.com/en-us/data-center/rubin/
→ Power references: NVIDIA Rubin (R200) ≈ 1,800 W TDP (2,300 W Max-P); AMD Instinct MI455X ≈ 1,700 W TDP.
Per-accelerator BOM from Morgan Stanley’s VR200 NVL72 estimate (≈ $7.8M / rack ÷ 72 GPUs): GPU silicon
≈ $55,000 + HBM4 ≈ $27,800 → Rubin BOM ≈ $82,800; MI455X scaled to 432 GB HBM4 ≈ $96,700. (These are
rack-price allocations including vendor margin; the Sophon BOM is a pre-silicon build cost.)

[18] JEDEC Solid State Technology Association. JESD270-4: HBM4 Standard (2025).
https://www.jedec.org/standards-documents/docs/jesd270-4
→ HBM4 package bandwidth: Rubin (R200) ≈ 22 TB/s, MI455X ≈ 19.6 TB/s; HBM read energy ≈ 7 pJ/bit,
κ_HBM4 = 5.6×10⁻¹¹ J·tok⁻¹·param⁻¹; used as the ~ 191×/214× weight-bandwidth baseline.

[19] JEDEC. Roadmap: HBM4 and HBM5 — preliminary specifications.
https://www.jedec.org/news/pressreleases → Source for
HBM4/HBM4e/HBM5/HBM5e roadmap capacity figures used in §7.

F. Thermal model

[20] Pop, E. “Energy Dissipation and Transport in Nanoscale Devices.”
Nano Research 3, 147–169 (2010). DOI: 10.1007/s12274-010-1019-z.
https://doi.org/10.1007/s12274-010-1019-z → Source
for BEOL effective thermal conductivity baseline (k_BEOL ≈ 2.0 W/m·K).

[21] Mahajan, R., et al. (Intel). “Cooling a Microprocessor Chip.”
Proceedings of the IEEE 94(8), 1476–1486 (2006).
https://ieeexplore.ieee.org/document/1683998 →
Source for liquid cold-plate package thermal resistance (R_pkg ≈ 0.05 K/W).

[22] Bar-Cohen, A., et al. “Embedded Cooling for Wide Bandgap Power Amplifiers.”
IEEE Trans. Components, Packaging and Manufacturing Tech. 5(9), 1226–1239 (2015).
https://ieeexplore.ieee.org/document/7173025 →
Source for microfluidic R_pkg ≈ 0.02 K/W; two-phase immersion ≈ 0.01 K/W envelope.

G. Economics and yield

[23] Cunningham, J. A. “The Use and Evaluation of Yield Models in Integrated Circuit
Manufacturing.” IEEE Trans. Semiconductor Manufacturing 3(2), 60–71 (1990).
https://ieeexplore.ieee.org/document/55438 →
Negative-binomial yield model with clustering parameter α = 3; basis for the 49.5% base yield in §9.

[24] Stapper, C. H. “Modeling of Defects in Integrated Circuit Photolithographic Patterns.”
IBM Journal of R&D 28(4), 461–475 (1984).
https://ieeexplore.ieee.org/document/5390244 →
Murphy yield model used as cross-check (51.2% for A·D₀ = 0.75) in the audit calculations.

[25] TechInsights. 28 nm Foundry Wafer Cost Analysis, 2025–2026 Update.
TechInsights subscription report; public summary:
https://www.techinsights.com/wafer-cost-analysis
→ Source for the $3,500 28 nm 12-inch wafer cost.

[26] U.S. Energy Information Administration.
Average Industrial Electricity Price, 2025.
https://www.eia.gov/electricity/monthly/ → Source for
the $0.10/kWh industrial tariff baseline used in TCO (§9).

[27] Uptime Institute. Global Data Center Survey 2024 — PUE Trends.
https://uptimeinstitute.com/resources/research/global-data-center-survey-2024
→ Source for the PUE = 1.5 assumption (industry median for liquid-cooled facilities).

I. Workload-level accelerators

[29] Leviathan, Y., Kalman, M., Matias, Y. “Fast Inference from Transformers via
Speculative Decoding.” ICML 2023.
https://arxiv.org/abs/2211.17192 → Source for the
speculative-decoding speedup model, k = 4 draft length, 70% token-acceptance rate baseline used in §5.A.6
and Eq. 17.

[30] Lin, J., et al. “AWQ: Activation-aware Weight Quantization for LLM Compression and
Acceleration.” MLSys 2024.
https://arxiv.org/abs/2306.00978
→ Source for INT4 weight-only quantization quality bounds (≤ 1–2 perplexity points vs FP8 on 70B-class
instruction-tuned models) used in §5.A.6 and Eq. 17.

J. Radiation effects & space deployment

[31] Baumann, R. C. “Radiation-induced soft errors in advanced semiconductor technologies.”
IEEE Transactions on Device and Materials Reliability 5(3), 305–316 (2005). DOI:
10.1109/TDMR.2005.853449.
https://doi.org/10.1109/TDMR.2005.853449 → Source for
the single-event-upset mechanism: charge collection onto storage nodes, and the dependence of SEU
cross-section on sensitive-node volume, used in §10.1.

[32]
Schwank, J. R., Ferlet-Cavrois, V., Shaneyfelt, M. R., Paillet, P., Dodd, P. E. “Radiation
effects in SOI technologies.” IEEE Transactions on Nuclear Science 50(3), 522–538 (2003). DOI:
10.1109/TNS.2003.812930.
https://ieeexplore.ieee.org/document/1208574 →
Source for dielectric isolation effects: reduced charge-collection volume, elimination of substrate
funneling, and latch-up immunity of devices isolated from the bulk substrate, used in §10.2.

[33]
Komsa, H.-P., Kotakoski, J., Kurasch, S., Lehtinen, O., Kaiser, U., Krasheninnikov, A. V.
“Two-dimensional transition metal dichalcogenides under electron irradiation: defect production and doping.”
Physical Review Letters 109, 035503 (2012). DOI: 10.1103/PhysRevLett.109.035503.
https://doi.org/10.1103/PhysRevLett.109.035503
→ Source for displacement-threshold energies in TMD monolayers and the isolated-point-vacancy character of
irradiation damage in atomically thin sheets, used in §10.2.

[34] Vogl, T., Sripathy, K., Sharma, A., et al. “Radiation tolerance of two-dimensional
material-based devices for space applications.” Nature Communications 10, 1202 (2019). DOI:
10.1038/s41467-019-09219-5.
https://doi.org/10.1038/s41467-019-09219-5 →
Demonstrates negligible performance change in 2D-material devices after γ-ray, proton, and electron
irradiation at space-relevant doses, used in §10.

[35] Zhu, L., et al. “Radiation-tolerant atomic-layer-scale RF system for spaceborne
communication.” Nature 650, 346–352 (2026). DOI: 10.1038/s41586-025-10027-9.
https://www.nature.com/articles/s41586-025-10027-9
→ On-orbit demonstration: a wafer-scale monolayer MoS₂ RF transmit/receive system operated at ~ 517 km LEO
for 9 months with bit error rate < 10⁻⁸, with a predicted ~ 271-year lifetime in GEO flux, used in §10.

Eq. 1 — Planar memory density

$$D_{\text{Mb/mm}^2} = \frac{10^{12}}{F^{2} \cdot F_{\text{nm}}^{2} \cdot (1 + p)} \cdot b$$

where F² is the cell footprint in lithographic squares (8 for the 2T0C DRAM cell), F_nm is the
half-pitch in nm (28 nm baseline), p is the periphery overhead fraction (0.45 for DRAM), and b is bits per
cell (1 for 2T0C). The 10¹² factor converts nm² to mm².

Worked example — Sophon 2T0C DRAM: D = 10¹² / (8 × 28² × 1.45) × 1 = 110.0 Mb/mm². Source
for cell: [10] (analogous 2D-material 2T0C); validated by
[8][9].

Eq. 2 — Per-die capacity

$$C_{\text{GB}} = \frac{D_{\text{Mb/mm}^{2}} \cdot A_{\text{mem-tier}} \cdot N_{\text{mem-tiers}}}{8 \cdot 10^{3}}$$

where A_mem-tier is the full footprint of one memory tier (750 mm²) and N_mem-tiers =
32. The 64-tier stack interleaves dedicated logic and memory tiers (32 of each); only the 32 memory tiers
contribute to capacity.

Sophon: C = (110.0 × 750 × 32) / 8000 = 330.2 GB (rounded to 330 GB).

Eq. 3 — 2T0C retention time

$$\tau = \frac{C_{\text{node}} \cdot V_{dd}}{2 \cdot I_{\text{off}}}$$

The factor of 2 reflects the sense margin: data is reliably recovered while the stored voltage remains above
V_dd/2. Source: [8] (closed-form derivation);
[9] (empirical confirmation).

Worked example: C_node = 3.0 fF (sum of C_gs,RT ≈ 2.5 fF + C_j,WT
≈ 0.5 fF), V_dd = 0.6 V. The off-current is specified as a
width-normalized density J_off = 10⁻¹⁵ A/µm = 1 fA/µm for the
2D-TMD nFET [2][3]; with a
Read-Transistor channel width W_RT = 0.5 µm the absolute leakage is I_off = J_off
· W_RT = 0.5 fA (5 × 10⁻¹⁶ A) at 25 °C: τ = (3.0 × 10⁻¹⁵ × 0.6) / (2 × 5 × 10⁻¹⁶)
= 1.8 s at 25 °C.

This is ≈ 4,800× longer than a 1T1C DRAM cell and reflects the exceptional sub-threshold off-state of the
atomically-thin TMD channel (I_on/I_off > 10⁸, sub-threshold slope ≈ 75 mV/dec).
Retention derates with junction temperature at ≈ 2× per 10 °C (Arrhenius): τ ≈ 159 ms at 60 °C and ≈ 28 ms
at 85 °C.

Eq. 4 — Refresh power

$$P_{\text{refresh}} = C_{\text{bits}} \cdot f_{\text{refresh}} \cdot E_{\text{read/bit}}$$

with C_bits = capacity in bits, f_refresh = 1 / T_refresh.

Sophon: at 25 °C the retention τ = 1.8 s (Eq. 3) permits a relaxed refresh interval of
T_refresh = 1.0 s (1.8× margin). P = (330 × 8 × 10⁹ bits) × (1 / 1.0 Hz) × (30 ×
10⁻¹⁵ J/bit) = 0.079 W — effectively negligible. This is the decisive consequence of the 1
fA/µm off-current: refresh power drops by ≈ 3,300× relative to a conventional gain cell.
The on-die controller scales the interval with junction temperature (Eq. 3 derating); even in the worst hot
corner the refresh cost stays small — at 85 °C a 20 ms interval (50 Hz) gives P_refresh ≈
4.0 W, and at a 105 °C excursion a 5 ms interval (200 Hz) gives ≈ 15.8 W.
A nominal 1 W refresh allowance is carried in the power budget (§6) to cover warm
steady-state operation with margin.

Eq. 5 — Per-MAC energy decomposition

Total energy per MAC operation is the sum of memory access and compute.
Sophon uses pure digital CIM (binary sense amplifier + adder tree per column-group.

$$E_{\text{MAC}} = E_{\text{mem, read}} + E_{\text{compute, digital}} + (E_{\text{mem, write}} \text{ if backward pass})$$

Sophon BF16 forward MAC: E = (30 fJ/bit × 16 bits) + E_{adder-tree,BF16} = 0.480 pJ
+ 0.140 pJ = 0.620 pJ/MAC.

Sophon BF16 backward MAC: add gradient write E_write = 20 fJ/bit × 16 bits =
0.320 pJ → 0.940 pJ/MAC total per weight per training step.

Sophon FP8 inference MAC: E = (30 fJ/bit × 8 bits) + E_{adder-tree,FP8} = 0.240 pJ
+ 0.070 pJ = 0.310 pJ/MAC.

E_{adder-tree,FP8} is computed from per-bit binary adder energy in 28 nm CMOS at 0.6 V
[11] scaled to 2D-TMD: 8 fJ × 8 levels × 0.85 ≈ 0.054 pJ; with sign-bit
and mantissa pipeline overhead the effective figure is 0.070 pJ/MAC. The BF16 adder-tree figure (0.140 pJ)
is twice the FP8 figure because the bit-serial activation broadcast runs for 16 cycles instead of 8. The
fully digital adder tree is the primary energy improvement of the digital-CIM architecture.

Eq. 6 — Active chip power

$$P_{\text{active}} = R_{\text{op}} \cdot u \cdot E_{\text{perop}} + P_{\text{static}} + P_{\text{refresh}}$$

where R_op is the peak operation rate (FLOPS), u is the utilization fraction, E_{per op}
is per-FLOP energy (half of per-MAC energy, since 1 MAC = 2 FLOPs).

Sophon FP8 decode (55% util.): P = 4,200 × 10¹² × 0.55 × (0.310 / 2) × 10⁻¹² + 15 W (NoC +
static) = ≈ 373 W (matches §C.3 table: DRAM read 277 W + digital MAC 81 W + NoC 13 W +
static 2 W; the read is the full 0.240 pJ/MAC at the FP8 MAC rate, not halved).

Sophon BF16 forward (55% util.): P = 2,100 × 10¹² × 0.55 × (0.620 / 2) × 10⁻¹² + ~1 W
refresh + 20 W NoC + 2 W static = ≈ 379 W (refresh is negligible at ≈ 0.08 W thanks to the
1 fA/µm off-current).

Sophon backward (55% util.): + gradient write power 2,100 × 10¹² × 0.55 × (0.320 / 2) ×
10⁻¹² = + 185 W extra at FLOP rate, or 370 W at MAC rate. The §C.3 table uses 370 W →
≈ 749 W total.

Utilization 55% is from TPUv4i sustained workload data [12]; peak 100%
used for thermal worst-case.

Eq. 7 — Inference throughput (decode)

From Kaplan et al. [14]:

$$\text{tok/s} = \frac{R_{\text{FLOPS}} \cdot u}{2 \cdot N_{\text{params}}}$$

Sophon 80B FP8 decode: tokens/s = (4,200 × 10¹² × 0.55) / (2 × 80 × 10⁹) =
14,438 tokens/s. Sophon 80B BF16 decode: tokens/s = (2,100 × 10¹² × 0.55)
/ (2 × 80 × 10⁹) = 7,219 tokens/s.

Eq. 8 — Training throughput

From Patterson et al. [13]:

$$\text{traintok/s} = \frac{R_{\text{FLOPS}} \cdot u}{6 \cdot N_{\text{params}}}$$

The factor 6 (vs. 2 for inference) accounts for forward (2N) + backward (4N) compute.

Sophon 80B BF16: tokens/s = (2,100 × 10¹² × 0.55) / (6 × 80 × 10⁹) =
2,406 tokens/s/die.

Eq. 9 — Cluster training time

$$T_{\text{days}} = \frac{N_{\text{tokens}}}{N_{\text{dies}} \cdot R_{\text{tok/s/die}} \cdot 86400}$$

Examples:

• 256-die cluster, 1T tokens: 10¹² / (256 × 2406 × 86400) ≈ 18.8 days.
• 1,024-die cluster, 1T tokens: ≈ 4.7 days.
• 1,024-die cluster, 15T (Llama-3-class [15]): ≈
  70.6 days.

Eq. 10 — Energy per token

$$E_{\text{tok}} = \frac{P_{\text{active}}}{R_{\text{tok/s}}}$$

Sophon FP8 decode: E = 373 W / 14,438 tokens/s = 25.8 mJ/token.
Sophon BF16 decode: E = 379 W / 7,219 tokens/s = 52.5 mJ/token.
Sophon training (time-avg fwd + bwd): E = 564 W / 2,406 tokens/s =
0.234 J/token.

Eq. 11 — Yield (negative binomial with defect clustering)

$$Y = \left(1 + \frac{A \cdot D_{0}}{\alpha}\right)^{-\alpha}$$

Source: Cunningham [23]. A = 7.5 cm² die area, D₀ = 0.1 defect/cm²
(mature 28 nm), α = 3 (typical clustering).

Y = (1 + 0.75/3)⁻³ = 0.512 → 51.2% (negative-binomial).

Cross-check with Murphy/Stapper [24]: Y = ((1 − exp(−AD₀)) / AD₀)² =
0.495 → 49.5%. The more conservative Murphy/Stapper value is used as the base wafer yield.

Eq. 12 — M3D stack yield

$$Y_{\text{stack}} = Y_{\text{tier}}^{N_{\text{tiers}}}$$

With Y_tier = 0.997 (3 σ M3D process control achievable per imec
[7]): Y_stack = 0.997⁶⁴ = 0.825.

Combined yield (base × stack): 0.495 × 0.825 = 0.408 → 40.8% final die yield used in the
BOM calculation (§9).

Eq. 13 — BOM per die

$$\text{BOM} = \frac{C_{\text{wafer}}/N_{\text{dies/wafer}} + N_{\text{tiers}} \cdot C_{\text{tier}-\text{adder}}}{Y_{\text{final}}} + C_{\text{package}} + C_{\text{program}} + C_{\text{test}}$$

Sophon: BOM = ($51 + 64 × $52) / 0.408 + $60 + $0 + $25 = $8,273 + $85 =
$8,358.

Wafer cost from [25]; tier adder estimated from per-tier mask +
processing economics in [7][5].

Eq. 14 — 3-year TCO

$$\text{TCO}_{3y} = \text{BOM} + P_{\text{avg}} \cdot 26280 \cdot \text{PUE} \cdot c_{\text{kWh}}$$

with 26,280 hours = 3 years × 8,760 h/year, PUE = 1.5 [27], c_kWh
= $0.10/kWh [26]. P_avg
is the duty-weighted average power.

Sophon inference (30% busy FP8 decode, 70% idle): P_avg = 0.30 × 373 + 0.70 × 3 =
114.0 W → energy 2,996 kWh × $0.15 = $449 → TCO = $8,358 + $449 = $8,807.

NVIDIA Rubin (R200) same duty: P_avg = 0.30 × 1,800 + 0.70 × 250 = 715 W → energy
18,790 kWh × $0.15 = $2,819 → TCO = $82,800 + $2,819 = $85,619.

AMD Instinct MI455X same duty: P_avg = 0.30 × 1,700 + 0.70 × 250 = 685 W → energy
18,002 kWh × $0.15 = $2,700 → TCO = $96,700 + $2,700 = $99,400.

Sophon training (50% busy training, 50% idle): P_avg = 0.50 × 564 + 0.50 × 3 =
283.5 W → energy 7,450 kWh × $0.15 = $1,118 → TCO = $8,358 + $1,118 = $9,476.

Eq. 15 — Effective vertical thermal conductivity (BEOL stack)

$$k_{\text{eff}} = \phi_{\text{Cu}} \cdot k_{\text{Cu}} + (1 – \phi_{\text{Cu}}) \cdot k_{\text{BEOL}}$$

Parallel-conduction model with Cu fill fraction φ_Cu = 0.06 (Monolithic Inter-tier Via density ×
via cross-section / total area), k_Cu = 380 W/m·K, k_BEOL = 2.0 W/m·K
[20]:

k_eff = 0.06 × 380 + 0.94 × 2.0 = 24.7 W/m·K.

Eq. 16 — Steady-state junction temperature

$$T_j = T_{\text{ambient}} + P_{\text{die}} \cdot (R_{\text{pkg}} + R_{\text{stack}})$$

R_stack = (N_tiers × t_tier) / (k_eff × A_die) is the M3D
stack resistance; R_pkg is the package-to-coolant resistance from
[21][22].

Sophon FP8 decode (373 W, liquid R_pkg = 0.05 K/W): R_stack = (64 ×
0.35 × 10⁻⁶) / (24.7 × 7.5 × 10⁻⁴) = 0.00121 K/W (negligible) → T_j = 25 + 373 × 0.0512 =
44.1 °C.

Sophon training avg. (564 W): T_j = 25 + 564 × 0.0512 =
53.9 °C (well below T_jmax = 105 °C).

Eq. 17 — Effective decode throughput with workload-level accelerators

The raw dense FP8 baseline of Eq. 7 can be multiplied by three orthogonal workload-level accelerators on a
single Sophon die. Let s be the speculative-decoding multiplier, q be the quantization multiplier, and N_active
/ N_total be the MoE sparsity ratio. The effective decode throughput becomes:

$$\text{tok/s}_{\text{eff}} = \frac{R_{\text{FLOPS}} \cdot u \cdot s \cdot q}{2 \cdot N_{\text{active}}}$$

with assumed multiplier values supported by published technique benchmarks:

• s = 2.5 for speculative decoding with a 1 B-parameter draft model co-resident on the same
  die (k = 4 candidates, 70% mean acceptance per token; the draft consumes ~ 1.4% of the MAC budget)
  [29].
• q = 2.0 for INT4 weight quantization vs. FP8 (halves the bit-serial activation cycle
  count without changing the underlying MAC accuracy by more than 1–2 perplexity points on 80B-class
  instruction-tuned models) [30].
• N_active / N_total ∈ [0.05, 0.30] for production MoE configurations
  (Mixtral, DeepSeek-V3, frontier-MoE estimates).

Worked example — Sophon 80B dense, INT4 + speculative (FP8 mode): tokens/s = (4,200 × 10¹²
× 0.55 × 2.5 × 2.0) / (2 × 80 × 10⁹) = 72,188 tokens/s/die = ~ 5× raw FP8 baseline.

Worked example — Sophon DeepSeek-V3 MoE (671 B total / 37 B active), FP8 dense weights:
tokens/s = (4,200 × 10¹² × 0.55) / (2 × 37 × 10⁹) = 31,216 tokens/s/die = ~ 18× the
equivalent 671 B dense decode rate.

Note that the three multipliers do not all compose additively in every regime: speculative decoding’s
effective speedup depends on the small-model draft accuracy (which itself depends on the deployment domain),
and the q = 2 INT4 multiplier and the MoE sparsity multiplier compose only when the model architecture
supports both jointly. The benchmark table in §5.A.6 enumerates the realistic combinations.