V · Explorer

Shape explorer

Change the four sizes. The equations, the counts and the blocks below update as you go. This is the modern no-peephole teaching cell, not a verbatim historical model. The rules do not change, only the numbers beneath the symbols.

Bbatch Ttime steps Einput width Dhidden width

show axis names instead of numbers count two biases, as PyTorch does

The cell at these sizes

Named axes: D is the recurrent source, R the distinct projection destination, |R|=|D|. K=G·R has 4D entries; J=E+D. Unpack unfuses K→G×R, selects each gate, then explicitly renames R→D. Its backward adjoint packs in reverse, D→R and G×R→K. Number-only labels substitute sizes from this ledger; equal numbers never identify different axes.

16 \times 3072 z_{t} = 16 \times 512 x_{t} 512 \cdot 512 \times 3072 W + 16 \times 768 h_{t - 1} 768 \cdot 768 \times 3072 U + (16) \times 3072 b

16 \times 3072 z_{t} = 16 \times 1280 [x_{t} ∥ h_{t - 1}] 1280 \cdot 1280 \times 3072 [W; U] + (16) \times 3072 b

16 \times 3072 z_{t} unpack_{K \to G, R \to D} (16 \times 768 q_{t}^{i}, 16 \times 768 q_{t}^{f}, 16 \times 768 q_{t}^{g}, 16 \times 768 q_{t}^{o})

(i_{t}, f_{t}, g_{t}, o_{t}) = (σ (q_{t}^{i}), σ (q_{t}^{f}), tanh (q_{t}^{g}), σ (q_{t}^{o}))

16 \times 768 c_{t} = 16 \times 768 f_{t} ⊙ 16 \times 768 c_{t - 1} + 16 \times 768 i_{t} ⊙ 16 \times 768 g_{t}

16 \times 768 h_{t} = 16 \times 768 o_{t} ⊙ tanh (16 \times 768 c_{t})

16 \times 128 \times 768 H = stack_{T} (h_{1}, \dots, h_{128})

512 \times 3072 δ W = 16 \times 128 \times 512 X 16, 128 \cdot 16 \times 128 \times 3072 δ Z

Parameters

4 D (E + D + 1) = 3072 \times (512 + 768 + 1) = 3072 \times 1281 = 3, 935, 232

Parameters	3,935,232	15.7 MB
…of which W, U, b	1,572,864 + 2,359,296 + 3,072
Input X	1,048,576	4.2 MB
Output H	1,572,864	6.3 MB
Activations kept for BPTT (≈ T·B·6D)	9,437,184	37.7 MB
Matrix-product work per step, 2·B·4D·(E+D)	125.8 MFLOP
…per whole batch sequence (×T), forward only	16.11 GFLOP

Memory is for 32-bit floats; the BPTT estimate excludes inputs, workspaces, gradients and optimizer storage. FLOPs count a multiply and an add as two. Gate outputs are sigmoid for i,f,o and tanh for g, as on the forward page.

The blocks

This is the fused product, z = [x ‖ h] · [W ; U], drawn to scale as a Falk diagram. Each entry of z is the row of [x ‖ h] to its left, multiplied along the column of [W ; U] above it. The edges that are summed away are shaded the same: E against E, D against D. The batch edge is drawn thicker than true scale so that it stays visible (dashed outline).

The output of the whole sequence, H, is a block of 16 × 128 × 768 = 1,572,864 numbers: