VI · Translate

Translating papers

Primary equations on the left, a named-shape teaching presentation on the right. Left-hand displays normalise typesetting and select equations; they are not facsimiles. Batching, renaming, biases added by us and modern architectural variants are identified explicitly.

1 · Hochreiter & Schmidhuber, 1997

Checked against Long Short-Term Memory, §4, printed pp. 6–8 (the forward definitions are unnumbered); CEC: §3.2, pp. 5–6; truncated learning: Appendix A.1. The simplified presentation below takes one cell per block. The paper also describes S cells sharing input/output gates (§4, p. 8).

Let N denote the full set of source units u and C the selected memory cells. These are distinct coordinate sets even if their cardinalities happened to agree. The original g and h are squash/scale functions; h here is a function, not our modern hidden-state tensor.

1997 §4 · selected scalar equations

n e t_{i n_{j}} (t) = u \sum w_{i n_{j} u} y^{u} (t - 1)

y^{i n_{j}} (t) = f_{i n_{j}} (n e t_{i n_{j}} (t))

n e t_{c_{j}} (t) = u \sum w_{c_{j} u} y^{u} (t - 1)

s_{c_{j}} (t) = s_{c_{j}} (t - 1) + y^{i n_{j}} (t) g (n e t_{c_{j}} (t))

y^{c_{j}} (t) = y^{o u t_{j}} (t) h (s_{c_{j}} (t))

Vectorised, no batch · our presentation

C n^{in} (t) = C \times N W^{in} N \cdot N y (t - 1)

C y^{in} (t) = f_{in} (n^{in} (t))

C n^{c} (t) = C \times N W^{c} N \cdot y (t - 1)

C s_{c} (t) = s_{c} (t - 1) + y^{in} (t) ⊙ g (n^{c} (t))

C y^{c} (t) = C y^{o u t} (t) ⊙ h (s_{c} (t))

The output gate is formed analogously to the input gate. There is no learned forget gate on the old state. The isolated self-connection has weight one and identity activation: that is the original CEC, not a proof that all gradients everywhere in the network are constant. The original algorithm truncates other routes; our modern BPTT page does not. The §4 “Learning” paragraph calls the method a truncated variant of RTRL. The absence of a batch axis in this presentation is not a claim that batching is impossible.

2 · Graves, 2013, with diagonal peepholes

Checked against §2.1, Eqs. 7–11; diagonal W_c matrices are specified in the paragraph immediately after Eq. 11. Batching, row-oriented weights and q/v intermediates on the right are ours.

For each gate/candidate a∈{i,f,c,o}, declare Wᵃ:E×R, Uᵃ:D×R, bᵃ:(B)×R, vᵃ:B×R and qᵃ:B×D. Here |R|=|D|=768 but R≠D. Define the shared affine family and its explicit coordinate rename:

B \times R v_{t}^{a} = B \times E x_{t} E \cdot E \times R W^{a} + B \times D h_{t - 1} D \cdot D \times R U^{a} + (B) \times R b^{a}, B \times D q_{t}^{a} = ρ_{R \to D} (v_{t}^{a})

Each peephole pᵃ has stored shape D and effective shape (B)×D. It is diagonal and has only D parameters, not D².

Graves Eqs. 7–11 · normalised typesetting

i_{t} = σ (W_{x i} x_{t} + W_{hi} h_{t - 1} + W_{c i} c_{t - 1} + b_{i})

f_{t} = σ (W_{x f} x_{t} + W_{h f} h_{t - 1} + W_{c f} c_{t - 1} + b_{f})

c_{t} = f_{t} c_{t - 1} + i_{t} tanh (W_{x c} x_{t} + W_{h c} h_{t - 1} + b_{c})

o_{t} = σ (W_{x o} x_{t} + W_{h o} h_{t - 1} + W_{co} c_{t} + b_{o})

h_{t} = o_{t} tanh (c_{t})

Batched ledger form · affine family declared above

B \times D i_{t} = σ (q_{t}^{i} + (B) \times D p^{i} ⊙ c_{t - 1})

B \times D f_{t} = σ (q_{t}^{f} + (B) \times D p^{f} ⊙ c_{t - 1})

B \times D c_{t} = f_{t} ⊙ c_{t - 1} + i_{t} ⊙ tanh (q_{t}^{c})

B \times D o_{t} = σ (q_{t}^{o} + (B) \times D p^{o} ⊙ c_{t})

B \times D h_{t} = o_{t} ⊙ tanh (c_{t})

The output peephole uses the current c_t, so it waits for the cell update. The affine projections themselves may still be fused. At E=512,D=768, diagonal peepholes add 3D=2304 parameters: 4D(E+D+1)+3D=3,937,536. Our main forward/backward pages omit these peepholes and are labelled accordingly.

3 · Cho et al., 2014: reset before projection

Checked against §2.3, Eqs. 5–8 (shown in order 5,6,8,7). Biases are not displayed in the original equations. On the right we add a bias per block and choose φ=tanh as in the encoder specification, Appendix A.

Use the same D/R distinction and affine family for a∈{r,z}. All gates, candidates and hidden states below have B×D. Wⁿ:E×R, Uⁿ:D×R and bⁿ:(B)×R are declared for the candidate.

Cho Eqs. 5–8 · selected, reordered

r_{j} = σ ([W_{r} x]_{j} + [U_{r} h_{⟨ t - 1 ⟩}]_{j})

z_{j} = σ ([W_{z} x]_{j} + [U_{z} h_{⟨ t - 1 ⟩}]_{j})

h_{j}^{⟨ t ⟩} = ϕ ([Wx]_{j} + [U (r ⊙ h_{⟨ t - 1 ⟩})]_{j})

h_{j}^{⟨ t ⟩} = z_{j} h_{j}^{⟨ t - 1 ⟩} + (1 - z_{j}) h_{j}^{⟨ t ⟩}

Batched reset-before form · biases added

B \times D r_{t} = σ (q_{t}^{r}), B \times D z_{t} = σ (q_{t}^{z})

B \times R v_{t}^{n} = x_{t} E \cdot W^{n} + (r_{t} ⊙ h_{t - 1}) D \cdot U^{n} + b^{n}

B \times D n_{t} = tanh (ρ_{R \to D} (v_{t}^{n}))

B \times D h_{t} = z_{t} ⊙ h_{t - 1} + (1 - z_{t}) ⊙ n_{t}

PyTorch’s nn.GRU “Note” explicitly documents a different candidate: reset is applied after the recurrent projection. In our axes its candidate is:

B \times D n_{t} = tanh (ρ_{R \to D} (x_{t} E \cdot W^{n} + b^{in}) + r_{t} ⊙ ρ_{R \to D} (h_{t - 1} D \cdot U^{n} + b^{hn}))

Both biases are (B)×R before renaming. The reset-after candidate allows all three recurrent affine products to be computed together; reset-before generally cannot fuse the candidate with the other two. These are different functions, not interchangeable notation. There is a further historical nuance: Cho’s Appendix A.1.1 decoder already writes the reset gate after the recurrent projection (and adds context). Our comparison is specifically with §2.3 Eq. 8 and Appendix A.1’s encoder, not a claim that every formula in the paper uses reset-before. Counts at E=512,D=768: 3D(E+D+1)=2,951,424 with our added single bias; 3D(E+D+2)=2,953,728 for PyTorch’s two biases. The source equations with no displayed biases have 3D(E+D)=2,949,120 matrix parameters.

4 · Vaswani et al., 2017: attention

Checked against §3.2.1, Eq. 1; projections are in §3.2.2’s unnumbered multi-head equations, and causal masking is in §3.2.3. Our numerical example (model width 768, 12 heads) is not the paper’s base configuration (512, 8 heads, head width 64).

For one head, declare B=16, query positions T=128, key/value positions T′=128, model width M=768, key width F=64, and value width L=64. T≠T′ and F≠L despite their equal cardinalities. The three weights are W^Q:M×F, W^K:M×F, W^V:M×L; they have 3·768·64=147,456 entries without biases.

Eq. 1 · projections specialised from §3.2.2

Q = X W^{Q}, K = X W^{K}, V = X W^{V}

Attention (Q, K, V) = softmax (\frac{Q K ^{⊤}}{d _{k}}) V

Batched self-attention · our axis choices

B \times T \times F Q = B \times T \times M X M \cdot M \times F W^{Q}

B \times T^{'} \times M X^{'} = ρ_{T \to T^{'}} (X)

B \times T^{'} \times F K = X^{'} M \cdot M \times F W^{K}, B \times T^{'} \times L V = X^{'} M \cdot M \times L W^{V}

B \times T \times T^{'} S = \frac{Q F \cdot K}{64} + (B) \times T \times T^{'} mask

B \times T \times T^{'} A = softmax_{T^{'}} (S), B \times T \times L Y = A T^{'} \cdot V

The causal mask is zero at allowed keys and −∞ at disallowed keys; omit it for unrestricted encoder self-attention. Every query must have at least one allowed key. B is shared and retained during the F contraction. T′ is the normalised/reduced key axis; no transpose ambiguity remains. Chiang–Rush–Barak already use this example in §1 and give the named-axis attention function in §4.3, Eqs. 9–10.

For 12 heads add N=12 to Q,K,V and the corresponding weights. Each head retains its own F/L coordinates. The output projection uses distinct destination model coordinates M′:

B \times T \times M^{'} O^{'} = B \times T \times N \times L Y N, L \cdot N \times L \times M^{'} W^{O}, B \times T \times M O = ρ_{M^{'} \to M} (O^{'})

The three input projections total 1,769,472 parameters, and W^O adds 589,824, excluding all biases. Explicit renaming permits a later residual sum with X without identifying unrelated axes inside a dense map.