IV · Backward

Backpropagation through time

An additive memory path mitigates a problem; it does not abolish it. Here is the full reverse-mode pass for the modern no-peephole cell, followed by a careful distinction between its direct cell path and its full recurrent Jacobian.

These equations are our derivation for that specified cell, not equations copied from 1997. Graves 2013, §2.1, explicitly distinguishes the original approximate gradient from full BPTT and notes excessively large derivatives. His peepholes require additional derivative terms not present in this no-peephole derivation.

Adjoints have the shape of their variable

For a scalar loss L, δX denotes its total reverse-mode adjoint at the indicated computation-graph node. Its stored shape equals X’s. Use B=16, T=128, E=512, D=768, R=768, K=G·R=3072, with D and R distinct as in the ledger.

Let $B \times D ℓ_{t}$ be the adjoint arriving directly from losses/read-outs at h_t, excluding the next recurrent cell. Assume losses depend on the cell only through H, no direct loss on C, and fixed initial states. Direct cell-state losses or an explicit terminal loss on c add their own adjoints. Initialise future recurrent contributions to zero at T.

Collect hidden and cell errors, from T down to 1

B \times D δ h_{t} = ℓ_{t} + B \times K δ z_{t + 1} K \cdot D \times K U

B \times D δ o_{t} = δ h_{t} ⊙ tanh c_{t}

B \times D δ c_{t} = δ h_{t} ⊙ o_{t} ⊙ (1 - tanh^{2} c_{t}) + δ c_{t + 1} ⊙ f_{t + 1}

At t=T, the second term in the first and third lines is defined as zero; no actual gate at T+1 needs to be evaluated. δh_t already collects the hidden-state recurrent routes; δc_t adds the direct cell-state route separately.

Gate adjoints

B \times D δ f_{t} = δ c_{t} ⊙ c_{t - 1}, B \times D δ i_{t} = δ c_{t} ⊙ g_{t}, B \times D δ g_{t} = δ c_{t} ⊙ i_{t}

B \times D δ q_{t}^{i} = δ i_{t} ⊙ i_{t} ⊙ (1 - i_{t}), B \times D δ q_{t}^{f} = δ f_{t} ⊙ f_{t} ⊙ (1 - f_{t})

B \times D δ q_{t}^{g} = δ g_{t} ⊙ (1 - g_{t}^{2}), B \times D δ q_{t}^{o} = δ o_{t} ⊙ o_{t} ⊙ (1 - o_{t})

Reverse the coordinate rename, then fuse

B \times K δ z_{t} = pack_{D \to R, (G, R) \to K} (δ q_{t}^{i}, δ q_{t}^{f}, δ q_{t}^{g}, δ q_{t}^{o})

Each δq has axes B×D. The inverse bijection renames D→R; stacking in i,f,g,o order gives B×G×R and fusing gives B×K. This is the exact adjoint of the forward unpack.

Send the error to the input

B \times E δ x_{t} = δ z_{t} K \cdot E \times K W

δz_t contracted with U over K gives the B×D contribution to h at step t−1 in the first line. δc_t⊙f_t supplies its B×D direct cell contribution. No ambiguous repeated D axis is used.

Parameter gradients sum over batch and time

B \times T \times K δ Z = stack_{T} (δ z_{1}, \dots, δ z_{T}), B \times T \times D H_{-} = stack_{T} (h_{0}, \dots, h_{T - 1})

E \times K δ W = B \times T \times E X B, T \cdot δ Z

D \times K δ U = H_{-} B, T \cdot δ Z, K δ b = B, T \sum δ Z

512×3072, 768×3072 and 3072, respectively. The broadcast bias’s adjoint sums the virtual B axis and all times. If L is a mean loss, its scaling enters through ℓ; do not divide by B again.

The direct partial is not the full recurrent derivative

For one sequence, flatten only its hidden coordinates. Holding the gate values and candidate fixed gives the direct computational-graph edge:

\frac{\partial c _{t}}{\partial c _{t - 1}}_{i_{t}, f_{t}, g_{t} fixed} = diag (f_{t})

For the no-peephole cell, this is also the partial with h_t−1 and x_t held independent and fixed. For a batch it is block diagonal over B. Across k steps the direct-path product is:

J_{direct} = diag (s = t - k + 1 \prod t f_{s})

The product is coordinate-wise. If a forget coordinate is 0.9 for 100 steps, the multiplier is 0.9¹⁰⁰≈0.0000266. It does vanish with a long enough run of gates bounded below one. If the gates stay close to one, the direct route can carry an error much farther. Saturated sigmoid/tanh derivatives and a closed output gate can still suppress learning even then.

What the full Jacobian contains

The state is the pair (h,c). Its derivative includes gate dependencies through h, not just the diagonal cell edge. In this conventional column-coordinate Jacobian display, rows are output-state coordinates at time t and columns are a distinct input-state copy at time t−1; equal sizes do not identify those two roles. Let A=∂c_t/∂h_t−1 and O=∂o_t/∂h_t−1, with c_t−1 held fixed. For this no-peephole cell:

A = diag (c_{t - 1}) \frac{\partial f _{t}}{\partial h _{t - 1}} + diag (g_{t}) \frac{\partial i _{t}}{\partial h _{t - 1}} + diag (i_{t}) \frac{\partial g _{t}}{\partial h _{t - 1}}

\frac{\partial ( h _{t} , c _{t} )}{\partial ( h _{t - 1} , c _{t - 1} )} = [diag (tanh c_{t}) O + diag (o_{t} (1 - tanh^{2} c_{t})) A A diag (o_{t} (1 - tanh^{2} c_{t})) diag (f_{t}) diag (f_{t})]

Products of these full block Jacobians determine total state sensitivities. The full gradient can shrink or grow; the absence of a dense matrix on one selected edge is no proof otherwise. If c_t−1 is perturbed together with its earlier history, h_t−1 changes too. Peepholes add direct cell-to-gate derivatives, so their partial is not simply diag(f_t) unless those gates are explicitly held fixed.

The 1997 constant-error carousel is different

Hochreiter & Schmidhuber 1997, §3.2 (pp. 5–6), §4 (pp. 6–8), and Appendix A.2 (Eqs. 30–35) describe an identity activation with a fixed self-connection of weight one, not a learned modern forget gate. Its isolated self-loop derivative is exactly one. Their original learning method truncates certain recurrent gradient routes (Appendix A.1); it is not the full BPTT above. Learned access gates and the rest of the network still matter. Calling a modern forget-gated path “constant” without qualification is incorrect.

Checks and cost

The complete no-peephole reverse pass was checked against central finite differences for X,W,U,b,h₀,c₀; relative errors are below 10⁻⁶. The check now also verifies the gate-held partial, its attenuating product, and the full block Jacobian independently. These numerical checks support the stated model, not every LSTM variant.

Keeping i,f,g,o,c,h costs roughly T·B·6D=9,437,184 numbers (37.75 MB fp32), excluding X, workspaces, optimizer state, gradients and framework overhead. Checkpointing can trade recomputation for storage. Larger T makes activation memory significant; it is not universally larger than parameter/optimizer memory.