LSTMwith its shapes on

VII · Sources

Primary sources & adaptations

The primary papers were opened and checked for this edition. “Ledger” is our pedagogical presentation of shape-aware named tensors, not a claim of unprecedented mathematical invention.

Equation-level source map

  1. Hochreiter, S. & Schmidhuber, J. (1997), Long Short-Term Memory. Author-hosted paper; Neural Computation 9(8), 1735–1780.

    §3.2 (printed pp. 5–6 of the author PDF): identity activation, self-weight 1.0 and the CEC. §4 (pp. 6–8): unnumbered input/output gates, net input, state and output definitions; shared-gate memory blocks; truncated RTRL-style learning. Appendix A.1 specifies approximate/truncated derivatives; Appendix A.2 (Eqs. 30–35) discusses constant error under those truncations. Our translation groups scalar equations into C-vectors and selects one cell per block. It retains the original g/h function names and does not add a forget gate. Printed page numbers are the author PDF’s, not journal pagination.

  2. Graves, A. (2013), Generating Sequences With Recurrent Neural Networks. PDF v5 · §2.1 HTML.

    §2.1, Eqs. 7–11 are the peephole LSTM reproduced in normalised typesetting on /translate. The paragraph after Eq. 11 specifies diagonal cell-to-gate weights; the next discusses approximate original learning versus full BPTT and excessive derivatives/clipping. Our /problem, /forward, /backward and /explorer instead use a modern no-peephole variant: remove W_ci,W_cf,W_co, name the candidate g, add B, choose row-oriented layouts and fuse four output blocks. The reverse-mode equations are our derivation of this explicitly specified variant, not a transcription of the original paper.

  3. Cho, K. et al. (2014), Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. PDF v3 · §2.3 HTML.

    §2.3, Eqs. 5–8: reset, update, state and candidate. Our left display reorders 8 before 7 for pedagogy; the right batches them, chooses tanh (Appendix A.1 encoder), and adds a bias per block. Appendix A says biases were omitted for clarity, so “no displayed bias” does not imply the trained model had none. Appendix A.1.1’s decoder has reset-after projection, unlike the reset-before §2.3 Eq. 8 and encoder. We do not attribute one placement universally to the paper.

  4. Vaswani, A. et al. (2017), Attention Is All You Need. PDF v7 · §3.2 HTML.

    §3.2.1, Eq. 1: scaled dot-product attention. §3.2.2: unnumbered head/projection equations and weight dimensions. §3.2.3: self-attention and −∞ causal masking. Our X→Q,K,V equations specialise those projections to self-attention. Batching and M=768,N=12,F=L=64 are our example, not the paper’s base M=512,N=8. We name query/key position axes T/T′ and key/value widths F/L separately. All parameter counts exclude biases unless stated.

  5. Chiang, D., Rush, A. M. & Barak, B., Named Tensor Notation. First posted 2021; checked against v3 (2023) PDF · HTML.

    §2 defines name/size axes; §2.1 allows unambiguous size omission. §§3.1–3.2 formalise lifting and broadcasting; §3.3 reductions; §3.4 contraction; §3.6 renaming/reshaping. §4.2 already distinguishes hidden from hidden′, renaming output to input for recurrence. §4.3, Eqs. 9–10 give attention. §5 requires distinct input/output axes for derivatives. Our D/R recurrence follows that idea. Our explicit virtual-axis parentheses, oxblood underset shapes, gold contraction labels, separate dot/⊙ and E1–E5 are display and teaching policies layered on those ideas, not replacements for that calculus.

Implementation references

PyTorch 2.15 nn.LSTM: opening equations and Variables (gate order i,f,g,o, two bias vectors, stored matrix layouts). PyTorch 2.15 nn.GRU: opening equations and the “Note” on reset-after projection versus the reset-before comparison. Our single-bias parameter counts and PyTorch’s two-bias counts are separated.

Corrections made before publication

  • Removed the false claim that an element-wise cell path guarantees non-vanishing gradients. It multiplies forget gates and may vanish. The direct gate-held partial, full state Jacobian and original 1997 CEC are now distinguished.
  • Removed ambiguous dense D×D named maps. D and R are distinct coordinate identities; an explicit R→D bijection is built into unpack and inverted in backward pack.
  • Replaced claims of missing sizes, renaming or elision in Named Tensor Notation with an accurate comparison and acknowledgements.
  • Labelled all modernisations and numerical examples; none is presented as a facsimile of a historical paper.

Source check: 14 November 2026 (UTC+8). The 1997 author site had an expired HTTPS certificate at check time; its public PDF was read with certificate validation disabled for that download only. Linked PDFs remain with their publishers/authors and are not bundled into the deployment.