Residual design and latent propagation

1My Research Direction#

How should hidden state travel?

A few observations guide my work.

Observation 1Residual research is evolving into something that looks a lot like sequence modeling research.

It began with convolution-like structures: additive PreNorm, $x_{\ell+1} = x_\ell + F_\ell(N_\ell(x_\ell))$ , sums every earlier output with equal weight, much like a convolution over depth with kernel $[1, 1, 1, \ldots]$ . It then morphed into structures that resemble RNNs, such as HC, mHC and many others, which carry a learned, evolving state through depth. Now it is starting to experiment with attention: Kimi's Attention Residuals, multi-headed variants, and a growing number of new papers. Many of these approaches have stability trade-offs similar to their sequence-modeling equivalents: exploding and vanishing gradients, and issues with state magnitude.

Observation 2Latent propagation research is a subset of residual research, and it also resembles recurrence and sequence modeling research.

The goal is to integrate information from prior time steps effectively into the current time step's residual stream. We can view this as creating a continuous residual sequence in time across all blocks in the network (Figure 2).

Figure 2. Unrolling latent feedback. Three positions of a four-block model with Full-bandwidth-style feedback: each top state, gated by the next token, becomes the next position's input. Unrolled, every block lies on one residual sequence that continues from one position into the next. Hover any element for what it is.

Observation 3Current latent propagation research undervalues the ability to co-design with residual research.

Most current latent propagation approaches use PreNorm as their residual backbone, which seems suboptimal in my small-scale experiments. Most also require fixed source and target layers; more advanced residual setups can allow learned flexibility in addition to better performance.

2Notation#

Read, transform, write

Each mechanism below is described in three steps.

Read: Which vectors are available, and how are they selected?
Transform: What normalization, projection, gating, and expensive computation acts on them?
Write: Where does the result go, and what state is retained?

The mechanism diagrams in Sections 3 and 4 color these steps the same way. Arrows describe data dependencies, not measured throughput.

Table 1. Symbols used in the equations and diagrams.
Symbol	Meaning
$t$	Sequence position; Coconut includes silent positions.
$\ell,\ L$	Depth and final depth: sublayers for PreNorm, HC, and AttnRes; whole blocks for MUDDFormer and the temporal models.
$d,\ n$	Vector width and number of residual streams.
$k,\ K$	Refinement round and its forward budget; separate from sequence position and depth.
$N,\ \sigma,\ \odot$	Specified normalization, sigmoid, and elementwise multiplication.
$e,\ h,\ r$	Embedding, accumulated block state, and raw branch contribution. These are different sources.

Table 2. Three paths that should not be conflated. Figure 1 draws each one on the lattice of positions and depths.
Path	Read	Write / receiving computation
Residual routing	Earlier depth states at the same position	Later depth at that position
Causal token attention	Available keys and values from token positions	Attention output at the current depth
Temporal latent feedback	A completed source state at an earlier position	Chosen receiving depths at a later position, often including earlier computation

Remark. Ordinary attention already transfers continuous information through K/V. Latent feedback adds a direct high-depth-to-earlier-depth path; it does not create the model's only continuous channel across tokens. A new path can carry useful state and still carry unstable dynamics.

3Residual design#

What survives across depth?

One vector can accumulate every branch output, several streams can retain a learned mixture of past computation, or a depth bank can keep sources separately addressable. “More streams” and “more stored layers” are different forms of memory.

Training cost includes the router's arithmetic, stored activations, backward computation, memory traffic, and pipeline communication. A small forward-FLOP increase can coexist with a larger elapsed-time penalty. Stability likewise has several meanings: state magnitude, preservation of distinct stream directions, and the gradients through the complete update.

These routers learn jointly with the language-model objective. They change the computation across depth without introducing a new serial dependency between token positions. Retaining a depth bank can, however, change activation-checkpointing and communication costs.

3.1PreNorm#

Baseline · Pre-LN analysis, 2020 · [1]

Identity carry
Read: normalized copy
Write: branch output

Each unit copies the carried vector $x_\ell$ , normalizes the copy, runs attention or an FFN on it, and adds the output $y_\ell$ back. The carried vector is not normalized on the identity path; a final normalization typically precedes the vocabulary head.

Figure 3. PreNorm along depth. One position and four residual units (two Transformer blocks). The carried vector climbs the left; each branch reads a normalized copy and writes its output back.

Figure 4. One residual state, two paths. Conceptual diagram of one residual unit; the identity bypass and normalized branch join by addition.

Read. One accumulated vector $x_\ell$ arrives from the preceding residual unit. Copy it into a branch and an identity carry path.

Transform. Normalize the branch input, apply attention or an FFN, and add its output to the untouched carry.

Write. The sum becomes $x_{\ell+1}$ . A Transformer block normally contains two such units; a final normalization typically precedes the vocabulary head.

\tip{xnext}{x_{\ell+1}} = \tip{x}{x_\ell} + \tip{F}{F_\ell}\bigl(\tip{N}{N_\ell}(\tip{x}{x_\ell})\bigr)

(1)

Training and cost. No learned depth router or history bank. Moving normalization changes the optimization geometry, rather than adding more model traversals. Xiong et al. show that Pre-LN can train without warmup in their experiments; this is not a prescription to remove warmup from every modern recipe.

Stability. The identity route helps gradient flow, but neither bounds the accumulated state nor makes every layer equally useful. NormFormer [2] documents mismatched gradient scales across depth.

3.2MUDDFormer#

Multiway Dynamic Dense connections · 2025 · [3]

Read from stored outputs
Append block output
Stored output

Every block reads the embedding and all earlier block outputs from a growing bank, then appends its own output. The final readout is also a mixture of the bank. Compare the single carried vector of PreNorm (Figure 3).

Figure 5. MUDDFormer along depth. One position and four blocks. Block outputs are stored on the left and read by every later block.

Figure 6. Four reads from a depth history. Conceptual block diagram: four depth reads produce Q, K, V and the attention bypass; the equations specify FFN details.

Read. Retain the embedding and previous block outputs in a depth bank. The newest output generates coefficients for four independent reads: Q, K, V, and the attention residual.

Transform. RMSNorm → small GELU MLP → learned static prior produces unconstrained coefficients. Weighted sums supply separately normalized Q/K/V inputs to token attention; the residual mixture bypasses attention.

Write. Add attention to the residual mixture, run the normal FFN and skip, and append the resulting block output to the bank. The final readout uses a residual mixture.

\begin{aligned} \tip{call}{[c^{Q}; c^{K}; c^{V}; c^{R}]} &= \tip{W2}{W_2}\, \tip{gelu}{\mathrm{GELU}}\bigl(\tip{W1}{W_1}\, \tip{rms}{\mathrm{RMSNorm}}(\tip{hl}{h_\ell})\bigr) + \tip{prior}{\mathrm{prior}} \\ \tip{u}{u^{s}} &= \tip{sum}{\textstyle\sum_{j=0}^{\ell}} \tip{cj}{c^{s}_{j}}\, \tip{hj}{h_j}, \qquad \tip{s}{s \in \{Q, K, V, R\}} \\ \tip{v}{v} &= \tip{att}{\mathrm{Attention}}\bigl(\tip{N}{N}(\tip{uq}{u^{Q}}), \tip{N}{N}(\tip{uk}{u^{K}}), \tip{N}{N}(\tip{uv}{u^{V}})\bigr) + \tip{ur}{u^{R}} \\ \tip{hnext}{h_{\ell+1}} &= \tip{v}{v} + \tip{ffn}{\mathrm{FFN}}\bigl(\tip{N}{N}(\tip{v}{v})\bigr) \end{aligned}

(2)

Training and cost. The cited 2.8B configuration adds 0.23% parameters and 0.4% forward FLOPs, while a separate 2.8B benchmark reaches 84% of its Transformer++ baseline's training tokens/s. Dense reads are sensitive to memory traffic. The recipe also reallocates FFN width toward later layers.

Stability. Initialize reads to the latest output. A deep/narrow variant additionally normalizes before and after aggregation and restores a skip. These coefficients are not softmax probabilities.

3.3Hyper-Connections#

HC · 2024 / ICLR 2025 · [4]

Carry between streams
Read gates $a$
Write gates $b$

Each unit reads a weighted sum of the four streams (violet), runs one ordinary branch, writes the output back to every stream with separate weights (terracotta), and mixes the carried streams with a 4 × 4 matrix $R$ (gray). In HC all three are unconstrained; in this illustrative carry the row sums are not one.

Figure 7. Hyper-Connections across four streams. One position and three units of depth, each with one ordinary branch. Weights are illustrative.

Figure 8. Read, carry and write across streams. Conceptual diagram of one hyper-connection unit: read one branch, mix the carry, and write back to

n

streams.

Read. Replicate the embedding into $n$ residual streams. Read a weighted sum of the current streams into one width- $d$ branch; there is no separately addressable complete depth history.

Transform. Learn a read vector $a$ , carry matrix $R$ , and write vector $b$ . Dynamic HC normalizes each stream across features, then uses linear projections, tanh, and static priors to adjust all three.

Write. Mix the carried streams and distribute the single branch output among them. Sum the streams at the network output before final normalization. The expensive branch runs once.

\begin{aligned} \tip{u}{u} &= \tip{a}{a_\ell} \tip{X}{X_\ell} \\ \tip{y}{y} &= \tip{F}{F_\ell}\bigl(\tip{N}{N_\ell}(\tip{u}{u})\bigr) \\ \tip{Xnext}{X_{\ell+1}} &= \tip{R}{R_\ell} \tip{X}{X_\ell} + \tip{b}{b_\ell^{\top}} \tip{y}{y} \end{aligned}

(3)

with $X \in \mathbb{R}^{n\times d}$ , $a, b \in \mathbb{R}^{1\times n}$ , and $R \in \mathbb{R}^{n\times n}$ . $R_{ij}$ routes source stream $j$ to destination stream $i$ ; the paper's convention is $R = A_r^{\top}$ .

Dynamic routes

\begin{aligned} \tip{Xbar}{\bar{X}} &= \tip{norm}{\text{feature-wise normalization of each stream}}(\tip{X}{X}) \\ \tip{a}{a} &= \tip{static}{a_{\text{static}}} + \tip{eps}{\varepsilon_\alpha} \tip{tanh}{\tanh}(\tip{Xbar}{\bar{X}} \tip{Wread}{W_{\text{read}}})^{\top} \\ \tip{b}{b} &= \tip{static}{b_{\text{static}}} + \tip{eps}{\varepsilon_\beta} \tip{tanh}{\tanh}(\tip{Xbar}{\bar{X}} \tip{Wwrite}{W_{\text{write}}})^{\top} \\ \tip{R}{R} &= \tip{static}{R_{\text{static}}} + \tip{eps}{\varepsilon_\alpha} \tip{tanh}{\tanh}(\tip{Xbar}{\bar{X}} \tip{Wcarry}{W_{\text{carry}}})^{\top} \end{aligned}

(4)

Training and cost. Start from identity carry, a rotating single-stream read, and zero dynamic projections. Four-stream OLMo-1B adds 0.200% forward FLOPs in the paper, but measured memory increases 26.1% in that configuration.

Stability. Carry matrices are unconstrained. The later mHC study [5] finds amplification under their repeated composition at larger scale. Small routing FLOPs do not imply negligible memory or communication cost.

3.4Manifold-Constrained Hyper-Connections#

mHC · 2025 / 2026 · [5]

Carry between streams
Read gates $a$
Write gates $b$

mHC keeps HC's layout but bounds the gates (reads in $(0, 1)$ , writes in $(0, 2)$ ) and pushes the carry toward a doubly stochastic matrix with 20 Sinkhorn–Knopp iterations: nonnegative entries whose rows and columns sum to one.

Figure 9. mHC across four streams. The layout of Figure 7 with bounded gates and a doubly stochastic carry. Weights are illustrative.

Figure 10. Read, carry and write across streams. Conceptual mHC unit: read and write gates are bounded; only the carry matrix targets doubly stochastic constraints.

Read. Keep HC's $n$ streams. Routing coefficients come from the RMS-normalized, flattened $n \times d$ state. Sigmoid read gates feed one branch.

Transform. Exponentiate carry logits and alternate column/row normalization for 20 Sinkhorn–Knopp iterations. The target is a nonnegative matrix whose rows and columns sum to one.

Write. Add twice-sigmoid-gated branch output to the mixed carry. Read gates lie in $(0, 1)$ and write gates in $(0, 2)$ ; neither set is normalized to sum to one.

\begin{aligned} \tip{q}{q} &= \tip{rms}{\mathrm{RMSNorm}}\bigl(\tip{vec}{\mathrm{vec}(X)}\bigr) \\ \tip{z}{z_s(q)} &= \tip{eps}{\varepsilon_s}\, \tip{q}{q} \tip{W}{W_s} + \tip{bias}{\mathrm{bias}_s}, \qquad \tip{sset}{s \in \{\text{read}, \text{write}, \text{carry}\}} \\ \tip{a}{a} &= \tip{sig}{\sigma}\bigl(\tip{zr}{z_{\text{read}}(q)}\bigr), \qquad \tip{b}{b} = \tip{two}{2}\tip{sig}{\sigma}\bigl(\tip{zw}{z_{\text{write}}(q)}\bigr) \\ \tip{Z}{Z_{\text{carry}}} &= \tip{reshape}{\mathrm{reshape}}(\tip{zc}{z_{\text{carry}}},\, \tip{nn}{n \times n}) \\ \tip{R}{R} &= \tip{sink}{\mathrm{Sinkhorn}_{20}}\bigl(\tip{exp}{\exp}(\tip{Z}{Z_{\text{carry}}})\bigr) \\ \tip{Xp}{X'} &= \tip{R}{R} \tip{X}{X} + \tip{b}{b^{\top}} \tip{F}{F}\bigl(\tip{N}{N}(\tip{a}{a}\tip{X}{X})\bigr) \end{aligned}

(5)

$\varepsilon_s$ : small, learned dynamic scales; biases: learned static routing priors.

Training and cost. Four streams widen residual storage. DeepSeek reports 6.7% additional training time with its optimized fused kernels, recomputation, and communication overlap. This is a setup-specific elapsed-time figure.

Stability. An exact doubly stochastic carry preserves stream mean and cannot amplify Euclidean norm on its own. Finite iterations approximate that constraint. Stream differences can contract; input-dependent routes and branch writes still affect the full Jacobian.

3.5mHC-lite#

Permutation mixtures · 2026 · [6]

Carry between streams
Read gates $a$
Write gates $b$

mHC-lite builds the carry as a softmax-weighted blend of fixed permutation matrices, each of which moves whole streams. Three of the 24 permutations are drawn with illustrative weights; every such blend is doubly stochastic by construction, without iterative normalization.

Figure 11. mHC-lite across four streams. The carry is a blend of whole-stream permutations. Weights are illustrative.

Figure 12. Read, carry and write across streams. Conceptual mHC-lite unit: fixed permutations replace iterative carry normalization; branch details follow the equations below.

Read. Use the same flattened, normalized stream state and sigmoid read/write pathways as mHC.

Transform. Softmax weights blend fixed permutation matrices. Each permutation rearranges whole streams. With four streams, all $4! = 24$ permutations give a doubly stochastic carry by construction, up to roundoff.

Write. Use the same carry-plus-branch-write update. The read and write gates remain unnormalized; the exact matrix constraint applies to the carry alone.

\begin{aligned} \tip{q}{q} &= \tip{rms}{\mathrm{RMSNorm}}\bigl(\tip{vec}{\mathrm{vec}(X)}\bigr) \\ \tip{z}{z_{\text{perm}}} &= \tip{eps}{\varepsilon_{\text{perm}}}\, \tip{q}{q} \tip{W}{W_{\text{perm}}} + \tip{bias}{\mathrm{bias}_{\text{perm}}} \\ \tip{p}{p} &= \tip{softmax}{\mathrm{softmax}}(\tip{z}{z_{\text{perm}}}) \\ \tip{R}{R} &= \tip{sum}{\textstyle\sum_{k=1}^{n!}} \tip{pk}{p_k} \tip{P}{P_k} \\ \tip{Xp}{X'} &= \tip{R}{R} \tip{X}{X} + \tip{b}{b^{\top}} \tip{F}{F}\bigl(\tip{N}{N}(\tip{a}{a}\tip{X}{X})\bigr) \end{aligned}

(6)

Read/write logits use the same scaled affine construction as mHC.

Training and cost. Native matrix operations replace iterative normalization. The paper evaluates roughly 45M–360M models. Its throughput comparison uses its own PyTorch mHC implementation, rather than DeepSeek's fused kernels. Enumerating permutations grows factorially, and “lite” does not mean uniformly fewer routing parameters.

Stability. The construction removes finite-iteration constraint error. It preserves the carry's mean and nonexpansiveness, while still allowing stream differences to vanish. The full trained recurrence is not guaranteed stable by this matrix property.

3.6Kimi Attention Residuals#

Full and Block AttnRes · 2026 · [7]

Read, width shows weight
Store raw output
Stored source

Unit 6 reads the embedding and every earlier raw sublayer output. Its learned query $q_6$ scores RMS-normalized copies, and the softmax weights (illustrative here) sum to one and blend the raw values. Every unit does the same with its own query; PreNorm would give each source weight one (Figure 3).

Figure 13. Attention Residuals along depth. One position; raw sublayer outputs are stored on the left. Line widths show illustrative softmax weights into unit 6.

Figure 14. Normalized keys, raw depth values. Conceptual Full AttnRes diagram; Block AttnRes substitutes grouped depth sums. Ordinary token attention is inside the branch.

Read. Full AttnRes retains the embedding and earlier raw sublayer outputs. Each destination has one learned query vector $q_\ell$ ; the default query is not projected from the current state.

Transform. RMS-normalize source keys, score them against the query, and softmax across depth sources. Values stay raw. Their weighted sum supplies the next sublayer input, followed by its input normalization.

Write. Store the new sublayer output as another source. Block AttnRes instead stores completed block sums and a live partial sum, sacrificing access to individual outputs inside completed blocks.

Full

\begin{aligned} \tip{r0}{r_0} &= \tip{e}{e_t}, \qquad \tip{ri}{r_i} = \tip{F}{F_i}\bigl(\tip{N}{N_i}(\tip{xi}{x_i})\bigr), \quad i \ge 1 \\ \tip{key}{k_i} &= \tip{rmsk}{\mathrm{RMSNorm}}(\tip{ri}{r_i}), \qquad \tip{alpha}{\alpha_{\ell i}} = \tip{softmax}{\mathrm{softmax}_{i<\ell}}\bigl(\tip{q}{q_\ell^{\top}} \tip{key}{k_i}\bigr) \\ \tip{x}{x_\ell} &= \tip{sumi}{\textstyle\sum_{i=0}^{\ell-1}} \tip{alpha}{\alpha_{\ell i}}\, \tip{ri}{r_i} \end{aligned}

(7)

Block

\begin{aligned} \tip{b0}{b_0} &= \tip{e}{e_t}, \qquad \tip{bj}{b_j} = \tip{sumb}{\textstyle\sum_{i \in \mathcal{B}_j}} \tip{ri}{r_i}, \qquad \tip{p}{p_\ell} = \tip{sump}{\textstyle\sum_{i \in \mathcal{B}_m,\; i<\ell}} \tip{ri}{r_i} \\ \tip{S}{S_\ell} &= \{\tip{b0}{b_0}, \tip{bj}{b_1}, \ldots, \tip{bj}{b_{m-1}}, \tip{p}{p_\ell}\} \\ \tip{x}{x_\ell} &= \tip{sums}{\textstyle\sum_{s \in S_\ell}} \tip{softmax}{\mathrm{softmax}_s}\bigl(\tip{q}{q_\ell^{\top}}\, \tip{keys}{\mathrm{RMSNorm}(s)}\bigr)\, \tip{src}{s} \end{aligned}

(8)

Training and cost. Full routing is $O(L^2 d)$ per token with $O(Ld)$ retained sources. For $N$ depth blocks, Block routing is $O(LNd)$ , with $O(Nd)$ source storage. Tokens retain ordinary parallel training; optimized inference batches the known queries.

Stability. Convex value mixing locally bounds the mixture by its largest source norm. This does not bound new branch outputs or the complete gradient dynamics. Current block sums can still grow.

4Latent propagation#

What survives across positions?

A temporal route needs a source depth, a carried state, a receiving depth, and a merger with current information. It also needs a training schedule. Teacher-forced token identities are known in advance; a model's own recurrent hidden states are not.

Exact latent recurrence follows positions in order. Parallel approximation instead initializes state estimates and refines them. The forward budget controls the states the model sees; the backward budget controls how far credit can travel. Both differ from adding extra silent reasoning positions or extra decoder loops at inference.

4.1Coconut#

Continuous latent reasoning · 2024 / 2025 · [8]

Residual stream
Final state → next input
On the recurrent chain
One forward call

Between <bot> and <eot>, each silent position takes the previous position's final normalized state $z$ as its whole input instead of a token embedding, then runs the full stack again. Token embeddings resume at <eot>.

Figure 15. Coconut across positions. Two silent thoughts between the boundary tokens; four blocks drawn. Blue curves carry the final normalized state into the next input.

Figure 16. A continuous state replaces a silent-position embedding. Conceptual latent-position diagram; ordinary causal attention and positional processing remain inside the model stack.

Read. At a silent latent position, take the preceding position's final normalized hidden state. Language-mode positions still use token embeddings.

Transform. Replace the next latent position's embedding with that state. There is no sampled token, projection, or fusion gate at the silent position; the model evaluates its full stack again.

Write. The final normalized state feeds the following position. Beginning/end markers delimit latent mode, after which ordinary text resumes.

\begin{aligned} \tip{z}{z_t} &= \tip{Nf}{N_{\text{final}}}(\tip{hL}{h^{L}_t}) \\ \tip{h0}{h^{0}_t} &= \tip{zprev}{z_{t-1}} && \tip{latentpos}{\text{(latent position)}} \\ \tip{h0}{h^{0}_t} &= \tip{e}{e_t} && \tip{langpos}{\text{(language position)}} \end{aligned}

(9)

Training and cost. A curriculum replaces initial explicit reasoning steps with latents. Cross-entropy on remaining text supervises them indirectly; no target vector reproduces the removed text. The released implementation resolves dependencies serially with K/V reuse and gradients: $n$ latents require $n + 1$ forward calls, not that many full-sequence passes.

Stability. Final normalization controls state scale. Aggressive curriculum transitions produced loss spikes. Each silent position still costs a model step, and reported benefits are task-specific.

4.2T²MLR#

Temporal Middle-Layer Recurrence · v2, 2026 · [9]

Residual stream
Cache route
Blocks a to b, recomputed

At decode time the cache is exact and serial. After block $b$ , the residual is added to the incoming cache and RMS-normalized; the result is merged into the next position just before block $a$ . Only blocks $a \ldots b$ repeat along the chain, and the cache accumulates contributions from every earlier position.

Figure 17. T²MLR across positions. Middle interval

a = 2

,

b = 4

of six blocks. The cache runs between positions and enters each one before block

a

.

Figure 18. Accumulated cache returns to an earlier middle block. Conceptual causal route from a previous middle source to a current earlier block; ordinary attention and later blocks are omitted.

Read. Take recurrent cache $R_{t-1}$ and the current stream $h$ before the chosen middle interval $[a, b]$ . The cache is a normalized accumulation, not a raw layer output.

Transform. Project and coordinatewise-gate the cache into $h$ , retaining an identity shortcut and a separately gated current-stream contribution. Both scalar gate strengths start at zero.

Write. Run blocks $a \ldots b$ , then add their output to the old cache and RMS-normalize to create $R_t$ . That cache enters the next token before block $a$ .

\begin{aligned} \tip{gcur}{g_{\text{cur}}} &= \tip{tanh}{\tanh}(\tip{gamma}{\gamma_{\text{cur}}})\, \tip{sig}{\sigma}\bigl(\tip{f}{f_{\text{cur}}}(\tip{cat}{[h, R_{t-1}]})\bigr) \\ \tip{grec}{g_{\text{rec}}} &= \tip{tanh}{\tanh}(\tip{gamma}{\gamma_{\text{rec}}})\, \tip{sig}{\sigma}\bigl(\tip{f}{f_{\text{rec}}}(\tip{cat}{[h, R_{t-1}]})\bigr) \\ \tip{Phi}{\Phi(h, R_{t-1})} &= \tip{h}{h} + \tip{gcur}{g_{\text{cur}}} \odot \tip{h}{h} + \tip{grec}{g_{\text{rec}}} \odot \tip{Wrec}{W_{\text{rec}}} \tip{R}{R_{t-1}} \\ \tip{Rt}{R_t} &= \tip{rms}{\mathrm{RMSNorm}}(\tip{hb}{h^{b}_t} + \tip{R}{R_{t-1}}) \end{aligned}

(10)

Training and cost. A no-feedback initializer precedes parallel Jacobi refinements of the middle interval. Defaults use forward depth 16 and backward depth 4; these are not 16 full-stack passes. Reported training takes roughly 2–4× wall-clock. A training-time-matched baseline beats the 135M variant on zero-shot NLP.

Stability. Normalized caches and bounded gate amplitudes control local scale, not the entire recurrent Jacobian. Finite refinement and truncated gradients create distinct approximation errors. Generated tokens need one stack traversal plus fusion.

4.3Full-bandwidth Transformer#

Top-state feedback · v1, 2026 · [10]

Residual stream
Top state → next input
On the recurrent chain

At decode time the next token's embedding gates a projection of the previous position's top-layer state, and the normalized product is the entire input to the stack. There is no additive embedding path. The recurrent chain therefore passes through every block.

Figure 19. Full-bandwidth Transformer across positions. Four blocks drawn. The top-layer state, gated by the next token, becomes that position's entire input.

Figure 20. Token-as-gate, latent-as-value. Conceptual decoding route with the paper's input normalization recipe; ordinary attention and K/V caching remain inside the stack.

Read. The previous token's top-layer state supplies the value; the current token embedding supplies a sigmoid gate.

Transform. Project the state, multiply it by the embedding-dependent gate, and normalize the fused input. There is no additive embedding shortcut. The detailed recipe also normalizes the gate's token input.

Write. Feed the product to the bottom of the current stack. Its top state becomes the next carried state while ordinary text and K/V caching continue.

\begin{aligned} \tip{u}{u_t} &= (\tip{WU}{W_U} \tip{z}{z_{t-1}}) \odot \tip{sig}{\sigma}\bigl(\tip{WG}{W_G}\, \tip{Ng}{N_{\text{gate}}}(\tip{e}{e_t})\bigr) \\ \tip{h0}{h^{0}_t} &= \tip{Nin}{N_{\text{in}}}(\tip{u}{u_t}) \\ \tip{zt}{z_t} &= \tip{top}{\text{current top-layer state}} \end{aligned}

(11)

Training and cost. Plain first passes initialize whole-stack feedback passes using right-shifted previous-pass states. Every pass is supervised, with gradients across feedback. The 200B-token recipe's 75%/22%/3% one/two/three-pass mix averages 1.28 stack passes, or 256B token-equivalent compute. Random plain prefixes train the prompt transition.

Stability. A single/double-pass schedule without triples fails at longer feedback horizons in the paper's diagnostic; a 3% three-pass component stabilizes it. The contraction interpretation is empirical. Depth scaling, normalization, tied embeddings, and latent jitter also matter. Decode adds projection/gating work; extra prompt passes cost separately.

4.4Latent Recurrent Transformer#

Residual and K/V injection · v1, 2026 · [11]

Residual stream
Memory route
On the recurrent chain
Receives memory only

At decode time the state after an intermediate source block becomes the memory $m_{t-1}$ for the next position, where it enters every block twice: scaled into the residual input and projected into the current keys and values. Blocks up to the source lie on the recurrent chain; later blocks receive memory but do not feed it forward.

Figure 21. Latent Recurrent Transformer across positions. Source block 4 of six. Memory from one position enters every block of the next.

Figure 22. Residual and key/value injection. Conceptual view of both memory routes for one current block; ordinary past K/V participates in attention, with details in the equations.

Read. Carry $m_{t-1}$ from a chosen source block. The default source is intermediate, not necessarily the top. Every current block can consume it alongside normal local state and K/V history.

Transform. One route adds scaled memory to the residual input before RMSNorm. Another projects memory into K/V and gates it together with local K/V. Keys then receive QK normalization and current-position RoPE.

Write. The mixed K/V occupies ordinary current-token cache entries, so no extra cache slots are needed. The chosen source block writes $m_t$ . Default recurrent projections are shared across depth; gates remain block-specific.

\begin{aligned} \tip{m}{m_t} &= \tip{hsrc}{h^{\ell_{\text{src}}}_t}, \qquad \tip{xbar}{\bar{x}^{\ell}_t} = \tip{alpha}{\alpha_\ell}\, \tip{x}{x^{\ell}_t} + \tip{gamma}{\gamma_\ell}\, \tip{mprev}{m_{t-1}} \\ [\tip{gl}{g_{\text{local}}}, \tip{gr}{g_{\text{rec}}}] &= 2\tip{sig}{\sigma}(\tip{Wg}{W^{\ell}_g}\, \tip{x}{x^{\ell}_t}) \\ \tip{kt}{\tilde{k}_t} &= \tip{gl}{g_{\text{local}}} \odot \tip{kl}{\tilde{k}_{\text{local}}} + \tip{gr}{g_{\text{rec}}} \odot \tip{Wk}{W_{k,\text{rec}}}\, \tip{mprev}{m_{t-1}} \\ \tip{v}{v_t} &= \tip{gl}{g_{\text{local}}} \odot \tip{vl}{v_{\text{local}}} + \tip{gr}{g_{\text{rec}}} \odot \tip{Wv}{W_{v,\text{rec}}}\, \tip{mprev}{m_{t-1}} \\ \tip{k}{k_t} &= \tip{rope}{\mathrm{RoPE}_t}\bigl(\tip{qk}{\mathrm{QKNorm}}(\tip{kt}{\tilde{k}_t})\bigr) \end{aligned}

(12)

Initialization

\tip{m0}{m_0} = 0, \quad \tip{alpha}{\alpha_\ell} = 1, \quad \tip{gamma}{\gamma_\ell} = 0.1, \qquad \tip{Wg}{W^{\ell}_g} = 0 \;\Rightarrow\; \tip{gl}{g_{\text{local}}} = \tip{gr}{g_{\text{rec}}} = 1

(13)

Training and cost. One full initialization pass fills a buffer; interleaved token subsets refine and write back each token once. This is about 2× ideal token compute, with more serial stages than the baseline. Approximate refinement differs from exact causal decoding.

Stability. Conservative memory initialization and normalization support scale control. The two K/V gates lie in $(0, 2)$ and are not complementary weights. Neither they nor the residual mixture prove long-horizon stability.

5Recurrence and sequence mixing#

What a single layer carries across positions

These mechanisms change how one layer mixes information across positions rather than how state moves through depth. DeltaNet and Gated DeltaNet replace the growing K/V cache with a fixed-size matrix state updated by an error-correcting rule; 2-simplicial attention keeps a cache but scores pairs of positions at once.

5.1DeltaNet#

Delta-rule linear attention · 2021 / 2024 · [13]

Token input
Carried state
Matrix state $S$

Each token writes into the carried state $S$ with its key, value and step size $\beta_t$ , then reads it with its query. Unlike attention there is no growing list of past keys and values: everything the layer remembers lives in one $d_v \times d_k$ matrix per head.

Figure 23. DeltaNet across positions. One layer at four positions. The matrix state

S

is the only thing carried forward.

Figure 24. An error-correcting fast-weight state. One layer at one token: the matrix state is the layer's whole memory, corrected and carried forward.

Read. Each token projects a query, key and value through a short convolution and SiLU, with L2-normalized queries and keys, and a write strength $\beta_t \in (0, 1)$ . The layer's whole memory of the past is one matrix $S_{t-1} \in \mathbb{R}^{d_v \times d_k}$ .

Transform. The delta rule, introduced for fast-weight programmers [12], reads what the state currently returns for $k_t$ , compares it with $v_t$ , and writes the error back along $k_t$ : a rank-one correction with step size $\beta_t$ , rather than linear attention's plain addition $v_t k_t^{\top}$ .

Write. The output is the corrected state applied to the query, $S_t q_t$ . $S_t$ is carried to the next token, so the cache is a fixed $d_v \times d_k$ matrix per head instead of a growing K/V list.

\begin{aligned} \tip{q}{q_t} &= \tip{l2}{\mathrm{L2Norm}}\bigl(\tip{phi}{\phi}(\tip{Wq}{W_q} \tip{x}{x_t})\bigr), \quad \tip{k}{k_t} = \tip{l2}{\mathrm{L2Norm}}\bigl(\tip{phi}{\phi}(\tip{Wk}{W_k} \tip{x}{x_t})\bigr), \quad \tip{v}{v_t} = \tip{phi}{\phi}(\tip{Wv}{W_v} \tip{x}{x_t}) \\ \tip{beta}{\beta_t} &= \tip{sig}{\sigma}(\tip{Wb}{W_\beta} \tip{x}{x_t}) \\ \tip{S}{S_t} &= \tip{Sprev}{S_{t-1}} \bigl(\tip{I}{I} - \tip{beta}{\beta_t} \tip{k}{k_t} \tip{kT}{k_t^{\top}}\bigr) + \tip{beta}{\beta_t} \tip{v}{v_t} \tip{kT}{k_t^{\top}} \\ \tip{o}{o_t} &= \tip{S}{S_t} \tip{q}{q_t} \end{aligned}

(14)

The same update as error correction

\tip{S}{S_t} = \tip{Sprev}{S_{t-1}} + \tip{beta}{\beta_t} \bigl(\tip{v}{v_t} - \tip{pred}{S_{t-1} k_t}\bigr) \tip{kT}{k_t^{\top}}

(15)

Training and cost. The recurrence is sequential, but the product of $I - \beta_t k_t k_t^{\top}$ factors is handled with a compact WY representation, so training runs chunkwise in parallel on matrix multiplies without materializing a state per token. Per-token decode cost and memory are constant in sequence length.

Stability. With unit-norm $k_t$ and $\beta_t \in (0, 1)$ , $I - \beta_t k_t k_t^{\top}$ has eigenvalue $1 - \beta_t$ along $k_t$ and 1 elsewhere, so the transition never expands the state. A fixed-size state still has to overwrite: recall degrades once more associations than dimensions compete for it.

5.2Gated DeltaNet#

Gated delta rule · 2024 / ICLR 2025 · [14]

Token input
Carried state
Matrix state $S$

Before writing, each step multiplies the carried state by a token-dependent decay $\alpha_t \in (0, 1)$ . Small $\alpha_t$ clears memory quickly; $\alpha_t$ near 1 keeps it, and the delta rule then edits individual associations.

Figure 25. Gated DeltaNet across positions. One layer at four positions. Each step decays the state, then applies the delta correction.

Figure 26. Decay, then correct. One layer at one token: DeltaNet's update with a per-head decay applied first, and a gated read-out.

Read. The same normalized queries, keys and values and write strength $\beta_t$ as DeltaNet, plus a per-head decay $\alpha_t \in (0, 1)$ computed from the token with Mamba2's parameterization.

Transform. The gated delta rule first decays the whole state by $\alpha_t$ and then applies the delta correction. Decay can clear memory quickly; the delta rule edits one association precisely. Setting $\alpha_t = 1$ recovers DeltaNet.

Write. Read $S_t q_t$ , normalize it per head, multiply by a SiLU output gate from the token, and project with $W_o$ . The decayed, corrected state carries to the next token.

\begin{aligned} \tip{alpha}{\alpha_t} &= \tip{exp}{\exp}\bigl(-\tip{A}{e^{A_h}}\, \tip{sp}{\mathrm{softplus}}(\tip{wa}{w_\alpha^{\top}} \tip{x}{x_t} + \tip{bh}{b_h})\bigr), \qquad \tip{beta}{\beta_t} = \tip{sig}{\sigma}(\tip{wb}{w_\beta^{\top}} \tip{x}{x_t}) \\ \tip{S}{S_t} &= \tip{Sprev}{S_{t-1}} \bigl(\tip{alpha}{\alpha_t} (\tip{I}{I} - \tip{beta}{\beta_t} \tip{k}{k_t} \tip{kT}{k_t^{\top}})\bigr) + \tip{beta}{\beta_t} \tip{v}{v_t} \tip{kT}{k_t^{\top}} \\ \tip{y}{y_t} &= \tip{Wo}{W_o} \bigl(\tip{rms}{\mathrm{RMSNorm}}(\tip{S}{S_t} \tip{q}{q_t}) \odot \tip{silu}{\mathrm{SiLU}}(\tip{Wg}{W_g} \tip{x}{x_t})\bigr) \end{aligned}

(16)

Training and cost. The decay is a scalar per head and step, so it folds into DeltaNet's chunkwise WY algorithm as cumulative products; training stays parallel over chunks and decoding keeps a constant-size state. The paper also builds hybrids that interleave these layers with sliding-window attention (and Mamba2).

Stability. Since $\alpha_t < 1$ and the delta factor is non-expansive, every transition contracts the state, which bounds it but also limits how long an association survives without being rewritten.

5.32-Simplicial Attention#

Trilinear attention · 2019 / 2025 · [16]

Read
Position

Ordinary attention scores the query at $i$ against one key at a time: each earlier position gets a single weight.

Figure 27. 2-Simplicial attention at one position. Positions run left to right. Line widths show illustrative weights.

Figure 28. Score pairs of positions. One query position: scores over pairs of earlier positions replace scores over single positions.

Read. Each position projects a query $q_i$ , two keys $k_j$ and $k'_k$ , and two values $v_j$ and $v'_k$ . In the efficient version, $j$ ranges over a window of the last $w_1$ positions and $k$ over the last $w_2$ .

Transform. Score every pair $(j, k)$ against the query with a trilinear form [15]: multiply $q_i$ , $k_j$ and $k'_k$ channel by channel and add up the $d$ products. Equivalently, dot the query with the elementwise product of the two keys, $q_i^{\top}(k_j \circ k'_k)$ , so a pair scores high only when both keys line up with the query on the same channels. Softmax over all pairs jointly. Ordinary attention is the special case $k'_k = v'_k = \mathbf{1}$ .

Write. Mix elementwise products of the two values, $v_j \circ v'_k$ , with the pair weights. The output feeds the residual stream like ordinary attention; the K/V cache simply holds two keys and two values per position.

\begin{aligned} \tip{q}{q_i} &= \tip{Wq}{W_q} \tip{x}{x_i}, \qquad \tip{k}{k_j} = \tip{Wk}{W_k} \tip{x}{x_j}, \qquad \tip{kp}{k'_k} = \tip{Wkp}{W_{k'}} \tip{x}{x_k} \\ \tip{A}{A_{ijk}} &= \tip{scale}{\tfrac{1}{\sqrt{d}}} \tip{tri}{\langle q_i, k_j, k'_k \rangle} = \tip{scale}{\tfrac{1}{\sqrt{d}}} \tip{sum}{\textstyle\sum_{l=1}^{d}} \tip{ql}{q_i[l]}\, \tip{kl}{k_j[l]}\, \tip{kpl}{k'_k[l]} = \tip{scale}{\tfrac{1}{\sqrt{d}}}\, \tip{qT}{q_i^{\top}} \bigl(\tip{k}{k_j} \tip{had}{\circ} \tip{kp}{k'_k}\bigr) \\ \tip{S}{S_{ijk}} &= \tip{sm}{\mathrm{softmax}_{(j,k)}}(\tip{A}{A_{ijk}}), \qquad \tip{win}{j \in \mathcal{W}_1(i),\ k \in \mathcal{W}_2(i)} \\ \tip{out}{\tilde{v}_i} &= \tip{sumjk}{\textstyle\sum_{j,k}} \tip{S}{S_{ijk}}\, (\tip{v}{v_j} \tip{had}{\circ} \tip{vp}{v'_k}), \qquad \tip{v}{v_j} = \tip{Wv}{W_v} \tip{x}{x_j}, \quad \tip{vp}{v'_k} = \tip{Wvp}{W_{v'}} \tip{x}{x_k} \end{aligned}

(17)

Training and cost. Full 2-simplicial attention costs $O(n^3)$ in sequence length. The Triton kernel restricts $j$ and $k$ to windows of $w_1 = 512$ and $w_2 = 32$ positions, $O(n\, w_1 w_2)$ , which the paper sizes to cost about as much as full attention at 48k context.

Stability. The softmax still produces a convex mixture, now over pairs. Standard RoPE is not invariant under a trilinear product; the paper discusses determinant-based alternatives that restore rotation invariance.

6Training the temporal route#

The same dependency, different schedules

Consider the toy recurrence $m_t = a\, m_{t-1} + b_t$ , with $m_{-1} = 0$ . Sequential execution uses the state it just computed. A Jacobi round updates every position simultaneously using the previous round's estimates (Figure 29).

Carry gain

a

0.70

Inputs $b_t$

1 · 0 · 0 · 1 · 0 · 0 · 0 · 0

Parallel estimate after 0 rounds

0 · 0 · 0 · 0 · 0 · 0 · 0 · 0

Sequential result

1 · .70 · .49 · 1.34 · .94 · .66 · .46 · .32

Start from zero estimates, then advance a round.

Enable JavaScript to change the recurrence and advance its estimates.

Figure 29. Parallel rounds versus sequential evaluation. A toy recurrence, not model measurements. Each Jacobi round updates all eight positions from the previous round's estimates; outlined cells already match the sequential result.

In this adjacent-position toy, each round resolves one more dependency level from zero initialization. Real models have learned initializers and causal attention, so this is not a universal bound on how far information travels in one pass.

Table 3. Count the work actually executed.
Mechanism	Training schedule	Generation / additional work
Coconut	Resolve silent positions serially, reusing K/V	One stack evaluation for each silent position
T²MLR	Refine a middle interval; truncate backward history	One stack per emitted token plus fusion
Full-bandwidth	Mix plain and whole-stack feedback passes	One stack per emitted token; optional extra prompt passes
LRT	Full initialization plus one aggregate token refinement	One stack per emitted token plus residual/K/V injection

These papers use different backbones, datasets, objectives, and cost accounting. Parameter- or token-matched quality does not automatically imply quality at matched training FLOPs, elapsed time, or deployment latency.

7The co-design opportunity#

Make memory a first-class source

Taken together, these mechanisms expose four choices: what to retain, who may read it, how it changes, and how to train that dependency. A residual reader could treat incoming temporal memory and current-token depth outputs as competing sources. A writer could learn which sources become the next carried state. Neither improvement follows automatically from adding gates or widening the state.

The research question is whether those choices improve quality at the relevant cost. That calls for ablations of source/target selection and mixing, causal-rollout validation, and comparisons under explicit pretraining and inference budgets. The architecture and its training approximation should be designed together.

References#

[1]Ruibin Xiong, Yunchang Yang, Di He, et al. On Layer Normalization in the Transformer Architecture. ICML 2020, PMLR 119.
[2]Sam Shleifer, Jason Weston, and Myle Ott. NormFormer: Improved Transformer Pretraining with Extra Normalization. arXiv:2110.09456, 2021.
[3]Da Xiao, Qingye Meng, Shengping Li, and Xingyuan Yuan. MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections. ICML 2025, PMLR 267. Official code.
[4]Defa Zhu, Hongzhi Huang, Zihao Huang, et al. Hyper-Connections. ICLR 2025; arXiv:2409.19606v3.
[5]Zhenda Xie, Yixuan Wei, Huanqi Cao, et al. mHC: Manifold-Constrained Hyper-Connections. arXiv:2512.24880v2, 2025.
[6]Yongyi Yang and Jianyang Gao. mHC-lite: You Don't Need 20 Sinkhorn-Knopp Iterations. arXiv:2601.05732v1, 2026. Official implementation.
[7]Kimi Team. Attention Residuals. arXiv:2603.15031, 2026. Official pseudocode.
[8]Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, et al. Training Large Language Models to Reason in a Continuous Latent Space. COLM 2025; arXiv:2412.06769v3.
[9]Ziyang Cai, Xingyu Zhu, Yihe Dong, Yinghui He, and Sanjeev Arora. T²MLR: Transformer with Temporal Middle-Layer Recurrence. arXiv:2607.15178v2, 2026. Official code.
[10]Xi Wang, Ziyang Cai, Zheng Zhan, et al. Full-bandwidth transformer. arXiv:2608.08888v1, 2026.
[11]Zeyi Huang, Xuehai He, LiLiang Ren, et al. Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior. arXiv:2605.26797v1, 2026.
[12]Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear Transformers Are Secretly Fast Weight Programmers. ICML 2021, PMLR 139.
[13]Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. NeurIPS 2024. Reference kernels (FLA).
[14]Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated Delta Networks: Improving Mamba2 with Delta Rule. ICLR 2025. Official code.
[15]James Clift, Dmitry Doryn, Daniel Murfet, and James Wallbridge. Logic and the 2-Simplicial Transformer. ICLR 2020.
[16]Aurko Roy, Timothy Chou, Sai Surya Duvvuri, et al. Fast and Simplex: 2-Simplicial Attention in Triton. arXiv:2507.02754, 2025.

Appendix#

A About the figures

The mechanism diagrams are redrawn computational schematics using the notation in Table 1. They summarize the mechanisms, while the cited papers and implementations define the complete recipes. Versions are identified where the account depends on them.

The interactive examples and full-Jacobian caveats are mathematical illustrations, not experimental results from these papers.