My Research Plan

Observations#

What motivates the plan

These observations, from the background post, motivate the hypotheses.

Observation 1Residual research is evolving into something that looks a lot like sequence modeling research.

It began with convolution-like structures: additive PreNorm, $x_{\ell+1} = x_\ell + F_\ell(N_\ell(x_\ell))$ , sums every earlier output with equal weight, much like a convolution over depth with kernel $[1, 1, 1, \ldots]$ . It then morphed into structures that resemble RNNs, such as HC, mHC and many others, which carry a learned, evolving state through depth. Now it is starting to experiment with attention: Kimi's Attention Residuals, multi-headed variants, and a growing number of new papers. Many of these approaches have stability trade-offs similar to their sequence-modeling equivalents: exploding and vanishing gradients, and issues with state magnitude.

Observation 2Latent propagation research is a subset of residual research, and it also resembles recurrence and sequence modeling research.

The goal is to integrate information from prior time steps effectively into the current time step's residual stream. We can view this as creating a continuous residual sequence in time across all blocks in the network (Figure 1).

Figure 1. Unrolling latent feedback. Three positions of a four-block model with Full-bandwidth-style feedback: each top state, gated by the next token, becomes the next position's input. Unrolled, every block lies on one residual sequence that continues from one position into the next. Hover any element for what it is.

Observation 3Current latent propagation research undervalues the ability to co-design with residual research.

Most current latent propagation approaches use PreNorm as their residual backbone, which seems suboptimal in my small-scale experiments. Most also require fixed source and target layers; more advanced residual setups can allow learned flexibility in addition to better performance.

Contents

1Hypothesis 1#

Better residual stream architecture can have a multiplicative effect with latent propagation.

Experiments.

Run an experiment with the residual passing from the Latent Recurrent Transformer [1] on a PreNorm backbone and get an improvement.
Run an experiment with the same residual passing on an Attention Residuals backbone [2] and get an improvement.
Demonstrate that the relative improvement of the Attention Residuals setup is higher.

Table 1. Planned runs. Each improvement is measured against the same backbone without latent passing.
Residual backbone	Without latent passing	With LRT residual passing
PreNorm	Baseline	Experiment 1
Attention Residuals	Baseline	Experiment 2

Evidence. The relative improvement from latent passing is larger on the Attention Residuals backbone than on PreNorm (experiment 3).

2Hypothesis 2#

Borrowing ideas from time recurrences is a good idea for improving latent propagation.

Experiments.

Get a solid baseline using modified versions of residual passing from existing research (background). My preferred form is gatherRMSNorm× embeddingFFN
Demonstrate improved learning from introducing Householder transforms on the prior banks, pulling the architecture from the current s3e42. That run carries two banks through the s3e37 Householder compactor, described in Section 4.1.

Evidence. The Householder variant learns better than the baseline.

3Hypothesis 3#

Latent propagation makes sense from a scaling-law perspective, given sufficient weight on inference or when factoring in test-time compute.

Experiments.

Run a set of experiments with the architecture from Hypothesis 2 across different model sizes and numbers of iterative input estimates (Jacobi passes; background).
Create a scaling-law model for it and for the baseline.
Intersect the two and figure out when latent propagation makes sense.

Table 2. Sweep axes for the scaling-law fits.
Axis	Values
Model size	Several sizes, for both models
Iterative input estimates	Several Jacobi pass counts, for the latent-propagation model
Model	Architecture from Hypothesis 2; baseline

Evidence. The fitted laws cross: beyond some weight on inference and test-time compute, latent propagation is the better use of compute.

4Current architecture#

The s3e42 starting point

Hypotheses 2 and 3 build on the architecture of my current run, s3e42. It keeps the s3e37 architecture: two memory slots carried between tokens through a Householder compactor, on an Attention Residuals backbone [2] with pairwise queries. s3e42 trains it with a mixed three- and four-pass Jacobi schedule and full gradients.

4.1s3e42: Householder slot compaction#

Current run · parent s3e37 · 144,915,932 parameters

Residual stream
Bank → slots → every block
On the recurrent chain
Token conditioning

At decode time the compactor $C$ between positions reads the embedding and every block output of position $t-1$ (and its incoming slots), conditions on the token at $t$ , and emits two slots $z_1, z_2$ . Both slots are sources for every residual reader at $t$ , so the whole stack lies on the recurrent chain.

Figure 2. s3e42 across positions. Four of the eight blocks drawn. The compactor

C

gathers the whole bank of one position and hands two slots to every block of the next.

Figure 3. Two Householder slots between tokens. One token boundary for one slot; the decoder's ordinary causal attention and K/V cache are unchanged.

SourceFrozen recipe s3e42-jacobi-bank2-mixed34-fullgrad-lr5e-3; the equations follow its compactor and reader code.

Read. At token $t$ , each of the two slots reads the complete bank $R_{t-1}$ left at $t-1$ : the embedding, all eight attention and eight FFN outputs, and the two slots that arrived there. Each slot has its own query and bias for every source; keys are RMS-normalized and values stay raw.

Transform. The raw gather $g_s$ is RMS-normalized, gated featurewise by a learned lookup of the current token $x_t$ (one table shared by both slots), and projected by $W_s$ to a unit direction $n_s$ . The raw gather is then reflected across the hyperplane normal to $n_s$ [3]. For a fixed direction this preserves its norm; the projection chooses how the state turns rather than replacing it.

Write. The reflected slots $z_1, z_2$ join the current token's source bank. Every Attention Residuals reader in the decoder can read them alongside the embedding and earlier sublayer outputs, and they are part of the bank the next token's compactor reads.

Compactor, per slot

\begin{aligned} \tip{l}{\ell_{si}} &= \bigl(\tip{q}{q_{si}^{\top}}\, \tip{N}{\mathrm{RMSNorm}}(\tip{r}{r_i}) + \tip{b}{b_{si}}\bigr) / 16, \qquad \tip{r}{r_i} \in \tip{R}{R_{t-1}} \\ \tip{g}{g_s} &= \textstyle\sum_i \tip{alpha}{\alpha_{si}}\, \tip{r}{r_i}, \qquad \tip{alpha}{\alpha_{s}} = \tip{sm}{\mathrm{softmax}_i}(\tip{l}{\ell_{si}}) \\ \tip{v}{v_s} &= \tip{W}{W_s}\bigl(\tip{m}{m(x_t)} \odot \tip{N}{\mathrm{RMSNorm}}(\tip{g}{g_s})\bigr), \qquad \tip{n}{n_s} = \tip{v}{v_s} / \lVert \tip{v}{v_s} \rVert_2 \\ \tip{z}{z_s} &= \tip{g}{g_s} - 2\, \tip{n}{n_s} (\tip{n}{n_s^{\top}} \tip{g}{g_s}), \qquad s = 1, 2 \end{aligned}

(1)

Decoder residual readers

\begin{aligned} \tip{h}{h_a} &= \textstyle\sum_i \tip{sm}{\mathrm{softmax}_i}\bigl((\tip{qa}{q_{ai}^{\top}}\, \tip{N}{\mathrm{RMSNorm}}(\tip{r}{r_i}) + \tip{ba}{b_{ai}}) / 16\bigr)\, \tip{r}{r_i} \\ i &\in \{\tip{e}{e_t},\ \tip{y}{y_{<a}},\ \tip{z}{z_1},\ \tip{z}{z_2}\} \end{aligned}

(2)

Jacobi passes and token recurrence

\begin{aligned} \tip{R1}{R^{1}} &= \tip{D}{D}(\tip{xs}{x_{1:T}}) \\ \tip{Rk}{R^{k}} &= \tip{D}{D}\bigl(\tip{xs}{x_{1:T}},\, \tip{C}{C}(\tip{Rprev}{R^{k-1}_{t-1}}, \tip{x}{x_t})\bigr), \qquad k = 2, 3, 4 \\ \tip{Rt}{R_t} &= \tip{D}{D}\bigl(\tip{x}{x_t},\, \tip{C}{C}(\tip{R}{R_{t-1}}, \tip{x}{x_t})\bigr) \end{aligned}

(3)

The first line is the bank-free first pass; the last is the token-by-token recurrence used at inference.

Training and evaluation. Each update has eight microbatches: six run three passes with losses weighted 0.3, 0.3 and 0.4, and two run four passes with loss on the fourth only. Gradients cross every pass and bank boundary without damping or detachment. Validation reports pass 4, the mean of passes 4 to 16, and the token recurrence.

Size. Eight layers at width 1,024, eight query heads sharing one K/V head, and 4,096-token sequences in 524,288-token batches. The two $d \times d$ slot projections are repaid by narrowing the first FFN from 3,072 to 2,304 hidden units; query, bias and token-lookup parameters are counted separately.

References#

[1]Zeyi Huang, Xuehai He, LiLiang Ren, et al. Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior. arXiv:2605.26797v1, 2026.
[2]Kimi Team. Attention Residuals. arXiv:2603.15031, 2026. Official pseudocode.
[3]Zakaria Mhammedi, Andrew Hellicar, Ashfaqur Rahman, and James Bailey. Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections. ICML 2017, PMLR 70.