Contents

Overview#

Shared setup, and where every run landed

Data
FineWeb-Edu, 16k BPE
Batch
128 × 4,096 tokens
Length
76,294 updates · 40B tokens
Width
1,024 · 8 query heads, 1 K/V head
Residuals
Attention Residuals, pairwise queries + biases
Optimizer
Muon, peak LR 5 × 10⁻³
Validation
every 5k updates · 400 batches

Reading the charts. Each line is a run's validation loss minus its baseline's at the same update, in nats per token, so the dashed zero line is the baseline and lower is better. Every run sees the same tokens in the same order, so updates are directly comparable. For latent runs the plotted loss is the fourth parallel pass; their token-by-token recurrence stays within 0.001 nats of it at every checkpoint. The first chart weighs quality against an estimated decode cost, which is what a deployed model pays per token; its other view shows measured training time, which reflects the kernels each run used, so treat it as indicative.

  • Alternative feedback paths
  • Training schedules
  • Bank width
  • Depth and compute controls
  • 16-layer foundations
−0.250−0.200−0.150−0.100−0.0500+0.05000.10.20.30.40.50.6estimated forward GFLOP per generated tokenΔ validation loss at 30k vs s3e33 (nats)↓ better · → costliers3e33s3e49s3e41s3e50s3e48s3e42s3e32s3e38s3e31s3e37s3e36s3e34s3e4s3e1s3e9s3e3s3e2s3e35
Table of values
RunSetGFLOP / tokenΔ at 30k
s3e216-layer foundations0.545−0.196
s3e316-layer foundations0.545−0.172
s3e916-layer foundations0.545−0.131
s3e116-layer foundations0.545−0.128
s3e416-layer foundations0.545−0.094
s3e36Bank width0.290−0.084
s3e37Bank width0.289−0.084
s3e31Bank width0.289−0.083
s3e38Bank width0.290−0.080
s3e32Bank width0.290−0.078
s3e35Depth and compute controls0.417−0.078
s3e42Training schedules0.289−0.077
s3e48Alternative feedback paths0.289−0.077
s3e50Alternative feedback paths0.289−0.072
s3e41Training schedules0.289−0.063
s3e34Depth and compute controls0.545−0.062
s3e49Alternative feedback paths0.289−0.048
s3e33Depth and compute controls0.2890

Decode cost is estimated from each recipe: two FLOPs per weight in every matrix a generated token passes through (attention, FFNs, latent projections and the LM head) plus attention over a 2,048-token cache. Latent runs decode with one decoder pass per token, so their repaid projections leave them at the plain eight-layer model's cost; s3e34 runs its eight layers twice. Prompt prefill, reader scoring and norms are left out.

Figure 1. Quality against cost at 30k updates. Every run reached 30k, so this is the one step where all eighteen can be compared. Validation loss is relative to the plain eight-layer model (s3e33). Hover a point for its run.
  • Alternative feedback paths
  • Training schedules
  • Bank width
  • Depth and compute controls
  • 16-layer foundations
  • still running
Sep 16Sep 21Sep 28Oct 5s3e48s3e49s3e50s3e41s3e42s3e32s3e31s3e36s3e37s3e38s3e33s3e35s3e34s3e1s3e3s3e4s3e2s3e9
Figure 2. When each run trained. Spans are estimated from each run's last logged update and its measured speed. Runs are grouped by set, newest set first.

1Alternative feedback paths#

What shape should the carried state take?

1–4 October · baseline s3e42 (2 slots, mixed34) · 3 runs

With s3e42 as the reference two-slot model, these runs change what crosses the token boundary while keeping its mixed three/four-pass schedule. s3e48 keeps far more state: sixteen memories, one per attention or FFN output, each updated by projecting out a low-rank direction generated from a single gather. s3e50 keeps s3e48's memories but lets every reader take two separate softmaxes, one over the current pass and one over the memories, mixed by learned per-channel gains. s3e49 goes the other way, towards the Full-bandwidth Transformer: a single gathered vector, gated by the next token, replaces the embedding as source 0.

s3e42

Two slots, three/four-pass schedule, full gradients.

Δ vs s3e42
baseline
Validation
2.4583 at 75k
Cost
3.91 s / update
Recurrent − pass 4
−0.0002

finished

s3e48

Per-output memories with projected-out directions.

Δ vs s3e42
+0.0010 at 40k
Validation
2.6526 at 40k
Cost
4.44 s / update
Recurrent − pass 4
−0.0003

paused at 40.5k

s3e50

s3e48 memories read through a separate softmax.

Δ vs s3e42
+0.0050 at 30k
Validation
2.7006 at 30k
Cost
4.67 s / update
Recurrent − pass 4
−0.0003

paused at 31.9k

s3e49

One gated feedback vector replaces the embedding.

Δ vs s3e42
+0.0327 at 60k
Validation
2.5699 at 60k
Cost
3.77 s / update
Recurrent − pass 4
+0.0000

running · 63.6k

Two positions of an eight-layer decoder. Each view shows how one experiment moves information from the completed computation at t−1 into the computation at t: two Householder slots, sixteen projected memories, one gated vector that replaces the embedding, or sixteen memories read through a separate softmax.et − 1etCz1z2gather the whole bank → reflect into 2 slots
  • Residual stream
  • Feedback route
  • Token conditioning

The reference: a compactor gathers the whole bank of position t−1t-1 (embedding and all sixteen sublayer outputs, plus incoming slots), and two Householder-reflected slots become sources for every reader at tt.

Figure 3. Feedback paths. Two positions of the eight-layer decoder; each view is one experiment's route from t−1t-1 to tt.

s3e48 · low-rank memory update

g=∑iαi rt−1,i,ℓ=Wd(RMSNorm(g)⊙M[xt])xk=Uk ℓ(l),mt,k=vt−1,k−xkxk⊤vt−1,kxk⊤xk,k=1,…,16\begin{aligned} \tip{g}{g} &= \textstyle\sum_i \tip{alpha}{\alpha_i}\, \tip{r}{r_{t-1,i}}, \qquad \tip{l}{\ell} = \tip{Wd}{W_d}\bigl(\tip{N}{\mathrm{RMSNorm}}(\tip{g}{g}) \odot \tip{M}{M[x_t]}\bigr) \\ \tip{xk}{x_k} &= \tip{U}{U_k}\, \tip{l}{\ell^{(l)}}, \qquad \tip{m}{m_{t,k}} = \tip{v}{v_{t-1,k}} - \tip{xk}{x_k} \frac{\tip{xk}{x_k^{\top}} \tip{v}{v_{t-1,k}}}{\tip{xk}{x_k^{\top}} \tip{xk}{x_k}}, \qquad k = 1, \ldots, 16 \end{aligned}
(1)

s3e49 · gated feedback vector

zt=(Wu RMSNorm(g))⊙σ(Wg RMSNorm(E[xt]))replaces E[xt]\tip{zt}{z_t} = \bigl(\tip{Wu}{W_u}\, \tip{N}{\mathrm{RMSNorm}}(\tip{g}{g})\bigr) \odot \tip{sig}{\sigma}\bigl(\tip{Wg}{W_g}\, \tip{N}{\mathrm{RMSNorm}}(\tip{E}{E[x_t]})\bigr) \quad \text{replaces } \tip{E}{E[x_t]}
(2)

s3e50 · split reads

hj=aj⊙∑i∈selfsoftmax(⋅)i ri+bj⊙∑k∈priorsoftmax(⋅)k mk\tip{h}{h_j} = \tip{a}{a_j} \odot \textstyle\sum_{i \in \text{self}} \tip{sm}{\mathrm{softmax}}(\cdot)_i\, \tip{r}{r_i} + \tip{b}{b_j} \odot \textstyle\sum_{k \in \text{prior}} \tip{sm}{\mathrm{softmax}}(\cdot)_k\, \tip{m}{m_k}
(3)
  • s3e48 16 low-rank memories
  • s3e50 split reads
  • s3e49 gated vector
  • s3e42 baseline
−0.0100+0.010+0.020+0.030+0.040010k20k30k40k50k60k70ktraining updateΔ validation loss vs baseline (nats)↓ better than baselines3e48 +0.001s3e50 +0.005s3e49 +0.033s3e42 (baseline)
Table of values
Updates3e48s3e50s3e49
5k−0.001+0.009+0.028
10k−0.001+0.008+0.028
15k−0.001+0.006+0.030
20k−0.003+0.005+0.028
25k+0.000+0.004+0.028
30k+0.000+0.005+0.029
35k+0.001–+0.030
40k+0.001–+0.030
45k––+0.030
50k––+0.031
55k––+0.032
60k––+0.033
Figure 4. Alternative feedback paths. Validation loss minus s3e42's (2 slots, mixed34) at each checkpoint; lower is better.

2Training schedules#

How much parallel-pass compute does the two-slot model need?

27 September – 4 October · baseline s3e37 (2 slots, mixed64) · 2 runs

The two-slot model (s3e37) learns the token recurrence through Jacobi passes: every position runs in parallel and reads the previous pass's bank one position to the left. Its mixed64 schedule is expensive: six microbatches run four supervised passes, and two run eight, supervising the last four. s3e42 trims this to three supervised passes, plus two microbatches of four passes with loss only on the fourth, and keeps full gradients through every pass. s3e41 tries the subset refinement from the Latent Recurrent Transformer: a bank-free pass over every token, then even positions and then odd positions are refined once each, reading the freshest bank and K/V cache.

s3e37

Two Householder slots, mixed64 schedule.

Δ vs s3e37
baseline
Validation
2.5620 at 55k
Cost
5.29 s / update
Recurrent − pass 4
−0.0007

paused at 55.1k

s3e42

Two slots, three/four-pass schedule, full gradients.

Δ vs s3e37
+0.0076 at 55k
Validation
2.4583 at 75k
Cost
3.91 s / update
Recurrent − pass 4
−0.0002

finished

s3e41

LRT-style even/odd refinement.

Δ vs s3e37
+0.0204 at 30k
Validation
2.7093 at 30k
Cost
3.43 s / update
Recurrent − pass 4
−0.0000

paused at 31.6k

Each row is one of the eight microbatches in an update; each cell is one parallel pass. Filled cells are supervised, with their loss weight; hatched cells run without gradients; the interleaved schedule refines even positions, then odd positions.pass 1pass 2pass 3pass 4pass 5pass 6pass 7pass 8s3e37 · mixed645.29 s / updatemb 10.20.20.30.3mb 20.20.20.30.3mb 30.20.20.30.3mb 40.20.20.30.3mb 50.20.20.30.3mb 60.20.20.30.3mb 7no gradno gradno gradno grad0.250.250.250.25mb 8no gradno gradno gradno grad0.250.250.250.25
  • Supervised pass (weight shown)
  • Run without gradients

Six microbatches supervise passes 1–4 with weights 0.2, 0.2, 0.3, 0.3. Two run passes 1–4 without gradients, detach the bank, then supervise passes 5–8 at 0.25 each.

Figure 5. Pass schedules. Rows are the eight microbatches of one update, cells the parallel passes.

Mixed Jacobi objective (s3e37, s3e42)

L=18∑m=18  ∑kwm,k  CE(Rmk),Rk=D(x1:T, C(Rt−1k−1,xt))\tip{L}{\mathcal{L}} = \frac{1}{8} \sum_{\tip{mb}{m=1}}^{8} \; \sum_{\tip{k}{k}} \tip{w}{w_{m,k}}\; \tip{CE}{\mathrm{CE}}\bigl(\tip{Rk}{R^{k}_m}\bigr), \qquad \tip{Rk}{R^{k}} = \tip{D}{D}\bigl(x_{1:T},\, \tip{C}{C}(\tip{Rprev}{R^{k-1}_{t-1}}, x_t)\bigr)
(4)

Interleaved objective (s3e41), seven of eight microbatches

Lm=0.3  CE(init)+0.7  CE(even∪odd)\tip{L}{\mathcal{L}_m} = 0.3\; \tip{CE}{\mathrm{CE}}(\tip{init}{\text{init}}) + 0.7\; \tip{CE}{\mathrm{CE}}(\tip{refined}{\text{even} \cup \text{odd}})
(5)
  • s3e42 2 slots, mixed34
  • s3e41 interleaved
  • s3e37 baseline
−0.0050+0.005+0.010+0.015+0.020+0.025010k20k30k40k50k60k70ktraining updateΔ validation loss vs baseline (nats)↓ better than baselines3e42 +0.008s3e41 +0.020s3e37 (baseline)
Table of values
Updates3e42s3e41
5k+0.000+0.011
10k+0.003+0.017
15k+0.005+0.018
20k+0.007+0.018
25k+0.007+0.019
30k+0.007+0.020
35k+0.006–
40k+0.007–
45k+0.008–
50k+0.007–
55k+0.008–
Figure 6. Training schedules. Validation loss minus s3e37's (2 slots, mixed64) at each checkpoint; lower is better.

3Bank width#

How many slots should cross each token boundary?

26 September – 1 October · baseline s3e37 (2 slots, mixed64) · 4 runs

These five runs share one recipe and change only the number of slots SS the compactor writes between tokens: one, two, four, eight or sixteen. Each slot is a pairwise-softmax gather over the previous token's whole bank, reflected by a token-conditioned Householder transform (the s3e42 section of the plan draws one slot in detail). Every slot adds a d×dd \times d projection, repaid by narrowing the lowest FFNs so parameter counts stay matched. All five beat the plain eight-layer model by about 0.08 nats at 30k (see the overview), so the chart measures them against the two-slot run, s3e37, the width later kept for s3e42.

s3e37

Two Householder slots, mixed64 schedule.

Δ vs s3e37
baseline
Validation
2.5620 at 55k
Cost
5.29 s / update
Recurrent − pass 4
−0.0007

paused at 55.1k

s3e38

One Householder slot, mixed64.

Δ vs s3e37
+0.0037 at 40k
Validation
2.6478 at 40k
Cost
5.18 s / update
Recurrent − pass 4
−0.0006

paused at 43.5k

s3e36

Four Householder slots, mixed64.

Δ vs s3e37
+0.0006 at 55k
Validation
2.5298 at 60k
Cost
5.39 s / update
Recurrent − pass 4
−0.0007

paused at 60.3k

s3e31

Eight Householder slots, mixed64.

Δ vs s3e37
+0.0037 at 55k
Validation
2.4561 at 75k
Cost
5.87 s / update
Recurrent − pass 4
−0.0009

paused at 76.1k

s3e32

Sixteen Householder slots, mixed64.

Δ vs s3e37
+0.0051 at 35k
Validation
2.6732 at 35k
Cost
6.73 s / update
Recurrent − pass 4
−0.0010

paused at 38.0k

Two positions of the eight-layer decoder. The compactor turns the bank at t−1 into S slots that every reader at t can address. Below, each layer's FFN width shows where the slot projections' parameters were repaid.et − 1etFFN width by layerL1L8CS = 1s3e38 · 1 slot−256 hidden unitsrepay 1 × d² projections
  • Residual stream
  • Slots
  • Reads the slots

s3e38: a single slot. Its one projection is repaid by trimming the first FFN to 2,816 hidden units.

Figure 7. Slots between tokens. The compactor CC writes SS slots that every reader at tt can address; the bars show each layer's FFN width after repayment.

Compactor, slot s of S

gs=∑iαsi ri,αsi∝exp⁡((qsi⊤RMSNorm(ri)+bsi)/16)ns∝Ws(m(xt)⊙RMSNorm(gs)),zs=gs−2 ns(ns⊤gs)\begin{aligned} \tip{g}{g_s} &= \textstyle\sum_i \tip{alpha}{\alpha_{si}}\, \tip{r}{r_i}, \qquad \tip{alpha}{\alpha_{si}} \propto \exp\bigl((\tip{q}{q_{si}^{\top}} \tip{N}{\mathrm{RMSNorm}}(\tip{r}{r_i}) + \tip{b}{b_{si}}) / 16\bigr) \\ \tip{n}{n_s} &\propto \tip{W}{W_s}\bigl(\tip{M}{m(x_t)} \odot \tip{N}{\mathrm{RMSNorm}}(\tip{g}{g_s})\bigr), \qquad \tip{z}{z_s} = \tip{g}{g_s} - 2\, \tip{n}{n_s} (\tip{n}{n_s^{\top}} \tip{g}{g_s}) \end{aligned}
(6)

Parameter repayment

S d2  ≈  3 d∑ℓ(3072−fℓ)\tip{S}{S}\, \tip{d}{d}^2 \;\approx\; 3\, \tip{d}{d} \textstyle\sum_{\ell} \bigl(3072 - \tip{f}{f_\ell}\bigr)
(7)
  • s3e38 1 slot
  • s3e36 4 slots
  • s3e31 8 slots
  • s3e32 16 slots
  • s3e37 baseline
−0.007−0.005−0.0030+0.003+0.005+0.007010k20k30k40k50k60k70ktraining updateΔ validation loss vs baseline (nats)↓ better than baselines3e38 +0.004s3e36 +0.001s3e31 +0.004s3e32 +0.005s3e37 (baseline)
Table of values
Updates3e38s3e36s3e31s3e32
5k−0.001−0.005−0.004+0.000
10k+0.002−0.003−0.002+0.002
15k+0.003−0.002−0.002+0.003
20k+0.003−0.000−0.001+0.004
25k+0.004−0.001−0.001+0.005
30k+0.004−0.000+0.001+0.005
35k+0.004−0.001+0.001+0.005
40k+0.004−0.001+0.002–
45k–+0.001+0.003–
50k–−0.000+0.003–
55k–+0.001+0.004–
Figure 8. Bank width. Validation loss minus s3e37's (2 slots, mixed64) at each checkpoint; lower is better.

4Depth and compute controls#

What do extra layers, or reused layers, buy without any latent state?

26–28 September · baseline s3e33 (8 layers) · 3 runs

The latent runs are eight-layer models that run their stack several times, so they need plain references at matched depth and matched compute. s3e33 is the eight-layer model itself. s3e34 runs the same eight layers twice, each of its 33 readers with its own queries and biases and the second loop reading every earlier output. s3e35 and s3e1 add depth instead: twelve and sixteen layers. All four use the same Attention Residuals reader.

s3e33

Plain eight-layer model, no latent state.

Δ vs s3e33
baseline
Validation
2.5308 at 75k
Cost
1.29 s / update

finished

s3e34

The same eight layers run twice.

Δ vs s3e33
−0.0675 at 75k
Validation
2.4633 at 75k
Cost
2.46 s / update

finished

s3e35

Plain twelve-layer model.

Δ vs s3e33
−0.0954 at 75k
Validation
2.4354 at 75k
Cost
1.83 s / update

finished

s3e1

The 16-layer reference.

Δ vs s3e33
−0.1549 at 75k
Validation
2.3759 at 75k
Cost
2.52 s / update

finished

Columns of layers for the plain baselines: eight layers, the same eight layers run twice, twelve layers, and sixteen layers. Every reader attends over all earlier sublayer outputs.es3e338 layers1.30 s / updatees3e348 × 2 loop2.47 s / updateagaines3e3512 layers1.83 s / updatees3e116 layers2.52 s / update
  • Residual stream
  • Shared weights, second loop
Figure 9. Depth controls. Eight layers, the same eight run twice (shaded), twelve, and sixteen layers, with measured cost per update.

Attention Residuals reader (every run)

ha=∑isoftmaxi((qai⊤ RMSNorm(ri)+bai)/16) ri,i∈{et,y<a}\tip{h}{h_a} = \textstyle\sum_{i} \tip{sm}{\mathrm{softmax}_i}\bigl((\tip{q}{q_{ai}^{\top}}\, \tip{N}{\mathrm{RMSNorm}}(\tip{r}{r_i}) + \tip{b}{b_{ai}}) / 16\bigr)\, \tip{r}{r_i}, \qquad i \in \{\tip{e}{e_t}, \tip{y}{y_{<a}}\}
(8)

s3e34 · loop 8 × 2

Fℓ(2)=Fℓ(1),y<a={y1..16(1), y<a(2)}\tip{F}{F^{(2)}_{\ell}} = \tip{F}{F^{(1)}_{\ell}}, \qquad \tip{y}{y_{<a}} = \{\tip{y}{y^{(1)}_{1..16}},\, \tip{y}{y^{(2)}_{<a}}\}
(9)
  • s3e34 8 layers × 2
  • s3e35 12 layers
  • s3e1 16 layers, LR 5e-3
  • s3e33 baseline
−0.200−0.150−0.100−0.0500+0.050010k20k30k40k50k60k70ktraining updateΔ validation loss vs baseline (nats)↓ better than baselines3e34 −0.068s3e35 −0.095s3e1 −0.155s3e33 (baseline)
Table of values
Updates3e34s3e35s3e1
5k−0.057−0.071−0.119
10k−0.058−0.074−0.122
15k−0.057−0.075−0.122
20k−0.059−0.076−0.125
25k−0.060−0.079−0.127
30k−0.062−0.078−0.128
35k−0.061−0.080−0.130
40k−0.062−0.081−0.132
45k−0.064−0.083−0.135
50k−0.064−0.084−0.137
55k−0.065−0.087−0.140
60k−0.065−0.089−0.144
65k−0.066−0.092−0.149
70k−0.067−0.094−0.153
75k−0.068−0.095−0.155
Figure 10. Depth and compute controls. Validation loss minus s3e33's (8 layers) at each checkpoint; lower is better.

516-layer foundations#

Fixing the learning rate and the attention values

17–22 September · baseline s3e1 (16 layers, LR 5e-3) · 4 runs

Setting 3 started with a sixteen-layer Attention Residuals model and a learning-rate sweep around 5 × 10⁻³ (s3e1–s3e4), followed by one architectural probe: s3e9 adds a learned per-head RMSNorm to the attention values after their short convolution, alongside the existing query and key norms.

s3e1

The 16-layer reference.

Δ vs s3e1
baseline
Validation
2.3759 at 75k
Cost
2.52 s / update

finished

s3e2

Learning rate 2 × 10⁻³.

Δ vs s3e1
+0.0037 at 75k
Validation
2.3796 at 75k
Cost
2.49 s / update

finished

s3e3

Learning rate 3 × 10⁻³.

Δ vs s3e1
−0.0005 at 75k
Validation
2.3754 at 75k
Cost
2.51 s / update

finished

s3e4

Learning rate 7 × 10⁻³.

Δ vs s3e1
+0.0019 at 75k
Validation
2.3778 at 75k
Cost
2.51 s / update

finished

s3e9

Per-head RMSNorm on attention values.

Δ vs s3e1
−0.0030 at 65k
Validation
2.4266 at 65k
Cost
2.51 s / update

paused at 65.2k

s3e9 · value normalization

vh←γv⊙vhRMS(vh),vh=conv4(Wv x)h∈R128\tip{v}{v_h} \leftarrow \tip{gamma}{\gamma_v} \odot \frac{\tip{v}{v_h}}{\tip{rms}{\mathrm{RMS}}(\tip{v}{v_h})}, \qquad \tip{v}{v_h} = \tip{conv}{\mathrm{conv}_4}(\tip{Wv}{W_v}\, x)_h \in \mathbb{R}^{128}
(10)
  • s3e2 LR 2e-3
  • s3e3 LR 3e-3
  • s3e4 LR 7e-3
  • s3e9 value RMSNorm
  • s3e1 baseline
−0.100−0.075−0.050−0.0250+0.025+0.050010k20k30k40k50k60k70ktraining updateΔ validation loss vs baseline (nats)↓ better than baselines3e2 +0.004s3e3 −0.000s3e4 +0.002s3e9 −0.003s3e1 (baseline)
Table of values
Updates3e2s3e3s3e4s3e9
5k−0.041−0.029+0.026−0.003
10k−0.059−0.039+0.033−0.003
15k−0.067−0.044+0.034−0.004
20k−0.069−0.043+0.036−0.003
25k−0.070−0.044+0.035−0.002
30k−0.068−0.044+0.035−0.003
35k−0.067−0.042+0.034−0.003
40k−0.062−0.039+0.032−0.002
45k−0.058−0.037+0.030−0.003
50k−0.052−0.034+0.026−0.003
55k−0.045−0.030+0.023−0.003
60k−0.034−0.023+0.019−0.003
65k−0.021−0.016+0.014−0.003
70k−0.006−0.007+0.007–
75k+0.004−0.000+0.002–
Figure 11. 16-layer foundations. Validation loss minus s3e1's (16 layers, LR 5e-3) at each checkpoint; lower is better.
  • at 30k updates
  • at the last checkpoint
−0.080−0.060−0.040−0.0200+0.020+0.040s3e1← betters3e2LR 2e-3s3e3LR 3e-3s3e4LR 7e-3s3e9value RMSNorm
Table of values
RunSettingΔ at 30kΔ at end
s3e2LR 2e-3−0.068+0.004
s3e3LR 3e-3−0.044−0.000
s3e4LR 7e-3+0.035+0.002
s3e9value RMSNorm−0.003−0.003
Figure 12. Early leads fade. Gap to s3e1 at 30k and at each run's last checkpoint (75k; 65k for s3e9).