# CUDA chain, the one-pass prompt attention (cc_attn_f2) on sm_75: the base, a 64-token prompt and its 32 teacher-forced steps, then a 512-token prompt and its steps, then a 64-token one, in one process (2026-10-28)
# Google Colab T4 high-RAM VM: Intel(R) Xeon(R) CPU @ 2.00GHz (4 cores / 8 threads, 38.5 MiB L3) + NVIDIA Tesla T4 (15360 MiB, sm_75, up to PCIe Gen3 x16), 51 GiB; Ubuntu 24.04.4 LTS, CUDA 13.0 (nvcc V13.0.88), driver 580.82.07, gcc 13.3.0.
# Model: Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64 (int4, 64-input groups; 40 layers of 256 experts, 10 of them full attention with 16 query heads and 2 kv heads of 256); CUDA_EXPERT_GB=auto: 6605 of the 10240 experts on the card, 3635 in the page-locked host pool, no heat file; the KV cache and its mirrors float16 (origin/dev's default).
# This change's commit is 60c72fa6 (c/ tree df3fc684854104c832825d064485ff5fab970b52), on origin/dev 2e981752; the runs measured it before its review, 3655d4e8 (correctness.txt prints the review's diff).
# Text: promessi_sposi.txt (sha256 f53e0d80cb2d4492d24ebd63c7000c397b16ae70f9bf09b3763e5d8323ec209f), speed-bench/promessi_sposi.txt of antirez's ds4.
# A warm-up run of v0 (not counted), then v0 v1 v1 v0 v0 v1 v1 v0, each once the card was cool (bench waits for it) and then 60 s of rest.
#   Every run's CSV and stderr in full, in the order they ran; the steps files by their sha256. The summary at the end.
# The measured change: ladder-up.txt.
# Harness: bench.c beside this file (sha256 8bfbeab882467558e8647dd03543ded67696e64917c4a4c2b137d146de2db458), built against each version's c/ by bench's driver (make CUDA=1 CUDA_ARCH=sm_75).

## measure folder zona1
# v0 = origin/dev 2e981752 (2e981752, c/ tree f52d06702fb1f564dd5f573ab5eab90be89beb30)
# v1 = origin/dev 2e981752 + this change before its review (3655d4e8, c/ tree b462be1166044e1ab76a948dd07d9af31e3ebe52)
# setup.txt:
#   bench 1cd0e3d05691; engine qwen36; Linux (Intel(R) Xeon(R) CPU @ 2.00GHz, Tesla T4); 2026-10-28 16:47
#   model /content/q36; text promessi_sposi.txt (sha256 f53e0d80cb2d4492); lengths 64,512,64; gen 32; teacher-forced
#   runs: a warm-up of v0 (does not count), then 4 per version, the order turning at every round (ABBA); before every run: wait until cool (at most 900 s), then 60 s of rest
#   v0 a: /content/f2/bin-a/qwen36-bench (commit 2e98175273386a32899e907ace17d17d758b7b49; harness qwen36_bench.c 8bfbeab88246; CUDA); env: COLI_CUDA=1 COLI_GPUS=0 CUDA_EXPERT_GB=auto OMP_WAIT_POLICY=ACTIVE OMP_PROC_BIND=close
#      command: env -i PATH=/usr/bin:/bin HOME=/root LD_LIBRARY_PATH=/usr/lib64-nvidia COLI_CUDA=1 COLI_GPUS=0 CUDA_EXPERT_GB=auto OMP_WAIT_POLICY=ACTIVE OMP_PROC_BIND=close /content/f2/bin-a/qwen36-bench --model /content/q36 --text /content/bench/promessi_sposi.txt --lengths 64,512,64 --gen 32 --teacher --steps /content/f2/zona1/v0-runN.steps
#   v1 b: /content/f2/bin-b/qwen36-bench (commit 3655d4e852045ed2b32bb3895d2a209f71372af9; harness qwen36_bench.c 8bfbeab88246; CUDA); env: COLI_CUDA=1 COLI_GPUS=0 CUDA_EXPERT_GB=auto OMP_WAIT_POLICY=ACTIVE OMP_PROC_BIND=close
#      command: env -i PATH=/usr/bin:/bin HOME=/root LD_LIBRARY_PATH=/usr/lib64-nvidia COLI_CUDA=1 COLI_GPUS=0 CUDA_EXPERT_GB=auto OMP_WAIT_POLICY=ACTIVE OMP_PROC_BIND=close /content/f2/bin-b/qwen36-bench --model /content/q36 --text /content/bench/promessi_sposi.txt --lengths 64,512,64 --gen 32 --teacher --steps /content/f2/zona1/v1-runN.steps

=== run 1 of 8: v0-run1 (v0) ===
--- thermal-before: 2026-10-28 16:49:44 | temp=61 sm=300 reasons=0x1 | swap_used=0.00M swapouts=0
--- thermal-after: 2026-10-28 16:50:15 | temp=67 sm=585 reasons=0x0 | swap_used=0.00M swapouts=0
--- csv
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
64,64,0.818,78.21,32,65.8,39.294,0,0.00,ffb816333a929871,5cb23346ab6be1d8,2.69296,0.469
512,512,2.892,177.05,32,22.4,46.553,0,0.00,c15c55e684fbd2c5,7a8848450b863c49,2.96104,0.438
64,64,0.854,74.95,32,20.4,49.793,0,0.00,b8bb564b85d5dbec,5cb23346ab6be1d8,2.68845,0.469
--- stderr
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[OMP] qwen36: 4 physical-core threads instead of 8 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8192 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.7s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: Tesla T4, 15.6 GB VRAM, sm_75
[qtier] dev 0: 14.5 GB free, allowance 13.4 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 11.64 GB left for experts
[qtier] dev 0: budget 11.64 GB for experts (~6605 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.383 ms on CUDA dev 0, 0.766 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] host pool: 3635 experts of the RAM cache page-locked for the device to copy a prompt chunk's experts from (COLI_CUDA_HOST_POOL=0 off)
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 6605 in VRAM -- 9.2 s
[bench] CUDA ready in 20.05 s; text 6182 tokens; lengths 64,512,64; gen 32; each length from an empty state
[qwen36] KV cache: float16, 0.00 GiB at 96 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 2048 rows per prompt chunk (286 KiB a row, 1.7 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[bench] peak RSS 19.48 GB
[qtier] host pool: 3635 experts (5.99 GB of the RAM cache) page-locked for the device's copies
--- steps sha256 9c65fa4c335c1ef08195d064da544904423dc49097bbbaf2adbf9fc00bcd2411 (2185 bytes)

=== run 2 of 8: v1-run1 (v1) ===
--- thermal-before: 2026-10-28 16:51:15 | temp=60 sm=300 reasons=0x1 | swap_used=0.00M swapouts=0
--- thermal-after: 2026-10-28 16:51:45 | temp=66 sm=585 reasons=0x0 | swap_used=0.00M swapouts=0
--- csv
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
64,64,0.839,76.26,32,64.8,39.516,0,0.00,2de56042834c385e,5cb23346ab6be1d8,2.69981,0.469
512,512,2.879,177.87,32,22.2,46.757,0,0.00,a5935566e1fbd665,7a8848450b863c49,2.94298,0.438
64,64,0.856,74.80,32,20.3,49.698,0,0.00,1e06a7a3ccabe108,5cb23346ab6be1d8,2.70021,0.469
--- stderr
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[OMP] qwen36: 4 physical-core threads instead of 8 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8192 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.6s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: Tesla T4, 15.6 GB VRAM, sm_75
[qtier] dev 0: 14.5 GB free, allowance 13.4 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 11.64 GB left for experts
[qtier] dev 0: budget 11.64 GB for experts (~6605 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.456 ms on CUDA dev 0, 0.808 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] host pool: 3635 experts of the RAM cache page-locked for the device to copy a prompt chunk's experts from (COLI_CUDA_HOST_POOL=0 off)
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 6605 in VRAM -- 9.3 s
[bench] CUDA ready in 20.16 s; text 6182 tokens; lengths 64,512,64; gen 32; each length from an empty state
[qwen36] KV cache: float16, 0.00 GiB at 96 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 2048 rows per prompt chunk (286 KiB a row, 1.7 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[bench] peak RSS 19.48 GB
[qtier] host pool: 3635 experts (5.99 GB of the RAM cache) page-locked for the device's copies
--- steps sha256 37c1b1e9f7ee238cc3021727f480aab81e1dd18a22b505bf117d7b42c964a2ba (2185 bytes)

=== run 3 of 8: v1-run2 (v1) ===
--- thermal-before: 2026-10-28 16:52:45 | temp=60 sm=300 reasons=0x1 | swap_used=0.00M swapouts=0
--- thermal-after: 2026-10-28 16:53:15 | temp=65 sm=585 reasons=0x0 | swap_used=0.00M swapouts=0
--- csv
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
64,64,0.841,76.07,32,64.9,39.609,0,0.00,2de56042834c385e,5cb23346ab6be1d8,2.69981,0.469
512,512,2.878,177.91,32,22.2,46.622,0,0.00,a5935566e1fbd665,7a8848450b863c49,2.94298,0.438
64,64,0.855,74.87,32,20.3,49.792,0,0.00,1e06a7a3ccabe108,5cb23346ab6be1d8,2.70021,0.469
--- stderr
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[OMP] qwen36: 4 physical-core threads instead of 8 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8192 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.6s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: Tesla T4, 15.6 GB VRAM, sm_75
[qtier] dev 0: 14.5 GB free, allowance 13.4 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 11.64 GB left for experts
[qtier] dev 0: budget 11.64 GB for experts (~6605 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.424 ms on CUDA dev 0, 0.822 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] host pool: 3635 experts of the RAM cache page-locked for the device to copy a prompt chunk's experts from (COLI_CUDA_HOST_POOL=0 off)
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 6605 in VRAM -- 9.3 s
[bench] CUDA ready in 20.10 s; text 6182 tokens; lengths 64,512,64; gen 32; each length from an empty state
[qwen36] KV cache: float16, 0.00 GiB at 96 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 2048 rows per prompt chunk (286 KiB a row, 1.7 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[bench] peak RSS 19.48 GB
[qtier] host pool: 3635 experts (5.99 GB of the RAM cache) page-locked for the device's copies
--- steps sha256 a468f1f177d189a2862027b9f45d171fd7a8536001acca2bd218f93586098c7e (2185 bytes)

=== run 4 of 8: v0-run2 (v0) ===
--- thermal-before: 2026-10-28 16:54:16 | temp=59 sm=300 reasons=0x1 | swap_used=0.00M swapouts=0
--- thermal-after: 2026-10-28 16:54:46 | temp=64 sm=585 reasons=0x0 | swap_used=0.00M swapouts=0
--- csv
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
64,64,0.813,78.73,32,64.9,39.560,0,0.00,ffb816333a929871,5cb23346ab6be1d8,2.69296,0.469
512,512,2.876,178.04,32,22.5,46.696,0,0.00,a91c46c5619c2bdf,7a8848450b863c49,2.95784,0.438
64,64,0.848,75.44,32,20.2,49.820,0,0.00,b8bb564b85d5dbec,5cb23346ab6be1d8,2.68845,0.469
--- stderr
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[OMP] qwen36: 4 physical-core threads instead of 8 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8192 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.6s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: Tesla T4, 15.6 GB VRAM, sm_75
[qtier] dev 0: 14.5 GB free, allowance 13.4 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 11.64 GB left for experts
[qtier] dev 0: budget 11.64 GB for experts (~6605 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.401 ms on CUDA dev 0, 0.797 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] host pool: 3635 experts of the RAM cache page-locked for the device to copy a prompt chunk's experts from (COLI_CUDA_HOST_POOL=0 off)
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 6605 in VRAM -- 9.2 s
[bench] CUDA ready in 20.03 s; text 6182 tokens; lengths 64,512,64; gen 32; each length from an empty state
[qwen36] KV cache: float16, 0.00 GiB at 96 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 2048 rows per prompt chunk (286 KiB a row, 1.7 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[bench] peak RSS 19.48 GB
[qtier] host pool: 3635 experts (5.99 GB of the RAM cache) page-locked for the device's copies
--- steps sha256 f9cdd30ac89c56ee25ff4d0b353782953cbedf95be968349f6927dda1e3a418a (2185 bytes)

=== run 5 of 8: v0-run3 (v0) ===
--- thermal-before: 2026-10-28 16:55:46 | temp=57 sm=300 reasons=0x1 | swap_used=0.00M swapouts=0
--- thermal-after: 2026-10-28 16:56:16 | temp=62 sm=585 reasons=0x0 | swap_used=0.00M swapouts=0
--- csv
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
64,64,0.837,76.46,32,64.8,39.639,0,0.00,ffb816333a929871,5cb23346ab6be1d8,2.69296,0.469
512,512,2.874,178.17,32,21.8,46.232,0,0.00,c15c55e684fbd2c5,7a8848450b863c49,2.96104,0.438
64,64,0.851,75.19,32,20.5,49.953,0,0.00,b8bb564b85d5dbec,5cb23346ab6be1d8,2.68845,0.469
--- stderr
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[OMP] qwen36: 4 physical-core threads instead of 8 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8192 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.6s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: Tesla T4, 15.6 GB VRAM, sm_75
[qtier] dev 0: 14.5 GB free, allowance 13.4 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 11.64 GB left for experts
[qtier] dev 0: budget 11.64 GB for experts (~6605 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.410 ms on CUDA dev 0, 0.790 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] host pool: 3635 experts of the RAM cache page-locked for the device to copy a prompt chunk's experts from (COLI_CUDA_HOST_POOL=0 off)
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 6605 in VRAM -- 9.2 s
[bench] CUDA ready in 19.98 s; text 6182 tokens; lengths 64,512,64; gen 32; each length from an empty state
[qwen36] KV cache: float16, 0.00 GiB at 96 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 2048 rows per prompt chunk (286 KiB a row, 1.7 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[bench] peak RSS 19.48 GB
[qtier] host pool: 3635 experts (5.99 GB of the RAM cache) page-locked for the device's copies
--- steps sha256 bfd37d5950a46729dd7a719eef47f23a9010f7beb27f0bdc8b059b8fb8cb8ec3 (2185 bytes)

=== run 6 of 8: v1-run3 (v1) ===
--- thermal-before: 2026-10-28 16:57:16 | temp=55 sm=300 reasons=0x1 | swap_used=0.00M swapouts=0
--- thermal-after: 2026-10-28 16:57:46 | temp=60 sm=585 reasons=0x0 | swap_used=0.00M swapouts=0
--- csv
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
64,64,0.822,77.81,32,64.9,39.699,0,0.00,2de56042834c385e,5cb23346ab6be1d8,2.69981,0.469
512,512,2.865,178.69,32,22.2,46.652,0,0.00,a5935566e1fbd665,7a8848450b863c49,2.94298,0.438
64,64,0.849,75.34,32,20.5,50.021,0,0.00,1e06a7a3ccabe108,5cb23346ab6be1d8,2.70021,0.469
--- stderr
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[OMP] qwen36: 4 physical-core threads instead of 8 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8192 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.6s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: Tesla T4, 15.6 GB VRAM, sm_75
[qtier] dev 0: 14.5 GB free, allowance 13.4 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 11.64 GB left for experts
[qtier] dev 0: budget 11.64 GB for experts (~6605 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.476 ms on CUDA dev 0, 0.774 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] host pool: 3635 experts of the RAM cache page-locked for the device to copy a prompt chunk's experts from (COLI_CUDA_HOST_POOL=0 off)
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 6605 in VRAM -- 9.2 s
[bench] CUDA ready in 19.99 s; text 6182 tokens; lengths 64,512,64; gen 32; each length from an empty state
[qwen36] KV cache: float16, 0.00 GiB at 96 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 2048 rows per prompt chunk (286 KiB a row, 1.7 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[bench] peak RSS 19.48 GB
[qtier] host pool: 3635 experts (5.99 GB of the RAM cache) page-locked for the device's copies
--- steps sha256 f4962d9790f4768ec72918d0b31564049d8b15fc5c60538f17e2a166dbd2c45b (2185 bytes)

=== run 7 of 8: v1-run4 (v1) ===
--- thermal-before: 2026-10-28 16:58:46 | temp=55 sm=300 reasons=0x1 | swap_used=0.00M swapouts=0
--- thermal-after: 2026-10-28 16:59:17 | temp=59 sm=585 reasons=0x0 | swap_used=0.00M swapouts=0
--- csv
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
64,64,0.804,79.63,32,65.3,39.599,0,0.00,2de56042834c385e,5cb23346ab6be1d8,2.69981,0.469
512,512,2.868,178.53,32,22.4,46.412,0,0.00,a5935566e1fbd665,7a8848450b863c49,2.94298,0.438
64,64,0.853,75.07,32,20.3,50.040,0,0.00,1e06a7a3ccabe108,5cb23346ab6be1d8,2.70021,0.469
--- stderr
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[OMP] qwen36: 4 physical-core threads instead of 8 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8192 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.6s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: Tesla T4, 15.6 GB VRAM, sm_75
[qtier] dev 0: 14.5 GB free, allowance 13.4 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 11.64 GB left for experts
[qtier] dev 0: budget 11.64 GB for experts (~6605 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.444 ms on CUDA dev 0, 0.802 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] host pool: 3635 experts of the RAM cache page-locked for the device to copy a prompt chunk's experts from (COLI_CUDA_HOST_POOL=0 off)
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 6605 in VRAM -- 9.2 s
[bench] CUDA ready in 20.02 s; text 6182 tokens; lengths 64,512,64; gen 32; each length from an empty state
[qwen36] KV cache: float16, 0.00 GiB at 96 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 2048 rows per prompt chunk (286 KiB a row, 1.7 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[bench] peak RSS 19.48 GB
[qtier] host pool: 3635 experts (5.99 GB of the RAM cache) page-locked for the device's copies
--- steps sha256 852a2fe10e602108376d390f7cbc13fa218d1c69aa76cb29ff98b8b0b157504e (2185 bytes)

=== run 8 of 8: v0-run4 (v0) ===
--- thermal-before: 2026-10-28 17:00:17 | temp=54 sm=300 reasons=0x1 | swap_used=0.00M swapouts=0
--- thermal-after: 2026-10-28 17:00:47 | temp=59 sm=585 reasons=0x0 | swap_used=0.00M swapouts=0
--- csv
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
64,64,0.804,79.64,32,65.5,39.534,0,0.00,ffb816333a929871,5cb23346ab6be1d8,2.69296,0.469
512,512,2.890,177.18,32,22.0,46.358,0,0.00,c15c55e684fbd2c5,7a8848450b863c49,2.96104,0.438
64,64,0.852,75.15,32,20.2,50.045,0,0.00,b8bb564b85d5dbec,5cb23346ab6be1d8,2.68845,0.469
--- stderr
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[OMP] qwen36: 4 physical-core threads instead of 8 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8192 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.5s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: Tesla T4, 15.6 GB VRAM, sm_75
[qtier] dev 0: 14.5 GB free, allowance 13.4 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 11.64 GB left for experts
[qtier] dev 0: budget 11.64 GB for experts (~6605 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.515 ms on CUDA dev 0, 0.796 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] host pool: 3635 experts of the RAM cache page-locked for the device to copy a prompt chunk's experts from (COLI_CUDA_HOST_POOL=0 off)
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 6605 in VRAM -- 9.3 s
[bench] CUDA ready in 19.99 s; text 6182 tokens; lengths 64,512,64; gen 32; each length from an empty state
[qwen36] KV cache: float16, 0.00 GiB at 96 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 2048 rows per prompt chunk (286 KiB a row, 1.7 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[bench] peak RSS 19.48 GB
[qtier] host pool: 3635 experts (5.99 GB of the RAM cache) page-locked for the device's copies
--- steps sha256 e7a488d503125d2d150ed7b00754fc6b555d1a816006a795b5a4a5635df0aacd (2185 bytes)

=== summary.md ===
# bench: zona1

- bench 1cd0e3d05691; engine qwen36; Linux (Intel(R) Xeon(R) CPU @ 2.00GHz, Tesla T4); 2026-10-28 16:47
- model q36; text promessi_sposi.txt (sha256 f53e0d80cb2d4492); lengths 64,512,64; gen 32; teacher-forced
- runs: a warm-up of v0 (does not count), then 4 per version, the order turning at every round (ABBA); before every run: wait until cool (at most 900 s), then 60 s of rest

## v0 a: 4 runs (median, min–max)

| tokens | new | prompt tok/s | first step ms | decode tok/s | experts read | reads' s | nll | top1 | fingerprint |
|---|---|---|---|---|---|---|---|---|---|
| 64 | 64 | 78.47 (76.46–79.64) | 65 (65–66) | 39.547 (39.294–39.639) | 0 | 0.00 | 2.6930 | 0.469 | ffb816333a929871 |
| 512 | 512 | 177.61 (177.05–178.17) | 22 (22–22) | 46.456 (46.232–46.696) | 0 | 0.00 | 2.9610 | 0.438 | DIFFERENT between runs: ['a91c46c5619c2bdf', 'c15c55e684fbd2c5'] |
| 64 after 512 | 64 | 75.17 (74.95–75.44) | 20 (20–20) | 49.886 (49.793–50.045) | 0 | 0.00 | 2.6885 | 0.469 | b8bb564b85d5dbec |

peak memory 19948 MiB, model ready in 20.01 s (medians)

decode by steps (step 1 gives the second new word; medians of 4 runs):

| tokens | steps | experts read | decode tok/s | reads' s |
|---|---|---|---|---|
| 64 | 2 to 32 | 0 | 39.547 | 0.000 |
| 512 | 2 to 32 | 0 | 46.455 | 0.000 |
| 64 after 512 | 2 to 32 | 0 | 49.886 | 0.000 |

## v1 b: 4 runs (median, min–max)

| tokens | new | prompt tok/s | first step ms | decode tok/s | experts read | reads' s | nll | top1 | fingerprint |
|---|---|---|---|---|---|---|---|---|---|
| 64 | 64 | 77.03 (76.07–79.63) | 65 (65–65) | 39.604 (39.516–39.699) | 0 | 0.00 | 2.6998 | 0.469 | 2de56042834c385e |
| 512 | 512 | 178.22 (177.87–178.69) | 22 (22–22) | 46.637 (46.412–46.757) | 0 | 0.00 | 2.9430 | 0.438 | a5935566e1fbd665 |
| 64 after 512 | 64 | 74.97 (74.80–75.34) | 20 (20–20) | 49.907 (49.698–50.040) | 0 | 0.00 | 2.7002 | 0.469 | 1e06a7a3ccabe108 |

peak memory 19948 MiB, model ready in 20.06 s (medians)

decode by steps (step 1 gives the second new word; medians of 4 runs):

| tokens | steps | experts read | decode tok/s | reads' s |
|---|---|---|---|---|
| 64 | 2 to 32 | 0 | 39.604 | 0.000 |
| 512 | 2 to 32 | 0 | 46.637 | 0.000 |
| 64 after 512 | 2 to 32 | 0 | 49.906 | 0.000 |

## v1 b against v0 a (medians; positive = faster)

| tokens | prompt | first step | decode | results |
|---|---|---|---|---|
| 64 | -1.8%, ranges overlap | +0.5%, ranges overlap | +0.1%, ranges overlap | same words, logits differ; nll 2.6930 → 2.6998, top1 0.469 → 0.469 |
| 512 | +0.3%, ranges overlap | +0.0%, ranges overlap | +0.4%, ranges overlap | same words, logits differ; nll 2.9610 → 2.9430, top1 0.438 → 0.438 |
| 64 after 512 | -0.3%, ranges overlap | -0.0%, ranges overlap | +0.0%, ranges overlap | same words, logits differ; nll 2.6885 → 2.7002, top1 0.469 → 0.469 |

| tokens | steps | experts read | decode | reads' s |
|---|---|---|---|---|
| 64 | 2 to 32 | 0 → 0 | +0.1%, ranges overlap | 0.000 → 0.000 s |
| 512 | 2 to 32 | 0 → 0 | +0.4%, ranges overlap | 0.000 → 0.000 s |
| 64 after 512 | 2 to 32 | 0 → 0 | +0.0%, ranges overlap | 0.000 → 0.000 s |

warm-up (does not count): 64 tokens: prompt 77.62 tok/s, decode 39.576 tok/s; 512 tokens: prompt 176.95 tok/s, decode 46.804 tok/s; 64 tokens: prompt 75.00 tok/s, decode 49.940 tok/s
runs not cool: none
pages written to swap during the measurement: 0, 0 of them in the runs that count
