# CUDA chain, the one-pass prompt attention (cc_attn_f2) on sm_75: the same lengths in falling order (8192 first) (2026-10-28)
# Google Colab T4 high-RAM VM: Intel(R) Xeon(R) CPU @ 2.00GHz (4 cores / 8 threads, 38.5 MiB L3) + NVIDIA Tesla T4 (15360 MiB, sm_75, up to PCIe Gen3 x16), 51 GiB; Ubuntu 24.04.4 LTS, CUDA 13.0 (nvcc V13.0.88), driver 580.82.07, gcc 13.3.0.
# Model: Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64 (int4, 64-input groups; 40 layers of 256 experts, 10 of them full attention with 16 query heads and 2 kv heads of 256); CUDA_EXPERT_GB=auto: 6605 of the 10240 experts on the card, 3635 in the page-locked host pool, no heat file; the KV cache and its mirrors float16 (origin/dev's default).
# This change's commit is 60c72fa6 (c/ tree df3fc684854104c832825d064485ff5fab970b52), on origin/dev 2e981752; the runs measured it before its review, 3655d4e8 (correctness.txt prints the review's diff).
# Text: promessi_sposi.txt (sha256 f53e0d80cb2d4492d24ebd63c7000c397b16ae70f9bf09b3763e5d8323ec209f), speed-bench/promessi_sposi.txt of antirez's ds4.
# A warm-up run of v0 (not counted), then v0 v1 v1 v0 v0 v1 v1 v0, each once the card was cool (bench waits for it) and then 60 s of rest.
#   Every run's CSV and stderr in full, in the order they ran; the steps files by their sha256. The summary at the end.
# The measured change: ladder-up.txt.
# Harness: bench.c beside this file (sha256 8bfbeab882467558e8647dd03543ded67696e64917c4a4c2b137d146de2db458), built against each version's c/ by bench's driver (make CUDA=1 CUDA_ARCH=sm_75).

## measure folder scala-giu
# v0 = origin/dev 2e981752 (2e981752, c/ tree f52d06702fb1f564dd5f573ab5eab90be89beb30)
# v1 = origin/dev 2e981752 + this change before its review (3655d4e8, c/ tree b462be1166044e1ab76a948dd07d9af31e3ebe52)
# setup.txt:
#   bench 1cd0e3d05691; engine qwen36; Linux (Intel(R) Xeon(R) CPU @ 2.00GHz, Tesla T4); 2026-10-28 16:06
#   model /content/q36; text promessi_sposi.txt (sha256 f53e0d80cb2d4492); lengths 8192,7168,6144,5120,4096,3072,2048; gen 128; teacher-forced
#   runs: a warm-up of v0 (does not count), then 4 per version, the order turning at every round (ABBA); before every run: wait until cool (at most 900 s), then 60 s of rest
#   v0 a: /content/f2/bin-a/qwen36-bench (commit 2e98175273386a32899e907ace17d17d758b7b49; harness qwen36_bench.c 8bfbeab88246; CUDA); env: COLI_CUDA=1 COLI_GPUS=0 CUDA_EXPERT_GB=auto OMP_WAIT_POLICY=ACTIVE OMP_PROC_BIND=close
#      command: env -i PATH=/usr/bin:/bin HOME=/root LD_LIBRARY_PATH=/usr/lib64-nvidia COLI_CUDA=1 COLI_GPUS=0 CUDA_EXPERT_GB=auto OMP_WAIT_POLICY=ACTIVE OMP_PROC_BIND=close /content/f2/bin-a/qwen36-bench --model /content/q36 --text /content/bench/promessi_sposi.txt --lengths 8192,7168,6144,5120,4096,3072,2048 --gen 128 --teacher --steps /content/f2/scala-giu/v0-runN.steps
#   v1 b: /content/f2/bin-b/qwen36-bench (commit 3655d4e852045ed2b32bb3895d2a209f71372af9; harness qwen36_bench.c 8bfbeab88246; CUDA); env: COLI_CUDA=1 COLI_GPUS=0 CUDA_EXPERT_GB=auto OMP_WAIT_POLICY=ACTIVE OMP_PROC_BIND=close
#      command: env -i PATH=/usr/bin:/bin HOME=/root LD_LIBRARY_PATH=/usr/lib64-nvidia COLI_CUDA=1 COLI_GPUS=0 CUDA_EXPERT_GB=auto OMP_WAIT_POLICY=ACTIVE OMP_PROC_BIND=close /content/f2/bin-b/qwen36-bench --model /content/q36 --text /content/bench/promessi_sposi.txt --lengths 8192,7168,6144,5120,4096,3072,2048 --gen 128 --teacher --steps /content/f2/scala-giu/v1-runN.steps

=== run 1 of 8: v0-run1 (v0) ===
--- thermal-before: 2026-10-28 16:12:20 | temp=62 sm=300 reasons=0x1 | swap_used=0.00M swapouts=0
--- thermal-after: 2026-10-28 16:15:47 | temp=74 sm=585 reasons=0x0 | swap_used=0.00M swapouts=0
--- csv
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
8192,8192,13.798,593.70,128,66.3,41.149,0,0.00,333543ed22bfca98,e54687bacc388530,2.42150,0.484
7168,7168,38.460,186.37,128,24.6,43.418,0,0.00,3b6b2bb4e1098947,05f26324095f3256,1.99007,0.531
6144,6144,32.259,190.46,128,26.8,41.331,0,0.00,1bf4892f8a150aed,54bf2b608342c377,1.28508,0.680
5120,5120,26.790,191.12,128,24.4,46.634,0,0.00,cb72d4307fe145ad,3eab0ef0e0213c20,2.19770,0.539
4096,4096,21.023,194.83,128,25.2,45.986,0,0.00,923ef5de07ebbe13,45601e6c456735d4,1.52905,0.609
3072,3072,15.634,196.49,128,21.3,50.194,0,0.00,f412eedc955e5165,52c1fbb2f3747d50,2.25481,0.492
2048,2048,10.152,201.73,128,20.9,50.902,0,0.00,b7168dd7af226bf1,d6661a686430b1fa,2.40331,0.484
--- stderr
[bench] Q36_MAXT=8384 (the longest length and the generated tokens)
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[OMP] qwen36: 4 physical-core threads instead of 8 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8384 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.5s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: Tesla T4, 15.6 GB VRAM, sm_75
[qtier] dev 0: 14.5 GB free, allowance 13.4 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 11.64 GB left for experts
[qtier] dev 0: budget 11.64 GB for experts (~6605 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.454 ms on CUDA dev 0, 0.789 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] host pool: 3635 experts of the RAM cache page-locked for the device to copy a prompt chunk's experts from (COLI_CUDA_HOST_POOL=0 off)
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 6605 in VRAM -- 9.2 s
[bench] CUDA ready in 19.93 s; text 79393 tokens; lengths 8192,7168,6144,5120,4096,3072,2048; gen 128; each length from an empty state
[qwen36] KV cache: float16, 0.16 GiB at 8320 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 2048 rows per prompt chunk (286 KiB a row, 1.7 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[bench] peak RSS 19.92 GB
[qtier] host pool: 3635 experts (5.99 GB of the RAM cache) page-locked for the device's copies
--- steps sha256 f2e91eaeed415e5ec58635489e91b05982a80054ac8f639b3652dba802c84722 (21819 bytes)

=== run 2 of 8: v1-run1 (v1) ===
--- thermal-before: 2026-10-28 16:16:57 | temp=62 sm=300 reasons=0x1 | swap_used=0.00M swapouts=0
--- thermal-after: 2026-10-28 16:20:09 | temp=75 sm=585 reasons=0x0 | swap_used=0.00M swapouts=0
--- csv
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
8192,8192,9.120,898.29,128,65.9,41.140,0,0.00,52dde184018ab698,e54687bacc388530,2.41221,0.484
7168,7168,34.605,207.14,128,24.6,43.486,0,0.00,e8ae7cfcd208897e,05f26324095f3256,1.98708,0.523
6144,6144,29.768,206.40,128,27.4,41.428,0,0.00,125e04ca6cf235a9,54bf2b608342c377,1.28047,0.688
5120,5120,25.019,204.64,128,24.1,46.824,0,0.00,e7943b93161c07ac,3eab0ef0e0213c20,2.20141,0.547
4096,4096,19.840,206.46,128,24.9,46.246,0,0.00,75ddcc777c118ba5,45601e6c456735d4,1.53093,0.602
3072,3072,15.063,203.94,128,21.8,50.226,0,0.00,d1cd4e31a2aa4070,52c1fbb2f3747d50,2.25691,0.500
2048,2048,9.902,206.82,128,20.9,51.041,0,0.00,5f101036c23d0b53,d6661a686430b1fa,2.40945,0.477
--- stderr
[bench] Q36_MAXT=8384 (the longest length and the generated tokens)
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[OMP] qwen36: 4 physical-core threads instead of 8 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8384 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.7s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: Tesla T4, 15.6 GB VRAM, sm_75
[qtier] dev 0: 14.5 GB free, allowance 13.4 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 11.64 GB left for experts
[qtier] dev 0: budget 11.64 GB for experts (~6605 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.402 ms on CUDA dev 0, 0.811 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] host pool: 3635 experts of the RAM cache page-locked for the device to copy a prompt chunk's experts from (COLI_CUDA_HOST_POOL=0 off)
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 6605 in VRAM -- 9.3 s
[bench] CUDA ready in 20.17 s; text 79393 tokens; lengths 8192,7168,6144,5120,4096,3072,2048; gen 128; each length from an empty state
[qwen36] KV cache: float16, 0.16 GiB at 8320 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 2048 rows per prompt chunk (286 KiB a row, 1.7 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[bench] peak RSS 19.92 GB
[qtier] host pool: 3635 experts (5.99 GB of the RAM cache) page-locked for the device's copies
--- steps sha256 19277942fe5b6a452b7580d1ab88b541fcec1e488d4303035a20ebca101dde53 (21817 bytes)

=== run 3 of 8: v1-run2 (v1) ===
--- thermal-before: 2026-10-28 16:21:19 | temp=62 sm=300 reasons=0x1 | swap_used=0.00M swapouts=0
--- thermal-after: 2026-10-28 16:24:31 | temp=74 sm=585 reasons=0x0 | swap_used=0.00M swapouts=0
--- csv
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
8192,8192,9.106,899.67,128,67.1,40.801,0,0.00,52dde184018ab698,e54687bacc388530,2.41221,0.484
7168,7168,34.731,206.38,128,24.2,43.269,0,0.00,e8ae7cfcd208897e,05f26324095f3256,1.98708,0.523
6144,6144,29.690,206.94,128,27.4,41.231,0,0.00,125e04ca6cf235a9,54bf2b608342c377,1.28047,0.688
5120,5120,25.011,204.71,128,24.3,46.672,0,0.00,e7943b93161c07ac,3eab0ef0e0213c20,2.20141,0.547
4096,4096,19.798,206.89,128,25.3,46.391,0,0.00,75ddcc777c118ba5,45601e6c456735d4,1.53093,0.602
3072,3072,15.067,203.89,128,21.4,50.085,0,0.00,d1cd4e31a2aa4070,52c1fbb2f3747d50,2.25691,0.500
2048,2048,9.914,206.58,128,20.6,50.896,0,0.00,5f101036c23d0b53,d6661a686430b1fa,2.40945,0.477
--- stderr
[bench] Q36_MAXT=8384 (the longest length and the generated tokens)
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[OMP] qwen36: 4 physical-core threads instead of 8 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8384 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.6s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: Tesla T4, 15.6 GB VRAM, sm_75
[qtier] dev 0: 14.5 GB free, allowance 13.4 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 11.64 GB left for experts
[qtier] dev 0: budget 11.64 GB for experts (~6605 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.394 ms on CUDA dev 0, 0.805 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] host pool: 3635 experts of the RAM cache page-locked for the device to copy a prompt chunk's experts from (COLI_CUDA_HOST_POOL=0 off)
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 6605 in VRAM -- 9.3 s
[bench] CUDA ready in 20.14 s; text 79393 tokens; lengths 8192,7168,6144,5120,4096,3072,2048; gen 128; each length from an empty state
[qwen36] KV cache: float16, 0.16 GiB at 8320 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 2048 rows per prompt chunk (286 KiB a row, 1.7 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[bench] peak RSS 19.92 GB
[qtier] host pool: 3635 experts (5.99 GB of the RAM cache) page-locked for the device's copies
--- steps sha256 97c64d28ddc724dc5eb60b61edec4c9299b7f831145d4ca0d49d1a4d64b6b52d (21817 bytes)

=== run 4 of 8: v0-run2 (v0) ===
--- thermal-before: 2026-10-28 16:25:41 | temp=62 sm=300 reasons=0x1 | swap_used=0.00M swapouts=0
--- thermal-after: 2026-10-28 16:29:08 | temp=74 sm=585 reasons=0x0 | swap_used=0.00M swapouts=0
--- csv
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
8192,8192,13.759,595.39,128,66.3,41.048,0,0.00,333543ed22bfca98,e54687bacc388530,2.42150,0.484
7168,7168,38.618,185.61,128,25.0,43.390,0,0.00,3b6b2bb4e1098947,05f26324095f3256,1.99007,0.531
6144,6144,32.268,190.40,128,27.6,41.547,0,0.00,1bf4892f8a150aed,54bf2b608342c377,1.28508,0.680
5120,5120,26.844,190.73,128,23.9,46.796,0,0.00,b0bfefe493bd9c1f,3eab0ef0e0213c20,2.19772,0.539
4096,4096,20.966,195.37,128,25.6,46.199,0,0.00,b47ba1322382be7d,45601e6c456735d4,1.53045,0.609
3072,3072,15.604,196.87,128,22.0,50.057,0,0.00,f6ec3d5af9818813,52c1fbb2f3747d50,2.25262,0.492
2048,2048,10.139,202.00,128,20.6,50.830,0,0.00,154b852aa11aa7cf,d6661a686430b1fa,2.40313,0.484
--- stderr
[bench] Q36_MAXT=8384 (the longest length and the generated tokens)
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[OMP] qwen36: 4 physical-core threads instead of 8 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8384 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.7s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: Tesla T4, 15.6 GB VRAM, sm_75
[qtier] dev 0: 14.5 GB free, allowance 13.4 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 11.64 GB left for experts
[qtier] dev 0: budget 11.64 GB for experts (~6605 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.424 ms on CUDA dev 0, 0.792 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] host pool: 3635 experts of the RAM cache page-locked for the device to copy a prompt chunk's experts from (COLI_CUDA_HOST_POOL=0 off)
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 6605 in VRAM -- 9.2 s
[bench] CUDA ready in 20.09 s; text 79393 tokens; lengths 8192,7168,6144,5120,4096,3072,2048; gen 128; each length from an empty state
[qwen36] KV cache: float16, 0.16 GiB at 8320 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 2048 rows per prompt chunk (286 KiB a row, 1.7 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[bench] peak RSS 19.92 GB
[qtier] host pool: 3635 experts (5.99 GB of the RAM cache) page-locked for the device's copies
--- steps sha256 0ff3e65e82f133b494597e93d1e4e81998f367b3df38410a330975fbb46c730f (21819 bytes)

=== run 5 of 8: v0-run3 (v0) ===
--- thermal-before: 2026-10-28 16:30:18 | temp=62 sm=300 reasons=0x1 | swap_used=0.00M swapouts=0
--- thermal-after: 2026-10-28 16:33:44 | temp=74 sm=585 reasons=0x0 | swap_used=0.00M swapouts=0
--- csv
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
8192,8192,13.804,593.46,128,67.5,40.903,0,0.00,333543ed22bfca98,e54687bacc388530,2.42150,0.484
7168,7168,38.520,186.08,128,24.8,43.619,0,0.00,3b6b2bb4e1098947,05f26324095f3256,1.99007,0.531
6144,6144,32.220,190.69,128,27.9,41.335,0,0.00,1bf4892f8a150aed,54bf2b608342c377,1.28508,0.680
5120,5120,26.801,191.04,128,24.6,46.743,0,0.00,cb72d4307fe145ad,3eab0ef0e0213c20,2.19770,0.539
4096,4096,20.974,195.29,128,24.9,46.183,0,0.00,b47ba1322382be7d,45601e6c456735d4,1.53045,0.609
3072,3072,15.578,197.20,128,21.4,50.441,0,0.00,f7dfee4b18e9112b,52c1fbb2f3747d50,2.25497,0.492
2048,2048,10.113,202.51,128,20.2,50.790,0,0.00,7ddd20f48fbacd5c,d6661a686430b1fa,2.40315,0.484
--- stderr
[bench] Q36_MAXT=8384 (the longest length and the generated tokens)
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[OMP] qwen36: 4 physical-core threads instead of 8 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8384 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.7s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: Tesla T4, 15.6 GB VRAM, sm_75
[qtier] dev 0: 14.5 GB free, allowance 13.4 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 11.64 GB left for experts
[qtier] dev 0: budget 11.64 GB for experts (~6605 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.444 ms on CUDA dev 0, 0.778 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] host pool: 3635 experts of the RAM cache page-locked for the device to copy a prompt chunk's experts from (COLI_CUDA_HOST_POOL=0 off)
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 6605 in VRAM -- 9.3 s
[bench] CUDA ready in 20.19 s; text 79393 tokens; lengths 8192,7168,6144,5120,4096,3072,2048; gen 128; each length from an empty state
[qwen36] KV cache: float16, 0.16 GiB at 8320 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 2048 rows per prompt chunk (286 KiB a row, 1.7 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[bench] peak RSS 19.92 GB
[qtier] host pool: 3635 experts (5.99 GB of the RAM cache) page-locked for the device's copies
--- steps sha256 ec7f21c99618bc392719f73e9359c4b3296d44338e198cc61758e3e36dc464f8 (21819 bytes)

=== run 6 of 8: v1-run3 (v1) ===
--- thermal-before: 2026-10-28 16:34:54 | temp=62 sm=300 reasons=0x1 | swap_used=0.00M swapouts=0
--- thermal-after: 2026-10-28 16:38:06 | temp=75 sm=585 reasons=0x0 | swap_used=0.00M swapouts=0
--- csv
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
8192,8192,9.154,894.88,128,65.5,41.001,0,0.00,52dde184018ab698,e54687bacc388530,2.41221,0.484
7168,7168,34.691,206.62,128,24.3,43.316,0,0.00,e8ae7cfcd208897e,05f26324095f3256,1.98708,0.523
6144,6144,29.647,207.24,128,27.1,41.831,0,0.00,125e04ca6cf235a9,54bf2b608342c377,1.28047,0.688
5120,5120,24.900,205.62,128,24.0,46.836,0,0.00,1ea6ccdccf93b510,3eab0ef0e0213c20,2.20121,0.547
4096,4096,19.779,207.09,128,24.8,46.269,0,0.00,75ddcc777c118ba5,45601e6c456735d4,1.53093,0.602
3072,3072,15.042,204.22,128,21.5,50.253,0,0.00,d1cd4e31a2aa4070,52c1fbb2f3747d50,2.25691,0.500
2048,2048,9.849,207.95,128,20.2,51.020,0,0.00,5f101036c23d0b53,d6661a686430b1fa,2.40945,0.477
--- stderr
[bench] Q36_MAXT=8384 (the longest length and the generated tokens)
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[OMP] qwen36: 4 physical-core threads instead of 8 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8384 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.6s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: Tesla T4, 15.6 GB VRAM, sm_75
[qtier] dev 0: 14.5 GB free, allowance 13.4 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 11.64 GB left for experts
[qtier] dev 0: budget 11.64 GB for experts (~6605 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.409 ms on CUDA dev 0, 0.769 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] host pool: 3635 experts of the RAM cache page-locked for the device to copy a prompt chunk's experts from (COLI_CUDA_HOST_POOL=0 off)
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 6605 in VRAM -- 9.2 s
[bench] CUDA ready in 20.02 s; text 79393 tokens; lengths 8192,7168,6144,5120,4096,3072,2048; gen 128; each length from an empty state
[qwen36] KV cache: float16, 0.16 GiB at 8320 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 2048 rows per prompt chunk (286 KiB a row, 1.7 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[bench] peak RSS 19.92 GB
[qtier] host pool: 3635 experts (5.99 GB of the RAM cache) page-locked for the device's copies
--- steps sha256 0191776caea10d79909eac197df9049251901e1a879cfd5541e59665ea31b853 (21817 bytes)

=== run 7 of 8: v1-run4 (v1) ===
--- thermal-before: 2026-10-28 16:39:16 | temp=62 sm=300 reasons=0x1 | swap_used=0.00M swapouts=0
--- thermal-after: 2026-10-28 16:42:28 | temp=75 sm=585 reasons=0x0 | swap_used=0.00M swapouts=0
--- csv
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
8192,8192,9.110,899.24,128,67.0,41.198,0,0.00,52dde184018ab698,e54687bacc388530,2.41221,0.484
7168,7168,34.655,206.84,128,24.5,43.469,0,0.00,e8ae7cfcd208897e,05f26324095f3256,1.98708,0.523
6144,6144,29.705,206.83,128,26.9,41.456,0,0.00,125e04ca6cf235a9,54bf2b608342c377,1.28047,0.688
5120,5120,25.011,204.71,128,24.4,46.680,0,0.00,1ea6ccdccf93b510,3eab0ef0e0213c20,2.20121,0.547
4096,4096,19.860,206.24,128,25.4,46.180,0,0.00,e335539488a6a4fd,45601e6c456735d4,1.53059,0.602
3072,3072,15.142,202.88,128,21.7,50.166,0,0.00,d1cd4e31a2aa4070,52c1fbb2f3747d50,2.25691,0.500
2048,2048,9.913,206.60,128,20.7,50.883,0,0.00,5f101036c23d0b53,d6661a686430b1fa,2.40945,0.477
--- stderr
[bench] Q36_MAXT=8384 (the longest length and the generated tokens)
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[OMP] qwen36: 4 physical-core threads instead of 8 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8384 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.5s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: Tesla T4, 15.6 GB VRAM, sm_75
[qtier] dev 0: 14.5 GB free, allowance 13.4 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 11.64 GB left for experts
[qtier] dev 0: budget 11.64 GB for experts (~6605 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.444 ms on CUDA dev 0, 0.794 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] host pool: 3635 experts of the RAM cache page-locked for the device to copy a prompt chunk's experts from (COLI_CUDA_HOST_POOL=0 off)
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 6605 in VRAM -- 9.2 s
[bench] CUDA ready in 19.90 s; text 79393 tokens; lengths 8192,7168,6144,5120,4096,3072,2048; gen 128; each length from an empty state
[qwen36] KV cache: float16, 0.16 GiB at 8320 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 2048 rows per prompt chunk (286 KiB a row, 1.7 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[bench] peak RSS 19.92 GB
[qtier] host pool: 3635 experts (5.99 GB of the RAM cache) page-locked for the device's copies
--- steps sha256 f881c636bce6684c481e7fa604507f0932c4e484269720984f398ae5ae3dc358 (21817 bytes)

=== run 8 of 8: v0-run4 (v0) ===
--- thermal-before: 2026-10-28 16:43:38 | temp=63 sm=300 reasons=0x1 | swap_used=0.00M swapouts=0
--- thermal-after: 2026-10-28 16:47:04 | temp=74 sm=585 reasons=0x0 | swap_used=0.00M swapouts=0
--- csv
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
8192,8192,13.794,593.88,128,66.4,41.033,0,0.00,333543ed22bfca98,e54687bacc388530,2.42150,0.484
7168,7168,38.491,186.23,128,25.7,43.414,0,0.00,3b6b2bb4e1098947,05f26324095f3256,1.99007,0.531
6144,6144,32.247,190.53,128,27.2,41.592,0,0.00,1bf4892f8a150aed,54bf2b608342c377,1.28508,0.680
5120,5120,26.710,191.69,128,24.3,46.861,0,0.00,7f9012b6891080ec,3eab0ef0e0213c20,2.19772,0.539
4096,4096,20.906,195.93,128,25.6,46.453,0,0.00,b47ba1322382be7d,45601e6c456735d4,1.53045,0.609
3072,3072,15.543,197.64,128,22.0,50.338,0,0.00,ec44bf40f77b4480,52c1fbb2f3747d50,2.25511,0.492
2048,2048,10.114,202.49,128,20.8,51.009,0,0.00,7ddd20f48fbacd5c,d6661a686430b1fa,2.40315,0.484
--- stderr
[bench] Q36_MAXT=8384 (the longest length and the generated tokens)
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[OMP] qwen36: 4 physical-core threads instead of 8 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8384 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.6s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: Tesla T4, 15.6 GB VRAM, sm_75
[qtier] dev 0: 14.5 GB free, allowance 13.4 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 11.64 GB left for experts
[qtier] dev 0: budget 11.64 GB for experts (~6605 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.385 ms on CUDA dev 0, 0.796 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] host pool: 3635 experts of the RAM cache page-locked for the device to copy a prompt chunk's experts from (COLI_CUDA_HOST_POOL=0 off)
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 6605 in VRAM -- 9.1 s
[bench] CUDA ready in 19.85 s; text 79393 tokens; lengths 8192,7168,6144,5120,4096,3072,2048; gen 128; each length from an empty state
[qwen36] KV cache: float16, 0.16 GiB at 8320 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 2048 rows per prompt chunk (286 KiB a row, 1.7 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[bench] peak RSS 19.92 GB
[qtier] host pool: 3635 experts (5.99 GB of the RAM cache) page-locked for the device's copies
--- steps sha256 62aabc12ce1a552dc2f20f3ef043560e6f8ceab26236dd81bc5d52c8b0b04b81 (21819 bytes)

=== summary.md ===
# bench: scala-giu

- bench 1cd0e3d05691; engine qwen36; Linux (Intel(R) Xeon(R) CPU @ 2.00GHz, Tesla T4); 2026-10-28 16:06
- model q36; text promessi_sposi.txt (sha256 f53e0d80cb2d4492); lengths 8192,7168,6144,5120,4096,3072,2048; gen 128; teacher-forced
- runs: a warm-up of v0 (does not count), then 4 per version, the order turning at every round (ABBA); before every run: wait until cool (at most 900 s), then 60 s of rest

## v0 a: 4 runs (median, min–max)

| tokens | new | prompt tok/s | first step ms | decode tok/s | experts read | reads' s | nll | top1 | fingerprint |
|---|---|---|---|---|---|---|---|---|---|
| 8192 | 8192 | 593.79 (593.46–595.39) | 66 (66–68) | 41.041 (40.903–41.149) | 0 | 0.00 | 2.4215 | 0.484 | 333543ed22bfca98 |
| 7168 | 7168 | 186.16 (185.61–186.37) | 25 (25–26) | 43.416 (43.390–43.619) | 0 | 0.00 | 1.9901 | 0.531 | 3b6b2bb4e1098947 |
| 6144 | 6144 | 190.50 (190.40–190.69) | 27 (27–28) | 41.441 (41.331–41.592) | 0 | 0.00 | 1.2851 | 0.680 | 1bf4892f8a150aed |
| 5120 | 5120 | 191.08 (190.73–191.69) | 24 (24–25) | 46.770 (46.634–46.861) | 0 | 0.00 | 2.1977 | 0.539 | DIFFERENT between runs: ['7f9012b6891080ec', 'b0bfefe493bd9c1f', 'cb72d4307fe145ad'] |
| 4096 | 4096 | 195.33 (194.83–195.93) | 25 (25–26) | 46.191 (45.986–46.453) | 0 | 0.00 | 1.5305 | 0.609 | DIFFERENT between runs: ['923ef5de07ebbe13', 'b47ba1322382be7d'] |
| 3072 | 3072 | 197.03 (196.49–197.64) | 22 (21–22) | 50.266 (50.057–50.441) | 0 | 0.00 | 2.2549 | 0.492 | DIFFERENT between runs: ['ec44bf40f77b4480', 'f412eedc955e5165', 'f6ec3d5af9818813', 'f7dfee4b18e9112b'] |
| 2048 | 2048 | 202.25 (201.73–202.51) | 21 (20–21) | 50.866 (50.790–51.009) | 0 | 0.00 | 2.4032 | 0.484 | DIFFERENT between runs: ['154b852aa11aa7cf', '7ddd20f48fbacd5c', 'b7168dd7af226bf1'] |

peak memory 20398 MiB, model ready in 20.01 s (medians)

decode by steps (step 1 gives the second new word; medians of 4 runs):

| tokens | steps | experts read | decode tok/s | reads' s |
|---|---|---|---|---|
| 8192 | 2 to 32 | 0 | 37.300 | 0.000 |
| 8192 | 33 to 64 | 0 | 42.472 | 0.000 |
| 8192 | 65 to 128 | 0 | 42.394 | 0.000 |
| 8192 | 2 to 128 | 0 | 41.041 | 0.000 |
| 7168 | 2 to 32 | 0 | 42.761 | 0.000 |
| 7168 | 33 to 64 | 0 | 43.983 | 0.000 |
| 7168 | 65 to 128 | 0 | 43.546 | 0.000 |
| 7168 | 2 to 128 | 0 | 43.416 | 0.000 |
| 6144 | 2 to 32 | 0 | 39.325 | 0.000 |
| 6144 | 33 to 64 | 0 | 41.879 | 0.000 |
| 6144 | 65 to 128 | 0 | 42.440 | 0.000 |
| 6144 | 2 to 128 | 0 | 41.441 | 0.000 |
| 5120 | 2 to 32 | 0 | 44.122 | 0.000 |
| 5120 | 33 to 64 | 0 | 48.414 | 0.000 |
| 5120 | 65 to 128 | 0 | 47.297 | 0.000 |
| 5120 | 2 to 128 | 0 | 46.769 | 0.000 |
| 4096 | 2 to 32 | 0 | 45.194 | 0.000 |
| 4096 | 33 to 64 | 0 | 45.774 | 0.000 |
| 4096 | 65 to 128 | 0 | 46.906 | 0.000 |
| 4096 | 2 to 128 | 0 | 46.191 | 0.000 |
| 3072 | 2 to 32 | 0 | 50.471 | 0.000 |
| 3072 | 33 to 64 | 0 | 49.971 | 0.000 |
| 3072 | 65 to 128 | 0 | 50.315 | 0.000 |
| 3072 | 2 to 128 | 0 | 50.266 | 0.000 |
| 2048 | 2 to 32 | 0 | 51.008 | 0.000 |
| 2048 | 33 to 64 | 0 | 50.202 | 0.000 |
| 2048 | 65 to 128 | 0 | 51.138 | 0.000 |
| 2048 | 2 to 128 | 0 | 50.866 | 0.000 |

## v1 b: 4 runs (median, min–max)

| tokens | new | prompt tok/s | first step ms | decode tok/s | experts read | reads' s | nll | top1 | fingerprint |
|---|---|---|---|---|---|---|---|---|---|
| 8192 | 8192 | 898.76 (894.88–899.67) | 66 (66–67) | 41.070 (40.801–41.198) | 0 | 0.00 | 2.4122 | 0.484 | 52dde184018ab698 |
| 7168 | 7168 | 206.73 (206.38–207.14) | 24 (24–25) | 43.392 (43.269–43.486) | 0 | 0.00 | 1.9871 | 0.523 | e8ae7cfcd208897e |
| 6144 | 6144 | 206.88 (206.40–207.24) | 27 (27–27) | 41.442 (41.231–41.831) | 0 | 0.00 | 1.2805 | 0.688 | 125e04ca6cf235a9 |
| 5120 | 5120 | 204.71 (204.64–205.62) | 24 (24–24) | 46.752 (46.672–46.836) | 0 | 0.00 | 2.2013 | 0.547 | DIFFERENT between runs: ['1ea6ccdccf93b510', 'e7943b93161c07ac'] |
| 4096 | 4096 | 206.68 (206.24–207.09) | 25 (25–25) | 46.258 (46.180–46.391) | 0 | 0.00 | 1.5309 | 0.602 | DIFFERENT between runs: ['75ddcc777c118ba5', 'e335539488a6a4fd'] |
| 3072 | 3072 | 203.91 (202.88–204.22) | 22 (21–22) | 50.196 (50.085–50.253) | 0 | 0.00 | 2.2569 | 0.500 | d1cd4e31a2aa4070 |
| 2048 | 2048 | 206.71 (206.58–207.95) | 21 (20–21) | 50.958 (50.883–51.041) | 0 | 0.00 | 2.4095 | 0.477 | 5f101036c23d0b53 |

peak memory 20398 MiB, model ready in 20.08 s (medians)

decode by steps (step 1 gives the second new word; medians of 4 runs):

| tokens | steps | experts read | decode tok/s | reads' s |
|---|---|---|---|---|
| 8192 | 2 to 32 | 0 | 37.195 | 0.000 |
| 8192 | 33 to 64 | 0 | 42.628 | 0.000 |
| 8192 | 65 to 128 | 0 | 42.423 | 0.000 |
| 8192 | 2 to 128 | 0 | 41.071 | 0.000 |
| 7168 | 2 to 32 | 0 | 42.692 | 0.000 |
| 7168 | 33 to 64 | 0 | 43.882 | 0.000 |
| 7168 | 65 to 128 | 0 | 43.447 | 0.000 |
| 7168 | 2 to 128 | 0 | 43.393 | 0.000 |
| 6144 | 2 to 32 | 0 | 39.175 | 0.000 |
| 6144 | 33 to 64 | 0 | 41.982 | 0.000 |
| 6144 | 65 to 128 | 0 | 42.356 | 0.000 |
| 6144 | 2 to 128 | 0 | 41.442 | 0.000 |
| 5120 | 2 to 32 | 0 | 44.118 | 0.000 |
| 5120 | 33 to 64 | 0 | 48.522 | 0.000 |
| 5120 | 65 to 128 | 0 | 47.303 | 0.000 |
| 5120 | 2 to 128 | 0 | 46.752 | 0.000 |
| 4096 | 2 to 32 | 0 | 45.281 | 0.000 |
| 4096 | 33 to 64 | 0 | 45.752 | 0.000 |
| 4096 | 65 to 128 | 0 | 47.075 | 0.000 |
| 4096 | 2 to 128 | 0 | 46.257 | 0.000 |
| 3072 | 2 to 32 | 0 | 50.611 | 0.000 |
| 3072 | 33 to 64 | 0 | 49.814 | 0.000 |
| 3072 | 65 to 128 | 0 | 50.204 | 0.000 |
| 3072 | 2 to 128 | 0 | 50.196 | 0.000 |
| 2048 | 2 to 32 | 0 | 51.084 | 0.000 |
| 2048 | 33 to 64 | 0 | 50.345 | 0.000 |
| 2048 | 65 to 128 | 0 | 51.209 | 0.000 |
| 2048 | 2 to 128 | 0 | 50.958 | 0.000 |

## v1 b against v0 a (medians; positive = faster)

| tokens | prompt | first step | decode | results |
|---|---|---|---|---|
| 8192 | +51.4%, every run faster | -0.2%, ranges overlap | +0.1%, ranges overlap | same words, logits differ; nll 2.4215 → 2.4122, top1 0.484 → 0.484 |
| 7168 | +11.1%, every run faster | +2.0%, ranges overlap | -0.1%, ranges overlap | same words, logits differ; nll 1.9901 → 1.9871, top1 0.531 → 0.523 |
| 6144 | +8.6%, every run faster | +0.6%, ranges overlap | +0.0%, ranges overlap | same words, logits differ; nll 1.2851 → 1.2805, top1 0.680 → 0.688 |
| 5120 | +7.1%, every run faster | +0.6%, ranges overlap | -0.0%, ranges overlap | same words, logits differ; nll 2.1977 → 2.2013, top1 0.539 → 0.547 |
| 4096 | +5.8%, every run faster | +1.2%, ranges overlap | +0.1%, ranges overlap | same words, logits differ; nll 1.5305 → 1.5309, top1 0.609 → 0.602 |
| 3072 | +3.5%, every run faster | +0.5%, ranges overlap | -0.1%, ranges overlap | same words, logits differ; nll 2.2549 → 2.2569, top1 0.492 → 0.500 |
| 2048 | +2.2%, every run faster | +0.2%, ranges overlap | +0.2%, ranges overlap | same words, logits differ; nll 2.4032 → 2.4095, top1 0.484 → 0.477 |

| tokens | steps | experts read | decode | reads' s |
|---|---|---|---|---|
| 8192 | 2 to 32 | 0 → 0 | -0.3%, ranges overlap | 0.000 → 0.000 s |
| 8192 | 33 to 64 | 0 → 0 | +0.4%, ranges overlap | 0.000 → 0.000 s |
| 8192 | 65 to 128 | 0 → 0 | +0.1%, ranges overlap | 0.000 → 0.000 s |
| 8192 | 2 to 128 | 0 → 0 | +0.1%, ranges overlap | 0.000 → 0.000 s |
| 7168 | 2 to 32 | 0 → 0 | -0.2%, ranges overlap | 0.000 → 0.000 s |
| 7168 | 33 to 64 | 0 → 0 | -0.2%, ranges overlap | 0.000 → 0.000 s |
| 7168 | 65 to 128 | 0 → 0 | -0.2%, ranges overlap | 0.000 → 0.000 s |
| 7168 | 2 to 128 | 0 → 0 | -0.1%, ranges overlap | 0.000 → 0.000 s |
| 6144 | 2 to 32 | 0 → 0 | -0.4%, ranges overlap | 0.000 → 0.000 s |
| 6144 | 33 to 64 | 0 → 0 | +0.2%, ranges overlap | 0.000 → 0.000 s |
| 6144 | 65 to 128 | 0 → 0 | -0.2%, ranges overlap | 0.000 → 0.000 s |
| 6144 | 2 to 128 | 0 → 0 | +0.0%, ranges overlap | 0.000 → 0.000 s |
| 5120 | 2 to 32 | 0 → 0 | -0.0%, ranges overlap | 0.000 → 0.000 s |
| 5120 | 33 to 64 | 0 → 0 | +0.2%, ranges overlap | 0.000 → 0.000 s |
| 5120 | 65 to 128 | 0 → 0 | +0.0%, ranges overlap | 0.000 → 0.000 s |
| 5120 | 2 to 128 | 0 → 0 | -0.0%, ranges overlap | 0.000 → 0.000 s |
| 4096 | 2 to 32 | 0 → 0 | +0.2%, ranges overlap | 0.000 → 0.000 s |
| 4096 | 33 to 64 | 0 → 0 | -0.0%, ranges overlap | 0.000 → 0.000 s |
| 4096 | 65 to 128 | 0 → 0 | +0.4%, ranges overlap | 0.000 → 0.000 s |
| 4096 | 2 to 128 | 0 → 0 | +0.1%, ranges overlap | 0.000 → 0.000 s |
| 3072 | 2 to 32 | 0 → 0 | +0.3%, ranges overlap | 0.000 → 0.000 s |
| 3072 | 33 to 64 | 0 → 0 | -0.3%, ranges overlap | 0.000 → 0.000 s |
| 3072 | 65 to 128 | 0 → 0 | -0.2%, ranges overlap | 0.000 → 0.000 s |
| 3072 | 2 to 128 | 0 → 0 | -0.1%, ranges overlap | 0.000 → 0.000 s |
| 2048 | 2 to 32 | 0 → 0 | +0.1%, ranges overlap | 0.000 → 0.000 s |
| 2048 | 33 to 64 | 0 → 0 | +0.3%, ranges overlap | 0.000 → 0.000 s |
| 2048 | 65 to 128 | 0 → 0 | +0.1%, ranges overlap | 0.000 → 0.000 s |
| 2048 | 2 to 128 | 0 → 0 | +0.2%, ranges overlap | 0.000 → 0.000 s |

warm-up (does not count): 8192 tokens: prompt 594.95 tok/s, decode 41.161 tok/s; 7168 tokens: prompt 186.11 tok/s, decode 43.563 tok/s; 6144 tokens: prompt 190.44 tok/s, decode 41.565 tok/s; 5120 tokens: prompt 190.70 tok/s, decode 46.695 tok/s; 4096 tokens: prompt 195.22 tok/s, decode 46.198 tok/s; 3072 tokens: prompt 196.40 tok/s, decode 50.347 tok/s; 2048 tokens: prompt 202.38 tok/s, decode 51.180 tok/s
runs not cool: none
pages written to swap during the measurement: 0, 0 of them in the runs that count
