# qwen36 CUDA chain, the MTP head's adaptive draft list, prose, the most likely token, one draft a step, warm counts: raw evidence (2026-10-26, UTC)
# Google Colab L4 VM: Intel(R) Xeon(R) CPU @ 2.20GHz (6 cores / 12 threads, 38.5 MiB L3) + NVIDIA L4 (23034 MiB, sm_89), 52 GiB; Ubuntu 24.04.4 LTS, Linux 6.6.122+ x86_64 (Google Colab), CUDA 13.0 (nvcc V13.0.88), driver 580.82.07, gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0.
# Model: Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64 (CUDA_EXPERT_GB=auto: all 10240 experts in VRAM), with its
#   MTP head (mtp.* of Qwen/Qwen3.6-35B-A3B, convert_qwen36.py --mtp-only); Q36_MTP=1 Q36_MTP_DRAFTS=1.
# v0 = 16d9f670: origin/dev 7bbea341 + the CUDA chain's small copies + the MTP draft picked on the
#   device, the changes this one builds on; v1 = 16d9f670 + this change (this commit's code).
#   Harness: bench.c beside this file (sha256
#   edbe7ddcc146af34d10c3d8a871fb1d3b93404537e374f9bc1504cf2c6886c17), built against each version's c/ with:
#   cd c && make qwen36_tier.o backend_cuda.o cuda_chain.o cu_xb.o cu_tier.o CUDA=1 CUDA_ARCH=sm_89 && gcc -O3 -march=native -fopenmp -pthread -Wall -Wextra -Wno-unused-parameter -Wno-misleading-indentation -Wno-unused-function -DCOLI_CUDA -I. <bench.c> qwen36_tier.o backend_cuda.o cuda_chain.o cu_xb.o cu_tier.o -o qwen36-bench -lm -fopenmp -pthread -L/usr/local/cuda/lib64 -Wl,-rpath,/usr/local/cuda/lib64 -lcudart -lcublasLt -lcublas -lstdc++
#   (Its comments name the driver and scripts it came with; the runs below need only this file.)
# Counts: both versions were given the same counts file, coli_vocab beside this file (sha256
#   9f32c38c5b516c41fc61f66c1b4c9cbadb2b86b847a0ce733cb5ade1a93843c1; the same before and after the runs), with
#   COLI_VOCAB_USAGE=<it> USAGE_SAVE=0; v0 does not read it. This change's engine wrote it on the same VM
#   before the runs, as a user's earlier runs would (counts.txt beside this file).
# Text: promessi_sposi.txt, speed-bench/promessi_sposi.txt of antirez's ds4 at commit 0aaea5a (sha256
#   f53e0d80cb2d4492d24ebd63c7000c397b16ae70f9bf09b3763e5d8323ec209f); for each length its first N tokens from an empty state, then 32 decode
#   steps of the most likely token, never an end of text. One process a run, the two lengths in order.
# A warm-up run of v0 (not counted), then v0 v1 v1 v0 v0 v1 v1 v0, each once the card was under 70 C
#   with no thermal clock event, then 60 s of rest. Every run's CSV and stderr in full, in the order they ran;
#   decode tok/s = gen_steady_tps (steps 2 to 32), the summary at the end.

===== setup
bench 482e1711d589; engine qwen36; Linux (Intel(R) Xeon(R) CPU @ 2.20GHz, NVIDIA L4); 2026-10-26 03:00
model /content/q36mtp; text promessi_sposi.txt (sha256 f53e0d80cb2d4492); lengths 64,512; gen 32; free (most likely token, never the end)
runs: a warm-up of v0 (does not count), then 4 per version, the order turning at every round (ABBA); before every run: wait until cool (at most 900 s), then 60 s of rest
v0 c4: /content/bin-c4/qwen36-bench (commit 16d9f6703aeb716135d5df22092e742be9cc9b5b; harness qwen36_bench.c edbe7ddcc146; CUDA); env: COLI_CUDA=1 COLI_GPUS=0 CUDA_EXPERT_GB=auto OMP_WAIT_POLICY=ACTIVE OMP_PROC_BIND=close Q36_MTP=1 Q36_MTP_DRAFTS=1 COLI_VOCAB_USAGE=/content/warm/coli_vocab USAGE_SAVE=0
   command: env -i PATH=/usr/bin:/bin HOME=/root LD_LIBRARY_PATH=/usr/lib64-nvidia COLI_CUDA=1 COLI_GPUS=0 CUDA_EXPERT_GB=auto OMP_WAIT_POLICY=ACTIVE OMP_PROC_BIND=close Q36_MTP=1 Q36_MTP_DRAFTS=1 COLI_VOCAB_USAGE=/content/warm/coli_vocab USAGE_SAVE=0 /content/bin-c4/qwen36-bench --model /content/q36mtp --text /content/bench/promessi_sposi.txt --lengths 64,512 --gen 32 --steps /content/c6caldo-prosa-libera-1/v0-runN.steps
v1 c6: /content/bin-c6/qwen36-bench (commit 8aaf4d788308ddac1e0e9e63ea507e152fda9cc5; harness qwen36_bench.c edbe7ddcc146; CUDA); env: COLI_CUDA=1 COLI_GPUS=0 CUDA_EXPERT_GB=auto OMP_WAIT_POLICY=ACTIVE OMP_PROC_BIND=close Q36_MTP=1 Q36_MTP_DRAFTS=1 COLI_VOCAB_USAGE=/content/warm/coli_vocab USAGE_SAVE=0
   command: env -i PATH=/usr/bin:/bin HOME=/root LD_LIBRARY_PATH=/usr/lib64-nvidia COLI_CUDA=1 COLI_GPUS=0 CUDA_EXPERT_GB=auto OMP_WAIT_POLICY=ACTIVE OMP_PROC_BIND=close Q36_MTP=1 Q36_MTP_DRAFTS=1 COLI_VOCAB_USAGE=/content/warm/coli_vocab USAGE_SAVE=0 /content/bin-c6/qwen36-bench --model /content/q36mtp --text /content/bench/promessi_sposi.txt --lengths 64,512 --gen 32 --steps /content/c6caldo-prosa-libera-1/v1-runN.steps

===== warmup.csv (sha256 3a8d6c616bd7235b8828e6b90cb1409e5572498ff6d71698a06698cdff4378c1)
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
64,64,0.608,105.31,32,26.7,83.688,0,0.00,d9d3351b46e6980d,1190b702cc495428,0.00000,0.000
512,512,0.327,1564.10,32,20.6,71.900,0,0.00,14c3a19d195d62a3,f57099df7a2d1304,0.00000,0.000
===== warmup.err (sha256 f56505be9126c9b37fd1110385e4e8f42a532debc60041e4ae61b2f5eab28d4b)
[tok] loaded 248044 pieces (max id 248069) from /content/q36mtp/tokenizer.json
[OMP] qwen36: 6 physical-core threads instead of 12 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8192 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36mtp/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.2s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: NVIDIA L4, 23.7 GB VRAM, sm_89
[qtier] dev 0: 21.8 GB free, allowance 20.8 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 19.02 GB left for experts
[qtier] dev 0: budget 19.02 GB for experts (~10794 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.113 ms on CUDA dev 0, 0.415 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 10240 in VRAM -- 14.4 s
[mtp] Qwen3.6 checkpoint head: 256 experts
[bench] CUDA ready in 25.96 s; text 5871 tokens; lengths 64,512; gen 32; each length from an empty state
[qwen36] KV cache: float16, 0.00 GiB at 96 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 4096 rows per prompt chunk (286 KiB a row, 3.1 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[mtp] the head runs on the CUDA chain's device, its experts in the frame (format 4)
[qwen36 mtp] 64 tokens: 1.60 tokens/forward (20 forwards per 32 tokens) | acceptance 66.7% (12/18 drafts) | gate T(1) 14.9 ms, rows 2:1.35, rho 0.50, acceptance by position 0.69 0.60 0.50 0.50 0.50
[mtp] head rows 18 (2.44 ms each), fed rows 83 (K/V only, 0.05 ms each), drafts 18
[qwen36 spec gate] 64 tokens: T(1) 14.9 ms, rows 2:1.35, rho 0.50, acceptance by position 0.50 0.50 0.50 0.50 0.50
[qwen36 lookup] 64 tokens: 1.60 tokens/forward (20 forwards per 32 tokens) | acceptance 0.0% (0/0 drafts in 18 verifies) | gate on, 0 declined, 0 probes
[qwen36 mtp] 512 tokens: 1.39 tokens/forward (23 forwards per 32 tokens) | acceptance 40.9% (9/22 drafts) | gate T(1) 15.0 ms, rows 2:1.32, rho 0.50, acceptance by position 0.43 0.60 0.50 0.50 0.50
[mtp] head rows 40 (2.44 ms each), fed rows 617 (K/V only, 0.03 ms each), drafts 40
[qwen36 spec gate] 512 tokens: T(1) 15.0 ms, rows 2:1.32, rho 0.50, acceptance by position 0.50 0.50 0.50 0.50 0.50
[qwen36 lookup] 512 tokens: 1.39 tokens/forward (23 forwards per 32 tokens) | acceptance 0.0% (0/0 drafts in 22 verifies) | gate on, 0 declined, 0 probes
[bench] peak RSS 20.03 GB

===== v0-run1.csv (sha256 e87e21c6652da777b203199467ab9d648e5e9a7520683a76ff32538c52b870c5)
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
64,64,0.605,105.87,32,25.9,83.118,0,0.00,d9d3351b46e6980d,1190b702cc495428,0.00000,0.000
512,512,0.323,1583.09,32,20.7,71.465,0,0.00,14c3a19d195d62a3,f57099df7a2d1304,0.00000,0.000
===== v0-run1.err (sha256 f471891626aaad2d24348f9be90dc0f26fbc32f5d6bc1f851ab8fbfc7ccb069a)
[tok] loaded 248044 pieces (max id 248069) from /content/q36mtp/tokenizer.json
[OMP] qwen36: 6 physical-core threads instead of 12 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8192 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36mtp/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.2s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: NVIDIA L4, 23.7 GB VRAM, sm_89
[qtier] dev 0: 21.8 GB free, allowance 20.8 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 19.02 GB left for experts
[qtier] dev 0: budget 19.02 GB for experts (~10794 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.112 ms on CUDA dev 0, 0.416 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 10240 in VRAM -- 14.5 s
[mtp] Qwen3.6 checkpoint head: 256 experts
[bench] CUDA ready in 25.97 s; text 5871 tokens; lengths 64,512; gen 32; each length from an empty state
[qwen36] KV cache: float16, 0.00 GiB at 96 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 4096 rows per prompt chunk (286 KiB a row, 3.1 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[mtp] the head runs on the CUDA chain's device, its experts in the frame (format 4)
[qwen36 mtp] 64 tokens: 1.60 tokens/forward (20 forwards per 32 tokens) | acceptance 66.7% (12/18 drafts) | gate T(1) 15.0 ms, rows 2:1.35, rho 0.50, acceptance by position 0.69 0.60 0.50 0.50 0.50
[mtp] head rows 18 (2.45 ms each), fed rows 83 (K/V only, 0.05 ms each), drafts 18
[qwen36 spec gate] 64 tokens: T(1) 15.0 ms, rows 2:1.35, rho 0.50, acceptance by position 0.50 0.50 0.50 0.50 0.50
[qwen36 lookup] 64 tokens: 1.60 tokens/forward (20 forwards per 32 tokens) | acceptance 0.0% (0/0 drafts in 18 verifies) | gate on, 0 declined, 0 probes
[qwen36 mtp] 512 tokens: 1.39 tokens/forward (23 forwards per 32 tokens) | acceptance 40.9% (9/22 drafts) | gate T(1) 14.8 ms, rows 2:1.34, rho 0.50, acceptance by position 0.43 0.60 0.50 0.50 0.50
[mtp] head rows 40 (2.44 ms each), fed rows 617 (K/V only, 0.03 ms each), drafts 40
[qwen36 spec gate] 512 tokens: T(1) 14.8 ms, rows 2:1.34, rho 0.50, acceptance by position 0.50 0.50 0.50 0.50 0.50
[qwen36 lookup] 512 tokens: 1.39 tokens/forward (23 forwards per 32 tokens) | acceptance 0.0% (0/0 drafts in 22 verifies) | gate on, 0 declined, 0 probes
[bench] peak RSS 20.03 GB

===== v1-run1.csv (sha256 5fd681e655642cbeef6b36e8878265bbf542b9d3a8d75bbef828ee75f9e16c6e)
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
64,64,0.617,103.68,32,24.3,96.822,0,0.00,d9d3351b46e6980d,1190b702cc495428,0.00000,0.000
512,512,0.324,1580.04,32,18.8,79.295,0,0.00,14c3a19d195d62a3,f57099df7a2d1304,0.00000,0.000
===== v1-run1.err (sha256 0ecd8338c59c6b19c8eaab80b5a627d9dbbeefd5dc16933ef335969d9e330ec9)
[tok] loaded 248044 pieces (max id 248069) from /content/q36mtp/tokenizer.json
[OMP] qwen36: 6 physical-core threads instead of 12 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8192 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36mtp/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.2s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: NVIDIA L4, 23.7 GB VRAM, sm_89
[qtier] dev 0: 21.8 GB free, allowance 20.8 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 19.02 GB left for experts
[qtier] dev 0: budget 19.02 GB for experts (~10794 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.113 ms on CUDA dev 0, 0.416 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 10240 in VRAM -- 14.5 s
[mtp] draft vocabulary: top 16000 + 3/4 rule + conversation, counts from /content/warm/coli_vocab (1588 tokens, 5346 distinct)
[mtp] Qwen3.6 checkpoint head: 256 experts
[bench] CUDA ready in 26.03 s; text 5871 tokens; lengths 64,512; gen 32; each length from an empty state
[qwen36] KV cache: float16, 0.00 GiB at 96 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 4096 rows per prompt chunk (286 KiB a row, 3.1 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[mtp] the head runs on the CUDA chain's device, its experts in the frame (format 4), drafts score an adaptive vocabulary
[qwen36 mtp] 64 tokens: 1.68 tokens/forward (19 forwards per 32 tokens) | acceptance 76.5% (13/17 drafts) | gate T(1) 14.8 ms, rows 2:1.22, rho 0.50, acceptance by position 0.77 0.60 0.50 0.50 0.50
[mtp] head rows 17 (0.55 ms each), fed rows 82 (K/V only, 0.05 ms each), drafts 17
[mtp] draft list 16000 tokens on average, 96.9% of picks inside, 1 ids appended, 0 rebuilds
[qwen36 spec gate] 64 tokens: T(1) 14.8 ms, rows 2:1.22, rho 0.50, acceptance by position 0.50 0.50 0.50 0.50 0.50
[qwen36 lookup] 64 tokens: 1.68 tokens/forward (19 forwards per 32 tokens) | acceptance 0.0% (0/0 drafts in 17 verifies) | gate on, 0 declined, 0 probes
[qwen36 mtp] 512 tokens: 1.39 tokens/forward (23 forwards per 32 tokens) | acceptance 40.9% (9/22 drafts) | gate T(1) 14.8 ms, rows 2:1.20, rho 0.50, acceptance by position 0.43 0.60 0.50 0.50 0.50
[mtp] head rows 39 (0.54 ms each), fed rows 616 (K/V only, 0.03 ms each), drafts 39
[mtp] draft list 16000 tokens on average, 100.0% of picks inside, 0 ids appended, 0 rebuilds
[qwen36 spec gate] 512 tokens: T(1) 14.8 ms, rows 2:1.20, rho 0.50, acceptance by position 0.50 0.50 0.50 0.50 0.50
[qwen36 lookup] 512 tokens: 1.39 tokens/forward (23 forwards per 32 tokens) | acceptance 0.0% (0/0 drafts in 22 verifies) | gate on, 0 declined, 0 probes
[bench] peak RSS 20.03 GB

===== v1-run2.csv (sha256 a19df10522db96195a388c244b23fe737cc345811ddcf8ff1902f77b49385f8d)
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
64,64,0.618,103.51,32,23.9,97.340,0,0.00,d9d3351b46e6980d,1190b702cc495428,0.00000,0.000
512,512,0.319,1603.80,32,18.7,79.836,0,0.00,14c3a19d195d62a3,f57099df7a2d1304,0.00000,0.000
===== v1-run2.err (sha256 4fd137bc8638ba2465ce21be6b7881e2dabb971fb8032d3dcd3d0dc88b3744e5)
[tok] loaded 248044 pieces (max id 248069) from /content/q36mtp/tokenizer.json
[OMP] qwen36: 6 physical-core threads instead of 12 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8192 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36mtp/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.2s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: NVIDIA L4, 23.7 GB VRAM, sm_89
[qtier] dev 0: 21.8 GB free, allowance 20.8 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 19.02 GB left for experts
[qtier] dev 0: budget 19.02 GB for experts (~10794 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.113 ms on CUDA dev 0, 0.415 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 10240 in VRAM -- 14.5 s
[mtp] draft vocabulary: top 16000 + 3/4 rule + conversation, counts from /content/warm/coli_vocab (1588 tokens, 5346 distinct)
[mtp] Qwen3.6 checkpoint head: 256 experts
[bench] CUDA ready in 25.97 s; text 5871 tokens; lengths 64,512; gen 32; each length from an empty state
[qwen36] KV cache: float16, 0.00 GiB at 96 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 4096 rows per prompt chunk (286 KiB a row, 3.1 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[mtp] the head runs on the CUDA chain's device, its experts in the frame (format 4), drafts score an adaptive vocabulary
[qwen36 mtp] 64 tokens: 1.68 tokens/forward (19 forwards per 32 tokens) | acceptance 76.5% (13/17 drafts) | gate T(1) 14.6 ms, rows 2:1.24, rho 0.50, acceptance by position 0.77 0.60 0.50 0.50 0.50
[mtp] head rows 17 (0.54 ms each), fed rows 82 (K/V only, 0.05 ms each), drafts 17
[mtp] draft list 16000 tokens on average, 96.9% of picks inside, 1 ids appended, 0 rebuilds
[qwen36 spec gate] 64 tokens: T(1) 14.6 ms, rows 2:1.24, rho 0.50, acceptance by position 0.50 0.50 0.50 0.50 0.50
[qwen36 lookup] 64 tokens: 1.68 tokens/forward (19 forwards per 32 tokens) | acceptance 0.0% (0/0 drafts in 17 verifies) | gate on, 0 declined, 0 probes
[qwen36 mtp] 512 tokens: 1.39 tokens/forward (23 forwards per 32 tokens) | acceptance 40.9% (9/22 drafts) | gate T(1) 14.6 ms, rows 2:1.21, rho 0.50, acceptance by position 0.43 0.60 0.50 0.50 0.50
[mtp] head rows 39 (0.53 ms each), fed rows 616 (K/V only, 0.03 ms each), drafts 39
[mtp] draft list 16000 tokens on average, 100.0% of picks inside, 0 ids appended, 0 rebuilds
[qwen36 spec gate] 512 tokens: T(1) 14.6 ms, rows 2:1.21, rho 0.50, acceptance by position 0.50 0.50 0.50 0.50 0.50
[qwen36 lookup] 512 tokens: 1.39 tokens/forward (23 forwards per 32 tokens) | acceptance 0.0% (0/0 drafts in 22 verifies) | gate on, 0 declined, 0 probes
[bench] peak RSS 20.06 GB

===== v0-run2.csv (sha256 c6f67a1f2afce96bd12463b44f0f893a613d844802696423745f57c366c77fbc)
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
64,64,0.607,105.38,32,25.8,83.800,0,0.00,d9d3351b46e6980d,1190b702cc495428,0.00000,0.000
512,512,0.321,1592.69,32,20.7,71.695,0,0.00,14c3a19d195d62a3,f57099df7a2d1304,0.00000,0.000
===== v0-run2.err (sha256 317687d327034d7058abcf816ada55b38a2c8cda45a38434ba4c3837757314a1)
[tok] loaded 248044 pieces (max id 248069) from /content/q36mtp/tokenizer.json
[OMP] qwen36: 6 physical-core threads instead of 12 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8192 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36mtp/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.2s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: NVIDIA L4, 23.7 GB VRAM, sm_89
[qtier] dev 0: 21.8 GB free, allowance 20.8 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 19.02 GB left for experts
[qtier] dev 0: budget 19.02 GB for experts (~10794 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.113 ms on CUDA dev 0, 0.417 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 10240 in VRAM -- 14.6 s
[mtp] Qwen3.6 checkpoint head: 256 experts
[bench] CUDA ready in 26.10 s; text 5871 tokens; lengths 64,512; gen 32; each length from an empty state
[qwen36] KV cache: float16, 0.00 GiB at 96 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 4096 rows per prompt chunk (286 KiB a row, 3.1 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[mtp] the head runs on the CUDA chain's device, its experts in the frame (format 4)
[qwen36 mtp] 64 tokens: 1.60 tokens/forward (20 forwards per 32 tokens) | acceptance 66.7% (12/18 drafts) | gate T(1) 14.9 ms, rows 2:1.34, rho 0.50, acceptance by position 0.69 0.60 0.50 0.50 0.50
[mtp] head rows 18 (2.44 ms each), fed rows 83 (K/V only, 0.05 ms each), drafts 18
[qwen36 spec gate] 64 tokens: T(1) 14.9 ms, rows 2:1.34, rho 0.50, acceptance by position 0.50 0.50 0.50 0.50 0.50
[qwen36 lookup] 64 tokens: 1.60 tokens/forward (20 forwards per 32 tokens) | acceptance 0.0% (0/0 drafts in 18 verifies) | gate on, 0 declined, 0 probes
[qwen36 mtp] 512 tokens: 1.39 tokens/forward (23 forwards per 32 tokens) | acceptance 40.9% (9/22 drafts) | gate T(1) 14.7 ms, rows 2:1.34, rho 0.50, acceptance by position 0.43 0.60 0.50 0.50 0.50
[mtp] head rows 40 (2.44 ms each), fed rows 617 (K/V only, 0.03 ms each), drafts 40
[qwen36 spec gate] 512 tokens: T(1) 14.7 ms, rows 2:1.34, rho 0.50, acceptance by position 0.50 0.50 0.50 0.50 0.50
[qwen36 lookup] 512 tokens: 1.39 tokens/forward (23 forwards per 32 tokens) | acceptance 0.0% (0/0 drafts in 22 verifies) | gate on, 0 declined, 0 probes
[bench] peak RSS 20.02 GB

===== v0-run3.csv (sha256 6370143b0858e550331242311a509a77e766824899df0bd6dd80597b3e267632)
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
64,64,0.610,104.90,32,26.1,83.923,0,0.00,d9d3351b46e6980d,1190b702cc495428,0.00000,0.000
512,512,0.321,1594.81,32,20.8,71.702,0,0.00,14c3a19d195d62a3,f57099df7a2d1304,0.00000,0.000
===== v0-run3.err (sha256 707f58317914bdd552c4525e127e8cb30bf919f52f696171555ec1b5fc326043)
[tok] loaded 248044 pieces (max id 248069) from /content/q36mtp/tokenizer.json
[OMP] qwen36: 6 physical-core threads instead of 12 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8192 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36mtp/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.1s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: NVIDIA L4, 23.7 GB VRAM, sm_89
[qtier] dev 0: 21.8 GB free, allowance 20.8 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 19.02 GB left for experts
[qtier] dev 0: budget 19.02 GB for experts (~10794 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.113 ms on CUDA dev 0, 0.414 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 10240 in VRAM -- 14.4 s
[mtp] Qwen3.6 checkpoint head: 256 experts
[bench] CUDA ready in 25.88 s; text 5871 tokens; lengths 64,512; gen 32; each length from an empty state
[qwen36] KV cache: float16, 0.00 GiB at 96 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 4096 rows per prompt chunk (286 KiB a row, 3.1 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[mtp] the head runs on the CUDA chain's device, its experts in the frame (format 4)
[qwen36 mtp] 64 tokens: 1.60 tokens/forward (20 forwards per 32 tokens) | acceptance 66.7% (12/18 drafts) | gate T(1) 14.9 ms, rows 2:1.35, rho 0.50, acceptance by position 0.69 0.60 0.50 0.50 0.50
[mtp] head rows 18 (2.44 ms each), fed rows 83 (K/V only, 0.05 ms each), drafts 18
[qwen36 spec gate] 64 tokens: T(1) 14.9 ms, rows 2:1.35, rho 0.50, acceptance by position 0.50 0.50 0.50 0.50 0.50
[qwen36 lookup] 64 tokens: 1.60 tokens/forward (20 forwards per 32 tokens) | acceptance 0.0% (0/0 drafts in 18 verifies) | gate on, 0 declined, 0 probes
[qwen36 mtp] 512 tokens: 1.39 tokens/forward (23 forwards per 32 tokens) | acceptance 40.9% (9/22 drafts) | gate T(1) 14.7 ms, rows 2:1.34, rho 0.50, acceptance by position 0.43 0.60 0.50 0.50 0.50
[mtp] head rows 40 (2.44 ms each), fed rows 617 (K/V only, 0.03 ms each), drafts 40
[qwen36 spec gate] 512 tokens: T(1) 14.7 ms, rows 2:1.34, rho 0.50, acceptance by position 0.50 0.50 0.50 0.50 0.50
[qwen36 lookup] 512 tokens: 1.39 tokens/forward (23 forwards per 32 tokens) | acceptance 0.0% (0/0 drafts in 22 verifies) | gate on, 0 declined, 0 probes
[bench] peak RSS 20.03 GB

===== v1-run3.csv (sha256 79c495172c3dce25e10e9df625d09e0c2845bcf3bb3fabae0d9646253f83cba5)
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
64,64,0.620,103.27,32,24.0,97.638,0,0.00,d9d3351b46e6980d,1190b702cc495428,0.00000,0.000
512,512,0.317,1617.48,32,18.6,80.510,0,0.00,14c3a19d195d62a3,f57099df7a2d1304,0.00000,0.000
===== v1-run3.err (sha256 cfc4c24c955653aefdb17054fbd82bdba6cb13c9be87d2d0493f2944f0f10b80)
[tok] loaded 248044 pieces (max id 248069) from /content/q36mtp/tokenizer.json
[OMP] qwen36: 6 physical-core threads instead of 12 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8192 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36mtp/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.2s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: NVIDIA L4, 23.7 GB VRAM, sm_89
[qtier] dev 0: 21.8 GB free, allowance 20.8 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 19.02 GB left for experts
[qtier] dev 0: budget 19.02 GB for experts (~10794 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.112 ms on CUDA dev 0, 0.421 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 10240 in VRAM -- 14.5 s
[mtp] draft vocabulary: top 16000 + 3/4 rule + conversation, counts from /content/warm/coli_vocab (1588 tokens, 5346 distinct)
[mtp] Qwen3.6 checkpoint head: 256 experts
[bench] CUDA ready in 25.97 s; text 5871 tokens; lengths 64,512; gen 32; each length from an empty state
[qwen36] KV cache: float16, 0.00 GiB at 96 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 4096 rows per prompt chunk (286 KiB a row, 3.1 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[mtp] the head runs on the CUDA chain's device, its experts in the frame (format 4), drafts score an adaptive vocabulary
[qwen36 mtp] 64 tokens: 1.68 tokens/forward (19 forwards per 32 tokens) | acceptance 76.5% (13/17 drafts) | gate T(1) 14.8 ms, rows 2:1.22, rho 0.50, acceptance by position 0.77 0.60 0.50 0.50 0.50
[mtp] head rows 17 (0.54 ms each), fed rows 82 (K/V only, 0.05 ms each), drafts 17
[mtp] draft list 16000 tokens on average, 96.9% of picks inside, 1 ids appended, 0 rebuilds
[qwen36 spec gate] 64 tokens: T(1) 14.8 ms, rows 2:1.22, rho 0.50, acceptance by position 0.50 0.50 0.50 0.50 0.50
[qwen36 lookup] 64 tokens: 1.68 tokens/forward (19 forwards per 32 tokens) | acceptance 0.0% (0/0 drafts in 17 verifies) | gate on, 0 declined, 0 probes
[qwen36 mtp] 512 tokens: 1.39 tokens/forward (23 forwards per 32 tokens) | acceptance 40.9% (9/22 drafts) | gate T(1) 14.7 ms, rows 2:1.19, rho 0.50, acceptance by position 0.43 0.60 0.50 0.50 0.50
[mtp] head rows 39 (0.53 ms each), fed rows 616 (K/V only, 0.03 ms each), drafts 39
[mtp] draft list 16000 tokens on average, 100.0% of picks inside, 0 ids appended, 0 rebuilds
[qwen36 spec gate] 512 tokens: T(1) 14.7 ms, rows 2:1.19, rho 0.50, acceptance by position 0.50 0.50 0.50 0.50 0.50
[qwen36 lookup] 512 tokens: 1.39 tokens/forward (23 forwards per 32 tokens) | acceptance 0.0% (0/0 drafts in 22 verifies) | gate on, 0 declined, 0 probes
[bench] peak RSS 20.02 GB

===== v1-run4.csv (sha256 6f9bf7d79e15b8d4226655a75c5bf9f9e301495388da6128b39a80147e43c705)
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
64,64,0.620,103.19,32,24.0,97.459,0,0.00,d9d3351b46e6980d,1190b702cc495428,0.00000,0.000
512,512,0.318,1609.57,32,18.5,80.382,0,0.00,14c3a19d195d62a3,f57099df7a2d1304,0.00000,0.000
===== v1-run4.err (sha256 666a1c7af53fd812da6d3b4d892fc972bb360b593c37a9c2bcf9cc8cf00cbf48)
[tok] loaded 248044 pieces (max id 248069) from /content/q36mtp/tokenizer.json
[OMP] qwen36: 6 physical-core threads instead of 12 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8192 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36mtp/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.2s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: NVIDIA L4, 23.7 GB VRAM, sm_89
[qtier] dev 0: 21.8 GB free, allowance 20.8 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 19.02 GB left for experts
[qtier] dev 0: budget 19.02 GB for experts (~10794 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.114 ms on CUDA dev 0, 0.418 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 10240 in VRAM -- 14.4 s
[mtp] draft vocabulary: top 16000 + 3/4 rule + conversation, counts from /content/warm/coli_vocab (1588 tokens, 5346 distinct)
[mtp] Qwen3.6 checkpoint head: 256 experts
[bench] CUDA ready in 25.89 s; text 5871 tokens; lengths 64,512; gen 32; each length from an empty state
[qwen36] KV cache: float16, 0.00 GiB at 96 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 4096 rows per prompt chunk (286 KiB a row, 3.1 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[mtp] the head runs on the CUDA chain's device, its experts in the frame (format 4), drafts score an adaptive vocabulary
[qwen36 mtp] 64 tokens: 1.68 tokens/forward (19 forwards per 32 tokens) | acceptance 76.5% (13/17 drafts) | gate T(1) 14.7 ms, rows 2:1.23, rho 0.50, acceptance by position 0.77 0.60 0.50 0.50 0.50
[mtp] head rows 17 (0.55 ms each), fed rows 82 (K/V only, 0.05 ms each), drafts 17
[mtp] draft list 16000 tokens on average, 96.9% of picks inside, 1 ids appended, 0 rebuilds
[qwen36 spec gate] 64 tokens: T(1) 14.7 ms, rows 2:1.23, rho 0.50, acceptance by position 0.50 0.50 0.50 0.50 0.50
[qwen36 lookup] 64 tokens: 1.68 tokens/forward (19 forwards per 32 tokens) | acceptance 0.0% (0/0 drafts in 17 verifies) | gate on, 0 declined, 0 probes
[qwen36 mtp] 512 tokens: 1.39 tokens/forward (23 forwards per 32 tokens) | acceptance 40.9% (9/22 drafts) | gate T(1) 14.6 ms, rows 2:1.20, rho 0.50, acceptance by position 0.43 0.60 0.50 0.50 0.50
[mtp] head rows 39 (0.53 ms each), fed rows 616 (K/V only, 0.03 ms each), drafts 39
[mtp] draft list 16000 tokens on average, 100.0% of picks inside, 0 ids appended, 0 rebuilds
[qwen36 spec gate] 512 tokens: T(1) 14.6 ms, rows 2:1.20, rho 0.50, acceptance by position 0.50 0.50 0.50 0.50 0.50
[qwen36 lookup] 512 tokens: 1.39 tokens/forward (23 forwards per 32 tokens) | acceptance 0.0% (0/0 drafts in 22 verifies) | gate on, 0 declined, 0 probes
[bench] peak RSS 20.04 GB

===== v0-run4.csv (sha256 0a6c2375dbe4bb64cd4d7fe7528e2bbae411248fb6fdcb0d82f134596f394b8d)
ctx_tokens,prefill_tokens,prefill_s,prefill_tps,gen_tokens,gen_first_ms,gen_steady_tps,expert_miss,disk_s,fingerprint,tokens,nll,top1
64,64,0.606,105.66,32,25.8,84.032,0,0.00,d9d3351b46e6980d,1190b702cc495428,0.00000,0.000
512,512,0.316,1618.42,32,20.5,72.139,0,0.00,14c3a19d195d62a3,f57099df7a2d1304,0.00000,0.000
===== v0-run4.err (sha256 ebfa3fc2c060523b2d088a4c1dfa180798b4a7e4788dd71dc57e02b78c83099d)
[tok] loaded 248044 pieces (max id 248069) from /content/q36mtp/tokenizer.json
[OMP] qwen36: 6 physical-core threads instead of 12 logical CPUs; SMT can halve decode throughput on some CPUs (#718); set OMP_NUM_THREADS=<n> to override
== qwen36 Phase-2 engine | cache=256/layer bits=4 ctx=8192 pilot=0 wide=1 hot=0 smooth=0.30 conf=0.92 ==
[tok] loaded 248044 pieces (max id 248069) from /content/q36mtp/tokenizer.json
[meta] loaded: q_heads=16 kv_heads=2 head_dim=256 q_head_dim=512 k_head_dim=256 v_head_dim=256 o_in=4096 rotary_dim=64 n_experts=256 topk=8 inter=512 shared_inter=512 n_group=1 topk_group=1 attn_output_gate=1 n_active=10
[meta] DeltaNet: vheads=32 kheads=16 kdim=128 vdim=128 convk=4 conv_dim=8192
[qwen36] embedding: rows read in place from the checkpoint, COLI_EMBED_MMAP=0 builds the f32 table
[dense-i8] 291 matrices quantized during load, 7.2 GB f32 freed
[qwen36] group-scaled experts: gs=64
resident weights loaded in 9.2s | RSS after load: 3.03 GB
[qwen36] expert format on disk: int4 packed (tier fmt=4)
[qwen36] expert kernel: planar int4 in RAM (expert_ffn.h), QWEN_EXPERT_KERNEL=0 restores int8 unpack
[CUDA] device 0: NVIDIA L4, 23.7 GB VRAM, sm_89
[qtier] dev 0: 21.8 GB free, allowance 20.8 GB
[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 19.02 GB left for experts
[qtier] dev 0: budget 19.02 GB for experts (~10794 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
[place] probe: one [8192 x 2048] int8 GEMV takes 0.112 ms on CUDA dev 0, 0.415 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (30 dnout, 40 attnproj, 120 shexp; 0.61 GB VRAM)
[chain] qwen36: a decode step's routed experts in the frame (the tier's residents by address, format 4, group 64; COLI_CUDA_CHAIN_MOE=0: the host's step)
[chain] qwen36: 40 layers on CUDA device 0 (10 attention), lm_head too; 4.4 MiB of parameters, 35.3 MiB of router, b|a and gate rows, 66 MB of DeltaNet state
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[qtier] warmstart (parallel): all 10240 experts in RAM (planar int4, the only copy), 10240 in VRAM -- 14.5 s
[mtp] Qwen3.6 checkpoint head: 256 experts
[bench] CUDA ready in 25.96 s; text 5871 tokens; lengths 64,512; gen 32; each length from an empty state
[qwen36] KV cache: float16, 0.00 GiB at 96 positions (half of float32; Q36_KV=f32 keeps float32)
[chain] qwen36: 4096 rows per prompt chunk (286 KiB a row, 3.1 GB free on the device; COLI_CUDA_CHAIN_ROWS sets it)
[chain] qwen36: up to 4 experts a step promoted into the card by this conversation's routes, the coldest residents giving way, none while the last 64 steps hit 75 % or more (COLI_CUDA_CHAIN_PROMOTE=0 keeps the warm start's set)
[mtp] the head runs on the CUDA chain's device, its experts in the frame (format 4)
[qwen36 mtp] 64 tokens: 1.60 tokens/forward (20 forwards per 32 tokens) | acceptance 66.7% (12/18 drafts) | gate T(1) 14.6 ms, rows 2:1.37, rho 0.50, acceptance by position 0.69 0.60 0.50 0.50 0.50
[mtp] head rows 18 (2.44 ms each), fed rows 83 (K/V only, 0.05 ms each), drafts 18
[qwen36 spec gate] 64 tokens: T(1) 14.6 ms, rows 2:1.37, rho 0.50, acceptance by position 0.50 0.50 0.50 0.50 0.50
[qwen36 lookup] 64 tokens: 1.60 tokens/forward (20 forwards per 32 tokens) | acceptance 0.0% (0/0 drafts in 18 verifies) | gate on, 0 declined, 0 probes
[qwen36 mtp] 512 tokens: 1.39 tokens/forward (23 forwards per 32 tokens) | acceptance 40.9% (9/22 drafts) | gate T(1) 14.8 ms, rows 2:1.33, rho 0.50, acceptance by position 0.43 0.60 0.50 0.50 0.50
[mtp] head rows 40 (2.44 ms each), fed rows 617 (K/V only, 0.03 ms each), drafts 40
[qwen36 spec gate] 512 tokens: T(1) 14.8 ms, rows 2:1.33, rho 0.50, acceptance by position 0.50 0.50 0.50 0.50 0.50
[qwen36 lookup] 512 tokens: 1.39 tokens/forward (23 forwards per 32 tokens) | acceptance 0.0% (0/0 drafts in 22 verifies) | gate on, 0 declined, 0 probes
[bench] peak RSS 20.04 GB

===== summary (bench's report of the runs above)
# bench: c6caldo-prosa-libera-1

- bench 482e1711d589; engine qwen36; Linux (Intel(R) Xeon(R) CPU @ 2.20GHz, NVIDIA L4); 2026-10-26 03:00
- model q36mtp; text promessi_sposi.txt (sha256 f53e0d80cb2d4492); lengths 64,512; gen 32; free (most likely token, never the end)
- runs: a warm-up of v0 (does not count), then 4 per version, the order turning at every round (ABBA); before every run: wait until cool (at most 900 s), then 60 s of rest

## v0 c4: 4 runs (median, min–max)

| tokens | new | prompt tok/s | first step ms | decode tok/s | experts read | reads' s | fingerprint |
|---|---|---|---|---|---|---|---|
| 64 | 64 | 105.52 (104.90–105.87) | 26 (26–26) | 83.862 (83.118–84.032) | 0 | 0.00 | d9d3351b46e6980d |
| 512 | 512 | 1593.75 (1583.09–1618.42) | 21 (20–21) | 71.698 (71.465–72.139) | 0 | 0.00 | 14c3a19d195d62a3 |

peak memory 20511 MiB, model ready in 25.96 s (medians)

decode by steps (step 1 gives the second new word; medians of 4 runs):

| tokens | steps | experts read | decode tok/s | reads' s |
|---|---|---|---|---|
| 64 | 2 to 32 | 0 | 83.861 | 0.000 |
| 512 | 2 to 32 | 0 | 71.698 | 0.000 |

## v1 c6: 4 runs (median, min–max)

| tokens | new | prompt tok/s | first step ms | decode tok/s | experts read | reads' s | fingerprint |
|---|---|---|---|---|---|---|---|
| 64 | 64 | 103.39 (103.19–103.68) | 24 (24–24) | 97.400 (96.822–97.638) | 0 | 0.00 | d9d3351b46e6980d |
| 512 | 512 | 1606.68 (1580.04–1617.48) | 19 (18–19) | 80.109 (79.295–80.510) | 0 | 0.00 | 14c3a19d195d62a3 |

peak memory 20516 MiB, model ready in 25.97 s (medians)

decode by steps (step 1 gives the second new word; medians of 4 runs):

| tokens | steps | experts read | decode tok/s | reads' s |
|---|---|---|---|---|
| 64 | 2 to 32 | 0 | 97.400 | 0.000 |
| 512 | 2 to 32 | 0 | 80.109 | 0.000 |

## v1 c6 against v0 c4 (medians; positive = faster)

| tokens | prompt | first step | decode | results |
|---|---|---|---|---|
| 64 | -2.0%, every run slower | +7.7%, every run faster | +16.1%, every run faster | same results |
| 512 | +0.8%, ranges overlap | +11.0%, every run faster | +11.7%, every run faster | same results |

| tokens | steps | experts read | decode | reads' s |
|---|---|---|---|---|
| 64 | 2 to 32 | 0 → 0 | +16.1%, every run faster | 0.000 → 0.000 s |
| 512 | 2 to 32 | 0 → 0 | +11.7%, every run faster | 0.000 → 0.000 s |

warm-up (does not count): 64 tokens: prompt 105.31 tok/s, decode 83.688 tok/s; 512 tokens: prompt 1564.10 tok/s, decode 71.900 tok/s
runs not cool: none
pages written to swap during the measurement: 0, 0 of them in the runs that count
