微型引擎，巨型模型。在仅配备约 25 GB 内存的消费级机器上运行 GLM-5.2（744B 参数的 MoE 模型）—— 纯 C 语言实现，零依赖，通过从磁盘流式加载专家模块实现。
Colibrì 是一个轻量级、保质量的 MoE 运行时，它将显存、内存和存储视为统一管理的内存层级。快速内存不足可能会降低速度，但默认策略绝不会静默改变模型精度或路由器语义。
$ ./coli chat
🐦 colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
✓ ready in 32s · resident 9.9 GB
› ciao!
◆ Ciao! 😊 Come posso aiutarti oggi?
查看运行效果
网页仪表盘（`./coli web`）：一个 744B 参数的模型在 6× RTX 5090 上以端到端 4+ tok/s 的速度作答 —— 包含实时 token 指标、硬件面板以及显存/内存/磁盘专家层级。
大脑页面：全部 19,456 个专家构成一个活体皮层 —— 颜色代表存储层级，亮度代表路由热度，每一轮被路由到的专家会闪烁白光。悬停可查看该专家测量到的主题亲和度。
目录
- 核心理念
- 查看运行效果
- 已实现功能
- 真实数据
- 下载模型
- 网页仪表盘
- 有更好的机器？
核心理念
一个 744B 参数的混合专家模型，每个 token 仅激活约 40B 参数 —— 其中每个 token 之间变化的参数仅约 11 GB（即被路由的专家模块）。因此：
- 密集部分（注意力机制、共享专家、嵌入层 —— 共 17B 参数）以 int4 精度常驻内存（9.9 GB）；
- 21,504 个被路由的专家模块（75 个 MoE 层 × 256 个专家 + MTP 头，每个 int4 精度下 19 MB）存储在磁盘上（共 370 GB），按需流式加载，并配备每层 LRU 缓存、可选的固定热存储区，以及作为免费 L2 缓存的 OS 页面缓存。
引擎是一个单一的 C 文件（`c/glm.c`，约 2,400 行）加上少量头文件。无需 BLAS，运行时无需 Python，无需 GPU（存在一个可选的 CUDA 层级用于固定专家模块 —— 见下文）。
已实现功能
- 忠实还原 GLM-5.2（`glm_moe_dsa`）前向传播 —— 已通过 transformers 参考实现验证 token 级别精确性（在具有真实架构的小型随机模型上，teacher-forcing 32/32 通过，greedy 20/20 通过）。
- MLA 注意力机制（q/kv-LoRA，交错式部分 RoPE）配合压缩 KV-cache：每个 token 仅需 576 个浮点数，而非 32,768 个（缩小 57 倍 —— GLM-5.2 有 64 个注意力头且无 GQA）。
- DeepSeek-V3 风格的 sigmoid 路由器（`noaux_tc`、`routed_scaling_factor`）、共享专家、前 3 层为密集层。
- 原生 MTP 投机解码——GLM-5.2 自身的多 token 预测头（第 78 层）会草拟 token，主模型通过一次批量前向传播进行验证。该预测头必须采用 int8 精度（转换器默认执行此操作）：在 int4 精度下，草稿接受率降至 0–4%，投机机制无法生效；而在 int8 精度下，接受率为 39–59%，每次前向传播可生成 2.2–2.8 个 token（社区实测数据，#8）。在精确算术运算下无损——但在实际应用中，其输出与无投机贪婪解码并非字节级一致（#100）。这并非 MTP 独有问题：colibrì 的量化整数内核具有形状依赖性，因此任何批量（S>1）或 GPU 前向传播的舍入方式都与单 token 路径略有不同，而 int4 精度的 GLM-5.2 又非常接近 argmax 平局状态，这种舍入变化足以翻转一个 token。MTP、CUDA 专家级推理以及批量预填充是触发相同敏感性的三种不同方式（社区在 #100 中证实：仅更换内核族就会在 3/5 的提示词上分叉贪婪解码的输出结果，且完全不涉及投机）。每个输出的 token 仍然是有效前向传播的 argmax——后续生成内容保持正确——只是不再是相同的输出流。如需实现字节级可复现性：设置 DRAFT=0（关闭投机），若还需消除内核族/GPU 依赖性，则同时设置 IDOT=0 COLI_CUDA=0。在采样模式下，拒绝采样可保持分布正确性。来自同一实测的诚实说明：在冷缓存状态下，每个被验证的草稿都会路由到额外的专家（约从 660 次专家加载/token 增至约 1100 次专家加载/token），因此在缓存/固定机制预热之前，投机解码可能反而导致净时间损失。
- 语法强制推测草稿（GRAMMAR=file.gbnf, #48）——在约束输出工作负载（JSON/NDJSON、函数调用、结构化提取）中，语法本身是第三个草稿来源：只要语法恰好允许一个合法字节（花括号、引号、键名、枚举体），该强制跨度就会被分词并作为预接受草稿注入，接受率约 1.0——无需草稿头，无需查找表，并且即使使用 #8 中的 int4 MTP 头也能生效。它从不约束采样：强制跨度与任何草稿一样，在相同的批量联合前向传播中验证，因此错误或不同步的语法不会改变输出——最坏情况是草稿被拒绝，并且自适应防护会在接受率低于 50% 时关闭该来源。字节级 GBNF 子集（字面量、字符类、| ( ) ? * +、注释）；GRAMMAR_DRAFT=n 限制每次前向传播的强制跨度（默认 24）。与 DRAFT/MTP 组合使用，后者填充强制跨度之间的自由文本间隙。完整参考——机制、实测 A/B 对比、适用场景、先前工作：docs/grammar-draft.md。
- 真实采样——温度 + 核采样，默认值针对 int4 现实调优（0.7 / 0.90；官方 1.0 / 0.95 会从尾部采样量化噪声）。
- 整数点积内核（Q8_0 风格的 int8 激活、AVX2 maddubs）：int8 矩阵乘法速度提升 1.4–2.5 倍（实测 119 GFLOP/s），int4 在批量中提升 1.8 倍——路由根据每个形状的测量结果决定（int4 单行保持 f32：实测速度更慢）。
- MLA 权重吸收（DeepSeek 技巧）用于解码：无需逐 token 的 k/v 重建——查询吸收 kv_b，上下文在注意力之后投影。经验证精确无误：TF 32/32 和生成 20/20，强制在所有位置使用吸收。
- 异步专家预读：当一组专家正在被乘运算时，内核已经在读取下一组专家（WILLNEED）。
- 量化内核：int8 / 打包 int4 / 打包 int2、逐行缩放、AVX2、按需反量化。打包经验证与 int8 容器位级一致。
- DSA 稀疏注意力 — GLM-5.2 的闪电索引器，严格遵循参考模型 glm_moe_dsa：每层选择前 2048 个因果键（全/共享索引器层），从 out-idx-* 权重中自动检测（--indexer 转换器模式，从 FP8 仓库中提取约 189 MB）。经精确验证：强制保留所有键的选择可逐 token 复现密集注意力。DSA=0 禁用，DSA_TOPK 覆盖默认值。
- KV 缓存持久化 — 对话在引擎重启后仍保持热状态：服务模式在每次对话轮次后将压缩的 MLA KV 追加到 .coli_kv 文件（约 182 KB/token，崩溃安全），并在启动时恢复，无需重新预填充。经验证与不间断会话字节完全一致。KVSAVE=0 禁用。
- 路由器预取（PILOT=1，实验性）— 下一层的路由有 71.6% 的概率可从当前层的注意力后状态预测（经测量）；一个专用 I/O 线程在当前层计算时预取这些专家。
- 批量联合 MoE：在预填充（以及 MTP 验证）中，批次中每个唯一的专家只读取一次，并应用于路由到该专家的每个位置。
- 基于 C 语言的字节级 BPE 分词器（GPT-2 风格，使用 Unicode 属性正则表达式，32 万次合并）。
- 内存安全：专家缓存在启动时根据 MemAvailable 自动调整大小 — 一个诚实的峰值预估（工作集、KV、MTP 行、重建缓冲区），确保内核 OOM 杀手永远不会触发。
- 离线 FP8→int4 转换器（c/tools/convert_fp8_to_int4.py）：一次下载一个分片（约 5 GB），反量化（128×128 块尺度），重新量化到引擎的容器格式，然后删除该分片 — 756 GB 的 FP8 检查点无需同时存在于磁盘上。支持断点续传。
真实数据（WSL2，12 核，25 GB 内存，通过 VHDX 的 NVMe）
详细 GPU 实验：GLM-5.2 在 6 块 RTX 5090 上 — 专家完全驻留在 VRAM+RAM 中，单请求解码达到 6.84 tok/s。
| 指标 | 数值 |
|---|---|
| 磁盘上的模型（int4 容器） | 约 370 GB |
| 常驻内存（密集，int4） | 9.9 GB |
| 加载时间 | 约 30 秒 |
| 对话期间峰值 RSS | 约 20 GB（自动限制） |
| 冷解码成本 | 约 11 GB 磁盘读取/token（75 层 × 8 个专家） |
| 磁盘上限（此开发机的驱动器） | 约 1 GB/s → 冷启动约 0.05–0.1 tok/s |
| MTP 推测解码（int8 头） | 测量值为 2.2–2.8 tok/前向传播（第 8 次） |
这并不慢。这是一款 744B 参数的尖端级模型，在一台成本低于一块 H100 风扇的机器上正确作答。热缓存、固定热专家和 MTP 技术显著降低了有效响应延迟；剩下的就交给磁盘的物理特性了。
SSD 说明
冷启动时随机读取负载很高（约 11 GB/token），但读取操作并不会对 SSD 造成实质性磨损——colibrì 的流式传输是只读的。高负载下的真正担忧在于：（1）系统内存不足时产生的交换流量（写入操作会磨损硬盘——请保持合理的 --ram 预算；colibrì 的自动预算设计旨在避免使用交换空间）；（2）持续的热量：长时间满读取负载会使廉价硬盘发热。请监控硬盘温度与健康状态。
下载模型
一个为 colibrì 预转换的 GLM-5.2 int4 模型已在 Hugging Face 上提供——请使用带有 int8 MTP 头（matey-0 的克隆版本）的版本：
⚠️ MTP 头必须是 int8 格式。原始镜像（jlnsrk/GLM-5.2-colibri-int4）提供的是 int4 MTP 头，这会导致草稿接受率为 0%——推测过程会静默失效，你将失去约 2 倍的 MTP 加速效果。这是最常见的“为什么 MTP 卡在 0%？”的报告来源（#8, #102）。int8 头可实现实测的 39–59% 接受率。上面的 matey-0 克隆版本是原始的 int4 模型，但三个 out-mtp-* 文件已替换为 int8 版本——下载这个版本即可直接使用。
检查你拥有的文件：ls -l <model>/out-mtp-* · int8（正确）：3527131672 / 5366238584 / 1065950496 · int4（0% 接受率）：1765523544 / 2686077736 / 536747200——如果看到这些数字，请从 int8 镜像中替换这三个文件。
下载仓库并将 COLI_MODEL 指向其目录：
COLI_MODEL=/path/to/GLM-5.2-colibri-int4-with-int8-mtp ./coli chat
这完全跳过了 FP8 到 int4 的转换步骤。感谢 DatPat 提供的原始镜像以及 matey-0 提供的 int8 头克隆版本。
快速开始
cd c
./setup.sh # checks gcc/OpenMP, builds, self-tests
# ONE command does everything model-side: downloads GLM-5.2-FP8 shard by shard
# (never needs the full 756 GB at once), converts to the int4 container, then
# converts the MTP head for speculative decoding. Resumable at any point.
# Conversion (only) needs python with: pip install torch safetensors huggingface_hub numpy
./coli convert --model /nvme/glm52_i4 # ~400 GB free on a real ext4/NVMe path
# chat — RAM budget, expert cache and MTP are all detected automatically:
COLI_MODEL=/nvme/glm52_i4 ./coli chat
在加载模型之前，请检查计划的存储层级结构：
COLI_MODEL=/nvme/glm52_i4 ./coli plan
COLI_MODEL=/nvme/glm52_i4 ./coli plan --gpu 0,1 --ram 128 --vram 48 --json
# apply the bounded plan to the normal runner
COLI_MODEL=/nvme/glm52_i4 ./coli chat --auto-tier
coli plan 仅读取 safetensors 文件头，并报告模型的精确稠密/专家参数量、运行时 RAM 预留量、安全专家缓存上限以及有界 VRAM 热层。其带版本号的 JSON 输出设计供 CLI、API 服务器、Web UI 和桌面 shell 共享；它不会分配模型张量或启动推理。--auto-tier 将同一方案应用于聊天、运行、服务和基准测试。它会立即设置 RAM 预算和上下文；VRAM 层仅在当前 glm 二进制文件与 CUDA 链接时才启用。显式标志和环境变量的优先级高于自动值。
在加载模型之前，coli doctor 执行一次只读就绪检查，并说明所选的磁盘/RAM/VRAM 放置方案是否可运行：
COLI_MODEL=/nvme/glm52_i4 ./coli doctor
COLI_MODEL=/nvme/glm52_i4 ./coli doctor --gpu 0 --ram 128 --json
Doctor 验证模型目录、配置、分词器、safetensors 文件头、引擎可执行文件、可用 RAM、请求的 NVIDIA 设备、CUDA 链接以及 coli plan 所使用的相同放置预算。它从不启动 glm、读取张量载荷、导入模型框架或创建 CUDA 上下文。带版本号的 JSON 报告使用稳定的检查 ID 以实现自动化。警告保持退出状态码 0；缺少需求或不安全的 RAM 预估返回 1，而无效的 CLI 值返回 2。
运行时的引擎是纯 C 语言——Python 仅用于一次性转换器。
Windows 11（原生，无需 WSL）
colibrì 在 Windows 11 x86-64 上使用 MinGW-w64 原生构建和运行。该移植在 c/compat.h 中添加了一个 _WIN32 兼容层，将 POSIX I/O 映射到 Windows API（pread → ReadFile+OVERLAPPED，posix_fadvise 为空操作，对齐分配，MoveFileEx 重命名，GlobalMemoryStatusEx RAM 检测）。所有平台差异都保留在 compat.h 中；引擎源代码保持不变。
工具链：通过 winlibs 或 MSYS2 MinGW-w64 使用 GCC。已使用 GCC 16.1.0（x86_64-ucrt-posix-seh）测试通过。
# One-time toolchain install (pick one):
scoop install mingw-winlibs # portable, no shell needed
# or: pacman -S mingw-w64-x86_64-gcc make # via MSYS2
# Build (from c/ directory):
make glm.exe # GLM-5.2 engine (static, no DLL dependencies)
make olmoe.exe # OLMoE engine (same shims)
make iobench.exe # disk I/O benchmark
make test-c # run C tests
make test-python # run Python tests (requires python)
# AVX-VNNI: Intel Alder Lake+ (and Meteor Lake+) CPUs have a 128-bit int8
# dot-product instruction (VPDPBUSD) the engine can use for ~1.3x faster
# quantized matmul. The x86-64-v3 default (portable AVX2) compiles it out;
# build for THIS machine to enable it:
make glm.exe ARCH=native # banner prints "idot: avx-vnni"
# Verify (tiny model, 2.4 MB):
pip install torch transformers safetensors huggingface_hub
python tools/make_glm_oracle.py # generate tiny oracle
SNAP=./glm_tiny TF=1 ./glm.exe 64 16 16 # expect "32/32 positions"
# Run with real model:
SNAP=D:\glm52_i4 ./glm.exe 64 4 16 # batch inference
python coli chat --model D:\glm52_i4 # interactive chat
python coli serve --model D:\glm52_i4 # OpenAI-compatible API
预热（隔夜缓存预填充）：引擎的专家缓存会从你的工作负载中学习。附带的 warmup.ps1 脚本以循环方式使用多种提示词运行 coli run，以无人值守方式构建 .coli_usage 直方图，从而使下一次真实会话开始时拥有一个庞大且准确的活跃专家固定集。每次运行在干净完成时都会原子化地保存使用情况。
.\warmup.ps1 -Rounds 1 -Ngen 32 # ~60-90 min, durable progress
NVIDIA GPU（可选，通过运行时 DLL）：在 Windows 上，引擎使用 MinGW gcc 构建，但 CUDA 内核需要 MSVC + nvcc。两者分工明确：将 CUDA 后端构建为独立的 coli_cuda.dll（nvcc + MSVC），然后主机程序 glm.exe 在运行时通过 LoadLibrary 加载该 DLL（c/backend_loader.c）。主机从不直接链接 cudart；如果该 DLL 不存在，引擎会无错误地回退到 CPU。
# Prerequisites: CUDA Toolkit + MSVC Build Tools (cl.exe) + nvcc on PATH.
# Build the DLL from a shell with the MSVC environment set (vcvars64.bat or
# "x64 Native Tools Command Prompt for VS"):
make cuda-dll CUDA_HOME="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.8" CUDA_ARCH=sm_120
# Build the host with the runtime loader (CUDA_DLL=1 adds -DCOLI_CUDA and
# links backend_loader.o instead of cudart):
make glm.exe CUDA_DLL=1 ARCH=native
# Run with the GPU expert tier (8 GB VRAM budget here; scale to your free VRAM):
$env:COLI_CUDA="1"; $env:COLI_GPU="0"; $env:CUDA_EXPERT_GB="8"
python coli chat --model D:\glm52_i4 --topp 0.7
该 DLL 导出了 11 个 extern "C" 符号（如 coli_cuda_init、coli_cuda_matmul 等）；backend_loader.c 在首次使用时通过 GetProcAddress 解析它们。ColiCudaTensor* 对主机是不透明的（仅存储，从不解引用），因此由 MSVC 分配的结构体在 ABI 边界上是安全的。CUDA_ARCH 必须与你的 GPU 计算能力匹配（例如，Blackwell / RTX 50 系列使用 sm_120，Ada / RTX 40 系列使用 sm_89）。
状态：第一阶段已完成（可编译、结果正确、静态链接）。Windows GPU 层级（通过 LoadLibrary 加载运行时 coli_cuda.dll）已在 RTX 50 系列（sm_120）上实现并验证。O_DIRECT（第二阶段）和针对 transformers 参考实现的完整模型验证是独立的工作流。
兼容 OpenAI 的 API
coli serve 保持一个模型进程加载状态，并暴露一个纯文本的、兼容 OpenAI 的 HTTP API。网关仅使用 Python 标准库；推理仍在同一个无依赖的 C 引擎中运行。
cd c
COLI_MODEL=/nvme/glm52_i4 COLI_API_KEY=local-secret ./coli serve \
--host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri
curl http://127.0.0.1:8000/v1/chat/completions \
-H 'Authorization: Bearer local-secret' \
-H 'Content-Type: application/json' \
-d '{
"model": "glm-5.2-colibri",
"messages": [{"role": "user", "content": "Hello"}],
"stream": true
}'
已实现的端点包括 GET /v1/models、GET /v1/models/{model}、POST /v1/chat/completions 以及旧版 POST /v1/completions。聊天和补全请求支持 JSON 响应、SSE 流式传输、用量统计、max_tokens/max_completion_tokens、temperature 和 top_p。扩展参数 enable_thinking: true 可启用 GLM-5.2 的推理块；标准的 reasoning_effort 字段也会启用它，除非设置为 none。
第一个版本特意只支持文本，并且一次只服务一个生成请求：744B 模型驻留在一个持久进程中，因此并发 HTTP 请求会排队，而不会加载重复的模型副本。工具、图像/音频输入、自定义停止序列、对数概率和 token 惩罚会返回明确的错误，而不是被静默忽略。默认绑定地址是 localhost；在将服务器暴露到本机之外前，请设置 COLI_API_KEY。
Vite 开发服务器和 Tauri 本地源的浏览器访问默认已启用。重复使用 `--cors-origin https://your-ui.example` 可允许另一个精确源，或仅在受信任的本地网络上使用 `--cors-origin '*'`。
引擎拥有一个可变的 KV 上下文，因此 HTTP 生成使用有界 FIFO 准入队列，而不是假装运行不安全的并行序列。通过 `--max-queue N`（默认 8）和 `--queue-timeout SECONDS`（默认 300）进行配置，或使用 `COLI_MAX_QUEUE` / `COLI_QUEUE_TIMEOUT` 环境变量。饱和和超时的请求在流式传输头部发送之前会收到 OpenAI 格式的 HTTP 429 错误。`GET /health` 会暴露活跃/排队/已完成/被拒绝的计数器，成功的生成响应会包含 `x-colibri-queue-wait-ms`。
隔离的 KV 上下文
`coli serve --kv-slots N` 最多分配 16 个独立的序列上下文。请求通过可选的整数 `cache_slot` 字段选择一个；普通的 OpenAI 客户端会省略该字段并保持原始的 slot 0 行为。
{
"model": "glm-5.2-colibri",
"messages": [{"role": "user", "content": "Continue this conversation"}],
"cache_slot": 1
}
每个 slot 拥有自己的 token 历史、压缩的 MLA/DSA KV 内存、MTP 窗口以及崩溃安全的持久化文件（`.coli_kv`、`.coli_kv.1` 等）。引擎仍然一次执行一个序列；这建立了明确的 KV 所有权，而不假装线程化 HTTP 是连续批处理。RAM 准入会考虑每个已配置的 slot。使用 `COLI_KV_SLOTS=N` 作为等效环境变量。从小值开始：在默认的 4096 token 上下文下，每个 slot 会消耗数百 MB。
实验性 Metal 后端（Apple Silicon）
在 Apple Silicon 上，解码阶段受矩阵乘法（matmul）限制，统一内存消除了让 CUDA 的流式专家停留在 CPU 上的 PCIe 拷贝开销——因此 colibrì 提供了一个可选加入的 Metal 后端，可在 GPU 上运行路由专家 SwiGLU（批处理，从 RAM slab 零拷贝）、融合解码注意力（一个命令缓冲区中的完整 MLA 层，S≤4）以及预填充阶段的大型 GEMM 操作。与 CPU 路径相比，token 结果完全一致。
cd c
make glm METAL=1 # macOS only; no Xcode needed (shader compiles at runtime)
make metal-test # standalone kernel/attention correctness vs CPU reference
COLI_METAL=1 COLI_MODEL=/path/glm52_i4 ./coli chat --ram 96
在 M4 Max（128 GB，缓存预热，MTP 开启）上测得：CPU 0.30 → Metal 0.42 tok/s（约 1.4 倍）（最佳配置添加 DIRECT=1；相比本机首次冷启动约 3 倍）。关键设计要点：Metal 约 5 毫秒的提交延迟使得逐矩阵乘法调度得不偿失——所有操作被批量合并为每层少量命令缓冲区，且常驻专家的 GPU 工作会在未命中专家的磁盘读取之前提交，从而实现 I/O 与计算重叠。COLI_METAL_GEMM_MIN 用于调整预填充 GEMM 行阈值（默认 16）。流式传输、缓存、MTP、DSA 及持久化格式保持不变；所有 GPU 路径在发生任何故障时都会回退到逐块 CPU 处理。数值计算采用反量化→f32-MAC（与 CUDA 层级相同）；贪婪解码输出与 CPU 引擎逐字节一致。
实验性常驻 CUDA 后端
colibrì 包含一个可选的 CUDA 后端，用于处理模型常驻张量。流式专家暂时仍保留在原始 CPU 路径上：每次使用都将专家从 NVMe 复制到 GPU，只会把磁盘瓶颈替换为 PCIe 瓶颈。常驻量化张量采用惰性上传方式，上传一次后重复使用。
cd c
make cuda-test CUDA=1 # q8/q4/q2/f32 kernel correctness
make CUDA=1
# optional dense-path experiment (hot experts are configured below)
COLI_CUDA=1 COLI_GPU=0 CUDA_DENSE=1 SNAP=/nvme/glm52_i4 ./glm 64 4 4
要求：Linux 系统、NVIDIA 驱动程序以及位于 /usr/local/cuda 的 CUDA 工具包（可通过 CUDA_HOME=/path/to/cuda 覆盖）。CUDA_ARCH=native 针对当前机器中的 GPU 进行编译；交叉编译时需指定显式架构。若在仅 CPU 的二进制文件、无效设备或不可用运行时上请求 CUDA，将在启动时失败，而非静默回退。
正常的 make 构建和运行时行为保持不变。CUDA 默认作为仅限专家的加速器。CUDA_DENSE=1 会额外将常驻密集/注意力投影张量以轮询方式分配到所选设备上；其投影占用空间在放置专家层级之前预留。在六块 RTX 5090 上，搭配 150 GB 专家层级，预热后的两请求/64 token GLM-5.2 运行从 1.650 提升至 2.157 聚合 tok/s（+30.8%），同时保留完整的专家层级。在投影密集集和每设备 2 GB 运行时预留空间适配目标 GPU 之前，请将此视为可选功能。经过测量的 PIN 配置文件可将其最热门的专家提升至持久 VRAM 层级，同时将其他专家保留在 RAM 中。
STATS=stats.txt SNAP=/nvme/glm52_i4 ./glm 64 4 4 # collect routing frequencies first
COLI_CUDA=1 COLI_GPU=0 CUDA_EXPERT_GB=16 \
PIN=stats.txt PIN_GB=160 SNAP=/nvme/glm52_i4 ./glm 64 4 4
# multi-GPU expert tier, 150 GB total budget across six 32 GB devices
COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=150 \
CUDA_DENSE=1 PIN=stats.txt PIN_GB=300 RAM_GB=226 \
SNAP=/nvme/glm52_i4 ./glm 64 4 4
# large-RAM host: fill safe VRAM, then keep every remaining expert in RAM
COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=auto \
CUDA_DENSE=1 COLI_CUDA_ATTN=1 PIN=stats.txt PIN_GB=all RAM_GB=auto \
SNAP=/nvme/glm52_i4 ./glm 64 4 4
选定的专家会在启动时上传，因此容量故障发生在推理之前，日志会报告它们精确的张量占用空间。在预留了预估的密集常驻集以及每个选定设备 2 GB 运行时余量后，预算会被限制在空闲 VRAM 范围内。使用 COLI_GPUS 时，CUDA_EXPERT_GB 是整个设备集的总预算；专家会被完整分配给能够容纳它们且负载最低的设备。多 GPU 运行也默认启用 PIN_FILL=1：首先放置实测的热点集，然后用零热度专家填充未使用的 VRAM。CUDA_RELEASE_HOST=1（多 GPU 默认值）会在成功上传后释放 RAM 副本，并且仅在后续 CUDA 失败时才从磁盘重新加载。将任一变量设为 0 可恢复保守行为。当释放主机备份时，放置是分离且分阶段的：加载最热的前缀，上传到 VRAM，然后在下一个排名的后缀加载到 RAM 之前释放。因此，PIN_GB 描述的是合并后的排名集，而非重复的 RAM 和 VRAM 副本。在一台 256 GB 双插槽主机上，从 150 GB VRAM + 130 GB RAM 的放置方案切换到 150 GB VRAM + 150 GB RAM，固定 token 重放速度从 1.87 tok/s 提升至 2.16 tok/s（+15.7%），专家磁盘等待时间从 5.144 秒减少到 3.948 秒，并且预估的 RAM 峰值保持在 RAM_GB=226 以下。缓存上限会自动下调（在该次运行中从 54 降至 40），以便更大的固定层不会超出进程预算。在可用 RAM 较少的主机上，请从更低的数值开始。
CUDA_EXPERT_GB=auto 会为每个选中的设备自动填充，上限为该设备测得的空闲内存减去预估的稠密张量以及 2 GB 运行时余量。PIN_GB=all 随后会将所有剩余的路由专家加载到 RAM 中，从而在主机预算允许的情况下消除解码时的磁盘未命中。常规的 RAM_GB 保护机制仍然会限制每层工作缓存，并拒绝不安全的预估；此模式专为专用高内存推理主机设计，而非运行其他工作负载的桌面电脑。在一台配备六张 RTX 5090 的专用 251 GiB 主机上，此模式选择了 176.7 GB 的 VRAM 专家层级和 191.3 GB 的 RAM 层级（所有 19,456 个专家均驻留）。该模式还会每生成 16 个 token 就调整一次 VRAM 层级，方法是将热门的 RAM 专家交换到现有的 GPU 插槽中。一次真实的 64 token 贪婪解码 GLM-5.2 生成任务测得解码速度为 6.00 tok/s，而此前使用 150 GB 层级时的端到端速度为 2.20 tok/s；专家命中率为 100%，磁盘等待时间为零。提示词预填充速度另行报告。这是一个特定主机的容量结果，并非可移植的默认值。
文本模式的时间报告会将预填充与解码分开统计。解码速率从提示词 KV 构建完成后开始计算，因此其吞吐量可与 REPLAY 相媲美，且不隐藏首 token 生成时间。MTP 推测解码在 CUDA 上默认关闭，因为冷草稿路由会增加专家流量；显式的 DRAFT=n 仍会覆盖此默认设置。
在六张 RTX 5090 32 GB 显卡上，使用 GLM-5.2 int4 模型，一个 150 GB 的热优先层级在 64 token 的多样化提示词上维持了 0.94 token/s 的速度（专家命中率 87.8%），并在预热后的短提示词上达到了 1.64 token/s（命中率 99.3%）。同样的容量在没有路由热数据填充的情况下，仅能达到 0.29 token/s，因此配置文件的质量比原始 VRAM 容量更为重要。这些是单次运行的工程测量结果，并非可移植的性能保证。
当前限制：设备使用独立的上下文和同步的主机级激活副本——尚无 P2P/NCCL 依赖。独立的专家组跨设备并发执行，但单个专家不会被分片。这些内核是以正确性优先的自定义内核，而非 cuBLAS/Tensor Core 内核。
为了在不使用完整检查点的情况下实现可复现的后端 A/B 测试，请生成确定性的 3.13 亿参数 glm_moe_dsa 测试夹具，并运行固定 token 重放：
cd c
python tools/make_glm_bench_model.py --output /nvme/colibri-bench-medium --device cuda
python tools/benchmark_cuda_fixture.py --model /nvme/colibri-bench-medium --gpu 0
该测试夹具具有随机权重，并非语言模型。它仅用于保留真实的 MLA/MoE/流式形状，并使用相同的重放 token 来比较 CPU 流式、纯密集 CUDA、CPU 热存储和 CUDA 热专家执行。
网页界面
web/ 目录包含一个社区贡献的浏览器 UI（React + TypeScript，约 390 行源码，纯 API 客户端——它从不直接接触引擎）：
cd web
npm ci && npm run dev # then point it at an OpenAI-compatible endpoint
它采用标准 OpenAI Chat Completions 协议，支持 SSE 流式传输，因此可配合 colibrì OpenAI 兼容服务器（正在审核中，#21）或任何其他兼容端点使用。数据不会离开你配置的端点。终端 coli 聊天仍然是第一等接口。
实用调节参数（环境变量或标志）：--temp T 设置 token 采样温度（默认 0.7 + nucleus 0.90——针对 int4 调优；0 为贪婪模式），--topp 0.7 自适应专家 top-p（减少 30–40% 磁盘用量），--ngen N 每次回答的最大 token 数（:more 可延续被截断的对话），--repin N 每生成 N 个 token 重新调整 RAM/VRAM 热专家，AUTOPIN=0 禁用学习缓存的自动固定，THINK=1 启用 GLM-5.2 的推理模块，DRAFT=n MTP 草稿深度，GRAMMAR=g.gbnf 语法约束草稿用于生成受限 JSON/NDJSON 输出（GRAMMAR_DRAFT=n 限制强制跨度长度），TF=1 教师强制验证，PILOT=1 路由器预取磁盘（实验性——见下文），CAP_RAISE=0 不自动扩展专家缓存。
资源策略
coli plan 会报告计划的热（VRAM）、温（RAM）和冷后备（磁盘）层级、每项放置的原因以及预期的瓶颈。默认的 --policy quality 和 --policy balanced 模式会保留检查点量化和路由器决策，除非传入了 --topk 或 --topp；这些显式的有损覆盖会打印警告并继续执行。
自动层级规划会根据物理核心数调整 OpenMP 大小，并将工作线程绑定到各核心上。当 SMT 兄弟线程竞争有限的内存通道时，内存受限的量化内核性能可能会急剧下降；显式的 OMP_* 设置始终具有更高优先级。
coli plan --model /models/glm52_i4 --policy quality
coli run --auto-tier --policy quality "Explain MoE offloading"
# Explicit research-only router reduction:
coli run --policy experimental-fast --topk 4 "Benchmark prompt"
磁盘是一种不可变的恢复源，而非正常的解码目标。如果方案将冷专家字节留在磁盘上，速度取决于缓存命中率；输出质量则不受影响。
冷专家读取采用延迟流水线：常驻 RAM/VRAM 的专家执行，同时缺失的专家在有限的后台 I/O 池中加载，随后冷结果在该层完成前汇入。当存在前台任务时，`IO_THREADS=n` 可覆盖默认的 8 个加载线程。性能分析报告会同时显示磁盘服务时间和较小的前台可见等待时间，从而使重叠部分明确可见，而非被归因于无法解释的加速。
`--policy balanced` 启用无损实时放置（`REPIN=64`）。在安全的请求边界处，每层的 LFRU 分数会结合衰减的会话频率与近期访问，并替换最多四个足够冷的固定专家。`--policy quality` 默认关闭实时替换；`REPIN=0` 始终禁用该功能。持久的 `.coli_usage` 历史记录与会话本地的 LFRU 状态保持分离。
对于单 token 的 q4 CPU 专家，门控投影与升维投影共享一个 OpenMP 分发，同时保留相同的逐行 AVX2/NEON 算术运算。这消除了每个 RAM 专家的一次线程团队启动，且无需激活重量化或降低精度回退。这是迈向持久原生 CPU 专家池的一块垫脚石，而非其替代方案。
专家缓存会根据你的 RAM 自动调整大小（自 2026-07-10 起）：引擎现在会提高 LRU 上限以填满你的 `--ram` 预算，而不仅仅是降低它。在此修复之前，一台 128 GB 的机器与一台 16 GB 的机器运行着相同的每层 8 专家缓存（问题 #12）——如果你在此日期之前对 colibrì 进行了基准测试，请重新运行：你的数据曾受到上限限制。
路由器预取（PILOT=1，实验性功能）：GLM-5.2 的专家路由具有可测量的提前可预测性——将第 L+1 层的路由器应用于第 L 层注意力后状态，可召回 71.6% 的真实 top-8 专家（相比之下，“与上一 token 相同的专家”策略仅为 41.3%）。PILOT=1 利用这一特性，在当前层计算的同时，通过专用 I/O 线程发出下一层专家预读请求。在我们的开发机上，磁盘已接近 80% 饱和，因此该功能测量结果为中性；在计算与磁盘负载均衡的机器上（如 issue #12 中的 Ryzen AI 9：磁盘 43% / 矩阵乘法 46%），它应该能实现实际工作重叠——欢迎提供测量数据。
学习型缓存：引擎会记录你的使用过程实际路由到了哪些专家（在模型旁生成 `.coli_usage` 文件，每轮对话更新），并在启动时自动将最热门的专家固定在空闲 RAM 中。colibrì 用得越多，速度就越快。
实时层级自适应（`--repin N`，可选加入）：在安全的对话轮次边界，一个衰减的会话热力图会将冷门固定专家替换为更热门的流式专家。替换操作将专家从磁盘加载到现有 RAM 槽位中；GPU 后端槽位会立即刷新相同的 VRAM 层级预算。25% 的迟滞区间和四次交换上限可防止层级抖动。持久化的 `.coli_usage` 文件仍作为长期信号保留，不会衰减。
对话热恢复（`.coli_kv`，自 2026-07-10 起）：coli chat 在每轮对话后将压缩的 MLA KV 缓存持久化到磁盘（约 182 KB/token，增量追加，崩溃安全）。关闭对话，第二天重新打开——模型仍能记住整个对话内容，且无需重新预填充：经验证，与不间断会话的字节级结果完全一致。`:reset` 可清除该缓存，`KVSAVE=0` 可禁用此功能。
网页仪表盘
一条命令即可在同一个端口上提供 OpenAI 兼容 API 和网页控制台，引擎就绪后自动打开浏览器：
cd web && npm install && npm run build # once
./coli web --model <model-dir>
你将获得：
- 带实时指标的聊天：生成过程中闪烁的 token 计数器，随后显示 tok/s、首 token 生成时间、提示词→补全计数以及队列等待时间；
- 运行时面板：你的硬件信息（CPU、GPU + VRAM、RAM、核心数）、调度器，以及实时专家层级条——当前 19,456 个专家中有多少位于 VRAM / RAM / 磁盘中；
- 大脑：整个模型是一个 76×256 的皮层，每个专家对应一个细胞。颜色代表层级，亮度代表路由热度，每一轮被路由到的专家会闪烁白色然后衰减——你可以亲眼看到模型在"思考"。悬停在任何细胞上，可查看其层级、热度以及测量到的话题亲和度（代码、中文、数学、法律等领域的专家分布在 11–22 层）。
仪表盘通过两条极简的协议线（TIERS、EMAP/HITS）和纯 JSON 端点与引擎通信——其负载不会超过引擎本身。
如果你有更好的机器，试试看——以下是预期效果
colibrì 是在刻意低调的硬件上构建的（12 核、25 GB 内存、一块位于 WSL2 VHDX 之后的无 DRAM 旧款 NVMe，该驱动器上测得随机读取约 1 GB/s——注意 WSL2 VHDX 本身并不慢：一台社区 5090 机器通过它测得 10.5 GB/s 的 O_DIRECT 速度，#101）。这些限制中的每一项，都是你的机器可以调高的旋钮。引擎需要：Linux（或 WSL2）、macOS 或原生 Windows 11（MinGW-w64）；gcc 支持 OpenMP、AVX2、≥16 GB 内存，以及本地 NVMe（ext4/NTFS——绝不能是网络/9p 挂载）上约 370 GB 的 int4 模型。
如何按顺序测试：
cd c && ./setup.sh # build + architecture self-test (expects 32/32)
# 1) measure YOUR disk the way the engine uses it (parallel 19 MB random reads):
gcc -O2 -fopenmp iobench.c -o iobench
./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 0 # buffered, 8 threads
./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 1 # O_DIRECT (bypass cache)
# Caveat (#86): iobench reads a bounded ~1 GB shard, so buffered reads on a big-RAM box
# report the PAGE CACHE, not the disk. Use the O_DIRECT run (arg 1) for a true number, and
# run it on a shard you haven't touched this session (a prior buffered run caches its pages).
# On macOS there is no O_DIRECT — iobench uses F_NOCACHE, which stops *new* caching but can't
# evict pages a prior buffered run already resident-mapped, so a macOS "O_DIRECT" figure right
# after a buffered run still reads cache. Reboot or use a fresh shard for a real cold read.
# 2) chat; watch the per-turn stats line (tok/s, expert hit-rate, RSS):
COLI_MODEL=/path/to/glm52_i4 ./coli chat
# 3) record expert usage, then pin the hottest experts in your spare RAM:
STATS=stats.txt ./coli chat
PIN=stats.txt PIN_GB=20 ./coli chat # scale PIN_GB to your free RAM
# 4) quality benchmarks (MMLU/HellaSwag/ARC):
./coli bench
粗略估算（解码受磁盘限制：一个冷 token 需要读取约 11.4 GB 的专家数据；缓存预热后，MTP 推测解码大致可将有效成本减半；RAM 能将冷读取变成免费的缓存命中）：
| 机器 | 预期 |
|---|---|
| 本开发机（WSL2 VHDX，约 1 GB/s，25 GB 内存） | 约 0.05–0.1 tok/s 冷启动——已验证的基准 |
| 原生 Linux，PCIe4 NVMe（随机读取约 3–5 GB/s），32 GB 内存 | 约 0.5–1 tok/s |
| PCIe5 NVMe 或 2×NVMe RAID0（约 8–12 GB/s），64 GB 内存（固定约 40 GB 的热门专家） | 约 2–4 tok/s |
| 128–256 GB 内存，12 核（热门专家已缓存） | 约 2–4 tok/s——受矩阵乘法限制：约 80 GFLOP/token 对比我们 AVX2 内核的约 250 GFLOP/s |
| 相同内存 + 24–32 核，或 AVX-512/VNNI 内核 | 约 5–15 tok/s——可交互；内核优化是倍增器 |
这些是估算值，并非实测数据——如果你在高端硬件上运行 colibrì，请提交 issue 附上你的数据：来自更好机器的真实数据点正是这个项目下一步所需要的。
社区基准测试（实测）
来自真实机器的真实数据，默认构建（setup.sh，gcc 13），贪婪解码，--ngen 32，MTP 开启：
| 机器 | 磁盘（iobench，19 MB × 64，8 线程） | 配置 | 实测 |
|---|---|---|---|
| 英特尔酷睿 Ultra 7 270K Plus（24 线程）· WSL2 · 24 GB 内存 · NVMe VHDX（#2） | 1.96 GB/s 缓冲 · 2.74 GB/s O_DIRECT | 默认 | 0.07 tok/s · 专家命中率 3–4% · RSS 14.1 GB |
| 同上 | 同上 | --topp 0.7 | 0.11 tok/s · 专家命中率 11% · RSS 14.7 GB |
| Apple M5 Max（18 核）· macOS · 128 GB 统一内存 · 内置 SSD（#4, #5） | 约 4 GB/s 冷启动（14.2 GB/s 的读取速度受缓存影响——参见注释） | 默认，MTP 关闭 | 1.06 tok/s · 专家命中率 23% · RSS 21.8 GB |
| Apple M5 Max · macOS · 128 GB 统一内存 · 2 TB SSD · Metal 后端（#72, #87） | （macOS 的 O_DIRECT 数值不可靠——参见注释） | Metal 开启 · --ram 96 · 39.7 GB 热固定 · MTP 关闭 | 1.83 tok/s · 专家命中率 66% · 预热后从 1.11 升至 1.83（运行过程中） |
| 同上 · 46.9 GB 固定（2.94M 选择历史）· --ram 110，1024 token 运行（#103） | 同上 | Metal 开启（专家 + 注意力）· MTP 关闭 | 2.06 tok/s · 命中率 72.5% · 输出连贯 · 迄今为止最快的数据点（仍在 rebase 前的 Metal 分支上） |
| Mac Mini M4 Pro · macOS · 48 GB 统一内存 · Metal 后端（#107） | 6.59 GB/s F_NOCACHE（新分片） | Metal 开启 · --ram 38 | 0.30 tok/s（对比纯 CPU 的 0.18）—— 该条目显示，内存仅为三分之一的 Apple Silicon 击败了 32 核 9950X 行 |
| Epyc 9654 ES · Linux · 4x16GB DDR5-4800-rdimm · 三星 PCIe Gen3 x4 NVME SSD | — | MTP=1 DIRECT=1 | 0.31 tok/s · 专家命中率 35% · RSS 21.52 GB |
| Ryzen AI 9 HX 370（Framework 13）· Arch Linux · 128 GB · WD SN850X，BTRFS zstd（#12） | — | int8 MTP 头 · --cap 32 · 46.7 GB 自动学习 PIN | 0.37 tok/s · 专家命中率 66% · MTP 接受率 52%（2.59 tok/fw）· RSS 105 GB |
| Ryzen 9 9950X（32 线程）· Linux · 123 GB · Crucial P3 QLC Gen3（#31） | 1.51 GB/s 缓冲 | 默认，冷启动运行 2 次 | 0.10 tok/s · 命中率 53% · 分析：66% 磁盘 |
| 同上，同一台机器，模型移至三星 9100 PRO PCIe 5.0（#31） | 8.81 GB/s O_DIRECT | 同上（使用历史保留） | 0.28 tok/s · 命中率 57% · 分析翻转：32% 磁盘 / 57% 矩阵乘法 |
| Ryzen AI Max+ 395（Framework Desktop）· Ubuntu · 128 GB LPDDR5x · 英特尔 Optane 905p PCIe 3.0（#39） | 3.27 GB/s 缓冲 | int8 MTP 头 · 全新历史（纯 LRU，自动提升上限 65） | 0.16 tok/s · 命中率 57% · 分析：49% 磁盘 / 47% 矩阵乘法 |
| 同上，五次运行后 —— 学习固定 47.6 GB（#39） | 同上 | --temp 0.7 --topp 0.7 | 0.40 tok/s · 命中率 71% · 非 Apple 设备中最快数据点 |
| Ryzen 7 9800X3D（16T）· WSL2 · 70 GB 内存 · 三星 9100 PRO PCIe 5.0 · RTX 5090（#101） | 10.51 GB/s O_DIRECT | MTP 关闭 · 固定内存 24 GB · 命中率 54% · OMP 热团队开启 | 0.41 tok/s · 磁盘瓶颈（磁盘 36.5 秒 vs 矩阵乘法 24.0 秒）· CUDA 专家层级 ≈ 0%（AVX-512 CPU 与 5090 持平）· --topp 0.7 → 0.52 tok/s |
| EPYC 7443（24核/48线程，Zen3 AVX2）· Linux · 430 GB 内存 · 通过 TrueNAS 虚拟机实现的 NVMe RAID-Z1（#104） | 约 1 GB/s（虚拟机开销） | 固定内存 77.5 GB · 上限自动提升至 194/层 · MTP 关闭 | 1.00 tok/s · 命中率 98% · 磁盘瓶颈消除 → 受限于内存带宽 + 矩阵乘法（Zen3 无 AVX-512/VNNI） |
| Intel i5-12600K（10核/16线程，AVX2）· 原生 Windows 11，无 WSL · 32 GB · MinGW GCC 16.1（#113） | 缓冲模式（MinGW 上无 O_DIRECT） | int8 MTP 头 · 冷启动，小内存（上限约 2/层） | 0.08 tok/s · 命中率 3.7% · MTP 接受率 57%——首个原生 Windows 数据点，移植验证通过 |
| Ryzen 9 9950X3D2（16核/32线程，avx512-vnni）· 原生 Linux · 121 GB · Samsung 9100 PRO PCIe Gen5 · RTX 5090（28 GB 专家层级，固定内存 1475）（#120） | 11.48 GB/s O_DIRECT | MTP=0 DIRECT=1 PIPE_WORKERS=16 PREFETCH=1 | 1.23 tok/s · 关闭 MTP 在磁盘瓶颈场景胜出 · 目前最快的 x86 数据点 |
| Ryzen AI Max+ 395（Strix Halo，16核/32线程 Zen5，avx512-vnni）· Arch Linux · 128 GB 统一 LPDDR5x 内存 · SK hynix P41 PCIe 4.0（#124） | — | DIRECT=1 PIPE=1 --topp 0.7 · 自动固定内存 | 冷启动 0.06 → 稳定 1.10 tok/s · 首个 Strix Halo / gfx1151 数据点（统一内存：无独立显存层级） |
| Intel Core Ultra 9 185H（16核/22线程，avx-vnni）· 原生 Windows 11，无 WSL · 32 GB · Crucial P3 QLC NTFS · RTX 5070 Ti（未使用）（#128） | — | int8 MTP 头 · 配合 #131（管道 + 内存修复），缓存预热，无 GPU | 冷启动 0.03 → 预热后 0.5 tok/s（约 7 次提示词预热）· 修复可移植性阻塞后，原生 Windows 上的缓存预热——在 #131 之前，主线版本在 \r\n READY 哨兵处挂起 |
| Dell Pro Max GB10（DGX Spark：Grace 10×X925 + 10×A725，aarch64 i8mm/sve2）· Linux · 121 GB 统一 LPDDR5x 内存 · Dell OEM 4 TB NVMe · GB10 sm_121（#136） | 5.58 GB/s O_DIRECT（#76 中的 NVIDIA-OEM 设备为 10.74——同一平台，不同 SSD） | int8 MTP 头 · 缓存预热 | 冷启动 0.21 → 预热后 0.50 tok/s · 命中率 83% · MTP 接受率 73%（3.20 tok/前向传播）· 矩阵乘法瓶颈（矩阵乘法 130 秒 vs 磁盘 58 秒）——统一内存，CUDA 放置层级中性；此处杠杆在于 i8mm 计算内核，而非放置策略 |
要点总结：在拥有 24 GB 内存的机器上，引擎会自动将专家缓存上限设为每层 2 个槽位，因此即使使用比开发机快 2–2.7 倍的磁盘，解码阶段仍然处于冷状态——在小内存机器上，内存上限（而非磁盘）才是真正的约束瓶颈，这与上表预测完全一致；仅 `--topp 0.7` 一项参数就带来了 1.6 倍的端到端速度提升。M5 Max 的数据点恰好落在表格的第二行：在笔记本 SSD 上运行 744B 模型，速度约为 1 token/秒——其 14 GB/s 的磁盘将瓶颈转移到了内存预算和内核上。Framework 13 的各行数据则是在同一台机器上端到端验证了缓存理论：仅通过为缓存分配内存（int8 MTP 头 + 更大的容量上限 + 学习到的固定策略），速度就从 0.29 token/秒提升至 0.37 token/秒（命中率从 28% 提升至 66%，推测机制终于在 52% 的接受率下开始生效）。容量上限部分现已实现自动化（容量自动提升功能，2026-07-10）。9950X 组合是目前最清晰的瓶颈实验——同一台机器，相同的运行历史，仅更换了磁盘：5.8 倍的磁盘带宽换来了 2.9 倍的 token 数，且性能分布从 66% 的磁盘瓶颈转变为 57% 的矩阵乘法瓶颈。但这一转折点取决于 CPU 内核：第 #101 行的 9800X3D 数据显示，在启用 OMP 热团队调优后，AVX-512 CPU 矩阵乘法速度足够快，以至于即使是 10 GB/s 的 NVMe 固态硬盘也仍然是磁盘瓶颈——此时 CUDA 专家层级带来的提升约为 0%，因为 CPU 在专家矩阵乘法上已经与 5090 相当。GPU 层级只有在 CPU 成为薄弱环节时才能发挥其显存优势，而非默认情况。（对 #101 的诚实更正：该报告的早期版本在关闭 OMP 调优的情况下运行，人为制造了虚假的矩阵乘法瓶颈转折点以及虚假的 14% CUDA 提升——这两点在重新干净运行后均不复存在。）
质量基准测试——寻求帮助
第一项测量结果已出炉（#108，感谢 dnnspaul）：int4 容器在 hellaswag/arc/mmlu 上取得了 62.5% 的平均 acc_norm（0-shot 对数似然，n=40）——低于全精度 GLM-5.2 已公布的 85–95%，但这一差距目前还不能归因于量化。有两个混淆因素横亘其中：（1）0-shot 对数似然多项选择评分方式严重不利于 GLM-5.2 这类推理模型（它根本没有思考的机会），因此即使在 fp16 下，出现较大差距也是意料之中；（2）n=40 的误差幅度为 ±14 个百分点。决定性的实验是在同一测试框架下对 OLMoE 进行 fp16 与 int4 的 A/B 对比（该模型足够小，两种精度都能运行）——两者之间的差值就是量化代价，同时评分协议带来的影响已被抵消。在完成这项实验之前，62.5% 只是一个数据点，而非最终定论。
代码已就绪，一条命令即可端到端运行（首次使用时会自动下载数据集）：
cd c
pip install tokenizers datasets # in addition to the convert deps above
./coli bench # hellaswag, arc_challenge, mmlu — 40 questions each
./coli bench hellaswag --limit 200 # one task, more questions
./coli bench mmlu arc_challenge --ram 100 # pick tasks, set a RAM budget
它会输出每个任务的准确率（对数似然评分，EleutherAI-harness 风格）。如果你能运行 OLMoE 的 fp16 与 int4 A/B 对比（或进行大样本量的 GLM 运行），请通过 issue 提交数据——这项测量结果将决定 62.5% 究竟是"int4 没问题，评分方式导致的假象"，还是"量化是瓶颈，分组量化才是优先方向"。
支持本项目
colibrì 是一个单人项目，完全在一台 12 核、25 GB 内存的笔记本电脑上编写和测试——上述数据是我在家所能测量的上限。如果你觉得这个项目有用或有趣，并希望支持其开发（更好的测试硬件将直接转化为更快的引擎，惠及所有人：真实的 NVMe 扩展数据、更大的固定缓存、在真实基准上对 int2/int3 质量进行扫描），你可以：
- ⭐ 给仓库点星并分享它；
- 🐛 提交 issue，附上你硬件上的基准测试数据；
- 💬 如果你想赞助开发或捐赠硬件，请通过 GitHub issues 联系我。
每一份贡献，无论是一个数据点还是一块硬盘，都在推动上限的提升。
仓库布局
Makefile root build/check entry point
c/
├── glm.c single-file GLM engine
├── st.h, tok.h, json.h runtime headers
├── backend_cuda.* optional CUDA tier
├── Makefile build and local checks
├── coli user-facing CLI
├── openai_server.py OpenAI-compatible HTTP gateway
├── setup.sh one-command local setup
├── tools/ offline conversion, fixtures and benchmarks
├── scripts/ long-running conversion helpers
└── tests/ dependency-free C and Python tests
web/ browser UI (pure OpenAI-API client, community-maintained)
运行时路径刻意保持扁平且易读：glm.c 及其少量头文件。辅助性的 Python 和 Shell 工具被单独分组，且永远不会成为引擎的运行时依赖。
在仓库根目录下，make、make check 和 make clean 会委托给引擎的 Makefile 执行。从 c/ 目录运行的现有命令仍可正常工作，无需修改。
为什么叫“colibrì”
蜂鸟只有几克重，能悬停在空中，每天访问上千朵花。这个引擎让一个拥有 7440 亿参数的庞然大物，仅靠蜂鸟般的“口粮”就能运行：25 GB 内存、十二个 CPU 核心，以及大量的磁盘耐心。
许可证
Apache 2.0。GLM-5.2 权重由 Z.ai 以 MIT 许可证发布。
Tiny engine, immense model. Run GLM-5.2 (744B-parameter MoE) on a consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by streaming experts from disk.
Colibrì is a lightweight, quality-preserving MoE runtime that treats VRAM, RAM, and storage as one managed memory hierarchy. Insufficient fast memory may reduce speed, but the default policy never silently changes model precision or router semantics.
$ ./coli chat
🐦 colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
✓ ready in 32s · resident 9.9 GB
› ciao!
◆ Ciao! 😊 Come posso aiutarti oggi?
See it running
The web dashboard (./coli web): a 744B model answering at 4+ tok/s end-to-end on 6× RTX 5090 — with live token metrics, the hardware panel, and the VRAM/RAM/disk expert tiers.
The Brain page: all 19,456 experts as a living cortex — colour is the storage tier, brightness is routing heat, and every expert routed in a turn flashes white. Hovering shows the expert's measured topic affinity.
Contents
- The idea
- See it running
- What's implemented
- Honest numbers
- Download the model
- Web dashboard
- Got a better machine?
The idea
A 744B Mixture-of-Experts model activates only ~40B parameters per token — and only ~11 GB of those change from token to token (the routed experts). So:
- the dense part (attention, shared experts, embeddings —
17B params) stays resident in RAM at int4 (9.9 GB); - the 21,504 routed experts (75 MoE layers × 256 experts + the MTP head,
19 MB each at int4) live on disk (370 GB) and are streamed on demand, with a per-layer LRU cache, an optional pinned hot-store, and the OS page cache as a free L2.
The engine is a single C file (c/glm.c, ~2,400 lines) plus small headers. No BLAS, no Python at runtime, no GPU required (an opt-in CUDA tier for pinned experts exists — see below).
What's implemented
- Faithful GLM-5.2 (
glm_moe_dsa) forward — validated token-exact against atransformersoracle (teacher-forcing 32/32, greedy 20/20 on a tiny-random model with the real architecture). - MLA attention (q/kv-LoRA, interleaved partial RoPE) with compressed KV-cache: 576 floats/token instead of 32,768 (57× smaller — GLM-5.2 has 64 heads and no GQA).
- DeepSeek-V3-style sigmoid router (noaux_tc, routed_scaling_factor), shared expert, first-3-dense layers.
- Native MTP speculative decoding — GLM-5.2's own multi-token-prediction head (layer 78) drafts tokens that the main model verifies in one batched forward. The head must be int8 (the converter does this by default): at int4 draft acceptance collapses to 0–4% and speculation never engages; at int8 it's 39–59% acceptance, 2.2–2.8 tokens/forward (community-measured, #8). Lossless in exact arithmetic — but not byte-identical to non-speculative greedy in practice (#100). This isn't MTP-specific: colibrì's quantized integer kernels are shape-dependent, so any batched (S>1) or GPU forward rounds slightly differently from the single-token path, and int4 GLM-5.2 sits close enough to argmax ties that such a rounding change can flip a token. MTP, the CUDA expert tier, and batched prefill are three different ways to trip the same sensitivity (community-confirmed in #100: swapping only the kernel family forks greedy output on 3/5 prompts, with zero speculation). Every emitted token is still the argmax of a valid forward — the continuation stays correct — it just isn't the same stream. For byte-exact reproducibility:
DRAFT=0(no speculation), plusIDOT=0 COLI_CUDA=0if you also want kernel-family/GPU independence. Under sampling, rejection sampling keeps the distribution correct. Honest caveat from the same measurement: on a cold cache each verified draft routes to extra experts (~660 → ~1100 expert-loads/token), so speculation can be a net time loss until the cache/pin warms up. - Grammar-forced speculative drafts (
GRAMMAR=file.gbnf, #48) — on constrained-output workloads (JSON/NDJSON, function calling, structured extraction) the grammar itself is a third draft source: wherever it admits exactly one legal byte (braces, quotes, key names, enum bodies), that forced span is tokenized and injected as pre-accepted drafts with ~1.0 acceptance — no draft head, no lookup table, and it engages even with the int4 MTP head from #8. It never constrains sampling: forced spans are verified in the same batch-union forward as any draft, so a wrong or out-of-sync grammar cannot change the output — worst case is rejected drafts, and an adaptive guard turns the source off below 50% acceptance. Byte-level GBNF subset (literals, char classes,| ( ) ? * +, comments);GRAMMAR_DRAFT=ncaps the forced span per forward (default 24). Composes withDRAFT/MTP, which fill the free-text gaps between forced spans. Full reference — mechanism, measured A/Bs, when it pays, prior art: docs/grammar-draft.md. - True sampling — temperature + nucleus, defaults tuned for int4 reality (0.7 / 0.90; the official 1.0 / 0.95 samples quantization noise from the tail).
- Integer-dot kernels (Q8_0-style int8 activations, AVX2
maddubs): int8 matmuls 1.4–2.5× faster (119 GFLOP/s measured), int4 1.8× in batch — routing decided per shape by measurement (int4 single-row stays f32: it measured slower). - MLA weight absorption (DeepSeek trick) for decode: no per-token k/v reconstruction — the query absorbs
kv_b, context is projected after attention. Validated exact: TF 32/32 and generation 20/20 with absorption forced everywhere. - Async expert readahead: while one block of experts is being multiplied, the kernel is already reading the next (
WILLNEED). - Quantization kernels: int8 / packed int4 / packed int2, per-row scales, AVX2, dequant-on-use. Packing validated bit-identical to the int8 container.
- DSA sparse attention — GLM-5.2's lightning indexer, faithful to the reference
glm_moe_dsamodeling: per-layer top-2048 causal key selection (full/shared indexer layers), auto-detected from theout-idx-*weights (--indexerconverter mode, ~189 MB extracted from the FP8 repo). Validated exact: forcing the selection to keep every key reproduces dense attention token-for-token.DSA=0disables,DSA_TOPKoverrides. - KV-cache persistence — conversations reopen warm across engine restarts: serve mode appends the compressed MLA KV to
.coli_kvafter every turn (~182 KB/token, crash-safe) and resumes it at startup with zero re-prefill. Validated byte-identical to an uninterrupted session.KVSAVE=0disables. - Router-lookahead prefetch (
PILOT=1, experimental) — the next layer's routing is 71.6% predictable from the current layer's post-attention state (measured); a dedicated I/O thread prefetches those experts while the current layer computes. - Batch-union MoE: in prefill (and MTP verification), each unique expert of the batch is read once and applied to every position that routes to it.
- Byte-level BPE tokenizer in C (GPT-2-style with Unicode-property regex, 320k merges).
- RAM safety: the expert cache is auto-sized from
MemAvailableat startup — an honest peak projection (working set, KV, MTP row, reconstruction buffers) so the kernel OOM-killer never fires. - Offline FP8→int4 converter (
c/tools/convert_fp8_to_int4.py): downloads one shard at a time (~5 GB), dequants (128×128 block scales), requantizes to the engine's container, deletes the shard — the 756 GB FP8 checkpoint never needs to exist on disk at once. Resumable.
Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)
Detailed GPU experiment: GLM-5.2 on 6x RTX 5090 — full expert residency across VRAM+RAM reaches 6.84 tok/s single-request decode.
| metric | value |
|---|---|
| model on disk (int4 container) | ~370 GB |
| resident RAM (dense, int4) | 9.9 GB |
| load time | ~30 s |
| peak RSS during chat | ~20 GB (auto-capped) |
| cold decode cost | ~11 GB disk reads/token (75 layers × 8 experts) |
| disk ceiling (this dev box's drive) | ~1 GB/s → ~0.05–0.1 tok/s cold |
| MTP speculation (int8 head) | 2.2–2.8 tok/forward measured (#8) |
This is not fast. It is a 744B frontier-class model answering correctly on a machine that costs less than one H100 fan. Warm cache, pinned hot experts and MTP push the useful-response latency down considerably; the physics of the disk does the rest.
SSD note
Cold starts are heavy on random reads (~11 GB/token), but reads don't meaningfully wear an SSD — colibrì's streaming is read-only. The real concerns under heavy use are (1) swap traffic if the system runs out of RAM (writes do wear the drive — keep a sane --ram budget; colibrì's auto-budget is designed to stay clear of swap) and (2) sustained thermals: hours at full read duty cycle will heat cheaper drives. Monitor drive temperature and health.
Download the model
A pre-converted GLM-5.2 int4 model for colibrì is available on Hugging Face — use the version with the int8 MTP heads (matey-0's clone):
⚠️ The MTP head must be int8. The original mirror (jlnsrk/GLM-5.2-colibri-int4) ships int4 MTP heads, which give 0% draft acceptance — speculation silently never engages and you lose the ~2× MTP lever. This is the single most common "why is MTP stuck at 0%?" report (#8, #102). The int8 head gives the measured 39–59% acceptance. matey-0's clone above is the original int4 model with the three
out-mtp-*files already swapped to int8 — download that one and you're done.Check what you have:
ls -l <model>/out-mtp-*· int8 (correct):3527131672 / 5366238584 / 1065950496· int4 (0% acceptance):1765523544 / 2686077736 / 536747200— if you see these, replace just those three files from the int8 mirror.
Download the repository and point COLI_MODEL to its directory:
COLI_MODEL=/path/to/GLM-5.2-colibri-int4-with-int8-mtp ./coli chat
This skips the FP8 → int4 conversion step entirely. Thanks to DatPat for the original mirror and matey-0 for the int8-head clone.
Quick start
cd c
./setup.sh # checks gcc/OpenMP, builds, self-tests
# ONE command does everything model-side: downloads GLM-5.2-FP8 shard by shard
# (never needs the full 756 GB at once), converts to the int4 container, then
# converts the MTP head for speculative decoding. Resumable at any point.
# Conversion (only) needs python with: pip install torch safetensors huggingface_hub numpy
./coli convert --model /nvme/glm52_i4 # ~400 GB free on a real ext4/NVMe path
# chat — RAM budget, expert cache and MTP are all detected automatically:
COLI_MODEL=/nvme/glm52_i4 ./coli chat
Inspect the planned storage hierarchy before loading the model:
COLI_MODEL=/nvme/glm52_i4 ./coli plan
COLI_MODEL=/nvme/glm52_i4 ./coli plan --gpu 0,1 --ram 128 --vram 48 --json
# apply the bounded plan to the normal runner
COLI_MODEL=/nvme/glm52_i4 ./coli chat --auto-tier
coli plan reads only safetensors headers and reports the model's exact dense/expert footprint, runtime RAM reserve, safe expert-cache cap, and bounded VRAM hot tier. Its versioned JSON output is intended to be shared by the CLI, API server, Web UI, and desktop shell; it does not allocate model tensors or start inference. --auto-tier applies the same plan to chat, run, serve, and benchmarks. It sets the RAM budget and context immediately; the VRAM tier is enabled only when the current glm binary is linked with CUDA. Explicit flags and environment variables keep precedence over automatic values.
Before loading the model, coli doctor performs a read-only readiness check and explains whether the selected Disk/RAM/VRAM placement is runnable:
COLI_MODEL=/nvme/glm52_i4 ./coli doctor
COLI_MODEL=/nvme/glm52_i4 ./coli doctor --gpu 0 --ram 128 --json
Doctor validates the model directory, config, tokenizer, safetensors headers, engine executable, available RAM, requested NVIDIA devices, CUDA linkage, and the same placement budget used by coli plan. It never starts glm, reads tensor payloads, imports a model framework, or creates a CUDA context. The versioned JSON report uses stable check IDs for automation. Warnings keep exit status 0; missing requirements or an unsafe RAM projection return 1, while invalid CLI values return 2.
The engine at runtime is pure C — python is only used by the one-time converter.
Windows 11 (native, no WSL)
colibrì builds and runs natively on Windows 11 x86-64 with MinGW-w64. The port adds a _WIN32 compatibility layer in c/compat.h that maps POSIX I/O to the Windows API (pread → ReadFile+OVERLAPPED, posix_fadvise no-op, aligned allocation, MoveFileEx rename, GlobalMemoryStatusEx RAM detection). All platform differences stay in compat.h; the engine source is unchanged.
Toolchain: GCC via winlibs or MSYS2 MinGW-w64. Tested with GCC 16.1.0 (x86_64-ucrt-posix-seh).
# One-time toolchain install (pick one):
scoop install mingw-winlibs # portable, no shell needed
# or: pacman -S mingw-w64-x86_64-gcc make # via MSYS2
# Build (from c/ directory):
make glm.exe # GLM-5.2 engine (static, no DLL dependencies)
make olmoe.exe # OLMoE engine (same shims)
make iobench.exe # disk I/O benchmark
make test-c # run C tests
make test-python # run Python tests (requires python)
# AVX-VNNI: Intel Alder Lake+ (and Meteor Lake+) CPUs have a 128-bit int8
# dot-product instruction (VPDPBUSD) the engine can use for ~1.3x faster
# quantized matmul. The x86-64-v3 default (portable AVX2) compiles it out;
# build for THIS machine to enable it:
make glm.exe ARCH=native # banner prints "idot: avx-vnni"
# Verify (tiny model, 2.4 MB):
pip install torch transformers safetensors huggingface_hub
python tools/make_glm_oracle.py # generate tiny oracle
SNAP=./glm_tiny TF=1 ./glm.exe 64 16 16 # expect "32/32 positions"
# Run with real model:
SNAP=D:\glm52_i4 ./glm.exe 64 4 16 # batch inference
python coli chat --model D:\glm52_i4 # interactive chat
python coli serve --model D:\glm52_i4 # OpenAI-compatible API
Warmup (overnight cache priming): the engine's expert cache learns from your workload. The included warmup.ps1 script runs coli run in a loop with diverse prompts to build the .coli_usage histogram unattended, so the next real session starts with a large, accurate hot-expert pin. Each run saves usage atomically on clean completion.
.\warmup.ps1 -Rounds 1 -Ngen 32 # ~60-90 min, durable progress
NVIDIA GPU (optional, via runtime DLL): on Windows the engine is built with MinGW gcc but CUDA kernels require MSVC + nvcc. The split is clean: build the CUDA backend into a standalone coli_cuda.dll (nvcc + MSVC), then the host glm.exe loads it at runtime via LoadLibrary (c/backend_loader.c). The host never links cudart directly; if the DLL is absent the engine falls back to CPU without error.
# Prerequisites: CUDA Toolkit + MSVC Build Tools (cl.exe) + nvcc on PATH.
# Build the DLL from a shell with the MSVC environment set (vcvars64.bat or
# "x64 Native Tools Command Prompt for VS"):
make cuda-dll CUDA_HOME="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.8" CUDA_ARCH=sm_120
# Build the host with the runtime loader (CUDA_DLL=1 adds -DCOLI_CUDA and
# links backend_loader.o instead of cudart):
make glm.exe CUDA_DLL=1 ARCH=native
# Run with the GPU expert tier (8 GB VRAM budget here; scale to your free VRAM):
$env:COLI_CUDA="1"; $env:COLI_GPU="0"; $env:CUDA_EXPERT_GB="8"
python coli chat --model D:\glm52_i4 --topp 0.7
The DLL exports 11 extern "C" symbols (coli_cuda_init, coli_cuda_matmul, etc.); backend_loader.c resolves them via GetProcAddress on first use. ColiCudaTensor* is opaque to the host (stored, never dereferenced), so the MSVC-allocated struct is safe across the ABI boundary. CUDA_ARCH must match your GPU's compute capability (e.g. sm_120 for Blackwell / RTX 50-series, sm_89 for Ada / RTX 40-series).
Status: Phase 1 complete (compiles, correct, static-linked). The Windows GPU tier (runtime coli_cuda.dll via LoadLibrary) is implemented and verified on RTX 50-series (sm_120). O_DIRECT (Phase 2) and full-model validation against the transformers oracle remain separate workstreams.
OpenAI-compatible API
coli serve keeps one model process loaded and exposes a text-only OpenAI-compatible HTTP API. The gateway uses only the Python standard library; inference still runs in the same dependency-free C engine.
cd c
COLI_MODEL=/nvme/glm52_i4 COLI_API_KEY=local-secret ./coli serve \
--host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri
curl http://127.0.0.1:8000/v1/chat/completions \
-H 'Authorization: Bearer local-secret' \
-H 'Content-Type: application/json' \
-d '{
"model": "glm-5.2-colibri",
"messages": [{"role": "user", "content": "Hello"}],
"stream": true
}'
Implemented endpoints are GET /v1/models, GET /v1/models/{model}, POST /v1/chat/completions, and legacy POST /v1/completions. Chat and completion requests support JSON responses, SSE streaming, usage counts, max_tokens/max_completion_tokens, temperature, and top_p. The extension enable_thinking: true enables GLM-5.2's reasoning block; the standard reasoning_effort field also enables it unless set to none.
The first version is deliberately text-only and serves one generation at a time: the 744B model stays in one persistent process, so concurrent HTTP requests queue instead of loading duplicate model copies. Tools, image/audio input, custom stop sequences, log probabilities, and token penalties return an explicit error rather than being silently ignored. The default bind address is localhost; set COLI_API_KEY before exposing the server beyond the machine.
Browser access from the Vite development server and Tauri local origins is enabled by default. Repeat --cors-origin https://your-ui.example to allow another exact origin, or use --cors-origin '*' only on a trusted local network.
The engine owns one mutable KV context, so HTTP generation uses a bounded FIFO admission queue instead of pretending to run unsafe parallel sequences. Configure it with --max-queue N (default 8) and --queue-timeout SECONDS (default 300), or the COLI_MAX_QUEUE / COLI_QUEUE_TIMEOUT environment variables. Saturated and timed-out requests receive OpenAI-shaped HTTP 429 errors before streaming headers are sent. GET /health exposes active/queued/completed/rejected counters, and successful generation responses include x-colibri-queue-wait-ms.
Isolated KV contexts
coli serve --kv-slots N allocates up to 16 independent sequence contexts. Requests select one with the optional integer cache_slot field; ordinary OpenAI clients omit it and keep the original slot 0 behavior.
{
"model": "glm-5.2-colibri",
"messages": [{"role": "user", "content": "Continue this conversation"}],
"cache_slot": 1
}
Each slot owns its token history, compressed MLA/DSA KV memory, MTP window, and crash-safe persistence file (.coli_kv, .coli_kv.1, ...). The engine still executes one sequence at a time; this establishes explicit KV ownership without pretending that threaded HTTP is continuous batching. RAM admission accounts for every configured slot. Use COLI_KV_SLOTS=N as the environment equivalent. Start with a small value: at the default 4096-token context, every slot costs hundreds of MB.
Experimental Metal backend (Apple Silicon)
On Apple Silicon the decode profile is matmul-bound, and unified memory removes the PCIe copy tax that keeps CUDA's streaming experts on the CPU — so colibrì has an opt-in Metal backend that runs the routed-expert SwiGLU (batched, zero-copy from the RAM slabs), the fused decode attention (full MLA layer in one command buffer, S≤4), and prefill's large GEMMs on the GPU. Token-exact vs the CPU path.
cd c
make glm METAL=1 # macOS only; no Xcode needed (shader compiles at runtime)
make metal-test # standalone kernel/attention correctness vs CPU reference
COLI_METAL=1 COLI_MODEL=/path/glm52_i4 ./coli chat --ram 96
Measured on an M4 Max (128 GB, warm cache, MTP on): CPU 0.30 → Metal 0.42 tok/s (~1.4×) (best config adds DIRECT=1; ~3× vs this machine's first cold run). Key design points: Metal's ~5 ms submit latency makes per-matmul dispatch a loss — everything is batched into few command buffers per layer, and the resident experts' GPU work is submitted before the missed experts' disk reads so I/O and compute overlap. COLI_METAL_GEMM_MIN tunes the prefill GEMM row threshold (default 16). Streaming, cache, MTP, DSA and the persistence formats are unchanged; every GPU path falls back to the CPU per-block on any fault. Numerics are dequant→f32-MAC (same as the CUDA tier); greedy outputs are byte-identical to the CPU engine.
Experimental resident CUDA backend
colibrì includes an opt-in CUDA backend for model-resident tensors. Streaming experts deliberately remain on the original CPU path for now: copying an expert from NVMe to the GPU on every use would only replace the disk bottleneck with a PCIe bottleneck. Resident quantized tensors are uploaded lazily once and reused.
cd c
make cuda-test CUDA=1 # q8/q4/q2/f32 kernel correctness
make CUDA=1
# optional dense-path experiment (hot experts are configured below)
COLI_CUDA=1 COLI_GPU=0 CUDA_DENSE=1 SNAP=/nvme/glm52_i4 ./glm 64 4 4
Requirements: Linux, an NVIDIA driver, and a CUDA Toolkit under /usr/local/cuda (override with CUDA_HOME=/path/to/cuda). CUDA_ARCH=native builds for the GPU in the current machine; set an explicit architecture when cross-compiling. Requesting CUDA with a CPU-only binary, an invalid device, or an unavailable runtime fails at startup instead of silently falling back.
The normal make build and runtime behavior are unchanged. CUDA defaults to an expert-only accelerator. CUDA_DENSE=1 additionally distributes resident dense/attention projection tensors round-robin across the selected devices; their projected footprint is reserved before the expert tier is placed. On six RTX 5090s with a 150 GB expert tier, a warmed two-request/64-token GLM-5.2 run improved from 1.650 to 2.157 aggregate tok/s (+30.8%) while retaining the full expert tier. Treat this as an opt-in until the projected dense set and the 2 GB per-device runtime reserve fit the target GPUs. A measured PIN profile can promote its hottest experts into the persistent VRAM tier while keeping the rest in RAM:
STATS=stats.txt SNAP=/nvme/glm52_i4 ./glm 64 4 4 # collect routing frequencies first
COLI_CUDA=1 COLI_GPU=0 CUDA_EXPERT_GB=16 \
PIN=stats.txt PIN_GB=160 SNAP=/nvme/glm52_i4 ./glm 64 4 4
# multi-GPU expert tier, 150 GB total budget across six 32 GB devices
COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=150 \
CUDA_DENSE=1 PIN=stats.txt PIN_GB=300 RAM_GB=226 \
SNAP=/nvme/glm52_i4 ./glm 64 4 4
# large-RAM host: fill safe VRAM, then keep every remaining expert in RAM
COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=auto \
CUDA_DENSE=1 COLI_CUDA_ATTN=1 PIN=stats.txt PIN_GB=all RAM_GB=auto \
SNAP=/nvme/glm52_i4 ./glm 64 4 4
Selected experts are uploaded during startup, so capacity failures occur before inference and the log reports their exact tensor footprint. The budget is clamped against free VRAM after reserving the projected dense resident set and 2 GB of runtime headroom per selected device. With COLI_GPUS, CUDA_EXPERT_GB is a total budget across the device set; experts are assigned whole to the least-loaded device that can hold them. Multi-GPU runs also default to PIN_FILL=1: the measured hot set is placed first, then unused VRAM is filled with zero-heat experts. CUDA_RELEASE_HOST=1 (the multi-GPU default) releases the RAM copy after a successful upload and reloads it from disk only if CUDA later fails. Set either variable to 0 to restore the conservative behavior. When host backing is released, placement is disjoint and staged: the hottest prefix is loaded, uploaded to VRAM, and freed before the next-ranked suffix is loaded into RAM. PIN_GB therefore describes the combined ranked set rather than duplicate RAM and VRAM copies. On a 256 GB dual-socket host, moving from a 150 GB VRAM + 130 GB RAM placement to 150 GB VRAM + 150 GB RAM raised fixed-token replay from 1.87 to 2.16 tok/s (+15.7%), reduced expert disk wait from 5.144s to 3.948s, and kept the projected RAM peak below RAM_GB=226. The cache cap adjusts down automatically (54 to 40 in that run) so the larger pinned tier does not exceed the process budget. Start lower on hosts with less available RAM.
CUDA_EXPERT_GB=auto fills each selected device only up to its measured free memory minus projected dense tensors and 2 GB of runtime headroom. PIN_GB=all then loads every remaining routed expert into RAM, eliminating decode-time disk misses when the host budget permits it. The regular RAM_GB guard still clamps the per-layer working cache and rejects unsafe projections; this mode is intended for dedicated high-memory inference hosts, not desktops running other workloads. On a dedicated 251 GiB host with six RTX 5090s, this mode selected a 176.7 GB VRAM expert tier and a 191.3 GB RAM tier (all 19,456 experts resident). The mode also adapts the VRAM tier every 16 emitted tokens by swapping hot RAM experts into existing GPU slots. A real 64-token greedy GLM-5.2 generation measured 6.00 tok/s decode, up from 2.20 tok/s end-to-end with the earlier 150 GB tier; expert hit rate was 100% and disk wait was zero. Prompt prefill is reported separately. This is a host-specific capacity result, not a portable default.
Text-mode timing reports prefill separately from decode. The decode rate starts after the prompt KV is built, so it is comparable to REPLAY throughput without hiding time-to-first-token. MTP speculation defaults off on CUDA because cold draft routes increase expert traffic; an explicit DRAFT=n still overrides the default.
On six RTX 5090 32 GB cards with GLM-5.2 int4, a 150 GB hot-first tier sustained 0.94 token/s over a 64-token varied prompt (87.8% expert hit rate), and reached 1.64 token/s on a warmed short prompt (99.3% hit rate). The same capacity filled without routing heat managed only 0.29 token/s, so profile quality matters more than raw VRAM capacity. These are single-run engineering measurements, not a portable performance guarantee.
Current limitations: devices use independent contexts and synchronous host-staged activation copies—there is no P2P/NCCL dependency yet. Independent expert groups execute concurrently across devices, but a single expert is not sharded. The kernels are correctness-first custom kernels rather than cuBLAS/Tensor Core kernels.
For a reproducible backend A/B without the full checkpoint, generate the deterministic 313M-parameter glm_moe_dsa fixture and run fixed-token replay:
cd c
python tools/make_glm_bench_model.py --output /nvme/colibri-bench-medium --device cuda
python tools/benchmark_cuda_fixture.py --model /nvme/colibri-bench-medium --gpu 0
The fixture has random weights and is not a language model. It exists only to preserve the real MLA/MoE/streaming shapes and compare CPU streaming, dense-only CUDA, CPU hot-store, and CUDA hot-expert execution with identical replay tokens.
Web interface
web/ contains a community-contributed browser UI (React + TypeScript, ~390 lines of source, a pure API client — it never touches the engine directly):
cd web
npm ci && npm run dev # then point it at an OpenAI-compatible endpoint
It speaks the standard OpenAI Chat Completions protocol with SSE streaming, so it works against the colibrì OpenAI-compatible server (in review, #21) or any other compatible endpoint. Nothing leaves the endpoint you configure. The terminal coli chat remains the first-class interface.
Useful knobs (env or flags): --temp T token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy), --topp 0.7 adaptive expert top-p (30–40% less disk), --ngen N max tokens per answer (:more in chat continues a truncated one), --repin N adapt RAM/VRAM hot experts every N emitted tokens, AUTOPIN=0 disable the learning cache's auto-pin, THINK=1 enable GLM-5.2's reasoning block, DRAFT=n MTP draft depth, GRAMMAR=g.gbnf grammar-forced drafts for constrained JSON/NDJSON output (GRAMMAR_DRAFT=n caps the forced span), TF=1 teacher-forcing validation, PILOT=1 router-lookahead disk prefetch (experimental — see below), CAP_RAISE=0 don't auto-grow the expert cache.
Resource policy
coli plan reports the planned hot (VRAM), warm (RAM), and cold backing (disk) tiers, the reason for each placement, and the expected bottleneck. The default --policy quality and --policy balanced modes preserve checkpoint quantization and router decisions unless --topk or --topp is passed; those explicit lossy overrides print a warning and proceed.
Auto-tier plans size OpenMP from physical cores and bind workers across cores. Memory-bound quantized kernels can regress sharply when SMT siblings compete for limited memory channels; explicit OMP_* settings always take precedence.
coli plan --model /models/glm52_i4 --policy quality
coli run --auto-tier --policy quality "Explain MoE offloading"
# Explicit research-only router reduction:
coli run --policy experimental-fast --topk 4 "Benchmark prompt"
Disk is an immutable recovery source, not a normal decode target. If the plan leaves cold expert bytes on disk, speed depends on cache hit rate; output quality does not.
Cold expert reads use a deferred pipeline: resident RAM/VRAM experts execute while missing experts are loaded in a bounded background I/O pool, then the cold results join before the layer completes. IO_THREADS=n overrides the default eight loader threads when foreground work exists. Profiling reports both disk service time and the smaller foreground-visible wait time so overlap is explicit rather than credited as unexplained speedup.
--policy balanced enables lossless live placement (REPIN=64). At safe request boundaries, a per-layer LFRU score combines decaying session frequency with recent access and replaces at most four sufficiently colder pinned experts. --policy quality leaves live replacement off by default; REPIN=0 always disables it. Persistent .coli_usage history and session-local LFRU state remain separate.
For single-token q4 CPU experts, gate and up projections share one OpenMP dispatch while retaining the same per-row AVX2/NEON arithmetic. This removes one thread-team launch per RAM expert without activation requantization or a lower-precision fallback. It is a stepping stone toward a persistent native CPU expert pool, not a replacement for one.
The expert cache auto-sizes to your RAM (since 2026-07-10): the engine now raises the LRU cap to fill your --ram budget instead of only lowering it. Before this fix a 128 GB machine ran with the same 8-experts/layer cache as a 16 GB one (issue #12) — if you benchmarked colibrì before this date, rerun: your numbers were capped.
Router-lookahead prefetch (PILOT=1, experimental): GLM-5.2's expert routing is measurably predictable ahead of time — applying layer L+1's router to layer L's post-attention state recalls 71.6% of the true top-8 (vs 41.3% for "same experts as last token"). PILOT=1 uses this to issue next-layer expert readahead from a dedicated I/O thread while the current layer computes. On our dev box the disk is already ~80% saturated, so it measures neutral; on machines where compute and disk are balanced (like the Ryzen AI 9 in issue #12: 43% disk / 46% matmul) it should overlap real work — measurements welcome.
The learning cache: the engine records which experts your usage actually routes to (.coli_usage next to the model, updated every turn) and at startup automatically pins the hottest ones in spare RAM. colibrì literally gets faster the more you use it.
Live tier adaptation (--repin N, opt-in): at safe turn boundaries, a decaying session heat map replaces cold pinned experts with hotter streamed experts. Replacement loads the expert from disk into the existing RAM slot; GPU-backed slots immediately refresh the same VRAM tier budget. A 25% hysteresis and a four-swap limit prevent tier thrashing. Persistent .coli_usage remains the long-term signal and is not decayed.
Conversations reopen warm (.coli_kv, since 2026-07-10): coli chat persists the compressed MLA KV-cache to disk after every turn (~182 KB/token, appended incrementally, crash-safe). Close the chat, reopen it tomorrow — the model still remembers the whole conversation and zero re-prefill happens: validated byte-identical to an uninterrupted session. :reset clears it, KVSAVE=0 disables it.
Web dashboard
One command serves the OpenAI-compatible API and the web console on the same port, then opens your browser when the engine is ready:
cd web && npm install && npm run build # once
./coli web --model <model-dir>
What you get:
- Chat with live metrics: a flashing token counter while generating, then tok/s, time-to-first-token, prompt→completion counts and queue wait;
- Runtime panel: your hardware (CPU, GPUs + VRAM, RAM, cores), the scheduler, and the live expert-tier bar — how many of the 19,456 experts sit in VRAM / RAM / disk right now;
- Brain: the whole model as a 76×256 cortex, one cell per expert. Colour = tier, brightness = routing heat, and the experts routed in each turn flash white and decay — you watch the model think. Hover any cell for its tier, heat and measured topic affinity (specialists for code, Chinese, math, law… live in layers 11–22).
The dashboard talks to the engine over two tiny protocol lines (TIERS, EMAP/HITS) and plain JSON endpoints — nothing heavier than the engine itself.
Got a better machine? Try it — here's what to expect
colibrì was built on deliberately humble hardware (12 cores, 25 GB RAM, an older DRAM-less NVMe behind a WSL2 VHDX that measured ~1 GB/s random on this drive — note WSL2 VHDX is not inherently slow: a community 5090 box measured 10.5 GB/s O_DIRECT through one, #101). Every one of those constraints is a knob your machine can turn up. The engine needs: Linux (or WSL2), macOS, or Windows 11 natively (MinGW-w64); gcc with OpenMP, AVX2, ≥16 GB RAM, and the ~370 GB int4 model on a local NVMe (ext4/NTFS — never a network/9p mount).
How to test it, in order:
cd c && ./setup.sh # build + architecture self-test (expects 32/32)
# 1) measure YOUR disk the way the engine uses it (parallel 19 MB random reads):
gcc -O2 -fopenmp iobench.c -o iobench
./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 0 # buffered, 8 threads
./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 1 # O_DIRECT (bypass cache)
# Caveat (#86): iobench reads a bounded ~1 GB shard, so buffered reads on a big-RAM box
# report the PAGE CACHE, not the disk. Use the O_DIRECT run (arg 1) for a true number, and
# run it on a shard you haven't touched this session (a prior buffered run caches its pages).
# On macOS there is no O_DIRECT — iobench uses F_NOCACHE, which stops *new* caching but can't
# evict pages a prior buffered run already resident-mapped, so a macOS "O_DIRECT" figure right
# after a buffered run still reads cache. Reboot or use a fresh shard for a real cold read.
# 2) chat; watch the per-turn stats line (tok/s, expert hit-rate, RSS):
COLI_MODEL=/path/to/glm52_i4 ./coli chat
# 3) record expert usage, then pin the hottest experts in your spare RAM:
STATS=stats.txt ./coli chat
PIN=stats.txt PIN_GB=20 ./coli chat # scale PIN_GB to your free RAM
# 4) quality benchmarks (MMLU/HellaSwag/ARC):
./coli bench
Back-of-envelope predictions (decode is disk-bound: a cold token costs ~11.4 GB of expert reads; MTP speculation roughly halves the effective cost once the cache is warm; RAM turns cold reads into free cache hits):
| machine | expected |
|---|---|
| this dev box (WSL2 VHDX, ~1 GB/s, 25 GB RAM) | ~0.05–0.1 tok/s cold — proven baseline |
| native Linux, PCIe4 NVMe (~3–5 GB/s random), 32 GB | ~0.5–1 tok/s |
| PCIe5 NVMe or 2×NVMe RAID0 (~8–12 GB/s), 64 GB (PIN ~40 GB of hot experts) | ~2–4 tok/s |
| 128–256 GB RAM, 12 cores (hot experts cached) | ~2–4 tok/s — matmul-bound: ~80 GFLOP/token vs ~250 GFLOP/s of our AVX2 kernels |
| same RAM + 24–32 cores, or AVX-512/VNNI kernels | ~5–15 tok/s — interactive; kernel work is the multiplier |
These are estimates, not measurements — if you run colibrì on serious hardware, please open an issue with your numbers: real datapoints from better machines are exactly what this project needs next.
Community benchmarks (measured)
Real numbers from real machines, stock build (setup.sh, gcc 13), greedy decoding, --ngen 32, MTP active:
| machine | disk (iobench, 19 MB × 64, 8 threads) | config | measured |
|---|---|---|---|
| Intel Core Ultra 7 270K Plus (24 threads) · WSL2 · 24 GB RAM · NVMe VHDX (#2) | 1.96 GB/s buffered · 2.74 GB/s O_DIRECT | default | 0.07 tok/s · expert hit 3–4% · RSS 14.1 GB |
| 〃 | 〃 | --topp 0.7 | 0.11 tok/s · expert hit 11% · RSS 14.7 GB |
| Apple M5 Max (18 cores) · macOS · 128 GB unified · internal SSD (#4, #5) | ~4 GB/s cold (the 14.2 GB/s reading was cache-influenced — see note) | default, MTP off | 1.06 tok/s · expert hit 23% · RSS 21.8 GB |
| Apple M5 Max · macOS · 128 GB unified · 2 TB SSD · Metal backend (#72, #87) | (macOS O_DIRECT figure unreliable — see note) | Metal on · --ram 96 · 39.7 GB warm pin · MTP off | 1.83 tok/s · expert hit 66% · warmed 1.11 → 1.83 over the run |
〃 · 46.9 GB pin (2.94M-selection history) · --ram 110, 1024-token run (#103) | 〃 | Metal on (experts + attention) · MTP off | 2.06 tok/s · hit 72.5% · coherent output · fastest datapoint yet (still on the pre-rebase Metal branch) |
| Mac Mini M4 Pro · macOS · 48 GB unified · Metal backend (#107) | 6.59 GB/s F_NOCACHE (fresh shard) | Metal on · --ram 38 | 0.30 tok/s (vs 0.18 CPU-only) — entry Apple Silicon on a third the RAM beats the 32-core 9950X row |
| Epyc 9654 ES · Linux · 4x16GB DDR5-4800-rdimm · Samsung PCIe Gen3 x4 NVME SSD | — | MTP=1 DIRECT=1 | 0.31 tok/s · expert hit 35% · RSS 21.52 GB |
| Ryzen AI 9 HX 370 (Framework 13) · Arch Linux · 128 GB · WD SN850X, BTRFS zstd (#12) | — | int8 MTP head · --cap 32 · 46.7 GB auto-learned PIN | 0.37 tok/s · expert hit 66% · MTP acceptance 52% (2.59 tok/fw) · RSS 105 GB |
| Ryzen 9 9950X (32 threads) · Linux · 123 GB · Crucial P3 QLC Gen3 (#31) | 1.51 GB/s buffered | default, 2 runs from cold | 0.10 tok/s · hit 53% · profile 66% disk |
| 〃 same machine, model moved to a Samsung 9100 PRO PCIe 5.0 (#31) | 8.81 GB/s O_DIRECT | 〃 (usage history retained) | 0.28 tok/s · hit 57% · profile flips: 32% disk / 57% matmul |
| Ryzen AI Max+ 395 (Framework Desktop) · Ubuntu · 128 GB LPDDR5x · Intel Optane 905p PCIe 3.0 (#39) | 3.27 GB/s buffered | int8 MTP head · fresh history (pure LRU, auto-raised cap 65) | 0.16 tok/s · hit 57% · profile 49% disk / 47% matmul |
| 〃 five runs later — learned pin 47.6 GB (#39) | 〃 | --temp 0.7 --topp 0.7 | 0.40 tok/s · hit 71% · fastest non-Apple datapoint |
| Ryzen 7 9800X3D (16T) · WSL2 · 70 GB RAM · Samsung 9100 PRO PCIe 5.0 · RTX 5090 (#101) | 10.51 GB/s O_DIRECT | MTP off · learned pin 24 GB · hit 54% · OMP hot-team on | 0.41 tok/s · disk-bound (36.5 s disk vs 24.0 s matmul) · CUDA expert tier ≈ 0% (AVX-512 CPU matches the 5090) · --topp 0.7 → 0.52 tok/s |
| EPYC 7443 (24C/48T, Zen3 AVX2) · Linux · 430 GB RAM · NVMe RAID-Z1 via TrueNAS VM (#104) | ~1 GB/s (VM overhead) | 77.5 GB pin · cap auto-raised to 194/layer · MTP off | 1.00 tok/s · hit 98% · disk eliminated → RAM-bandwidth + matmul bound (no AVX-512/VNNI on Zen3) |
| Intel i5-12600K (10C/16T, AVX2) · native Windows 11, no WSL · 32 GB · MinGW GCC 16.1 (#113) | buffered (no O_DIRECT on MinGW) | int8 MTP head · cold, small-RAM (cap ~2/layer) | 0.08 tok/s · hit 3.7% · MTP 57% acceptance — first native-Windows datapoint, port validated |
| Ryzen 9 9950X3D2 (16C/32T, avx512-vnni) · native Linux · 121 GB · Samsung 9100 PRO PCIe Gen5 · RTX 5090 (28 GB expert tier, 1475 pinned) (#120) | 11.48 GB/s O_DIRECT | MTP=0 DIRECT=1 PIPE_WORKERS=16 PREFETCH=1 | 1.23 tok/s · MTP-off wins disk-bound · fastest x86 datapoint yet |
| Ryzen AI Max+ 395 (Strix Halo, 16C/32T Zen5, avx512-vnni) · Arch Linux · 128 GB unified LPDDR5x · SK hynix P41 PCIe 4.0 (#124) | — | DIRECT=1 PIPE=1 --topp 0.7 · auto-pin | 0.06 cold → 1.10 tok/s sustained · first Strix Halo / gfx1151 datapoint (unified memory: no discrete VRAM tier) |
| Intel Core Ultra 9 185H (16C/22T, avx-vnni) · native Windows 11, no WSL · 32 GB · Crucial P3 QLC NTFS · RTX 5070 Ti (unused) (#128) | — | int8 MTP head · with #131 (pipe + RAM fixes), warm cache, no GPU | 0.03 cold → 0.5 tok/s warm (~7-prompt warmup) · cache-warming on native Windows once the portability blockers are fixed — stock main hung on the \r\n READY sentinel before #131 |
| Dell Pro Max GB10 (DGX Spark: Grace 10×X925 + 10×A725, aarch64 i8mm/sve2) · Linux · 121 GB unified LPDDR5x · Dell OEM 4 TB NVMe · GB10 sm_121 (#136) | 5.58 GB/s O_DIRECT (NVIDIA-OEM unit in #76 was 10.74 — same platform, different SSD) | int8 MTP head · warm cache | 0.21 cold → 0.50 tok/s warm · hit 83% · MTP 73% (3.20 tok/fw) · matmul-bound (matmul 130 s vs disk 58 s) — unified memory, CUDA placement tier neutral; the lever here is an i8mm compute kernel, not placement |
Takeaways: with 24 GB of RAM the engine auto-caps the expert cache to 2 slots/layer, so decode stays cold even on a disk 2–2.7× faster than the dev box — on small-RAM machines the RAM cap, not the disk, is the binding constraint, exactly as the table above predicts; --topp 0.7 alone bought a clean 1.6× end-to-end speedup. The M5 Max datapoint lands right on the table's second row: ~1 tok/s of a 744B model on a laptop SSD — and its 14 GB/s disk shifts the bottleneck back to RAM budget and kernels. The Framework 13 rows are the cache thesis proven end-to-end on one machine: 0.29 → 0.37 tok/s (hit 28% → 66%, speculation finally engaging at 52% acceptance) just by giving the cache its RAM — int8 MTP head + a bigger cap + the learned pin. The cap part is now automatic (cap auto-raise, 2026-07-10). The 9950X pair is the cleanest bottleneck experiment yet — same machine, same history, only the disk swapped: ×5.8 disk bandwidth bought ×2.9 tokens, and the profile flipped from 66% disk to 57% matmul. But the crossover depends on the CPU kernel: the 9800X3D row (#101) shows that with the OMP hot-team tuning on, the AVX-512 CPU matmul is fast enough that even a 10 GB/s NVMe stays disk-bound — and there the CUDA expert tier buys ≈ 0%, because the CPU already matches the 5090 on expert matmul. The GPU tier earns its VRAM only when the CPU is the weak link, not by default. (Honest correction from #101: an earlier version of that report ran with the OMP tuning off, which manufactured a false matmul-bound crossover and a false +14% for CUDA — neither survived a clean re-run.)
Quality benchmark — help wanted
First measurement is in (#108, thanks dnnspaul): the int4 container scored 62.5% mean acc_norm on hellaswag/arc/mmlu (0-shot log-likelihood, n=40) — below the 85–95% published for full-precision GLM-5.2, but the gap is not yet attributable to quantization. Two confounds sit in the way: (1) 0-shot log-likelihood MC scoring badly underserves a reasoning model like GLM-5.2 (it never gets to think), so a large gap is expected even at fp16; (2) n=40 is ±14pp. The decisive experiment is the OLMoE fp16-vs-int4 A/B under this same harness (small enough to run both precisions) — that delta is the quantization cost with the scoring protocol cancelled out. Until it's run, 62.5% is a datapoint, not a verdict.
The code is here and ready; one command runs it end to end (it auto-downloads the datasets on first use):
cd c
pip install tokenizers datasets # in addition to the convert deps above
./coli bench # hellaswag, arc_challenge, mmlu — 40 questions each
./coli bench hellaswag --limit 200 # one task, more questions
./coli bench mmlu arc_challenge --ram 100 # pick tasks, set a RAM budget
It prints per-task accuracy (log-likelihood scoring, EleutherAI-harness style). If you can run the OLMoE fp16-vs-int4 A/B (or a large-n GLM run), please open an issue with the numbers — it's the measurement that turns 62.5% into either "int4 is fine, scoring artifact" or "quantization is the ceiling, grouped-scale is the priority."
Supporting the project
colibrì is a one-person project, written and tested entirely on a 12-core laptop with 25 GB of RAM — the numbers above are the ceiling of what I can measure at home. If this project is useful or interesting to you and you'd like to support its development (better test hardware translates directly into a faster engine for everyone: real NVMe scaling data, bigger pinned caches, int2/int3 quality sweeps on real benchmarks), you can:
- ⭐ star the repo and share it;
- 🐛 open issues with benchmark numbers from your hardware;
- 💬 reach out via GitHub issues if you'd like to sponsor development or donate hardware.
Every contribution, from a datapoint to a disk, moves the ceiling.
Repo layout
Makefile root build/check entry point
c/
├── glm.c single-file GLM engine
├── st.h, tok.h, json.h runtime headers
├── backend_cuda.* optional CUDA tier
├── Makefile build and local checks
├── coli user-facing CLI
├── openai_server.py OpenAI-compatible HTTP gateway
├── setup.sh one-command local setup
├── tools/ offline conversion, fixtures and benchmarks
├── scripts/ long-running conversion helpers
└── tests/ dependency-free C and Python tests
web/ browser UI (pure OpenAI-API client, community-maintained)
The runtime path intentionally stays flat and readable: glm.c plus its small headers. Auxiliary Python and shell tooling is grouped separately and is never a runtime dependency of the engine.
From the repository root, make, make check, and make clean delegate to the engine Makefile. Existing commands run from c/ continue to work unchanged.
Why "colibrì"
The hummingbird weighs a few grams, hovers in place, and visits a thousand flowers a day. This engine keeps a 744-billion-parameter giant alive on hummingbird rations: 25 GB of RAM, twelve CPU cores, and a lot of disk patience.
License
Apache 2.0. GLM-5.2 weights are released by Z.ai under MIT.