mxfp4

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

A SOTA quantization toolkit for high-accuracy low-bit LLM inference|简洁且高效的量化工具包

The Kimi K3 and DeepSeek-V4-Flash-0731 architectures, reimplemented from scratch in pure Rust (zero dependencies) and verified 1:1 against the official code.

CPU inference for Kimi K3, a 2.78T-parameter MoE LLM, in pure Rust. No GPU, no BLAS, no PyTorch. Streams the checkpoint from disk. Byte-identical port of kimi-k3-in-c.

Tiny engine, immense models — run large MoE LLMs (gpt-oss, Mixtral, Qwen3-MoE) on ordinary machines by streaming experts from disk. OpenAI-compatible server with tool calling + hybrid cloud relay; CPU, Apple Silicon & CUDA (MLX).

vLLM for AMD RDNA4 (gfx1201): Radeon AI PRO R9700 & RX 9070 XT — native MXFP4 linear kernel + spec-decode verify fix, carried as rebased branches (fork-carry model)

面向 8×RTX 4090（SM89）的 DeepSeek-V4-Flash-0731 MXFP4 SGLang 实验分支

Run Kimi-K3 (2.8T MoE) from SSD with a tiny readable C++17 engine—native MXFP4, io_uring, and O_DIRECT.

Run the official Kimi K3 MoE checkpoint on one 128 GB AMD Strix Halo box. ROCm-resident static weights, MXFP4 experts streamed from NVMe via io_uring. C engine, chat client, OpenAI-compatible server. Very experimental.

Qwen3-8B quantization study across vLLM, TensorRT-LLM, AutoRound, INT8, and MXFP4

Local inference engine for DeepSeek-V4-Flash (284B MoE, MXFP4 experts streamed from disk) on consumer hardware — C11, zero deps, bit-exactness gated. macOS / Linux / Windows.

Run a 2.78T parameter MoE LLM on CPU in pure Rust with only 8 GB RAM.

Field notes from optimizing LLM inference on an NVIDIA DGX Spark. Everything measured on the machine.

GRIT — Grouped Reduced-precision Interchange Type: a checkable 64-byte descriptor + contract for block-scaled, sparse, sub-byte tensors. Spec, conformance vectors, and 5 zero-dependency implementations (C, C++, Rust, Python, TypeScript). grit scan audits GGUF/safetensors files you already have.

CPU-first Rust research implementation of OpenAI GPT-OSS with exact MXFP4 kernels, reproducible benchmarks, and archived accelerator research.

Run OpenAI's gpt-oss-20b on AMD Radeon (RDNA2) via a from-scratch Vulkan compute backend. No CUDA, no ROCm, no Triton.

MXFP/NVFP/FP16 to GGUF conversion with automatic safe routing

gpt-oss-120b on one NVIDIA DGX Spark at 65 tok/s single-stream and 300 tok/s aggregate at 30 users — five patches on upstream vLLM plus SM121 CUTLASS MXFP4 kernels, with the full measurement campaign and every negative result

A one-command inference server and benchmark harness for running large language models that don't fit in your GPU's VRAM. Optimized for RTX PRO 6000 and Deepseek v4 flash

Serving DeepSeek-V4-Flash-0731 at 1M context across 2x DGX Spark (GB10) with TP=2, DSpark speculative decoding and NVFP4 KV cache. Measured findings: why PP is impossible, DSpark's 32-stream ceiling, speculation 1.4-1.9x, MXFP4-vs-NVFP4.