FreeToken: Edge-Native MoE Serving Runs 753B Models on a Single Workstation
Published by AINave Editorial • Reviewed by Ramit
FreeToken, a new edge-native mixture-of-experts serving engine from UC Berkeley and UT Austin, claims to run models as large as 753B GLM-5.2 on a single workstation GPU FreeToken runs 753B GLM-5.2 on a single RTX PRO 6000. For AI builders, this could mean running frontier open-weight models locally without leasing datacenter hardware, but the performance claims come from a single article and need independent verification.
What FreeToken Brings to Local Inference
FreeToken treats a personal machine as a unified, elastic inference platform rather than a fixed small GPU. It continuously maps computation and model state across whatever GPU, CPU, memory, and interconnect bandwidth the machine has. The result: a 35B model at interactive speed on an 8 GB laptop GPU, 284B on a gaming desktop, and 753B on a single workstation card.
The magic is in mixture-of-experts sparsity. DeepSeek-V4-Flash activates only 6 of its 256 routed experts in each of 43 layers, so just 13B of its 284B parameters participate in any single token. This sparsity makes local inference arithmetically feasible even though the full model requires roughly 140 GB at FP4.
FreeToken is Apache-2.0 licensed on GitHub, published on PyPI as freetoken v0.1.2 (pip install "freetoken[accel]"), and ships as a one-click desktop app for Windows and Linux at flashml.ai. The CLI exposes OpenAI- and Anthropic-compatible endpoints on port 1919 via ft serve, and ft launch can wire Claude Code, Codex, OpenCode, or OpenClaw against your own box.
Performance Numbers and Where to Be Skeptical
On an RTX 5090, FreeToken sustains 77-83 tok/s on Qwen3.6-35B-A3B and 22-25 tok/s on DeepSeek-V4-Flash -- 1.5-2.3x the strongest baseline tested. Decode stays within 12% of the single-turn rate across three agentic workloads, and worst-case time-to-first-token stays below 44 seconds in every cell tested. By comparison, llama.cpp hit 232 s, Ollama 179 s, and KTransformers 946 s in some cases.
On an 8 GB RTX 4060 laptop, the NVFP4 build serves 35B at 39.3 tok/s -- above the 33 tok/s median decode speed measured for Codex in production traces. On a single RTX PRO 6000, GLM-5.2 (753B, 40B active) runs at 14.9 tok/s versus llama.cpp's 7.3.
These benchmarks are reported in one article. They are not yet independently confirmed. Builders should treat them as directional until tested on their own hardware.
Who Benefits from a Local Serving Engine
FreeToken targets solo developers, startups, and SMB engineering teams whose per-token API bills already exceed the cost of a GPU they own. Enterprises are advised to use it as an air-gapped or regulated-workload path rather than a datacenter replacement. The strongest industry fits include healthcare, legal, defense, finance, and IP-heavy R&D where data must never leave the machine.
Typical use cases: local coding agents, private code review, offline contract analysis, synthetic-data generation, and batch evals.
What You Need to Run FreeToken
The CLI targets Linux x86_64 with an NVIDIA GPU on driver r580+ (CUDA 13). Specific CPU and RAM requirements are not detailed in the available source. The desktop app covers Windows and Linux. FreeToken uses FP4 quantization for large models, which may affect output quality compared to higher precision. Also, while only a small fraction of parameters are active per token, inactive experts still occupy roughly 140 GB of host memory for DeepSeek-V4-Flash, so the machine needs enough RAM to hold them.
FreeToken is an interesting engineering contribution, not a finished product. If the benchmarks hold under real-world agentic workloads, it could meaningfully reduce inference costs for builders running local coding agents or private analysis pipelines. For now, treat the claims as intriguing but unverified.
