Pipette changes how we benchmark on-device AI: treat deployment as the unit of measurement
Published by AINave Editorial • Reviewed by Ramit
On-device model performance has always been a guess. Model card numbers come from server GPUs at full precision, not from a phone thermal-throttling through a Q4KM quant. Liquid AI's Pipette changes that assumption: treat on-device behavior as a property of the full deployment stack, not just the model alone.
What Pipette actually measures
Pipette is an open-source benchmarking platform for edge devices, built with Artificial Analysis as an independent methodology validator. Its unit of measurement is a full configuration: model + quantization + runtime + device. The launch dataset covers five on-device performance metrics across more than 1,000 model x quantization x runtime x device x context configurations, spanning 30+ models, llama.cpp builds for macOS, iOS, Windows and Android, and context lengths from 256 to 8,192 tokens.
Initial verified results come from a MacBook Pro with M5 Max, an iPhone 17 Pro, and a Galaxy S26 Ultra. One data point shows two 350M models at the same quantization on the same phone retaining 78.4% and 33.8% of decode throughput at 4,096 tokens. That gap is the difference between a usable on-device experience and a frustrating one.
Why this matters for your edge deployments
If you are shipping an on-device agent, a local RAG pipeline, or a mobile chatbot, your deployment decision matters more than the model name. Pipette makes that measurable and reproducible. Every submission records benchmark version, token shape, model artifact, quantization, runtime version and settings, and device hardware and OS. Performance runs follow a published protocol with fixed token shapes, greedy decoding, a discarded warm-up, and readiness gating that verifies thermal and load conditions before each timed repetition. Runs that fail the gate are not published.
The project also ties quality metrics back to on-device configurations using IFBench, GPQA Diamond, and MATH-500. Note that those quality scores currently come from llama.cpp evaluation runs on NVIDIA H100 80GB reference systems, then get matched to on-device runs sharing the same model and quantization. A quality number shown next to phone throughput was not produced on the phone.
What's still missing
Pipette is useful but young. Community-submitted results are still in beta. AMD Ryzen AI Max+ 395 and Radeon 8060S results are listed as coming soon. The framework itself is clearly documented, but the community contribution pipeline is not yet fully open. If you are building for a specific niche device, you may need to run your own benchmarks and wait for broader coverage. The methodology is sound, but the dataset is still thin for less common hardware.
For builders shipping on-device AI, Pipette provides the first serious tool to stop guessing and start measuring the full deployment configuration. The source code is Apache 2.0, and the public dataset is available today. Whether you use it or not, the shift in thinking -- treat deployment as the unit of measurement -- is worth adopting.
