Kog Inference Engine Claims 30x Faster LLM Inference on Standard GPUs
Published by AINave Editorial • Reviewed by Ramit
French startup Kog is betting that software optimization can squeeze far more inference performance out of standard datacenter GPUs than most developers expect. The company's Kog Inference Engine (KIE) claims up to 30x faster LLM decoding on existing hardware, but the evidence so far is limited to a small custom model.
The Claim: 3,000 Tokens Per Second on Commodity GPUs
Kog demonstrated its engine in May using Laneformer 2B, a model with roughly 2 billion parameters, on AMD MI300X and Nvidia H200 GPUs. According to TechCrunch, it achieved 3,000 tokens per second per request for single-request decoding. The company says this represents a 30x speedup over baseline inference on the same hardware. CEO Gael Delalleau, whose background combines solid-state physics and DEFCON hacking, believes newer GPUs have memory bandwidth that is left untapped by conventional software stacks.
Why Builders Should Pay Attention
If KIE's approach generalizes to larger models, the practical implications are significant. Faster single-request decoding directly reduces latency for professional AI workflows like code generation, game creation, and interactive app building. Kog reports 200 tangible business leads from its tech preview, indicating real demand. The company is now focusing on larger models after learning that prospective customers are unwilling to fine-tune small models. Delalleau told TechCrunch that software engineering will likely be the first use case, targeting users who today wait hours for tools like Claude Code.
The Road Ahead: 10x Speed by September
Kog plans to ship its first major model at 10x speed by September, a milestone that would also enable the company to demonstrate customer traction and raise a Series A round. The startup's seed round was co-led by Varsity VC with involvement from Kamel Zeroual, and it has backing from Scaleway, Bpifrance, and the French Tech 2030 program. Longer term, Kog hopes to use agent-based pipelines to support more GPU architectures and models, which could contribute to European AI sovereignty goals.
Caveats: A Long Way to Go
The current evidence rests entirely on a demo with Laneformer 2B. Whether the same software optimization can deliver comparable speedups on large language models with hundreds of billions of parameters is unproven. Kog's approach is also highly manual: each new GPU generation requires weeks to months of deep engineering research at the assembly and binary level. With a team of 11, this limits how many chips the company can support in the near term. Skeptics will note that many inference optimization startups have made bold claims that did not generalize. Kog's September test with a major model will be the real proof point.
