French AI Startup Kog Bets on Software to Accelerate GPU Inference

AI inference acceleration competition is heating up. French startup Kog has opted not to develop its own chips but instead bets on optimizing underlying software to further unleash the inference capabilities of existing data center GPUs.

Targeting High Latency Scenarios in Enterprises

The company gained attention in May this year due to a technology preview. Kog showcased that using standard data center GPUs like AMD MI300X and Nvidia H200 can achieve extremely fast single-request decoding speeds. The company later stated that this demonstration generated about 200 clear business leads.

Kog is currently prioritizing professional workflows that require high response speeds, particularly in software engineering scenarios. According to the company, some heavy code generation users may wait for results for hours, and inference latency has become a significant issue affecting user experience and costs.

The company is also working with several design partners to advance application deployment. These clients allow users to generate games or applications through prompts, and improved inference speeds typically mean higher conversion efficiency and revenue potential.

Small Models Achieving 3000 TPS

Kog previously set a goal of "accelerating large model inference by 30 times," but current public demonstrations are still mainly based on small models. The demonstration used a Laneformer 2B model with about 2 billion parameters, achieving a single-request inference speed of 3000 tokens per second. This model is now open-sourced.

However, market demand has not remained focused on fine-tuning small models. Kog stated that potential clients are not prepared to focus on small models, so since the release of the demonstration, the company has shifted its main efforts toward accelerating the development of larger models to meet actual demand.

Validating Large Model Capabilities Around September

Kog believes that the potential of GPUs in inference decoding has not yet been fully realized. CEO Gaël Delalleau stated that as the memory bandwidth of the next generation of GPUs continues to improve, there is still significant room for optimization at the software level.

The cost of this approach is a longer R&D cycle. Kog stated that for each new GPU adaptation, the team often needs to invest several weeks or even months in hardware-level research. Given the current team size of 11, the number of chips the company can support in the short term remains limited.

Kog plans to gradually integrate this method into agent-based R&D processes in the future to support more chips and models. The more critical milestone now is to first prove that this approach can work with large models. Delalleau expects the company to achieve the first major model acceleration of about 10 times around September and use this to showcase client progress, driving subsequent Series A funding.