This week (17/07) Moonshot announced the launch of Kimi K3. 2.8 trillion parameters, 1 million context window and frontier-level performance, all under an open-source license. Whilst Moonshot remains humble about the performance in respect to recent proprietary models, the benchmarks show clear-cut performance, just behind the flagship models. If you ask us, reason enough to be excited. Let's take a deep-dive into the technical side of Kimi K3!
Kimi K3 has 2.8 trillion parameters in its architecture, which is significantly more compared to the latest open-source release of DeepSeek 1.6T.
Now you might be asking yourself: why would I care about parameter count? Well, generally speaking (if we simplify a few highly technical limitations) more parameters means more performance. You may visualize this as pixels on a TV screen. With 1 pixel, you can't really display a lot. Moving up to 100 pixels, you start getting a better idea of what message needs to be conveyed, and when we have a full 4K screen, you get a super sharp picture which instantly tells you what is happening on screen. Note that in this metaphor, you can convey a message perfectly fine on the 1000 pixel screen, but when the message gets larger and more complicated, the 1000 pixel screen becomes a limitation. This is, in a very simplified manner, why increasing the parameter count leads to better performance; the model has more "pixels" to convey an increasingly complex message!
Alongside this astonishing parameter count, Moonshot introduced two architectural upgrades as well: Kimi Delta Attention (KDA) and Attention Residuals. KDA handles the heavy lifting for the 1-million-token context window—decoding up to 6.3x faster over long inputs—while Attention Residuals boost training efficiency by roughly 25% at a negligible added compute cost.
Sticking to our notoriously bad metaphor, you might be wondering: "Why not increase the amount of screens?". And we'd say: Go figure a job in AI development! Because this is exactly the thought-process most conventional AI models go through and actively engineer around. The technical term is Mixture-of-Experts (MoE), a specific architecture used inside models to enhance performance whilst lowering compute costs. Traditional AI models are quite simple in nature and pull every request through the entire neural network. In a MoE architecture, the neural network is divided into Experts. These are smaller neural networks which are specifically trained to excel at a specific topic. The model determines, via a router, which experts the input requires, routes the information to those experts, and afterwards consolidates this into a response. Having a literal Mixture of Experts in the model, allows the model to efficiently route different queries to different experts, leading to faster and more accurate response. An added benefit is that instead of activating the entire neural network, only a small amount of experts relating to the topic at hand is activated, decreasing compute cost. To give you the template AI explainer; instead of hauling the entire toolbox from the shed to fix a lightbulb, you simply grab the pliers.
To put real numbers to that toolbox, Kimi K3’s MoE framework features a staggering 896 total experts. When you send a query, the internal router selectively activates just 16 experts at a time. This means that while the total architecture holds 2.8 trillion parameters, it only fires up about 50 billion active parameters per token, drastically cutting down compute.
Let's now talk about the practical side of Kimi K3. First off: Performance
Kimi K3 performs excellent on all conventional benchmarks. Whilst trailing behind Fable 5 and GPT's 5.6 SOL, you can still expect better or equivalent performance compared to GPT 5.5 and Claude Opus 4.8. One of the most notable performances of Kimi K3, is the coding. Due to the massive context size, long coding sessions can easily be maintained without losing accuracy. On top of this, Moonshot made extra efforts in implementing Visual Reasoning, allowing coders to use screenshots to further refine the codebase. On the moonshot blogpost, you can see some case studies in action. And they look promising to say the least. Do note that these figures are reported by Moonshot, so until July 27, we have to take their word for it.
First, a quick technical distinction: Kimi K3 is technically an open-weight model rather than completely open-source. Moonshot is releasing the weights under a permissive, modified MIT license so developers can build on top of it, but the underlying training datasets and codebase remain proprietary.
That nuance aside, open weights mean that, in theory, we can run this at home. Moonshot is currently finetuning the technical details but promised to release the full model weights on July 27, 2026. Whilst we encourage everybody to try, the hard truth remains that you essentially need a NASA lab computer to be able to run models with that many parameters. For reference, local LLM engineers occasionally drop $5000 on their setup, running 600 billion parameters. Mileage may vary and there are many more things to consider when running local, but for most consumer and business markets, we are still miles away from pulling the strength of models like Kimi K3 on a local device.
Leaning into the practicality point, most of us will be using Kimi K3 in the cloud. To be honest here, without the release of the open weights, we can't really make a good estimate on the costs. Moonshot prices currently sit at $3 per million input tokens, $15 per million output tokens. Looking at earlier models, we would expect this to drop the moment other inference providers get their hands on the open weights. Given Fable 5 and GPT 5.6, these costs are not too bad, but it does make one wonder where the line lies to do a task manually compared to using an AI
We are only a few days into the Kimi K3 era, so much remains to be seen. But honestly, pushing the border on open source models is almost never a bad thing. Reddit is already overflowing with discussions on Kimi and engineers are excited to build, so it is looking like another push in the right direction. More interesting is Google's response to all this, as the current Gemini model is lacking and the community is expecting a major version before end of year. When that happens, we are sure to write about it. Until next time!
A small note from us: Please check out the blogpost by the Moonshot team, they provided an awesome, in-depth blogpost about Kimi K3. We've linked it through this article, and will do it once more here!