Kimi K3.1 Pro’s strong benchmarks meet the harder test of finished work

A software demonstration can look finished before the work is sound. A phone render may have attractive lighting and the wrong hardware details. A chart may be neatly drawn from numbers its creator invented. When an AI assistant can write code, run tools and assemble a deliverable, the final inspection becomes part of the product.

Kimi K3.1 Pro puts that question in focus. Kimi is the AI assistant developed by Moonshot AI, and the new Pro model is aimed at coding and longer assignments involving software tools. These systems are often called agents: the model chooses actions, inspects the results and decides what to do next. Their usefulness depends on the whole sequence, including the decision that a task is complete.

The Kimi Code release log dates the rollout to October 23, while the research index lists the launch article on October 24. Moonshot reports a context window of one million tokens, the chunks of text a model processes, and substantial improvements across coding, 3D creation and financial work. Hosted access is already available. The release’s practical significance is a stronger candidate for work that spans many files and steps, with performance that readers can now examine beyond the launch announcement.

Two kinds of evidence are especially revealing. Artificial Analysis reports large improvements on its evaluation suite. In Matthew Berman’s video review, useful outputs sit alongside conspicuous errors in geometry, data and finishing. Together, they describe an advance whose value varies with the job and the checks surrounding it.

The benchmark progress deserves a close reading

Moonshot’s launch report places K3.1 Pro at 65.7% on Terminal-Bench 4.0, compared with 66.4% for Claude Opus 5.5 and 58.2% for GPT-6 Astra. The benchmark tests agents on tasks in a computer terminal. Its footnotes say Kimi ran through Claude Code, the software coordinating its tool use, while comparison figures came from other published results. The report generally uses maximum reasoning effort.

That distinction helps explain what a score measures. The model supplies reasoning and actions; the surrounding software supplies tools, instructions and execution limits. A result belongs to that combination. A small gap between two published figures deserves more scrutiny when the runs use different arrangements.

The stronger evidence for broad progress comes from Artificial Analysis’s own October 23 evaluation. It gives K3.1 Pro an Intelligence Index score of 55, up from 44 for K3 and fourth among the models in that release comparison. It also reports 67% on Terminal-Bench 4.0 and a rise from 58% to 73% on AutomationBench-AA, which evaluates software-service workflows.

Those findings give the launch substance. They show improvement under an outside evaluator’s conditions, including tasks that require tools and multiple steps. The index combines a particular set of tests, so a buyer still needs to connect its strengths to the work being purchased. A high aggregate score can coexist with a costly weakness in the one task an organization repeats every day.

Where Berman’s demonstrations become useful

Berman starts with a success. In the interactive Rubik’s Cube demonstration at 0:57, he praises the result. It is a useful example of an agent turning a brief into an interactive artifact, the kind of work for which a functional first version can save substantial time.

His iPhone rendering task, beginning at 4:04, makes the distinction between appearance and fidelity more visible. The output looks attractive, but Berman identifies incorrect details. The task took 28 minutes and 45 seconds. He requests a revision; the video does not show its outcome. For a designer using the result as a concept image, the polish may be valuable. For a product illustration that must accurately depict a specific device, the incorrect details determine whether the work is usable.

The same issue becomes sharper in three dimensions. In the Blender climbing-knot task at 7:15, Berman finds problems with the rope’s topology and attachment. Topology concerns how the parts connect and pass around one another. Attractive materials cannot repair that underlying relationship. A rope can look convincing at a glance while following an impossible or incorrect path.

He then tries a knot animation at 9:09. Kimi’s crossings remain difficult to follow. A same-prompt comparison with GPT-6.1 Sol at medium effort begins at 10:03, and Berman finds that sequence much clearer; Kimi’s revision still leaves the operation unclear. The comparison concerns the clarity of these generated animations. Any climbing instruction would need separate validation by a qualified source.

A browser game shown at 13:12 works, although Berman points out collision and flickering faults after a 48-minute run. His five-minute parody-video project at 15:25 demonstrates a different strength: assembling a production using external tools. The achievement is orchestration, including the ability to coordinate the parts of a media workflow.

The most consequential failure for research work appears in the chart task at 16:58. Kimi supplies invented benchmark scores and subsequently acknowledges them as placeholders. Completing the task requires supplied source data and further prompting. A polished chart amplifies the problem because its appearance encourages a reader to treat the numbers as settled evidence.

This is where the review becomes more than a collection of visual successes and failures. Each example asks whether the agent checked the property that made the output useful: product details in the render, connected geometry in the knot, dependable behavior in the game and sourced values in the chart. The cost of a weak check falls on the person receiving the file.

Settings matter, and so does the size of the sample

Berman’s assessment from 18:35 onward is that these practical tests fall short of Claude Opus 5.5 and GPT-6.1 Sol. His discussion at 20:23 contrasts the benchmark claims with his experience and raises benchmark optimization as a possible explanation.

The demonstrations support his account of the outputs he reviewed. Establishing a broader ranking would require repeated runs, comparable tools, fixed effort settings and predefined success criteria. Berman’s small, non-blinded set provides concrete failure cases and useful leads for further evaluation. JetsonsAge has not run these model comparisons.

At 5:29, Berman explicitly identifies the iPhone task as K3.1 Pro at high effort, rather than max. That statement establishes the setting for that task. Moonshot’s Code documentation lists high as the default and offers low, high and max. The launch report’s usual max setting is therefore a relevant difference, but the video provides too little controlled evidence to measure how much that difference explains.

Readers can draw a practical lesson without settling a universal model ranking. Test the exact version, effort and tool environment that will do the work. Define success in terms the output can actually satisfy. For a chart, every value should trace to the supplied dataset. For a model of an object, geometry should match the reference from more than one view. For software, important behaviors should be exercised after the first successful launch.

Faster output and cheaper tokens leave another bill to count

Moonshot describes its Highspeed version as roughly two to three times faster, with the same model capabilities. The same documentation explains the boundary: Highspeed accelerates model output, while tool calls and scripts retain their own execution time. A long render or a sequence of revisions can dominate the elapsed time even when text arrives quickly.

Artificial Analysis provides a useful economic counterpoint. It reports about $1.98 per Intelligence Index task, roughly equal to K3. Lower token prices were accompanied by greater output-token use. Its methodology calculates cost from the tokens actually consumed across its weighted workload.

For a working team, the next measure is cost per accepted deliverable. That includes model use, external tools and the human time spent checking, correcting and rerunning the assignment. Berman’s elapsed times illustrate why that measure matters, while remaining specific to his demonstrations. A fast, correct first draft and a visually impressive file that needs extensive repair create very different savings.

The weights are a separate release

Moonshot says K3.1 Pro is available through its hosted products and API, with full model weights promised by November 4, 2026. As of October 24, that downloadable release remains ahead. Weights are the learned numerical parameters needed to run a model; their availability can give outside teams more control over deployment and evaluation.

The eventual files and accompanying license will determine the practical permissions and requirements for that use. The launch’s open-weight description alone leaves those questions unresolved. A team considering self-hosting needs the released license, supported software and measured resource requirements before it can make a reliable deployment plan.

For now, the most defensible reading is also the most useful. K3.1 Pro has strong outside benchmark evidence and demonstrated ability to build worthwhile artifacts. Berman’s review identifies specific places where an agent’s judgment about completion still needs help. The next competitive advantage will come from reducing that checking burden: delivering files whose facts, behavior and structure survive inspection as well as their presentation does.