According to a Kilo Blog comparison, a two-model setup using Kimi K3 for planning and Grok 4.5 for implementation produced an embedded database with nearly the same crash-safety results as a version built entirely by Claude Opus 5—at a fraction of the reported cost.
Kimi K3 and Grok 4.5 Nearly Matched Claude Opus 5 on a Database Build
Kilo gave both setups the same two-phase assignment: design and build kvd, an embedded key-value store written in Go. The system had to support a Redis-style text protocol over TCP, persist data to disk, recover from crashes, process atomic batches, and scale to 1 million keys.
Kilo’s test awarded Claude Opus 5 a score of 98/100 and the Kimi K3–Grok 4.5 combination 93/100. Both implementations passed 64 of 65 automated conformance checks and survived every crash-recovery test. Both also shipped the same critical protocol bug.
The reported costs were $31.71 for Claude Opus 5 and $1.27 for the Kimi K3–Grok 4.5 workflow—roughly 4% as much, or about 25 times cheaper. The lower-cost setup also completed the assignment in 23% of the time.
A database designed around failure cases
Kilo selected the database project because its requirements could be tested directly. The specification required:
- A single Go binary with commands for reading, writing, deleting, batching, statistics, and compaction
- Durable writes that could not be acknowledged before being flushed to disk
- Recovery of every acknowledged write after a
kill -9 - Atomic batches of up to 1,000 writes and deletes
- Detection and removal of partially written log records through checksums
- Support for 1 million keys and datasets larger than available RAM
- Recovery within 60 seconds
- Use of the standard library without third-party storage engines
The test focused heavily on failure paths. The database had to discard a torn final record without losing earlier data and preserve all-or-nothing behavior when a process stopped during a batch.
Planning was separated from implementation
Each setup used a strict handoff between planning and coding.
Claude Opus 5 handled both phases. Kimi K3 created the plan, while Grok 4.5 received only that plan and implemented it. Neither implementer received the original specification.
Kilo reports that Claude Opus 5’s plan was more than twice as long as Kimi K3’s. It included decision tables, details about where fsync should occur, and explanations of recovery behavior at different crash points. Kimi K3’s plan was shorter but reportedly covered the main durability, protocol, and batch requirements.
Both planning sessions contained an error involving oversized batches. Kimi K3’s plan also included a compaction design flaw that could allow a deleted key to reappear after a restart if deletion occurred while compaction was running.
Claude Opus 5 initially hit Kilo CLI’s default 32,000-token output limit during planning and failed to produce a plan in early attempts. Kilo raised the limit to 128,000 tokens before rerunning the session. One step in the successful run produced about 37,000 output tokens, according to the post.
Grok 4.5 corrected a flaw in its plan
The implementation phase produced a notable difference between the two workflows.
Claude Opus 5 took 76 minutes and built a larger test suite covering 14 crash scenarios, along with a benchmark for the full 1-million-key workload. Kilo reports that it faithfully implemented the compaction design from its own plan, including the flaw.
Grok 4.5 finished in 11 minutes and produced a codebase roughly one-quarter the size. It changed the compaction design so that compaction ran serially with writes and old data files were cleaned up. Kilo reports that this change closed the flaw in Kimi K3’s plan.
The models also differed in how much they revisited their work. Claude Opus 5’s planning run took 49 steps, including 23 edits to its existing plan. Its implementation used 150 steps, with repeated cycles of building, testing, and fixing. Kimi K3 produced its plan in seven steps, while Grok 4.5 completed the implementation, tests, and README in 22 steps.
Crash testing produced a tie
Kilo’s Python harness ran 65 checks across protocol conformance, crash durability, batch atomicity, compaction, consistency, and scale.
For durability tests, the harness recorded writes and treated them as acknowledged only after the server responded. It then killed and restarted the process at random points, checking whether every acknowledged write remained intact. The test was repeated during concurrent writes.
According to Kilo, neither implementation lost an acknowledged write. Both recovered a million-key dataset in less than 1.5 seconds and handled additional garbage appended to their data files.
The code review also found that both implementations flushed data before acknowledging writes, used checksummed records to identify torn writes, and stored each batch as a single record.
Both implementations failed the same protocol check.
The shared oversized-batch bug
The specification required batches larger than 1,000 operations to be rejected without applying any operations. Both servers returned an error, but neither consumed the rejected operations from the TCP connection.
As a result, those operations remained in the input stream and could be interpreted as separate commands. Kilo’s scripted reproduction found that Claude Opus 5’s server eventually applied all 1,001 operations from a rejected batch. Grok 4.5’s server applied two before the connection closed.
The error originated in the planning phase. Claude Opus 5’s plan explicitly instructed the server to keep the connection open without consuming the operations. Kimi K3’s plan did not specify how pending operations should be handled. Neither model’s own test suite covered an oversized batch with its operations attached.
Kilo’s external harness found the defect by testing the implementations against the specification rather than relying on the models’ tests.
Where Claude Opus 5 gained points
The five-point difference came from code review rather than crash safety or basic protocol correctness.
Claude Opus 5 produced a deeper test suite. It killed the server at multiple points, truncated and modified data files, and fuzzed the protocol parser. Grok 4.5’s tests included one crash test but did not terminate the server during a batch or compaction operation.
The documentation also favored Claude Opus 5. Its 746-line README reportedly matched the code after a line-by-line review. Grok 4.5’s README covered the required topics but described an fsync behavior on macOS that the implementation did not provide.
Kilo also found dead code and internal commentary in Grok 4.5’s codebase. A narrow compaction error path could leave an old data file behind. Claude Opus 5’s output was judged cleaner.
Grok 4.5 had one resource-use advantage. After recovering a 512MB dataset, its server used 223MB of memory compared with 428MB for Claude Opus 5’s. Kilo attributed the difference to a mis-sized memory estimate in Claude Opus 5’s recovery path.
Cost reduced the trade-off to maintainability
Kilo’s results suggest that the cheaper setup did not sacrifice crash safety in this particular assignment. Both implementations passed the same automated checks, survived the same crash tests, and contained the same critical bug.
Claude Opus 5 produced the more complete maintainability package, with broader tests, more accurate documentation, and cleaner code. Those benefits accounted for its higher score, while the core durability results were effectively tied.
Kilo concludes that Claude Opus 5 would be the preferred default when cost is secondary. However, the reported 25-fold price difference makes the cheaper workflow difficult to dismiss, particularly because additional testing, documentation review, and code cleanup could be requested separately. The database’s crash-safety properties were the harder part to recover after the fact, and the Kimi K3–Grok 4.5 combination matched Claude Opus 5 there.
The comparison also leaves open whether a more iterative prompt would improve the lower-cost workflow. Claude Opus 5 reviewed and revised its work repeatedly by default, while Kimi K3 and Grok 4.5 followed a much shorter process. Kilo did not test whether adding similar review steps would narrow the quality gap.
The shared oversized-batch defect offers the clearest warning from the exercise: independent planners made the same mistake, and both implementations’ internal tests failed to catch it. The defect surfaced only through an external harness written before either model began.
Source: Kilo Blog
