OpenAI’s cloud-based coding agent, Codex, looks like “just” another model-assisted developer tool from the outside. Under the hood, it’s closer to a carefully orchestrated distributed system: a model (codex-1, a version of o3 fine-tuned for software engineering), wrapped in a harness that manages tools, state, permissions, and multiple UI surfaces. That scaffolding is where most of the engineering complexity lives.
Codex, in practical terms
Codex is designed to write features, fix bugs, answer questions about a codebase, and propose pull requests. Each task runs in its own isolated cloud sandbox that’s preloaded with the repository, and multiple tasks can run in parallel with progress visible as the work unfolds.
That “agent” framing matters: Codex isn’t limited to generating a patch once. It can iteratively inspect a repo, run tests, and revise changes until it has something coherent to hand back.
The agent loop: model reasoning meets tool execution
At the center is the agent loop: user input becomes a prompt, the model responds, and often that response is a tool call rather than a final answer—think “run this shell command” or “execute the test suite.”
When that happens, the harness executes the request, collects output, appends it to the ongoing context, and calls the model again. This repeats—sometimes dozens of cycles—until the loop ends with a user-facing result. The important separation is that the model decides what to do next, while the harness handles execution, output capture, permissions, and termination conditions.
Codex also structures interaction into “turns.” Each new user follow-up includes the entire history of prior turns, including tool calls and outputs—useful for continuity, but expensive for context management.
Prompt construction and the cost of “remembering everything”
Codex’s prompts are layered. Alongside the user message, the system includes layers like environment context (cwd, shell), sandbox permission rules, tool definitions, developer instructions, and AGENTS.md files placed in the repo to convey project conventions and preferred commands.
Because Codex chooses stateless requests—supporting requirements like Zero Data Retention—each turn resends the full conversation history. That makes the total data sent grow quadratically over time.
Two key mitigations keep this workable:
- Prompt caching: the prompt grows by appending, so earlier content remains an exact prefix; compute can be reused to keep effective inference cost closer to linear.
- Context compaction: once the context window limit is reached, Codex replaces full history with a smaller representation, preserving understanding via an encrypted payload carrying latent state.
Caching is also shown to be fragile: even small inconsistencies—like tool lists being emitted in a different order—can break cache hits.
One agent, many surfaces: Codex core and the App Server
Codex started as a CLI, then needed to work in VS Code, a web app, a macOS desktop app, and third-party IDEs. An early attempt to expose the agent via MCP didn’t fit certain interaction needs, including streaming progress, pausing for user approval, and emitting structured diffs.
The solution was a multi-surface architecture built around:
- Codex core: the shared codebase containing the agent loop, tool execution, thread management, configuration, and authentication.
- An App Server wrapper that exposes core via a bidirectional JSON-RPC protocol over stdio (or via HTTP streaming in the web deployment model).
This bidirectionality enables the server to ask the client for approval mid-task; execution pauses until the user responds “allow” or “deny.” The protocol is also designed to be backward compatible, letting partners keep clients stable while updating server binaries independently.
The system is the product
Codex’s architecture underlines a theme that keeps repeating in AI-assisted coding: the model is a component; the agent is the system. Practical performance comes from orchestration details—how context is constructed, how tools are mediated, and how interaction patterns are preserved across terminals, IDEs, and web surfaces.
Original source: How OpenAI Codex Works
