Pando: How OpenAI Rebuilt Git Infrastructure to Keep Pace with AI Agents
In the world of modern development, the massive increase in the use of AI agents (like Codex) has created an unexpected challenge: unprecedented load on code management systems. OpenAI discovered that existing tools were simply not designed to handle the volume of Pushes and Fetches generated when dozens of agents and developers work simultaneously on the same Monorepo. Their solution? Building Pando, a new Git infrastructure that changes the rules of the game [1, 4, 10].
Why did we need this?
The problem was simple yet critical: as the use of Codex grew, so did the number of branches and code changes. Traditional Git systems tend to "choke" under such a load, as they attempt to maintain global consistency throughout the entire repository. The result was long wait times, failures in Merges, and a decrease in system reliability [5, 7].
Pando: What is under the hood?
Instead of replacing Git, OpenAI chose to upgrade its Backend. Pando was built in Rust in just 10 weeks, and it presents two key engineering insights [4, 6, 9]:
- •Causal Consistency: Instead of locking the entire repository during an update, Pando allows parallel updates to different branches. If two developers (or agents) push code to separate branches, the system processes them in parallel. Only pushes to the same branch are kept in chronological order [1, 3, 5].
- •Reuse of Packfiles: In standard Git, the server generates a unique compressed file (Packfile) for every Fetch request. Pando changes the approach: it keeps pre-prepared Packfiles and serves them to many users through a network of regional Caches. This is similar to the performance of a CDN, allowing it to serve almost a petabyte of data per day with minimal resource consumption [3, 4, 8].
Results in the field
The transition to Pando has yielded impressive results for development teams at OpenAI [4, 6]:
- •Performance Improvement: Median Push time was cut by nearly 80% [1, 8].
- •Stability: Merge reliability reached 99.99% [6].
- •Scalability: The system is capable of serving over 57 terabytes of data in just one hour [4, 8].
It is important to note that all of this happened without changing the way developers work. They continue to use the same familiar Git commands, while Pando handles the complexity behind the scenes [3, 5, 10].
Summary and Conclusions
The story of Pando is a reminder that adopting AI technologies requires us to re-examine even our most basic infrastructures. As AI agents become an integral part of development processes, bottlenecks in code management will worsen. OpenAI chose not only to "put out fires," but to build a modern infrastructure adapted to the Agentic AI era.
For those who want to dive into the technical details, OpenAI published a detailed overview of the process on their engineering blog [4].
Relevant links: