Software WorldStudying Agent Collaboration with
Software Dependency Chain Optimization
Motivation: optimizing software ecosystem
The development of expert-level coding agents opens the opportunity to automatically improve the efficiency of software packages that millions people use everyday. Existing efforts tend to focus on developing and maintaining individual packages, but software is not build in a vacuum. Packages exist within a broader dependency ecosystem, in which a given package might take on others as dependencies while still others downstream might depend on itself. We believe that improving software packages require improving the broader ecosystem.
More than just the implications on software engineering, optimizing software ecosystems provide an exciting setting for studying and evaluating multi-agent coordination. Ecosystems can naturally be broken up, with each agent optimizing its own package. The social structure of an ecosystem is ripe with opportunities for agents to collaborate to exchange usage information, jointly discover goals for improvement, and independently validate each others' changes. Moreover, ecosystems enable robust evaluation. We can hold out certain downstream packages—that is, not optimize them—and use them as an extrinsic evaluation. If agents' individual and collective efforts improve the ecosystem, these held out packages, that potentially use dependencies in unique ways, should also benefit.
To study this problem, we create several software worlds of Python packages sub-sampled based on real dependency data, and we deploy agent teams focused on optimizing code efficiency. We find that agents are capable of identifying code deficiencies and come up with patches individually. Notably, depending on the base models, some agents can engage in meaningful collaboration, soliciting downstream usage patterns that lead to discovering algorithmic improvements for their code. Over long-running sessions that last from a few days to weeks, agent teams can consistently improve the code, leading to 5-10% drop-in improvements measured by held-out downstream benchmarks. Through these simulations, we begin to understand how the future of agent software engineering may look like.
Creating software worlds
Selecting Repositories in the Software World. We construct software worlds by subsampling real packages from the Python ecosystem. Our candidate catalog combines packages from prior software-engineering benchmarks (Sehgal et al., 2026; He et al., 2025; Deng et al., 2025; Ma et al., 2025; Shetty et al., 2025; Nangia et al., 2026) with popular PyPI packages, for which we collect releases, source repositories, tests, benchmarks, and dependency metadata. We organize the resulting repositories into candidate communities using their package-level dependency graph, producing a shortlist of 302 repositories. Finally, we enrich the graph with static and runtime interface-usage information and manually group packages and their dependencies into software worlds of different sizes with controlled structure, composition, and code-usage patterns.
A proxy society around requests → urllib3
Society A, aligned with the held-out consumers (world overlap 0.180): two requests-only consumers, one urllib3-only, one consumer of both — six repositories, six edges, density 0.4.
Simulating software worlds
Agent Implementation. Each repository in a software world is assigned to a persistent maintainer agent that operates asynchronously in discrete work episodes, which we call wakes. Agents receive only the high-level objective of improving software efficiency while preserving correctness. At the beginning of each wake, an agent loads its persistent memory, reviews the current state of its repository and recent collaboration activity, and independently decides what to investigate. Using Harbor (Harbor Framework Team, 2026), each agent runs in a sandboxed Docker container where it can inspect and edit code, execute tests and profilers, manage version-control state, and publish new releases.
Agent Collaboration. We host the repositories on a modified version of Forge, an open-source, GitHub-like service that provides version control, issue tracking, pull requests, and release management. Agents can inspect other repositories, open and discuss issues, propose changes, and review others' pull requests via the provided forge cli. Communication is therefore asynchronous and artifact-bound: an agent that needs something from a neighbour opens an issue on that neighbour's repository, the exchange accumulates as comments on the issue or on a pull request, and every claim carries the code, benchmark, or profile it rests on. There is no direct channel between agents; what one agent learns from another, it learns from the forge, on its next wake. Only the assigned maintainer can merge changes and publish releases for its repository. These affordances ground collaboration in concrete software artifacts; in practice, we find agents exchange usage and profiling evidence, request upstream changes, discuss proposed improvements, and independently validate releases against their own workloads.
Collaboration Examples. A few kinds of exchange recur across models. Each is shown here by one real story from the runs, one frame per event.
Two agents spent two days establishing where pluggy's time is not, and left pytest with two profilers and one real target.
About to optimize its next release, pluggy-agent asks pytest — its most important downstream on the forge — which pluggy paths show up in pytest's profiles, which hooks fire most, and what must not change, rather than guessing at hot paths.
“I would love to make those improvements target workloads pytest actually cares about rather than guessing hot paths.”
Analyzing agent simulations
Evaluating Agent Work with Downstream Impacts. For each software world, we curate a set of evaluation repositories with human-authored performance benchmarks that exercise one or more libraries in the world. Evaluation repositories and their benchmark tasks remain fixed and unseen to the agents in the society. Because wall-clock measurements can be noisy, we use Callgrind's instruction-read count (Ir) collected in standardized container environments as our primary performance metric. Each evaluation sweep installs the world's latest releases under the fixed evaluation repositories and reports the geometric-mean speedup of their benchmarks against the world at seed; a benchmark that breaks scores 1.0, so breakage is never rewarded.
Two central providers (requests → urllib3) and four society consumers; four frozen downstream packages — requests-cache, cachecontrol, zeep, awswrangler — are the hidden battery.
| run | model | treatment | wakes | duration | ledger | releases | cross-repo | final |
|---|---|---|---|---|---|---|---|---|
| 15.1 | terra | shared · A | 600 | 9.0 h | $221† | 103 | 0 | 1.034 |
| 16.1 | sonnet | shared · A | 599 | 13.5 h | $510 | 81 | 4 | 1.050 |
| 18.1running | opus | shared · A | 263 | 5.9 d | $1,734† | 118 | 144 | 1.123 |
adoption geomean = exp(mean(log(seed / current))) over every hidden task's CPU instruction count; a red-gated consumer scores 1.0 on all its tasks; sweeps missing a task are dropped; series end with the run. † ledger dollars at list price for runs on a subscription (real spend $0). Click a row to add or remove that run from the plot; click a line to pin it.
References
- Deng, X., Da, J., Pan, E., He, Y. Y., Ide, C., Garg, K., Lauffer, N., Park, A., Pasari, N., Rane, C., Sampath, K., Krishnan, M., et al. (2025). SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? arXiv:2509.16941. arxiv.org
- Harbor Framework Team (2026). Harbor: A Framework for Evaluating and Optimizing Agents and Models in Container Environments. Software, version v0.16.1. doi.org
- He, X., Liu, Q., Du, M., Yan, L., Fan, Z., Huang, Y., Zheng, Y., Yuan, Z., and Ma, Z. (2025). SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories? arXiv:2507.12415. arxiv.org
- Ma, J. J., Hashemi, M., Yazdanbakhsh, A., Swersky, K., Press, O., Li, E., Reddi, V. J., and Ranganathan, P. (2025). SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads? arXiv:2511.06090. arxiv.org
- Nangia, A., Mishra, S., Gokrani, A., and Chopra, P. (2026). ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads? arXiv:2602.19594. arxiv.org
- Sehgal, A., Hou, J., Sarkar, A., Mantripragada, I., Chaudhuri, S., Sun, J. J., and Yue, Y. (2026). FormulaCode: Evaluating Agentic Optimization on Large Codebases. arXiv:2603.16011. arxiv.org
- Shetty, M., Jain, N., Liu, J., Kethanaboyina, V., Sen, K., and Stoica, I. (2025). GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents. arXiv:2505.23671. arxiv.org