Authors: Shengran Hu, Cong Lu, Jeff Clune · arXiv: 2408.08435 · Source: https://arxiv.org/abs/2408.08435
Fetched: 2026-09-05
Note: This ingest preserves the abstract and key formulation excerpts from the arXiv abstract page / paper text. Full PDF is at https://arxiv.org/pdf/2408.08435. Do not treat this file as a translation.
Abstract
Researchers are investing substantial effort in developing powerful general-purpose agents, wherein Foundation Models are used as modules within agentic systems (e.g. Chain-of-Thought, Self-Reflection, Toolformer). However, the history of machine learning teaches us that hand-designed solutions are eventually replaced by learned solutions. We describe a newly forming research area, Automated Design of Agentic Systems (ADAS), which aims to automatically create powerful agentic system designs, including inventing novel building blocks and/or combining them in new ways. We further demonstrate that there is an unexplored yet promising approach within ADAS where agents can be defined in code and new agents can be automatically discovered by a meta agent programming ever better ones in code. ... We present a simple yet effective algorithm named Meta Agent Search to demonstrate this idea, where a meta agent iteratively programs interesting new agents based on an ever-growing archive of previous discoveries. Through extensive experiments across multiple domains including coding, science, and math, we show that our algorithm can progressively invent agents with novel designs that greatly outperform state-of-the-art hand-designed agents. Importantly, we consistently observe the surprising result that agents invented by Meta Agent Search maintain superior performance even when transferred across domains and models...
ADAS triad (paper §2)
Formulation: Automated Design of Agentic Systems (ADAS) involves using a search algorithm to discover agentic systems across a search space that optimize an evaluation function.
- Search Space: which agentic systems can be represented (here: code defining the entire agentic system, e.g. a
forwardfunction). - Search Algorithm: how the space is explored (here: FM meta agent iteratively generating new agents conditioned on an archive).
- Evaluation Function: how candidates are scored on objectives such as performance; "a simple method is to calculate the accuracy rate on the validation data for a task."
Meta Agent Search loop (paper §3)
(1) Archive optionally initialized with baselines (CoT, Self-Refine, etc.). (2) Meta agent designs a new agent (description + code) with self-reflection for novelty. (3) Generated agent evaluated on validation data from the target domain; runtime errors trigger meta-agent self-reflection (up to five times). (4) Agent added to archive with metrics; iterate.
ARC setup (paper §4.1): sample validation set and test set with 20 and 60 questions respectively for searching and testing; evaluate all discovered agents on the held-out test set; report median accuracy and 95% bootstrap CI on held-out test by evaluating agents five times.
Reported gains (paper abstract / tables): e.g. DROP F1 +13.6/100; MGSM accuracy +14.4%; transferred GSM8K +25.9% and GSM-Hard +13.2% over hand-designed baselines (as stated by authors).
Isolation note (for KB indexing)
Candidate = discovered agent programs in code. Optimizer = meta agent (separate from the agents being scored). Eval & control = validation scoring during search + held-out test reporting / transfer tests. Meta agent is not rewritten by the search in the base Meta Agent Search algorithm (higher-order ADAS discussed as future work).