mixture-of-agents — Per-Turn Model Rotation for AI Agents

The problem

Every model has a characteristic failure mode: a reasoning shortcut it always takes, a library it always hallucinates, a refactor it always over-engineers. Run a hundred turns through one model and those failures don't average out — they compound, because each turn builds on the last turn's output. The agent digs a straight line into its own blind spot and never sees the wall.

Why it mattered

A subscription pool is paid for whether or not it is used, and the model that is best at planning is rarely the one that is best at writing a regex. Switching by hand means remembering to switch, which means not switching. The cost is a session that quietly plateaus at the ceiling of its single model.

What I built

A pi extension that hooks turn_start — the moment before each LLM call — draws a random model from a configurable pool, and calls pi.setModel(). If the draw matches the model already loaded, the switch is skipped, so the extension prefers switching away rather than idling on a lucky repeat. The active model shows in the footer as MoA: <label>, a notification fires on each real switch, /moa toggles the whole thing at runtime, and --no-moa starts a session with it disabled. A model whose API key is missing is reported once, then dropped from the draw and replaced from the remaining pool — a missing key degrades the pool instead of killing the turn.

1. Step 1
   Draw before the call

   On turn_start — fired before each LLM request — the extension draws a random entry from the pool. If the draw equals the currently loaded model, it redraws, preferring to switch away rather than idle on a repeat.

2. Step 2
   Swap the model, tell the human

   pi.setModel() loads the drawn model, the footer updates to read `MoA: <label>`, and a brief notification fires on each real switch so the operator always knows which model produced which turn.

3. Step 3
   Degrade, don't fail

   If a drawn model is unconfigured or its API key is missing, the extension reports it once, drops it from the pool, and draws again from what remains. A broken provider shrinks the rotation instead of killing the session.

Key decisions

Random rotation over task-aware routing. The obvious design is to route by task — the strong reasoner plans, the fast model edits. I rejected it. Task-aware routing needs a classifier that labels each turn, and the classifier needs ground truth about which model is actually better at which task. I have no such data, and inventing the labels would have meant encoding my guesses as if they were measurements. Random rotation costs zero classification, requires zero ground truth, and still achieves the thing that actually matters: decorrelating consecutive turns, so one model's blind spot can't compound across the session. It is the honest version of the idea I could actually build.

A lifecycle hook over a routing proxy. I could have put a proxy in front of the provider API and rotated there. That means infrastructure — a process to run, a port to hold, a failure mode between the agent and its model. Hooking turn_start inside the agent uses models the runtime already has configured, adds no moving parts, and hot-reloads with /reload.

Outcome

What a production version needs

The name overpromises, and I'd rather say so than let a reader assume. In the literature, mixture-of-agents means several models each propose an answer and an aggregator synthesizes them. This does no ensembling and no aggregation — it is per-turn random routing across a pool. The technique is real and the name is borrowed.

No evidence it works. There is no A/B harness, no benchmark, no measured difference between a rotated session and a fixed-model one. The argument for it — that decorrelating turns prevents a single model's failure mode from compounding — is a hypothesis. Testing it properly means a task suite, many seeds, and a scored comparison against fixed-model baselines. That is the next piece of work, and until it exists this is an untested idea with a clean implementation.

The pool is hardcoded. It lives as a const POOL at the top of the source file, so changing models means editing TypeScript rather than a config file. Cost per turn also becomes nondeterministic once the pool spans models with different pricing — the extension does not track or cap spend.

There are no tests, and CI runs only npm install and npx tsc --noEmit.