I run a multi-agent coding system on a single AMD Strix Halo box: 128 GB of unified memory, one iGPU, four agent roles (architect, coder, QA, investigator), all driven by local models through Ollama. The system plans projects, decomposes them into tasks, dispatches work to executor agents, and reconciles completions. It took me weeks to get it working well, and the single biggest improvement had nothing to do with prompt engineering, model selection, or fine-tuning. It was removing the LLM from the loop.
Control plane vs data plane
If you’ve built distributed systems, you know this split. The control plane decides what should run and where. The data plane does the actual work. Kubernetes has it. Network switches have it. Service meshes have it. The control plane is fast, deterministic, and boring on purpose, because it’s the single point where a wrong decision cascades into everything downstream. The data plane is where complexity, latency, and judgment live.
Multi-agent LLM systems have the same split. They just don’t know it yet. Most frameworks run an LLM for both: an LLM decides what to do (control plane), then an LLM does it (data plane). When everything is an API call to a cloud provider with functionally infinite concurrency, you can get away with that. On constrained hardware, it falls apart.
My architecture ended up here:
Dispatcher (control plane -- deterministic Go daemon)
|
+---- Architect (planning, LLM)
| |
| +---> tasks
|
+---- Coder (data plane -- LLM)
+---- QA (data plane -- LLM)
+---- Investigator (data plane -- LLM)
The dispatcher is infrastructure. It doesn’t think. It snapshots state, applies fixed rules, and spawns processes. The agents are the ones that read specs, write code, make judgment calls, and produce artifacts. Separating the two turned a fragile system into a reliable one.
I didn’t start there. I started with an LLM doing both.
The original design
The first version used a hermes cron job that fired every five minutes, executing through a model turn. On each tick, the system loaded the architect model, fed it a skill file describing the coordination protocol, and asked it to scan the filesystem for spec files, check task status, and decide what to do next. The architect would read a spec, decompose it into tasks, write them to Consul KV, and exit.
I had put a probabilistic component at the single point where determinism mattered most. I had put an LLM in the control plane.
What went wrong
The scheduler’s job boils down to five questions, asked every tick, for every project:
- Are there stale tasks whose executor died? Reset them to pending.
- Has the spec changed since last planning? Wake the architect.
- Are there tasks in error state? Wake the architect.
- Are there no tasks at all? Wake the architect.
- Are there pending tasks? Spawn one executor per profile.
Every one of these is a comparison, a hash check, a set membership test. There is no judgment anywhere in the list. Running a 120-billion-parameter model to answer five boolean questions is like hiring a lawyer to flip a light switch. It works, but the hourly rate is hard to justify.
Worse, the costs compound in ways that aren’t obvious until you live with the system:
GPU contention. On a single-GPU box, loading the scheduler model evicts whatever the coder was running. Every five-minute tick, every time, whether or not there was work. My agents spent more time swapping in and out of memory than they spent thinking.
Invisible failures. A hallucinated tool call in a scheduling tick is silent corruption. The system doesn’t crash; it just makes a wrong decision that propagates forward. One skipped recovery, one duplicate dispatch, one forgotten workspace, and state drifts. You don’t notice until something downstream fails for no apparent reason.
Zombie processes. The cron was registered inside the agent harness. When I deleted the skill files it was based on, the registration survived. It kept firing every five minutes, loading a 51 GB model, evicting whatever I was testing, and I didn’t find it for days because it didn’t show up in crontab or systemd timers. A model-driven scheduler outlived the code that defined it and degraded the system silently from the inside.
No child supervision. Even if the cron reliably decided to spawn executors, it couldn’t track them. Cron ticks are stateless. “Is a coder already running for this workspace?” requires holding PIDs across ticks. The model-driven version reconstructed this from lock files, which brought its own staleness bugs. The scheduler needs to be a long-lived process, and an LLM session is not that.
Building a real control plane
I replaced the cron with a small Go daemon. It does exactly what the cron did, minus the model. Every 60 seconds it calls workspace_reconcile on the MCP server (which atomically recovers stale tasks and returns a snapshot), hashes the spec file, runs the five-question decision function, and spawns agent processes for the results. The decision function is 40 lines of Go, table-tested, deterministic, boring.
The daemon holds a singleton lease so two instances can’t run simultaneously. It tracks child PIDs so it never spawns a duplicate. It caps total children at one (configurable), because on my hardware only one model fits on the GPU at a time without thrashing. When a child exits, the slot opens on the next tick.
The entire scheduler costs zero GPU. It doesn’t load a model. It doesn’t run inference. It runs in microseconds per tick and the only network calls are a few MCP tool invocations over localhost HTTP. The GPU is now 100% dedicated to agents doing actual work.
The data plane is where LLMs belong
The point isn’t that LLMs are bad at scheduling. They’re fine at it. The point is that scheduling doesn’t need what LLMs provide. It needs correctness, speed, and predictability. LLMs provide judgment, creativity, and language understanding. Those are data plane properties.
In this system, the LLM lives in two places: the architect (which decomposes specs into tasks, reconciles errors, and decides when a project is done) and the executors (which claim tasks, write code, run tests, and report results). Both roles require reading natural language, making judgment calls, and producing artifacts that didn’t exist before.
The dispatcher doesn’t touch any of it. It doesn’t read specs, doesn’t evaluate quality, doesn’t decide if a task decomposition is good enough. It answers “should something run right now?” and gets out of the way. Clean separation: the control plane is fast and reliable, the data plane is as creative and expensive as it needs to be.
The hardware lesson
This matters more on consumer hardware than it does in the cloud. If you’re running agents against an API with functionally infinite concurrency, the cost of a scheduling inference is still a few cents and a couple of seconds. Annoying but survivable. On a single-GPU box, it’s catastrophic: every scheduling tick competes with real work for the one resource that everything needs.
My Strix Halo box has 128 GB of unified memory. That sounds like a lot. It was enough for exactly one model I chose at a time, because Ollama’s VRAM accounting reported 62.5 GiB (the default GTT ceiling, which was itself misconfigured due to a kernel parameter using the wrong module prefix). For weeks, every model load evicted whatever was resident, and the cron’s five-minute ticks guaranteed constant eviction. Fixing the GTT ceiling to 105 GiB and removing the cron turned the same hardware from barely functional to running two models simultaneously with instant warm starts.
The lesson generalizes: on constrained hardware, every inference you eliminate is GPU time returned to the work that needs it.
What I’d tell someone building this today
Start with the decision function, not the prompt. Write down what your scheduler needs to decide. If every decision is a comparison, a threshold, or a state check, you don’t need a model. If some decisions require reading documents and making judgment calls, split the system: deterministic scheduler, model-driven planner.
The scheduler is the wrong place for probability. It’s your single point of failure. If it makes a wrong decision, everything downstream is wrong. If it crashes, nothing self-heals. Make it boring. Make it tested. Make it the thing you don’t have to debug.
Spec files stay in git. Task state lives in your coordination layer. Don’t mix them. The spec is a human-authored document that should be versioned, diffed, and reviewed. Tasks are ephemeral machine state that should be queried, locked, and garbage-collected. Two different lifecycles, two different storage systems.
Watch the spec, not the tree. My dispatcher hashes the spec file each tick and wakes the architect when it changes. It deliberately does not hash the project directory, because executors modify the repo as their job. Tree-hashing creates a feedback loop where the architect is woken by its own agents work. Only hash the thing humans change.
Recovery is a server-side operation. A dead executor leaves a locked task. The coordination server should detect and reset it atomically, not the scheduler. CAS-guarded writes mean a concurrent live update always wins over a recovery attempt. This is the kind of thing Consul (or any CP store with compare-and-swap) gives you for free, and it’s the kind of thing an LLM-driven scheduler gets wrong in ways that are very hard to reproduce and debug.
The numbers
Before: 45 seconds to 13 minutes per scheduling tick (model load + inference + eviction cascade), 10.6-36 tokens/second depending on which model survived the eviction lottery, silent state corruption from a zombie cron, constant GPU contention.
After: under 3 seconds warm inference for actual agent work, 36-60 tokens/second with stable model residency, zero GPU cost for scheduling, deterministic recovery, and a dispatcher whose entire decision logic has eight table-driven unit tests and has never made a wrong call.
The LLM got better at its job the moment I stopped asking it to do mine.
Every distributed system eventually learns this. The thing that decides what to run cannot be the most expensive, slowest, least predictable component in the stack. Kubernetes doesn’t run a neural network to schedule pods. BGP doesn’t consult a language model to propagate routes. The control plane is boring because boring is the point. It’s the foundation everything else trusts.
The AI agent ecosystem is rebuilding distributed systems from scratch, and it’s repeating the mistakes the infrastructure world made twenty years ago: putting too much intelligence in the coordination layer and wondering why the system is fragile. The fix is the same fix it’s always been. Separate the control plane from the data plane. Make the control plane deterministic. Let the data plane be as creative, expensive, and probabilistic as it needs to be.
LLMs are extraordinary tools for reasoning, planning, and producing artifacts. They are terrible infrastructure. Stop using them as infrastructure.