Multi-Agent Systems
Over the past year, I have worked on several AI agents in different organizations - from startups to enterprises. And essentially, the problem was always the same: ChatGPT knows a lot, but it can’t tell you how many leads you had last month or what the delivery status of these orders is.
The solution is to develop a multi-agent system that communicates with the company’s various data sources and makes everything available via a chat interface.
It could look something like this:
Interactive diagram A deterministic shell around stochastic agents Open the diagram to see where orchestration, specialized agents, and deterministic checks fit into the request path.
User question
↓
Chat UI → API endpoint → Orchestrator
├─ SQL agent → validated query → database
├─ Chart agent → visualization spec
└─ Policy checks → permissions + output contract
↑
Final answer ← response composer ← normalized agent outputsThe important design choice is not to make every part creative. Keep routing, permissions, schemas, and output contracts deterministic; reserve stochastic reasoning for the places where ambiguity is useful.
The user asks a question through a chat UI. This question is then forwarded to the backend. There, in turn, are several agents specialized in different areas. One agent, for example, is specialized in making SQL queries, another in visualizing the data as charts, and so on. There is also an orchestration agent that monitors the individual agents and distributes tasks. In the end, a response is generated from the outputs of all agents and sent back to the client.
The Stochastic Nature of LLMs
LLMs are stochastic “by design”. They are built to give slightly different answers to the same question every time. They are “token guessers” that try to figure out, using statistical models, what the probably most sensible next token is based on the previous tokens. And there is always some leeway.
And when you use a multi-agent system that chains several LLM queries one after another, these small deviations can of course compound.
Interactive loop
Where variability enters an agent workflow
stochastic
Interpret intent
The model translates a human question into a goal. This is useful because users rarely ask in perfect system language.
But this often clashes with the expectations of end users, who expect deterministic answers.
Case Study
Take, for example, a tool through which users can ask things like “How were my leads in the last 3 months?”. The program then taps into the relevant data sources and visualizes the result in a pretty dashboard with charts. So far, so good. But the next time, the user asks the same question - and this time the dashboard looks a bit different. Or the data is broken down by channel this time instead of aggregated. But then the user complains: “When I ask about leads, I always want exactly this dashboard”. So they expect a deterministic answer.
The Tension: Deterministic Expectations vs. Stochastic Nature
The example above shows that expectations are placed on the system that contradict its nature.
Furthermore, users don’t want “ChatGPT with internal info” - they want a replacement for PowerBI.
Deterministic vs. stochastic responsibilities
The Solution
The first approach is usually to keep inflating the prompts with guardrails and use-case-specific examples. But that usually fails quite quickly.
As of March 2026, sooner or later it comes down to switching to a skill-based approach. A good introduction to skills can be found here. But the basic idea is to create a skill with detailed instructions for every prompt that needs a deterministic answer.
Is this the final, perfect solution? Hardly. But at this point in time, it’s the only way to reconcile users’ requirements with the nature of LLMs.