Boris MasnecOur first version was extremely simple.
We had no idea how much the architecture would have to change before this became a production AI product. What started as an experiment with an LLM answering questions about business data would eventually force us to rethink what the model should actually be responsible for.
We wanted to learn what users might expect from AI operating on their business data. So we built a small internal application where our first testers could upload a CSV file, type an open-ended question and get an answer from the data.
The architecture was essentially a frontend, an API, file storage and a call to an LLM.
On small examples, it worked well enough to be interesting.
Then we tried larger files.
Even medium-sized datasets were cut off before the model could process them completely. The model tried to fill gaps in the data, which led to hallucinations. Calculations became unreliable, and results varied depending on how the user asked the question.
For calculations, we needed a reproducible answer. We were expecting a probabilistic LLM to behave deterministically. So instead of fighting the probabilistic nature of LLMs, we embraced it.
The first change was straightforward. We started preprocessing the data before sending anything to the LLM.
Calculations moved into code and were exposed to the LLM as tools. We sampled and aggregated data where necessary. The prompt received calculated results together with instructions on how to interpret them.
The difference was immediate. Calculations were accurate and hallucinations decreased.
When software could produce an answer deterministically, we were better off letting software produce it and letting AI interpret the result.
Initial prototype
- CSV / user data
- LLM
- Answer
After the first failure
- User data
- Preprocessing / deterministic calculations
- Calculated results + prepared data
- LLM interpretation
- Answer
Then we added conversation
The early prototype had another obvious limitation. Without a chat-like conversation it wasn’t particularly useful.
So we added conversation history and allowed users to continue asking questions. That immediately made the experience more natural, but it also exposed the next set of problems.
Our testers wanted to combine different kinds of data. Context filled quickly. The parts of the system that worked well on prepared visit data became much less reliable once a conversation moved outside that narrow area.
Our response was not to make the prototype broader. We narrowed it. If we wanted reliable answers, we first needed a domain where we could prepare the relevant data, define the workflows and understand what a good answer looked like.
Our customers manage field teams that visit hundreds or thousands of retail locations. They need to decide which representatives should cover which locations, how frequently they should visit them and how to minimize unnecessary travel.
Territory planning became our first real use case. We wanted to help customers analyze their existing territories and find opportunities to improve coverage, workload distribution and travel efficiency.
We pre-aggregated the data we knew was required for territory planning. We added an analytical database and tools the model could call. Optimization itself was handled by deterministic algorithms. The LLM’s job was to guide the user through the workflow, manage optimization settings and interpret the recommendations, not calculate them.
We introduced a structured optimization workflow and strict instructions around it. By the time we reached broad internal testing, we had a single agent that could understand a question, retrieve data, use tools and guide a user through territory optimization.
It looked like a reasonable architecture.
Then people started using it.
Users didn’t stay inside the workflow
We had designed a product around territory optimization: analyze the current setup, identify problems, adjust settings, generate recommendations.
Internal users had other ideas and our first alpha customers confirmed the pattern.
They started by exploring the data. They asked why certain places were not being visited, compared reps, looked for gaps and asked questions such as how they might reduce costs. We had designed the product around optimization, but users first wanted to understand what was happening in their business. Territory optimization was often secondary.
This mattered technically. Data exploration, optimization and configuration were not just different questions. They were different jobs, with different data, instructions and workflows, happening inside the same conversation.
A user could start with an optimization, switch to investigating a rep, ask for different insights from the data, change an optimization setting and then refer to something discussed much earlier. What looked like one chat session was actually several different workflows sharing a conversation.
The single agent started failing in ways our development testing had not exposed. As the conversation accumulated, it became biased by previous context. It sometimes ignored workflow instructions when the topic changed. It forgot data it had retrieved earlier. Business relationships that were obvious to a domain expert were not obvious from the raw data available to the model.
We had hit the limit of what we could achieve by crafting a single agent system prompt.
One agent became several
This wasn’t a decision to adopt a multi-agent architecture because multi-agent systems were more sophisticated.
We couldn’t reliably fit three different behaviors, their workflows and enough business knowledge into one agent and expect it to remain consistent through a long conversation.
So we separated the behaviors that had started competing with each other.
An Optimization Wizard focused on the optimization workflow. A Data Explorer handled open-ended analysis. An Optimization Settings specialist handled configuration. A Coordinator sat in front of them, interpreted the user’s intent, selected the appropriate specialist, prepared its context and combined the result back into one conversation.
The user still experienced one assistant. Internally, the responsibilities were no longer mixed together.
- Optimization Wizard
- orData Explorer
- orOptimization Settings
Splitting the system into several agents also forces a question: if several agents participate in one conversation, what exactly do they need to remember and share?
Conversation history was not memory
At first, it was tempting to treat chat history as the agents’ shared memory. Keep enough previous messages and the agents should know what happened.
That stopped working for us fairly quickly.
Some information was useful only for the current turn. Some events from earlier in the conversation mattered later. A user could leave one topic, spend several turns with another specialist and then return to the first one. Some things were not conversational at all: they were facts retrieved from customer data. And the business knowledge needed to interpret those facts was something different again.
Keeping all of that as one growing conversation history made the model slower, less focused and increasingly biased by whatever happened to be in the context.
So we separated it.
We kept the immediate conversation context small and preserved important events from earlier interactions separately. Facts retrieved from customer data were managed independently of the conversation, as was the shared state needed to continue workflows across specialists.
Business knowledge was another concern. Instead of putting everything into the agents’ instructions, we retrieved only the domain knowledge relevant to the current task.
What we had originally called memory had turned into several different architectural concerns.
Trust required showing the data behind the answer
While we were solving context problems, users were showing us another requirement.
They wanted to know how the AI had arrived at an answer.
This was not an abstract explainability question. Territory decisions affect coverage, workload and budget. During early use, people often asked questions about things they already knew. They were checking whether the AI saw the same issues they already knew existed.
If the answer matched their understanding, they became willing to explore things they didn’t know. If the narrative contradicted data they trusted, confidence disappeared very quickly.
We needed to make the factual basis of an answer much more explicit.
We had already introduced what we called Factsets to ground the model’s reasoning. Tools queried or calculated data deterministically and returned the results as Facts. The model could reason over those Facts, but the factual basis did not originate in the model.
Now we exposed the same data to users through Data Artefacts: tables they could inspect and export.
That changed grounding from an invisible engineering control into part of the product. The same mechanism that constrained the model also gave users a way to verify its reasoning.
The AI could explain what it saw in the data. The user could inspect the same data.
Some problems were not agent problems
The architecture also exposed problems that adding another agent would not solve.
The richer the analysis became, the more work happened behind a response: multiple database calls, tools and, in some cases, several agents. Some interactions were simply slow, and we couldn’t make them fast enough.
So we improved the perception of latency more than latency itself. We introduced streaming so users could see progress while the work was happening. It did not materially reduce the actual processing time, but it helped users understand what was going on.
We also questioned one of our earlier infrastructure decisions. We had introduced an analytical database because we expected AI to work with data much like a traditional BI system.
It turned out that we needed to prepare data differently. Instead of querying a traditional analytical model, the AI was accessing prepared features selectively through tools, depending on what the user was asking.
For that access pattern, the analytical database was overkill. A regular database could handle the workload without the additional cost and complexity.
What the model was left to do
Looking back, the direction of the architecture is slightly counterintuitive.
We started with a model, some customer data and a question.
Over time, calculations moved into code. Optimization moved into deterministic algorithms. Data access moved behind tools. Facts became explicit. Business knowledge was managed separately from behavioral instructions. Different workflows moved into specialist agents. Conversation history stopped being treated as one large memory.
The system became considerably more sophisticated, but the model’s responsibility became narrower.
That did not make the AI less important. It made its role clearer.
The model was useful for understanding intent, navigating an open-ended conversation, choosing what capability to use, interpreting results and reasoning over grounded information. The surrounding software handled the parts where we needed reproducibility, explicit state or a factual answer we could inspect.
We did not design that division of responsibility up front. Each boundary came from something that had failed, or from something users did that we had not anticipated.
Our first prototype tried to let the model answer from the data.
Now the production system does much more, while the AI itself does less.