How AI changes the way we build business software products

A different kind of software

For decades, business software has worked on a straightforward bargain. You filled in a form, clicked a button, and the system saved the result. It stored facts and enforced rules, while people did the interpretation, prioritisation and decision-making: reading the reports, spotting problems, working out what to do, and clicking through screens to make it happen.

AI is changing that bargain, gradually rather than all at once. Business software can now help detect exceptions and emerging risks, estimate what is likely to happen, suggest a response and, within limits the business sets, carry out some of the work.

None of this makes existing, reliable software less important. If anything, it matters more. AI is only useful for real work when the systems underneath it are dependable, secure and accountable.

Business software is moving from recording what happened to helping decide what happens next, safely.

Take a retail manager who is worried about running out of popular products before the weekend. Today she would open several reports, compare sales with stock levels, check supplier delivery times and place orders by hand. The information is all there, but the software doesn't know what she is trying to achieve.

With an AI-native product, she can simply ask: "Which stores are likely to run out of our best-selling products before Saturday, and what should we reorder?" The system brings together her sales, stock and supplier data, uses forecasting models to estimate which stores are most at risk, shows the evidence and policy checks behind its suggestions, and prepares draft orders. She reviews them, makes changes if needed, and approves. The orders then go through the normal purchasing process. Afterwards, the product compares the forecast and recommendation with actual sales and availability, building up evidence to evaluate and improve its models.

The AI did the investigating and the paperwork. She kept the decision.

The principle underneath this is simple to state:

AI can investigate, recommend and prepare. Trusted business workflows validate, authorise, execute and record every consequential action.

Those workflows use rules, permissions and, where needed, human approval. A useful way to think about it is a capable new analyst: they can research, draft and propose, but they don't sign off purchases on their own, and their proposals go through the same checks as anyone else's. The AI side of the system can keep its own drafts, notes and traces, but the core business system stays the single source of truth for orders, prices, payments and stock.

How much the AI should do

The choice isn't between doing everything by hand and automating everything. In practice, businesses tend to increase autonomy one task at a time, as the system shows it can be trusted.

The level should be configurable, not fixed: by customer, user role, workflow, monetary value, product category and risk threshold.

Three kinds of intelligence

It is easy to assume that a chat assistant is the whole of "the AI". In a well-designed product it is one of three capabilities, each doing a different job.

A language model shouldn't guess next week's demand. It should ask a forecasting model, combine the answer with current data and company policy, and explain what it found.

Over time, comparing what was predicted with what actually happened gives the evidence needed to improve the forecasts.

The Pivot Stack

The Pivot Stack is the platform we have built for this kind of product, and most of what this article describes comes from the experience of building it: years of enterprise backend work, adopting new open-source patterns, and learning how AI models and machine learning behave in real business software. We describe it here as a reference architecture so the ideas are useful beyond our own product. Its main idea is to keep the everyday, predictable work of the business separate from AI-driven work, with a clear checkpoint between the two.

It is built for multi-tenant SaaS, where each customer is a tenant: its data, configuration, permissions, policies, usage budgets and AI context must stay separate from every other tenant's. The architecture sections below use "tenant" in that sense.

Two design choices do most of the work. First, there are two paths. Everyday transactions, such as saving an order or changing a price on a screen, go straight to the core without touching AI, so they stay fast and predictable. Only AI-initiated work goes through the seam. Second, the seam only lets AI use a registered list of specific, well-described actions, such as "check stock position" or "create a draft order", each with its own permissions. It never gets general access to the database.

The product holds the conversation

In the Pivot Stack, the chat is part of our product, not a direct line to an AI provider. People talk to the product, and the product talks to the language model on their behalf.

In our architecture, the model is used statelessly. It has no memory of the conversation and no access to business systems beyond what the product sends on each call. Some AI providers can hold conversation state or call external tools directly, but we deliberately keep both on the product side, so that every piece of data and every action passes through the same governed checkpoint.

So before each call, a context builder in the seam assembles what the model needs for that request: the relevant tenant data, the policies and limits that apply, the recent conversation, and the list of approved tools it may ask to use. Each piece is checked against what this user is allowed to see, and sensitive details that aren't needed are left out. Retrieved documents and data are clearly marked as information to use, not instructions to follow, which helps protect against instructions hidden inside them. That package goes to the LLM API with the question, and it is what grounds the answer in the customer's actual business rather than in general knowledge.

The response comes back the same way. If the model asks to use a tool, for example to look up stock or create a draft order, the product runs that request through the tool registry and policy checks, returns the result to the model, and the cycle may repeat several times before a final answer is ready. This is one reason a single question can turn into many internal calls. Output checks validate the final response, and only then does anything reach the person or the core.

Keeping the conversation and context on the product side has two further benefits. The AI provider becomes much easier to replace, because history and business knowledge live with the product rather than with the provider, although prompts tuned for one model still need re-testing on another. And cost stays manageable: the amount of context sent is one of the biggest drivers of AI cost, so the context builder sends only what each request needs, and repeated content such as standing instructions and policies can use the prompt caching many providers offer to reduce cost further.

Choosing the right model for each task

Every AI call has a cost, and for software sold on subscription those costs can add up quickly. Sending every request to the largest available model tends to be slower and more expensive than necessary, and makes pricing harder.

Most AI work inside a business product is fairly narrow: reading an invoice, sorting a request, filling in a structured form. For tasks like these, smaller models, often combined with tightly written instructions, structured outputs, relevant company data and sometimes task-specific tuning, can be much more economical than a large general-purpose model. Some tasks don't need language AI at all, because a rule or a forecasting model already has the answer. Larger models are best kept for work that needs them, like investigating something unusual or explaining a complex situation. The right mix depends on volume, accuracy needs, the data available and the cost of getting it wrong.

The model router in the seam makes this choice. It also means the business can change AI providers without rebuilding the product, and it decides what happens when AI isn't available. A good rule of thumb is to fail closed for consequential actions and fail gracefully for assistance. Core screens, APIs and rules keep working; AI features show that they are temporarily unavailable; non-urgent AI work is queued for later; and nothing risky quietly falls back to a less safe route.

What changes in production

More calls, and more waiting

When a person uses a traditional application, one click usually means one request. With AI involved, a single task can trigger several model calls, a forecast, stock and supplier lookups, policy checks and retries. Agent-based workflows can also produce bursts of activity that look very different from human usage. So API traffic can grow substantially even when the number of users stays the same, and many of those calls are slow.

In a traditional "blocking" design, each request holds a server thread for the whole time it waits. With enough slow AI calls, the threads run out and everything slows down, including ordinary transactions. A restaurant is a helpful comparison: a waiter who stands by the kitchen until each dish is ready can serve one table at a time, while one who passes the order on and moves to the next table can look after the whole room. Non-blocking, or reactive, programming works like the second waiter.

The practical measures are: non-blocking services (such as Spring WebFlux in Java) where the system spends its time waiting; asynchronous work through queues and event streams like Kafka for anything that doesn't need an immediate answer; backpressure, so a part that falls behind asks for less work rather than failing; quotas, timeouts and priorities that put people ahead of background agent work; and caching plus duplicate-safe retries to avoid repeated work.

Reactive programming isn't free. It is harder to write, test and debug than traditional code, so it makes most sense in the parts of the system that spend most of their time waiting: the front door, the AI seam and calls to external services. The core business logic can often stay simpler.

Tenants need more careful management

AI raises the bar for multi-tenancy in two ways. The first is fairness. One tenant running a large batch of agent tasks can slow the product for everyone else, so each tenant needs its own quotas, priorities and AI spending budget, enforced before work reaches shared services. Most tenants can share a pool of capacity; very large or regulated tenants may need dedicated capacity.

The second is separation. It used to be mostly about the database. Now it also covers document indexes, AI memory, caches, prompts, evaluation data and usage costs. One tenant's data should never appear in another tenant's AI answer, even indirectly. Each tenant also needs its own settings for autonomy levels, permitted models, and whether its data may be used to improve the product.

Responsiveness and review

AI tasks can take several seconds or more, so it helps to show progress as it happens ("checking stock… running forecast… checking policy"), let people interrupt or correct the work, and make it clear whether something is a suggestion, waiting for approval, or done. If someone is asked to approve fifty suggestions a day, they will start approving without reading, so review screens should draw attention to what is unusual or high-risk.

Long-running work and recovery

An AI task might investigate a problem, wait overnight for approval, then carry out the result. The system needs to remember where each task has got to, pick up again after a failure, time out sensibly, and never carry out the same action twice. Before any action is automated, it also needs a defined recovery path: reversal where possible, a compensating action where not (a refund rather than a reversed payment, an amended order rather than a deleted one), escalation to a person where neither is safe, and approval-only handling for actions that can't be undone, such as messages already sent to customers.

Security

Instructions can be hidden inside an uploaded document, a search can surface files a user isn't allowed to see, and an AI system with broad tool permissions can trigger actions outside the user's intent or authority. The practical answer is that the context builder only includes what the person being helped is allowed to see, and the AI only uses the approved actions in the registry.

External agents

As business products begin to expose capabilities to partner and customer agents, those agents need to show who they are, whose behalf they are acting on and what they are allowed to do. Delegated authority should be explicit, scope-limited, short-lived where possible, and auditable, and never greater than the authority of the person or business behind the agent.

Testing, versioning and decision records

Models, prompts, retrieval settings and evaluation datasets should be versioned like application code. A team should be able to see which model, instructions, data sources and policy configuration produced a recommendation, compare versions, and roll back when quality or cost gets worse. AI changes need their own test scenarios before release and monitoring afterwards.

Each consequential action should also leave a decision record: what started it, which data, forecasts and model were used, which checks ran, what was recommended, who approved it and what happened next. "Why it acted" means this evidence, not a transcript of the model's internal reasoning. It supports customer questions, audits and growing expectations around transparency and human oversight. AI cost should be tracked by tenant and feature in the same way.

What it asks of engineering

AI coding tools can now produce a lot of routine code, and teams may get smaller for that kind of implementation work. The need for experienced judgement tends to grow, though: deciding what should be built, designing clear boundaries between AI and the core, verifying behaviour under load and failure, keeping data clean and tenants separate, and designing review experiences people will actually use. Mistakes can now be produced faster as well, so the ability to check the whole system matters more than the ability to write more of it.

Where the advantage comes from

As software gets faster to build, feature lists become easier to copy. What is harder to copy is a product that:

1. understands each customer's business and rules;
2. predicts outcomes from real operating data;
3. recommends and prepares sensible actions;
4. acts only through trusted, checked business workflows;
5. measures what happened and improves over time.

That advantage doesn't come from a single model, a chat interface or an agent framework. It comes from the trusted loop around them: domain knowledge, clean data, prediction, policy, human judgement, reliable execution and measured outcomes, built up over time with customers.

The business software that does well in the coming years probably won't be the one with the most impressive chat feature. It will be the one that understands each customer's context, helps them see what may happen next, prepares useful actions, and carries them out only through controls the business trusts. That is the shift from record-keeper to decision partner, and it is what we built the Pivot Stack to support.