An AI agent for questions about pay

Context

My role

On this project I owned the interface, the conversation design, and the design side of capability scoping.

Conversation design meant the tone of answers, the formats they rendered in, and the vocabulary they used. Scoping meant drawing the boundary around what the agent was allowed to respond to, and writing the plan for widening it over time. Remuner had no product manager at the time; the CEO and CTO covered that ground, and the agent's scope was decided between them, the engineers, and me.

The problem

Reaching an answer took too many screens

Reps could already see their commissions and plan setup in the platform. Getting from a question to an answer meant knowing where to look, and most questions spanned several screens.

We had been improving those screens step by step when a faster route appeared: let reps ask in plain language and skip the navigation. The information was already computed and stored; reps had no direct way to ask for it.

Leadership pushed hard for the agent, and competitors were shipping similar features, so the project was a market response as much as a user need. We worked from the assumption that a rep who catches the agent wrong about their own commission stops trusting it.

Interviews before release backed the concept: reps described it as a genuine tool for their day-to-day work. Whether they would trust the answers could only be tested in production.

Scope

Deciding what the agent answers, and what it refuses

The first design decision was which questions the agent would answer and which it would refuse.

We scoped it to commissions, compensation plans, and what-if payout scenarios using the simulator reps already had in the product. Nothing else. Any question outside that boundary got a standard refusal that told the rep what the agent could answer, instead of an improvised reply.

A wrong guess about pay would have cost more trust than any refusal, so refusing was the default for everything outside the boundary. The narrow launch scope came with a release path, a written plan for which question types would become answerable next.

The other boundary was computational. The agent calculated nothing itself; it read the results the platform had already computed.

Behind the conversation, engineering ran a separate agent for each vertical of the app (plans, the earnings simulator, line items, and others), with an orchestrator routing each question to the right one. That setup existed because a single agent had produced inconsistent answers, most likely from the amount of context it had to manage.

Trust

The anatomy of an answer

Every consequential answer had the same structure: the number first, then how it was built, then the plan rules and context behind it.

A rep asking "how much am I making this quarter" got the figure immediately, with the calculation detail and the relevant plan conditions underneath. They could stop at the number or follow it down to the rule that produced it.

Answers rendered in the format the question called for. Line items came back as tables, breakdowns as lists, single figures as a sentence. Lists and tables already outperformed paragraphs elsewhere in the platform, validated with users, and a table crammed into paragraph form is hard to follow. We confirmed the same pattern in the chat prototype, testing formats with users before launch.

Vocabulary mattered more than I expected. The same object was a "deal" at one customer, a "contract" at another, an "item" at a third. An agent that calls your deal an item reads as a tool that doesn't know your business. Adapting terminology per customer became part of the iteration loop.

After release

Where answers failed, and what we did about it

Feedback after release was mixed. Three problems drove most of the iteration work.

The first was wrong numbers, and most of them traced back to custom code. Some clients had requirements that couldn't be configured through the product, so engineering had written custom calculations for them. The agent didn't know that code existed and answered from the standard calculation path, so for those clients its numbers could be wrong. Reading the platform's own calculations protected accuracy only as far as the standard configuration reached.

The second was comprehension. Compensation plans with tiers and multipliers produce figures that need explaining, and a rep could get a correct number and still not understand it. This drove most of the format and tone work: trying different ways to show the calculation until the explanation landed.

The third was data recency. When the ERP data behind a calculation was hours stale, the agent reported a stale number, and the rep saw a figure that didn't match what they expected. The platform showed a last-updated date, but globally, not inside the conversation where reps hit the mismatch. We never fully solved that inside the chat. If I built this again, answer-level freshness would be in the first release.

We had no ground truth for answer quality, so we instrumented the conversation instead. The chat had thumbs up and down buttons, and a thumbs-down opened a text field asking what went wrong. We also captured every question and answer the agents produced, and reviewed the feedback weekly against that full record. In that review, an answer with no follow-up question and no thumbs-down counted as valid.