ANDREI
TEKHTELEV
Back to engineering workAPPLIED AI · STEP

AI coaching in production.

People wanted useful answers about their own training inside the product, not a chat window bolted onto it. The model is one component of that system. The decision worth writing down is when the system should go and look something up — and when it should not.

Context
Step · AI coaching inside a consumer fitness product
My role
AI integration, backend and client delivery, output validation, performance investigation. Part of a team.
Material status
Written from my own account of the work. Each number below is listed with its source and limits at the end of the page. Job title and employment dates are not published here.

Problem

A user asks the coach something. Sometimes it is about them — what their week looked like, whether a plan is realistic given what they actually did. Sometimes it is general. Sometimes it is not really a question at all: "thanks", "got it", a follow-up that only refers to the previous message.

All of those arrive through the same input. The product needs the first kind to be grounded in real stored knowledge, and the last kind to just answer, quickly. The first version treated every message the same way: fetch context, then generate. That is the safe default, and it is what most integrations start with.

It is also where the cost is. Retrieval is work — it happens before generation, so the user waits for it whether or not it contributed anything. On a short acknowledgement it contributes nothing and costs the same. The problem was not that retrieval was slow; it was that retrieval was unconditional.

My role

The coaching features were a team effort. Split by what I did:

  • I implementedthe AI integration end to end for these features — the backend handler, the client surface, streaming of responses, and tool calling.
  • I implementedvalidation of model output, so that what reaches the product is checked against an expected shape instead of being trusted because it reads well.
  • I proposed and ledthe investigation into where the time in a coach response actually went, and the move from unconditional retrieval to a tool the model can call.
  • I proposed and ledthe argument for keeping the boundary explicit: the model may request retrieval, but it does not decide what the application is allowed to do with the result.
  • Shared with the teamproduct direction, the coaching content itself, and the surrounding platform.

Key decision

Use retrieval as a tool the model can call, instead of a step that runs on every message.

Always-on retrieval is easy to reason about: every generation sees context, so nothing is ever missing. The price is paid on every message — including the ones where no stored knowledge is relevant. And it hides a design question rather than answering it, because nobody has to decide which questions actually depend on retrieved knowledge.

As a tool, the shape inverts. The default path is answer directly; retrieval happens when the model requests it, and the request is a tool call the application handles explicitly. That makes retrieval visible and auditable — there is a call, with arguments, that either happened or did not — and it means the cost lands on the conversations that need it.

The boundaries I kept fixed: the model may ask for retrieval, but the application decides what a retrieval call is permitted to fetch, validates the arguments coming in, and validates the output going back before it becomes part of a response. A tool call is a request, not an instruction to obey.

How it works

A simplified request path. It is drawn to show where the boundaries are — which component may decide what — rather than to document the internal wiring, which I am not publishing.

  1. Client sends the message

    The app sends the user's message and the conversation it belongs to.

  2. Backend handler prepares the request

    The handler assembles the request for the model and declares the small set of tools available on this call — content search, knowledge retrieval, a read of the user's own precomputed insights, and a workout suggestion.

  3. Retrieval, only if requested

    If the model calls the retrieval tool, the application executes it with validated arguments and returns the result. On most messages this step does not happen.

  4. Output is validated

    The response is checked against the shape the product expects before any of it is used, and a validation failure is recorded as its own signal rather than passed through.

  5. Response streams back

    Validated output streams to the client so reading can start before generation finishes.

The conditional step is 03. Everything before it is the same on every message; everything after it is the same whether or not retrieval ran. That is what makes retrieval removable without redesigning the path.

In prose: the client sends a message, the backend decides what the model is allowed to reach for, the model answers — pausing to call retrieval only when it needs stored knowledge — and the result is validated and streamed back.

Two things are deliberately not in that description. I am not claiming a set of routing rules that classify messages before the model sees them; the decision is made through the tool call, not by a classifier I am describing here. And I am not stating the exact internal services — this is the boundary view, which is the part I can state accurately.

Adjacent work on the same feature, in one paragraph

Three other pieces of work sat next to this one and are not the subject of this page: precomputing coach intelligence so some context is ready before a conversation starts, routing by intent, and a video pipeline. Each would need its own explanation and its own evidence to be worth publishing.

They are mentioned here only so the picture is not misleading — the coaching feature was more than this one decision. But one decision explained properly is more useful than five listed, so this page stays on retrieval.

Verification

What is checked, what is reported, and what is still open.

  • Documented
    Schema validation on the response path

    Model output is validated against the expected shape before the product uses it, so a malformed or unexpected response fails a check instead of reaching the UI, and the failure is recorded as its own telemetry. By construction this runs on every response rather than when someone remembers to run it — but that is my description of the design, not an artifact you can inspect, which is why it is marked as documented rather than as a verified automated check.

  • Documented
    ~300–500 ms of retrieval overhead removed from about 90% of messages

    Read this as a design estimate, not a measurement. It describes the retrieval step's own cost on messages that no longer trigger it — not total response latency, not time-to-first-token. When I traced it back to its source for this page, it rests on the design decision record for the change rather than on a benchmark or a monitoring query, unlike neighbouring figures in the same notes that cite staging-verified measurements. The ~90% is the reported share of messages that do not need retrieval.

    Source inspected, not independently verified · Step Experience Master Document §1.5 / §3.B; §7 attributes the figure to the on-demand-retrieval design decision record · 2026-09-11

  • Open question
    Is retrieval ever skipped when it was needed?

    The risk this decision introduces, and the check that matters most. A harness for it exists — four scenarios, twenty queries, scored on expected content — but it writes nothing, runs in no pipeline, and has no recorded execution. Built, never turned into evidence. Stated as open rather than answered.

    Source inspected, not independently verified · Retrieval-quality script in the Step app repository (inspected 2026-09-24); Step Experience Master Document §3.K / Appendix G8 · 2026-09-24

  • Not presented
    Published latency or quality benchmark

    None. No measurement of this change is published on this site, so nothing here should be read as a verified performance result.

Outcome & limitations

What the work produced

  • Coaching shipped as part of the product, with retrieval, streaming responses and tool use working together inside an explicit application contract.
  • Retrieval became conditional and visible: it is a call that either happened or did not, on the conversations that need it.
  • Model output is validated before it is used, so the product's behaviour does not depend on the model being well-behaved.
  • Reported effect: roughly 300–500 ms of retrieval overhead removed from about 90% of messages — my own design estimate, defined above, and not a benchmark.

What this does not show

  • The latency figure is an estimate from the design decision, not a measurement you or I can inspect. It describes the retrieval step's overhead, not end-to-end response time or time-to-first-token.
  • Answer quality after the change is not demonstrated. The harness that would demonstrate it exists but has never produced a recorded result, which is a gap in my work rather than a detail of presentation.
  • The internal routing is described at the boundary level only. I am not publishing the exact services or rules.
  • Targets that existed for this path — a p99 under 50 ms, time-to-first-token under 200 ms — are targets. No measurement against them is presented, so they are not published as results.

Sources & available artifacts

Step's AI code and evaluation tooling are internal. What follows is what each claim rests on.

  • My own account
    My account of the integration and the retrieval decision

    The handler, tools and validation live in Step's private codebase.

    Not public
  • Technical write-up
    My own experience document for this work (§3.B, Appendices A, G)

    Inspected while writing this page. It is where the latency estimate and the harness are recorded — and where I confirmed that the estimate traces to a design decision rather than a measurement.

    Not public
  • Source code
    The retrieval-quality harness itself

    Exists in the app repository: four scenarios, twenty queries, keyword scoring. Inspected for this page. It prints to the terminal and stores nothing, it is not wired into the build, and no run of it is recorded anywhere — so there is no result to cite, and none is being withheld either. It simply does not exist.

    Not public
Every figure on this page, with what it measures and what it does not prove (3)

  • Making retrieval a tool instead of an unconditional step took roughly 300–500 ms of retrieval overhead off about 90% of coach messages.

    Verification
    Source inspected, not independently verified
    Source
    Step Experience Master Document §1.5 / §3.B; §7 attributes the figure to the on-demand-retrieval design decision record (2026-09-11)
    Environment
    production
    What is measured
    The cost of the retrieval step itself on messages that no longer trigger it, with ~90% being the reported share of messages that do not need retrieval. It is not total response latency and not time-to-first-token. Environment is not applicable: this is an estimate attached to a design decision, not a measurement taken in an environment.
    Limits
    This is a design estimate, not a measurement. The document's own source for it is a design-decision record rather than a benchmark or a monitoring query, unlike neighbouring figures that cite staging-verified measurements. A faster path is also not evidence that answer quality held.

  • A scenario-based retrieval-quality harness exists — four scenarios, twenty queries, keyword-relevance scoring with per-scenario retrieval depth.

    Verification
    Source inspected, not independently verified
    Source
    Retrieval-quality script in the Step app repository (inspected 2026-09-24); Step Experience Master Document §3.K / Appendix G8 (2026-09-24)
    Environment
    local
    What is measured
    A script that runs a fixed set of queries per scenario and scores whether retrieved content contains the expected keywords. Existence and size only — no result is being reported.
    Limits
    No result of running it exists anywhere: the script only prints to standard output and has no code path that writes a report, it is not wired into CI, and neither the repository history nor the source document records a single run. So the question it was built to answer — is retrieval ever skipped when it was needed — is open, not answered.

  • Targets of p99 under 50 ms and time-to-first-token under 200 ms.

    Verification
    Target / planned
    Source
    Step Experience Master Document, §4.4
    Environment
    unknown
    What is measured
    Stated service targets for the coach path.
    Limits
    Targets only. No measurement was found, so they must not be presented as achieved.