PR 346: token usage in the chat footer

torchiaf's PR, reviewed twice at 79968cc: Cody's process (in this session) and ours (a fresh agent that didn't see the first), then merged and re-checked. Nothing was posted to GitHub.

Preview: rancheraipr346r, built from 79968cc, backend 13.62 (torchiaf's AI Rancher, v2.15.1): its agent is a local build with /v1/api/chats/{id}/usage and llm-mock v1.2.3, planning off. Baseline: rancher-ai-main at 28ae86d (rebuilt today by another session), backend mf, whose AI agent is evicted, so the baseline only shows the footer, not a working chat.

Where it stands: works with the new agent; merge it after agent PR 507, and fix how failures and source names show

On a running build against an agent that has the endpoint, everything the PR describes works: the count appears after the welcome message (0, then 1.8K), refreshes after each answer (8.5K), matches the API to the token, resets on New chat, follows the chat when you open one from History, and survives closing the panel. It holds at a 250px panel and in dark.

What I'd raise, in order: 1 the new spec can't pass before agent PR 507 merges, and its exact counts will break on agent prompt changes; 2 any failed request shows "Tokens used: 0", even over a count it already had; 3 By Source shows raw ids because its label keys aren't in this PR; 4 the button's accessible name has no "tokens". Then three optional cleanups (5). Suggested as a Comment review, not Request changes: the merge order is something torchiaf already knows, since he wrote both PRs.

How the two passes compare

#FindingPass A (Cody)Pass B (ours)Lands as
1New spec needs agent PR 507; exact counts tied to the agent's promptsBase review: both partsB1 (blocker): the merge order onlyFix / merge order
2A failed usage request shows "Tokens used: 0"Base review: 404 from the startS1: also a 500 that wipes a shown 1.8KFix
3By Source shows raw ids: the label keys aren't in this PRBase review, live; also flagged supervisorS2, live + a jest test against the real en-us.yamlFix
4The button is named "1.8K in total", nothing about tokensa11y pass (AX tree)S3 (ariaSnapshot)Fix
5Unused fetch:usage emit, unused disabled prop, no unit test for fetchUsageSeen, not filed (no-nit rule)N1, N2, N3Optional
6Look: "Tokens spent", exact total, supervisor row, long names cut the numberDesign-fidelity passLook-only listFor the designer
7Popover: Escape doesn't close it, role="menu" with no itemsa11y pass, removed as already on mainNot seenFYI, not raised
CIScroll test redFails on main tooFails on main tooNot this PR
Pass A (Cody's process)Pass B (our process)
VerdictComment: 5 line comments + the demo pairRequest changes: 1 blocker, 3 suggestions, 3 nits
Checked by bothThe main flow live against the API (welcome, prompt, new chat, history, reopen), a stubbed 404, the raw source labels, the trigger's accessible name, the design screenshot, a long agent name truncating the number, and the CI tabulation (scroll test also red on main).Same.
Only one pass didBefore/after demo videos. A sourced acceptance table (issue + ux 559). The llm-mock source showing counts are len(body)/4, so the spec's numbers depend on the agent's prompts. His a11y probe: contrast clean in both themes, target size within the spacing rule, Escape and aria-required-children on the popover (both already on main). A 250 and 320px panel.A 500 after a shown count (wipes it to 0). A 7s-delayed response arriving after New chat (the guard works). A scratch jest test against the real en-us.yaml (the unit test mocks the missing key). The unused emit and prop, the untested fetchUsage, PR 507 making chatId required, the 19px taller console, a coverage grid.
MissedThe 500-after-a-count case (same cause as its 404 finding). The unit test that hides finding 3.That the spec's exact counts come from other repos' prompts. The pre-existing popover issues (out of scope anyway).
Outputacceptance-criteria.md, acceptance-verification.md, comments.md, preexisting-on-master.md, a11y probe JSON, 2 videosreview.md, coverage.md, checked.md, draft.md, 19 screenshots, a CI tabulation
CostIn this session, about 75 minutes including the shared setup (preview, choosing the backend, the design)A fresh agent: 16 minutes, 76 tool calls, about 214k tokens

Findings

1

The new spec can't pass before agent PR 507 merges, and its exact counts follow the agent's prompts

Found by Pass A (Cody)Found by Pass B (ours)
Fix / merge order

CI deploys rancher-ai-agent from main (.github/scripts/deploy-rancher-ai.sh clones it with no ref). Main's app/routers/chat.py has no usage route; PR 507's head adds @router.get("/chats/{chat_id}/usage"). So every usage call 404s, the label stays " 0 ", and the spec fails all 3 attempts in both runs since it was added (43471969845, 43834164964). The description doesn't mention the dependency.

Separately (Pass A only): llm-mock 1.2.3 computes input = (len(request body)+3)/4 (internal/handlers/models/usage.go). The body is the agent's system prompt, the MCP tool schemas and the page context, so assertions like 1,747, 8,384 and 41,690 change when anyone edits a prompt in the agent repo. Live with default mock replies I got 1,747 / 24 for the welcome and 8,385 / 72 after one prompt, where the spec expects 25 and 8,384, because its queued replies differ slightly.

2

A failed usage request shows "Tokens used: 0", even over a count it had

Found by Pass A (Cody)Found by Pass B (ours)
Fix

fetchChatUsage turns every failure into null, fetchUsage writes it over the current value, and TokenUsage renders null as 0 with the button disabled. I re-checked Pass B's case: a real 1.8K, then a stubbed 500 on the refresh after a prompt, and the footer reads "Tokens used: 0", disabled. With a stubbed 404 on every call (an agent without the endpoint) it stays at 0 for the whole chat and logs "Failed to fetch chat usage" after each answer. Note PR 507 itself answers 404 for a chat with no recorded usage.

Footer reading Tokens used: 0 after a refresh failed with a 500.
After a 500 on the refresh (stubbed): the 1.8K is gone.
Footer reading Tokens used: 0 with every usage call answered 404.
Every usage call 404 (stubbed), after a prompt.
3

By Source shows raw ids: the label keys aren't in this PR

Found by Pass A (Cody)Found by Pass B (ours)
Fix

sourceLabel looks up aiConfig.form.section.usage.sources.*. Those keys exist only on torchiaf's bu-feature-token-consumption branch, with the settings Usage page this PR left out, so every source shows its backend id: ui-tools, agent, planner, summarization. The design has "UI Tools" and "Planner". The unit test mocks the planner key into both t and i18n/exists, so it passes; Pass B's scratch test against the real en-us.yaml gets the raw ids back. The e2e asserts the raw ui-tools.

Same family, your call whether to add it: By Agent shows supervisor. In a single-agent chat all usage lands there, since PR 507 credits the orchestrator as supervisor and agentLabel only knows real agents. The issue's example calls it "Liz (General)"; the design shows only named agents. Pass B kept this as look-only, Pass A filed it; I left it out of the suggested text.

Popover listing supervisor, ui-tools and agent rows.
PR, after one prompt: supervisor, ui-tools, agent.
Design popover with SUSE Observability Agent, SUSE Security Agent, Planner and UI Tools.
Design (rancher/ux 559 screenshot).
4

The count's button is named "1.8K in total", with nothing about tokens

Found by Pass A (Cody)Found by Pass B (ours)
Fix

"Tokens used:" is a plain span next to the button and isn't linked to it. Tabbing from the prompt goes agent select, "Verify the results.", then "1.8K in total" (menu button), so a screen-reader user hears a number and no subject. The neighbouring "Verify the results." makes sense on its own. WCAG 2.4.6 / 1.3.1; a usability risk more than a strict failure.

5
Optional

'fetch:usage' is declared in Console's emits and nothing emits or listens to it. TokenUsage's disabled prop has no caller. fetchUsage (the reset, the guard for a chat the user left) has no unit test; the composable's test only gained the error log mock. Pass B checked the guard live: a response delayed 7s arrived after New chat and was ignored. Cody's no-nit rule drops these; ours keeps them, marked optional.

6

Look and wording differences from the design

Found by Pass A (Cody)Found by Pass B (ours)
For the designer

Kept out of the review: a UI-team PR whose design oboc-sts owns. See the next section.

7

The popover ignores Escape and is a menu with no items (already on main)

Found by Pass A (Cody)
FYI

Enter opens the token popover; Escape leaves it open with aria-expanded="true". Its content sits in RcDropdown's role="menu" with no menu items (axe aria-required-children, critical). Both come from the existing TextLabel popover and show the same on "Verify the results.", so they're not this PR's. Worth an issue of its own if you want.

For the designer, not in the review

From rancher/ux 559 (the Figma API token has expired, so this is against the screenshot on that issue): the footer reads "Tokens spent: 17,964 in total", the exact number like the popover; the PR reads "Tokens used: 8.5K in total", compact on purpose (its code comment says so). The design's By Agent has only named agents, no supervisor. Section titles and row sizes match. Both passes also saw a long agent name squeeze the number to "400,0…" or "34,5…" (stubbed names; the full value is in the tooltip). The extra footer line makes the console 19px taller, which the design also does.

PR title typo: "consumtion". It becomes the squash commit subject.

The ux 559 design screenshot with the token popover open over the Liz panel.
Design, rancher/ux 559.
PR popover with stubbed large numbers and a long agent name; its value reads 400,0….
PR with a stubbed long agent name: the value is cut.

Pass A: Cody's process

Followed my-pr-full-review from codyrancher/ai-skills at c04c11a, mapped to our setup: the PR preview for his dev server, rancher-ai-main for his master build, 13.62 behind the PR, local files instead of his harness, our recorder for the demos. It ran in this session (the passes in his table order, each written up on its own). His a11y probe ran from the clone against a Chromium with a CDP port.

Before this change: main.
With this change: the PR.

Acceptance, from issue 189, its comments and rancher/ux 559

Criterion (source)Verdict
AC1Input, output and total shown (issue: "Capture input, output, and total token counts")Met
AC2Updates after every interaction, welcome and sub-agents included (issue: "for every interaction")Met; sub-agent attribution only with a stubbed body
AC3Per-agent breakdown, Liz and each agent by name (issue: "Liz (General): 850 tokens")Partly met: the orchestrator shows as supervisor
AC4Matches rancher/ux 559 (samjustus: "needs UX ticket")Partly met: label, number format, source names, supervisor row
AC5New chat resets; an old chat shows its own count (implied)Met
AC6Console still works, light/dark/narrow (implied)Met
-Per-turn count under each answer (issue user flow) vs per-chat footer (design)Open question; the later design was graded

Pass B: our process

A fresh agent that didn't see Pass A, following prompts/code-review.prompt.md and your review rules. Verdict: Request changes, for the CI dependency.

CI

e2e is red with two final failures; i18n, lint and unit tests pass. The token-usage spec is this PR (finding 1). The scroll test also fails on main, so it isn't.

RunCommitToken usage specScroll test (expand/collapse planning)
4383416496479968cc (this PR)fails 3/3, label " 0 "fails 3/3
43471969845fd493f1 (this PR)fails 3/3passes
4283364566928ae86d (main, push)not in mainfails 3/3
428313021243593589 (main, push)not in mainfails 3/3
43104450587Dependabotnot in mainfails 3/3

Checked, and not checked

Not checked:

What I'd post

In your shape: a short intro, one inline per point, no code, no fix, no preview link. Comments 5 to 7 are the optional nits; drop them if you want it lighter. The two screenshots go in as attachments when posting. Before/after demo videos are on this page if you want them in the body.

Review body (Comment)

I tried it on a running build against an agent with rancher/rancher-ai-agent#507: the count, the popover, new chat, history and reopening the panel all work and match the API. A few points inline. The other red e2e (scroll handling) fails on main too, so it isn't this PR.

cypress/e2e/tests/features/chat/chat-token-usage.spec.ts:48

This spec can't pass until rancher/rancher-ai-agent#507 is merged: CI deploys agent main, which has no usage endpoint, so the label stays at 0 on all three attempts. The exact counts are also llm-mock's estimate of each request's size, so a later prompt or MCP tool change in the agent repo will fail it again.

pkg/rancher-ai-ui/composables/useChatMessageComposable.ts:912

A failed usage request lands here as null, so the footer drops to "Tokens used: 0" and disables the button, even right after it showed 1.8K (a 500 on the refresh). Against an agent without the endpoint it reads 0 for the whole chat, which looks like nothing was spent rather than unknown.

[screenshot: the footer after a 500 on the refresh]

pkg/rancher-ai-ui/components/console/TokenUsage.vue:59

These keys aren't in en-us.yaml, so By Source always shows the raw values: ui-tools, agent, planner, summarization. The unit test mocks the planner key, so it doesn't catch it.

[screenshot: the popover after one prompt]

pkg/rancher-ai-ui/components/console/TokenUsage.vue:100

The button's accessible name is only "1.8K in total": "Tokens used:" is a separate span, so tabbing to it announces a number with a menu popup and nothing about tokens.

pkg/rancher-ai-ui/components/panels/Console.vue:76

Optional: `fetch:usage` is declared but never emitted or listened to.

pkg/rancher-ai-ui/components/console/TokenUsage.vue:20

Optional: nothing passes `disabled`; Console only passes the usage and the agents.

pkg/rancher-ai-ui/composables/useChatMessageComposable.ts:897

Optional: `fetchUsage` (the reset and the check that drops a response for a chat the user already left) has no unit test.