Jarvis voice grounding architecture

Diagnosis

What actually produced the screenshot

Observed output

split dispatch

The provider started in rivvl and received I need you to check out Zivil. The screenshot proves those two facts. It does not prove that Companion's recent-project fallback chose Rivvl.

Full captured speech

The screenshot is from Full Desktop's compact voice flow, not the standalone Companion process analyzed in the supplied review.

Early client check missed

Capture may start with an empty catalog. The transcript is then queued, and grounding is not rerun after the catalog arrives.

0.0.47 Host skipped grounding

The installed code disabled inferred project grounding whenever context or reference thread IDs were present.

Provider saw raw text

Ambient context supplied Rivvl as the execution project while the ungrounded ASR objective remained unchanged.

Reproduction evidence Direct, deterministic calls against the current pure grounding seam.
Input	Context	Result	Meaning
I need you to check out Zivil	Fresh voice turn	needs input Did you mean Rivvl?	Current Host code catches the exact case.
I need you to check out Zivil	continueContext: true	ready project Rivvl, objective still Zivil	Safe only when continuation was explicitly requested and Zivil is literal task content. It is unsafe as an ambient default.
Same utterance, real eight-project catalog	Full client matcher	phonetic candidate Rivvl	The claimed matcher-margin failure does not reproduce with the installed catalog.

Release mismatch

The running AppImage is Jarvis-0.0.47-x86_64.AppImage, built on 27 Aug at 22:20. It contains the older context suppression. Commit 0784a00a2, pushed on 28 Aug, changes that rule so only deliberate continueContext suppresses project inference. A server-level test now proves that context and reference threads do not stop a spoken project from winning.

Pushback and salvage

What to keep from the supplied review

Keep

One closed interpretation before dispatch

Routing and provider text must come from one result. A project span consumed for routing cannot remain as conflicting free text.

Keep

Ambiguity is typed state

Confirmation must retain the original request, candidate IDs, expected slot, expiry, and request identity. It is not a prompt fragment.

Keep

Hotwords are not correctness

Parakeet context and better pronunciation matching reduce errors. They cannot authorize a project or replace clarification.

Keep

Soundex should not be the final matcher

Technical names need phoneme-aware candidates or learned pronunciations. That improves recall but does not change authority.

Where the review overreaches

Claim	Verdict	Correction
Companion recent-project fallback caused this screenshot.	Wrong path	The screenshot came from Full Desktop. Full uses JarvisManagerDialog → JarvisMesh → JarvisManager, not apps/companion/src/main.ts.
No unresolved entity-bearing ASR text may reach a provider.	Too broad	Branch names, files, symbols, tickets, and literal user terms often belong in the objective. The rule applies to spans Jarvis consumed as control or routing evidence.
Only execute-as-Rivvl or ask “Rivvl?” are legitimate.	False dichotomy	Zivil may be a branch or other literal target inside Rivvl. That third meaning is valid when Jarvis has not also treated the same span as the project selector.
A small LLM should be the Turn Interpreter.	Wrong authority	An optional model may propose a parse over bounded candidates. The deterministic Director still resolves IDs, confidence, ambiguity, authorization, and dispatch.
The 0.68 matcher threshold explains the failure.	Unproven	That threshold belongs to the client lexicon matcher. The Host uses different rules, and the actual catalog returns Rivvl as a phonetic client candidate.
Compile a generic inspect-project objective.	Needs product semantics	Compilation must preserve the user's task. If “check out <project>” is defined as an inspection command, encode that deliberately and test it. Do not silently invent a richer objective.

End architecture

Deepen the existing Director seam

Do not add a second authoritative “Turn Interpreter” beside prepareJarvisTurn. Turn that existing seam into one deep module that owns extraction, candidate generation, grounding, clarification, and objective compilation. Full and Companion may run it early for feedback. The Host must run or validate it before any provider command.

RecognitionEnvelope
  rawText, captureId, inputMode, ASR metadata
          │
          ▼
GroundedTurnDirector
  real ProjectRef / TaskRef catalogs
  provider + model catalogs
  explicit continuation state
  ambient focus and attention as defaults only
  confirmed pronunciation aliases
          │
          ├── Clarify(frame with candidates + original request)
          ├── Reject(reason)
          └── Execute(closed control plan)
                    │
                    ▼
          deterministic plan validator
                    │
                    ▼
          provider prompt compiler
                    │
                    ▼
          ordinary T3 orchestration command

Small external interface

Input

One recognition envelope plus one bounded context snapshot.

raw speechProjectRef catalogTaskRef catalogprovider catalogexplicit continuationambient defaultsaliases

Closed output union

Output

Callers never combine a project resolver result with an independently edited transcript.

executeclarifyrejecttyped control

Execution plan contents

Control facts

Action, exact node-qualified project, optional exact task, provider/model selection, and whether continuation was explicitly requested.

Canonical objective

Natural task text with consumed control spans removed or replaced. Literal branch, file, symbol, and ticket terms remain.

Grounding evidence

Each consumed span records heard text, role, canonical entity, match kind, and confirmation source. This is diagnostic data, not provider text.

Source metadata

Raw ASR text, capture and request IDs, and recognition metadata stay available for replay, metrics, and corrections.

Hard invariant

If a span selected or changed a project, task, provider, model, approval, or control action, the provider objective cannot retain that span with a conflicting meaning. Ambient focus may fill a missing route. It may never resolve an explicit but uncertain route silently.

Where an LLM fits

Not on the critical path by default. Add it later as an internal candidate-proposal adapter when deterministic parsing cannot label a long paraphrase. Give it bounded candidate handles, not raw authority. Reject invented handles. Re-score deterministically. Clarify on ties, timeout, malformed output, or model unavailability. The system must remain correct with that adapter disabled.

Acceptance

Cases the architecture must lock down

Utterance and state	Required result	Forbidden result
Voice: “check out Zivil”; Rivvl is a unique phonetic project candidate	Clarify before thread creation	Project Rivvl plus provider text Zivil
User confirms Rivvl	Resume the same request ID once, store scoped pronunciation, compile canonical objective	Start a second request or ask the provider to infer Zivil
“In Rivvl, check out branch Zivil”	Route to Rivvl and preserve branch Zivil as literal objective text	Rewrite both spans to Rivvl
Explicit continuation of the current task with “check out Zivil”	Keep current TaskRef; preserve literal Zivil unless another bounded entity catalog resolves it	Treat ambient focus as proof that Zivil means Rivvl
Capture completes before project catalog loads	Queue the envelope, then ground against the fresh catalog	Queue a pre-grounded raw transcript and skip reevaluation
Two Rivvl projects on two nodes	Clarify with node labels	Use recency to choose a node silently
Typed: git checkout zivil	Preserve literal command intent under normal approval policy	Run ASR phonetic project correction
Model parser disabled or timed out	Deterministic result or clarification	Loss of basic voice routing

Implementation sequence

Small slices, one authority

Lock the exact regression

Add Zivil tests at the Director and JarvisManager seams, with ambient context present. Repair the orphaned groundVoiceTurn suite.

Enforce plan invariants

Make provider prompt compilation accept only a closed plan. Assert route and consumed-span consistency before dispatch.

Collapse duplicate paths

Move Full and Companion extraction, matching, confirmation, and canonicalization behind the shared Director module.

Close the catalog race

Queue recognition envelopes and ground at dequeue time against the current catalog. Client preview may be discarded and recomputed.

Improve candidate recall

Add pronunciation-aware matching and telemetry after the correctness seam is fixed. Keep thresholds conservative.

Evaluate optional semantics

Only then test a bounded model proposal adapter against real captured utterances and compare it with deterministic clarification rates.

Current source verdict

main bug patchedarchitecture not done

The pushed commit fixes the specific Host rule that made ambient thread context suppress project grounding. It should not be called complete yet. The broken missing-module test, client catalog race, and duplicated grounding implementations leave the same class of split decision possible elsewhere.

Evidence

Local sources checked

apps/web/src/components/jarvis/JarvisManagerDialog.tsx, apps/web/src/components/jarvis/JarvisNativeCapture.ts, packages/jarvis-client-runtime/src/jarvis/voiceLexicon.ts, packages/jarvis-core/src/prepareJarvisTurn.ts, packages/jarvis-core/src/resolveProjectTarget.ts, apps/server/src/jarvis/Layers/JarvisManager.ts, apps/companion/src/voice-routing.ts, contracts and task-desk architecture docs.

Verification: five focused test files, 72 tests passed in 1.52 seconds. The separate groundVoiceTurn.test.ts suite failed before collecting tests because groundVoiceTurn.ts is absent.