On-Device AI (Gemma)

Pitcher exposes AI inference to apps through one JS-API surface — the app calls a method and doesn't care where the answer comes from. Impact owns the model on each platform:

iOS runs a Gemma model fully on-device through llama.cpp with Metal acceleration. Inference is offline, private (nothing leaves the device), and free per query.
Web has no local model — the same surface routes to the online (Bedrock) backend instead.

Your app talks to it through usePitcherApi(), exactly like any other bridge call.

import { usePitcherApi } from '@pitcher/js-api'
const api = usePitcherApi()

Which method should I use?

Method	Use it when	Platforms
`piaSearchAnswer()`	You want an answer to a rep's question over the instance's content. Start here — it handles on-device vs online routing for you.	iOS + Web
`aiGetCapabilities()`	You need to know whether/what AI is available before showing UI.	iOS + Web
`aiComplete()`	You need raw, low-level text completion on the on-device model.	iOS only (web rejects)

TIP

Most apps only need piaSearchAnswer(). Reach for aiComplete() only when you're driving the on-device model directly and are handling the iOS-only / web-fallback split yourself.

Field casing follows your SDK casing option

Responses (and raw ai.* event bodies) come back in snake_case by default (casing: 'snake'). If you initialize the API with casing: 'camel', they are camelCased instead — result.tokens_generated becomes result.tokensGenerated. Request payload keys are already snake_case (max_tokens) — don't rename them. The examples below use the default snake_case.

`piaSearchAnswer()` — the high-level call

Answers a natural-language question over the instance's content and returns a unified result, regardless of which runtime served it.

const result = await api.piaSearchAnswer({
  query: 'What is our latest pricing for oncology products?',
})

console.log(result.answer)          // the answer text
console.log(result.source)          // 'on_device' | 'online' — which runtime answered
console.log(result.cited_file_ids)  // files the answer drew on (online path)
console.log(result.confidence)      // 'high' | 'medium' | 'low' (online path)

Payload (PiaSearchAnswerPayload):

query (required) — the rep's question.
instance_id — defaults to the env's active instance.
context_text — client-assembled content context (file names/tags/summaries). Used only by the on-device path; the online route builds its own context server-side.
file_ids — narrows the online route's candidate pool.
max_tokens — token budget for the on-device completion.

Routing is driven by the pia_search_config LaunchDarkly flag (see Configuration):

on_device and iOS and a model is available → on-device Gemma over your context_text. Any failure (bridge timeout, model unavailable/downloading, completion error) transparently falls back to online.
online (or any on-device fallback) → the next-core Bedrock route, which assembles candidate context server-side.
off → the call rejects (pia_search_disabled).

Because fallback is automatic, a single piaSearchAnswer() call is safe to make on any platform — you only need to handle the answer and a possible rejection.

`aiGetCapabilities()` — is AI available?

const caps = await api.aiGetCapabilities()
if (!caps.available) {
  // Hide AI UI. caps.unavailability_reason explains why (iOS).
}

On web it resolves statically to the online runtime: { available: true, offline: false, runtime: 'bedrock-online' }.
On iOS it reflects the native on-device runtime (AIRuntimeCapabilities):
- available — inference is possible right now.
- offline — true for the on-device path.
- llm_model — the loaded model identifier.
- model_available_for_download, model_downloading, download_progress_percent — download state, so you can render "Tap to enable AI" / "Downloading 42%".
- unavailability_reason — human-readable "why not" when available is false.

`aiComplete()` — raw on-device completion (iOS only)

Low-level, text-in / text-out completion on the on-device model. iOS only — on web the host rejects; use piaSearchAnswer() for online inference.

const completion = await api.aiComplete({
  prompt: 'Summarize the following call notes in 3 bullets:\n' + notes,
  max_tokens: 256, // optional; on-device runtime defaults to 512
})
console.log(completion.text)

The payload is the exact iOS wire contract: { prompt, max_tokens?, stream? } → { text, tokens_generated?, elapsed_seconds?, time_to_first_token_seconds? }.

TIP

The runtime is model-agnostic and applies no chat template. For best results with Gemma, wrap your prompt yourself:

const prompt = `<start_of_turn>user\n${userText}<end_of_turn>\n<start_of_turn>model\n`

On-device constraints (enforced natively): prompt capped at 32,000 bytes; max_tokens defaults to 512, hard-capped at 1024; a 30s per-request budget; one inference at a time.

Configuration

These are set per instance (backend / LaunchDarkly), not from your app, but they govern behavior:

Setting	Values / default	Purpose
`pia_search_config` (LD)	`{ mode: 'off' \| 'on_device' \| 'online' }`	How `piaSearchAnswer()` serves inference.
`offline_ai_model_tag` (iOS)	default `offline-ai-model`	The Pitcher file tag the Gemma GGUF asset must carry, so the device can find and download it.
`offline_ai_cellular_download_policy` (iOS)	`always_ask` / `wifi_only` / `allow_cellular`	Whether to confirm the (multi-GB) model download on cellular.
`enable_meeting_notes_ai` (iOS)	default off	Gates the on-device meeting-notes capture/summarize pipeline.

The Gemma model file (~2.9 GB) is downloaded on demand the first time it's needed on a device; there's a native confirmation prompt subject to the cellular policy. Until it's present, aiGetCapabilities() reports model_available_for_download / model_downloading.

Advanced: raw bridge access (iOS)

The iOS on-device runtime exposes more than the typed SDK surface above — function calling, cancellation, token streaming, model-download events, and meeting-notes summaries. These are not part of the typed @pitcher/js-api surface; you reach them through the low-level bridge on usePitcherApi(). They are iOS-only and their shapes may change — prefer the typed methods above where they suffice.

Raw requests go through api.API.request(type, body); native events are consumed with api.on(type, cb) / api.off(type, cb).

Function calling (`ai.function_call`)

An agentic loop over a fixed, native-side Tier-1 CRM tool catalog (backed by the on-device SmartStore). The catalog is not supplied from JS.

const res = await api.API.request('ai.function_call', {
  prompt: 'Which of my accounts have opportunities closing this month?',
  // max_tokens_per_turn?: number, stream?: boolean
})
// res: { text, tool_calls: [{ function, arguments, status, result, elapsed_seconds }], iterations, ... }

Token streaming

Set stream: true on ai.complete / ai.function_call to receive incremental tokens. The originating request still resolves with the full text — streaming is additive.

const onToken = ({ delta }) => appendToUi(delta) // { request_id, delta, index }
api.on('ai.token', onToken)
api.on('ai.token_stream_completed', () => api.off('ai.token', onToken)) // { request_id } — fires once before the request settles

await api.API.request('ai.complete', { prompt, stream: true })

For ai.function_call, only the model's final natural-language turn streams; intermediate tool-call JSON turns don't.

Cancellation (`ai.cancel`)

Cancel an in-flight request by its id; it settles with the CANCELLED error code and any partial output is preserved. You obtain the id from the request_id on ai.token events, so cancellation is practical for streamed requests.

let activeRequestId
api.on('ai.token', ({ request_id }) => { activeRequestId = request_id })
// ...
if (activeRequestId) api.API.request('ai.cancel', { request_id: activeRequestId })

Model-download events

Fired while the on-device model is fetched on demand:

ai.model_download_progress — { bytes_downloaded, bytes_total, percent }
ai.model_download_completed — { bytes_total } (safe to retry your request)
ai.model_download_failed — { reason }

api.on('ai.model_download_progress', ({ percent }) => setProgress(percent))
api.on('ai.model_download_completed', () => retryMyRequest())

Meeting-notes summary

When enable_meeting_notes_ai is on, Impact captures call audio, transcribes it on-device (whisper.cpp), and summarizes it with Gemma — fully offline; audio and transcript never leave the device. Your app just receives the finished summary:

api.on('meeting_notes_summary_pending', () => showSpinner())
api.on('meeting_notes_summary_ready', ({ summary }) => {
  // e.g. append into a post-call Notes field (append below; never replace)
  notesField.value += summary
})

This is how the post-call app populates its Notes field.

Error codes

Raw ai.* requests reject with { code, message }. Branch on code:

Code	Meaning
`SESSION_ACTIVE`	Another inference is already running.
`PROMPT_TOO_LARGE`	Prompt over 32,000 bytes.
`TIMEOUT`	Exceeded the 30s budget.
`CANCELLED`	Result of `ai.cancel` (usually expected).
`INSUFFICIENT_MEMORY`	Device under memory pressure.
`MODEL_NOT_AVAILABLE` / `MODEL_NOT_DOWNLOADED` / `MODEL_DOWNLOADING`	Model isn't ready — a download may have been prompted; retry after `ai.model_download_completed`.
`MODEL_DOWNLOAD_DECLINED` / `MODEL_DOWNLOAD_INSUFFICIENT_STORAGE` / `MODEL_DOWNLOAD_OFFLINE` / `MODEL_DOWNLOAD_FAILED`	Download couldn't proceed — surface the matching "tap to retry" / "free up space" / "connect to a network" affordance.
`AI_DISABLED` / `MODEL_NOT_LOADED` / `MODEL_LOAD_FAILED` / `DECODE_FAILED` / `INVALID_REQUEST`	Runtime-level failures — treat as "AI unavailable".

On-Device AI (Gemma) ​

Which method should I use? ​

piaSearchAnswer() — the high-level call ​

aiGetCapabilities() — is AI available? ​

aiComplete() — raw on-device completion (iOS only) ​

Configuration ​

Advanced: raw bridge access (iOS) ​

Function calling (ai.function_call) ​

Token streaming ​

Cancellation (ai.cancel) ​

Model-download events ​

Meeting-notes summary ​

Error codes ​

See also ​