Chrome Prompt API and bundled Gemma: observed run record

Backends and artifacts

Skim Recap’s shipping backend uses Gemma 4 E4B through LiteRT-LM on WebGPU. The recorded model file is 2,969,059,328 bytes. The comparison backend uses Chrome’s LanguageModel Prompt API.

On the test profile, LanguageModel.availability() returned available. A clean Chrome profile was not used, so this run did not measure an initial Prompt API model download.

André Cipriani Bandarra stated after the run that Chrome stable was using Gemini Nano and Canary was using Gemma 4 during a migration. The runtime and exported JSON do not independently expose the Prompt API model identity; this is recorded as an external attribution.

Recorded setup

Prompts, sampling, and execution order

Both backends received system and user messages from the same src/prompts.ts module used by the extension. The recap prompt requests two to four passage-bounded points. The explain prompt requests two to three short paragraphs and permits definitions of terms that the passage names without defining.

Prompt API sessions request temperature: 0 and topK: 1. LiteRT-LM requests SamplerType.GREEDY through sessionConfig.samplerParams, and the export records that request beside whatever the engine resolved to.

In the first run LiteRT-LM received no sessionConfig at all, so its sampler was a WASM default that appeared in neither the library types, the source, nor the export. The second run sets it and produces byte-identical output, which establishes that the default was greedy — previously an inference from identical repeats rather than a value anyone had read.

Backend order alternates within each passage-mode group, and which backend leads flips with the group index and the repeat, so each leads exactly 18 of 36 slots. Counterbalanced rather than randomised: the schedule stays a function of the corpus, so two runs remain comparable and the order can be reconstructed from the export, where each record carries its slot number and whether it led.

The first run did not do this. Every repeat of one backend ran before the other started, so LiteRT-LM held the earlier slot in all 18 groups:

LiteRT repeat 1
LiteRT repeat 2
Prompt API repeat 1
Prompt API repeat 2

One generation per backend was discarded before the measured loop. Each measured generation created a new session. TTFT starts before session creation and excludes model download and availability checks.

Repeat equality and character counts

The Prompt/LiteRT character-count ratio was 1.31× for recap and 1.64× for explain/Feynman. Counts include trailing whitespace. Two repeats per condition do not describe behavior under other sampler settings, browser versions, devices, or additional repeats.

Recorded TTFT pairings

First-third and last-third TTFT medians were 969 ms and 1,878 ms for LiteRT-LM, and 783 ms and 1,495 ms for Prompt API — both roughly 1.9× slower across the session. These are reported as pairs rather than medians for that reason.

The counterbalanced schedule also measures what the position was worth. Across all 72 generations the leading slot in a group had a median TTFT of 1,328 ms against 1,452 ms for the following slot: +125 ms for going second. In the first run that advantage went to LiteRT-LM in all 36 of its generations.

Under the fixed order the Prompt API was first to token in 28 of 36 pairs; with the order split evenly it is first in 31 of 36. That is not evidence the second number is the true one. It is evidence that a single sequential session on one machine moved the answer when the only thing that changed was which generation happened to run first — which means the first number was never trustworthy either.

The per-conversation warm-up seen in the first run — LiteRT repeat 2 a median 114 ms faster than repeat 1 — does not survive the change: the medians are now −18 ms for LiteRT-LM and +8 ms for the Prompt API, neither distinguishable from noise. That was a real effect of the fixed order, read at the time as a property of the runtime. Withdrawn, and left here rather than deleted, because publishing a measurement that later turns out to have been measuring the harness is the failure mode this section exists to show.

Session-creation results

As individual examples from these probes, ja session creation succeeded for both declarations, while ko raised NotSupportedError for both declarations.

These probes ran in a top-level extension page. Prompt API execution in an offscreen document and a content-script isolated world was not tested. An earlier setup that declared all eight Skim Recap languages raised NotSupportedError before generation; those attempts are not included in the 72-run final export.

Context values and their sources

The Prompt API session reported contextWindow: 9216 in the export. A separate loaded LiteRT-LM engine.settings reading recorded mainExecutorSettings.maxNumTokens: 4096. The LiteRT-LM value is not present in the final JSON because the engine was not loaded when backend description ran.

Review procedure and recorded excerpts

The author reviewed the outputs with backend identity visible. The review was not exhaustive: not every generation was read closely. There was no predefined rubric, second reviewer, blinding procedure, or agreement statistic. Four output claims received an external factual check. The repository verifier checks only that the quoted fragments occur in the attributed backend output.

An earlier draft attributed a sentence containing WeInfer to the Prompt API. No exported output contains that term, so that sentence is not used here. The missing input passage text prevents an independent passage-to-output groundedness review from the repository alone.

Capture-stage records

• Several captured bodies were between 3,957 and 3,997 characters against a 4,000-character budget.
• Observed substituted headings included Latest posts and Subscribe to newsletter.
• A bibliography passed the article-container, minimum-length, and link-density checks.
• Approximately 17 captures were reviewed before nine fixtures were selected.

Rejected passage text is not stored in the final export. The selection history is recorded in the findings log.

What is and is not reproducible

The JSON preserves generated output text, timings, chunk counts, errors, environment fields, capture timestamps, extension versions, and — in the second export — the nine input passages themselves.

The first export did not carry them. The passages had been captured into browser storage and never promoted into src/fixtures.ts, so the results file recorded which passages had been used but not what they said. Nobody could check whether a definition in an output came from the model or from the source text.

They are now committed as literal constants, each with its source URL, capture timestamp, and a licence note. Each also carries three lists written against the passage rather than assumed from its subject area: terms the passage names and never explains, terms it names and explains, and the facts a correct explain answer has to add. The middle list is what separates supplying outside knowledge from paraphrasing the source, which one list could not do.

Writing them against the text corrected the previous set. Terms named in the old notes appear in none of the passages those notes described; one term is expanded in one passage and only abbreviated in another, so it is undefined in one fixture and defined in another; and four passages define their own key terms and are now marked bounded rather than undefined-term.

A fresh clone can now build the harness, load it unpacked, and rerun the same nine passages. That costs a 2.97 GB model download, a WebGPU machine, and a Chrome build where the Prompt API reports as available.

eval/verify-claims.py recomputes a declared set of metrics whose expected values are encoded in the script. It does not parse this article and does not validate every number or prose statement. A zero mismatch count means that those encoded expectations match the newest JSON export.

Download the JSON export · View the harness, report, and findings log

What changed, and what did not

Maud Nalpas of the Chrome team reviewed the first run and raised the missing corpus, the unconfigured sampler and the fixed order. All three were corrected and the same nine passages were run again on 12 August: 72 generations, 0 failures.

Nothing the models produced moved

All 72 outputs are byte-identical across the two runs, so every content measurement above is unchanged — and the sampler question is settled. Requesting greedy explicitly produced exactly what the library default produced, which means the default was greedy. That had been an inference from repeat equality; it is now a comparison of two runs.

The timing did not replicate

With the order counterbalanced the Prompt API is first to token in 31 of 36 pairs rather than 28, and the leading slot is measurably cheaper: a median 125 ms — a slot the first run gave to LiteRT-LM in all 36 of its generations. I do not read this as the second run correcting the first into a trustworthy number. I read it as two runs of the same setup disagreeing, which is what a weak instrument does. The right conclusion is not “31/36 is right”; it is that a single sequential session on one machine cannot be trusted to report a latency finding at all, and both runs above should be read that way.

One finding did not survive the second run. LiteRT-LM’s second generation in a group had been a median 114 ms faster than its first, read at the time as a per-conversation warm-up cost. Under the corrected schedule that gap is −18 ms, and +8 ms on the Prompt API side — gone. It was the fixed order, and I had reported an artefact of the schedule as a property of the runtime. Withdrawn, and kept here rather than deleted, because this is the failure mode the rest of this section is arguing against.

What is still open

The corpus was not enlarged and summarization quality still has no metric. Both runs measure the same nine passages, so a second run of an unchanged corpus is not further evidence about the models — it is evidence about how little a single sequential session on one warm laptop can be trusted to say about latency. What that setup would need to become trustworthy — repeated sessions, a cooldown between them, or simply not reporting absolute latency until it can — is an open question, not a solved one.

Conditions not covered by this run

• The corpus contains nine selected passages rather than a representative sample.
• Two repeats within a condition are not independent passages.
• Nine passages come from five independent sources: three are sections of one review, two are sections of one paper, and two are overlapping slices of one post.
• Content review was unblinded and four claims received an external factual check.
• Timing was recorded on one machine in one sequential session, with drift of roughly 1.9× across it.
• The Prompt API model identity is externally attributed rather than exposed in the artifacts.
• The production offscreen extension context was not tested.

Items not answered by the artifacts

1. Does LanguageModel run in an extension offscreen document or content-script isolated world?
2. Can unsupported input languages be handled per request rather than at session creation?
3. What additional input and output languages are planned?
4. Will the Prompt API expose an output-token limit?
5. What model and runtime identifiers can an extension record?
6. What availability state, download behavior, and model size are observed in a clean Chrome profile?
7. How should Stable and Canary be compared when browser version, runtime, and model may change together?