How good is it, honestly
Every number below is read from the evaluation results in the repository when this page is built: nothing is typed by hand. All the test data is synthetic, written or voiced for the purpose and labelled as such; no real person's notes are in it. Where a model is involved we run it three times and show the range, because a result that changes from run to run is a result too.
How to read it: each row shows the mean, then the lowest and highest run in brackets. The floor is what our tests require of the worst run; a ✗ would mean it failed.
Speech to text
What was measured
Audio was transcribed by ElevenLabs Scribe v2 and compared with the correct text, after folding half-spaces, ی/ي, ک/ك and digits. Names, English words and numbers were scored on their own, once plain and once with a names hint (a list of names passed to the service).
On what data synthetic data
34 Persian work notes with English words in them, read aloud by a computer voice (TTS): clean, no background noise.
| What | Result (3 runs: mean, range) | Floor (worst run) |
|---|---|---|
| Character error rate (no names hint) | 3.8% (3.7%–4.1%) | ✓ ≤ 0.08 |
| Character error rate (with names hint) | 4.2% (3.8%–4.4%) | — |
| Word error rate (no hint) | 8.5% (8.3%–8.7%) | — |
| Names kept right (no hint) | 76% (76%–76%) | — |
| Names kept right (with hint) | 84% (82%–88%) | ✓ ≥ 0.75 |
| English words kept (no hint) | 65% (64%–65%) | — |
| English words kept (with hint) | 78% (73%–80%) | — |
| Numbers and dates kept (no hint) | 100% (100%–100%) | — |
| Numbers and dates kept (with hint) | 93% (89%–100%) | — |
Known weaknesses
- Clean computer voices, not a real phone in a warehouse or a car: real speech will do worse.
- English words are the weakest part: 65% kept without the hint, 78% with it.
- The names hint lifts names from 76% to 84%, but once, in one run, it returned an empty transcript for clear speech; such a note is now heard again without the hint.
Reading a note
What was measured
Claude (Opus) read each note and pulled out what the speaker promised, who they're waiting on, who they talked with and what they talked about. Compared with labels a person wrote by hand.
On what data synthetic data
34 invented Persian notes, labelled by hand: most from an invented trading company in Dubai.
| What | Result (3 runs: mean, range) | Floor (worst run) |
|---|---|---|
| Speaker's promises found | 100% (100%–100%) | ✓ ≥ 0.9 |
| “Waiting on” items found | 97% (92%–100%) | ✓ ≥ 0.85, spread ≤ 0.1 |
| People the speaker talked with | 100% (100%–100%) | ✓ ≥ 0.9 |
| Key things (clients, goods, jobs) | 97% (96%–97%) | ✓ ≥ 0.9 |
| Someone else's promise read as the speaker's | 0 (0–0) | ✓ ≤ 0 |
| Promises invented | 0 (0–0) | ✓ ≤ 1 |
Known weaknesses
- Cheaper models miss “waiting on” (found: Sonnet 75%–83%, Haiku 58%–67%), so Opus stays on this step.
- Dates and topics aren't in the labels yet, so they aren't measured.
- The set is small: one mistake moves a figure by several points.
Patterns: three people, separately
What was measured
We planted issues in the notes (the system was never told) and checked which became a pattern: three or more people raising it separately. We also planted cases that must not be found: people who had talked to each other, only two people, something that isn't a problem, someone who left.
On what data synthetic data
6 invented companies, 38 planted cases. Deterministic, no AI model, so one run.
| What | Result (1 runs: mean, range) | Floor (worst run) |
|---|---|---|
| Planted issues found (after a person's review) | 85% | ✓ ≥ 0.7 |
| The same, without review | 70% | — |
| Cases that must not be found, and weren't | 100% | ✓ ≥ 1 |
| False alarms | 0% | ✓ ≤ 0 |
| People's names in patterns | 0 | ✓ ≤ 0 |
Known weaknesses
- The same problem said in entirely different words isn't joined: 0 of 3.
- The system knows only the links people mention: three people who heard it from one source and didn't say so count separately. Three is a floor, not proof.
- The model mode (Claude proposing that two phrasings are one problem) hasn't had its three runs yet, so its spread is unknown.
Privacy: what the owner never sees
What was measured
Everything the owner can reach, and every prompt given to a model, was searched for note text, unshared pages and people's names. A deliberately broken version was run too, to prove the check sees a leak.
On what data synthetic data
The same 6 invented companies, with adversarial questions. Deterministic, one run.
| What | Result (1 runs: mean, range) | Floor (worst run) |
|---|---|---|
| Leaks: a private note, an unshared page or a name | 0 | ✓ ≤ 0 |
| Leaks the control (deliberately broken) caught | 13 | ✓ ≥ 1 |
| Owner requests checked | 273 | — |
| Prompts to the model checked | 96 | — |
| Forbidden strings searched for | 253 | — |
Known weaknesses
- Claude doesn't answer in this check: the prompts were checked, not the real model's replies to adversarial questions.
- What someone shared with the whole company isn't a leak and can be quoted (labelled as one person's account).
Asking the company
What was measured
The owner asked, and Claude answered from the pages the owner may see. A separate Claude call set each sentence beside the page it cited and judged it.
On what data synthetic data
10 questions about an invented trading company in Dubai; some deliberately have no answer in the pages.
| What | Result (3 runs: mean, range) | Floor (worst run) |
|---|---|---|
| Cited sentences backed by their page | 93% (89%–97%) | ✓ ≥ 0.85 |
| Sentences with no source and no “conclusion” label | 0 (0–0) | ✓ ≤ 0 |
| “Conclusion” label used correctly | 86% (82%–92%) | ✓ ≥ 0.8 |
| Required facts in the answer | 100% (100%–100%) | ✓ ≥ 0.8 |
| Unanswerable questions answered “I don't know” | 100% (100%–100%) | ✓ ≥ 0.9 |
| Runs that kept right who promised | 100% (100%–100%) | ✓ ≥ 1 |
Known weaknesses
- The judge is Claude too; a person hasn't checked every sentence.
- Few questions, all about one company.
- The demo carries a real, published mistake: once, a promise the speaker made was put on the supplier.
Not measured yet
- Whether the pages written from a note stay faithful to it (no added facts, who promised kept).
- How long a note takes from recording to being read.
- Whether a goal marked “no trace” really has no matching talk.
- The privacy check with Claude itself answering adversarial questions.
- Real phone recordings from real workplaces: the speech numbers here are on clean computer voices.
Source files: evals/results/. Thresholds: evals/thresholds.json. Rerun with python evals/run.py.
Start with one team.
A free two-week pilot with three or more of your own people. Each person decides for themselves, and anyone can stop at any time.