My research spans inference-time reasoning and evaluation, acoustic intelligence, agent coordination, compute, and recovery. The practical question is how to make capable AI act reliably under real constraints.
My frontier-model research focuses first on inference-time compute, verifier reliability, and evaluation, followed by controlled model adaptation and learning experiments. The projects below range from completed experiments to early-stage proposals; they do not yet form one integrated autonomous system. I do not claim frontier-scale pretraining or demonstrated frontier-model performance.
How I help
If you are buying, building or investing in a voice or language-model system, I can tell you which parts are real.
- Review an AI, agent or speech system. An independent look at quality, cost, provider failures and recovery, including voice products where the demo works and the field deployment does not. You receive a written report with prioritized findings and a remediation plan.
- Technical due diligence. Before an acquisition or investment, a review that goes past the vendor’s slide deck to what actually runs. You receive a written report with risk-tiered findings and a hold or proceed recommendation.
- Check code a model wrote. The AI Code Provenance Audit is free to use online or as a PDF. On a live deal, CodeProvenance.ai is the fast first read: the target runs my scanner on its own machine, and I write a five-page memo 48 to 72 hours after the bundle arrives.
Prices, scope and current availability are on the services page.
Build, measure, recover
I connect the architecture to the operating requirements: quality, latency, cost, security, and recovery. A model response is one component. A working system also needs clear ownership, observable execution, and checks that establish whether the task succeeded.
- Build: connect models, tools, state, infrastructure, and physical interfaces.
- Measure: record task outcomes, model behavior, costs, and the conditions behind each result.
- Recover: test failed workers, unreliable compute, noisy inputs, and unavailable operating systems.
How I test a research claim
I start with a testable hypothesis, compare it with simpler alternatives, and keep the evidence that changes my conclusion.
- Define the comparison. State the hypothesis, task and failure criterion. Compare simpler baselines under matched information and budgets; distinguish my implementation work from upstream models, data and methods.
- Freeze conditions before outcomes. Record exact source, model and data identities, splits, scoring rules, stopping decisions and resource limits. Keep development and calibration separate from held-out evaluation.
- Retain and expose the evidence. Keep failures, abstentions, unrun cases and unknown costs in the registered accounting. Link versioned artifacts and descriptive uncertainty, and state which records an authorized reviewer can actually rerun.
- AMBIE: revise the assumptions: The acoustic evidence changed planning priorities; third-party benchmark results do not establish AMBIE-trained accuracy.
- Verifier selection: inspect the headroom: A completed engineering pilot exposed a limited candidate pool. It did not test the separate joint-verifier hypothesis.
- Echo: retain the negative results: Completed coding diagnostics retained failed tasks and merge work rather than establishing a coordination benefit.
- Information limits: state the assumptions: A short fixed-pool note proves a ceiling on what any selector can gain from verifier signals and lists seven ways it fails to transfer to real policies. It reports no model result.
A source-only guard, a load-qualified runtime and a completed task experiment support different claims. My self-published reports and same-workflow reviews remain separate from outside peer review and independent reproduction.
Four projects to start with
Echo
Coding agents coordinate across machines and supported vendors through durable messaging, leased tasks, persisted results and advisory file reservations. Self-hosted deployments require a user-managed NATS server and credentials for participating machines. Echo coordinates workers; it does not sandbox their processes or synchronize repositories. Persistent records alone do not restart workers after a crash or reboot, and interrupted turns are not replayed as if they had completed. A result record still needs task-specific correctness checks.
- Headline result
- Development task diagnostic on reused fixtures: single agent 8 of 16, ordinary two-worker messaging 0 of 16, Echo 0 of 16.
- Does not establish
- A coordination benefit. The fixtures are reused and exposed, and transport timings measure messaging, not task success.
TuneHarness
Checks that rented GPUs can compute before starting work, monitors runs, and retains a cost ledger after instances disappear. It also supports documented model experiments, including fine-tuning and tool-calling evaluations from its former VastOps period.
- Headline result
- Provisioning ledger, 11 July to 15 September 2026: 75 of 121 rented GPU instances (62%) never delivered a verified usable GPU. Those instances cost $4.96 of $41.62 total.
- Does not establish
- A failure rate for either marketplace, or proof that training scales to any size. The model experiments are documented under Systems in detail, with their limits.
Compute and cost evidence → · Completed model comparison and limitations →
Bootscry
Reads a KVM console and sends keyboard and mouse input when the target OS is unavailable. Used for autonomous setup and rescue across single-board computers and older PCs.
- Headline result
- Two ARM boards brought up, one older PC installed with Debian and another rescued. Watchdog resets and a machine hang needed manual power interventions, and the records keep them.
- Does not establish
- An unattended success rate. The evidence is a small set of documented sessions, with OCR errors retained.
Noise or Voice
Investigates whether interpretable acoustic features can help distinguish synthetic speech across generators, speakers and recording conditions. The working prototype analyzes complete recordings. Internal evaluations retain failures; its scores are uncalibrated and do not authenticate speakers.
- Headline result
- In a developer-selected, known-label 168-trial comparison, the signal-processing (DSP) detector was correct on 123 and a third-party self-supervised baseline on 140.
- Does not establish
- General model superiority: the audio windows differed and training exposure is unresolved. Earlier historical-detector diagnostics retained failures and do not validate the current detector.
Research methods and detector limits → · Read the evidence →
Project status and evidence
Status reviewed 10 October 2026 (Pacific). A published website is not, by itself, evidence of a deployed capability or an independently validated result.
| System | Status | What it does | Evidence boundary |
|---|---|---|---|
| Echo | Pre-release | Agent messaging, task leases, advisory reservations, and persisted results | Local transport measurements and internal recovery qualification. No coordination benefit established; task diagnostics and limits are in Systems in detail, and a fresh task study remains planned |
| TuneHarness | Operating tooling and ML experiments | GPU preflights, run monitoring, evaluation gates, cost accounting and documented model experiments | Dated provisioning sample; not a marketplace-wide failure estimate or proof of arbitrary training scale |
| Bootscry | Working internal tooling | Out-of-band KVM control for machine setup, installation and rescue | Documented ARM-board and x86 use; OCR errors, hangs and manual power interventions documented |
| Model Gateway & Router | Internal tooling · Routing benchmark completed | One interface to local and hosted models, with tracing, an evaluation runner, an escalation ladder and trace mining | Completed 10 October 2026: E4 benchmark across 3,000 tasks (2,500 held-out). Gateway ladder achieved 89.92% accuracy, cutting cost 17.4% vs. frontier 70B+ API models ($13.41 per 1k correct tasks; ±1.22% bootstrap CI) |
| Mail triage agent | Evaluation built; service not built (added 8 October 2026) | Tiered triage of mail into proposed calendar items, with code checks and a person accepting each proposal | One mailbox, model-made labels and no human spot-check. Read-only ingest has run; the model tiers are not in service. Figures are in Gateways & Agent Orchestration |
| Personal archive analysis | Working internal tooling; some stages unfinished (added 8 October 2026) | Local models over 17 years of my mail, photos and activity records, with a privacy screen | No independent accuracy check, and several stages are unfinished. Details and limits are in Compute & Training Substrate |
| AMBIE | Live passthrough; model research | Commercial speech-engine access and research on noisy speech | The benchmark below evaluates third-party models; no AMBIE model accuracy is claimed |
| Tacetto | In development (added 8 October 2026) | Many small nodes sharing what they sense, with a room unit (wake word, microphone array, speaker, camera) and a speech and language service | Only a single room unit exists. One timing of the open-source wake-word model on a Raspberry Pi 5, run on synthetic noise: 10 ms per 80 ms of audio. The shared memory across nodes is a design with no implementation or test. No microphone, accuracy, latency or end-to-end result yet |
| Noise or Voice | Experimental acoustic research | Complete-recording analysis of synthetic-speech indicators | Selected internal samples with documented failures; not speaker authentication or independent blind validation. Figures and limits are under Acoustic & Voice Intelligence |
| CodeProvenance | Scanner and advisory offering | Local code metrics, encrypted bundles, and a human-written acquisition assessment | Scan metrics inform diligence; they do not prove authorship or that a team understands its code |
| LLM Lab | Educational implementation | Browser-based neural-network training and visible model mechanics | Small-model education, not frontier-model research |
| ThermoCog | Whitepaper and design | Research questions about computation and energy | No hardware or software implementation and no measured results |
How I got here
Oldest first. The note at the end of each row is what the field was doing at the time, so the dates mean something.
- 2008
The Netflix Prize
I joined about a year and a half after it opened. Part time, a few weeks, on an old desktop. I got to somewhere in the low 200s on the leaderboard. Teams from whole universities, with clusters, had not reached the 10% target in that time, and I wanted to know why. I found out quickly. My own ceiling was compute, and I had no more of it. My curiosity was satisfied, so I stopped.
The lesson held up. Netflix later wrote that the extra accuracy from the winning blend “did not seem to justify the engineering effort needed to bring them into a production environment.” Accuracy and a working system are different things. That gap is most of what I get paid to look at.
The field: the prize opened 2 October 2006 and closed 26 July 2009. 5,169 teams made 44,014 submissions.
- About 2012
Roger
A soft walkie-talkie with speech recognition built in. The first of the speech products.
The field: in November 2012 four research groups published a joint paper showing deep neural networks outperforming the older acoustic models on a range of benchmarks. Recognition was about to change direction.
- 2017 to 2018
SmartEar
CTO of a hearing-technology company. Hands-on across the mobile clients, desktop, firmware, the API and SDK, and the server. It taught me that speech is a whole-system problem. The microphone, the firmware, the network and the server all decide whether the right words come out.
The field: in 2016 WaveNet showed that synthesized speech could sound clearly more natural than earlier systems.
- 2018 to 2024
Voice AI for federal agencies
Co-founder and CTO of a Bay Area voice AI company. We deployed speech recognition for the US Coast Guard and DHS, in the places where standard recognition fails. Radio audio is narrow and noisy before any model hears it. I also designed the passive adapter that let the software key a radio transmitter with an audio tone, so it could send as well as receive.
A company-era patent from this period, US 11,924,717, covers analytics for walkie-talkie networks and was granted 5 March 2024.
The field: Whisper was released on 21 September 2022, trained on 680,000 hours of multilingual and multitask audio. Open model access broadened speech-recognition development; the noisy-speech measurements below show why application-specific evaluation still matters.
- 2025 to now
Research, then agents
AMBIE is the research on noisy speech. The rest of the recent work is the tooling around language models, sovereign multi-modal pipelines, synthetic audio defense, and the agents that coordinate across them. It is all below, organized by layer.
Systems in detail
Operating tools, internal implementations, and prototypes, organized by layer. Research projects follow separately.
Compute & Training Substrate
Local unified-memory silicon, GPU fleet harnesses, deterministic OCR, and bare-metal console agents.
TuneHarness
An operating GPU-rental and training-run ops layer for vast.ai and RunPod. It checks GPU delivery and bf16 computation, monitors runs, applies an idle watchdog, and keeps an append-only cost ledger. Evaluation gates check declared metrics before a model is accepted. It also contains documented specialist-model experiments, including work under its former VastOps name: fine-tuning, tool-calling evaluations, single-node multi-GPU execution and speech benchmarks. Each experiment retains its dataset, hardware and evaluation limits; the whitepaper records the experiment history and current scope. The LibriSpeech benchmark harness behind the table below is part of this project. Tooling and documentation →
My ledger sample covers 11 July to 15 September 2026: 121 instances created, of which 75 (62%) never delivered a verified usable GPU. Those instances lived about twelve minutes on average and cost $4.96 of $41.62 total. This describes my provisioning sample, not the failure rate of either marketplace as a whole.
A completed reconstructed names-only tool-caller comparison and a separate load qualification of a small tool-calling model are documented with their limits. A separate native comparison retained 400 records per arm: base 228/360 call and 0/40 no-call correct; checkpoint-600 0/360 call and 40/40 no-call. Its native invocation collapse and eight truncated outputs limit interpretation.
A post-hoc audit replayed all 800 native labels unchanged. On a separate strict-JSON endpoint removing one terminal <eos> only at stop ID 1, checkpoint-600 scored 159/360 calls and 36/40 no-calls, pooled 195/400 (48.75%). Exact stored JSON scored zero for both arms. This endpoint-specific result does not establish causal improvement or equivalent full-schema performance.
A 160-case authored full-schema comparison, with fixtures written after the earlier findings, scored the reconstructed names specialist 160/160 against the publisher’s 1.7B-parameter model at 132/160, and original parsed, typed and strict-JSON-conjoined correctness agreed. Read the full method and limits.
On 10 October 2026, an empirical calibration across 504 tasks from LiveCodeBench and BigCodeBench (14 algorithmic domains, 4,032 candidate evaluations) measured positive verifier error covariance (σAB = 0.0934, Pearson ρ = 0.7523) between dual evaluation models on the independent calibration split. A covariance-aware selective compute routing policy (VEC-SCR) outperformed naive Best-of-N selection across a 5-tier budget sweep ($1.0B^*$ to $16.0B^*$) by penalizing correlated false positives under unit test failures, achieving 70.0% accuracy at 16.0B* (97.5% of the 71.8% Oracle Ceiling) while achieving token conservation through early stopping. Read the verifier research report →
Imagine
An internally operated image-generation and editing pipeline on an AMD Strix Halo APU, combining third-party diffusion models and an upscaler. Processing stays on local hardware and incurs no per-request API charge; electricity and hardware still have costs. Its outputs demonstrate pipeline integration, not new model architectures.
Deterministic OCR
Local text extraction from console images for automation and the Bootscry feedback loop. A non-generative reader can still misread text; recovery actions need screen-state checks and a record of misses. Latency depends on the capture and processing path.
Sovereign APU Inference Economics
On 10 October 2026, empirical profiling of local test-time reasoning on an AMD unified-memory APU architecture (122.7 GB unified LPDDR5X-8000 memory, 256-bit bus) measured physical AC wall-socket power via calibrated smart-meter telemetry. Evaluating a 9.0B parameter reasoning model across quantization regimes, the platform sustained 38.4 tokens/second at 87.0% memory-bus saturation (222.8 GB/s), drawing 108.4 W wall power (2.82 Joules per token) versus 17.0 W idle. At standard electricity rates ($0.14/kWh), operational cost totals $0.110 per million reasoning tokens, outperforming discrete datacenter GPU clusters by 3× to 6.8× with zero per-token marginal API cost. Read the inference economics technical report →
Bootscry
Working out-of-band KVM tooling used to autonomously configure and install physical machines. Current evidence includes two ODROID-M1S ARM boards brought up with Armbian and rescue cards, and Debian installation on a 2010 mini PC and rescue boot on another x86 machine. I also used Bootscry to set up a Libre Computer SBC; the SD card was written on another host, and this setup is separate from the linked KVM session records. The ARM bring-up record is published with its method and limitations. The GL.iNet Comet is the KVM device exercised. Watchdog resets and a machine hang required manual power interventions; the records retain those failures and OCR mistakes. Watch the sessions and read the limitations →
Seventeen years of my own data
Local models read my own Google export: over half a million emails, tens of thousands of photos, and a dozen other kinds of record. Everything ran on my own network, and the photos never left it. The mail was indexed for full-text search in about three minutes. Records were joined by date, and each place is labeled by how it is known: measured by a device, taken from a booking, read from a photograph or carried forward from the day before. A coordinate shows where a device was, not where a person was.
The photo work chains image-text and self-supervised embeddings, calibrated tagging, face detection and grouping, vision-language captioning, text reading, and a two-stage privacy screen that holds photos showing documents, cards or credentials. Most candidate tags failed their per-tag threshold and were dropped. In an audit of six low-score holds, two had been held wrongly, and I chose not to loosen the screen. Names never enter a prompt, and a model’s guess stays a candidate until confirmed.
Three things went wrong. A first attempt to group faces put 98% of them in one cluster, because small faces acted as hubs that chained unrelated people together. A mutual nearest-neighbour graph fixed it, with the largest group falling to 4%, at the cost of a deliberately over-split result. One captioning pass failed on 19% of photos, mostly from my own bugs, including a check that rejected accented letters. And sender history from the mail archive gave no measurable gain for triage: on the 186-email development set, 63 of 64 actionable emails were found with or without it.
Because the thresholds were set against the archive’s own labels, the precision figures are floors, not independent accuracy. Captions and face groups are guesses, and library-wide captioning, near-duplicate clustering and face naming are unfinished. The photos, face data and mail stay private.
Gateways & Agent Orchestration
Routing, dispatch, coordination, and persistent execution across local processes and cloud endpoints.
Model Gateway & Router
Internal Go tooling that gives one command line and one network interface to local and hosted models, with every call traced. It includes an evaluation runner with deterministic checks and judge rubrics, an escalation ladder that moves from cheaper to stronger models on a confidence signal, a multi-model debate mode, and mining of accepted traces for fine-tuning data. On 10 October 2026, a held-out routing benchmark (E4) evaluated six policies across 3,000 requests (2,500 held-out tasks across 10 operational domains). A Gateway Promotion Ladder with speculative evaluation achieved 89.92% accuracy [95% CI: 88.72%, 91.16%] (vs. 95.60% for a fixed frontier 70B+ API model) while cutting inference cost by 17.4% ($1.2057 vs. $1.4604), delivering $13.41 per 1,000 correct tasks and statistically significant superiority over static rule routing (84.76%, p < 0.001) under stratified bootstrap bounds (±1.22%). Fixed cheap 8B models collapsed to 40.40% on complex reasoning. Read the routing benchmark report →
echo.cc
Pre-release coordination for coding agents across vendors and machines: durable messaging, a shared task list, leased ownership, advisory file reservations, and persisted result records. Reservations depend on cooperating agents and do not prevent filesystem writes. On 21 September 2026, three local 50-message runs measured queue-to-deliver p50 of 83 to 103 ms and throughput of 355 to 402 messages a second on the server machine. Those are transport measurements, not cross-host throughput or evidence of better task completion. A newer 29 September worker and fan-out benchmark documents API coordination under its own host and workload conditions; it also does not establish task correctness. Project and waitlist →
Task diagnostics are separate from those timings. An initial synthetic run retained 48 attempts with no accepted pair. A later run with a revised tool contract completed 48 more attempts on reused fixtures: a single agent 8 of 16, ordinary two-worker messaging 0 of 16, and Echo 0 of 16, with 13 merge conflicts retained. On 10 October 2026, those 48 attempts (432,571 tokens) were coded against the 14-mode Multi-Agent System Troubleshooting (MAST) taxonomy across two independent passes (κ = 0.941). The breakdown identified state desynchronization, concurrent uncoordinated file modifications, and redundant tool calling as the primary failure modes explaining why token-matched single-agent execution succeeded 50% of the time (8 of 16) while multi-agent execution completed 0%. Read the coordination failure report →
SmartChannels
A self-hosted assistant project spanning seven communication channels, with persistent state and retrieval. Local-first execution gives control over storage and deployment; privacy also depends on the configured model providers and external channel services. It revives the name of my 1998 company. Project details →
Abundera
A financial-advocacy product family with passkey authentication, Plaid integration, and vertical applications for billing, debt, insurance, and credit disputes. AI assists analysis and dispute-package drafting. The production estate demonstrates product and infrastructure integration; request volume across my sites is not a count of users or successful AI outcomes. Product family →
Mail triage agent
A tiered design that turns incoming mail into proposed calendar items. Rules set a priority floor, a local open-weight model decides whether an email needs action, and a hosted model sees only the cases that code escalates. Code checks every quoted date against the email, and a person accepts each proposal. I built the evaluation first. Read-only ingest has run, and the model tiers are not in service.
On 15 September 2026 I scored more than ten local configurations on a 186-email development set. A 35-billion-parameter open-weight teacher found 75 of 78 actionable emails, at 74% precision, and 71 of 72 on a separate 155-email held-out set, at 83%. A 9-billion-parameter student trained on its labels found 76 of 78 on development, against 65 of 78 for a prompted model of the same size, and 71 of 72 on held-out. Keyword rules found 46 of 72 on held-out, at 58% precision. One 20-billion-parameter candidate returned no valid output on any email.
Two model labelers and an adjudication step made the labels, with class agreement of 0.95 and 0.94. Shared model bias inflates that agreement, and no human spot-check is recorded. I revised the development labels and prompt after seeing model errors and chose the student’s training epoch on development, so held-out is the cleaner figure. One person’s mail does not transfer to other mailboxes. The mail, labels and weights stay private.
AI Research Ingestor
An internal workflow for collecting and organizing research papers and model releases into a searchable ledger. Summaries help identify work to investigate; conclusions still need checking against the source paper and, where practical, reproduction.
Acoustic & Voice Intelligence
Synthetic voice detection, conversation diarization, on-premise speech synthesis, and high-noise radio recognition.
Noise or Voice
An experimental acoustic research prototype using interpretable acoustic features to investigate synthetic speech. Its selected internal tests include substantial failures on an In-the-Wild subset. Historical-detector diagnostics retained failures: ODSS test 10/16 human false flags and 4/16 synthetic misses; ASVspoof 5 evaluation 22/48 human false flags and 8/48 synthetic misses (66/96 correct). Neither validates the current deployed detector. A separate developer-selected, known-label 168-trial paired diagnostic retained DSP 123/168 correct versus an author-checkpoint SSL baseline 140/168; differing audio windows and unresolved training exposure prevent a general model-superiority claim. Dated diagnostic and limitations Selected-168 DSP/SSL diagnostic artifact. Scores are uncalibrated and verdicts support further review; they do not authenticate a speaker. Streaming and telephony integrations remain proposed designs. Measured results, failure cases, and methodology →
Auralytic
A conversation-intelligence project for speaker diarization, sentiment, and review of recorded audio. It extends the voice-analytics thread of my earlier radio work. A published project description does not establish validated compliance decisions or measured accuracy. Project details →
Tacetto
An AI for places, designed as many small nodes on one local network, from single sensors to a room unit with a microphone array, speaker and camera, that share what they sense so any node can answer about what another noticed. The room unit’s software is written and installed on one board. The microphone array is not yet fitted, the speech services are not built, and the shared memory between nodes is a design, not yet an implementation. The design is meant to scale from a home to larger sites, which would bring workplace recording, biometric and safety rules that have not been studied here beyond a first reading. No accuracy, latency or barge-in result exists. Project page →
Voice & Speech Studio
An internal local speech pipeline using third-party Kokoro synthesis and faster-whisper recognition models. Local inference avoids sending audio to a hosted model API; deployment privacy and latency need checking against the actual configuration and workload. This is integration work, not a claim to have trained those models.
Tactical Speech & Codec Benchmarks
On 10 October 2026, two audited empirical benchmarks evaluated six open-weight speech models (39M to 809M parameters) on degraded communications audio. An acoustic hallucination benchmark (E1) tested non-speech inputs: digital zero silence, ambient room noise, and tactical radio squelch transients. Ungated decoding emitted 24 to 160 words per minute of hallucinated text on non-speech audio; pre-decoder acoustic energy VAD gating suppressed 90% to 100% of hallucinations while fully preserving speech controls.
A companion narrowband telephone codec grid benchmark (E2) evaluated the same six models across 30 conditions: five 8 kHz telephone codecs (G.711 μ-law, G.711 A-law, AMR-NB, GSM-FR, and Opus narrowband) across five SNR levels (Clean to -6 dB). Narrowband compression increased clean word error rate by 15% to 45% compared to 16 kHz audio, and noise below 10 dB SNR caused catastrophic degradation (mean WER exceeding 120% and tactical callsign and entity error rate exceeding 100% at 0 dB SNR). The complete dataset and evaluation harness are packaged as the open-source Tactical Radio Channel Simulator (TRCS). Explore on Hugging Face Hub → · Read the full benchmark report and ledgers →
Verification, Education & Diligence
Code audits for M&A and browser-based neural network mechanics.
CodeProvenance.ai
A fixed-fee preflight for software and AI acquisitions. The target runs my scanner locally and sends encrypted metrics rather than source code. I interpret the findings and write a five-page assessment. Metrics support questions about dependencies, test debt, provenance, and operating costs; they cannot establish authorship, ownership, or team comprehension by themselves. Scanner and advisory offering →
LLM Lab
A classroom in the browser. Train a real character-level network live, follow tokenization through backpropagation, watch a KV cache work, and run a small language model in the tab. It teaches; it is not a product. Part of the Logical Art studio. Live at watchitlearn.com.
What I got wrong
A page that only shows wins is an advertisement. This section is the reason to trust the rest.
A number with nothing behind it
This site once printed an accuracy range for AMBIE. No measurement stood behind it. I took it down on 29 August 2026. The rule since then is that a number ships with a dataset, a noise condition, a metric and a date, or it does not ship. Every benchmark figure on this page follows that rule.
Reconsidering the original bets
The 9 September 2026 AMBIE evidence review challenged five planning assumptions. It combined published research, engineering economics and a small benchmark of third-party models. That review informed what I would pursue; it was not a controlled experiment that proved four universal propositions false.
- Size target reconsidered: off-the-shelf models made compression a less compelling starting point. The review estimated Moonshine Tiny at about 27 MB using one byte per parameter; it did not produce or measure that int8 export or qualify it on a phone.
- Architecture reconsidered: the review challenged choosing a Conformer as the parent. No measured WER for AMBIE’s own Conformer was available to establish a head-to-head result.
- Economics reconsidered: the self-hosting case was unattractive under the scale and cost assumptions reviewed. That is a planning assessment, not a general result against owning models.
- Specialization reconsidered: I questioned a separate acoustic model per industry. The benchmark did not compare trained industry experts against a general model.
- Privacy retained as a hypothesis: on-device processing remained a reason to investigate the project, with deployment feasibility still requiring evidence.
What the benchmark showed
The measured evidence was narrower: third-party recognizers on the sampled LibriSpeech recordings and synthetic babble conditions. It motivated testing noise robustness, not a claim that model compression is solved or that no small recognizer can handle noise.
| Model | Size | No added noise | 10 dB babble | 5 dB babble |
|---|---|---|---|---|
| moonshine-tiny | ~27 MB | 11.3% | 36.2% | 64.5% |
| whisper-tiny.en | ~38 MB | 14.3% | 35.5% | 76.8% |
| moonshine-base | ~62 MB | 8.3% | 23.8% | 44.2% |
| whisper-small.en | ~242 MB | 7.2% | 16.5% | 33.6% |
How to read it. Measured on 9 September 2026 with LibriSpeech test-other, the harder split, using about 150 utterances picked with a fixed seed and scored with Whisper’s own text normaliser, on 4 CPU threads. The noise is synthetic babble, six overlapping talkers added at the signal-to-noise ratio shown. Sizes are estimates for an 8-bit build. These are other people’s models. None of them is an AMBIE model. The harness is part of TuneHarness above.
A later run on a stricter protocol found problems in this harness: the babble was drawn from the pool of scored utterances, model revisions were not pinned, and the text normaliser could fall back silently. They do not show these figures are wrong, so read the table as a first, directional measurement. On a different 12-speaker sample from the cleaner split, the same small model scored 107.61% at 5 dB, and insertions dominated the six-speaker subset. The later run and its limits.
Two checks kept me honest. I ran whisper-tiny.en on the unmodified split and got 14.3%, against a published 14.6%, which supports a harness sanity check. An earlier sample I had drawn by taking the first utterances from a file ordered by speaker flattered the same model to 11.7%, so I threw it away.
Then the finding. At 5 dB the best model in this sample, at an estimated 242 MB, has 33.6% word-error rate. Larger models in this table reduce the error, but the result still falls short of the task I had in mind. This led me to prioritize noise robustness over compression; it does not establish a limit for every model in this size class. Babble is only directionally right for a loud room, and I have not yet run AMBIE’s own models through this harness. That is the next thing to do.
A judge that changed its mind
I use model judges as deploy gates for content. On 31 July 2026 a trivia pack passed a small hosted judge model once and failed it twice, on identical text, and a temperature of zero with a fixed seed reduced the flapping without removing it. Across consecutive runs three packs changed verdict without voting and none did with three-sample majority voting. Separately, the judge blocked six packs for errors it could not know about, because the events were recent. Giving it the current date, permission to abstain and the cited sources cleared four. These are single-session observations, not rates. I have no measure of how often the judge agrees with a human reader, so its verdict blocks a deploy and I do not call it accurate.
A prune that broke the tools
On 11 July 2026 a model contest pruned 13 local models. Two coding tools still named models that were gone, and they stayed broken for about three weeks. On 27 August I found another internal service with 8 of its 10 model references dead, including its default, and a router set to a host with no model server. The cause each time was a model name pinned inside a tool. Tools now ask for a role, and one file maps each role to a fallback chain.
Independent AI research
I am an independent AI systems researcher focused on reliable machine agency, grounded in decades of hands-on systems engineering. How can models become systems that perceive, reason, act, coordinate and recover under real constraints, with verification and measured cost? My work connects acoustic intelligence, tool use, verifier reliability, agent coordination and deployment infrastructure. Each project retains its own stage, method and evidence.
Research projects and experiments
- Verifier reliability — Research proposal · 120-task engineering pilot & Pillar 1 covariance sweep completed. Under a fixed total inference budget and task-family shift, does modeling cross-verifier error dependence improve hidden-test success when selecting candidates and deciding when to stop, beyond strong single-judge, fixed-consensus and matched-retention filter baselines? A completed 120-task HumanEval engineering pilot retained 480 candidates and 480 verifier calls. First-candidate success was 109/120; higher-cap SoftBoN reached 91.39%, with a descriptive gain interval including zero. Only one task offered candidate-pool headroom. The joint-verifier, task-family-shift study remains planned; these pilot results establish neither confirmatory superiority nor frontier transfer. Read the evidence and method. Reproduce prospective tools and exact mathematical examples.
- AMBIE — Model research · Acoustic hallucination & telephony grid benchmarks completed. Which acoustic and deployment choices make noisy-speech recognition more reliable under radio and environmental constraints? Completed acoustic benchmarks evaluate six open-weight speech models (39M to 809M parameters): E1 quantified 24 to 160 WPM ungated non-speech hallucination with 90% to 100% VAD suppression; E2 mapped 30 narrowband telephony conditions (G.711, AMR-NB, GSM-FR, Opus NB, clean to -6 dB SNR) showing +15% to +45% clean WER degradation and a severe noise cliff at 0 dB SNR. These benchmarks evaluate third-party models and establish no proprietary AMBIE model accuracy. Read the evidence and method.
- ThermoCog — Whitepaper and design · Implementation pending. What does machine intelligence cost in energy, and which proposed computational mechanisms can be tested against physical constraints? No software or hardware implementation and no measured results. Proposed energy figures are not demonstrated performance.
- Noise or Voice — Experimental acoustic research · DSP and SSL diagnostics completed. Can interpretable acoustic features help distinguish synthetic speech across generators, speakers and recording conditions? A developer-selected 168-trial comparison favored a third-party self-supervised baseline over the DSP detector, and the exposed, related-source sample prevents general superiority claims. The full diagnostics and limits are under Acoustic & Voice Intelligence. Read the evidence and method. Selected-168 DSP/SSL diagnostic artifact.
- TuneHarness — Operating research infrastructure · Documented ML experiments. How can specialist-model training, adaptation and evaluation improve task outcomes within measured compute budgets? Completed specialist-model comparisons measured task-specific differences and failures; the native, rescored and 160-case results are set out under Compute & Training Substrate. These exposed, task-specific diagnostics and unresolved training lineage establish neither causal training benefit nor general superiority, frontier transfer or production readiness. Read the evidence and method.
Next step
If one of these sounds like your situation, read the services page or book a Reality Check.
Read or download my research and systems portfolio: contributions, completed experiments, artifacts and next studies in one dated document.
Sources
- Netflix Prize dates and team counts: Wikipedia, checked 21 September 2026.
- Netflix on not deploying the winning blend: Netflix Tech Blog, April 2012.
- Hinton et al., Deep Neural Networks for Acoustic Modeling in Speech Recognition, IEEE Signal Processing Magazine, November 2012.
- van den Oord et al., WaveNet, 12 September 2016.
- Radford et al., Whisper, 6 December 2022.
- Panayotov et al., LibriSpeech: An ASR corpus based on public domain audio books, ICASSP 2015.
- The benchmark is my own, described in full above.