The Harlow Hospitality ASR Benchmark

Summary#

A hotel request is a few seconds of speech spoken in a corridor, a kitchen or a lobby, often by a speaker whose first language is not English. What matters is whether a handful of facts in it survive. The benchmark scores exactly that, on 2,920 synthetic request clips from one fictional property under ten acoustic conditions and four line types, and on 1,386 clips from 77 real speakers with eleven first languages.

• With the vocabulary and at matched repeat rates, Harlow proceeds silently wrong on 2.6% of hotel requests against 5.1% for the best gated single recognizer (paired difference 2.5 points, 95% interval 1.6 to 3.5), and on 3.4% against 4.5% of real accented clips (1.1 points, 95% interval 0.1 to 2.1).
• On hotel requests it is ahead or level in every acoustic condition and line type, most clearly in heavy noise, with a guest talking nearby and on mixed-language requests.
• On hotel requests the property vocabulary is the largest single lever for every recognizer, cutting silent-wrong rates 1.6- to 5.1-fold. On real speakers it helps by 1.3- to 1.5-fold, but the choice of recognizer matters at least as much.
• How a recognizer writes room numbers is a second lever of the same kind. When rooms written as clock times are counted as rooms for every recognizer, the headline comparison is 2.6% against 4.1% (1.5 points, 95% interval 0.8 to 2.2).
• Word error rate and silent-wrong rate rank the systems differently.
• In a separate stress test that cuts the end of speech, every system fails most often, and Harlow stays ahead (18.4% against 23.8%).

Why word error rate is not enough#

For a growing class of applications the transcript is not the product; it is an intermediate representation from which a small number of facts are extracted and acted on. A hotel is a clear instance. A staff member wearing a microphone says "Room 1214 needs two extra towels for Mrs. Okonkwo", and the system files a task against a room, for a person, with a count. The value of that utterance depends on whether four or five tokens survive intact.

Word error rate counts every word equally, so it measures neither what matters nor what it costs when it goes wrong. A transcript with a 5% word error rate can file every request against the wrong room; a transcript with a 20% word error rate can file every request correctly. The task here is deliberately not "transcribe the audio"; it is "either get every fact right or say so".

Each clip therefore has an expected text and a fact record: a set of room numbers, a set of must-appear phrases (names, items with counts, hospitality terms) and a negation flag. A transcript is correct if every fact is recoverable from it; otherwise it is silent-wrong. A system that asks the speaker to repeat is not wrong; it is slower.

The scorer is lenient about form and strict about facts. Number words and digits are equal; a room matches however its digits are grouped ("twelve fourteen", "12 14", "1214"); hyphenation, spacing and plurals are ignored. A room written as a clock time or a date ("12:14", "12/14") does not match the room, because whatever consumes the transcript would read a time. Fact recall and word error rate on normalised tokens are reported as secondary metrics, over the clips a system went ahead with.

The benchmark#

The benchmark has two tracks: hotel requests spoken by synthetic voices, and real accented speakers reading a standard paragraph. There are 5,066 clips in all: Track A 3,680 including the stress test, Track B 1,386.

Track A: hotel requests, synthetic speech

Track A speech is synthetic: AI-generated voices from Azure neural text-to-speech and OpenAI gpt-4o-mini-tts, not recordings of people. Each line is spoken by a native US voice and by an accented voice, and the voices alternate between the two synthesis vendors, so results can be split by synthesis vendor.

Property

One fictional property with 64 rooms on eight floors, ten staff members, fifteen in-house guest surnames and twenty hospitality terms (rollaway, PTAC, turndown, bell cart, late checkout, do not disturb and others)

Request lines

120 request lines drawn from seeded templates, plus self-corrections, short commands and mixed-language lines

Voices

24 synthetic voice configurations (8 native US, 16 accented), 21 after the intelligibility screen, plus 4 synthetic guest voices for the bystander conditions

Accents

Spanish, Tagalog, Russian, Mandarin, Vietnamese, Indian, Nigerian, Haitian, Portuguese

Clips

3,200 rendered and 2,920 scored in the main set; 436 more scored in the stress test

Duration

Median 4.65 s (5th to 95th percentile 2.88 to 6.75 s), 288.9 minutes in all

A screen removes synthetic voices that are not accents but failures of synthesis: a voice is dropped if the median standalone recognizer, without any vocabulary, gets at most 20% of its clean clips correct. The screen uses only the standalone recognizers, so it cannot favour Harlow. It removed three Azure voices (pt-BR-FranciscaNeural, ru-RU-DmitryNeural, vi-VN-NamMinhNeural), leaving 2,920 scored clips of the 3,200 rendered.

Noise is mixed at a fixed signal-to-noise ratio against the speech level, from real recordings in the DEMAND corpus; the stairwell is a synthetic impulse response. Measured on the clips built on the same synthesis as their clean clip, signal-to-noise ratios match their targets to within 0.56 dB and bystander levels to within 0.24 dB. Half the guest lines name a different room from the wearer's.

Track B: real accented speakers

Track B is real people reading the Speech Accent Archive's "Please call Stella" paragraph: read speech, not requests. The archive is maintained by Steven H. Weinberger and Matthew C. Kelley at George Mason University.

Speakers

77 of 89 downloaded; the 12 for whom Scribe v2's segmentation pass did not recover every boundary word are excluded (see Limitations)

First languages

English 10, Arabic 8, French 8, Hindi 8, Mandarin 7, Portuguese 7, Tagalog 7, Amharic 6, Russian 6, Spanish 5, Vietnamese 5

Segments

Six clips per speaker, cut at sentence and comma boundaries, 1.8 to 14.3 s long (median 4.7 s); each has its own fact record (names, items with counts, a day, a place)

Conditions

Clean (as recorded, no added noise), a lobby crowd 5 dB quieter than the speaker, and restaurant noise as loud as the speaker (0 dB)

Clips

1,386, of which 462 clean

Duration

Median 4.72 s (5th to 95th percentile 2.51 to 8.53 s), 114.2 minutes in all

Systems#

Five commercial recognizers from three vendors are evaluated, each run plain and with the property's vocabulary, configured as a careful integrator would: language pinned to English, everything else at the vendor default, and the vocabulary passed through the vendor's own channel.^[1]

Harlow, the authors' production system, returns either a transcript or a request to repeat and is evaluated as a black box. It receives the same audio file as every other system and, in the vocabulary arm, the same vocabulary. It exposes no confidence to gate on; when to ask is its own decision, so it appears in every comparison at a single operating point.

Scribe is a product of ElevenLabs; MAI-Transcribe of Microsoft; gpt-4o-transcribe and whisper-1 of OpenAI. The report measures those services as served on the stated dates and makes no claim about them beyond those measurements. No vendor has reviewed or endorsed this benchmark.

How systems are compared#

Same vocabulary

Every system runs twice: plain, and with the property vocabulary (122 terms on Track A, 99 on Track B), passed through each vendor's own biasing field and cut to what that field accepts. The plain run also represents a property on its first day.

Same repeat budget

Gated recognizers send back as many clips as Harlow asks to repeat on the same track, with thresholds chosen on held-out folds.

Same stimuli

Every system hears the identical file.

Each system's own output

Every system, Harlow included, is scored on what it returned for the clip, and the benchmark adds no processing to any of them.

A standalone recognizer that reports a confidence may send back its least-confident clips, as many as Harlow asks to repeat on the same track. The threshold is chosen on one half of the sentences (Track A) or speakers (Track B) and applied to the other half, so a threshold never sees the clip it is scored on. Where a choice exists, every gate is tuned in the standalone systems' favour.

The headline does not depend on how the folds are split: for Scribe v2 with the vocabulary the held-out gate gives 5.10% silent-wrong, the in-sample gate 5.17% and a held-out gate whose folds split each synthesis vendor's lines 5.14%.

MAI-Transcribe 1.5 and MAI-Transcribe 2 report a confidence of zero under their enhanced mode, so they cannot be gated; the benchmark reports this rather than constructing a gate for them. What they file, they file silently.

Intervals in the tables are 95% percentile bootstrap intervals over clips with 1,000 resamples. For the headline comparisons the paired difference between two systems on the same clips is also reported, with a percentile interval from 2,000 resamples of whole sentences (Track A) or speakers (Track B).^[2]

Results#

Headline comparison at matched repeat rates

With the vocabulary, Harlow proceeds silently wrong on 2.6% of hotel requests at a 5.3% repeat rate. The best gated single recognizer, Scribe v2, is at 5.1% when it sends back 5.5%: a paired difference of 2.5 points (95% interval 1.6 to 3.5). Among the requests each system went ahead with, the error is 2.7% against 5.4%. Without a gate Scribe v2 is at 6.9%; the recognizers that cannot be gated range from 17.9% to 28.0%.

Show the dataHide the data

Budget	Scribe v2 (A)	gpt-4o-transcribe (A)	whisper-1 (A)	Scribe v2 (B)	gpt-4o-transcribe (B)	whisper-1 (B)
0%	6.8% (0.0%)	10.7% (0.1%)	15.7% (0.0%)	19.8% (0.1%)	31.1% (0.1%)	27.9% (0.0%)
5%	5.2% (5.1%)	8.3% (5.0%)	14.2% (5.6%)	15.3% (4.8%)	26.5% (5.0%)	22.7% (5.3%)
10%	4.2% (11.1%)	6.6% (12.2%)	12.7% (10.7%)	11.0% (10.2%)	21.5% (10.0%)	18.2% (10.3%)
20%	2.9% (20.3%)	3.5% (28.6%)	10.1% (20.9%)	5.7% (20.1%)	12.8% (20.6%)	12.3% (20.3%)
30%	1.8% (30.1%)	2.4% (40.6%)	7.8% (32.3%)	3.2% (30.2%)	7.6% (29.1%)	8.4% (31.0%)
40%	1.3% (40.1%)	1.7% (41.4%)	6.5% (40.5%)	2.3% (39.5%)	3.0% (40.0%)	5.7% (40.1%)
50%	0.7% (49.9%)	1.1% (51.3%)	5.2% (47.8%)	1.3% (49.9%)	1.4% (49.7%)	3.3% (50.1%)

The ordering of Track A survives on real speech. Harlow is ahead of gated gpt-4o-transcribe and whisper-1 (3.4% against 10.3% and 11.0%) with non-overlapping intervals, and ahead of gated Scribe v2 by 1.1 points, with overlapping marginal intervals and a paired interval that reaches down to 0.1 points.

Fact recall and word error rate for Harlow are over the clips it did not send back, which excludes its hardest clips, so they are not comparable with the other systems' values and no ranking is drawn from them. They are in the report.

The property vocabulary

Show the dataHide the data

System	Plain	With vocabulary
Scribe v2	34.9%	6.9%
gpt-4o-transcribe	36.2%	10.8%
whisper-1	41.9%	15.8%
MAI-Transcribe 2	38.3%	17.9%
MAI-Transcribe 1.5	44.1%	28.0%
Harlow	18.7%	2.6%

Every system runs twice: plain, which also represents a property on its first day, and with the property's vocabulary (122 terms on Track A, 99 on Track B). On hotel requests the vocabulary cuts silent-wrong rates 1.6- to 5.1-fold, and it matters more than the choice of recognizer: every recognizer does better with it than the best recognizer does without it (MAI-Transcribe 1.5 with the vocabulary at 28.0% against Scribe v2 without it at 34.9%).

On real speakers the vocabulary helps every recognizer, by 1.3- to 1.5-fold and up to 10.8 points, but the choice of recognizer matters at least as much: MAI-Transcribe 2 without the vocabulary (24.5%) does better than gpt-4o-transcribe and whisper-1 with it (31.2% and 27.9%). On Track B the vocabulary is the paragraph's own names and items, so the vocabulary arm is told in advance the facts it must recover; this applies to every system equally.

Like every recognizer, Harlow is worse without the vocabulary, 7.2 times as often silently wrong: 18.7% against gated Scribe v2's 24.7% at 15.7% and 16.0% repeats (6.0 points, 95% interval 3.9 to 8.1).

By acoustic condition

Show the dataHide the data

Condition	Harlow	Scribe v2, gated	gpt-4o-transcribe, gated
Clean	0.9% (1.8% repeats)	1.4% (0.5%)	4.1% (1.4%)
Hallway noise, 10 dB	0.9% (0.9% repeats)	1.4% (1.4%)	1.8% (0.5%)
Lobby crowd, 5 dB	2.8% (2.3% repeats)	4.6% (1.8%)	5.5% (0.9%)
Restaurant, 0 dB	3.2% (14.7% repeats)	6.9% (11.0%)	14.2% (10.1%)
Kitchen, 5 dB	1.4% (1.8% repeats)	2.8% (2.3%)	5.1% (0.9%)
Kitchen, 0 dB	2.3% (1.4% repeats)	2.3% (1.4%)	5.5% (0.9%)
Stairwell echo	1.8% (1.8% repeats)	2.8% (2.3%)	4.1% (2.3%)
Guest talking nearby	1.8% (7.3% repeats)	4.1% (2.3%)	8.3% (2.8%)
Guest in the background	1.4% (3.2% repeats)	2.3% (4.1%)	4.6% (0.5%)
Guest nearby plus lobby crowd	0.5% (4.6% repeats)	2.8% (3.2%)	6.0% (1.8%)

In quiet and in mild noise Harlow and gated Scribe v2 are within about half a point of each other (0.9% and 1.4% on clean audio, 0.9% and 1.4% in hallway noise 10 dB quieter than the speaker). The separation is largest where single recognizers fail silently: in restaurant noise as loud as the speaker (3.2% against 6.9% for gated Scribe v2, at 14.7% and 11.0% repeats), with a guest talking nearby (1.8% against 4.1%, at 7.3% and 2.3% repeats) and with the guest and the lobby crowd together (0.5% against 2.8%, at 4.6% and 3.2% repeats).

With a guest nearby Harlow asks for more repeats; among the requests it goes ahead with, it still errs less (2.0% against 4.2%). At a matched overall repeat rate, gated Scribe v2 is silently wrong at least as often as Harlow in every condition; the two are level in kitchen noise as loud as the speaker (2.3% each). Gated gpt-4o-transcribe is at 4.1% on clean audio, and ungated MAI-Transcribe 2, which has no confidence to gate on, is at 9.6% before any noise is added and 23.4% with a guest nearby.

Harlow's repeats are concentrated where the audio is hardest: on hotel requests it asks on 1.8% of clean clips, 7.3% with a guest nearby and 14.7% in restaurant noise as loud as the speaker. The scorer asks only that the wearer's room be present, so the guest cells understate what a guest's voice costs a system that acts on the first room it finds.

By line type

Show the dataHide the data

Line type	Clips	Harlow	Scribe v2, gated	gpt-4o-transcribe, gated
Request	872	1.9% (3.2%)	3.1% (1.5%)	5.9% (1.5%)
Self-correction	292	2.1% (6.5%)	2.7% (4.1%)	9.9% (3.4%)
Short command	288	7.6% (9.4%)	18.8% (19.1%)	12.8% (27.1%)
Mixed language	160	6.9% (14.4%)	11.9% (17.5%)	27.5% (12.5%)

On short commands Harlow is at 7.6% against 18.8% for gated Scribe v2 and 12.8% for gated gpt-4o-transcribe, while sending back fewer of them (9.4% against 19.1% and 27.1%). A two-word utterance gives a confidence score little to work with, so a gate tuned to the overall repeat budget spends a disproportionate share of it there.

Mixed-language requests show the widest gap (6.9% against 11.9% and 27.5%), consistent with the known difficulty of code-switched speech for monolingual decoding. Self-corrections are handled well by Scribe v2 and Harlow and poorly by gpt-4o-transcribe.

Real accented speakers

In restaurant noise as loud as the speaker Harlow and both gated recognizers send back more than half of the real-speaker clips; Harlow asks on 1.7% of clean segments and 57.6% under restaurant noise.

Show the dataHide the data

First language	Speakers	Clips	Harlow	Scribe v2, gated
Amharic	6	108	4.6% (28.7%)	7.4% (30.6%)
Arabic	8	144	8.3% (36.8%)	6.9% (35.4%)
English	10	180	2.2% (15.0%)	2.2% (13.3%)
French	8	144	3.5% (22.9%)	2.1% (29.9%)
Hindi	8	144	0.7% (22.9%)	2.1% (22.2%)
Mandarin	7	126	1.6% (19.1%)	5.6% (17.5%)
Portuguese	7	126	4.0% (24.6%)	4.8% (24.6%)
Russian	6	108	3.7% (19.4%)	7.4% (18.5%)
Spanish	5	90	2.2% (33.3%)	4.4% (30.0%)
Tagalog	7	126	1.6% (15.9%)	3.2% (19.1%)
Vietnamese	5	90	5.6% (30.0%)	5.6% (30.0%)

Harlow is ahead overall but not uniformly. Its point estimate is lower than gated Scribe v2's for Mandarin, Russian, Amharic, Spanish, Tagalog and Hindi first languages (by 4.0, 3.7, 2.8, 2.2, 1.6 and 1.4 points), within a point for English, Portuguese and Vietnamese, and higher for French and Arabic (by 1.4 points each).

Word error rate and silent-wrong rate

On hotel requests MAI-Transcribe 2 has the lowest word error rate of any single recognizer (0.071) and the lowest median latency (331 ms), yet files 17.9% of requests silently wrong against Scribe v2's 6.9% at a word error rate of 0.152, 2.1 times as high. gpt-4o-transcribe and whisper-1 likewise have a lower word error rate than Scribe v2 and file more requests silently wrong. Scribe v2's word errors evidently fall outside the facts.

Rooms written as clock times

Some recognizers write a room number as a clock time ("12:05" for room 1205), which the scorer reads as a time, because whatever consumes the transcript would read a time. Each system is scored on what it returns, so this is counted against it. As a robustness check, such readings are also counted as the room for every recognizer: gated Scribe v2 then moves from 5.1% to 4.1%, and the Track A comparison is 2.6% against 4.1% (1.5 points, 95% interval 0.8 to 2.2). The difference stays above zero, so the result does not rest on the recognizers' number formatting.

The two Microsoft recognizers often write a room as a clock time: reading such rooms as rooms takes MAI-Transcribe 1.5 from 28.0% to 8.1% silent-wrong with the vocabulary and MAI-Transcribe 2 from 17.9% to 7.7%; Scribe v2 gains 1.4 points, gpt-4o-transcribe and whisper-1 nothing. A comparison on each system's own output measures its formatting along with its recognition, which is why both views are reported. Without the vocabulary the comparison is 18.7% against 24.7% on each system's own output, and 18.7% against 21.3% with room readings counted for every recognizer. Track B is unaffected.

Trading repeats for errors

On hotel requests (Figure 1) gated Scribe v2 needs about 24% repeats to reach the silent-wrong rate Harlow reaches at 5.3%. A gate is worth less than the vocabulary, but it is the only lever that turns a silent mistake into a question.

Latency

Harlow's latency is not reported, because it is not measured on the same basis as the standalone services. The arms were run at different times, so differences of this size are within the services' own variation.

The stress test#

Two further Track A conditions remove the first 200 ms or the last 300 ms of speech, measured from the detected speech edges. They model a capture window that opens late or closes early, such as a push-to-talk radio released before the last word or voice activation that triggers after the first. The badge Harlow ships on starts capturing when the wearer presses it and stops when the wearer presses again after speaking, so these conditions are reported separately from the main set: 218 clips each, 436 in all, synthetic speech.

With the last 300 ms removed every system fails most often, and Harlow remains ahead: 18.4% against 23.8% for gated Scribe v2 and 35.8% for gated gpt-4o-transcribe. No system detects the truncation reliably; each often proceeds with a reading that lacks the final number, and Harlow asks for a repeat on 12.8% of these clips. A capture window that closes after the speaker stops removes the condition at its source.

Truncation at the start costs Harlow and gated Scribe v2 little (1.8% and 3.2%) but takes gated gpt-4o-transcribe to 9.2% and ungated MAI-Transcribe 2 to 17.4%. With the two cut conditions included in the main set (3,356 clips), the comparison is 3.6% against 6.2% at 6.0% repeats each (2.6 points, 95% interval 1.7 to 3.6), and 3.6% against 5.1% with room readings counted for every recognizer (1.6 points, 95% interval 0.9 to 2.2).

What the results mean#

Three things an integrator controls move the silent-wrong rate: the property's vocabulary, which is worth more than the choice of recognizer for every recognizer; normalising how numbers are written, which is worth as much for the recognizers that write rooms as times and nothing for those that do not; and a way to decline, which is worth less than either but is the only one of the three that turns a silent mistake into a question. One of the three vendors evaluated, Microsoft, returns no usable confidence, so its two recognizers cannot be gated.

Because Harlow is evaluated only through what it returns, the benchmark says how often it is silently wrong and how often it asks, and nothing about why. The same holds for the standalone services, whose models are equally undisclosed. The comparison is between products as served, on identical audio, with identical vocabulary, each scored on what it returns, at matched repeat rates; it is not a comparison of methods.

Whether a repeat costs the wearer two seconds or the request depends on the deployment; the benchmark measures how often it is asked, not what it costs.

Limitations#

• Synthetic accents. Track A's accented voices are AI-generated synthesis voices reading English; they are consistent and controllable but are not people. Track B is real speech but read, not requests, and its speakers' misreadings count against every system.
• Track B selection and vocabulary. The 77 Track B speakers are those for whom Scribe v2, a system under test, recovered every boundary word in one pass; the twelve it did not are excluded, which removes the speech that recognizer found hardest. The Track B vocabulary consists of the paragraph's own names and items, so the vocabulary arm is told the facts it must recover. Both apply to every system equally, and both make Track B's absolute rates optimistic. The first favours Scribe v2, and because Harlow is evaluated as a black box, it cannot be ruled out that it favours Harlow as well. On 7 clean segments no system is right, because the speaker departed from the printed text or, in one case, the boundary cuts into the first word.
• Black-box evaluation. Harlow's design is not disclosed, so a reader cannot attribute its results to a method. Its verdict for every clip is part of the data release; its output and the system are not, and it cannot be run again by others.
• Number formatting. Each system is scored on what it returns, so a recognizer that writes rooms as clock times is charged for it, although an integrator could add a step that reads them as rooms; with that reading given to every recognizer the hotel-request comparison is 2.6% against 4.1%. The room list that reading consults is the set of rooms the corpus names, not a property's full list.
• Scorer rules. Two scorer rules are lenient. Negation is satisfied by any negating word, and 10 of the 20 negated lines contain "do not disturb", so a transcript that drops a leading "Don't" can still pass; the cut-start condition may be understated for every system. A transcript that contains the wearer's room passes even if it also contains the bystander's, so the guest conditions understate the cost of transcribing the guest. Both rules apply to every system; neither is corrected.
• Statistics. Table intervals are marginal and treat clips as independent, although clips share lines and speakers; paired, clustered differences are reported for the headline comparisons only. Each standalone recognizer's reported output is a single response per clip.
• No device path. Every clip is clean 16 kHz audio with noise mixed in. No microphone, codec, automatic gain control or room from an actual wearable device is in the chain.
• Other capture devices. The main set leaves out speech cut short because the badge's press-to-start, press-to-stop capture does not produce it. A push-to-talk or voice-activated client could cut speech; for such a deployment the stress-test figures are the relevant ones.
• Fixed signal-to-noise ratio. Noise is mixed at a constant level per clip; real environments vary within an utterance.
• Stairwell rendering. The reverberated clips are coarsely quantised: a stairwell clip holds a median of 663 distinct sample values in steps of about 43, where a clean clip holds 13,240 in steps of 1. The quantisation noise is 47 dB below the signal at the median clip and 32 dB at the worst, the same for every system.
• Non-determinism. Harlow's output is not deterministic; its two runs moved no total by more than 0.4 points, less than the bootstrap intervals. The vendors' recognizers are not all repeatable either: replaying 200 clips changed the words of Scribe v2's transcript on 28 and its score bucket on 5, while whisper-1 returned its stored output on 199.
• Latency scope. Harlow's latency is not reported, because it is not measured on the same basis as the standalone services.
• Take pairing. For the OpenAI voices, the clips of one line are not all built on one synthesis: 1,404 of the 1,620 mixed OpenAI-voice clips use another take of the same sentence, by the same voice, than their clean clip. Per-condition differences for those voices therefore include take-to-take variance, and a new rendering will not reproduce those clips; the clips themselves are the reference. It does not touch the comparison between systems, which hear identical files.
• Moving targets. Every number is tied to the vendors' services as served on 30 September and 1 October 2026. A run on a later date is a new result.
• One property. Track A is one fictional property with one vocabulary. Properties with larger rosters or more homophonous room numbers may behave differently.

Data and access#

The benchmark is released with each system's verdict for every clip, so that the silent-wrong, repeat and correct rates in its tables can be recomputed, and with the audio of the real-speaker clips recorded without added noise. Both are in the dataset on Hugging Face.

Related work and the report's appendices A to G are in the full report: report (PDF).

What the data release contains

Report

The full report as a PDF, 27 pages

Per-clip verdicts

60,792 rows: one per clip, system and vocabulary setting (correct, silent-wrong or repeat, and the verdict after the confidence gate where one applies)

Clip list

5,066 rows: track, condition, line type, voice or speaker, accent or first language, speech source, main-set and screen flags, and for Track A the reference text and expected facts

Systems

Identifiers for the 6 systems used in the tables

Aggregate results

The results behind the report's main tables and figures, the same with the two stress-test conditions included, and paired differences between Harlow and each gated recognizer

Real-speaker audio

The 462 Track B clips recorded without added noise (38.1 minutes, 16 kHz mono WAV) with their metadata, under CC BY-NC-SA 4.0

Checking script

Recomputes every silent-wrong, repeat and correct rate in the report's tables from the per-clip verdicts and checks it against the results files (Python 3.9 or later, no packages needed). It does not recompute the risk-coverage curves or the intervals

Re-running the scorer on the stored per-clip outputs of the 12 arms reproduces its stored per-clip results (60,792 rows, 0 vendor errors). Not in the release: the text any system returned, the Track B clips with background noise mixed in, the synthetic Track A audio (available on request, below) and the Harlow system. Of the audio, only the real-speaker clips recorded without added noise are in the release, so from the release alone the standalone systems can be tested again on those clips and no others.

Request the synthetic speech

The synthetic Track A speech, before any noise is mixed in, is available to researchers on request under a data-use agreement, with the settings used to add the noise. It is AI-generated speech, not recordings of people.

1. Email support@tryharlow.com
   With your name, your organisation and a sentence on the project.
2. We send the data-use agreement
3. You sign it
4. We countersign and send a private download link
   To the synthetic speech before any noise is mixed in, plus the settings used to add the noise.

The agreement's terms, in short:

• Evaluation only.
• No training, fine-tuning or distilling of any model.
• No building a speech product with it.
• No passing it on.
• Say it is AI-generated speech whenever it is played.
• Credit the benchmark.
• Delete on request.
• Two-year term.

Licences

Report, tables and checking script

CC BY 4.0, Harlow Technologies, Inc. (licence)

Real-speaker clips

CC BY-NC-SA 4.0, as the Speech Accent Archive (Steven H. Weinberger and Matthew C. Kelley, George Mason University) licenses them (licence). Converted to 16 kHz mono 16-bit, trimmed to the read paragraph and cut into six segments per speaker; nothing else changed.

Noise

DEMAND (J. Thiemann, N. Ito and E. Vincent, 2013), Zenodo record 1227121 (DOI 10.5281/zenodo.1227121), under its authors' licence. Not redistributed

Names

Product and company names are the property of their owners and are used only to identify the systems evaluated

Conflict of interest#

Harlow is the production system of the authors' company, and the authors are its developers. The benchmark was built by the team that develops Harlow and is not independent of it. Harlow is evaluated as a black box: its design is proprietary and outside the scope of the report. The standalone systems are commercial services that anyone with vendor credentials can test again; Harlow cannot be run by others. Where a choice exists, every confidence gate is tuned in the standalone systems' favour, and each system's verdict for every clip is part of the data release. No vendor has reviewed or endorsed this benchmark.

Citation#

If you use the benchmark, its results or its data, please cite the report.

Acknowledgements and contact#

Track B uses recordings from the Speech Accent Archive, by Steven H. Weinberger and Matthew C. Kelley at George Mason University. The background noise comes from the DEMAND corpus by J. Thiemann, N. Ito and E. Vincent.

Questions about the report, its data or a replication: support@tryharlow.com.