How to judge humanized text quality: the editors' reading rubric

Each output is read against its input by two editors who see a blind ID, not a tool name. Each gives one whole-number score from 1 to 5. The two scores are averaged for the run table and both are published in ratings.csv.

Dropped claims and invented facts are also counted as whole numbers, separately from the score, and named in the tool's review. A 3 that comes from one dropped supporting example is a different fault from a 3 that comes from one invented statistic, and the review says which.

The reading itself follows one procedure. The editor lists the input's claims as short lines before opening the output, then ticks each line off in the output, then reads the output once more looking only for sentences that have no source line. That last pass is where invented facts turn up, and they turn up more often than dropped ones, because a rewriter that is asked to vary a sentence tends to add rather than remove.

Twelve classes, in two groups. The first six are meaning faults and feed the meaning score above. The second six are writing faults and feed the errors-per-100-words figure, which carries 20 of the 100 points on the method page.

Because humanizers change length. An output that runs 150 words longer than its input has more room for faults, and a raw count would punish the tool for being wordy rather than for being wrong. Every count is therefore divided by the output's length in hundreds of words, so a 600-word output with six marked errors and a 300-word output with three both read as 1.0 per 100 words.

Readability is handled differently. The change between input and output is reported beside each tool and never scored. A text that got simpler is not better for it, and a text that got harder is not worse; the reader who asked for a research abstract and the reader who asked for a product description want opposite things.

A gap of 1 point stands and is averaged; the two scores are published as given. A gap of 2 or more points sends the output to a third editor, who reads it blind in the same way, and the third score is published alongside the first two with a note that it was a third read. The run notes list how many outputs went to a third read, per tool.

Weber-Wulff and colleagues' 2023 test is the reason the sentence-level record matters. Their 14 detection tools all scored below 80 percent accuracy, and the paper's error analysis, not its headline number, is what showed where each tool failed (Weber-Wulff et al. 2023, Testing of detection tools for AI-generated text, read 9 November 2026). A rating without the sentence behind it cannot be checked by anyone, including the vendor whose tool it describes.

The same way, with the same table. Put input and output side by side. Write the input's claims as a list before you read the output, then tick them off; anything in the output you cannot tick is an invented fact or a changed one. Then read the output alone for the six writing classes and count what you find, dividing by the output's length in hundreds of words.

Two habits make the reading more reliable. Read the output first for meaning and only afterwards for errors, since a reader hunting for grammar faults stops noticing what the sentences say. And read any sentence that sounds smoother than the input twice, because smoothing is where a rewriter most often drops a qualifier. The open-source blader/humanizer editing skill puts the same rule in its own terms: a name, number, date, quote or citation "must come from the source or the writer", and a rewriter that needs a missing detail should ask rather than invent one (blader/humanizer README, read 9 November 2026).

A text that reads well and says something different from what you meant has not been humanized; it has been replaced. No detector score should be the only evidence in a misconduct case, and no detector score should be the only evidence that a humanizer did its job either. The guide to a humanizer that actually works uses this rubric to say what "works" means; the runs page will carry every rating once the first run is complete. The guides hub holds the rest of this series.