I asked three AI judges to score my blog against twenty others, with letters in place of names. Mine tied for sixth of twenty-one. The blog in first place was nan.fyi, by Nanda Syahrasyad. Five blogs scored above mine, by two to eight points, and the judges put most of that gap in visual craft.
I took all four changes the top blogs pointed to, and the blog you’re reading is built on them.
The problem with a reviewer who knows
In my experience, a model that reviews something it wrote finds small things and misses big ones.
One study points the same way. In 2023, a team at UC Berkeley and other universities published a study of models used as judges. Strong models agreed with human judgments more than 80 percent of the time, about as often as the humans agreed with each other. The study also named the ways those judges go wrong. There is a position bias, where a judge favors an answer for where it sits, most often first, and a self-enhancement bias, where a model may favor answers it wrote itself.
So the judging here takes away what it can. Shuffling the order handles the first bias. The second I can only reduce, because the judges are the same kind of model that helped build this site. That is one more reason their scores are only a check.
How the judging works here
Before any judge sees a page, a rubric is written down: six criteria, each with a line on what a ten looks like. Three are visual craft. The other three ask whether the list helps you choose a post, whether the post is easy to read, and whether what the page does would carry to this site.
Then the pages are captured in a real browser, on a laptop and on a phone, and each one gets a letter instead of a name. The order of the letters is shuffled, and each of three judges reads them in a different order, so no page sits first for everyone. The judges start with no memory of the build. They score and rank the pages, and they list the defects of the weakest. That is the part to copy. Put letters where the names were, give each judge the pages in a different order, and ask each for a list of defects as well as a score. Then make the call yourself.
What the scores are for
I use the scores as a check on the page. On the Student page of this site, the judges’ scores didn’t track what I would accept. I rebuilt it on the reference they scored highest: a statement at display size with one button, and a column of text beside a tall portrait pinned at the right. I threw that version out. To me it looked elementary and sloppy. The build I kept was the site’s own layout carried to that page. The judges scored it below most of the references, and I kept it anyway. On the About page, built the same way, the page went back to the judges and came first for all three.
So I read the scores the way I read a test result. A low score tells me something is wrong, and the defect list tells me where. A high score tells me less, because the judges can’t see what I want the site to be.
The same idea at work
The writing system I built at EG Capital uses AI judges too. It drafts posts for the firm’s people. A panel of AI judges reads each draft, and a loop revises it until the changes stop helping. Then the system stops and asks the author. In a test run, the panel flagged an exchange in the draft that the author never had. The author decides what the firm says.
Sources
Next post Why a person checks every change before it ships All posts