A star rating averages everyone. Forkery counts only the people who cooked it.

What the build actually involved, the one decision that changed the result, and what the evaluation shows so far.

The finding that makes the case

Detroit Style Pizza on acozykitchen.com carries 4.74 stars from 100 raters. Forkery read all 152 comments and found 59 people who said they made it.

15 of those 59 independently added more water to the dough. Five said it failed outright. Not a coincidence, and not visible anywhere in the rating.

Definitely need to add 1 cup of warm water not the suggested 3/4. My dough was not squishy or hydrated enough.Nikki

The amount of water in the recipe does NOT create the "sticky dough" in the description.Taylor

The recipe is under-hydrated. Fifteen people worked that out on their own and wrote it down, and the number at the top of the page still says 4.74.

How it reads a page

Four stages, and only one of them involves a model. The fetcher pulls the recipe and its comments from the WordPress API or the YouTube API. A pre-filter drops, in ordinary code, what no model needs to see: the blogger's own replies, comments under four words, duplicates, anything not in English. What survives goes to TypeSafe's Jev, one call per comment. Everything after that is arithmetic.

The fetcher stops at 400 comments, and it takes the newest 400 when the comments carry dates; when a page shows them without dates, it takes the first 400 the page shows and says so. Most recipe pages are smaller than that and get read out. A popular video is not: reading it out would cost more than the answer is worth, so the card says which 400 it drew and how big the thread it drew them from was. A sample described as a sample is a finding; the same sample described as the thread is a claim the data does not support.

A single call carries the recipe title, its ingredient lines, the comment, and the parent comment if it is a reply. Against that it asks 37 questions at once for a twenty-ingredient recipe: ten about what the cook did, seven about how it went wrong, and one per ingredient. Jev answers the whole set in one parallel pass, so cost scales with comments rather than questions. 111 comments in eight seconds.

Every answer carries a confidence. A choice is accepted at 0.80, a yes/no at 0.70 and 0.30, and anything between is an abstention that the card counts and displays rather than rounding into a yes.

The decision that mattered

The spec asked "which ingredient did they change" as a single choice across all twenty-one options. Measured against hand labels, it was right 38% of the time.

Asked instead as twenty independent yes/no questions, one per ingredient, the same model on the same comments scored 91%.

A single choice cannot say that one comment changed four things, and plenty do. The shape of the question mattered far more than its wording, and no amount of prompt editing would have found that.

What the evaluation actually shows

The first round graded Jev against labels written by another model. It reported 91% agreement, and that number was misleading in a specific and instructive way: two models trained on similar data agree with each other more than either agrees with a person.

101 of the 200 comments are now labelled by hand. Against those:

Ten to twenty-two points of the original result was two models agreeing with each other. Hand labelling is the only thing that would have surfaced that.

The labels also named a failure mode nothing else would have: comment sections are full of people cooking right now. "Just made mine at 11:30pm to bake around 8am." "Making right now, does the recipe really call for 1/4 cup Italian seasoning?" Those read as cook reports and are not.

That produced a third question set with explicit rules for it, which lifted "would make it again" from 71 to 75% and "added something" from 80 to 85%. Worth reporting honestly: "did they cook it" did not move. It is still 80%, because Jev still reads prep-only comments as cooking. Naming the edge case was not the same as fixing it.

What is not finished

The release criterion in the spec is 600 hand-labelled comments across 30 recipes, with 100 double-labelled a week apart to measure how consistent the labelling itself is. There are 101 across 14. The keyword baseline and the frontier-model comparison the spec asks for are not done, and the confidence gates have not been tuned on a held-out split.

YouTube is weaker than the blogs: video descriptions often carry no ingredient list, so those cards can say how many cooked it and how it went, but not which ingredient anyone changed.

Accuracy targets of 90% and 85% are not met on the fields above. They are the targets, they are written down, and the current numbers are here rather than anywhere else.

Built on Cloudflare Workers with TypeSafe's Jev. Try it on a recipe.