A second AI checked the first one’s work
- What this shows
- The Checker (claude-opus-5) sorted 200 posts again on its own, without seeing how the Sorter (claude-haiku-4-5) had sorted them. Where they agree, the sorting is reliable. Where they don’t, we say so.
- How to read it
- Each number is how often the two agreed. Lower numbers mean that kind of label should be read with more caution.
agree on whether a post is about finding a photo (200 posts)
agree on whether the person describes one specific photo
overlap on what the person remembered
overlap on what went wrong, the least reliable label, so we lean on it less
Agreement beyond chance on the first question (Cohen’s kappa) is 0.59, usually read as moderate. Most disagreements are borderline posts like general complaints about the app.
How this is calculated
- What is counted
- The same 200 posts, each sorted by both models as about finding a photo or not.
Checker: yes Checker: no Total Sorter: yes 115 5 120 Sorter: no 32 48 80 Total 147 53 200 - Formula
- Observed agreement po = (115 + 48) / 200 = 0.815. Agreement expected by chance pe = (120/200 × 147/200) + (80/200 × 53/200) = 0.547. Cohen's kappa = (po − pe) / (1 − pe) = (0.815 − 0.547) / 0.453 = 0.59.
- Keep in mind
- Kappa discounts the agreement two raters would reach by luck, which is high here because both say "yes" often. 0.41 to 0.60 is conventionally read as moderate (Landis and Koch, 1977). Most of the 37 disagreements are the Checker counting borderline app complaints as relevant. Two different AI models, not two people.
20 of 49 “specific photo” stories held up when read by hand
- What this shows
- The AI flagged posts where someone describes one particular photo they were looking for. A person read every one and rejected those that were really about deleted photos, backups or settings advice.
4 rejected
Advice comment
3 rejected
Testing description field
2 rejected
Photos disappeared
2 rejected
Photos missing
1 rejected
Web geolocation display
1 rejected
Photo taken moments ago
Limits we know about
Public posts lean negative
People post when something breaks, and tend to blame the app. That is why we also ran our own survey and live interviews.
It shows what people remember, not what they forgot
A post can only mention what the person still recalls. Our survey asks people to write down their memory before searching, then check the real date.
Some groups are small
Only 20 posts describe a specific photo in detail. Those are shown as counts, never as percentages.
Few App Store reviews
Apple only shows the 20 most helpful reviews per country, and most are general praise. 200 reviews from 20 countries gave just 4 about finding photos.