Short heads-up on the objective-projection dataset (500 scenes, 200 EN + 300 TR, CC BY-NC-ND).
Every scene carries an applied_rules field from a deterministic rule-based detector. I tested how reliable those flags are against blind hand-labeling on 120 Turkish scenes, and shipped documentation for what I found:
-
The detector is fine for surface features (e.g. simile) but no better than chance for materialized metaphor (κ ≈ 0.00), and its emotion rule over-fires (precision 0.17). Those flags are now explicitly marked as detector output, not validated labels.
-
A schema guide for the EN/TR field asymmetry, and a category audit that found the two halves categorize on different axes (EN = emotion only; TR = 80% scenario/genre).
It’s a single-rater pilot, Turkish-only, and I built the detector myself — all stated in the docs. A full write-up with the confusion tables is in preparation.
Full reasoning in the blog post: What happens when you check whether your own detector detects what it claims
If you annotate writing quality and want to be a second rater or poke holes in the labels, that’s exactly the feedback I’m after.