Hi John,
I wanted to thank you again for all the time you’ve invested in reviewing the project. Your comments have genuinely helped transform what initially looked like a broad conceptual proposal into a much more structured and testable research program.
I thinking about your comments regarding prompt-only conditions versus the fine-tuned condition.
My current view is that, for small models, the fine-tuned checkpoint remains an essential reference condition. This isn’t only a theoretical intuition—it comes from previous experiments I documented during a pilot study. What I consistently observed is that smaller models often treat the PCE axioms as an explicit list of instructions rather than integrating them as an operational reasoning framework. The fine-tuning appeared to internalize that functional organization much more effectively.
Of course, I don’t see this as something that should simply be assumed. Quite the opposite: I think it should be treated as an empirical hypothesis that the evaluation protocol can either support or challenge. That’s why I would still keep prompt-only conditions as mandatory controls, exactly as you suggested.
There is one area where I think your help could make a real difference.
One of the reviewers of my work suggested that the evaluation dilemmas should be as neutral as possible and ideally written by someone independent from the theoretical framework itself. I completely agree with that recommendation.
Would you be interested in designing two neutral benchmark batteries?
pce_benchmark_neutral_20.json(a small reference battery for the REO);
pce_benchmark_neutral_100.json (the full benchmark used in the Standard Experimental Protocol).
The idea would simply be to create balanced, methodology-oriented evaluation cases, independent of EPE itself. I think having someone external design them would significantly strengthen the credibility and reproducibility of the evaluation process.
Please don’t feel any obligation at all—I know you’ve already devoted a great deal of time to this project. I simply thought you would be the ideal person to ask, given how much you’ve contributed to shaping the methodology.
Thank you again for all your thoughtful feedback. It has had a much bigger impact on the project than you probably realize.
Best regards,
Allan