Calibrating a transformers model with scipy CalibratedClassifierCV?

Intermediate
1

Hi folks,

Can someone point to sample code to calibrate a pre-trained transformers model (e.g., RoBERTa) using scipy CalibratedClassifierCV class?

I’ve fumbled for a day following this outline:

  1. Wrap a transformers model in a class inheriting BaseEstimator and ClassifierMixin.

  2. Implement both predict() and predict_proba()

  3. Construct a calibrated classifier via CalibratedClassifierCV(clf, cv=‘prefit’) where clf is my wrapped transformers model.

  4. call fit() on calibration dataset → fail with error message sklearn.utils._param_validation.InvalidParameterError: The ‘estimator’ parameter of CalibratedClassifierCV must be an object implementing ‘fit’ and ‘predict_proba’, an object implementing ‘fit’ and ‘decision_function’ or None. …

Help!