AAIA Question of the Day: Confidence Calibration Testing

Quiz Yazan Ibrahim
AI audit training by expert Yazan Abu Ghosh for auditors with certifications.

AAIA exam practice question — AAIA Question of the Day: Confidence Calibration Testing

AAIA exam practice question: daily practice for the ISACA Advanced in AI Audit (AAIA) exam — domain: AI Auditing Tools and Techniques.

Question

You are auditing a production binary classifier that outputs probability scores for a rare positive class. The available labeled calibration data for that class is limited. Which approach is best for assessing and recalibrating the model's probability estimates without introducing overfitting?

  • A. Apply Platt scaling (parametric logistic calibration) using a held-out calibration set
  • B. Compute the Brier score and use bootstrap resampling to estimate calibration uncertainty
  • C. Train an isotonic regression calibrator on the full calibration set to map raw scores to probabilities
  • D. Tune the classification threshold on validation data to maximize F1 score for the rare class
Show the answer and explanation

Correct answer: A. Apply Platt scaling (parametric logistic calibration) using a held-out calibration set

Platt scaling is a parametric logistic calibration method with few parameters, so it is less prone to overfitting when calibration data for a rare class is limited; it both assesses and adjusts predicted probabilities. The Brier score quantifies calibration quality and can estimate uncertainty with bootstrapping, but it does not recalibrate probabilities. Isotonic regression is nonparametric and flexible, which makes it likely to overfit when calibration data are scarce, producing unreliable probability maps. Tuning the classification threshold optimizes decision outcomes (e.g., F1) but does not correct the underlying probability estimates.

Want more practice?

Prepare for the ISACA Advanced in AI Audit (AAIA) exam with AI Audit & Compliance Framework: Practical Methods & Evaluation.