AAIA Question of the Day: Labeling Quality Monitoring

Quiz Yazan Ibrahim
AI audit training by expert Yazan Abu Ghosh for auditors with certifications.

AAIA exam practice question — AAIA Question of the Day: Labeling Quality Monitoring

AAIA exam practice question: daily practice for the ISACA Advanced in AI Audit (AAIA) exam — domain: AI Operations.

Question

Your audit team reviews an AI operations process where a third‑party vendor continuously labels incoming customer support tickets for a supervised intent-classification model. Over the past quarter the model's online accuracy has been stable, but product owners suspect labeler quality may be degrading. Which control is most effective to detect systematic degradation in the vendor's labeling quality?

  • A. Periodic blind re-labeling of random samples by internal subject-matter experts with calculation of inter-annotator agreement metrics
  • B. Daily monitoring of the distribution of assigned labels in production to identify shifts from historical proportions
  • C. Alerting on drops in the model's production accuracy measured against delayed ground truth when it becomes available
  • D. Automated active sampling to send low-confidence examples back to the vendor more frequently without independent review
Show the answer and explanation

Correct answer: A. Periodic blind re-labeling of random samples by internal subject-matter experts with calculation of inter-annotator agreement metrics

Periodic blind re‑labeling by internal subject‑matter experts with inter‑annotator agreement directly measures label correctness and consistency independent of the vendor, making it the most reliable mechanism to detect systematic degradation. Monitoring label distribution (option 2) can show shifts but cannot distinguish true changes in data from labeler errors. Alerting on delayed production accuracy (option 3) may detect downstream effects but is reactive and can be delayed or confounded by other system changes. Automated sampling without independent review (option 4) increases vendor workload but does not provide an unbiased quality benchmark and therefore cannot reliably detect systematic errors.

Want more practice?

Prepare for the ISACA Advanced in AI Audit (AAIA) exam with AI Audit & Compliance Framework: Practical Methods & Evaluation.