AAIA Question of the Day: Audit Sampling for Model Outputs

Quiz Yazan Ibrahim
AI audit training by expert Yazan Abu Ghosh for auditors with certifications.

AAIA exam practice question — AAIA Question of the Day: Audit Sampling for Model Outputs

AAIA exam practice question: daily practice for the ISACA Advanced in AI Audit (AAIA) exam — domain: AI Auditing Tools and Techniques.

Question

You are auditing an enterprise credit-scoring model. Management asks you to review model outputs for fairness and accuracy across segments. The population includes several predicted score bands and small but important protected groups. Which sampling approach gives the auditor the best chance to detect differential performance across both predicted outcomes and protected groups?

  • A. Draw a simple random sample of model outputs across the entire population.
  • B. Use a stratified random sample based only on the model's predicted labels.
  • C. Use a stratified random sample that ensures representation by both predicted label and key protected attributes.
  • D. Select a convenience sample of recent errors and monitoring-flagged cases.
Show the answer and explanation

Correct answer: C. Use a stratified random sample that ensures representation by both predicted label and key protected attributes.

A stratified random sample that ensures representation by both predicted label and key protected attributes is best because it produces sufficient observations in each combination of outcome band and protected group, enabling reliable comparisons and detection of disparate performance. A simple random sample risks under-representing rare but important groups or outcome bands. Stratifying only by predicted label addresses outcome balance but can still miss imbalances across protected attributes, limiting ability to detect fairness issues. A convenience sample of recent errors is biased toward known issues and may not reflect systematic differences across all segments, so it is unsuitable for an objective audit.

Want more practice?

Prepare for the ISACA Advanced in AI Audit (AAIA) exam with AI Audit & Compliance Framework: Practical Methods & Evaluation.