AAIA Question of the Day: Synthetic Data Validation

Quiz Yazan Ibrahim
AI audit training by expert Yazan Abu Ghosh for auditors with certifications.

AAIA practice question — AAIA Question of the Day: Synthetic Data Validation

AAIA practice question: daily practice for the ISACA Advanced in AI Audit (AAIA) exam — domain: AI Auditing Tools and Techniques.

Question

You are auditing an AI classification model trained on sensitive customer records. The data science team supplies a synthetic dataset intended to replace real records for external testing. Which approach most reliably evaluates whether the synthetic data preserves the model's predictive utility while limiting disclosure risk?

  • A. Compare distributional similarity metrics (e.g., KS, Wasserstein) between original and synthetic feature sets
  • B. Train the intended predictive model on the synthetic data and compare its performance on a withheld real test set to the baseline trained on real data
  • C. Perform disclosure-risk testing only, such as nearest-neighbor record linkage, to quantify privacy leakage
  • D. Review the synthetic generator's configuration, code and random seeds to verify sampling logic and reproducibility
Show the answer and explanation

Correct answer: B. Train the intended predictive model on the synthetic data and compare its performance on a withheld real test set to the baseline trained on real data

The best approach is to train the same model on synthetic data and evaluate it against a withheld real test set: this directly measures whether synthetic data preserves the predictive relationships the model needs (utility) while comparison to the baseline indicates degradation. Option A (distributional tests) can detect marginal or feature-level differences but can miss preserved or broken predictive relationships. Option C (disclosure-risk testing) assesses privacy leakage but does not measure whether the synthetic data supports equivalent model performance. Option D (code/config review) helps with reproducibility and process assurance but cannot by itself prove model utility or quantify privacy risk.

Want more practice?

Prepare for the ISACA Advanced in AI Audit (AAIA) exam with AI Audit & Compliance Framework: Practical Methods & Evaluation.