AAIA Question of the Day: Canary Deployment Response

Quiz Yazan Ibrahim
AI audit training by expert Yazan Abu Ghosh for auditors with certifications.

AAIA exam practice question — AAIA Question of the Day: Canary Deployment Response

AAIA exam practice question: daily practice for the ISACA Advanced in AI Audit (AAIA) exam — domain: AI Operations.

Question

During a canary deployment of a fraud-detection model, the canary cohort shows a 12% higher false-positive rate than the incumbent model. Which immediate operational action best balances user impact mitigation with effective diagnosis?

  • A. Pause the rollout, keep the canary isolated, capture detailed telemetry and run targeted diagnostics before deciding on rollback or full deployment
  • B. Immediately rollback the canary and revert all production instances to the previous model version
  • C. Increase the model decision threshold in production to reduce false positives and continue the rollout to collect more data
  • D. Proceed with the full rollout to all users to accelerate data collection, then analyze results at scale
Show the answer and explanation

Correct answer: A. Pause the rollout, keep the canary isolated, capture detailed telemetry and run targeted diagnostics before deciding on rollback or full deployment

Pausing the rollout while isolating the canary and collecting detailed telemetry is best: it limits user impact, preserves the experimental conditions, and enables root-cause analysis before taking disruptive actions. Option 1 (correct) provides controlled diagnostics and avoids premature system-wide changes. Option 2 is too drastic because reverting all production instances ignores that the issue is currently limited to the canary and loses diagnostic data. Option 3 risks masking the underlying problem by changing decision behavior without understanding cause, which may introduce bias or degrade other metrics. Option 4 is unsafe because expanding exposure multiplies user harm and confounds causal analysis before you understand the failure.

Want more practice?

Prepare for the ISACA Advanced in AI Audit (AAIA) exam with AI Audit & Compliance Framework: Practical Methods & Evaluation.