Master Test & QA Strategy — Project Catalyst

The governing quality document for Project Catalyst. It defines how the program tests — across three AI delivery legs, a shared data and cloud platform, and a governance layer — and it applies to every workstream, vendor, and release for the life of the program. Individual test plans (per BRD, per release) sit beneath this strategy and inherit from it; where a test plan is silent, this document governs. Structured in line with ISO/IEC/IEEE 29119, the international software testing standard series that superseded IEEE 829 and added explicit organizational test policy and strategy layers.

Test Organization

QA 18 · UAT 11 · IMV 6

Test Levels

Unit → UAT

AI Model Dimensions

Beyond functional testing

Lines of Defense

Delivery QA + independent IMV

Approach

Risk-based

Per ISO 29119-1:2022

The central premise. Testing an AI program is not testing software with extra steps. A traditional application is deterministic: given an input, a correct output exists and can be asserted. A model is probabilistic — it can be accurate on average and still be unfair, poorly calibrated, brittle under adversarial input, or silently degrading against last quarter's data. This strategy therefore runs two distinct but connected test disciplines: conventional software testing of the applications and integrations, and model validation of the AI itself. Both must pass before anything reaches production.

1 · Purpose, Scope & Objectives

Purpose

To define a single, consistent, auditable approach to quality across Project Catalyst so that AI capability reaches ACME's members, providers, and underwriters only when it has been demonstrated to be functionally correct, performant, secure, fair, explainable, and compliant — and so that it stays that way after go-live.

In scope

All three delivery legs: BRD-01 Claims & Prior Authorization AI, BRD-02 Member & Provider Experience AI, BRD-03 Underwriting & Risk AI.
The Data & Cloud AI Platform, its data pipelines, feature store, and MLOps tooling.
Integrations with ACME claims, member, provider, and underwriting systems.
AI/ML models, including training data quality, model performance, and post-deployment behavior.
Non-functional characteristics: performance, security, accessibility, and regulatory compliance.

Out of scope

CMS-0057-F baseline compliance testing — a separate ACME project; only the integration interface is tested here.
Testing of ACME legacy systems beyond the interfaces this program consumes or writes to.
Vendor-internal testing of the hyperscaler or foundation-model platform, which is evidenced through vendor attestation and contractual assurance rather than re-tested by the program.

Objectives

Objective	How it is met
No critical defect reaches production	Tiered test levels with enforced exit criteria and a production readiness gate
No model reaches production without independent validation	Second-line IMV sign-off is a mandatory, non-waivable gate
Quality is evidenced, not asserted	Traceability from requirement → test case → result → defect → release record
Testing scales across three parallel delivery legs	Shared automation framework, common environments, reusable regression suites
Quality persists after go-live	Continuous monitoring, drift detection, and scheduled re-validation

2 · Standards & Framework Alignment

The strategy is anchored to recognized external standards rather than invented convention, so that ACME's internal audit, regulators, and external reviewers can map it to frameworks they already accept.

Framework	Applies to	How this program uses it
ISO/IEC/IEEE 29119 (Parts 1–4)	Software testing process & documentation	Structural basis for this strategy, the test process model, test documentation set, and test design techniques. Part 1 (2022) supplies the risk-based approach; Part 3 supplies documentation templates.
NIST AI Risk Management Framework	AI trustworthiness	Source of the model test dimensions in Section 5 — validity, reliability, safety, security, fairness, explainability. NIST's position that AI systems be tested before deployment and regularly while in operation drives Section 12.
ISO/IEC 42001	AI management system	Governance wrapper — links test evidence to the AI management system and model registry.
SR 11-7 model risk principles	Model risk management	Basis for the two-line-of-defense split: delivery teams build and test; Independent Model Validation independently validates.
HIPAA / PHI handling	Test data	Governs Section 9 — test data management, masking, and environment segregation.
Program-internal	Governance	AI Governance & JAD Charter, Methodology Guide, RAIDD Log

3 · Test Organization & Responsibilities

Thirty-five people across three independent groups. The separation is deliberate: the team that builds a model cannot be the team that clears it for production.

Group	Size	Reports to	Responsibility	Line
QA & Testing	18	QA Lead (Pulaski)	Test planning, case design, execution across unit through system/E2E, automation framework, performance and regression testing, defect triage	First
User Acceptance Testing	11	Operational Testing Manager (ACME)	Business-scenario validation, operational readiness, UAT sign-off. Deliberately independent of the delivery teams and on loan from Operations, not from IT.	Business
Independent Model Validation	6	P. Okafor, reporting outside delivery	Independent replication, fairness re-testing, calibration and robustness assessment, production gate sign-off for every model	Second
Cybersecurity / InfoSec	10	CISO	Security and penetration testing, threat modelling, controls assurance	Supporting
Clinical / Medical Policy	6	Chief Medical Officer	Clinical validity review of decision logic and medical-necessity criteria	Supporting
Data Privacy Office	5	Chief Privacy Officer	DPIA, PHI handling assurance in test environments	Supporting

Independence is a control, not an org-chart detail. UAT reports through Operations and IMV reports outside the delivery line specifically so that neither can be pressured by a delivery deadline. If a delivery lead could overrule an IMV finding, the second line would provide no assurance at all.

4 · Test Levels (Application & Integration)

Conventional software test levels, applied to the applications, APIs, workflows, and integrations that surround the models.

Level	Owner	Scope	Approach	Automation Target
Unit	Delivery engineers	Individual components, functions, transformations	Developer-written, executed in CI on every commit	≥ 95%
Component	QA & Testing	Services and modules in isolation with stubbed dependencies	Automated suite in CI	≥ 85%
Integration	QA & Testing	Service-to-service, API contracts, data pipeline handoffs, legacy interfaces	Contract testing plus scenario-based integration suites	≥ 75%
System / End-to-End	QA & Testing	Complete business workflows across the assembled solution	Scripted functional scenarios plus exploratory testing	≥ 60%
Non-functional	QA + InfoSec	Performance, load, resilience, security, accessibility	Specialist tooling per characteristic (Section 6)	≥ 70%
User Acceptance	UAT team (ACME)	Real business scenarios executed by operational staff	Manual, business-scenario driven, two cycles plus regression confirmation	Manual by design

5 · AI / ML Model Test Dimensions

These are additional to — never a substitute for — the levels above. A model that passes functional testing can still be unfit for production on any of these eight dimensions. Each is assessed by the delivery team (first line) and independently re-assessed by IMV (second line) before the production gate.

#	Dimension	Question it answers	Method	Gate
1	Data quality & representativeness	Is the training and evaluation data complete, accurate, and representative of the population the model will serve?	Profiling, completeness and distribution analysis, sub-population coverage checks against ACME's member mix	Pre-training
2	Performance & generalization	Does it perform on data it has never seen, not just on its training set?	Hold-out and cross-validation; precision, recall, F1, AUC against agreed thresholds per use case	Pre-pilot
3	Calibration	When the model says it is 80% confident, is it right about 80% of the time?	Reliability curves, calibration error — critical because confidence thresholds drive the human-in-the-loop routing rules	Pre-pilot
4	Robustness & adversarial resistance	Does it hold up under noisy, edge-case, malformed, or hostile input?	Perturbation testing, edge-case suites, adversarial probes; prompt-injection testing for language interfaces	Pre-production
5	Fairness & bias	Does performance hold across protected and clinically relevant sub-populations?	Disaggregated performance analysis across sub-populations, agreed fairness metrics, disparity thresholds set with Compliance and Clinical	Pre-production — blocking
6	Explainability	Can a human understand and defend why a decision was made?	Feature-attribution review, decision-path inspection, clinical reviewer comprehension testing	Pre-production
7	Inference integrity & reproducibility	Does the same input reliably produce the same output, and can a past decision be reproduced?	Determinism checks, versioned model + data + config lineage, replay of historical decisions	Pre-production
8	Drift & stability	Is it still behaving as validated, months after go-live?	Continuous data and prediction drift monitoring with alert thresholds; scheduled re-validation (Section 12)	Post-production, continuous

Fairness is a blocking gate. Of the eight dimensions, fairness and independent validation are the two that cannot be waived by any delivery authority. A fairness finding stops the release until remediated and re-validated, regardless of schedule pressure — a control the program expects to exercise at least once, since RSK-06 anticipates exactly this scenario on the underwriting model.

6 · Test Types by Quality Characteristic

Characteristic	What is tested	Acceptance basis
Functional correctness	Business rules, decision logic, workflow routing, data transformations	100% of critical-path requirements traced and passing
Performance & scalability	Inference latency, throughput at projected volume, batch window completion, concurrency	Meets service-level targets at 1.5× projected peak volume
Security	Authentication, authorization, encryption, injection resistance, model endpoint protection	Zero high or critical findings open at release
Privacy	PHI handling, minimum necessary access, data residency, audit logging	DPIA complete; Data Privacy Office sign-off
Accessibility	Member and provider interfaces	WCAG 2.2 AA conformance
Regulatory compliance	Prior-auth decisioning rules, disclosure and disclaimer behavior, record retention	Compliance and Legal sign-off per release
Operability	Monitoring, alerting, failover, rollback, runbook accuracy	Operational readiness review passed

7 · Risk-Based Test Approach

ISO/IEC/IEEE 29119-1 frames testing as the primary means of treating product risk, and this program allocates test effort accordingly rather than spreading it evenly. Test depth is set by the consequence of failure.

Risk Tier	Characteristics	Examples	Test Depth
Tier 1 — Critical	Affects a coverage or payment decision, patient care, regulatory obligation, or PHI	Prior-auth decisioning, claims adjudication logic, underwriting risk scoring	All levels, all eight model dimensions, IMV mandatory, clinical review, full regression, shadow-mode operation before live decisioning
Tier 2 — Significant	Member- or provider-facing, or feeds a Tier 1 process	Member virtual assistant, provider portal AI, agent-assist	All levels, model dimensions 1–7, IMV mandatory, human-in-the-loop controls verified, staged rollout
Tier 3 — Standard	Internal-facing, no direct decision or member impact	Internal analytics, operational dashboards, developer tooling	Standard levels, automated regression, risk-proportionate model checks

Risk tiers are assigned at requirements sign-off and recorded in the requirements traceability matrix. Product risks that materialize become entries on the RAIDD Log; RSK-01 (data fragmentation), RSK-03 (chatbot liability), and RSK-06 (model bias) each map directly to test dimensions above.

8 · Entry & Exit Criteria

Criteria are enforced, not advisory. A level does not begin until entry criteria are met, and does not close until exit criteria are evidenced.

Level	Entry Criteria	Exit Criteria
Integration	Components unit-tested and passing; interfaces stable; environment provisioned; test data available	≥ 95% planned cases executed; no open critical or high defects; interface contracts verified
System / E2E	Integration exit met; full workflow deployed; requirements traceability current	100% critical-path scenarios passed; ≥ 98% overall pass rate; no open critical or high defects; performance targets met
Model validation (IMV)	Model card complete; first-line results submitted; training data lineage documented; fairness analysis provided	Independent replication successful; fairness within agreed thresholds; calibration acceptable; formal IMV sign-off issued
UAT	System exit met; IMV sign-off issued; UAT environment loaded with representative data; users trained	All business scenarios executed; no open critical or high defects; business sign-off recorded
Production release	All above complete; operational readiness review passed; rollback rehearsed	Three-board production readiness approval (ESB, AIGB, EARB); hypercare staffed and monitoring live

9 · Test Environments & Data Management

Environment	Purpose	Data	Refresh
Development	Engineer workstation and shared dev	Synthetic only	On demand
Test / QA	Component, integration, system execution	Masked / de-identified production-derived	Weekly
Model training & evaluation	Model development and validation	Governed PHI under approved access, onshore-only	Per training cycle
Performance	Load, stress, endurance	Volume-scaled synthetic	Per release
UAT / Staging	Business acceptance; production mirror	Masked production-derived, full referential integrity	Per release cycle
Production	Live operation and monitoring	Live PHI	—

Test data principles

PHI never leaves approved boundaries. Offshore engineering works exclusively with synthetic or fully de-identified data; onshore-only access applies to any environment containing real claims-decision data — the control that RSK-08 anticipates being tested by privacy audit.
Masking preserves statistical shape. De-identification must retain distributional properties, or model evaluation on masked data becomes meaningless.
Representativeness is verified, not assumed. Evaluation datasets are checked for sub-population coverage before use in fairness testing.
Data, model, and config are versioned together. Any validated result must be reproducible from its recorded lineage.

10 · Defect Management

Severity	Definition	Response	Release Impact
Critical	Incorrect coverage/payment decision, PHI exposure, safety or regulatory breach, or complete loss of service	Immediate; work stops until resolved	Blocks release absolutely
High	Major function unusable, significant model performance shortfall, no workaround	Within 2 business days	Blocks release
Medium	Function impaired with an acceptable workaround	Within the release cycle	Release with documented workaround
Low	Cosmetic or minor usability issue	Backlog	No impact

Defects are triaged daily during active test windows by QA Lead, delivery lead, and business analyst. Any defect arising from model behavior — rather than code — is additionally routed to the AI Governance Board, because the remedy is usually retraining or threshold adjustment rather than a code fix, and that is a governance decision.

11 · Test Automation Strategy

Automation exists to make regression affordable across three parallel delivery legs and a three-year program. It is not a target in itself.

Automate the repeatable, explore the novel. Regression, contract, and data-pipeline checks are automated; first-pass validation of new AI behavior is exploratory and human-led.
CI/CD gating. Unit and component suites run on every commit; integration and regression suites run nightly and as a release gate. A failing gate blocks promotion.
Model evaluation is automated as a pipeline stage, so performance, calibration, and fairness metrics are produced on every training run rather than assembled by hand at gate time.
Test assets are versioned with the code they exercise and owned by the team that owns the component.
The automation framework is shared across BRD-01/02/03 — built once during BRD-01, reused thereafter, which is a substantial part of why Year 2 can run two delivery legs in parallel.

12 · Continuous Testing in Production

NIST's framing is that AI systems be tested before deployment and regularly while in operation. For a probabilistic system, release is not the end of testing — it is the point at which testing changes form.

Activity	Cadence	Owner	Trigger for action
Data & prediction drift monitoring	Continuous, automated	Platform / MLOps	Drift beyond threshold → investigation and possible retraining
Live performance monitoring against validated baselines	Continuous	Delivery + CoE	Degradation beyond tolerance → AI Governance Board review
Human-override rate tracking (prior auth)	Weekly	Clinical + CoE	Elevated override rate → decision-logic review; this is the early-warning signal for RSK-04
Fairness re-assessment on production data	Quarterly	IMV	Disparity beyond threshold → blocking remediation
Full model re-validation	Annually, and on any material retrain	IMV	Failure → model withdrawn from autonomous operation
Production regression suite	Per release and monthly	QA & Testing	Failure → rollback consideration

13 · Metrics & Reporting

Metric	Definition	Target	Reported
Requirements test coverage	Requirements with at least one linked, executed test	100% critical path	Per release
Test execution progress	Cases executed vs planned	≥ 95% at exit	Weekly in test windows
Pass rate	Passed vs executed	≥ 98% at system exit	Weekly
Defect escape rate	Defects found post-release vs total found	≤ 5%; zero critical	Per release + monthly
Defect density	Defects per unit of delivered scope	Trend downward release over release	Monthly
Automation coverage	Automated vs total regression cases	Per level targets (Section 4)	Monthly
IMV turnaround	Submission to sign-off	5–10 business days	Monthly
Model performance vs baseline	Live metrics vs validated baseline	Within tolerance	Continuous, reviewed monthly

Test metrics feed the Program Dashboard and the monthly quality section of Steering Board reporting.

14 · Test Deliverables

Deliverable	Produced by	When	29119 Alignment
Master Test & QA Strategy (this document)	QA Lead + Program Director	Phase 0; reviewed annually	Organizational test strategy
Test plan (per BRD / per release)	QA Lead	Per delivery leg	Part 3 test plan
Test cases & test data specifications	QA & Testing	Design phase	Part 3 / Part 4 techniques
Model validation report	IMV	Per model, pre-production	Program-specific
Test execution & defect reports	QA & Testing	Continuous in test windows	Part 3 test results
Test summary / exit report	QA Lead	Per level exit	Part 3 test completion report
UAT sign-off record	Operational Testing Manager	Per release	Acceptance evidence

15 · Assumptions, Dependencies & Risks to the Test Approach

Assumptions

Representative, de-identified production-derived data can be made available to test environments within privacy constraints.
UAT staff are released from operational duties for the agreed cycles — historically the most common cause of UAT slippage on programs of this size.
Fairness thresholds and model performance targets are agreed with Compliance and Clinical at requirements sign-off, not negotiated at gate time.

Dependencies

Platform and environment availability per the program schedule; environment delay compresses test windows rather than moving gates.
IMV capacity — the second line is 6 people validating models from three delivery legs; the 5–10 day queue is a planning constraint, not a buffer.
Legacy data remediation (RSK-01) completing sufficiently to support representative model evaluation.

Risks to the approach

Risk	Impact on testing	Mitigation
Test window compression from upstream delay	Pressure to reduce coverage on the highest-risk scope	Exit criteria are non-negotiable; the release date moves, the criteria do not
IMV becomes a bottleneck in Year 2 (two legs in parallel)	Production gates queue	Staggered BRD-02/03 validation windows; early submission of model documentation
Insufficient sub-population data for fairness testing	Fairness conclusions unreliable	Coverage verified at dataset governance sign-off, before training
Automation debt accumulating across three legs	Regression cost grows faster than delivery	Shared framework, automation coverage tracked as a reported metric

Governing relationship. This strategy is subordinate to the Program Charter and the Program Management Plan, and is the parent of every test plan produced under it. Changes to this strategy follow the program's change-control process and require AI Governance Board endorsement, because test standards are a model-risk control.