Fifteen lessons captured across the life of Project Catalyst and consolidated at closeout. Many trace directly to risks anticipated at program start — several of which materialized and were managed — providing a clear line from the original RAIDD register to realized experience. Each lesson records what happened, its impact, and a recommendation for future AI programs. This register transfers to ACME's AI Center of Excellence as reusable organizational knowledge.
15
Lessons Captured
5
Theme Areas
7
From Anticipated Risks
9
Practices to Keep
6
Improvements for Next Time
Governance & Risk
| ID | Lesson — What Happened | Impact | Recommendation |
|---|---|---|---|
| LL-01 | The two-line-of-defense model-risk structure (delivery as first line, Independent Model Validation as an independent second line) held throughout. | Cited by the AI Governance Board as the program's key control; no model reached production without independent sign-off. | KEEP — make independent model validation a standing structure, not a project role. |
| LL-02 | Independent validation flagged a fairness/bias issue in the underwriting model before production (RSK-06 realized). | Forced a retraining cycle and short delay; avoided a far costlier post-production fairness problem. | KEEP — budget retraining cycles as contingency; treat validation queue time as investment. |
| LL-03 | Phase gates requiring sign-off from all three boards occasionally slowed decisions. | Some decision latency early on, but prevented downstream rework and scope drift. | IMPROVE — pre-brief boards and batch decisions to reduce gate latency without weakening control. |
AI / ML Delivery
| ID | Lesson — What Happened | Impact | Recommendation |
|---|---|---|---|
| LL-04 | Human-in-the-loop, disclaimer, and confidence-threshold controls were designed into member-facing AI from the outset, informed by the Air Canada precedent (RSK-03 mitigated). | No hallucination/liability incident occurred across the program's member-facing deployments. | KEEP — design HITL and disclaimer controls in at requirements, never retrofit. |
| LL-05 | The underwriting risk model ran in shadow mode against live inputs before acting on any output. | Real-world performance was proven safely, de-risking the BRD-03 go-live. | KEEP — mandate shadow-mode for any consequential decisioning model. |
| LL-06 | Model cards and MLOps monitoring (drift, performance, retraining triggers) were standardized early. | Enabled a clean steady-state handover to ACME; models are maintainable without the delivery team. | KEEP — require a model card and monitoring plan as a Definition-of-Done gate. |
Data
| ID | Lesson — What Happened | Impact | Recommendation |
|---|---|---|---|
| LL-07 | Legacy claims-data fragmentation was the program's single largest realized risk (RSK-01 realized); early Phase-0 profiling surfaced it before Build. | A funded remediation effort (drawn from contingency) protected the BRD-01 timeline; had it surfaced later, the slip would have been severe. | KEEP — front-load data profiling in Phase 0 as a default on any AI program. |
| LL-08 | A privacy audit found an offshore data-access gap (RSK-08 realized); onshore-only staffing for PHI claims-decision roles proved correct. | Offshore work was briefly paused and re-scoped; no PHI exposure occurred. | KEEP — set data-residency and access boundaries by role before offshore mobilization. |
Financial & Commercial
| ID | Lesson — What Happened | Impact | Recommendation |
|---|---|---|---|
| LL-09 | GenAI compute/token spend proved highly variable and pressured the platform budget (RSK-02 realized). | A mid-program FinOps adjustment (within contingency) was required; cost telemetry added before scale-up prevented overrun. | IMPROVE — treat AI compute as a variable cost with telemetry and alerting from day one, not a fixed line. |
| LL-10 | The 9.1% contingency reserve absorbed two formal re-baselining events without increasing the authorized budget. | Program closed $0.6M under budget; contingency sizing was adequate. | KEEP — size contingency to realized-risk scenarios, and gate its release at the steering board. |
| LL-11 | A platform/SaaS vendor slipped a commitment mid-program (RSK-05 realized). | Required schedule rework and a contract renegotiation; recovered without an envelope breach. | IMPROVE — build milestone-linked flexibility and remedies into AI vendor contracts up front. |
Change & Adoption
| ID | Lesson — What Happened | Impact | Recommendation |
|---|---|---|---|
| LL-12 | Frontline adoption resistance appeared as capabilities rolled out (RSK-07 realized). | A phased, opt-in rollout with added change-management investment achieved adoption without mandate. | KEEP — phase adoption and co-design with frontline users; avoid big-bang cutovers for AI. |
| LL-13 | An executive sponsor transition occurred mid-program (RSK-09 realized). | A formal re-baselining and re-confirmation of scope preserved continuity with no delivery disruption. | KEEP — maintain a living charter and re-baseline formally on any sponsor change. |
| LL-14 | Shadow-AI scope pressure recurred as business units requested additional use cases (RSK-10 realized). | The CoE intake process either folded requests into the roadmap or deferred them, protecting scope. | KEEP — run a CoE intake/triage process to absorb demand without scope creep. |
| LL-15 | Change-management effort was underestimated in the initial plan. | Additional OCM investment was needed to hit adoption targets. | IMPROVE — budget organizational change management more generously on AI programs than on traditional IT. |
Disposition. Nine lessons are "keep" practices to be codified as CoE standards; six are improvements folded into the CoE's playbook for future AI initiatives. Seven of the fifteen trace directly to risks anticipated in the original RAIDD register — evidence that disciplined forward risk identification at program start paid off in managed, rather than surprise, outcomes.