Record as at June 10, 2024 — ten days after the EDI translation layer cut over to the integrated path on May 31, 2024. Fourteen interfaces were specified in the Integration Architecture; this is the record of how each was built, tested and cut over, what broke, and which of them the program is still carrying. It is written after the fact for the interfaces that are done and as a live record for the three that are not.
Contents
- The defect record
- What interface testing proved, and what it did not
- The failure that does not announce itself
1. Status of the fourteen interfaces
| Ref | Interface | Type | Wave | Status |
|---|---|---|---|---|
| IF-01 | Enrollment — target to ACME | X12 834 | 1 | Cut over; stable |
| IF-03 | Eligibility inquiry / response | X12 270/271 | 1 | Cut over; stable |
| IF-04 | Claims — target platform to ACME | X12 837 | 1 | Cut over; stable |
| IF-05 | Remittance advice | X12 835 | 1 | Cut over; stable |
| IF-10 | Premium billing and payment | X12 820 | 1 | Cut over; two defects, both closed |
| IF-02 | Enterprise member identifier broadcast | API | 2 | Cut over; volume higher than modeled |
| IF-06 | Accumulator synchronization | API | 2 | Cut over; one defect, closed |
| IF-07 | Provider master distribution | Batch | 2 | Cut over; stable |
| IF-11 | Financial posting to general ledger | Batch | 2 | Cut over; three defects, all closed |
| IF-12 | Member correspondence and ID cards | File | 2 | Cut over; carried the card run |
| IF-08 | Authorization status | API | 3 | Cut over; stable |
| IF-09 | Care management member feed | Batch | 3 | Cut over; preserved-capability dependency |
| IF-13 | Data warehouse ingestion | Batch | 3 | Cut over; silent-failure defect, remediated |
| IF-14 | Write-back to the retiring platform | API | 3 | Live and load-bearing — see section 6 |
Wave 1 cut over inside the Day 1 window on October 2, 2023. Wave 2 followed at the full production cutover on November 2, 2023, after parallel operations concluded on November 1, 2023. Wave 3 ran through the first half of 2024, gated behind the host claims processing retirement on April 30, 2024 and completed for the transaction set at the EDI cutover on May 31, 2024.
2. Why the waves were ordered as they were
The ordering was not by difficulty or by readiness. It was by what a member or a provider would notice within a day of it failing.
Wave 1 is the transaction spine: enrollment in, eligibility answered, claims received, remittance issued, premium billed. If any of these is down, a member is turned away at a point of care or a provider is not paid, and neither has a workaround. These were cut over first, in the window where the whole program was standing by and a rollback was still available.
Wave 2 carries interfaces whose failure is serious but not same-day visible — an accumulator that drifts, a general ledger posting that has to be corrected at month end, a provider file a week stale. Wave 3 holds everything whose failure is invisible to members entirely.
Sequencing by blast radius rather than by readiness is a deliberate inversion of the usual instinct. The natural order is to cut over what is ready, which front-loads the easy interfaces and leaves the hard ones for the window with least contingency behind it. Ordering by consequence does the opposite: it puts the interfaces that cannot be allowed to fail into the window where the entire program is watching and a rollback path is still open, and defers the ones that can tolerate a bad week into windows where a bad week is survivable. The cost is that the hardest work happens under the most pressure. That was accepted.
3. The defect record
| Ref | Interface | Defect | Class | Status |
|---|---|---|---|---|
| D-I-01 | IF-10 | Group-level premium adjustments applied at the subscriber level, producing correct totals against incorrect member detail | Semantic | Closed |
| D-I-02 | IF-10 | Retroactive termination transactions processed in receipt order rather than effective-date order | Semantic | Closed |
| D-I-03 | IF-11 | Three revenue codes with no counterpart in the target chart of accounts posted to suspense without alerting | Semantic | Closed |
| D-I-04 | IF-11 | Reversals posted to the period of correction rather than the period of the original entry | Semantic | Closed |
| D-I-05 | IF-11 | Undocumented local convention: a trailing record used as a control total by the source, treated as data by the target | Semantic | Closed |
| D-I-06 | IF-06 | Accumulator credits from the retiring platform applied without honoring the plan-year boundary | Semantic | Closed |
| D-I-07 | IF-13 | Partial file accepted and reported as a successful load; see section 5 | Silent | Remediated |
Every defect that reached production was semantic, and not one was structural. The X12 interfaces — the ones that look hardest, carry the most specification, and consumed the most build estimate — produced no production defects at all. They are governed by published implementation guides, both organizations had been exchanging them for years, and a conforming transaction is machine-checkable. The defects came from the batch and file interfaces, where the format is whatever the two systems agreed to locally, sometimes a decade ago, sometimes by a person who has left. Difficulty in an interface is not proportional to its complexity; it is proportional to how much of its meaning lives outside its specification. D-I-05 is the clearest instance: a trailing record that one system had always used as a control total and the other read as another row of data. Nothing in the file said which it was.
4. What interface testing proved, and what it did not
Every interface passed its test cycle before cutover. Six of them subsequently produced production defects. That is not a testing failure in the sense of tests being skipped — it is a statement about what the tests were able to assert.
Interface regression is run by the offshore test team, which by the program's inherited data constraint works against synthetic and de-identified datasets. Synthetic data is generated from the specification. It therefore conforms to the specification by construction, and it cannot contain the thing that causes semantic defects: a real record carrying a local convention nobody wrote down.
A structural test and a semantic test answer different questions, and passing the first says nothing about the second. Structural testing asks whether the transaction conforms — segments present, values in range, totals balancing. All fourteen passed that, and the X12 interfaces were never going to fail it. Semantic testing asks whether the receiving system arrives at the same meaning the sending system intended, and it can only be run against real data, reconciled by someone who knows what the numbers are supposed to be. That is the parallel run, it needs member-level claims, and it therefore cannot be done offshore. The defects that reached production are concentrated exactly where the parallel run had least coverage — the financial and file-based interfaces, which were reconciled at aggregate level while the claims interfaces were reconciled claim by claim.
The remediation applied from Wave 2 onward was to require, for any batch or file interface, a reconciliation against a sample of real production output performed by someone from the originating organization — in practice J. Amonett or a member of his team — before the interface is accepted. That check found D-I-05 before it reached production for two further interfaces.
5. The failure that does not announce itself
D-I-07 is the defect worth keeping. The data warehouse ingestion job ran on schedule, completed without error, and reported a successful load. It had loaded a partial file. The upstream extract had been truncated by a change in the source system's export window, and the ingestion job — which validated that the records it received were well-formed — had no basis on which to notice that records it never received were missing.
It went undetected for eleven days. It was found not by monitoring but by an analyst who noticed a membership count that looked low against a figure she remembered.
Alerting on absence is harder than alerting on error, and it is the thing most monitoring omits. An error has an event attached to it — something threw, something failed a check, something can be caught and raised. An absence has no event; nothing happens, and nothing is exactly what a healthy quiet system also looks like. The data warehouse requirements anticipated this and called for alerting on expected-but-absent data; the requirement had been implemented for feed arrival, so a feed that failed to appear would have alerted, but not for record volume within a feed that did appear. The interface was monitored for whether it ran, not for whether what it delivered was the size it should have been. Volume-band alerting against a trailing average was added for all four batch interfaces, and this is the class of defect the Steering Committee should assume exists elsewhere and has not yet been found.
6. IF-14 and the interface that should not exist
Thirteen of these interfaces move data out of the retiring platform. IF-14 moves data back into it.
It exists because some functions continued to run on the Cumberland Valley platform after Day 1 while their replacements were being built, and those functions need to see the results of work performed in the target system. The Integration Architecture treats the whole layer as temporary, built to be deleted once the retirement completes. IF-14 is the point where that argument is under most strain.
A write-back interface converts the system being retired into a system that must keep working, and it does so quietly. Every other interface makes the retiring platform less necessary. This one makes it more so: as long as something writes into it, whatever reads from it stays live, and each consumer that appears is a new blocker on a retirement that was supposed to be a decommissioning exercise rather than a negotiation. The controlled position is that IF-14 has a closed register of consumers and no new ones are added. Two requests to extend it have been refused since Day 1 and both refusals were recorded, because the pattern the register exists to prevent is one in which each individual extension is reasonable and the aggregate is a platform that can never be switched off. The register currently holds four consumers, down from seven.
IF-14 retires when the last of those four is migrated. That is a dependency the interface workstream does not own and cannot accelerate, and it is reported as a retirement blocker rather than as an interface defect — because the interface is working exactly as built, which is the problem.
7. Retirement dependencies
Three interfaces remain load-bearing against the transition rather than against a permanent design.
- IF-14 retires with its fourth consumer. Owner: K. Stallworth, with the migration dates held by the receiving workstreams rather than by integration.
- IF-02, the enterprise member identifier broadcast, is carrying materially more traffic than modeled. The identity work resolved 41% of records deterministically and a further 33% automatically, leaving a clerical review band of 26% — a queue of 109,200 records worked by stewards over months. Every steward resolution is an identifier change, and every identifier change is a broadcast. The interface is coping; the point is that its volume is now tied to the pace of a queue rather than to enrollment activity, and it cannot be sized down until that queue clears.
- IF-09, the care management member feed, serves a preserved capability rather than a migrating one. It has no retirement date because the capability it serves is not being retired, which makes it the one interface in this log that is permanent. It should be moved out of the integration layer's temporary architecture and onto a supported footing before the layer is decommissioned around it.
IF-09 is the trap in this list and it is easy to miss on a status report. It appears in every review as green, cut over, no defects — indistinguishable from the thirteen interfaces that are temporary. When the integration layer is decommissioned, twelve of these disappear with it by design, IF-14 blocks until its consumers move, and IF-09 breaks a capability the program deliberately chose to keep. An interface that is working is not the same as an interface that has somewhere to live afterward, and nothing in the interface register distinguishes the two.
Related artifacts: 25 — Integration Architecture · BRD-03 — Integration Layer and Interfaces · BRD-04 — Enterprise Data Warehouse · 26 — Quality Plan