Every degradation, written down.

Summary

Between 11:14 and 15:26 NZST, AOTEA‑10T returned 503 portfolio_unavailable for all requests touching both the Treasury and Social Development portfolios. Two minister models entered a mutual deference loop over a single spending item and neither yielded. The orchestrator has no timeout on deference depth, so the loop ran until the request pool was exhausted.

No determination was issued incorrectly. Every affected request failed closed, which is the designed behaviour and the only reason this is a SEV‑1 rather than a SEV‑0.

Impact

Approximately 14,200 requests failed over the window, of which 11,900 were retried successfully after resolution. 2,300 citizens received an error rather than a determination and were notified by email the following morning. A live coalition negotiation was suspended for the duration and resumed by the human participants without model support, which is the fallback the Cabinet Manual describes and the first occasion on which it was used.

Timeline

• 11:14ŌHANGA receives a determination request on a hardship grant top-up and routes it to MANAAKI, assessing it as a social support matter rather than a fiscal one.
• 11:14MANAAKI routes it back to ŌHANGA, assessing it as a fiscal matter. Deference depth begins incrementing. No ceiling is configured.
• 11:21Request pool saturation reaches 60%. No alert exists for this metric in isolation.
• 11:38First user-visible 503s. Availability monitoring records the drop but does not page, as the threshold is a five-minute rolling average and recovery blips had been common that week.
• 11:52A member of the public in Ōtautahi posts to the status dashboard that the entitlements endpoint has been failing for fourteen minutes.
• 12:33Internal monitoring pages the on-call engineer, 41 minutes after the public report.
• 13:05Root cause identified from the deference trace. Manual tie-break applied by the on-call engineer under delegated authority.
• 15:26Request pool drained and full service restored. Retry of queued determinations completes at 15:58.

Root cause

Portfolio deference was implemented as an unbounded recursive call. The design assumed that any request has exactly one correct owner and that each model can identify it. A hardship grant funded from a contingency line satisfies both models’ ownership tests and neither model’s exclusion test, and no component was responsible for noticing that the question had been asked twice.

What went wrong

• Deference depth was not instrumented, so a loop was indistinguishable from ordinary load.
• The availability alert averaged over five minutes, which is longer than a citizen’s patience and considerably longer than the party’s stated latency commitments.
• The public found this before the operators did. That is a monitoring failure and it is recorded as one.

Action items

Summary

Two matching workers held stale vacancy state for the same Papakura dwelling inside a single 22 ms window. Both cleared threshold. Both issued an allocation notice. One household had already given notice on their existing tenancy on the strength of that notice.

This is the most serious incident the housing programme has recorded, and it is the one the party is asked about least often, which is the wrong ratio.

Impact

One household moved into the dwelling. The second household, who had given 21 days’ notice on a private tenancy, was housed in temporary accommodation for six nights and then permanently housed in a comparable dwelling 4.1 km from their original match. The Crown met all costs, including the private landlord’s re-letting fee.

The party apologised in writing and in person. No part of that apology was drafted by a model, and the notice that caused the harm was not shown to a human before it was sent, which is the failure this postmortem is actually about.

Timeline

• 02:14:07.118Worker A reads vacancy state for dwelling PAP‑4471 and begins scoring.
• 02:14:07.131Worker B reads the same state. Worker A has not yet written its claim. Both records are valid at read time.
• 02:14:07.140Both workers clear threshold against different households. Both write. The second write succeeds silently.
• 02:14:09Two allocation notices are dispatched. No human confirmation step exists.
• 09:20Household B telephones to arrange collection of keys and is told the dwelling is occupied. This is the first anyone knows of the fault.
• 09:44Allocation writes suspended nationally. 31 pending notices held for manual review; all 31 were correct.
• 11:30Household B placed in temporary accommodation. Case assigned to a named caseworker, not a queue.
• 9 Feb, 14:00Household B permanently housed. Incident closed on the operational side; the postmortem remains open for a further eleven days.

Root cause

Dwelling records had no optimistic locking. The matching layer was designed around the assumption that vacancies are rare relative to throughput and that two workers would not contend for the same record. At 44,000 matches per quarter across 2,900 returning vacancies, that assumption was wrong by roughly the margin one would expect from never having tested it.

What went wrong

• No lock, no version check, no idempotency key on the allocation write.
• The allocation notice — an irreversible communication that people act on immediately — was classified as an output rather than an action, and so fell outside the human review boundary.
• Nothing in the system noticed. A person had to ring up.

Action items

Summary

During Prompt Time, MANA answered a question on council amalgamation by citing section 41B of the Local Government Act 2002. No such section exists. The real-time fact-check flagged the citation at 900 ms, the response was retracted on the chamber screens before the member had resumed their seat, and a correction was read into Hansard at 15:40 the same day.

The system worked as designed. It is a SEV‑2 anyway, because the design permitted an unverifiable citation to be spoken aloud in the House at all.

Timeline

• 23 Feb, 23:58The Local Government (Amalgamation Procedure) Amendment Act passes under urgency. The legislation index begins reindexing.
• 00:14Reindexing fails on a malformed schedule reference. The failure is logged at WARN and the replica continues serving the previous index.
• 14:22Prompt Time begins. MANA serves citations from the stale replica without any freshness assertion.
• 14:31:04.0MANA generates a citation to s41B, interpolating a plausible section number from an amendment it had partial knowledge of.
• 14:31:04.9Fact-check fails to resolve the citation against the live index. Hallucination score 0.31. Response retracted on screen.
• 15:40Correction read into Hansard by the countersigning member.
• 16:10Citations gated on index freshness across all portfolios. Prompt Time resumes.

Root cause

A failed reindex degraded to serving stale data rather than to refusing to serve. The replica had no way to express “I am 14 hours behind” and no consumer asked. The model then did what models do at the edge of their knowledge, and produced a section number that was structurally correct and factually invented.

What went well

Detection, retraction and correction took under ninety minutes end to end, of which 900 milliseconds was the technical component and the remainder was a person walking to the House. The public record was corrected the same sitting day. Under the Human Governance Baseline, the median time to correct an incorrect statutory reference made in the House is 4 to 6 weeks, and roughly a third are never corrected at all.

Action items

Summary

Manapōuri inflows fell below the trigger in the demand-response agreement. The sovereign compute campus shed 62% of load within four minutes, as contracted. Health triage and education tutoring ran at full capacity throughout. Climate simulation degraded to a reduced ensemble. Coalition simulation and the Sheep Happiness Index were suspended for nine hours.

Every technical control behaved correctly. The incident is filed as a SEV‑2 for a governance reason rather than a technical one.

Impact

No household or industrial consumer experienced reduced supply, and no firm supply was purchased by the campus at any point during the event. Climate ensemble members were reduced from 200 to 48, widening the confidence interval on the seasonal outlook published that week; the wider interval was published, with a note explaining why.

Timeline

• 04:02Inflow forecast crosses the dry-year trigger. Demand-response notice issued automatically to the campus.
• 04:05Load shedding begins in the published priority order. Health and education workloads pinned.
• 04:0662% of load shed. Campus draw falls to 38%.
• 06:30A journalist asks which workloads were dropped first. Operations cannot point to a published document, because the priority order exists only in the runbook.
• 13:11Inflows recover. Load restored progressively over 40 minutes to avoid a demand step.
• 13 MarLoad-shedding priority order published in full, with a commitment that any change requires 30 days’ notice.

Root cause

None technically. The governance fault is that the order in which public compute is sacrificed — a decision about whose service degrades first when the country is short of water — had been made correctly by engineers and never put in front of anybody else. A correct decision taken privately is still a decision taken privately.

Action items

Summary

The entitlements endpoint was unavailable for 2 hours and 48 minutes. Throughout, the public status dashboard displayed Democracy status: ONLINE and a rolling availability figure of 100.00%.

The availability metric was computed by a service running inside the cluster it was measuring. When the cluster stopped serving requests, it also stopped serving the failures, and an absence of recorded failures was treated as an absence of failures.

Impact

Approximately 6,100 citizens could not query their entitlements. The larger harm is to the credibility of every other number on that dashboard, which the party has spent the campaign asking people to trust. That harm is not measurable and is not claimed to have been resolved by the action items below.

Timeline

• 19:41A configuration push drops the entitlements route. Requests begin failing at the edge, before any in-cluster telemetry.
• 19:41Failure count remains zero, because failures are counted in-cluster and no request now reaches the cluster. Availability continues to compute as 100.00%.
• 19:55First public reports. The status page continues to display ONLINE, and several people report the outage a second time assuming their first report was not received.
• 21:02On-call engineer paged by a colleague who could not check their own entitlements from home.
• 22:29Route restored. Availability metric never registered the incident and had to be corrected by hand.

Root cause

The observer was inside the system it observed. This is a well-documented failure mode and the platform documented it too, in the reliability whitepaper, sixteen months before doing it.

What went wrong

• Availability was computed from failures the system recorded rather than from requests the public made.
• There was no external prober, and the party had published a commitment to run one.
• Public reports had no path to the on-call rotation. Fourteen people told the dashboard, and the dashboard told nobody.

Action items

Summary

A catchment authority updated its nitrate limits, publishing them in g/m³. TAIAO ingested the values as mg/L. The two are numerically identical, and the ingestion layer converted between them anyway, dividing every limit by one thousand. Every application in the affected catchments was then measured against a limit a thousand times stricter than the real one, and declined.

Impact

1,412 provisional declines were issued over forty minutes, each with reasoning that was internally consistent and entirely wrong. All were reinstated within 26 hours and every applicant was contacted directly. No applicant incurred a cost, and none had reached the appeal stage, largely because forty minutes is not long enough to file one.

Timeline

• 08:00Catchment limits ingested. Conversion applied to units that required no conversion.
• 08:02Consent assessment resumes against the corrupted limits. Decline rate rises from 6% to 94%.
• 08:31Anomaly monitor pages on decline-rate deviation. This alert had been added six weeks earlier following a near-miss.
• 08:40Assessment halted. Every determination issued since 08:00 quarantined.
• 29 Apr, 10:15All 1,412 applications reassessed against correct limits and reinstated. Applicants notified individually with an explanation of the fault.

Root cause

The ingestion schema carried no unit annotation. A conversion table inferred units from the source authority rather than from the data, and this authority had changed its published unit convention in 2024 without changing its identifier.

The deeper fault is that a 94% decline rate is self-evidently absurd to any person who has ever seen a consent register, and the system ran for 31 minutes without asking whether the world had changed or whether it had.

Action items