The $250K-Per-Outage Risk Hiding in the AI Buildout
The AI gold rush is now also a plumbing project. Every direct-to-chip cooled rack, every rear-door heat exchanger, every immersion tank quietly turns a data hall into something the industry has spent forty years working hard to keep dry: a room with pressurized water running above and around millions of dollars of compute. The economics still work — but only if the water stays inside the pipes.
Most of the conversation around liquid cooling focuses on PUE, sustainability, and chip density. The less glamorous risk is also the most immediate: a single failed fitting, a CDU pump seal, a contractor's wrench in the wrong place, and you have standing water in a room where standing water was never supposed to exist. The Uptime Institute's 2024 survey found roughly one in three data center outages now cost more than $250,000, with many topping $1 million. Cooling problems specifically account for 19% of outages — and that share is rising as facilities transition from air to liquid.
What changed: from 8 kW air to 130 kW liquid in five years
Five years ago, a 10 kW rack was considered high-density and a 15 kW rack required negotiation. Today, NVIDIA's GB200 NVL72 reference design pulls roughly 120 kW per rack, and roadmaps for the next generation push past 150 kW. Air simply cannot move enough heat to keep up. The Global Data Center Hub frames it bluntly: "Once racks cross roughly the 15–25 kW threshold where conventional air approaches start to lose efficiency, investors confront a step-change decision: either rebuild the mechanical concept, or accept that the facility's highest-return tenant class will lease elsewhere."
The market has answered. Dell'Oro now projects the data center liquid cooling market to approach $7 billion by 2029, up sharply from earlier forecasts. Roughly 75–90% of data centers worldwide already rely on water-based cooling somewhere in their stack — chilled water loops to CRAHs, evaporative cooling towers, and now increasingly to the rack itself. Liquid is no longer the exception. It is the architecture.
"In the AI era, the scarce input is effective heat rejection at density, under real-world constraints of water, permitting, and reliability. Cooling failures can trigger downtime with SLA penalties, reputational damage, and churn."
— Global Data Center Hub, January 2026
What can go wrong: three recent incidents
This is not theoretical. Three publicly documented incidents in the last 24 months show the shape of the failure mode.
Southeast Asia, November 2025 — GPU cluster wiped out. Liquid cooling pipes burst in a high-density hall, taking out roughly a dozen racks of GPU servers. In a widely-shared post-incident account, employees were photographed squeegeeing water down an aisle while operators raced to shut down the affected zone. At a few million dollars per GPU rack, even partial damage to one rack is an unrecoverable seven-figure loss. Industry observers noted that most production designs still run "single-circuit hot/cold pipe systems" — meaning one leak and the entire IT load on that loop has to stop.
CME Group, November 2025 — global trading disrupted. A cooling system malfunction at a U.S. data center serving CME Group, the largest exchange operator in the world, disrupted financial trading technology. CME has since installed additional cooling capacity specifically to harden against a repeat. The incident did not require a leak — just thermal excursion — to land in the BBC and trigger a capital project.
Equinix Singapore, October 2023 — 2.5 million failed bank transactions. During a planned upgrade, a contractor sent the wrong signal to valves on chilled-water buffer tanks. Sections of the hall heated up; DBS Bank and Citibank lost critical services for roughly two days, resulting in 2.5 million failed payment and ATM transactions and 810,000 failed digital-banking sessions. The Monetary Authority of Singapore later imposed a 1.8x multiplier on DBS's operational-risk capital — roughly S$1.6 billion in additional regulatory capital. One valve. One mis-sent signal.
"When liquid cooling leaks, the only thing flowing is panic. More than one liquid cooling pipe had burst, affecting a dozen racks. At a few million USD per rack, damage to even one would be an unmitigated disaster."
— Paul Mah, post-incident analysis, November 2025
Why water in a server room is uniquely expensive
A puddle in a warehouse is a mop and an insurance claim. A puddle in a high-density compute hall is a different category of event, for three reasons.
Density of capital at floor level. Modern liquid-cooled racks routinely cost $2–5 million fully populated. A 100-foot row at the new generation of densities can carry $50–100 million in capital. Water on the floor of that aisle is not a cleanup problem; it is a balance-sheet event.
Conductivity-adjacent risk. Even non-conductive coolants like PG25 (a propylene glycol mix) eventually break down, pick up contaminants, and become more conductive over time. The ASHRAE Liquid Cooling Technical Committee has been explicit that "operators express concerns about potential liquid leakage risks near sensitive IT equipment, despite advanced leak detection systems and use of non-conductive coolants." The risk isn't zero; it's residual, and it grows with system age.
Subfloor and adjacency damage. Most data halls still sit on raised floors over cable trays, power distribution units, and chilled-water mains. A leak above is rare. A leak that runs under the floor, into the adjacent electrical room or the row of UPSes downstream, can take out infrastructure that was never directly wetted. Standard water-damage restoration practice notes that water typically travels to the lowest accessible point and lingers under flooring for weeks — not the timeline a Tier III SLA can absorb.
And the leak is not the only water source. External flooding from a major storm event is a perennial threat to facilities sited near coastlines, riverbanks, or in low-lying industrial parks. Hurricane Helene's flash floods in September 2024 caused more than $59 billion in damage across North Carolina, including roughly $3 billion in damage to business property and equipment — a footprint that included several colocation and edge facilities in the southeast. Datacenter site selection has historically prioritized fiber and power; flood risk is climbing the list.
The standard response — and where it leaves a gap
Mature liquid-cooled facilities lean on a stack of mitigations: leak-detection sensors woven through under-rack and overhead piping, segmented cooling loops with shutoff valves, redundant CDUs, dry-break quick-disconnects on every rack manifold, and humidity sensors that flag a leak indirectly via dew point. ASHRAE TC 9.9's fifth edition formalizes much of this guidance, and the better operators run leak-detection telemetry into the same DCIM dashboards as thermal data.
What that stack is good at: detecting the leak fast. Detection is now routinely sub-60-seconds in modern halls. What it does not do is stop the water that has already escaped from reaching the next rack, the next aisle, or the subfloor cable tray. Between the moment a sensor trips and the moment a human in a control room actually closes the right valve, water still moves. On a polished concrete floor, water moves quickly. A burst CDU return line at 25–50 GPM can put hundreds of gallons on the floor before anyone gets to the affected zone.
That gap — between detection and containment — is where most of the incident-cost actually lives.
StormBag in the data hall: a containment layer, not a replacement
StormBag's role in a high-density facility is narrow but specific: a deployable, pre-staged water containment layer that turns a "water everywhere" event into a "water in one aisle" event. The bags are inert, dry-stored, and expand on contact with fresh water to a 17"-tall barrier within minutes. They are not a substitute for leak detection, segmented loops, or a properly designed CDU; they are the last physical line between a leak and the rest of the room.
Practical placement patterns we have seen work in real facilities:
- Perimeter rings around critical pods. A pre-positioned stack of StormBags at the entrance of each pod or aisle, ready to deploy as a curb if a sensor trips. Even a single 8" rise diverts flow back toward floor drains and away from adjacent pods.
- Subfloor and trench protection. Floor cutouts for cabling and chilled-water mains are often the path of least resistance for runaway water. Pre-positioned bags around known cutouts buy minutes during which staff can stage proper containment.
- UPS and switchgear room thresholds. The electrical room next to the data hall is usually where a "leak" becomes a "compound failure." A threshold barrier at every doorway, deployable in under five minutes, is cheap insurance against cascading outages.
- Loading dock and external entry points. For sites with flood exposure, perimeter staging at dock doors and any below-grade entry. StormBag cannot be hydrated with salt water, but once hydrated with fresh water will repel salt water — important for coastal sites with seawater exposure.
"Such occurrences are very, very rare — as are data center fires. Piping, tube, and interconnection components have been perfected. When there is an issue, 95% of the time it's related to human error, inexperience, someone cutting corners, a squirrel, or a rat."
— Joe Capes, on Paul Mah's post
The Capes point is worth dwelling on. The failure mode for liquid cooling is rarely a designed-in defect; it is a contractor, a maintenance event, a rodent. That is exactly the failure mode against which pre-positioned passive containment is most useful, because there is no engineering team in the loop at the moment the water starts moving.
The math: containment cost vs. incident cost
Numbers are useful here. A typical AI data center pod might be 20–40 racks. Outfitting that pod with perimeter StormBag containment and threshold barriers at every door and cable cutout runs roughly $3,000–8,000 in product, dry-stored, with a five-year shelf life in unopened cases. The same pod, if a CDU return line fails and water reaches even one rack of GPU servers, can lose $2 million in hardware in the first ten minutes — before counting SLA penalties, business interruption, or the regulatory follow-up that incidents like the Equinix Singapore event have made standard in financial-services colocation.
That is a roughly 250-to-1 ratio on the cheapest possible damage scenario. The ratio gets worse as rack density goes up. It is hard to find any other capital-protection layer that pencils that aggressively.
Quick-reference: where to stage StormBags in a liquid-cooled hall
| Location | Threat addressed | Recommended deploy |
|---|---|---|
| Aisle ends, per pod | CDU / manifold leak | Pre-staged 25-pack at each aisle |
| Cable cutouts / floor penetrations | Water tracking to subfloor | Single-bag ring around each cutout |
| Electrical / UPS room doorways | Cross-room cascade | Pre-staged threshold pack |
| Loading dock doors | External flood / storm | Perimeter ring, seasonal |
| Generator yard / fuel berm | Standing water near switchgear | Pre-staged at low points |
What this looks like in practice
For a facilities team evaluating this, the practical path is small and measurable. Start with one pod. Map the actual physical risk surface — where the CDU lines run, where the cable cutouts are, where the electrical room sits relative to the racks. Stage StormBags at the three or four highest-leverage points. Add the deployment sequence to the existing leak-response runbook so the on-call security or NOC team knows where the cases are and what to do in the first ninety seconds.
This is not a product that changes how a data center is engineered. It is a passive containment layer that closes the gap between "the sensor tripped" and "the right human closed the right valve." In a world where that gap can cost more than a million dollars per minute, the case for closing it is straightforward.
A closing thought
Liquid cooling is not retreating. Dell'Oro's $7 billion by 2029 projection assumes the opposite — that every meaningful AI buildout from here forward is liquid-first. The leak risk is structural to that transition, and the industry's instinct, correctly, has been to engineer around it with sensors, redundancy, and better materials.
The cheapest mitigation in the stack, though, is also the oldest one: a physical barrier between water and the equipment that water can destroy. The math has not changed since the Romans built aqueducts. What has changed is that the equipment behind the barrier is now worth a few million dollars per rack.
For facilities and operations teams
If you are responsible for a liquid-cooled facility — colocation, hyperscale, edge, or enterprise — StormBag offers volume pricing, palletized dry storage, and pre-staging consulting for any site over 5 MW. Our team has worked with operators across financial-services colocation, AI compute, and government high-availability environments. Review specifications and order on stormbag.co, or reach out via the contact form for facility-specific staging recommendations.
This post is part of StormBag's ongoing reporting on flood and water-damage mitigation. For real-time flood alerts at any U.S. address, see our free Flood Watch tool.
For the insurance and risk-management view of the same problem — what FM Global, Swiss Re and Bracewell are saying about coverage gaps in liquid-cooled facilities — see our companion piece, The Insurance Industry Has a Liquid Cooling Problem.
Selected sources
- Uptime Institute — Data Center Outages are Common, Costly, and Preventable
- Dell'Oro Group — Liquid Cooling Market to Approach $7B by 2029
- BBC News — Data centres: The new tech stopping chips from overheating
- Paul Mah — Southeast Asia GPU cluster leak post-incident analysis (Nov 2025)
- Envigilance — ASHRAE TC 9.9 Critical Data Center Thermal Guide (Equinix Singapore case study)
- Global Data Center Hub — The AI Data Center Crisis No One Is Talking About
- FWPCOA — Data Centers and Water Usage
- Global Market Insights — Data Center Liquid Cooling Market Report 2035
