The Great Flood of '05

Early Monday morning (December 5th), a sprinkler system water main burst in a closet in the back of the CMCL machine room which is located on the 3rd floor of Wean Hall. It was discovered when a knocking was heard coming from the inside of the closet door. The noise was caused by debris floating in several feet of water and banging against the door. Of course, no one knew the main had broken, so the door was opened to investigate.

A gush of water poured forth, destroying most of the CMCL machine room. This area has raised flooring and the water quickly filled the subfloor space and spread to the CMCL lab next door, out into the hallway, and across he hall into the SCS Operations Room, Helpdesk and the SCS machine room. SCS Operations staff quickly shut down all the servers in the SCS machine room, since all of the power feeds to the server racks run through the subfloor. Then power was shut down to that entire wing.

The water main was shut off, but because most of the steam tunnels on campus are around the level of the 3rd floor of Wean, and Wean is one of the low points on campus, we had to wait until all of campus drained out through our little corner of the world before the plumbers could repair the pipe. The water level made it up to the very top of the SCS Machine Room subfloor, but fortunately did not get high enough to crest the raised floor panels. A fire and water damage contractor was called in to rip out all the damp, stinky carpets and dry out the subfloor.

After things settled down a bit emergency power feeds were run overhead from the breaker panels to the racks that housed all of our core systems (network, DNS, NTP, DHCP, Kerberos, AFS, mail, web, zephyr, etc) and things were carefully powered back up. Many SCS Facilities staffers worked well past midnight to bring basic core services up before the workday started on Tuesday.