Oh No!! - I've been flooded!!!

Sorry for the provocative title, but I thought I'd cash in on the recent south-east Queensland flood crisis which seemed to dominate international news headlines just a couple of weeks ago. I live in Brisbane, which has been quite devastated (but with plenty of forewarning), but not nearly as dramatically so as a group of country towns located around an hour or two's drive west of Brisbane. Their problem was the "wall of water" (aptly described as an inland tsunami) which swept everything in it's path (including many complete houses) with virtually no forewarning.

In Brisbane, we had the experience of previous major riverine floods in 1974 and 1893 to tell us that the Brisbane river meanders through a flood plain and that if you build with the superb river and tributary views in mind, you are taking something of a major risk of periodic inundation. The 1974 flood should have been a wake-up call of what could happen.

People had been lulled into a false sense of security after 1974, because a very large dam had been built in one of the upper Brisbane river catchments which had two purposes - water storage and flood mitigation. Under normal circumstances the dam is limited to 100% capacity. Just a couple of years ago it was at 16% capacity at the height of a protracted drought. At the time of a flood event, it is designed to be able to go as high as 225% capacity. This extra 'flood capacity' was supposed to prvent another 1974 style flood by allowing a large extra volume of water to be temporarily stored. Obviously nobody decided to explain the rules to mother nature. During the leadup to the recent crisis, the dam was already overfull and the ground already saturated from previous weeks of abnormal rainfall. The engineers were actually desperately trying to keep the dam level under control by sustained releases of water and then the big rain dump came out of the blue. The peak dam level did actually get to about 216% (from memory) and there were considerable water releases going on since the engineers really didn't want to fully test that upper design limit. So the upshot of all this was that there was little ability to store more water when it was most needed. The flood didn't quite make it to the 1974 levels so the dam probably did have some mitigating effect but that's little consolation to the thousands of home owners who did get inundated.

Having lived through the 1974 event, and having experienced the desolation first hand (helping clean up inundated houses of friends) I have always made sure I wouldn't get caught in a repeat dose. My home is well and truly high and dry but I do own business premises quite close to the Brisbane river in South Brisbane. In 1974, there was water in the street outside these premises but the actual ground floor level of the building is about 1m above street level so when I bought the property I figured I'd be relatively safe.

When the recent event was unfolding, there were warnings that it could be 'worse than `74' so I did monitor the situation rather closely. A part of my 'farm' was in that building but when I watched what the river was actually doing in the leadup to the flood peak, it was quite obvious that the peak would actually be less than that of '74 - so I'd be OK. That turned out to be true so my hosts crunched right through the flood peak. However, the next day they cut off the power to the whole area for the next three days - apparently there was some basement high voltage 'assets' that were 'at risk' until the water had completely receded.

Another part of the farm is at my house - no power loss there so those machines were quite OK - until my own personal flood event occurred :-). I'm using a spare bedroom at the back of the house as a 'computer room'. I've got 26 hosts running there, mainly Q6600 and Q8400 quads. They generate quite a bit of heat so I have them strategically located adjacent to my high tech cooling solution - open windows :-). My wife is a keen gardener so there are lots of protective shrubs which offer quite good protection against even blowing rain - so open windows aren't usually a problem.

To cut a long story short, my wife managed to position a hose in just the exactly correct location that part of the stream was being deflected by something in the bushes straight into the open window. The hose wasn't pointing anywhere near the direction of the window and she was quite unaware of what was happening. She was chatting to a neighbour and shortly afterwards she turned the hose off and came inside to prepare dinner.

At that point she had lights but she couldn't get any power outlets to work. So she phoned me. I muttered something about 'those damn safety switches again' and I told her I'd come home shortly and reset everything.

When I got home I went into the 'computer room' and started to switch off all the PSUs so I would be able to reset the safety switches without anything tripping out immediately. Then I noticed it. There was water glistening on all the keycaps of a keyboard sitting on the desk. There was water all over the desktop. The carpet on the floor was actually squelching!! There were pools of water sitting on top of PSUs in the stacks of open computer cases. I could even see water sitting on top of CMOS batteries!! If I picked up one of the machines, there was water swilling around in the base of the case. This whole nightmare flashed through my mind and I remember being completely bamboozled by the source of the water. there were clear skies outside so where had the sudden storm come from??

My wife was completely oblivious to this 'disaster' so she thought I must be really losing my marbles to be asking whether we had a storm or not :-). It was only then that we worked out what must have happened.

That night I had a bucket of water after tipping out all the cases. Out of the 26 hosts, there were 10 that didn't have any obvious signs of water entry. The other 16 were spread out in the living room to drain with a couple of fans to help with the drying out. I managed to get the 10 'dry' machines back working before I went to bed late that night.

The next day I started working, one by one, on the others, thinking that most might need new motherboards and probably PSUs and maybe even other components. When you see water sitting on top of CMOS batteries, it doesn't really engender much confidence that all will be well.

The amazing thing is that all 26 machines are now back working, with very little actual hardware damage. I needed to replace just 4 PSUs and 2 motherboards. Some 'resurrections' were a bit of a struggle :-). The old trick of taking the battery out for >24 hours worked wonders in quite a few cases - seemingly dead boards being coaxed back to life. It looks like some PSUs may have taken out the safety switches early enough before the bulk of the water accumulated inside the cases and on the motherboards. I haven't actually opened up any of the PSUs yet. Maybe they might even be repairable :-).