The Server Room + an air-con with yearly mood swings

[redbeacon]

Geographic Location: North America

Setting: Server Room in a University. Outside this institution, this Uni faces 4 seasons. Cool Spring, sweltering-heat Summers, cool Fall, and frigid Winters.

 

Problem: Server Room is locked tight. Air-conditioner inside the Server Room failed one day as our hero Greg (not real name) gets employed as the Uni's IT Guy. He unravels the mystery of the on-time on-the-same-date failure of the air-conditioning unit every year...

 

Enjoy the rest of the story folks...

 

================

As our hero Greg saw the temperature readings rise on his computer screen, his body temperature did as well. Sweat dropped.

 

Several calls, and frantic finger-tapping-on-table later, somebody knocked on the door to the IT room.

 

"You're the I.T. guy right?" a guy in a jumpsuit with the words "We'll fix your air-conditioner right" stepped in.

 

Our hero went on to explain that he was receiving increasingly severe temperature alarms from the Server Room. First 86°F, then 93°F, then 97°F, and – just then – 103°F. He figured that the dedicated air conditioning unit had died. Normally, that wouldn’t be a problem, but he had no way to get in to fix it. The Server Room – which housed all of the university’s critical research, financial, and operational servers – was locked.

 

None of the elite group of key holders could be contacted to open the door. Fortunately, Greg was able to "unlock" it with his trusty PILOT-brand pen and let the maintenance guy in to fix the problem.

 

"All the servers, all the research grants," Greg thought while basking in his PILOT-brand pen-breaking-and-entering skills, "I could take them, they’d be mine, mine, mine!"

 

As Greg briefly wondered whether he should upgrade to burglary, the Maintenance Guy hastily worked to fan out the hot air from the room generated from the gradually overheating servers. A few minutes later, with the aide of a giant floor fan, the temperature dropped to a much more suitable level.

 

While Greg worked on rebooting the overheated servers, the Maintenance Guy inspected the nonfunctioning A/C unit.

 

“Aww crap,” he said after cycling and tripping the breakers a few times, “I bet it’s those damn tree leaves again cloggin’ up the compressor unit.” He told Greg that he’d have to go up on the roof and inspect things from there.

 

One thing to keep in mind about this certain university, is that it’s located within in a certain geography known for its super quick transitions from Winter to Spring. For example, on April 5th, there’s snow on the ground and the thermometer reads a brisk -6°C. The very next day on April 6th, the temperature jumps to a cool 21°C and everyone’s walking around in shorts.

 

And it’s usually on this day that the dedicated A/C unit decides to go dead.

 

Just as Greg was finishing rebooting the servers, the Maintenance Guy returned to the IT room from the roof.

 

“Yup,” he said casually, “it’s the leaves again. The whole A/C unit’s dead until we can get it cleaned out.” It took a day or so of A/C and server repairs, and the datacenter was back up and running.

 

The Spring months passed by and then, all of a sudden, it was Summer. The A/C did its work, cooling through the blistering heat and all the way throughout the Fall.

 

Sometime in late October during the Fall months, as expected, the A/C automatically shut down and entered its several-month hibernation. That way, the University can cut costs because the cold air from the frigid Winter coldness can effectively cool the server room.

 

And then, all of a sudden, April 6th hit again. A Maintenance Guy rushed to the building in response to a temperature alarm, frantically ran around trying to find someone to open the door to the so darn secure Server Room, and eventually ended up on the roof with the same diagnosis:

 

“the A/C is clogged up with leaves, again.” Greg said that as he shook his head.

 

After a panicked day or two of no cool, the A/C and servers were patched up and working again.

 

Inquiring further with the Maintenance Guy and the datacenter administrators responsible for the ownership of the Server Room, Greg the day-to-day I.T. guy learned that this whole routine had become a bit of a tradition at the university:

 

On or about April 6th – whenever that first blast of Spring weather hit – the A/C system would fire up and then quickly grind to a halt due to leaves. It had been going on for as long as anyone could remember, and everyone had assumed that air conditioners just work that way.

 

As Greg was leaving the building that day, he glanced up at the roofline. For the first time, he noticed that, when that old University was remodeled, the contractors added an architecturally-compatible brick cooling tower for the air conditioner. And perched directly above the cooling tower was a majestic, 200-year-old ash tree. It was perfectly positioned so that, every year, at least half of the tree’s leaves would fall directly into the cooling tower when Spring came on April the 6th...

 

It was then Greg figured out:

 

The unaware datacenter admins responsible for the Server room or A/C repair folks actually had missed the painfully obvious fact that a quick brushing-off the leaves *once* every year before starting the air-con would save thousands of dollars in equipment repair, a bunch of man hours, and whole lot of lost productivity time.

 

Greg then sent a friendly email to the maintenance and datacenter managers. All he got was “Thanks, but we know what we’re doing.”

 

Like clockwork, on or about the following April 6th, A/C unit died from clogged leaves again. Our dear hero Greg then decided to leave that job shortly thereafter, but is pretty sure the tradition continues to this day…

================

 

Sooo...SG admins.

This is sunny Singapore; I don't think you're spared as well.

Do you have your air-conditioners dustied-out periodically before they even fail?

[wait4me]

management thanks eh ...

 

coz its not them that rushed back when things go wrong ..

jus > " i will need a report on what happen ." to the guy that came back

[WildCard]

Normally, any good DC or even smaller server rooms would have a proper maintenance contract to service and maintain the facility equipment.

 

Now let me share a little anectode on some of the funny configs you can find in DCs & server rooms that belong to... oops, the powers that be:

 

----------------------------------------------------------

Normal compressor aircon (those residential home aircons) instead of a proper climate control & dehumidifier DC A/C units being used in a server room.

 

Stacking of all production servers into one rack and all other development & testing servers onto other racks. Sounds pretty reasonable? Not when you have over 15 production servers, each one as critical as the other. You have 6 racks with more than enough space to spread all of them with spare space and its all gotta be stacked into one rack.

 

An older server using RAID 5. Pretty normal, right? Problem is it has only one (1) harddisk. Figure that one out...

[jiasheng]

how do you raid 5 with 1 hdd?

[redbeacon]

An older server using RAID 5. Pretty normal, right? Problem is it has only one (1) harddisk. Figure that one out...

 

I'd say it might be a pseudo "RAID 5" setup.

Ya kno...not really that RAID 5, i guess.

 

Other than partitioning + tricking the OS into thinking it's multiple HDDs (with controllers each) that it's handling...

I dunno, i'm outta guesses.

[kmax]

Data Centers are really a joke here...

ask permission from one, then forward to another till end of the year..

 

i should try "pilot pen" method..

 

i heard one of important DC got power surge ???

[jiasheng]

yea... and in the terms and conditions it idenmify itself against damages brought about by power failure.. wtf....

[WildCard]

Its a SCSI RAID HDD, rite?

RAID does not care how many physical HDDs. It cares about total physical size, rite?

 

So there was this smart aleck who came out with this ingenious idea to partition a single large drive (sumting like 160GB) into several partitions and set the controller to RAID 5 them.

I have no idea how it was done. But when my managed service team came in to start taking over the systems from them, we found it like dat.

[jiasheng]

omg... then its no longer RAID liao.. its jsut R, redundant.. -___-;

[redbeacon]

In this case, it's...

 

RAID = Redundant Array from an Inexpensive Disk

 

("-_-)

 

 

** Note that it's "disk", and it's not in its plural form.

[WildCard]

omg... then its no longer RAID liao.. its jsut R, redundant.. -___-;

 

Bingo!

You figured it out.

[jiasheng]

lol.. that approach just does 1 thing. waste your bloody expensive scsi hdd.