Yes the Server is Down !!!~!~!~!ELVENTY

[Daskinor]

Excuse me I just need to vent!


The server from Hell

Good old ancient VAX server, you are essential to the daily lives of many in this building. Many of the managers revere you as a god, but also fear you as a vengeful god. Talk of replacing you was met with fear and disbelief, how dare we utter such blasphemy.

But like all false gods, they do not stand the test of immortality. A massive building wide power outage exhausted your batteries. Generators that are supposed to be tested every 3 months failed to turn on. And for the first time probably ever the system powered down. Nothing could be done to rouse the slumbering giant.

FIX IT!!!!

After informing the bosses of the servers death, only one answer filtered down. FIX IT! But how does one resurrect a fallen god? Did not enough believe in him then was cast to float in the astral plane?

First we stared at it, then we tried opening it up. But the old god was crafty and would not give up his secrets. For his case was locked and the key lost to the ages. Fortunately we had a +5 Server Avenger, no server could stand up to this screwdriver. We made a mental note of making a request for a sonic screwdriver and inside we went. We tighten every cable we could find, cleaned every component. We tried our luck again, no dice. We needed outside help.

Reinforcements

The VAX expert appeared a few hours after we called him. After agreeing to a a hefty fee per hour for his assistance, he gets to work. Twenty minutes into his 2 hour minimum he declares all 4 redundant power supplies broken!

How they have error lights when they fail, we check them!

The answer was simple, the error lights used bulbs and they all were blown out.

How do we fix the beast?

Again the answer was simple, we cant unless we dig up 2 new power supplies. Problem was such devices where near impossible to find.

Breaking the news

We hastily formed a meeting and called the department heads in. Did I mention this was a Sunday. Three hours later the room was full of disgruntled people.

"How could we let this happen" they ask.

Out comes the folder, the 300 hundred pages devoted to every proposed project this year and our requested budget. In the High Priority section, listed first was "Replace the VAX"

Actually we brought in the budget requests for the last few years. In the High Priority section of each, listed first was "Replace the VAX" Imagine that.

Listed first in the reasons why, probably failure of system and unavailability of parts. We also found out the reason it was denied. It was too expensive and it has worked fine for over 15 years!!!. Then all in IT around the world could feel the pain of a room of server administrators all head-desking in unison.

The Users

While we devised the plan we still had to deal with the users.

Most of them simply emailed us. "I can't connect to the network"

Hmm, then how did you send the email. Oh you mean you cant connect to the dedicated server that's down. I think we sent out 20 emails stating this. And even put notices on the doors to various departments.

I ponder, why is it the only response a user has for any network issue no matter how small is; "I cant connect to the network."

WHAT DO YOU MEAN ITS DOWN

Yes I know you need this system to do 5% of your daily amount of work. We have other databases you can reference. I know for particular info you need that system up, but you know who to call to request the info right from the government. I know it is extra work and you need to do.

Please do not send emails asking when it will be back up when we put in the email;

"Please do not send emails asking when the system will be back up, we will send out more info when it becomes available."

When you do we have to comb through many helpdesk tickets and emails for fixable problems other users are having.

The FIX!!

What do you do when you have a massive server with a failed set of power supplies. Especially if your patron saint is MacGyver!! All we needed to do was provide power to the chassis. We knew how much that was, so why not build one. Just grab some scientific grade power inverters, configure them to the right Voltage, Amps and Wattage. Then see if she passes the smoke test.

Its risky, Dangerous and down right crazy.

But we are the few the proud the GEEKS!

We turned on the inverter and she gently hummed to life, then coughed and sputtered.

Life Support

She was running, barley with a series of alligator clips and power inverters.

We send out an email to the bosses.

"The unit is back up. She is on life support, we need to replace the entire system ASAP"

While I wait for a response, what do you think they will say? Its already been a few days. Saying yes means spending a shitload of money. No means, its on their asses.

[Pedersen]

While I wait for a response, what do you think they will say? Its already been a few days. Saying yes means spending a shitload of money. No means, its on their asses.

My guess? "It's working now. Why should we spend any money? Why should it fail? Did you not do your job properly and make it reliable?"

In other words, no. And worded in such a way that when it does fail again, it's your fault. Again.

Document it all, unless you don't mind taking the heat.

[Eric the Grey]

Ahhh, the VAX. Truly the God of the old-school servers. Capable of running day and night for years without so much as a reboot, or so the worshipers claim. Who am I to say if it is true or not? Your worshipers become fewer and fewer every year, while those who still bow down to you rely on your data more and more and when you die, you don't go quietly.

While I wait for a response, what do you think they will say? Its already been a few days. Saying yes means spending a shitload of money. No means, its on their asses.

I'd bet that first they'll ask you to explain exactly what the risk is in leaving it running the way it is.

Copy all databases off the system into some other media ASAP, and I don't mean backup tapes. You want that data readable by whatever database app you switch over to and tapes may not be useful.



Eric the Grey

[Geek King]

Oh, Daskinor, I had to smile ruefully through your tale. We too have an ancient beast with old databases that no one wanted to spend money on converting, but yet several people still need to access. Just this week, we recieved the quote for continued support of this dark god--a figure of five places for one year of service!

Lo! The Management did loose a thunderous rage at the number, and have begun talks of torches and pitchforks to oust the dark lord from its place on the bottom half of a floor-to-celing server rack. Plans have been hatched, and mystic circles are being scribed in preparation.

[protege]

Ugh. As the resident IT guy, I feel your pain.

Our old Win2K server was a piece of shit. It would crash if the case's side panels were installed. Seems the drives would overheat, crashing the server. Pain in the ass, since it always happened without warning. Eventually, the drive situation got pretty bad--both failed one afternoon. Meaning, we were literally out of business.

I spent nearly 4 hours fucking around with that goddamn server, only to hear how it "shouldn't have failed in the first place," and that I should "schedule server failures at more convenient times"

Anyway, it was being difficult--as soon as it booted, it would attempt to load Winblows, but errors popped up that the drives were corrupted! Meaning, 10 years of data, that, although was backed up 2 days before, wasn't as current as I'd have liked. Meaning, that if I couldn't get it off the drive, we'd be out 2 days worth of trade data

I had to bust my ass to get that fixed. Just because I *can* rebuild the database from scratch...doesn't mean I necessarily *want* to. Not only is it a pain and takes a *long* time, it just sucks. Plus, then I get to deal with the "well, why isn't *this* trade in there?" from the boss every few minutes. My answer? "It's not in there...because the fucking server crashed!"

But, I can't say that. At least I got lucky--I was able to see the drive (why?), get that piece of crap onto the network, and move the *entire* database, spreadsheet and other file structure, onto my own workstation. Set that thing as the temporary server, and all was good.

Did I even get a "thank you?" Hell no--I got told (again) that I should schedule failures for more convenient times

At least that piece of shit eventually got replaced--the old one has been stripped of most parts (including those damn hard drives), and will eventually be rebuilt. However, I'm tempted to bash it apart

[Eric the Grey]

I forgot to mention that the company I work for has 23 of these beasts (Vax's, that is), 20 at outlying plants, and 3 here in my data center.

Recently, with all the migration we've been doing off of them and onto a UNIX solution (still older than One World, but not quite as old), it was decided to see how many people really use them. So they shut them all down to see who yelled.

Of the 20, only 6 remain active at plants today. Eventually those will go away as well. Hopefully then we can also get rid of the three that sit here in my data center.



Eric the Grey

[lordlundar]

Seeing as I have a war story nomination waiting on this, I'd just like to say please post updates.

[mattm04]

My school has a old VAX for one application used by a few people, maybe twice a week. The thing has duct tape holding the front on. Yet they (finance people) don't want to replace it.

[Trayol]

We turned on the inverter and she gently hummed to life

I must ask Daskinor, did you do a Frankenstein: It's alive..It's alive! IT'S ALIVE!!!!

[wagegoth]

I had to triple-check my eyes when I saw the word VAX in the story. I have not heard the word VAX in like forever.

This was almost 20 years ago (I told you I'm old): My first husband was hired to design a system for a certain high-end greeting card company. He was an employee, not contracted, and was to design and build the system to replace an absolutely ancient IBM. I think it was a 3600. He worked out a system that used Wang -- yes, Wang -- and would have cost a total of $100,000 between leasing to purchase, software, etc., everything included. It was to handle everything but running the machines in the plant.

Unfortunately, my husband's diabetes caught up with him and he became ill. Due to substandard care, he was in the hospital for weeks, ending up with one leg amputated below the knee. He was so happy to finally come home and plan his return to work. When he called into let his boss know that he would finally be returning, his boss asked how he was. My husband told him he was fine, but he would need to be in a wheelchair for a couple of weeks while he prosthetic leg was being completed. His boss said, "Oh, that's not good," then told him he would get back to him before hanging up. They fired him. I believe they killed him, because the heart just went out of my husband. He lived only a few more months. His doctor called me the day after he died in shock, because my husband had just been to see him for a check-up the day he died and was doing really well.

Well, the company was told by a couple of enterprising contractors that they could breathe continued life into the IBM. (One of my husband's coworkers kept me up to date.) By the time the coworker left the company, the contractors had been paid over $300,000 and the IBM was still not working.

I refuse to buy any products made by that company to this day. And I hope they ended up spending hundreds of thousands more to get a new system.

Karma, she be a bitch.

[Crab]

Life Support

She was running, barley with a series of alligator clips and power inverters.

We send out an email to the bosses.

"The unit is back up. She is on life support, we need to replace the entire system ASAP"

While I wait for a response, what do you think they will say? Its already been a few days. Saying yes means spending a shitload of money. No means, its on their asses.

"Nah, look, it's working now."
Now, you fuck with them. Randomly tamper with the alligator clips every half hour, blaming it on its last death spasms, until they accede to a new one...

[Daskinor]

sorry for the lack of updates, I have been so busy with replacing this server.

But this crap takes the cake. So pardon the language.


We start taking out some large parts of the vax that had been replaced over time. Just never removed.

WE FOUND FUCKING ASBESTOS

Turns out the server room used to be the bomb shelter. It makes sense, its already fire proof and has no existing fire suppression equipment. Throw in an AC unit. A raised tile floor and a halon unit and you are in business.

Unfortunately back in the 50s they threw the stuff everywhere. When the room was converted, the stuff was stable in the walls. Over time the vibration from the equipment has worked it out.

Suposetly the health risk is low, because of the AC filtering to keep the dust level non-exsistant. This I can belive because we had some highly dust sensitive equipment not to long ago.

However this is the fun part.

We can not go in the room till they remove all of it.

Abort, Retry, Fail?

Well thats a lie, somone can go. As long as the Asbestos guys are thier and lend us a suit.

[lordlundar]

However this is the fun part.

We can not go in the room till they remove all of it.

Abort, Retry, Fail?

Well thats a lie, somone can go. As long as the Asbestos guys are thier and lend us a suit.

Now you see, I see this as an opportunity. Get skilled with an elastic and a paper wasp and shoot the alligator clips from out of the room.

"The machine needs to be back up!"
"Can't go in to fix it until the asbestos is removed. You know, we wouldn't have this problem if we could have replaced it when we asked for it."

(My BOFH is strong today)