Failure is always an option.

[Daskinor]

Last Thursday is was my turn to stay late and do firmware and software updates on the server. Never takes over 2 hours and I get to go home early Friday.

So it was just me, by myself in the haunted server room, formally a bomb shelter. Well me and my anti Murphy nerf gun.

Servers check.
Switches Check.
Routers Check.
Firewalls Check.
And lastly the spam filter


10% 20% 30% 40% 50%
Function error please contact your system admin.
!!!!~!

This is not good

So I console in, and take a look around. Up until I am asked for my username and password. It accepts it then dumped me back to the login screen.

Frak, and I am hungry at this point! Double Frak!!

Ok lets call tech support.

Troubleshoot step one, connect it to the manufactures servers. Does not work.

Step 2, soft wipe. Does not work

Step 3, wipe everything- yay it works.

It comes up in factory mode and I restored the system backup it automatically made. Doesn't WORK. FRAK! FRAK! FRACK!

Fine, I reconfigure everything by hand and then update it to the latest version.

Doesn't Work. GRRR.

Tech Support again. They find that it error out on the final content update.

They try to fix it
They try to fix it
They try to fix it


Eventually its 1 am. Its the only spam filter we have. So the exchange server is getting 100% of the cheep Viagra and penor enlargement adds. Poor puppy is running hot.

They figured it out. HD 2 is failing. Not enough to trigger it as bad but its screwing crap up. So break the internal raid, and there she goes.


I get to spend the morning responding to the "OMG I have penors in my E-Mail." help desk tickets. Most of them tell the poor (L)user to refer to the global email and the IT bulletin on the company intranet homepage.

Monday rolls around. There is lots of spam. Spam everywhere. I wonder over to the spam filter. Its dead. DEAD DEAD! Like no lights no power.

FRACK.

[teh_blumchenkinder]

Eventually its 1 am. Its the only spam filter we have. So the exchange server is getting 100% of the cheep Viagra and penor enlargement adds. Poor puppy is running hot.
They figured it out. HD 2 is failing. Not enough to trigger it as bad but its screwing crap up. So break the internal raid, and there she goes.
I get to spend the morning responding to the "OMG I have penors in my E-Mail." help desk tickets. Most of them tell the poor (L)user to refer to the global email and the IT bulletin on the company intranet homepage.
Monday rolls around. There is lots of spam. Spam everywhere. I wonder over to the spam filter. Its dead. DEAD DEAD! Like no lights no power.
FRACK.

one in the morning? don't you go home at like 5 or something?
Also, I love the descriptions of what's going on. (Penors, lol) And wow, someone's fk'd, if only for the mass slough of spam... The way you're saying it puts me in the mind of dumping leaves, wet leaves, styrafoam filler bits and styrafoam peanuts willy-nilly in the server rooms.
I shouldn't be laughing.

[teh_blumchenkinder]

Any updates, D-man? >_> I hope you didn't get eaten by (l)users....

[mattm04]

one in the morning? don't you go home at like 5 or something?
Also, I love the descriptions of what's going on. (Penors, lol) And wow, someone's fk'd, if only for the mass slough of spam... The way you're saying it puts me in the mind of dumping leaves, wet leaves, styrafoam filler bits and styrafoam peanuts willy-nilly in the server rooms.
I shouldn't be laughing.

The latest/earliest for me was leaving 6:30 AM. This was before we had good power protection on the network and Mr. lightning decided to cause havoc. Oh and we HAD to run some critical apps at 9:00 AM. Cue massive pillaging efforts to get the equipment working to run the apps/reports until the parts arrived. Oh, and some good UPS gear.

[technical.angel]

Mr. Lightning caused havoc at my house too. I guess my line got hit and went through the phone line and took out my modem, which has been out of warranty since Feb. So, going to get a new modem/router tonight.

[blas]

Your title is wrong. It should say "There is always options with failure".

At least, that's our motto at work.

[Ironclad Alibi]

Mr. Lightning caused havoc at my house too. I guess my line got hit and went through the phone line and took out my modem, which has been out of warranty since Feb. So, going to get a new modem/router tonight.

I had a visit from Mr. Lightning many years ago. He came in on the phone line. Took out the phone and the internal modem. Fortunately, he stopped before getting to the motherboard. After that the phone line was routed through the surge protector. Don't have to worry about it now with a fiber optics hookup.

[Geek King]

I had a visit from Mr. Lightning many years ago. He came in on the phone line. Took out the phone and the internal modem. Fortunately, he stopped before getting to the motherboard. After that the phone line was routed through the surge protector.

A surge protector is a nice extra protection, but its not bulletproof. We had a site just a few months back that took a direct strike. The power surge blew two of their three PC's motherboards out, one of which also lost the power supply and RAM. It also fried their fax machine and screwed up that phone line. Kind of scary when you think about the fact it had to bypass not only the local surge protectors, but the building's main surge protection as well.

[Daskinor]

ahh most of the story is uneventful.

I called the manufacturer Monday, and Tuesday the unit was in.

Of course we got 139 helpdesk tickets about the spam. Even though we have on the helpdesk intranet page. Please do not make tickets about the spam.


At least they use the helpdesk system. Getting them off sending an email to a random person then whine when they are not helped in 10 minutes system was a pain in the ass.



As for lightning. Almost nothing will save your but from a direct hit unless that direct hit was to a lightning rod. I have had equipment behind gas filed arrestor modules that still got zapped. Stupid old ass buildings....

[lordlundar]

Kind of scary when you think about the fact it had to bypass not only the local surge protectors, but the building's main surge protection as well.

That's because there's two traits to lightning that make surge protectors a 50/50 effectiveness: It moves really fast and it's really powerful.

The speed aspect allows it to get past the breaker before it trips, rendering it ineffective at stopping it.

Even if the breaker trips or the fuse pops, there's a lot of juice going through the wires which can allow it to jump the gap.

It's why I prefer UPS (uninterrupted power supply) units. Any power coming in is used to charge a battery which is then used to power the devices plugged in. So if it gets past the surge protection part, the worst case scenario is a blown battery.

[Geek King]

It's why I prefer UPS (uninterrupted power supply) units. Any power coming in is used to charge a battery which is then used to power the devices plugged in. So if it gets past the surge protection part, the worst case scenario is a blown battery.

Nothing is going to stop a direct or near-miss lighting strike. Even a UPS can be overcome and bypassed, as the batteries can't charge fast enough, nor store enough energy to eat a full lightning bolt. If we could do that, there would be battery farms all over the tornado belt in the U.S. set to absorb lighting strikes for almost free energy. We're just not at the point of being able to convert and store that kind of energy that quickly.

BONUS: Article for fun science-geek reading.

[lordlundar]

Very true, but it's still a lot more surge resistant than a basic power bar.

[Difdi]

I had a visit from Mr. Lightning many years ago. He came in on the phone line. Took out the phone and the internal modem. Fortunately, he stopped before getting to the motherboard. After that the phone line was routed through the surge protector. Don't have to worry about it now with a fiber optics hookup.

I had a similar experience some years back, but in my case, it was the cable internet line. Seems the last tech to service it had not only disconnected the ground in the cable company's side of things, but also the surge protector installed on the customer side.

When Mr. Lightning came to visit the utility pole up the street, we lost three cable "modems", two computer power supplies (luckily they failed safe) an inline surge protector on a laptop, and two TVs.

Naturally, the cable company disclaimed any responsibility for the matter...

[mattm04]

ahh most of the story is uneventful.

I called the manufacturer Monday, and Tuesday the unit was in.

Of course we got 139 helpdesk tickets about the spam. Even though we have on the helpdesk intranet page. Please do not make tickets about the spam.


At least they use the helpdesk system. Getting them off sending an email to a random person then whine when they are not helped in 10 minutes system was a pain in the ass.



As for lightning. Almost nothing will save your but from a direct hit unless that direct hit was to a lightning rod. I have had equipment behind gas filed arrestor modules that still got zapped. Stupid old ass buildings....

You users actually use the help desk? Can I have them. Please!

[Daskinor]

You users actually use the help desk? Can I have them. Please!

This is the direct quote from big boss in the email he sent out a year or so ago. We since then printed it out and have it on our wall.

"If your computer does not work and you do not make a help desk ticket. You will not get it fixed. If you can't do your work you will then need to deal with me"

-Big Boss