PDA

View Full Version : Servers down! FFFFFFFFFU--


Jay 2K Winger
03-22-2011, 05:01 PM
Guh. These last two days have just been... guuuhhh.

Last week, I was the on-call at work. My first week doing it. Had a little pager and everything in case they couldn't reach me via my cell. Got called in Saturday. At 630 am. That was fun. </sarcasm>

I had seriously debated calling out Monday so I could have that extra day to decompress. Ended up going in anyway.

Good lord, we had no lulls in the calls all day long.

Now, we get calls all day, and sometimes we'll have little rushes where there'll be two to four people waiting in the queue even with everyone answering phones. When the Calls In Queue number gets up above six, it's serious. When the CIQ number gets into double-digits, it's a sign that there's a major problem. Usually a server going down.

Near as I can tell, there wasn't a major server problem in the morning, when we had about 12 CIQ fairly consistently for about two hours. Things settled down (still at least 2-3 CIQ during the mid-afternoon) until the evening.

Oh, and while I'm dealing with all of that, I get a intraoffice IM from a user I'd helped on Saturday. I'd taken two calls from her on Saturday, because she was having issues with her Word program. She'd had me stay on the line while the Repair was being run on MS Office, but I managed to deflect her request to keep me on the line while she rebooted. ("Sorry, ma'am, I would stay on the line, but I have other calls I need to take.") She called back later, because a key plug-in that our client uses wasn't working in her Word. So I performed the repair and got it working.

Well, she called back Monday, because the original problem (Word crashing) was starting up again, and she was losing work because of it. The secondary fix wasn't working either. I managed to get her to try using a different computer while I worked on the rest of the calls that morning.

About 4:30-5:00, right as most of the day crew had already left, we suddenly had about four servers go down. Including one where a number of important applications are hosted. Gyeh. Not fun.

For about an hour, we had a shitload of calls to take related to those servers being down until we got the incoming voice-message (which relates issues like this to the callers so they don't clog the incoming lines with redundant calls) updated and the rest of the incoming calls dealt with. Things had settled down by the time I left.

Fast forward to today. Late in the morning, the application server goes down again.

Shit.

And in going down, it takes out the IM network, thus severely limiting our communication ability in the office.

FFFFFFFFFFUU---

Thankfully, the incoming voice-message was updated quickly, thus paring out a lot of the calls we were dealing with. But still, I'm hoping the rest of the day goes less stressed.

Jay 2K Winger
03-23-2011, 04:55 PM
Still having server issues.

And there were other, un-related, safety issues going on at the facility today. I can't go into more detail than that, lest I reveal too much about where I work (and thus violate both CS.com policy and work policy), but we're still trying to patch things back up.

On top of everything else today, the system we use to track our trouble tickets has been hiccuping all morning.

FML. :cry:

protege
03-23-2011, 06:17 PM
...and people wonder why I threaten our equipment with a 40-pound fence maul :D

Jay 2K Winger
03-23-2011, 09:35 PM
Funny enough, people were considering taking a hammer or axe or BOMB to the servers today.

They seem to have stabilized for now, but I don't trust them. You just know they're Up To Something, those sneaky servers...

ETA:
I was right!! A different server has gone down, and is affecting users in multiple locations!

Argh! *shake fist*