Tweet
So, I love being the telco admin, as long as I don't have to do anything telco-y.
So, yesterday morning, we get calls that message waiting lights aren't working correctly. I try logging into the server to see what was going on, and instead of sending me to my profile, it creates a temporary one. I get in and, boom, voicemail goes down.
PANIC!!!
Our toll free phone tree works off the voicemail server.
PANIC!!!!
Server admin tries to remotely restart it. Fails. Manages to remotely shut it down. We have to physically go over and restart the server. Happiness!!!!!... for all of 5 minutes, and BAM! voicemail goes back down.
I remote back into the server to check out the services. Temp profile. I log out, go back in, and get my profile... and a message that C: was almost full.
PANIC!!!!
Server admin and I start dumping installers and stuff onto my home dir. We manage to free up 2 whole gigs of space from 0 free space.
I call the tech support at 10:15am (yes, I wrote down the exact time my ticket was entered.)
And proceeded to be chained to my desk (I left twice, once for a caffeine run, and once for a potty break) FOR FOUR HOURS!!!! staring at the phone, waiting for it to ring. Finally at 2pm, after what should have been A TWO HOUR TURNAROUND! I finally call back and ask at very vehement WTF???
I was told that my ticket was sent to a technician 15 minutes after it was called in and they were working on the issue. The technician will call me back soon.
WTF?? The problem is on my server. You don't have access to my server without me starting a webex session. How is anyone working on a problem THEY DON'T HAVE ACCESS TO?????
Finally, AN HOUR LATER!!! I get a call from the technician. Before he goes into anything, he says, "You need to know that we have changed our rates. We are now charging $250 an hour. Will you accept the charge?"
"Well, I don't really have a choice if I want my server to to stay up tonight."
"You do have other choices..."
"I'm aware I have other choices with service providers, and believe me, after the service I have received from this company, I AM looking."
Once I get the billing straightened out, he simply states, your service is back up, the size of free space hasn't changed from the morning, everything should be cool.
OH HELL NO!! I have a freaking TWO FREE GIGS ON MY SERVER! I want to know that the DAMN THING won't crash tonight! You THINK you know what caused the space to fill up (the server using itself as temp storage for voicemails, rather than the exchange sever), but me asking how to clean up that space is UNFATHOMABLE!
When the tech gets the idea that I'm not just going to say, "Oh, okay, yay."
He finally says that he'll send my ticket to an engineer and quickly ends the call.
Joy, I think, that means I'll be getting a call 15 minutes before I have to leave.
Well, 4:15pm comes and goes, and no call, 4:30 sneaks up behind me. NO CALL!!
I go home and make pizza, and kill lots of things in WoW.
7:45am comes around, and I walk into the work. I see no lit message waiting light on my phone, beckoning me to new messages.
As soon as the AA gets in, I call BACK to tech support to find out the HELL was going on.
"Well," said the now meek sounding receptionist, "according to the ticket, it was forwarded to something call the VIA Group. I don't know what that is. I'll forward you to the tech you talked to yesterday for more info."
At that point, I just hung up, because I knew who the VIA group is, and was very happy. They are the ones who installed the system and who know HOW THE FRACKING THING WORKS!!!
So, the tech who installed the system for us, who supported us for the first year of our contract gave me a call.
I told him what was going on.
"Ah," he said. "You guys installed a patch recently, haven't you?"
"Well, I wouldn't say recently, but yeah."
So, in short, guess what took up 60gb of storage space on our voicemail server?
LOGS!!!!!!!!!!!!!!!!!!!!!!
Apparently, when you install a patch, you have to go through and tone down the default logging settings. The
tech from the first company (who, incidentally, I've written/ranted about before) didn't know about this, so, since January 11th, the server had been creating 4 logs a minute, 24-7.
So, I was shown where the logs lived, set the recycle bin delete everything permanently, and deleted SIXTY GIGS OF LOG FILES!!
Voicemail server is happy now.
Now, as to what prompted my diatribe.
We sent out advisory emails so people would know that the voicemail was down, then later, that everything was back up.
Today, we got this email from a student... "please stop sending me updates about things that don't work. please
thank you"
:: death of Jenni ::
So, yesterday morning, we get calls that message waiting lights aren't working correctly. I try logging into the server to see what was going on, and instead of sending me to my profile, it creates a temporary one. I get in and, boom, voicemail goes down.
PANIC!!!
Our toll free phone tree works off the voicemail server.
PANIC!!!! Server admin tries to remotely restart it. Fails. Manages to remotely shut it down. We have to physically go over and restart the server. Happiness!!!!!... for all of 5 minutes, and BAM! voicemail goes back down.
I remote back into the server to check out the services. Temp profile. I log out, go back in, and get my profile... and a message that C: was almost full.
PANIC!!!! Server admin and I start dumping installers and stuff onto my home dir. We manage to free up 2 whole gigs of space from 0 free space.
I call the tech support at 10:15am (yes, I wrote down the exact time my ticket was entered.)
And proceeded to be chained to my desk (I left twice, once for a caffeine run, and once for a potty break) FOR FOUR HOURS!!!! staring at the phone, waiting for it to ring. Finally at 2pm, after what should have been A TWO HOUR TURNAROUND! I finally call back and ask at very vehement WTF???
I was told that my ticket was sent to a technician 15 minutes after it was called in and they were working on the issue. The technician will call me back soon.
WTF?? The problem is on my server. You don't have access to my server without me starting a webex session. How is anyone working on a problem THEY DON'T HAVE ACCESS TO?????
Finally, AN HOUR LATER!!! I get a call from the technician. Before he goes into anything, he says, "You need to know that we have changed our rates. We are now charging $250 an hour. Will you accept the charge?"
"Well, I don't really have a choice if I want my server to to stay up tonight."
"You do have other choices..."
"I'm aware I have other choices with service providers, and believe me, after the service I have received from this company, I AM looking."
Once I get the billing straightened out, he simply states, your service is back up, the size of free space hasn't changed from the morning, everything should be cool.
OH HELL NO!! I have a freaking TWO FREE GIGS ON MY SERVER! I want to know that the DAMN THING won't crash tonight! You THINK you know what caused the space to fill up (the server using itself as temp storage for voicemails, rather than the exchange sever), but me asking how to clean up that space is UNFATHOMABLE!
When the tech gets the idea that I'm not just going to say, "Oh, okay, yay."
He finally says that he'll send my ticket to an engineer and quickly ends the call.
Joy, I think, that means I'll be getting a call 15 minutes before I have to leave.
Well, 4:15pm comes and goes, and no call, 4:30 sneaks up behind me. NO CALL!!
I go home and make pizza, and kill lots of things in WoW.
7:45am comes around, and I walk into the work. I see no lit message waiting light on my phone, beckoning me to new messages.
As soon as the AA gets in, I call BACK to tech support to find out the HELL was going on.
"Well," said the now meek sounding receptionist, "according to the ticket, it was forwarded to something call the VIA Group. I don't know what that is. I'll forward you to the tech you talked to yesterday for more info."
At that point, I just hung up, because I knew who the VIA group is, and was very happy. They are the ones who installed the system and who know HOW THE FRACKING THING WORKS!!!
So, the tech who installed the system for us, who supported us for the first year of our contract gave me a call.
I told him what was going on.
"Ah," he said. "You guys installed a patch recently, haven't you?"
"Well, I wouldn't say recently, but yeah."
So, in short, guess what took up 60gb of storage space on our voicemail server?
LOGS!!!!!!!!!!!!!!!!!!!!!!
Apparently, when you install a patch, you have to go through and tone down the default logging settings. The
tech from the first company (who, incidentally, I've written/ranted about before) didn't know about this, so, since January 11th, the server had been creating 4 logs a minute, 24-7. So, I was shown where the logs lived, set the recycle bin delete everything permanently, and deleted SIXTY GIGS OF LOG FILES!!
Voicemail server is happy now.
Now, as to what prompted my diatribe.
We sent out advisory emails so people would know that the voicemail was down, then later, that everything was back up.
Today, we got this email from a student... "please stop sending me updates about things that don't work. please
thank you"
:: death of Jenni ::
until he resets the default option to NOT save a log file unless you tell it to.
Comment