PDA

View Full Version : Winter break?? What's that?? Epicness.


technical.angel
12-30-2010, 02:58 PM
So, being over in Education land, I get a grand week and a half off work for Christmas. This year, it was the 23rd through the 30th. The peasants rejoiced.

So, tell me, why at this moment, am I going into my 21st hour of work during my holiday???

Big boss (aka the CIO) is a mac user. This is all well and good. The newest version of Office came out for mac, and uh oh! Problem land! Outlook can only be used with Exchange 2010, and we're using Exchange 2003! BASTADGES!!

So, of course, we got an Exchange 2010 box up and running. Uh oh! Problem land! We use version 3.?? of Avaya Modular messaging for voicemail! The current version 5.2 (see a theme here?) is the one that supports Exchange 2010. BASTADGES!

So, CIO goes to the CFO, and gets the money needed to get the upgrade for the voicemail server. Our phone support company rep (who we're buying the upgrade from) begs, pleads and cries for us to go with the more expensive option, which is replacing the physical server. That way, our downtime is minimal and the multitude of problems we've had with voicemail will go away. AND, when go move to VoIP, we don't have to buy a new server. YAY! And the Telco Admin (me) rejoices!

HOWEVER, all big boss cares about is getting his Outlook to work. So, he decides to go with the cheaper option, which is just the upgrade. And the Telco Admin cries.

So, instead of downtime being limited to the amount of time it takes to shut down a server, remove it from the rack, put the new one in, turn it on, and maybe configure it for our mail server (if they couldn't do that remotely), we're looking at 4-8 hours of downtime. Let me repeat that for posterity.. 4-8 HOURS

Remember what I said earlier?? Yeah....

So, now we have to schedule the upgrade, planning on up to 8 hours of downtime. Well, for various reasons, during break is my only real choice. That way, no one's affected by the outage, and I'm not getting called away from the phone techs because someone can't turn on their friggin' printer by themselves.

I choose Monday, the 27th, because A: Our learning management system/super duper mega database software was getting upgraded on Tuesday and Wednesday. B: Work is the perfect excuse to leave my parents' sooner than they would want me to leave. So, perfect! YAY!

So the plan is that the main tech will be in his office, and he will send another tech on site, to do the physical work (swapping CDs, rebooting, etc.). That tech, J, tells me that we're only looking at about 2 hours for the upgrade. The Telco Admin rejoices! (oh, wait, did he knock on wood? Uh oh.)

I get him set in the phone room (aka the dungeon), and go back to my office, and spend 2 hours doing a lot of nothing. 2 hours later happens to be noon. So I go back, and see what's the what. As far I can understand, the first part of the process involves updating firmware or drivers on the cards that connect the server to the phone system to the new version. This went well. The second part involves running several installers. This went well until one of the installers decided it didn't want to run. The offsite tech was researching the issue, so we grabbed the server admin, who was onsite in case of exchange integration issues, and went out to lunch. We come back, and the two techs fought with the server for 5 more hours, and then decide that it's PROBABLY a bad CD, and the main tech will come out to the site later and do the upgrade personally. we schedule it for Wednesday. So far, I've clocked 8 hours.

Wednesday arrives, and due to the LMS upgrade, we have pretty much a full house, with the DBA, asst DBA, my manager, and the Server Admin (who had to work on Tuesday, also). Main tech arrives, and gets right to work. Two hours later, I drag him out of the dungeon for lunch, which Manager had thoughtfully put on the uni credit card. (FREE LUNCH!! PIZZA!!!!!!!!) Tech and Server Admin talk. The dreaded R word is brought up.

Server Admin is all for reformatting, as he's had OS based issues with the server since he started. I'm against a reformat, since the Tech tells me that all the greetings will be lost. A compromise is made, and they will try to repair the Windows install first.

Of course, it doesn't help.

The Tech had said he wanted to leave by 5pm. 4:50pm rolls around, and I hadn't heard anything recently, so once again, I descend to the dungeon level. The reformat worked. However, the install is like... HUGE. But since it was "almost done", he was going to stay and finish it, transfer the data back and configure, so it would be live by the time he left. Testing could wait for the morning.

2 and a half hours later, I'd been in the office for a total of 11.5 hours, when I had been planning on, at most, a 4 hour work day. The pizza was wearing thin, and the 90 calorie pack cookies were not cutting it. So, back down I go..., and find the Tech packing up. Apparently, the install time was just too much for him. So, he was going to let it run overnight, remote into the server in the morning and do the restore of the data, config, and everything else. I finally get to go home.

It's now 9:45am on Thursday. Tech and Server admin are doing their things. With luck :: knock on wood ::, everything will go according to plan, and I won't have to spend the entire day here. As of right now, I'm looking at 2 days and 5 hours of comp time. When I will be able to take that comp time? Who knows??? I do know that I'm going to have a heck of a time doing my timesheet!!

OH! And remember the part where I said the voicemail greetings are getting lost? Yeah, so that means when we get everything back up, I get to send out an email telling everyone this, and how to re-record their greetings.

This step requires a password. A password no one ever uses since they check their messages via their email. You can guess how many people remember it. So, I'll be "on call" from the time we get it back up until Monday morning, remoting into a server to reset passwords. AWESOMESAUCE!

Shoot me now, please???

dalesys
12-30-2010, 03:58 PM
Remember the first correction to Murphy's Law?



Murphy was a blithering optimist!

:hug:

trailerparkmedic
12-30-2010, 06:28 PM
I feel bad for all college techies this time of year.

My school is upgrading Blackboard (and probably other stuff we can't see) and something obviously went wrong since it was supposed to be down for two days, and it's still down on day four. The thing that really makes me cringe is that the new students in my program have been calling the help desk all week, asking for updates on when it will be back up, because they want to check if their spring classes are listed. Never mind that they don't NEED to check, and the professors can't post anything if the system is down.

I want to slap the new students.:salmon::salmon::salmon:

*knock on wood* Good luck with the passwords.

Rapscallion
12-31-2010, 10:26 AM
So, being over in Education land, I get a grand week and a half off work for Christmas. This year, it was the 23rd through the 30th. The peasants rejoiced.

Um, isn't that just one week for the dates described?

Anyway, do you get overtime for this extra, or time in lieu, or just shafted?

Rapscallion

Captain Trips
12-31-2010, 06:50 PM
And all because your boss wanted to use Outlook on a Mac. Shees!

Lachrymose
12-31-2010, 07:01 PM
For the want of Outlook, a server was lost...

Sapphire Silk
12-31-2010, 07:13 PM
<snip> The thing that really makes me cringe is that the new students in my program have been calling the help desk all week, asking for updates on when it will be back up, because they want to check if their spring classes are listed. Never mind that they don't NEED to check, and the professors can't post anything if the system is down.

I want to slap the new students.:salmon::salmon::salmon:

*knock on wood* Good luck with the passwords.

I know what you mean. My department chair wanted me to make sure my Moodle page was ready before I left for Christmas break. Can't figure out any reason why, since we return to the office a week before classes begin, and the student can't access Moodle until 8am on the first day of classes anyway. I said to heck with it, I'll work on it at home.

And of course, Moodle won't accept the upload of the course material I'm trying to import. I'm not losing any sleep over it.


Um, isn't that just one week for the dates described?

Anyway, do you get overtime for this extra, or time in lieu, or just shafted?

Rapscallion

Don't know about her, but I got paid extra for doing SOAR after faculty went on break :) It was regular time, though, not overtime :(

technical.angel
01-01-2011, 09:27 PM
Um, isn't that just one week for the dates described?

Anyway, do you get overtime for this extra, or time in lieu, or just shafted?

Rapscallion

Well, I'm counting Thursday and Friday as half a week, because it makes me feel better. :)

I'm leaning more towards shafted. Well, let me put it this way, as of this moment (okay, a few minutes from now), I have worked 20 hours. I left an hour and a half early on Wednesday, so that's 18.5 hours. That puts me at 2 days and 2.5 hours. I have no idea how I'm going to be able to take off 2 days the week after break. We'll just see what Thursday and Friday holds.

I really wanted overtime, but Mom explained that since the only thing putting me over is holiday time, it won't count. So, it's comp time or nothing. :(

At least I'll be able to go to the eye doctor this coming week. My left eye is blurry.

Raveni
01-03-2011, 12:53 AM
I'm not a techie, so the last time I did an upgrade like this, I reset everyone's passwords and made them go in and change them. Just because they can't remember their dog's name or some such doesn't mean you should be on call the whole holiday.

technical.angel
01-03-2011, 05:24 PM
Well, an update. Friday, they put the info back on, and the server freaked out again. So, another reformat of the server. Saturday morning, I get the call that everything is back up and working. So I sent out the email saying that the greetings have been lost, and to re-record. Well, my parents came down, and that evening, after they left, I went back in.

And found the greetings had NOT been lost. :D um... :cry: So I recalled the email.

Get in on Monday, and find the voicemail server is back down. But this time, we find it's because of a permission issue with our Exchange 2010 box. That's getting fixed now, and everything is good!