Tweet
And I mean insane. I haven't posted much lately about work, because of something that happened that was a very good thing.
I got told to take my head out of my ass by the new partner in the company (not in quite those words, but that was the message). I can be unhappy. I can be miserable. Just stop taking it out on others. You know what? He was right. So, I stopped being an asshole to everybody.
Things have turned around for the better since then. I still see a lot of stupidity, but I think people are listening to me a bit more now, and we're avoiding making the problems worse. We've been able to start fixing our problems, and did we ever have a shitload of them.
Our server room has numerous problems, with heat being only one of them. We've been struggling to keep the temperature below 80F all summer long. We've managed to be mostly successful. We've had the major UPS get too hot and shut down, taking the whole room with it (more on that in a minute).
We've managed to remove somewhere over 1000ft of wire (I'm sure, and I think it's over 2000ft), and we've still got a large amount more to go. Best estimate? Probably another 2000ft. We've managed to permanently shut down about a dozen servers so far, and get them out, resulting in much better air flow and cooling. We've got another 20 servers that are on temporary shutdown, and we're going to make them virtual soon, so that the physical machines don't have to come back into the room.
We've got another half dozen servers and devices that we're shutting down next week (again, permanently). And I think I can pull out another two or three on top of that.
In addition, I've been working on the other hardware problems (failed power supplies and the like). We should be getting almost the last of that tomorrow. We'll still have one hard drive that needs replacement, but that's workable.
Now, the server room shut down I mentioned above? Turns out there was a circuit breaker that needed replacement. To find this out, the repair tech had to come out and shut everything down in the room, then take the transfer box apart. Once found out, he didn't have the right part with him, so he had to get it ordered.
Next time he's getting ready to come out, he calls me and says "woops, the service center ordered the wrong part, we'll try again for next week". Next week comes, and he's got the right part, so we shut everything down again (second controlled shutdown, third overall). He takes things apart, and finds out the part he's been sent is still the wrong part.
So, he gets the right part ordered, comes back the next week for yet another shutdown, and replaces the part. Finally. Four shutdowns later, and we finally have the right part. Oh, and doing a clean shutdown of the entire server room takes 3 hours, and starting everything back up takes another two (assuming everything comes up properly).
So, 20 hours of my life on shutdowns and restarts in under a month. 15 of that is all after hours work (meaning I get to start shut downs at 7pm to reach a final shutdown by 10pm), and each time he comes in to work I have to wait 2 hours, so there's another 6.
Meanwhile, I'm cleaning up our Exchange setup. The people who installed that deserve sound beatings. Without getting technical, suffice to say that, because of how badly they did, I managed to shut down two physical servers from that setup alone.
I'm also looking at our backup setup, and finding out that the backup of our Exchange system is horribly broken. By horribly broken, I mean that the last full backup was done in August, 2010. Look at the calendar, and you will notice that we are in August, 2011. This means that if one of those servers went, we would have to find every tape for a full year to do a restore. I've fixed that.
I've also been working on our VMWare systems. They were poorly maintained, and showing it. Sluggish performance, periodic disconnects from the network for hosted virtual machines, and improperly set up (for those who know: Missing vmware tools in the guest OS) VMs. I've begun fixing that, too, and things are already looking better.
And that's all been in the past 2 months. I think it might have only been one, I'm not totally sure.
Way too insane lately. And that doesn't even get into the issue of dealing with coworkers who almost seem to be actively refusing to work with all of IT, and instead are focused only on the stuff they were doing before the merge.
I think I've started to ramble here, so am calling this post done for now. At least I've got tomorrow and Friday off. Yay!
EDIT! I forgot this little bit of joy. The final time that the electrician came by, we decided to move some servers around to reduce clutter, make things neater, etc. This required a specially configured pair of switches (standard vlans, vlan trunking, private vlans, and ether channeling, for the techies out there). We thought we tested it properly. We were wrong. The attached systems did not come up properly, and would not come up properly until five hours after we first tried. The end result was that I worked from 9am on Tuesday until 2:15pm on Wednesday getting everything fixed and back up and running.
I got told to take my head out of my ass by the new partner in the company (not in quite those words, but that was the message). I can be unhappy. I can be miserable. Just stop taking it out on others. You know what? He was right. So, I stopped being an asshole to everybody.
Things have turned around for the better since then. I still see a lot of stupidity, but I think people are listening to me a bit more now, and we're avoiding making the problems worse. We've been able to start fixing our problems, and did we ever have a shitload of them.
Our server room has numerous problems, with heat being only one of them. We've been struggling to keep the temperature below 80F all summer long. We've managed to be mostly successful. We've had the major UPS get too hot and shut down, taking the whole room with it (more on that in a minute).
We've managed to remove somewhere over 1000ft of wire (I'm sure, and I think it's over 2000ft), and we've still got a large amount more to go. Best estimate? Probably another 2000ft. We've managed to permanently shut down about a dozen servers so far, and get them out, resulting in much better air flow and cooling. We've got another 20 servers that are on temporary shutdown, and we're going to make them virtual soon, so that the physical machines don't have to come back into the room.
We've got another half dozen servers and devices that we're shutting down next week (again, permanently). And I think I can pull out another two or three on top of that.
In addition, I've been working on the other hardware problems (failed power supplies and the like). We should be getting almost the last of that tomorrow. We'll still have one hard drive that needs replacement, but that's workable.
Now, the server room shut down I mentioned above? Turns out there was a circuit breaker that needed replacement. To find this out, the repair tech had to come out and shut everything down in the room, then take the transfer box apart. Once found out, he didn't have the right part with him, so he had to get it ordered.
Next time he's getting ready to come out, he calls me and says "woops, the service center ordered the wrong part, we'll try again for next week". Next week comes, and he's got the right part, so we shut everything down again (second controlled shutdown, third overall). He takes things apart, and finds out the part he's been sent is still the wrong part.
So, he gets the right part ordered, comes back the next week for yet another shutdown, and replaces the part. Finally. Four shutdowns later, and we finally have the right part. Oh, and doing a clean shutdown of the entire server room takes 3 hours, and starting everything back up takes another two (assuming everything comes up properly).
So, 20 hours of my life on shutdowns and restarts in under a month. 15 of that is all after hours work (meaning I get to start shut downs at 7pm to reach a final shutdown by 10pm), and each time he comes in to work I have to wait 2 hours, so there's another 6.
Meanwhile, I'm cleaning up our Exchange setup. The people who installed that deserve sound beatings. Without getting technical, suffice to say that, because of how badly they did, I managed to shut down two physical servers from that setup alone.
I'm also looking at our backup setup, and finding out that the backup of our Exchange system is horribly broken. By horribly broken, I mean that the last full backup was done in August, 2010. Look at the calendar, and you will notice that we are in August, 2011. This means that if one of those servers went, we would have to find every tape for a full year to do a restore. I've fixed that.
I've also been working on our VMWare systems. They were poorly maintained, and showing it. Sluggish performance, periodic disconnects from the network for hosted virtual machines, and improperly set up (for those who know: Missing vmware tools in the guest OS) VMs. I've begun fixing that, too, and things are already looking better.
And that's all been in the past 2 months. I think it might have only been one, I'm not totally sure.
Way too insane lately. And that doesn't even get into the issue of dealing with coworkers who almost seem to be actively refusing to work with all of IT, and instead are focused only on the stuff they were doing before the merge.
I think I've started to ramble here, so am calling this post done for now. At least I've got tomorrow and Friday off. Yay!
EDIT! I forgot this little bit of joy. The final time that the electrician came by, we decided to move some servers around to reduce clutter, make things neater, etc. This required a specially configured pair of switches (standard vlans, vlan trunking, private vlans, and ether channeling, for the techies out there). We thought we tested it properly. We were wrong. The attached systems did not come up properly, and would not come up properly until five hours after we first tried. The end result was that I worked from 9am on Tuesday until 2:15pm on Wednesday getting everything fixed and back up and running.
.
Comment