The moose out front should of told ya

[BravoOrig]

I work as internet tech support for a cable provider in 23 states, and luckily I only cover about 4 or 5. So I come into work last Thursday, at 6am, and there's already a huge queue. It seems the DHCP server crashed earlier that morning, so anyone who power cycled their modem, whether for our phone or HSD service, could not get a connection back. This also applied to anyones IP address that was up for renewal, nothing to get it renewed from when that very server is down.

Okay, fine, at 6am, corporate to controls the messages on our phone system, isn't in the office yet, so I get to inform our customers firsthand that it is a known issue and we are working on it. But the calls did not dwindle when they did get that message on there. They put the message on first thing when you called, before it asks for your info, but people got al confused when later through the menus it didn't detect a problem in their area. Hold times were huge, and all my calls lasted 3 minutes at most, we each took 100+ calls and stopped confirming account info, just verified the lights on their modem.

Well atleast they fixed most of it that night, and I spent Friday with the same long queue fixing the problem.

[Horsetuna]

OT: What happened to the moose?

[alexdsoso]

heh. its a quote from national lampoons. they had a moose out front of wally world telling people it was closed. they bust in, and the janitor says, parks closed, the moose out front shoulda told you.

[Kilamon]

OK, I built an enterprise class, mirrored, LINUX based file server that handles DHCP, DNS and Windows domain in a day (for home use). I've done this same thing for work on a windows based system and I wager I could do it that way in less than 4 hours (assuming OS install, format, raid setup, etc). WTF was your IT group doing, aside from playing Quake?

[BravoOrig]

After the week of issues I've had to deal with, both internal and external, I just don't know. I've heard our DHCP server is in KC, but thats from an unreliable source who barely can tie their shoes without assistance. We use AT&T for our email provider and our internet backbone, so that maybe it, but I never get all the info. Now if IT could fix one of our tools that allows us to probe the modem thats been down for 3 days, I'd be happy and content. Then they can get around to fixing a problem I have on my work computer, which had a ticket placed for about 2 months ago.

[trunks2k]

WTF was your IT group doing, aside from playing Quake?

You mean they're supposed to be doing something else?

I've seen this sort of issue before. At <large pharma company> we had an entire week where the network home drives of emplyees at three major company campuses were down for half the morning. Unfortunately most employees kept all their work on those home drives, so that meant lots of calls about it. I'd come in the morning and see a huge queue of calls and just sigh. I'm fairly certain someone got fired over that.

[Kilamon]

I've seen this sort of issue before. At <large pharma company> we had an entire week where the network home drives of emplyees at three major company campuses were down for half the morning. Unfortunately most employees kept all their work on those home drives, so that meant lots of calls about it. I'd come in the morning and see a huge queue of calls and just sigh. I'm fairly certain someone got fired over that.

All major file access systems should not only be redundant (ready failover such as a pair of servers working in tandem to serve files from a single SAN) but easily repaired. I know that "home" drives in windows are mostly nonexistant unless you're doing roving profiles, which is dumb unless you have a gig-ethernet to all workstations. In Linux, I found out from installing my server at my house, you don't have to do any real administration. You just create the user in the normal linux tool, which creates the standard login/home folder for the user, and the samba daemon figures out automatically where that user's home folder is and directs them there.

Having said this, I can honestly say that unless something catastrophic occurs, all problems with file servers are the result of human error. I can provide an example.

Last year, one of the guys here was a little paranoid about all of his data since he was leaving the company. He deleted various files, folders, etc., from the server which would normally be OK except one of these folders was shared. On a clustered resource, you can't just delete a folder that's shared; you have to remove the resource in the cluster tool THEN delete the folder. You can imagine what happened. *crash* Users were asking me what happened to their home folders, all resources were nonresponsive, etc. The server basically had a heart attack while trying to activate a resource that no longer existed. Nice of Microsoft to program that one. Took me about 45 minutes to get stable again.

The best part of the whole deletion thing? Everything is backed up on tape. If I wanted to, I could restore everything he deleted. I don't know why people think the delete command is as powerful as they believe it.

Maybe I can make a business case to switch to Linux for file services....

[Der Cute]

Ya mean they dont have a way to monitor the CORE pieces of their network?
The DHCP, the File Servers...the AD servers....???

You're saying since you didnt get an error/alert about the DHCP....the IT dept doesnt have a monitoring on them?

That's a WTF to me.

Cutenoob