Announcement

Collapse
No announcement yet.

The Admin Meeting from Hell, or, Power Failures and You

Collapse
This topic is closed.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • The Admin Meeting from Hell, or, Power Failures and You

    So, as I mentioned in morons in management, I was hoping to have turned in my 2 week notice by now, and not have to attend the admin meeting yesterday. Short background: Admin meetings are pointless. They consist of 2 senior admins alternately BSing, and headbutting, and contain no information pertinent to me, or anyone but the 2 involved.

    So, I was driving down the interstate towards the new office where the admin meetings are held, and when I get off, I see all the traffic lights are out. This doesn't bode well, as the building is less than 1/4 mile from the first light that's out, and everything in all directions appears to be out. I pull up to the office, and there's developers, and various non phone people milling about outside, some playing hackey sack, some reading on their laptops, some smoking.

    When I walk in, all the lights are out, and it's 85 in the building, hotter near the data center, and 3 admins sprint by as I walk in.

    I head back to the admin office room, and ask what's up, and power has been out for 15 minutes now. Because we have no generator in the building, only the servers wired to the UPS systems are up. Everything else is down. Phones are powered via Ethernet, so they're still on, and customers are calling like mad.

    SA: Senior admin. One of the BSers in the background.
    MA: Mid Admin. A rather quiet guy I don't know much about, but is a genius.

    Me: What can I do to help?
    SA: Well...Go shut down backup boxes in the new data center (there's 2 rooms of servers here)
    Me: K.

    ...In the data center, there's no lights, save for power and ethernet lights. I'm navigating by the light of my Cell phone.

    MA: Which servers you shutting down?
    Me: Backup boxen. You?
    MA: Customer servers.

    Between the 2 of us, we get all 500 servers shut down about 20 minutes before power is restored. The data center was about 95F when we finished. Because of this mess, there was no real meeting. Navigating a data center WORSE than the one I've posted about previously via cell phone made admin meetings seem...not so bad, in fact. :-P

    As I'm leaving:

    *Lights come on*
    *Unanimous shout of "WOOOHOOOO!" is heard echoing from all parts of the building.*
    Coworker: Distro of choice?
    Me: Gentoo.
    Coworker: Ahh. A Masochist. I thought so.

  • #2
    500 servers, in what sounds like a co-location facility, and NO GENERATOR? Better yet, no backup power for the climate control either.

    That place screams "FAIL"

    Comment


    • #3
      Quoth bean View Post
      500 servers, in what sounds like a co-location facility, and NO GENERATOR? Better yet, no backup power for the climate control either.

      That place screams "FAIL"
      I work for a shared hosting company. There's a good reason I want to get out of here. :-P The data center I work in in the old Word Perfect complex (Not the provo office where tech support et al are, and where this took place) has proper backup generators, UPS, raised floors, and 70 tons of cooling split between 2 coolers - a 50 ton, and a 20 ton.

      The provo office has less than 30 tons of cooling, 2 small rooms, no raised floors, 2 UPS units for just servers, and obviously, no backup generators.

      The only reason the Orem data center has that much is because it was there when we moved in. The only reason the Provo office has as much as it does is because we had outages before where everything was down...so we bought one UPS...which only powered about half the servers. After the second outage, we bought the second UPS.

      2 generators are now on order as of today.

      I can't say much about what company without risk of losing my job. :-)
      Coworker: Distro of choice?
      Me: Gentoo.
      Coworker: Ahh. A Masochist. I thought so.

      Comment

      Working...
      X