Announcement

Collapse
No announcement yet.

118°

Collapse
This topic is closed.
X
X
Collapse
 
  • Page of 1
  • Filter
  • Time
  • Show
Clear All
new posts
Previous template Next
  • #1

    118°

    That's how hot it got in my school's server room last night.

    Around 4:30 p.m. yesterday, the compressor in the server room's primary AC went out. The compressor in the backup failed shortly afterward (conspiracy? We'll never know). Three hours later, error messages finally started going out. It took until an hour ago to get most of the servers up and running again, and we're still doing damage control trying to figure out what hasn't rebooted correctly. It also appears that one of the database servers was being funky to begin with, and this disaster was what finally revealed it.

    Strangely, I consider this fun. It's only the third day of my internship and I'm already dealing with something new. Something broke, and I get to help fix it. I'm getting to meet way more people than normal because everyone's in and out trying to figure out what's going on. I get an experience no other student gets and I get to have insider information on what's going on. It's just... fun.
    The fact that jellyfish have survived for 650 million years despite not having brains gives hope to many people.

    You would have to be incredibly dense for the world to revolve around you.
    Tags: None
  • #2
    118° That's the typical temp of the sauna in the gym I frequent
    Good to see that you're not seeing a bus underside over this.
    I'm trying to see things from your point of view, but I can't get my head that far up my keister!

    Who is John Galt?
    -Ayn Rand, Atlas Shrugged

    Comment

    • #3
      Might want to suggest that they get some temperature monitoring in there. We have multiple remote locations that we monitor and alarm that has saved our asses from having toasted servers, controllers and switches many times.
      Just sliding down the razor blade of life.

      Comment

      • #4
        Quoth taxguykarl View Post
        Good to see that you're not seeing a bus underside over this.
        We wouldn't, we're all coders/programmers and have nothing to do with the hardware, especially the AC in the server room. It's maintenance that's taking the heat for this (literally), since they apparently failed at ensuring that a Very Important Thing (TM) wouldn't fail.

        Quoth drunkenwildmage View Post
        Might want to suggest that they get some temperature monitoring in there.
        From what I understand, they do have temperature monitoring, and it's the only reason nothing is permanently messed up. I don't have exact details but if I'm right, the error messages went out well before it actually overheated, which gave the IT guys time to get there and shut everything down before it screwed up for good.
        The fact that jellyfish have survived for 650 million years despite not having brains gives hope to many people.

        You would have to be incredibly dense for the world to revolve around you.

        Comment

        • #5
          Quoth Aragarthiel View Post
          We wouldn't, we're all coders/programmers and have nothing to do with the hardware, especially the AC in the server room. It's maintenance that's taking the heat for this (literally), since they apparently failed at ensuring that a Very Important Thing (TM) wouldn't fail.
          The backup for the server room at my internship was... interesting. The server room was connected to the IT office. If the AC went out and the temp got above a certain number, two vents would open. One in the server room wall, the other in the IT office wall. Strong fans then turned on to exhaust the hot air through the IT office, outside. Oh, and the moisture that accumulates in the AC unit trickled down a small PVC pipe and dripped into the IT office sink.

          Also, I don't feel I learned very much from my internship other than "teachers be crazy" and that I hate WSUS. And that I have forgotten most useful things that I learned in my program. Yay.
          Replace anger management with stupidity management.

          Comment

          • #6
            Quoth notalwaysright View Post
            The backup for the server room at my internship was... interesting.
            Our server room is across the hall and through a storage closet from our office, and we got lucky that the backup plan meant big white tube-y things in the hallway pumping air in and out of the room.

            I actually have a really good internship. The guys I'm working under are great, I'm learning a lot, and even if I don't know the code itself (they use a lot of PHP and I'm just starting to learn that), I understand the logic enough that I can help point out mistakes. When they don't have anything for me to do, they're also available to help me with my current classwork. So it's not like I'm fetching coffee or filing papers or anything.
            The fact that jellyfish have survived for 650 million years despite not having brains gives hope to many people.

            You would have to be incredibly dense for the world to revolve around you.

            Comment

            • #7
              Quoth Aragarthiel View Post
              Strangely, I consider this fun. It's only the third day of my internship and I'm already dealing with something new. Something broke, and I get to help fix it. I'm getting to meet way more people than normal because everyone's in and out trying to figure out what's going on. I get an experience no other student gets and I get to have insider information on what's going on. It's just... fun.
              My 1st day at the jewelry company (basically almost as soon as I walked in the door and sat down at my desk) I found out the company system (an OLD IBM System/36) had killed a hard drive. OH WHAT fun that was. and yes I did learn a LOT about that particular piece of hardware. I leaned how to reload the OS, reload ALL the files (then near the end the backup tape drive also died), run all of the relevant programs to reintegrate all of the back logged data into the databases.

              all told we were down for 3 or 4 days cause things kept failing. Quite a learning experience for my first week.
              I'm lost without a paddle and headed up SH*T creek.
              -- Life Sucks Then You Die.

              "I'll believe corp. are people when Texas executes one."

              Comment

              • #8
                Our old data center had three chillers for the AC. One failed on a Thursday. Parts would be in on Monday. On Saturday, we started receiving thermal alerts from our servers. One of the other chillers had failed. We started shutting down all unnecessary equipment (Dev, QA, hot backups). Thankfully, it was January. They opened all the doors. Not sure how they moved the air. They were able to fix the second chiller in a few hours.
                Life is too short to not eat popcorn.
                Save the Ales!
                Toys for Tots at Rooster's Cafe

                Comment

                • #9
                  Quoth Aragarthiel View Post

                  From what I understand, they do have temperature monitoring, and it's the only reason nothing is permanently messed up. I don't have exact details but if I'm right, the error messages went out well before it actually overheated, which gave the IT guys time to get there and shut everything down before it screwed up for good.
                  Could also be that the servers started tossing out alarms from the heat. I'm surprised it could get that warm, some of our stuff starts borking when it gets to be around 100 degrees air temp. All our remote sites are either cylinder block, or concrete buildings about the size of a 1 car garage. so they can hold heat in pretty good. Even in the dead of winter we have to run our A/C in most of those buildings.

                  Even with the redundant systems in our remote sites, we have a HVAC contractor on call with us 24/7 365. Talking to one of the workers, they said that they have a rack with just extra parts for just for us and our systems at their officers.
                  Just sliding down the razor blade of life.

                  Comment

                  Previous template Next
                  Working...
                  X