Announcement

Collapse
No announcement yet.

That time we took down a bunch of banks

Collapse
This topic is closed.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • That time we took down a bunch of banks

    So, years ago when dinosaurs roamed the earth, and some of us wrote software for them, I worked for a US Government Agency. Enough time has passed, and enough brain cells euthanized, that many of the details are fuzzy or outright wrong, so please forgive any of those details that are bogus. The part of the Agency I worked for performed $(SPECIFIC_TASK), and was moving away from specialized equipment to these cool shiny new UNIX workstations.

    One of the cool things we could do with those was have them all talk together over Ethernet, rather than using serial connections between specific endpoints. That made it a lot easier to share information, to distribute software, and so on.

    Naturally, our work needed to be separated from the Outside World, so our Ethernet network was purely in-house and was set up however we felt like. (Foreshadowing: Allowing non-network software engineers to set up your network is Not A Great Idea.)

    This did work well for our internal use, and everyone was happy. Then some bright spark decided that it would be cool if we had a way to connect to the Outside World, over this cool Internet thingie. That would allow us to transfer data to other Agency sites and to get data from them more easily. However, we didn't want to just put all of our systems Out There, so we picked a dedicated machine and gave it two network interfaces, one on the internal network and one on the Internet. The system was set up to not allow direct traffic between the networks, just to keep everyone safe from everyone else.

    Due to the details of this setup, there were some peculiarities in how the system needed to be brought up. If done wrong, it could bridge the two networks and cause havoc in our internal one. Of course, it turned out that wasn't the only havoc it could cause...

    One day, there was a need for that particular system to be restarted, so one of the hardware guys wandered over to do that. While he did know how to deal with most of our UNIX systems, he had not been educated on the peculiarities of this dual-network server. During start-up, the system did all of the expected things, like ARP and IP advertising. ("Hey everyone, this is me and this is my IP address!") Our internal network got confused because all of a sudden there was an IP from a completely different IP range being advertised, and someone quickly had to run and shut the thing down and re-start the proper way.

    A few days later, we heard what had happened on the other side of that. It turned out that the internal IP address we had picked for the system was the same as some fairly important piece of network equipment somewhere in South America. And when our system came up and started telling everyone who it was, the South American network got very confused as well. So we wound up disconnecting a fairly large chunk of South American banking network due to our screw-up.

    At that point, our software people looked into the whole IP standard and worked out what we should have done in the first place, which was to pick addresses from the "local only" segments of the IP range for the internal network.

    Our apologies to anyone who had banking activity that was messed up that day.
    “There are two novels that can change a bookish fourteen-year old’s life: The Lord of the Rings and Atlas Shrugged.
    One is a childish fantasy that often engenders a lifelong obsession with its unbelievable heroes, leading to an emotionally stunted, socially crippled adulthood, unable to deal with the real world.
    The other, of course, involves orcs." -- John Rogers

  • #2
    The Register (theregister dot co dot uk) would like this story for their "On Call" or "Who, Me?" features.


    example
    I am not an a**hole. I am a hemorrhoid. I irritate a**holes!
    Procrastination: Forward planning to insure there is something to do tomorrow.
    Derails threads faster than a pocket nuke.

    Comment


    • #3
      Why is it

      There is always it seems some guy/girl who decides they know it all and go ahead to do something they have never done before but still do not bother to ask around if anyone knows if there is any special knowledge they need to know first.

      A good chunk of the repairs I used to do were because someone wrecked/damage a machine/software/database because they did not think or ask around and just went ahead on how they thought things worked. This really was a mess when they tried their ideas on a totally different computer PC<->Mac or Xenix<->VAX, yes that last one was not a mistake on my part, it was told to me from a friend who worked on VAX machines and the new hired lied about his skills and had only worked with Xenix.

      Comment


      • #4
        Heh, some of the systems we were using at the time of the above story were DPD-11s. As in, the "little brother" of the VAXen. We did have a few VAX systems that we used for supporting tasks. Eventually some of the Main Tasks were moved to "microVAX" systems. Those were our upgrades, mind you, and that was significantly after the incidents mentioned above.
        “There are two novels that can change a bookish fourteen-year old’s life: The Lord of the Rings and Atlas Shrugged.
        One is a childish fantasy that often engenders a lifelong obsession with its unbelievable heroes, leading to an emotionally stunted, socially crippled adulthood, unable to deal with the real world.
        The other, of course, involves orcs." -- John Rogers

        Comment


        • #5
          Quoth Nunavut Pants View Post
          Heh, some of the systems we were using at the time of the above story were DPD-11s.
          *cough*PDP-11s*cough*.

          Comment


          • #6
            Oops, typo. Yes, PDP. Coded on them using KED, an interesting modal text editor that used the number pad keys for a lot of stuff.
            “There are two novels that can change a bookish fourteen-year old’s life: The Lord of the Rings and Atlas Shrugged.
            One is a childish fantasy that often engenders a lifelong obsession with its unbelievable heroes, leading to an emotionally stunted, socially crippled adulthood, unable to deal with the real world.
            The other, of course, involves orcs." -- John Rogers

            Comment

            Working...
            X