Go Back   Customers Suck! > The Heart of the Site > Unsupportable

Thread Tools Display Modes

That time we took down a bunch of banks
Old 06-09-2020, 01:33 AM
Nunavut Pants's Avatar
Nunavut Pants Nunavut Pants is offline
Warning: He thinks he's funny.
Join Date: Nov 2015
Posts: 1,083
Default That time we took down a bunch of banks

So, years ago when dinosaurs roamed the earth, and some of us wrote software for them, I worked for a US Government Agency. Enough time has passed, and enough brain cells euthanized, that many of the details are fuzzy or outright wrong, so please forgive any of those details that are bogus. The part of the Agency I worked for performed $(SPECIFIC_TASK), and was moving away from specialized equipment to these cool shiny new UNIX workstations.

One of the cool things we could do with those was have them all talk together over Ethernet, rather than using serial connections between specific endpoints. That made it a lot easier to share information, to distribute software, and so on.

Naturally, our work needed to be separated from the Outside World, so our Ethernet network was purely in-house and was set up however we felt like. (Foreshadowing: Allowing non-network software engineers to set up your network is Not A Great Idea.)

This did work well for our internal use, and everyone was happy. Then some bright spark decided that it would be cool if we had a way to connect to the Outside World, over this cool Internet thingie. That would allow us to transfer data to other Agency sites and to get data from them more easily. However, we didn't want to just put all of our systems Out There, so we picked a dedicated machine and gave it two network interfaces, one on the internal network and one on the Internet. The system was set up to not allow direct traffic between the networks, just to keep everyone safe from everyone else.

Due to the details of this setup, there were some peculiarities in how the system needed to be brought up. If done wrong, it could bridge the two networks and cause havoc in our internal one. Of course, it turned out that wasn't the only havoc it could cause...

One day, there was a need for that particular system to be restarted, so one of the hardware guys wandered over to do that. While he did know how to deal with most of our UNIX systems, he had not been educated on the peculiarities of this dual-network server. During start-up, the system did all of the expected things, like ARP and IP advertising. ("Hey everyone, this is me and this is my IP address!") Our internal network got confused because all of a sudden there was an IP from a completely different IP range being advertised, and someone quickly had to run and shut the thing down and re-start the proper way.

A few days later, we heard what had happened on the other side of that. It turned out that the internal IP address we had picked for the system was the same as some fairly important piece of network equipment somewhere in South America. And when our system came up and started telling everyone who it was, the South American network got very confused as well. So we wound up disconnecting a fairly large chunk of South American banking network due to our screw-up.

At that point, our software people looked into the whole IP standard and worked out what we should have done in the first place, which was to pick addresses from the "local only" segments of the IP range for the internal network.

Our apologies to anyone who had banking activity that was messed up that day.
“There are two novels that can change a bookish fourteen-year old’s life: The Lord of the Rings and Atlas Shrugged.
One is a childish fantasy that often engenders a lifelong obsession with its unbelievable heroes, leading to an emotionally stunted, socially crippled adulthood, unable to deal with the real world.
The other, of course, involves orcs." -- John Rogers
Reply With Quote

Old 06-09-2020, 02:44 AM
dalesys's Avatar
dalesys dalesys is offline
Pune Massacreeist
Join Date: Nov 2008
Location: Salsa Lake City, UT, USA
Posts: 7,142

The Register (theregister dot co dot uk) would like this story for their "On Call" or "Who, Me?" features.

I am not an a**hole. I am a hemorrhoid. I irritate a**holes!
Procrastination: Forward planning to insure there is something to do tomorrow.
Derails threads faster than a pocket nuke.
Reply With Quote

Why is it
Old Today, 01:25 PM
earl colby pottinger earl colby pottinger is offline
Area Manager
Join Date: Feb 2007
Posts: 1,749
Default Why is it

There is always it seems some guy/girl who decides they know it all and go ahead to do something they have never done before but still do not bother to ask around if anyone knows if there is any special knowledge they need to know first.

A good chunk of the repairs I used to do were because someone wrecked/damage a machine/software/database because they did not think or ask around and just went ahead on how they thought things worked. This really was a mess when they tried their ideas on a totally different computer PC<->Mac or Xenix<->VAX, yes that last one was not a mistake on my part, it was told to me from a friend who worked on VAX machines and the new hired lied about his skills and had only worked with Xenix.
Reply With Quote

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT. The time now is 08:27 PM.

Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2020, vBulletin Solutions, Inc.