Announcement

Collapse
No announcement yet.

Why hasn't their customer fired them yet?

Collapse
This topic is closed.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Why hasn't their customer fired them yet?

    BTW, this is gonna be long.

    Anyone who works in or even near IT has probably heard of certain consulting companies, of a certain nationality, and their legendary incompetence. So far as I can tell, the full extent of their capabilities is opening tickets, during the course of which they are infamously pushy, especially when they're in the wrong. (Amusingly, one of these companies has a name that in certain regions, can be slang for female breasts. Quite apropos, when things go tits-up, IMO.)

    This however, is a story of a smaller such company, which I had never heard of until now. And I have to wonder how they even managed to figure out how to open a ticket. By the level of incompetence shown, I can just imagine them shouting, "Raise a case! Raise a case! Kindly check and revert at the earliest!" at their computer screens, thinking it will contact support. But anyway, let's get to it.

    It all started with this company deploying an ancient, buggy version of our WAF via AWS On Demand--meaning, they pay Amazon, and Amazon pays us, and there are certain restrictions which will be important later--and finding their website doesn't work properly.

    Alright buddy, who allowed you to use this old version anyway? Replace it post-haste, there are long-known and resolved bugs in this old crap that could easily cause this kind of problem.

    And so they do. Well, they upgrade the gateways, which is the important part. Unfortunately, the issue persists, but at least we've ruled out (quite literally) over a half-dozen bugs that could've been responsible for it. Now that they're on the latest and greatest, it's time to get serious about things. Difficulty: since they're "On Demand" customers, our boxes are sealed. They could be unsealed, but the ID number they provided doesn't bring up anything, so we can't get a key to unseal them. Can't activate debug traces while sealed. Doh! But we have a backup plan.

    We'll need to get packet captures on client and server side, as well as log bundles before and after the test. Do two sets of captures: one going through the gateway, one direct to the server.

    What do I get? Oh, I get my pcaps alright, but the client-side ones are HTTPS, and no keys of any kind were provided. Useless. They only included one log bundle, done after the test. The server-side captures are, amazingly, plain HTTP, so I spend quite a lot of time reading through and comparing them to see what the hell is wrong. During this process, they consulting firm keeps pestering us for updates, as if they haven't delayed their case more through their own incompetence every time a minor speedbump was hit (how the hell did you lose your SSH key for an AWS instance that only allows key-based login?!). It took me a little over a day, but I found what I thought to be the smoking gun, and a potential fix. So back to the consulting firm I go. I explain the problems with the previous dataset, explain the workaround, and in case it doesn't work I ask explicitly, for SSL session keys to be collected, providing instructions on how to get them.

    In response, I get the following:
    • Before and after log bundles (YAY they did something right)
    • Two sets of client and server pcaps, (Yay again!)
    • Not one, not two, but THREE different private SSL keys, two of which are encrypted, and one password (okay wtf?)
    • No context as to which of the above keys is which, or which key the password goes with (oh ffs)


    Leaving aside the fact I could now set up a server to impersonate theirs, and that the password only worked for one of the encrypted keys, the client-side pcaps were still not decryptable because the client used ECDHE ciphers. It's called Perfect Forward Secrecy, and means that even though I have the keys to the kingdom, they don't work to decrypt previously-captured communication. There's a reason I went to the trouble of providing instructions for getting session keys, dammit. Not to mention, their screw-ups with the pcaps and keys, were fully visible to their end customer (a European pharmaceutical company, I think).

    But wait, there's still more to come. Despite this handicap, I manage to build my case to the dev team and they don't push back for not having the client-side stuff. Unfortunately, after some days of analysis, they can't find the issue in the code, or reproduce the issue. They ask for the customer's export.

    We tell them so, and ask for the export. Again, since they are running restricted versions, their options for doing the export are limited. We provide specific instructions (again) complete with examples. Mind, their customer is on the email thread at this point, because this has been going on for a while.

    The consulting firm runs the wrong command (a different export command), it runs to completion successfully, and they email us saying "It didn't work". (The export it produced would've been useless to us, but they didn't know that).

    We tell them again, exactly what to do. It takes them days to get back to us. And when they do, we find that, while they didn't technically screw it up this time, the problem they had was one they could've solved for themselves literally in minutes. We advise them of the issue, and how they can work around it. That was yesterday morning. This morning, they sent this comment: "we're very unhappy with support and the way this has been handled. Kindly check your lab environment and share us the right command".

    Excuse me while I laugh my ass off. They tried the same exact thing, and it failed the same way, after we told them why it was failing, and what they should do about it. And they blame it on us.

    Soon, it'll be over a week that the consulting firm's incompetence will have delayed the investigation just over this one issue of getting the export, nevermind the issues gathering the original troubleshooting data. I really have to wonder why hasn't their customer fired them yet?

    This one is ongoing. Possibly more to come!
    Supporting the idiots charged with protecting your personal information.

  • #2
    Offshore is *supposed* to mean "completely at sea" ?
    I am not an a**hole. I am a hemorrhoid. I irritate a**holes!
    Procrastination: Forward planning to insure there is something to do tomorrow.
    Derails threads faster than a pocket nuke.

    Comment


    • #3
      You've probably heard "Do the needful" more times than is healthy.

      Comment


      • #4
        I'm sure that I have...
        “There are two novels that can change a bookish fourteen-year old’s life: The Lord of the Rings and Atlas Shrugged.
        One is a childish fantasy that often engenders a lifelong obsession with its unbelievable heroes, leading to an emotionally stunted, socially crippled adulthood, unable to deal with the real world.
        The other, of course, involves orcs." -- John Rogers

        Comment


        • #5
          ^ Ditto.

          Update on this: the problem wasn't even us. It was their AWS load balancer. If they had provided a proper ID in the first place, we could've proved that much earlier, much more easily.

          They haven't replied to my mail explaining the source of the problem. I bet they'll still try to blame us somehow.
          Supporting the idiots charged with protecting your personal information.

          Comment


          • #6
            Quoth otakuneko View Post
            ^ Ditto.

            Update on this: the problem wasn't even us. It was their AWS load balancer. If they had provided a proper ID in the first place, we could've proved that much earlier, much more easily.

            They haven't replied to my mail explaining the source of the problem. I bet they'll still try to blame us somehow.
            You mean you're not telepathic?
            "I don't have to be petty. The Universe does that for me."

            Comment


            • #7
              Nah, but I am psychic: they screwed up configuring their new load balancer and then tried to blame us when things didn't work. It kinda helps when you configure the load balancer to actually send traffic to a port that's open. You know, the one our WAF is expecting connections to come in on, that you chose yourself? That one? (/facepalm)

              And it appears their customer hasn't fired them because they're not paying attention at all. They replied on the email thread asking for updates, but they replied to an old email. My manager shut them up real quick by just copying them on the current email.
              Supporting the idiots charged with protecting your personal information.

              Comment


              • #8
                Aaaand an update: they finally got that issue resolved, but then promptly turn around and open another new case.

                This time they say they have created a policy to block some particular URLs. Right off the bat, they want a remote session to troubleshoot. Because of course--they want us to do their job for them. Not happening. Not on my team, not for this company, not on my watch. I tell the engineer assigned to send them some basic advice and ask for a screenshot of the policy.

                So they send said screenshot (and another request for remote session) and what do we find?

                The screenshot isn't of a security policy at all. It is a completely different area of the UI which is designed to help with some other tasks, but happens to have fields for hostnames and URLs.

                The stupid, it burns.

                I helped the engineer craft a reply explaining to them why they are wrong, what to do to accomplish their goal, and provide doc links. We also forward the issue to management and the company's account team. Support is not here to train you or do your job for you. We do offer training--as a separate, professional services engagement, billable by the hour.
                Supporting the idiots charged with protecting your personal information.

                Comment


                • #9
                  Wow. Just...wow. Have a cookie
                  My son thinks I'm Lucifer Morningstar. I'm not sure he's wrong.

                  Comment

                  Working...
                  X