Announcement

Collapse
No announcement yet.

Tests? We don't need no stinkin' tests!

Collapse
This topic is closed.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Tests? We don't need no stinkin' tests!

    My company is building $(BIG_COMPLEX_SOFTWARE_PRODUCT). As part of that, we have a MANDATORY set of automated tests that MUST BE RUN before every checkin of code. That is even enforced by the tests returning an encrypted token that must be included in your checkin command, FFS.

    Turns out that those tests haven't been actually passing for ... oh, a month? Longer? I can't tell how far back it goes, because the results web page loads SO SLOWLY.

    I have asked for advice, and I was told by other developers, "Oh, just ignore the results. If it fails in Step 10, you're fine and should check in."

    OH NO YOU DI'INT!

    Seriously, folks. We have these tests for a reason. Do you think, perhaps, one of the reasons that this development branch is always broken just might be because lots of people are checking in without running complete test suites? Even if this test isn't going to catch many of the problems that do get checked in, we now have devs that are completely OK with checking in code that doesn't pass testing. This is a dangerous attitude and really really needs to be stopped RIGHT NOW.

    I swear to Bog, some of these people need a Boot To The Head!
    “There are two novels that can change a bookish fourteen-year old’s life: The Lord of the Rings and Atlas Shrugged.
    One is a childish fantasy that often engenders a lifelong obsession with its unbelievable heroes, leading to an emotionally stunted, socially crippled adulthood, unable to deal with the real world.
    The other, of course, involves orcs." -- John Rogers

  • #2
    I can so relate. Had to troubleshoot a production problem today (by the way, I am on sick leave). B2B web site that was shutdown several months ago when customer left. We have a new customer. Been testing in QA for a while now. Went live on the prod site today.


    Nobody checked to see if the prod site was still functional.

    Life is too short to not eat popcorn.
    Save the Ales!
    Toys for Tots at Rooster's Cafe

    Comment


    • #3
      Have fun tracking down the changes that result in test failures. Always fun when you've got a bunch of changes in between.

      Comment


      • #4
        Fun fact that I didn't know before:

        We have an email list called "(this specific test suite) tiger team". I do wish one of them had actually bothered to look at the test results some time in the last month or so; they might have actually started doing something about it at that point!!

        So far all we have is a bunch of "well there are a lot of different things that could cause this to fail", "we should increase the timeout so the test might pass", "let's keep the system alive for 6 hours so someone can troubleshoot" (uh, most of these are run overnight so they're gone by the time we get back in!!), and so on. Well, it's possible that someone will actually track down the source of the error if they keep the system around...
        “There are two novels that can change a bookish fourteen-year old’s life: The Lord of the Rings and Atlas Shrugged.
        One is a childish fantasy that often engenders a lifelong obsession with its unbelievable heroes, leading to an emotionally stunted, socially crippled adulthood, unable to deal with the real world.
        The other, of course, involves orcs." -- John Rogers

        Comment


        • #5
          Since nobody can be bothered to make sure their code passes, the problems can be ANY FREAKIN WHERE in the whole program. And, of course, now that the stupidity has come to light, the greatest idea anyone can have is to pull the equivalent of putting fingers in ears and "La La La, can't hear you".
          Seriously, the code has GOT to have such glaring holes in it that... well, I'm sure the test suite is likely more stable and dependable than the friggin product right now. (Yes, I'm aware that the test suite is non-functional as it is)

          I'm thinking that the product code is going to have to be gone through from start to finish, every last module, from this crap.
          Last edited by Buzzard; 02-02-2019, 03:01 AM. Reason: Helps if I write the whole thing instead of just thinking I did.

          Comment


          • #6
            Since the end uses will end up being the beta testers to find of the problems with the software, I know which company you work for.

            I've narrowed it down to several dozen based on the programs and games I've used.
            "I don't have to be petty. The Universe does that for me."

            Comment


            • #7
              Nunavut...

              What testing suite are you using? Unit Tests should run fast. Even if there are hundreds or thousands of them, they shouldn't time out your system.

              Personally, if you're using a web page, you're probably doing it wrong. I know Visual Studio has built-in testing tools. And if you use NUnit, you can even create test cases for each test.

              It also sounds like if they're using a deployment script, that the deployment script may need work. They should have a policy in place (I don't know why they don't) that if the tests don't pass, the code won't build (or something like that). And in that case, it should kick back to the dev team or project manager to let them know.

              Unit tests should be small, fast, and have no conditional logic in them, except the "built-in" logic of the assert statements.
              Skilled programmers aren't cheap. Cheap programmers aren't skilled.

              Comment


              • #8
                We don't actually have unit tests. We have three different layers of test architecture that call themselves unit tests, but they are actually full-on functional tests. As in "deploy the software to an actual device, then throw commands and data at it and see if it does the right things".

                Of course, the whole thing is a huge mish-mash of homebrew scripts in six or seven different languages, ranging from Perl to Ruby to bash to Python to Curl scripting (blanking on the name for that, maybe JCL?) to C-language tests that get compiled and run once per decade whether they need it or not!

                To say that I am not overly enamored with the way we do stuff would be a bit of an understatement...
                “There are two novels that can change a bookish fourteen-year old’s life: The Lord of the Rings and Atlas Shrugged.
                One is a childish fantasy that often engenders a lifelong obsession with its unbelievable heroes, leading to an emotionally stunted, socially crippled adulthood, unable to deal with the real world.
                The other, of course, involves orcs." -- John Rogers

                Comment


                • #9
                  Wow. That sounds like haphazard "Integration Testing".

                  Is there anyone in your office you could convince that Unit Testing is a good idea? Start the ball rolling. I did it where I work, and they were rather receptive.
                  Skilled programmers aren't cheap. Cheap programmers aren't skilled.

                  Comment


                  • #10
                    There are quite a few people who agree that it is a fabulous idea, and that Somebody Else should put a whole lot of time and effort into developing it.

                    I tried to be that Someone Else for a little while, but wound up unable to put together anything useful...
                    “There are two novels that can change a bookish fourteen-year old’s life: The Lord of the Rings and Atlas Shrugged.
                    One is a childish fantasy that often engenders a lifelong obsession with its unbelievable heroes, leading to an emotionally stunted, socially crippled adulthood, unable to deal with the real world.
                    The other, of course, involves orcs." -- John Rogers

                    Comment


                    • #11
                      It can be difficult to do. I joined my current company about 18 years ago, and they didn't really have a Unit Test framework at the time. My job before it was heavy on Unit Testing and I did miss that framework at the time, but I was too junior to push it back then. Still a few years later, some of our teams started developing unit tests and making it our policy. There was some resistance, and it's taken a long time to get decent coverage. But now, everyone is grateful we have the coverage we have for the tests. It has allowed us the freedom and confidence to do some major refactoring and updating of our systems and know we haven't broken things worst than it was before.

                      At the moment the code base I work in the most has about 25,000+ unit tests; takes about 5 minutes to run fully without debugging in release. And some of our other big modules have 10k tests as well; all written in the past dozen years or so.

                      Comment


                      • #12
                        Quoth Nunavut Pants View Post
                        There are quite a few people who agree that it is a fabulous idea, and that Somebody Else should put a whole lot of time and effort into developing it.
                        The problem with that approach is that they usually want that Somebody Else to pay for it.
                        "I don't have to be petty. The Universe does that for me."

                        Comment


                        • #13
                          Quoth Jetfire View Post
                          ... But now, everyone is grateful we have the coverage we have for the tests. It has allowed us the freedom and confidence to do some major refactoring and updating of our systems and know we haven't broken things worst than it was before.
                          "The key to fearless refactoring is good test coverage."

                          Sounds like your current job is much like my first group at $(MAJOR_NETWORKING_COMPANY). We had good coverage, support from management to require people to test, and good people doing the work.

                          ... so of course the mandate came down from Very Upper Management that we had to increase our release cadence and reduce head count by half. The top guy in our $(BUSINESS_UNIT) told them it couldn't be done, so they just closed down $(BUSINESS_UNIT) and got out of the $(PRODUCT) market all together.
                          “There are two novels that can change a bookish fourteen-year old’s life: The Lord of the Rings and Atlas Shrugged.
                          One is a childish fantasy that often engenders a lifelong obsession with its unbelievable heroes, leading to an emotionally stunted, socially crippled adulthood, unable to deal with the real world.
                          The other, of course, involves orcs." -- John Rogers

                          Comment


                          • #14
                            Quoth Nunavut Pants View Post
                            There are quite a few people who agree that it is a fabulous idea, and that Somebody Else should put a whole lot of time and effort into developing it.

                            I tried to be that Someone Else for a little while, but wound up unable to put together anything useful...
                            There are lots of useful websites and such as to unit testing online. I know it's in C#, but one book I'd recommend is Roy Osherove's "The Art of Unit Testing". There are plenty of others, too.
                            Skilled programmers aren't cheap. Cheap programmers aren't skilled.

                            Comment


                            • #15
                              Going along with the testing theme--

                              My company recently released a new version of our agent which was supposed to support a specific version of Hive. A very big and influential customer has been waiting for this for quite some time. You can probably guess where this is going.

                              It didn't work. There were multiple issues with it, including in some cases crashing the server, but I dealt with one that was a real beauty. No crash, but it just plain didn't work. Threw a bunch of errors and sat in the corner pouting.

                              The problem? Oh, there was a code change in Hive, wherein a big B was changed to a little b. No big deal. It only broke our agent, which was supposed to support this specific version, which the customer had been waiting on for I don't even know how long.

                              Is it really a necro, when the thread is only #2 on the page?
                              Supporting the idiots charged with protecting your personal information.

                              Comment

                              Working...
                              X