Having issues with software quality? Have you tried test-driven development? Do you have test scripts based on test requirements? Do you have a rich sandbox with good test data? What’s your code coverage? How often is a fully tested version released into production with problems that were missed? How often does the new feature work adequately in production but with destructive side effects that integration testing missed? How long and expensive is the pipeline from programmer’s release to working production? Are you proud of being on the forefront of development methods with Agile, Scrum and the rest but still avoid releasing what comes out of each Sprint to production because you can’t risk another disaster?

If any of these apply to you, you may want to consider a decades-old, widely proven in production method of software QA that enables frequent, error-free releases to production with almost no overhead or traditional QA work. It’s not taught, there are no certifications and It’s ignored by mainstream software experts. But it works.

I used to call it “comparison-based QA.” The term is appropriate because the core of the method is comparing results of the production version of the code with a test version. A CTO with a data science background suggested a better term, which I will use henceforth: champion/challenger software QA. It’s like when you’ve got a good model, the champion, and you want to see if a new model, the challenger, yields better results. Or it’s like A::B testing of a consumer UI, when you want to see if a proposed variation works better. In both cases, you’ve done something new and you want to do two things: (1) see if the new thing works like it should, and (2) make sure nothing that used to work is broken. Sounds like feature testing and regression testing, doesn’t it? Yup!

Nobody writes test requirements or feature tests for champion/challenger. You just feed the new thing the same data you fed the old thing and compare the results. You expect differences. If something is better and nothing is worse, you could have a winner – you test with a wider range of data, and if the results hold up, the challenger becomes the new champion and you move on.

Here are the main beauties of champion/challenger:

• The tests are nothing but inputs and saved outputs of real-world processing. Nothing artificial, no carefully crafted "test data."
• You create the code for comparison of output data once. The code/model could get endlessly complex, but the output comparison identifies differences whether it has 10KB or 10GB to compare. No test scripts!
  
  • You can do the comparison with a UI with some extra work, but again one-and-done.
• The comparison pulls out all the differences. You look to see if each difference is something you expected: if everything new you expected is there, and crucially if there are any unexpected differences. If there are, you’ve got a regression failure to fix.
• You can do the comparison in batch with large samples of old data, giving you great coverage.
  
• You can do the comparison live, in a production environment, using the challenger code/model output for comparison and sending only the champion’s output to the user.
  • You can test the changes you've made on the challenger code alone, looking at the challenger code's output, to see if you're happy with the change you made. While you're doing that, the challenger code will simultaneously be running production data, which the comparison with the champion will make sure is still OK.
    
  • After you’ve done enough live parallel production testing, you can be confident that the challenger will do everything the champion did – with real-world data in the production environment — so you merely turn the switch so that challenger output becomes live and the old champion is retired.
  • You can do this in stages to crank the risk way down, so that the challenger becomes champion for just 10% of the inputs and then gradually turn up the fraction. This guarantees problem-free production releases.

I've written about about the details of how to get this done in a book.

I've written a post explaining the core concepts of the two main aspects of QA and how they're different and best served by different methods.

I've written about the path that led me to writing this and related books and summaries of the books.

I've written about how this method provides crucial fuel enabling startups to surpass tech groups hundreds of times their size along with specific strategies those groups use here and here.

The method is a "secret" — because the vast majority of industry elites choose to ignore it. Want to succeed against industry giants? Use secret methods like champion/challenger to enable rapid release of quickly evolving code with zero errors in production and minimal QA effort. You'll run circles around the lumbering, stumbling giants and then leave them in your dust.

I have found ample reason to mock the golden-glowed tech reputations of most of the tech giants, in addition to the supposed tech prowess of organizations such as the NSA. There is good reason to believe that old-style libraries are more secure.

I recently stumbled upon a rather glaring bug or result of hacking at Amazon — and found that Amazon provides no way I could find to report the problem.

The problem was glaring and amusing — a whole set of over 3,000 reviews of a book attached to a pest repelling product. As I'll describe in a future post, this isn't just a bug concerning bug repellers — it's the tip of an iceberg about the foundations of AI/ML.

Here's how I found the bug bug.

Stumbling on the bug at Amazon

I was looking for a product to scare away some pests that have found their way into my house. I gave a close look at the following product:

I immediately noticed and was impressed by the large number of reviews, and how favorable they trended. What a popular product! I've got to look at this one. I scrolled down:

Things are still looking good. More details:

Uh-oh! Right after we see that it has 3,051 reviews with an average rating of 4.8, which is super-good, we see that this pest repeller is #2,695 … in Books!! Something is seriously wrong. Scrolling down, we see some really weird answers to questions:

A couple of the people answering clearly tried to tell Amazon there was a problem.

This next one blew me away — over 3,000 reviews for a pest repeller??!!

Now we know for sure something's seriously wrong — a pest repeller is not Children's Literature!

Here is the nail in the coffin, the pictures of the product and a list of the phrases that frequently appear:

And here's one of the glowing reviews. It brought tears to the reviewer's eyes, and not because of the strong odor generated by the pest repeller:

What's going on, Amazon? I don't usually get to mock you. It's usually the other tech giants that receive my sarcasm, places like Google, Facebook, Twitter, Apple and Microsoft. Amazon has done a reasonable job maintaining quality and growing into new areas. This is such a peculiar bug — it's as though some disgruntled employee were messing with things to show his displeasure with the world.

As we'll see in a following post, this is not an isolated bug, and it has implications that go far beyond Amazon — implications concerning the glorious future promised by all the AI/ML enthusiasts.

QA is QA, right? Either you’re committed to quality or you’re not. If you are, you use the widely accepted tools and techniques and produce a high-quality product, or you accept the fact that you’ll churn out crap. What else is there to say?

Here’s what: if you accept this view and apply it in the usual software development environment, your QA processes metastasize, invade all parts of the process, and cause endless pain and suffering – not to mention crap software. Eliminating the cancer can only be done by leaving the cancer-causing environment. That means finally saying good-by to peace-time software and going to war.

Software Quality Assurance

Software QA is a BIG subject. There are lots of methods within a broad set of accepted practices. Experts assure us that “quality” goes way beyond testing; it infiltrates every aspect of the software SDLC.

But testing by itself, which is just part of quality, is a HUGE subject. You can get certified. Here are some of subjects involved in testing according a leading certification organization.

Here’s what’s involved just in test planning:

The SQA cancer

Most organizations aren’t aware of it, but the typical, expectations-based, peace-time software development organization is highly prone to QA and testing cancer.

Everybody is supposed to do good work. You’re supposed to take the flawless results of the previous group’s work, do what you do to it, and pass it on to the next group – error-free.

When it’s obvious you’ve been given crap, it’s easy to reject what you’ve been given and throw it back at the slugs who gave the crap to you. But all too often, the problems don’t emerge until you’re pretty far down the line of doing your own work. Then it’s upsetting and embarrassing. The schedule is at risk, and who’s at fault?

Once this happens a couple times, on the sending and receiving ends, most people respond by creating as many quality measure as they can, to be applied to the work they’re given. Then, having been blamed for producing bad work a couple times, they run all sorts of tests on the work they’ve produced before passing it on.
Why wouldn’t you? You don’t want to receive bad work, and you certainly don’t want to be blamed for giving bad work to someone else. But you can see how the QA/testing steps are reproducing.

Inside coding itself, it can be even worse. In addition to inspecting the code that's calling them and that they're calling, the programmers can "enhance" their code with additional code whose sole purpose it to check whether the inputs they're given — at run-time — are correct, and then again to test the outputs they're about to give.
As though that isn't enough, lots of quality-minded programmers are way into writing code that tests each individual piece of code; this can be before you write the code (i.e., test-driven development, which is currently fashionable, something programmers say they want to do to show everyone else how good they are) or after you write the code (i.e., plain old unit testing).

What's worse is, I've just skimmed the surface!

There is no escape

You think that's bad? Try to do something about it. Whenever anything goes wrong, you're likely to hear one of the following:

• I wasn't allowed to buy adequate test tools
• We weren't adequately trained on the tools
• I didn't have enough time to get enough coverage
• Lots of old tests had bugs, I had to disable them to meet the deadline
• We changed so much old code, I didn't have time to update all the tests for it.
  
• My regression testing has poor coverage, I need more time/money
• We have too much emphasis on testing — quality is something we need to build in

Then you'll find a group that isn't testing inputs or outputs. More time and money needed.

Get mad and remove any of these steps? You've just given the group a "get out of jail free" card if/when something goes wrong.

Cancer-free testing and quality

Testing cancer is one of the main reasons why peacetime software is a lengthy, organized expensive process … that produces disastrous results. Then, when you try to fix the problem so it doesn't happen next time — the cancer spreads. Current quality and testing methods are a cancer on software development, and there is only one known cure.

The cure is simple. It's documented. It's proven in practice, at many places over many years. But it's radical, and requires a shift to wartime software thinking. It requires shifting to global process optimization, and optimizing for speed. It takes less time and produces far superior results. Here are some of the key points:

• Forget "quality," whatever that is. Concentrate on testing.
• Move from correctness-based testing everywhere to change-based testing at just a couple key points.
• Move all testing from the lab to production.
• Move all testing from periodic to continuous.

There are a number of posts in this blog that spell this out. There are even books! The advance of software QA cancer is inexorable and unstoppable in environments that are friendly to it. There is only one cure: Wartime software and its associated methods.

Software Quality Assurance (QA) has grown wildly complex and specialized, while at the same time becoming increasingly ineffective. It's time to question a couple key assumptions and to establish and simple, clean, unifying framework to get the results we want from SQA: "get it right" and "keep it right."

Software QA Today

There's a huge amount going on with software QA. For example, the American Society for Quality has a Certification for Software Quality Engineer.The test has about 160 subject areas. Here is a sample of just four of them:

On one of the many websites devoted to software QA, I found links to over 100 recommended books.

Assumptions

If you accept the assumptions about software development that underly most modern QA, it doesn't seem completely insane. However, I challenge several of the core assumptions.

It is a historical fact that much of modern software development theory, including QA, is based on the notion that building software is somehow like a manufacturing process. There are a couple edge cases in which this assumption is useful; but for the general case, it's just wrong and should be discarded.

It is similarly clear that most modern software development theory, including QA, is based on the notion that the root fact of software, the fact from which everything else grows, is a set of requirements. There are important cases in which this assumption is useful (a large number of them are algorithms, think Knuth); but for most software, it causes more trouble than anything else, and should be forgotten like a bad dream.

Finally, much software is developed with the notion of predictability as central. This is a choice organizations make. If they want predictability, let them have it; it's a valid (if mostly brain-dead) choice. But for most of the software that makes a difference, speed is more important than predictability.

Instead of these assumptions, it is more fruitful in most cases to assume the following:

• Software is primarily a design process
• The requirements of the software evolve as you build it
• Speed is more important than predictability.
  

Core Software QA Function: Keep it Right

"Keep it Right" is one of the two major categories of software QA. It has its own distinct methods. It can and should be largely automated.

The core, transformative observation here is that your software need not (yet) be correct. When you alter the software, the main thing you want is to avoid changing its behavior (except as intended). The killer in all regression testing software is that you have to go function by function and write scripts to make sure the "right" thing happened. So your work is proportional to the size and complexity of your program.

Once you understand the role of "Keep it Right," you see that all you need to do is compare what the software did before with what the changed version does. There are always way to accomplish this. The key observation here is that writing comparisons (think diff) is something you do just once, for the point at which the differences in the software are being compared (for example, the UI or the DBMS). So your work is independent of the size and complexity of your program.

The key method of "keep it right" is to replace "correctness testing" with "comparison testing." This can be completely automated; it reduces programmer overhead; and it does what users do, viz., ask the question "what's different?"

Core Software QA Function: Get it right

"Get it Right" is one of the two major categories of software QA. It has its own distinct methods. It cannot be automated to any significant extent.

Scripts or automation of any kind are irrelevant to "get it right." Instead of having product people write requirements that someone down the line eventually turns into test plans and test cases, the people who would normally write requirements watch the software's behavior as it grows in capability and guide its development interactively. They see whether this iteration of the software does what they expected, and they also watch their own reactions as they use it, learning from those reactions. Outside people may also play a part in this process.

Meanwhile, of course, "keep it right" is running all the time, assuring that only forward steps are taken, and that unintended side-effects are caught so they can be corrected. Once the new round of changes have been "gotten right," they become part of the base set that is "kept right" using the comparison-based test-for-change method.

Conclusion

Keep it Right and Get it Right alter the organization, methods, work and results not just of QA, but of the entire software-producing organization. They change who does what and how they do it. When done reasonably well, they align quality efforts with customer expectations, which nearly always are based on the hopeful assumption that at least you will avoid breaking what used to work. At least as important, your product managers and programmers can "goal seek" their way rapidly to an effective solution with minimal overhead.