BDD and The Testing Matrix

2021-07-07 21-07-07
by Ken Pugh



The tests you create using Behavior Driven Development represent only a portion of the tests for a system.  There are a number of other tests that should be executed in order to ensure that the system reflects quality.       

The Testing Matrix below shows the various forms of tests for a system. It was originally created by Brian Marick, transformed by Lisa Crispin and Janet Gregory into Agile Testing Quadrants, added to by Gerard Meszaros, and then expanded more by me. The key aspect of all these representations is that there are many categories of tests that can be applied to a system. For example, Behavior Driven Development / Acceptance Test Driven Development creates one category of these tests and Test-Driven Development creates another category.  

The diagram below (from Crispin and Gregory) shows the quadrant version. Here’s a short explanation of this, and a further explanation if you want to get into more details. I’ll explain some of the details using the expanded matrix that is shown next.

The expanded version of the Testing Matrix shown below provides a different perspective of the aspects in the Testing Quadrant. On the left side are the functionality tests, on the right the cross-functional tests (also known as non-functional tests). In the middle column are tests that are versions of cross-functional that can be executed earlier in the DevOps pipeline – either on a developer’s or build machine. At the top are business facing or customer facing tests. At the bottom are the technology facing tests. We’ll illustrate these types of tests with an example of an on-line ordering system.

The Customer/User Tests in the top left (L1) indicate the business intent of the application. They are typically created using BDD/ATDD and automated using SpecFlow / Cucumber / Fit-FitNesse or similar frameworks. An example would be a test scenario for creating an order.   

The Component Tests in the middle (L2) are written by the architects or developers who design the partition of the solution into components. The components may be layers or microservices or some other division. These are typically not necessary for smaller solutions. Tests for the proper operation of the infrastructure of a tax computation microservice represent component tests. 

The Unit Tests (L3) are written by developers to specify the behavior of their units – classes, methods, or other modules. Typically, they are created and automated with Test Driven Development and the red-green-refactor cycle. A unit test could check a method returns the correct tax for a particular item ordered from a particular location.

The thought process behind Behavior Driven Development is one way of approaching Component and Unit Tests. The tests are created using behavior scenarios for the microservices or classes. We’ll examine this in another article.  

Moving to the top right are the Usability tests (R1), which may be initially done with paper or screen prototypes but are completed using the actual application. Users perform their workflows and see how easy it is to execute using the system. In the example context, the user would create an order and see whether it was a nice experience.   

At the bottom right is Property Testing (R3). This includes externally visible qualities such as performance, security, robustness, availability, scalability, and similar quality aspects.   Tools can execute many of these tests. Testing with a large number of customers ordering simultaneously is one form of property testing. Dynamic testing for security issues is another.

In the middle right is Exploratory Testing (R2). These tests are not pre-defined. From the description in Wikipedia, using cognitive engagement, the tester with experience and creativity generates new tests to run. Exploratory tests might include looking for any unexpected effects of adding and deleting items from an order numerous times.

In the middle column at the top (M1) are Specified Usability tests. This include timing tests, such as a page should take no more than two seconds to load, or tests for accessibility standards or corporate standards such as colors.

In the middle (M2) are Specified Exploratory Tests. This may sound like a contradiction with Exploratory Testing. These are tests that may be run during Exploratory Testing or any other phase. It is then determined that they should be run for all deployments. An example of this was a discovery during testing that a web session crashed when more than 30 tabs were opened during the same session. A test is then established that each deployment would be checked with 100 open tabs.  

At the bottom (M3) is Unit Property Testing. These include source code analysis tests that check internal qualities, such as maintainability and security issues. Based on performance requirements, individual units may be allocated preliminary performance goals. For example, if an order should be processed within one second, then the calculation of taxes might be tested to check that it executes within a tenth of a second. Developers can run these performance tests during development.   

The flow chart below shows an example of some stages in a DevOps pipeline. Each of the tests above are shown with an arrow that indicates starts with the stage where the tests are typically created and runs through the stage where they may be executed. The Specified tests are shown where they are typically executed.   They may have been created for other systems or by using standards.


Passing the Functionality Tests in the testing matrix are necessary to release a system, but not sufficient. The system needs to pass Cross Functional Tests in order to be released. Many tests can be created in a test first manner (such as the Customer/User tests) and used to check that the system meets the expectations specified in those tests. However, the system still needs to have users manually test it for usability and testers explore it.