The Customer/User Tests in the top left (L1) indicate the business intent of the application. They are typically created using BDD/ATDD and automated using SpecFlow / Cucumber / Fit-FitNesse or similar frameworks. An example would be a test scenario for creating an order.
The Component Tests in the middle (L2) are written by the architects or developers who design the partition of the solution into components. The components may be layers or microservices or some other division. These are typically not necessary for smaller solutions. Tests for the proper operation of the infrastructure of a tax computation microservice represent component tests.
The Unit Tests (L3) are written by developers to specify the behavior of their units – classes, methods, or other modules. Typically, they are created and automated with Test Driven Development and the red-green-refactor cycle. A unit test could check a method returns the correct tax for a particular item ordered from a particular location.
The thought process behind Behavior Driven Development is one way of approaching Component and Unit Tests. The tests are created using behavior scenarios for the microservices or classes. We’ll examine this in another article.
Moving to the top right are the Usability tests (R1), which may be initially done with paper or screen prototypes but are completed using the actual application. Users perform their workflows and see how easy it is to execute using the system. In the example context, the user would create an order and see whether it was a nice experience.
At the bottom right is Property Testing (R3). This includes externally visible qualities such as performance, security, robustness, availability, scalability, and similar quality aspects. Tools can execute many of these tests. Testing with a large number of customers ordering simultaneously is one form of property testing. Dynamic testing for security issues is another.
In the middle right is Exploratory Testing (R2). These tests are not pre-defined. From the description in Wikipedia, using cognitive engagement, the tester with experience and creativity generates new tests to run. Exploratory tests might include looking for any unexpected effects of adding and deleting items from an order numerous times.
In the middle column at the top (M1) are Specified Usability tests. This include timing tests, such as a page should take no more than two seconds to load, or tests for accessibility standards or corporate standards such as colors.
In the middle (M2) are Specified Exploratory Tests. This may sound like a contradiction with Exploratory Testing. These are tests that may be run during Exploratory Testing or any other phase. It is then determined that they should be run for all deployments. An example of this was a discovery during testing that a web session crashed when more than 30 tabs were opened during the same session. A test is then established that each deployment would be checked with 100 open tabs.
At the bottom (M3) is Unit Property Testing. These include source code analysis tests that check internal qualities, such as maintainability and security issues. Based on performance requirements, individual units may be allocated preliminary performance goals. For example, if an order should be processed within one second, then the calculation of taxes might be tested to check that it executes within a tenth of a second. Developers can run these performance tests during development.
The flow chart below shows an example of some stages in a DevOps pipeline. Each of the tests above are shown with an arrow that indicates starts with the stage where the tests are typically created and runs through the stage where they may be executed. The Specified tests are shown where they are typically executed. They may have been created for other systems or by using standards.
Summary
Passing the Functionality Tests in the testing matrix are necessary to release a system, but not sufficient. The system needs to pass Cross Functional Tests in order to be released. Many tests can be created in a test first manner (such as the Customer/User tests) and used to check that the system meets the expectations specified in those tests. However, the system still needs to have users manually test it for usability and testers explore it.
