How to deal with all the extra cases and exceptions? #GivenWhenThenWithStyle

The next challenge deals with a common issue teams face when they seriously start using feature files as a repository of specifications and tests.

The tester in our team wants to add a lot of extra exceptions to Given-When-Then files, but from a specification perspective these just create confusion. What’s the right way to balance clarity and test coverage?

Examples are very useful for several purposes in a software development cycle:

  1. They can be used to facilitate a collaborative analysis session and create a shared understanding when a team is trying to decide what to build.
  2. They can be used to define a solid proportion of acceptance tests for an upcoming feature, or regression tests for existing features.
  3. They can also be used to clarify documentation and make it easy to understand what a feature is supposed to do, to support maintenance and future development.

Balancing all these different roles and purposes can be quite a challenge.

Here’s an example:

Scenario Outline: Users should be able to download videos 
only when they have enough money to purchase them

Given a user account with balance of <balance>
When the owner wants to download a video costing <cost>
Then the transaction status should be <status>
And the user account balance should be <new balance>

Examples:

| balance | cost | status   | new balance | 
| 10      | 5    | approved | 5           |
| 5       | 6    | rejected | 6           |
| 10      | 10   | approved | 0           |

Someone approaching this scenario outline purely from a documentation perspective might argue that even three examples are too much, and that one of the “approved” examples could be safely deleted without losing any meaning.

Someone approaching the same document from a shared understanding perspective might want to keep the 10/10 case just to prove that people didn’t wrongly interpret the boundary.

Someone approaching it from a testing perspective will likely want to add examples such as 9.99/10 or 10.01/10, as well as negative amounts, to ensure that there are no bugs in rounding or data storage.

Someone approaching from a test coverage perspective might want to add a range of costs for the same balance (for example, all amounts starting from 0.01 to 10.00 with a step of 0.01, such as 0.02, 0.03, 0.04 and so on). After all, once the basic automation is in place, it’s almost free to keep adding cases.

Someone approaching this from an exploratory testing perspective might want to mess around with data types or structure, passing non-numerical values, fractional values with different separators (comma instead of a dot), amounts with one or three decimals, very large or very small amounts, words instead of numbers, null or empty values and so on. With basic automation in place, exploratory testing becomes much faster so people can try many more values.

All those examples are perfectly valid, but for different purposes. Also, the more cases we add, the more difficult the list becomes to understand (and the slower the overall test feedback).

How would you structure this information to ensure a solid test coverage, at the same time keeping it easy to understand and provide fast feedback? How would you decide which of the cases are worth including, and which not?

Do you have any challenges you’d like us to explore in one of our future posts?

Stay up to date with all the tips and tricks and follow SpecFlow on Twitter or LinkedIn.

PS: … and don’t forget to share the challenge with your friends and team members by clicking on one of the social icons below 👇