How to run only a subset of scenarios from an outline? #GivenWhenThenWithStyle

Instead of a new challenge, this week we’ll take a look at a nice trick to speed up feedback with complex scenario outlines. (You still have a few days to participate in the current challenge: How to write a good scenario title?)

When a suite of tests grows too large for fast feedback, teams often create a smaller subset so they can get a more immediate response. This allows developers to first run the smaller set, and if everything is OK with it, run the whole suite. In case of very large or slow systems, the entire suite can run overnight, or over a weekend on a special server. This technique can be quite effective to manage complex test suites, assuming the smaller subset is representative, and that adding or removing examples is not difficult.

With Given-When-Then tools, tags are the usual way of selecting a subset. SpecFlow and most other tools let you execute only scenarios with or without a specific tag during a test run. This works well for individual scenarios, but it can be a bit tricky with scenario outlines, leading to the question that we’ll answer this week:

How can I run only the first examples from each table?

For example, consider this scenario outline, but imagine it continues with a block of examples for each EU country.

Scenario outline: Digital VAT calculation

Given the customer is located in <country>
And the ebook base price is <net>
When the ebook pricing is shown
Then the final price should be <gross>

Examples:

| country | net    |  gross |
| IE      | 100    | 109.00 |
| IE      | 100.05 | 109.05 |
| IE      | 100.06 | 109.07 |
| IT      | 100    | 109.00 |
| IT      | 100.05 | 109.05 |
| IT      | 100.06 | 109.07 |
| AT      | 100    | 110.00 |
| AT      | 100.01 | 110.01 |
| AT      | 100.05 | 110.06 |
| AT      | 100.11 | 110.12 |
| BE      | 100    | 106.00 |
| BE      | 100.01 | 106.01 |
| BE      | 100.05 | 106.05 |
...

Although the list of examples might seem unnecessarily repetitive, in cases where the risk of a bug is huge (as it usually is with tax calculations), the business users might want to ensure that all those examples are checked. On the other hand, we might not need all those examples to quickly test if the code is still OK after a small change.

Because a tag applies to the whole scenario or outline, it’s not easy to just select a subset of examples in this case. One solution would be to copy a few cases into a separate scenario, and tag that new scenario. That’s not ideal, since it would create duplicated cases and slow down feedback for the full test run. It’s also error prone, as someone may update one instance and forget to modify the second set in the future.

A little known trick with Specflow is that we can apply a tag on a block of examples in the outline. In this series of articles, I’ve often suggested dividing a large set of outline examples into several blocks for easier understanding. We can combine those two techniques to select a subset for quick feedback:

Scenario outline: Digital VAT calculation

Given the customer is located in <country>
And the ebook base price is <net>
When the ebook pricing is shown
Then the final price should be <gross>

@smoke
Examples:

| Country | Net    |  gross |
| BE      | 100.05 | 106.05 |
| IE      | 100.05 | 109.05 |
| AT      | 100.05 | 110.06 |

Examples:

| IE      | 100    | 109.00 |
| IE      | 100.06 | 109.07 |
| IT      | 100    | 109.00 |
| IT      | 100.05 | 109.05 |
| IT      | 100.06 | 109.07 |
| AT      | 100    | 110.00 |
| AT      | 100.01 | 110.01 |
| AT      | 100.05 | 110.06 |
| AT      | 100.11 | 110.12 |
| BE      | 100    | 106.00 |
| BE      | 100.01 | 106.01 |
...

The previous outline has two blocks of examples. The first has only three cases, checking that the rates are correctly configured for just three countries, and proving that the amounts are rounded correctly for those cases. The second block contains all the remaining examples. This configuration lets us easily run the smaller subset of tests, by including the @smoke tag, and run everything at a later point. (The usual convention is to call this smaller set smoke tests, since they are not designed to prove that the system fully works, just that something is not horribly wrong. We’re testing that smoke does not come out once the code is plugged in).

Because we’re not duplicating cases, this approach does not slow down the feedback for the larger test run. We’re also still using just a single scenario outline, so it will not be error prone to update. Moving examples in or out the smoke test block is trivially easy. We just need to move a row of text above or below the second Examples header.

This approach also scales nicely with large test suites. Use the same tag in different feature files, and you’ll be easily able to run just the first few scenarios from each table of examples across hundreds of feature files.

Participate in the current challenge

The current Given-When-Then with style challenge is about writing a good scenario title. You still have time until next Monday to participate. This time, you can vote for your preferred title or provide an alternative. We’ll publish the solution next week.

Stay up to date with all the tips and tricks and follow SpecFlow on Twitter or LinkedIn.

PS: … and don’t forget to share the challenge with your friends and team members by clicking on one of the social icons below 👇