Avoid tedious test case documents through comprehensible living documentation

In one of my recent projects the customer asked for a test case specification. After further inquiry, we learned that what they actually wanted was documentation of the systems latest agreed upon behavior, to help them with their acceptance tests for implementation sign-off. After a quick demo of SpecFlow+ Living Documentation rendered in Azure Devops and the Gherkin language, they agreed to having the documentation in this format to serve this need.

We had used Gherkin and Living Documentation in the past, but mostly for internal documentation that was only used within the development team. On a few occasions, where we used BDD scenarios in conversations with our customer, they were often received as “too technical”.

The problem was, that back then we looked at Gherkins mainly from an automation point of view. When a customer had a question about a certain behavior, anyone in the team could answer the question by checking our automated tests. We grouped examples for a better overview of our test coverage which was also reflected in our scenario titles.

Once we saw our features and scenarios rendered in a documentation view (see Screenshots further down), it was easy to spot our weak points. So, in order for our customers to use the systems Living Documentation as basis for their acceptance tests, we adjusted how we organised our feature files and most importantly our scenario titles.

After a first round of customer acceptance tests, we received the following feedback:

• They mainly used the “headlines”, in the tree on the left side, for their acceptance tests. The information provided there was comprehensible and enough for their tests, because they knew which important or special cases, they wanted to check for anyway
• They looked closer at examples only
  • when the system behaved differently than expected
  • when they tested complicated domain calculations or statistical views
• They wanted to keep this form of documentation even after their acceptance tests were finished and the initial project had ended

This was a total win for us. We achieved a Living Documentation that allowed not only the project team who created it, but also stakeholders, to quickly sift through a huge amount of information. They easily found the information they needed, without having to read and analyse every detail all the way down to the single examples. All this was possible because of how we organised our feature files, chose our scenario titles and had a way to provide our customer with an accessible render of it.

Now let us go back and take a closer look at what we did exactly that worked for all stakeholders of this project.

1.     Have a clear structure in how you organise your feature files

Gojkos Given-When-Then with Style Challenge #16 “How to organise feature files?” asks this exact question. We organised our feature files like described in Option 4.

“Option 4 – split by functional area, group by capability. Files are organised into directories by functional areas, scenarios are grouped into files by the user capability they provide.”

An example of that would be:

For the following example I used the SpecFlow Book Shop Example Project on Github and the LivingDoc CLI Generator which was very fast and I didn’t need to setup an Azure DevOps build just for demonstration purposes (which I am sure will also come in handy next time we want to clarify specifications with customers early on in this form).

In the Book Shop Example, the functional areas are:

• Homescreen
• Book Search
• Book Details
• Shopping Cart

With their capabilities or actions being:

• Homescreen
  • displaying
• Book Search
  • searching
• Book Details
  • displaying
• Shopping Cart
  • Adding books
  • Displaying shopping cart
  • Editing shopping cart
  • Removing books

To better demonstrate what needs to be done in your IDE to achieve the aforementioned structure, let me illustrate how source code files and feature file structure gets represented in SpecFlow+ LivingDoc.

Why we didn’t choose an iteration or user story based structure?

When we started with BDD and Living Documentation back in the days, we grouped by iteration and user stories. The reasons why we moved away from it are:

• A user story is only one particular change of the system at a particular time. Many other user stories can follow that will change behavior implemented with earlier user stories.
• Structure by iteration and user story adds unnecessary work, e.g. moving around or renaming feature files and scenarios, without delivering extra value.
• If you want traceability back to your product backlog items, use tags instead. There even exists a feature in SpecFlow+ LivingDoc that can link tags to PBIs.

At the end of the day user stories are short lived. Once implemented we forget their IDs and stakeholders hardly ever knew them in the first place. What counts in the end is the behavior of the system and how we perceive and talk about it, which is usually by functional areas and capabilities.

Not part of the above mentioned Given-When-Then with Style challenge, but an integral part to a comprehensible Living Documentation are the scenario titles.

2.     Scenario titles describe your specifications

One approach we tried out was to use naming that allowed us to group our features and scenarios by implied test coverage. We ended up with scenario titles like:

• Add book – different one
• Add book – same one – increase quantity

Unfortunately, our customers found that too technical and not very useful to work with.

What worked much better was to formulate scenario titles as specifications. This resonated with them immediately.

Now by simply looking at the tree on the left side you can read how the system should behave, grouped by functional area and capability. This way our customers could easily use the Living Documentation as basis for their acceptance tests.

For our Living Documentation to be a full success for customer acceptance tests and to involve non-dev team stakeholders, we had to do a few more things. Which I will talk about in my follow up blog article “Avoid tedious test case documents through comprehensible living documentation – Part 2”.