Test Results in Living Documentation – The Key to Stakeholder Engagement!

When you work with Behavior Driven Development, chances are you will produce a living documentation. Sooner or later you will then run into the question “should we include test results in our living documentation? And if yes, in what form?”.

As the maintainer of Pickles, the open source living documentation generator, I spent a good deal of time answering that question – and implementing the answers. The SpecFlow team at Tricentis kindly invited me to share my experiences on this blog.

The question

Should we include test results in a living documentation? Should we include them in a minimalistic pass/fail form? Should we include detailed tracing information? To answer those questions, it’s a good idea to briefly reflect on the target audience for a living documentation.

Who is the primary consumer of a living documentation? How do they benefit from it? Once we have the answers to those questions clearly in mind, we can go look for answers to the original question about test results.

Why do we produce a living documentation?

Let’s go back one step further: why do we apply Behavior Driven Development? We apply BDD to create software that better addresses the needs of the users. We apply BDD to better understand those needs. The business stakeholders have a good understanding of those needs, so it is essential that they get involved in the software development process. In my experience, the lower the participation threshold for business stakeholders is, the more they will engage in the process.

Behavior Driven Development enables us to capture requirements in scenarios that are understandable to everyone involved in the software development process. SpecFlow is a natural choice to work with BDD scenarios in Visual Studio, and software developers feel comfortable working with an IDE like that. Business stakeholders will likely shut down mentally when they see Visual Studio.  As my BDD-savvy product owner at the time famously put it: “I can’t show Visual Studio to a bank manager”.

Living Documentation – for example in the form of a website – significantly lowers the threshold for a business stakeholder to participate. After all, everybody knows how to use a website. I’ve seen bank managers who wouldn’t touch “programming stuff” with a ten-foot pole, happily discuss BDD scenarios from the living documentation.

Why do we produce a living documentation? To get the invaluable collaboration from business stakeholders who are far removed from the nuts and bolts of software development. It follows from there that any information we put in the living documentation, should increase the participation of the business stakeholders.

Business people love status updates

One of the easiest ways of making a business stakeholder happy – and willing to collaborate – is to give them a clear picture of how far along the project is. “Where are we in the project” and its variants is easily the most-asked question in a software project. Answering that question quickly will make the business stakeholders happy. Answering that question efficiently will save the product owner a lot of time.

In projects that use Behavior Driven Development, the number of scenarios that are implemented successfully is an excellent metric to determine the progress of the project. The more scenarios pass, the further along the project is. Everybody understands this metric. Since this metric is hard to cheat, it is easy to trust it.

Living Documentation makes the BDD scenarios accessible to the business stakeholders. We can also use it to make the progress of the project visible. By simply including a pass/fail indicator for each scenario, we give the business stakeholders a reliable and accessible way to assess the progress of the project. Business stakeholders don’t really care for the technical wizardry of a program – nor should they – so they are happy with the level of “pass” (this is done) and “fail” (this is not done).

A pass/fail indicator can be as simple as a green checkmark and a red “x”. The effect of this simple indicator is remarkable. Business stakeholders love to refer to the status and the living documentation, and they begin to trust the process. Happy and trusting business stakeholders are people who like to collaborate with the software developers.

Keep it simple

In my years as maintainer of Pickles, I’ve lost count of the number of times that people requested more debug information in the living documentation. It’s a reasonable request: software developers love living documentation as well; they especially love the pass/fail indicator. It reminds them of the red/green indicator of the unit test sessions.

A unit test report typically includes an exception message or a stack trace when a test fails. It is only natural for software developers to want the same kind of technical information for failed BDD scenarios.

My clear advice here is: keep it simple. The primary consumer of the living documentation is the business stakeholder. Everything that increases engagement of the business stakeholder is desirable, everything that decreases engagement is not. Putting a highly technical stack trace in a scenario is a sure way of decreasing engagement from the business stakeholders.

Does that mean we can’t have any kind of technical information in a Living Documentation? No. You can include it, if you take care to keep a clean interface for the business stakeholders. One way to do this could be to hide the technical information in expanding areas. Those areas should be collapsed by default. That way, the non-technical people will see a clean interface without technical “ballast”. Technical people can drill down into the technical details and gain some understanding about why a scenario failed.

Even the best error message is only a starting point

I strongly advise to keep the technical information in a living documentation to a minimum. If you automatically verify your scenarios, then the scenario runner will produce a comprehensive report. Why not include that information in the living documentation and eliminate the need to consult two sources of information?

Experience shows that in most cases, the technical information is not enough to solve the problem. If a scenario fails because of a null reference exception in a property, then of course that bit of information will probably be enough for a software developer to immediately fix a problem. For more complex causes, the software developer will likely have to experiment quite a bit with the source code to solve the issue.

Trying to provide the perfect stack trace or error message that gives you all information to solve a problem, is a quixotic quest. Even the best error message is only a starting point. I advise strongly against providing a lot of technical information in a living documentation. The potential drawbacks of overloading a living documentation with technical information (and consequently losing the engagement of the business stakeholders) outweigh the possible time gains for the developers when determining the cause of a failed scenario.

From the trenches: an elegant compromise

The requests for more developer-oriented information in living documentation kept popping up on the Pickles project, though. There is a clear desire for it. One person suggested an elegant compromise that I would have implemented. had I had the resources to do it.

Briefly put, the suggestion was this: when a scenario fails, provide an indication which step failed. For example: make the passing steps green, the failing step bold red and the non-executed steps yellow and italic. For the failing step, provide a collapsible area with a stack trace, and collapse it by default.

A non-technical reader benefits from a clean reading experience, and may even satisfy some of his curiosity about which part of the scenario failed. A technical reader benefits from knowing which step failed and having a stack trace, so that the starting point for the problem hunt is clear.

This elegant compromise ensures that the non-technical readers still have a smooth experience, while at the same time providing valuable information to the technical readers.

Summary

We produce living documentation to increase engagement by the business stakeholders. Adding a simple pass/fail indicator is an effective way of increasing the value of the living documentation and of increasing engagement. There is a clear desire for more technical information. An elegant way of doing this could be to mark the failing step in a failed scenario. In any case, make sure to add that information in an unobtrusive way so that the reading experience of the business stakeholders remains first class.

Feel free to discuss this article in our Community Forum!