The State of Open Source Testing – key findings on BDD

We love Open Source Software. At SpecFlow we believe in creating great software through collaboration. We appreciate any creative input during our development journey.

Open Source does just that, simply on a larger scale, which is why we have been running our very own .NET BDD project Open Source from its inception on. Once the idea was born to ask a greater community about their Open Source preferences and experiences, it was a no-brainer for us to join our peer sponsors Tricentis, Flood, and TestProject in conducting this year’s “The State of Open Source Testing” survey.

Let’s start with the demographics. Together we were able to motivate over 1,800 people to participate. Most of our participants (66%) are working currently in a QA related role. The majority (61%) of our respondents lives in Asia and their preferred programming language is Java (67%). Lastly and most importantly 30% are heavy BDD users. With these demographics in mind we need to remind ourselves that BDD is a team activity and we are looking mostly through the tester’s eyes with our findings.

Here are our 5 key findings on Behavior Driven Development (BDD) and Open Source.

1) Writing good Gherkins requires practice

A sizable number of participants indicated to us that one of their main challenges in adopting BDD widely was to explain to their entire team how to write well written Gherkin examples. Users struggles are manifold, but the 2 main areas turned out to be the abstraction that comes with it and to write specific enough while keeping it simple. Once the practice was established communication amongst the team was much faster.

2) Living Documentation is a hidden gem

One of the most impactful benefits of BDD is to generate so-called Living Documentation from the Gherkin scenarios generated and used for automation. The Idea of Living Documentation is to always keep the specification of your software current – a historic pain point in software change management. Specification usually is current at initial release and then dilutes over time as software version numbers increase. The survey revealed that less than half (43%) of BDD users, that are either specifying their scenarios in Gherkin and/or are automating their scenarios, are using them to document the behavior of their application.

3) SpecFlow and Cucumber are the tools of choice

BDD has a growing community and ecosystem. The two Open Source projects spearheading the evangelization of the BDD market with an astonishing collective 94% are Cucumber (Java, JS, Ruby, …) and SpecFlow (C# .Net). Teams embracing BDD seem to align their preference with their choice of programming language and choose predominantly one of the two projects.

4) BDD is used in pieces

Even though BDD is a methodology that runs through the entire development cycle from Design to Release and wants to involve all 3 agile amigos (QA, DEV, BIZ) users seem to be choosing segments of its ideas while only exactly 50% of responses indicate complete use of all concepts of BDD. 51% are using it mostly for specification of behavioral examples in Given-When-Then format. Almost half (46%) use it predominantly for automation – which underlines the importance of acceptance test automation in today’s software development.

5) BDD projects run with 50% higher efficiency

The common consensus amongst the participants was that their projects applying BDD run substantially more efficient. On average 49%. The spread within these results was (not surprisingly) significant with users claiming anywhere from 0-100%. While investigating into the data we found a strong correlation between BDD proficiency level and (Proficient and Expert) and the efficiency claimed. With 76% of users being proficient their average was at 68%.

Our Conclusion

BDD has a growing community with lots of enthusiasm behind it. The community is clearly maturing and realizing strong benefits from BDD already. Cucumber and SpecFlow are nurturing the community actively and have strong followership. There are still some benefits that BDD teams are yet to enjoy and need to grow into using (e.g. Living Documentation). Agile teams need to embrace BDD together while Gherkin craftsman need to continue to practice their BDD authoring skills.

If you are interested in additional detail or would like to read more about the survey, check out the links below and follow us on social media (twitter, linkedin) to hear about our plans for the next survey in 2021.