This week’s challenge is the tricky problem of explaining a missing value.
What’s the best way to describe a Given for a value that’s not supposed to be there? For example, starting the specification from a point of a user who does not yet exist in a database, or a product that’s not been set up. “Given a user that doesn’t exist” sounds vague, but “Given user John that does not exist” sounds silly – how can we talk about John if he doesn’t exist yet?
Key examples should ideally be specific enough to avoid ambiguity, but it’s difficult to be specific about something that does not exist. Such problematic Given-When-Then scenarios are usually overly generic, and do not really help with shared understanding or testing. Here’s a typical trivial example that should ensure unique accounts:
Given user John does not exist When John tries to register Then the registration is successful Given user John exists When John tries to register Then the registration fails
This set of scenarios seems simple but it can hide many important constraints, and provide a false sense of shared understanding. A team might think they nailed down a feature, but develop something full of bugs. Scenarios such as these leave too much for later exploratory testing, overloading the team to discover seemingly unexpected problems that could have easily been predicted and documented with better examples.
As a trivial counter-example, consider what should happen if John tries to register with an email already assigned to a different user. For a more complex edge case, consider what should happen if two Johns tried to register at the same time. And what is “John” anyway in this case? Is it a personal name or a username? If it’s a personal name, should we really prevent two people with the same name from opening two different accounts?
Oversimplified examples often lead to overcomplicated test automation. Proving that a user successfully registered usually requires accessing a database, which means that the related tests will be slow and brittle. Accessing external systems is a huge performance penalty compared to in-process tests. Asynchronous networking introduces a whole set of technical edge cases that have nothing to do with user registration business rules.
Data persistence leads to issues around reproducibility. John might not exist in the database the first time you run a test, but it surely will be there after that test completes. To make such a test repeatable, you’ll either need to introduce complex test set-up or clean-up procedures or make the examples even more generic by introducing placeholders for random or unique values.
Database resources are usually difficult to set up from scratch, so team members and test automation systems sometimes share testing databases, which can lead to issues around reproducibility and isolation. If two people execute tests against the same database at the same time, one might add data that the other does not expect, causing tests to fail intermittently without a good explanation.
Read-on for a selection of good ideas we’ve received and our suggestions on how to handle scenarios such as this one.
PS: … and don’t forget to share the challenge with your friends and team members by clicking on one of the social icons below 👇