How to specify that something should be random? #GivenWhenThenWithStyle

Specification by Example works well for deterministic processes, where you can know an expected result upfront. The next challenge deals with a situation when there is no clear expected result – generating random numbers.

We have a chat-bot that needs to simulate human typing. Instead of sending the entire message at once, we want it to insert short pauses between each letter, say between 0.2 and 0.5 seconds. And these pauses should vary randomly, so the human on the other end would not suspect it’s chatting with a robot. How can we describe this with Gherkin?

Option 1: Specify overall process properties

One option would be to directly translate business expectations into a single statement, and hide the details of the test in the automation layer.

Given the bot receives a message "Hi there"
When the bot replies with "Hello"
Then the bot should insert random pauses from 0.2 to 0.5 seconds between each letter

Option 2: specify randomness in individual actions explicitly

Alternatively, we could expose more information in the scenario, and leave individual actions to prove for the automation layer.

Given the bot receives a message "Hi there"
When the bot replies with "Hello"
Then the pause between typing H and e should be random between 0.2 and 0.5 seconds
And the pause between typing e and l should be random between 0.2 and 0.5 seconds
And the pause between typing l and l should be random between 0.2 and 0.5 seconds
And the pause between typing l and o should be random between 0.2 and 0.5 seconds

Option 3: Test properties of a larger set

Although there’s no clear deterministic value for each individual action, non-deterministic processes may have some deterministic properties when applied to a large group. We could try specifying that instead of individual instances:

Given the bot receives 10000 messages
When the bot replies to all messages
Then the minimum delay between typing should be greater than 0.2 seconds
And the maximum delay between typing should be less than 0.5 seconds
And there should be at least 1000 unique delay values

Option 4: Fake the randomness for automation

Alternatively, we could make the process deterministic by providing a “fake” random number generator for testing purposes. In production, the system would use an actual random number generator. This would not prove end-to-end integration, but it would prove that we’re transforming the random number output (usually between 0 and 1) to the interval required by the scenario (0.2s to 0.5s).

Given the random number generator produces the following sequence: 0.9, 0.4, 0.1, 0, 0.6
When the bot replies with "Hello"
Then the folling pauses should be be inserted after:
| H | 0.47s |
| e | 0.32s |
| l | 0.23s |
| l | 0.20s |
| o | 0.38s |

Participate in the challenge

Choose one of the options you like the best, or propose your own solution – there are plenty of other ways to achieve the same thing. If you have an alternative proposal, perhaps publish a quick blog post and send us the link.

To participate, cast your vote using the link below before 25th January. We’ll publish the results on the 26th.

Stay up to date with all the tips and tricks and follow SpecFlow on Twitter or LinkedIn.

PS: … and don’t forget to share the challenge with your friends and team members by clicking on one of the social icons below 👇