SpecFlow+ Getting Started

This section guides you through the process of installing SpecFlow and SpecFlow+ Runner¹ and setting up a simple project in Visual Studio. The getting started guide for SpecFlow+ Excel can be found here.

The sample project is intentionally kept very simple and consists of a basic “calculator” that adds two numbers and returns the result. You will define a test scenario to automate the testing of the calculator’s code.

This guide contains the following sections:

Installation and Setup
Adding a Feature File
Executing Your First Test
Implementing the Automation and Application Code
Executing the Tests Again
Living Documentation
Additional SpecFlow+ Runner Features

^{1. Instead of SpecFlow+ Runner, you can also use other test engines, like MsTest, xUnit, or NUnit. These packages are installed in exactly the same manner as SpecFlow+ Runner. However to follow all the steps in this guide, you need to install SpecFlow+ Runner. The evaluation version of SpecFlow+ Runner delays the execution of your tests by a number of seconds each time. You can request a demo license key to remove this restriction (please include your name and company name, as this information is included in the license).}

Installation and Setup

Installing SpecFlow consists of two steps:

Install the IDE integration
Set up your Visual Studio project to work with SpecFlow

Installing the IDE Integration Packages

The easiest way to install the SpecFlow Visual Studio extension method is to select Extensions | Manage Extensions (or Tools | Extensions and Updates in older versions) from the menu in Visual Studio, switch to the Online search on the left and enter “SpecFlow” in the search field at the top right.

ExtensionsAndUpdatesDialog

Click on the Download button to install the extension.

Once the installation has completed:

Select Tools | Options | SpecFlow from the menu in Visual Studio.
Locate the Enable SpecFlowSingleFileGenerator CustomTool option and set it to false.

The Enable SpecFlowSingleFileGenerator CustomTool option is a legacy setting that is enabled by default for backwards compatibility with older SpecFlow projects. We strongly recommend using MSBuild to generate your tests, and this tutorial will set up your project accordingly. When using MSBuild to generate your tests, this option should be disabled.

Setting Up your SpecFlow Project

This section guides you through the first steps of setting up a Visual Studio project with SpecFlow and defining and executing your first test scenario. In this example, we will be using SpecFlow+ Runner, but you can use a number of other test execution frameworks, including NUnit, Visual Studio Test Window and Team Foundation Server (TFS) Build integration for SpecFlow. SpecFlow+ Runner’s advantages include integration with Visual Studio Test Runner and extensive integrated reports available from within Visual Studio.

SpecFlow tests are usually placed into one or more separate projects in your solution, and these projects are referred to as a “specification project” below. The easiest and most convenient way to set up these projects is to use our SpecFlow NuGet package or one of the specific helper packages, like SpecRun.SpecFlow or SpecFlow.NUnit.

To set up your specification project:

Add an “MSTest Test Project (.NET Core)” to your solution (e.g. “MyProject.Specs”).
Note: Adding a “Unit Test Project” simplifies the setup, as we will be using the .NET Core framework for this project. For a Full Framework project, select “Unit Test Project (.NET Framework)” instead.
Remove the UnitTest1.cs file, as it is not required.
Right-click on your solution (e.g. “MyProject.Specs”) and select Manage NuGet Packages for Solution.
Install the following packages (use the search field to filter the search results):
- SpecFlow
- SpecFlow.Tools.MsBuild.Generation
- SpecRun.SpecFlow
Microsoft .NET Test SDK 15 is also required. If you have not installed this package already, please install it as well.

SpecRun.Runner Package

This package is added to your project automatically when you install SpecRun.SpecFlow. This package configures SpecFlow+ Runner as your unit test provider.

Note: Instead of SpecFlow+ Runner, you can also use other test engines, like MsTest, xUnit or NUnit. These packages are installed in exactly the same manner as SpecFlow+ Runner. However to follow all the steps in this guide, you need to install SpecFlow+ Runner. The evaluation version of SpecFlow+ Runner delays the execution of your tests by a number of seconds each time. You can request a demo license key to remove this restriction (please include your name and company name).

SpecFlow.Tools.MsBuild.Generation

In order to run your tests, SpecFlow needs to generate so-called “code-behind” files. The SpecFlow.Tools.MsBuild.Generation NuGet package is responsible for generating these files whenever you build your solution.

Adding a SpecFlow Profile

SpecFlow+ Runner is configured using a profile. If you want to make changes from the default configuration, you need to add this file to your project manually:

Right-click on your project, and select Add | New Item from the menu.
Navigate to Installed | Visual C# Items | SpecFlow on the left.
Select the SpecFlow+ Runner Profile (.srprofile) entry from the list, and change its name to “Default.srprofile”.
Click on Add to add the profile to your project.
If your project name contains spaces or dashes etc., you will need to change the name of the test assembly in Default.srprofile. Open the file and locate the <TestAssemblyPath> element. Ensure that the name matches that of your project by replacing any underscores with spaces as required.

As we are going to use the default settings, there is no need to edit the profile.

Adding a Feature File

You now need to add a feature file to your specifications project that outlines a specific feature and includes a test scenario:

1. Right-click on your specifications project and select Add | New Item from the popup menu.
2. Select SpecFlow Feature File (restrict the entries by entering “SpecFlow” in the search field), give it a meaningful name (e.g. “Calculator.feature”) and click on Add.
  Note: Do not choose one of the feature files with a language in the name, as these files do not contain the skeleton code we will be using.

The feature file is added to your specification project. It includes a default scenario for adding two numbers. You can leave most of the content as is, but add the word “also” to the scenario on the line beginning with “And” (see red text below):


Feature: Calculator
       In order to avoid silly mistakes
       As a math idiot
       I want to be told the sum of two numbers

@mytag
Scenario: Add two numbers
       Given I have entered 50 into the calculator
       And I have also entered 70 into the calculator
       When I press add
       Then the result should be 120 on the screen

We will use this scenario to demonstrate the first development iteration. By adding “also” to the second Given statement, we have two separate statements that will be parsed differently (otherwise both statements would match the regular expression “I have entered (.*) into the calculator”). This both simplifies the code in our example and illustrates how different statements linked with “And” are handled.

You can also assign tags to scenarios (e.g. “@mytag” in the example above), which can be used to filter scenarios, and control how scenarios are executed and automated. For more details on using tags, see Hooks, Scoped Bindings, FeatureContext and ScenarioContext in the documentation.

Generating Step Definitions

In order to test our scenario, we need to create step definitions that bind the statements in the test scenario to the application code. SpecFlow can automatically generate a skeleton for the automation code that you can then extend as necessary:

Right-click on your feature file in the code editor and select Generate Step Definitions from the popup menu. A dialogue is displayed.
Enter a name for the class, e.g. “CalculatorSteps”.
Click on Generate and save the file. A new skeleton class is added to your project with steps for each of the steps in the scenario:

Executing Your First Test

SpecFlow+ Runner integrates with Visual Studio Test Explorer. After adding your first specification and building the solution, the business readable scenario titles will show up in Visual Studio Test Explorer:

Build your solution.
Select Test | Windows | Test Explorer to open the Test Explorer:

Scenarios are displayed with their plain text scenario title instead of a generated unit test name.
Click on Run All to run your test. As the automation and application code has not yet been implemented, the test will not pass successfully.
Note: The evaluation version of SpecFlow+ Runner delays the execution of your tests by a number of seconds each time. You can request a demo license key to remove this restriction (please include your name and company name).

Note: If you cannot see your tests, make sure there are no spaces or dashes in your project name!

Implementing the Automation and Application Code

In order for your tests to pass, you need to implement both the application code (the code in your application you are testing) and the automation code (binding the test scenario to the automation interface). This involves the following steps, which are covered in this section:

Reference the assembly or project containing the interface you want to bind the automation to (including APIs, controllers, UI automation tools etc.).
Extend the step definition skeleton with the automation code.
Implement the missing application code.
Verify that the scenario passes the test.

Adding a Calculator Class

The application code that implements the actual functions performed by the calculator should be defined in a separate project from your specification project. This project should include a class for the calculator and expose methods for initialising the calculator and performing the addition:

Right-click on your solution in the Solution Explorer and select Add | Project from the context menu. Choose to add a new class library and give your project a name (e.g. “Example”).
Right-click on the .cs file in the new project and rename it (e.g. “Calculator.cs”), and choose to rename all references.
Your new class should be similar to the following:

Referencing the Calculator Class

Right-click your specification project and select Add | Reference from the context menu.
Click on Projects on the left of the Reference Manager dialogue. The projects in your solution are listed.
Enable the check box next to the Example project to reference it from the specifications project.
Click on OK.
A reference to the Example project is added to the References node in the Solution Explorer.
Add a using directive for the namespace (e.g. “Example”) of your Calculator class to the CalculatorSteps.cs file in your specification project:
using Example;
Define a variable of the type Calculator in the CalculatorSteps class prior to the step definitions:
```
 
public class CalculatorSteps
{
    private Calculator calculator = new Calculator();
    [Given(@"I have entered (.*) into the calculator")]
…
```
Defining a variable outside of the individual steps allows the variable to be accessed by each of the individual steps and ensures the variable is persistent between steps.

Implementing the Code

Now that the step definitions can reference the Calculator class, you need to extend the step definitions and implement the application code.

Binding the First Given Statement

The first Given statement in the scenario needs to initialise the calculator with the first of the two numbers defined in the scenario (50). To implement the code:

Open CalculatorSteps.cs if it is not already open.
The value defined in the scenario is passed as a parameter in the automation code’s associated function, e.g.:
Rename this parameter to something more human-readable (e.g. “number”):
public void GivenIHaveEnteredIntoTheCalculator(int number)
To initialise the calculator with this number, replace ScenarioContext.Current.Pending(); in the step definition as follows:
```
public void GivenIHaveEnteredIntoTheCalculator(int number)
{
    calculator.FirstNumber = number;
}
```
Switch to the file containing your Calculator class (e.g. Calculator.cs) and add a public integer member to the class:
public int FirstNumber { get; set; }

You have now determined that the FirstNumber member of the Calculator class is initialised with the value defined in the scenario when the test is executed.

Binding the Second Given Statement

The second Given statement in the scenario needs to initialise the second number with the second value defined in the scenario (70). To implement the code:

Open CalculatorSteps.cs if it is not already open.
Locate the function corresponding to the second Given statement and rename the p0 parameter to “number”, as before.
To initialise the calculator with the second number, replace ScenarioContext.Current.Pending(); in the step definition as follows:
```
public void GivenIHaveAlsoEnteredIntoTheCalculator(int number)
{
    calculator.SecondNumber = number;
}
```
Switch to the file containing your Calculator class and add another public integer member to the class:
public int SecondNumber { get; set; }

You have now determined that the SecondNumber member of the Calculator class is initialised with the value defined in the scenario when the test is executed.

Binding the When Statement

The step for the When statement needs to call the method that performs the actual addition and store the result. This result needs to be available to the other final step in the automation code in order to verify that the result is the expected result defined in the test scenario.

To implement the code:

Open CalculatorSteps.cs if it is not already open.
Define a variable to store the result at the start of the CalculatorSeps class (before any of the steps):
private int result;
Defining a variable outside of the individual steps allows the variable to be accessed by each of the individual steps.
Locate the function corresponding to the When statement and edit it as follows:
```
public void WhenIPressAdd()
{
    result = calculator.Add();
}
```
Switch to the file containing your Calculator class and define the Add() method:
```
public int Add()
{
    return FirstNumber + SecondNumber;
}
```

You have now determined that the Add() method of the calculator class is called once the initial Given steps have been performed.

Binding the Then Statement

The step for the Then statement needs to verify that the result returned by the Add() method in the previous step is the same as the expected result defined in the test scenario. To implement the code:

Open CalculatorSteps.cs if it is not already open.
As the result will be verified using Assert, you need to add “using Microsoft.VisualStudio.TestTools.UnitTesting;” to the top of your automation code.
Locate the function corresponding to the Then statement. Rename the p0 parameter in the function call (this time to “expectedResult”) and edit the step definition as follows:


public void ThenTheResultShouldBeOnTheScreen(int expectedResult)
{
    Assert.AreEqual(expectedResult, result);
}

You have now implemented the final piece of the jigsaw – testing that the result returned by your application matches the expected result defined in the scenario.

Final CalculatorSteps.cs Code

Your CalculatorSteps.cs code should be similar to the following:

using System;
using TechTalk.SpecFlow;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Example;

namespace MyProject.Specs
{
    [Binding]
    public class CalculatorSteps
    {
        private int result; 
        private Calculator calculator = new Calculator();

        [Given(@"I have entered (.*) into the calculator")]
        public void GivenIHaveEnteredIntoTheCalculator(int number)
        {
            calculator.FirstNumber = number;
        }

        [Given(@"I have also entered (.*) into the calculator")]
        public void GivenIHaveAlsoEnteredIntoTheCalculator(int number)
        {
            calculator.SecondNumber = number;
        }

        [When(@"I press add")]
        public void WhenIPressAdd()
        {
            result = calculator.Add();
        }
        
        [Then(@"the result should be (.*) on the screen")]
        public void ThenTheResultShouldBeOnTheScreen(int expectedResult)
        {
            Assert.AreEqual(expectedResult, result);
        }
    }
}

Final Calculator.cs Code

Your Calculator.cs code should be similar to the following:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace Example
{
    public class Calculator
    {
        public int FirstNumber { get; set; }
        public int SecondNumber { get; set; }
        
        public int Add()
        {
            return FirstNumber + SecondNumber;
        }
    }
}

Executing the Tests Again

Now that the test steps have been bound to your application code, you need to rebuild your solution and execute the tests again (click on Run All in the Test Explorer). You should see that the test now passes (green).

TestExplorerInitialStatus

Click on Output in the Test Explorer to display a summary of the test:
TestSummary
This example is obviously very simple; at this point you would want to refactor your code before proceeding with the implementation of your remaining scenarios.

Living Documentation (Optional)

An easy way to share your feature files with all members of your team is to use the SpecFlow+ LivingDoc VSTS extension to generate living documentation from your feature files. This documentation can then be accessed from within VSTS/TFS. Your Gherkin tests constitute an important part of the documentation of your system, describing the intended behaviour in human-readable form and providing the basis for discussion with other project stakeholders. Many stakeholders will not be able to access your feature files directly in your repository, nor will they be using Visual Studio, so sharing them on a common platform is a big advantage.

Generating documentation involves two steps:

Define a build process. This build process needs to include the SpecFlow+ build step (included when installing the extension).
Queue the build. Once it is complete, select Test | SpecFlow+ from the menu in TFS to access the living documentation in VSTS/TFS.

The generated documentation includes syntax highlighting for Gherkin keywords, tables and the ability to preview scenarios using example data.

Additional SpecFlow+ Runner Features

SpecFlow+ Runner includes a number of features that make working in Visual Studio easier.

Grouping Tests by Category

When using SpecFlow+ Runner to execute your tests, you can group scenarios according to various attributes:

Select Group By Traits to group the SpecFlow scenarios by the tags defined at the scenario or feature level:
Select Group By Class to group the SpecFlow scenarios by feature (with the feature title in plain text:

SpecFlow Reports

SpecFlow+ Runner generates an advanced execution report for each test run. To view the report, select Tests in the Show output from field in the Output window:

Click on the link to the report file to view the report in Visual Studio:

You can configure the name of the report file in the SpecFlow+ Runner settings (ReportFile element in the .runsettings file). More details are available here.

The report provides a convenient side-by-side view of the individual scenario steps and the trace output for each step:

Step definitions can write to the trace output to provide a lower level technical representation of the executed step (e.g. a more verbose representation of data structures that are processed in a step).

Running SpecFlow Tests from the Command Line

You can also run runtests.cmd from the command prompt to execute your specification tests.

Additional SpecFlow+ Runner Features

SpecFlow+ Runner offers several additional features that improve SpecFlow’s integration with Visual Studio, such as:

Parallel test execution
Detect flickering (intermittently failing) scenarios
Smart test execution order (run failed tests first)
Execute tests in different runtime environments
Automatic deployment configuration
SpecFlow integration with Team Foundation Server

SpecFlow+ requires you to purchase a license to use. The evaluation version of SpecFlow+ Runner delays the execution of your tests by a number of seconds each time. You can request a demo license key to remove this delay (please include your name and company name).

Alternatively, you can instead use other test engines, like MsTest, xUnit or NUnit. These packages are installed in exactly the same manner as SpecFlow+ Runner, but do not require a license. However, they also lack the extended features found in SpecFlow+ Runner.

Installation and Setup

Installing the IDE Integration Packages

Setting Up your SpecFlow Project

SpecRun.Runner Package

SpecFlow.Tools.MsBuild.Generation

Adding a SpecFlow Profile

Adding a Feature File

Generating Step Definitions

Executing Your First Test

Implementing the Automation and Application Code

Adding a Calculator Class

Referencing the Calculator Class

Implementing the Code

Binding the First Given Statement

Binding the Second Given Statement

Binding the When Statement

Binding the Then Statement

Final CalculatorSteps.cs Code

Final Calculator.cs Code

Executing the Tests Again

Living Documentation (Optional)

Additional SpecFlow+ Runner Features

Grouping Tests by Category

SpecFlow Reports

Running SpecFlow Tests from the Command Line

Additional SpecFlow+ Runner Features

About SpecFlow

Manage your Backlog

Contact