SpecFlow+ And TFS VSTS

By default, TFS looks for a profile named TFS.srprofile to execute your tests; if none is found, TFS uses default.srprofile instead. If you have changed the name of your profile, you need to enter the name of your profile in your runner settings (<Profile> element) and enter the path to your .runsettings file in the Run Settings File field.

TFS/VSTS 2015 & 2017

Note: General information on running tests with your builds in TFS can be found here.

To configure your build process in TFS 2015 to execute tests using SpecFlow+ Runner:

  1. Open TFS and switch to your build definition.
  2. Click on Add build step to add a new step to the build definition. Click on Test in the list of categories and click on Add next to "Visual Studio Test".
  3. Configure the build step as follows:
    • Enter the path to your Test Assembly (the DLL containing your compiled specifications project with the test bindings).
    • If you are using both MSTest and SpecFlow+ Runner, enter the path to NuGet package folder in the Path to Custom Test Adapters field. If you do not enter the path here, only your MSTest tests will be executed as TFS cannot find the SpecFlow+Runner test adapter.
    • If you have renamed your .srprofile file (i.e. the name is not TFS.srprofile or default.srprofile), enter the name of your file in the Run Settings File field.
  4. Save your changes.

Once the test run is complete, links to the report file and logs generated by SpecFlow+ Runner are available from the test run as attachments.

TFS and XAML

To configure your build process in TFS with XAML:

  1. Define your build definition as described here
  2. In order for the tests to execute, the binaries (DLLs in the various SpecFlow packages) need to be available to the build agent. Either:
    • Enable NuGet restore, in which cases the files are downloaded automatically
    • Check in the DLLs in the corresponding SpecFlow, SpecRun and SpecFlow+ packages. These DLLs are located in the corresponding sub-folders of your solution's packages directory.
  3. Enter the name of your .srprofile file in the Run Settings File field if the name is not TFS.srprofile or default.srprofile (see above).

Known Issues

  • The build agents cache the test adapters, which means that the last test adapter to be used is used for each build. You can thus only use a single SpecFlow+ Runner version with the same build agent, as the cached version is always used. If you want to use a different version of SpecFlow+ Runner for different builds, you need to define separate build agents.
  • Upgrading SpecFlow to a newer version requires a restart to purge the cache.