Reports

SpecFlow can generates reports once your tests have finished executing that include a breakdown of the results of your tests. The default report includes a statistical overview of the status of all tests, as well as information on individual scenarios, including Gherkin test cases, statistics on the total number and percentage of successful tests, and the execution time for each step. When running tests from within Visual Studio, a link to the generated report(s) is included in the Output window once the tests have completed.

The report is output to your output folder (configured in your profile) and the name of the report is generated using the projectName and name defined in your profile plus a time stamp: <projectName>_<name>_YYYY-MM-DDTHHMMSS

If you want to generate multiple reports from a single test run, you also need to modify your profile to include the templates and output paths for these reports. The settings for multiple reports are defined in the <Report> element.

Report Templates

A Razor template is used to generate the reports, and the SpecFlow+ Runner NuGet package includes several report templates. The templates are located in the \packages\[SpecRun.Runner]\templates directory of your Visual Studio project and include the following:

  • ReportTemplate.cshtml: The default report template, this is a standard report that outputs information on the results of the test run as HTML, and is human-readable.
  • ReportTemplate_Json.cshtml: Outputs the results of the test run as JSON. This format is intended for post-processing purposes.
  • ReportTemplate_Xml.cshtml: Outputs the results of the test as XML. This format is intended for post-processing purposes.

You can customise the templates to meet your needs. If you customise a template, we recommend renaming the template file accordingly. For some examples of how you can customise your reports, refer to the Customising Reports tutorial.

The data available to your reports is accessible using SpecFlow+'s report API. If you edit the .cshtml template in Visual Studio, Intellisense will display a short description of the available properties and methods. You can also reference them in the online report API documentation.

Note: The old HTML report template for SpecFlow+ 1.2 can be found here. There are no XML or JSON report templates for version 1.2.

Defining Your Own Template

Please review the SpecFlow+ Runner Report API Documentation to find out about the classes and properties of the reporting model that can be referred in the reporting template.

To define your own report template:

  1. Create a copy of ReportTemplate.cshtml (in \packages\[SpecRun.Runner]\templates) and rename it accordingly.
  2. If desired, add the .cshtml file to your Visual Studio project so that you can edit in Visual Studio. If you do this, ensure that Copy to Output Directory in the file's properties is set to "Copy always" to ensure that the latest version of the template is always used to generate reports.
  3. Edit your .srprofile file and change the value of the reportTemplate attribute in the Settings element to the name of your template file. This path is relative to your base folder (default: \bin\).
  4. Change the name and projectName attributes in the Settings element to reflect your project. These values are used to name the generated report.
  5. Execute your tests to generate the report.

Using Intellisense

If you edit the template .cshtml file in Visual Studio, you can use Intellisense to access the properties and methods exposed by the API. To use Intellisense, reference the following DLLs from your project in VS:

  • RazorEngine.dll
  • System.Web.Razor.dll
  • TechTalk.SpecRun.Framework.dll
  • TechTalk.SpecRun.Framework.Interfaces.dll

These files are all included in the SpecFlow+ Runner NuGet package and are located in the \packages\[SpecRun.Runner]\tools directory once you have added the NuGet package to your solution. If you are using Resharper, referencing the DLLs is all that is necessary. If you are not using ReSharper, you will also need to follow the steps outlined here.

Generating a Single Report for Multiple Specification Projects

A SpecFlow+ report is generated for the .srprofile file used to execute the tests from the command line. You can add multiple assemblies to your .srprofile file, in which case the report will combine the results of the assemblies in a single report.

To generate a single report for multiple assemblies:

  1. Open the profile you want to edit and locate the <TestAssemblyPaths> section. This should already contain the path to the assembly linked to the project the profile belongs to.
  2. Add the test assemblies you want to include in the report as elements. You need to specify the paths as relative to the location of the assembly associated with the profile's project. If all your projects are located in the same root folder, the path should be ../../../<ProjectDirectory>/bin/debug/<ProjectSpecificationAssembly>.dll.
  3. Start runtests.cmd located in the directory of your edited profile. Your tests are executed and the report is generated.

Generating Multiple Reports in a Single Test Run

You can generate multiple reports from a single test run. To do so, you also need to modify your .srprofile file to reference the .cshtml templates and output paths for each reports. The settings for multiple reports are defined in the <Report> element of your profile. You can use these settings to determine the name of the report files and how to handle conflicts with existing files with the same name.