Reporting

SpecFlow provides various options to generate reports related to the acceptance tests.

Note: The specflow.exe command line tool that is used to generate reports can be found in the packages\Specflow.{version number}\tools directory, when you installed SpecFlow through NuGet. Start the tool with no parameters or use the --help option to display an overview of the available options.

Test Execution Report

This report generates an HTML test execution report. The report contains a summary of the executed tests and their results, as well as a detailed report of the execution of individual scenarios.

The following sub-sections cover generating the test execution report for different unit test providers.

NUnit Test Execution Report

NUnit 2

In order to generate this report, execute the acceptance tests with the nunit-console runner. This tool generates an XML summary of the test executions. To include detailed scenario execution traces, you need to capture the test output using the /out and the /labels options, e.g.

nunit-console.exe /labels /out=TestResult.txt /xml=TestResult.xml bin\Debug\BookShop.AcceptanceTests.dll

NUnit 3

In order to generate this report, execute the acceptance tests with the nunit3-console runner, and set it to output the results in nunit2 format. To include detailed scenario execution traces, you need to capture the test output using the --out and the --labels=All options (see the example below).

Important: The NUnit.Extension.NUnitV2ResultWriter package must be included in your project, otherwise you will receive the message: "Unknown result format: nunit2".

nunit3-console.exe --labels=All --out=TestResult.txt "--result=TestResult.xml;format=nunit2" bin\Debug\BookShop.AcceptanceTests.dll

SpecFlow Report Generator

Note: The examples and parameters are for version 2.4.* and higher. Older versions can be found by viewing past revisions in the GitHub wiki.

The report generation step is the same for both versions of NUnit. The two generated files can be used to invoke the SpecFlow report generation. If you use the default output file names shown above, you only need to specify information about the C# test project containing the *.feature files. Specflow uses the default TestResult.xml and TestResult.txt files to produce TestResult.html.

specflow.exe nunitexecutionreport --ProjectFile BookShop.AcceptanceTests.csproj
specflow.exe nunitexecutionreport --ProjectFile BookShop.AcceptanceTests.csproj --xmlTestResult CustomNunitTestResult.xml --testOutput CustomNunitTestOutput.txt --OutputFile CustomSpecflowTestReport.html

The following table contains the possible arguments for this command.

Attribute Value Description

-p, --ProjectFile

A path of the project file containing the *.feature files. Required.

-t, --xmlTestResult

NUnit XML test result file

The XML test result file generated by nunit-console. Optional.
Default: TestResult.xml

--testOutput

NUnit test output file

The labeled test output file generated by nunit-console. Optional.
Default: TestResult.txt

--OutputFile

HTML file

Generated Output File. Optional.
Default: TestResult.html

-x, --XsltFile

XSLT file

Custom XSLT file to use, defaults to built-in stylesheet if not provided. Optional.
Default: not specified

-n, --ProjectName

Project name

Project name which can be passed explicitly instead of implicitly getting it from --ProjectFile. Optional.
Default: not specified

-l, --FeatureLanguage

Feature language

Default: en-US

MsTest Test Execution Report


specflow.exe mstestexecutionreport BookShop.AcceptanceTests.csproj /testResult:result.trx /out:MyResult.html

The following table contains the possible arguments for this command.

Attribute Value Description
-p, --ProjectFile A path of the project file containing the feature files. This is a mandatory argument.
-t, --testResult MsTest test result (trx) file The TRX test result file generated by MsTest. Optional.
Default: TestResult.trx
-n, --ProjectName Project name Project name which can be passed explicitly instead of implicitly getting it from --ProjectFile. Optional.
Default: not specified
-o, --OutputFile HTML file Generated Output File. Optional.
Default: TestResult.html
-l, --FeatureLanguage Feature language Default: en-US
-x, --XsltFile XSLT file Custom XSLT file to use, defaults to built-in stylesheet if not provided. Optional.
Default: not specified

Step Definition Report

This report shows the usage and binding status of the steps in your entire project. You can use this report to find both unused code in the automation layer and scenario steps with no binding defined.

  • Steps with a red background are steps that exist in the automation layer but are not used in any feature files.
  • Steps with a yellow background are steps that exist in a feature file but do not have a corresponding binding defined.
  • Steps without a special backgrounds are steps that exist both in feature files and the automation layer. Ideally, all your steps are like this.
specflow.exe stepdefinitionreport BookShop.AcceptanceTests.csproj /BinFolder:bin/debug

The following table contains the possible arguments for this command.

Attribute Value Description

projectFile

A path of the project file containing the feature files. This is a mandatory argument.

/binFolder

A folder path (absolute or relative to the project folder)

A path for the compiled SpecFlow project. Optional.
Default: bin\Debug

/out

HTML file

Generated Output File. Optional.
Default: StepDefinitionReport.html

/xsltFile

XSLT file

Custom XSLT file to use, defaults to built-in stylesheet if not provided. Optional.
Default: not specified