Reporting

SpecFlow provides various options to generate reports related to the acceptance tests.

Note: The specflow.exe command line tool that is used to generate reports can be found in the packages\Specflow.{version number}\tools directory, when you installed SpecFlow through NuGet.

Test Execution Report

This report provides a formatted HTML report of a test execution. The report contains a summary about the executed tests and the result and also a detailed report for the individual scenario executions.

See the following sub-sections for how to generate the test execution report depending on the unit test provider used.

NUnit Test Execution Report

In order to generate this report you have to execute the acceptance tests with the nunit-console runner. This tool generates an XML summary about the test executions. To have the detailed scenario execution traces visible, you also need to capture the test output using the /out and the /labels options as it can be seen in the following example.

nunit-console.exe /labels /out=TestResult.txt /xml=TestResult.xml bin\Debug\BookShop.AcceptanceTests.dll

The two generated files can be used to invoke the SpecFlow report generation. If you use the output file names shown above it is enough to specify the project file path containing the feature files.

specflow.exe nunitexecutionreport BookShop.AcceptanceTests.csproj /out:MyResult.html

The following table contains the possible arguments for this command.

Attribute Value Description

projectFile

A path of the project file containing the feature files. This is a mandatory argument.

/xmlTestResult

NUnit XML test result file

The XML test result file generated by nunit-console. Optional.
Default: TestResult.xml

/testOutput

NUnit test output file

The labeled test output file generated by nunit-console. Optional.
Default: TestResult.txt

/out

HTML file

Generated Output File. Optional.
Default: TestResult.html

/xsltFile

XSLT file

Custom XSLT file to use, defaults to built-in stylesheet if not provided. Optional.
Default: not specified

MsTest Test Execution Report

specflow.exe mstestexecutionreport BookShop.AcceptanceTests.csproj /testResult:result.trx /out:MyResult.html

The following table contains the possible arguments for this command.

Attribute Value Description

projectFile

A path of the project file containing the feature files. This is a mandatory argument.

/testResult

MsTest test result (trx) file

The TRX test result file generated by MsTest. Optional.
Default: TestResult.trx

/out

HTML file

Generated Output File. Optional.
Default: TestResult.html

/xsltFile

XSLT file

Custom XSLT file to use, defaults to built-in stylesheet if not provided. Optional.
Default: not specified

Step Definition Report

This report shows the usage and binding status of the steps for the entire project. You can use this report to find both unused code in the automation layer and scenario steps that have no definition yet.

  • Steps with a red background are steps that exist in the automation layer but are not used in any feature files.
  • Steps with a yellow background are steps that exist in a feature file but do not have a corresponding definition.
  • Steps without a special backgrounds are steps that exist both in feature files and the automation layer. Ideally, all your steps are like this.
specflow.exe stepdefinitionreport BookShop.AcceptanceTests.csproj

Important for .NET 4.0 projects: Because specflow.exe is compiled for .NET 3.5, it cannot load .NET 4.0 assemblies by default. To generate this report for .NET 4.0 projects, you have to force specflow.exe to use the .NET 4.0 runtime by using the config file. Just copy the config below and create a specflow.exe.config file and put it next to your specflow.exe and you will be able to create the step definition report.

<?xml version="1.0" encoding="utf-8" ?> 
<configuration> 
    <startup> 
         <supportedRuntime version="v4.0.30319" /> 
    </startup> 
</configuration> 

The following table contains the possible arguments for this command.

Attribute Value Description

projectFile

A path of the project file containing the feature files. This is a mandatory argument.

/binFolder

A folder path (absolute or relative to the project folder)

A path for the compiled SpecFlow project. Optional.
Default: bin\Debug

/out

HTML file

Generated Output File. Optional.
Default: StepDefinitionReport.html

/xsltFile

XSLT file

Custom XSLT file to use, defaults to built-in stylesheet if not provided. Optional.
Default: not specified