VS Integration Breaking Changes – Affects ALL users!

The upcoming SpecFlow 3 release will require an update to the Visual Studio extension for SpecFlow. Because the extension is normally updated automatically whenever a new version is released, this change has the potential to affect all users, not just those that upgrade to version 3! Please read the following information in detail.

What will break?

The new extension will not be compatible with versions of SpecFlow earlier than 2.3.2. If you are using an earlier version of SpecFlow, you should make sure that you have disabled automatic updates for the SpecFlow extension in Visual Studio. To do so:

  1. Select Tools | Extensions and Updates from the menu in Visual Studio.
  2. Enter “SpecFlow’ in the search field to restrict the entries in the list.
  3. Click on the “SpecFlow for Visual Studio” entry and disable the Automatically update this extension check box.
  4. This will prevent newer versions of the extension from being installed automatically. Once you are ready to upgrade to SpecFlow 3, you can enable this option again.

What limitations are there?

Because the Visual Studio extension can only be installed once per Visual Studio installation, you will not be able to mix SpecFlow 3 projects with projects that use a version of SpecFlow prior to 2.3.2.

How will the update be handled?

Our intention is to release a preview version of the Visual Studio extension that will not trigger automatic updates in Visual Studio for the duration of the preview period. If you want to try out the preview version of SpecFlow 3, you will need to add the feed to Visual Studio manually to install the new version of the extension. We will provide additional details on how to do this once the preview is available.

Once SpecFlow 3 has been officially released, we will update the live Visual Studio extension with the new version. This will cause your extension to automatically update if you have not disabled automatic updates (see above). From this point on, users of older versions of SpecFlow will need to download and install the compatible version of the Visual Studio extension manually and ensure that automatic updates are disabled.

I see a potential pitfall. What should I do?

If you see any potential pitfalls in this approach, please, let us know now! If you have suggestions for how to make this process easier, we would like to hear them! You can contact us at support@specflow.org.

Targeting Multiple Browser with a Single Test

If you are testing a web app (e.g. with Selenium), you will normally want to test it in a range of browsers, e.g. Chrome, IE/Edge and Firefox. However, writing tests for all the browsers can be a time-consuming process. Wouldn’t it be much easier to just write just one test, and be able to run that test in all browsers?

That’s where using targets with the SpecFlow+ Runner comes in. Targets are defined in your SpecFlow+ Runner profile. They allow you to define different environment settings, filters and deployment transformation steps for each target. Another common use case is to define separate targets for X64 and x86.

Defining targets for each browser allows us to execute the same test in all browsers. You can see this in action in the Selenium sample project available on GitHub. If you download the solution and open Default.srprofile, you will see 3 different targets defined at the end of the file:

  <Target name="IE">
  <Target name="Chrome">
  <Target name="Firefox">

Each of the targets has a name and an associated filter (e.g. “Browser_IE”). The filter ensures that only tests with the corresponding tag are executed for that target.

For each target, we are going to transform the browser key in the appSettings section of our app.config file to contain the name of the target. This will allow us to know the current target and access the corresponding web driver for each browser. The corresponding section in the project’s app.config file is as follows:

    <add key="seleniumBaseUrl" value="http://localhost:58909" />
    <add key="browser" value="" />

To make sure that the name of the target/browser is entered, we transform the app.config file to set the browser key’s value attribute to the name of the target using the {Target} placeholder. This placeholder is replaced by the name of the current target during the transformation:

  <![CDATA[<?xml version="1.0" encoding="utf-8"?>
    <configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
        <add key="browser" value="{Target}"
        xdt:Transform="SetAttributes(value)" />

This section locates the key (browser) and sets its value attribute to the name of the target. This transformation occurs separately for all three targets, resulting in 3 app.config files, each with a different browser key (that is available to your application).

WebDriver.cs (located in the Drivers folder of the TestApplication.UiTests project) uses this key to instantiate a web driver of the appropriate type (e.g. InternetExplorerDriver). The key from the configuration file is passed to BrowserConfig, used in the switch statement:

  if (_currentWebDriver != null)
    return _currentWebDriver;

  switch (BrowserConfig)
    case "IE":
      _currentWebDriver = new InternetExplorerDriver(new InternetExplorerOptions() { IgnoreZoomLevel = true }) { Url = SeleniumBaseUrl };
    case "Chrome":
      _currentWebDriver = new ChromeDriver() { Url = SeleniumBaseUrl };
    case "Firefox":
      _currentWebDriver = new FirefoxDriver() { Url = SeleniumBaseUrl };
      throw new NotSupportedException($"{BrowserConfig} is not a supported browser");

  return _currentWebDriver;

Depending on the target, the driver is instantiated as either the InternetExplorerDriver, ChromeDriver or FirefoxDriver driver type. The bindings code simply uses the required web driver for the target to execute the test; there is no need to write separate tests for each browser. You can see this at work in the Browser.cs and CalculatorFeatureSteps.cs files:

public class CalculatorFeatureSteps
  private readonly WebDriver _webDriver;

  public CalculatorFeatureSteps(WebDriver webDriver)
    _webDriver = webDriver;
  [Given(@"I have entered (.*) into (.*) calculator")]
  public void GivenIHaveEnteredIntoTheCalculator(int p0, string id)
    _webDriver.Wait.Until(d => d.FindElement(By.Id(id))).SendKeys(p0.ToString());

To ensure that the tests are executed, you still need to ensure that the tests have the appropriate tags (@Browser_Chrome, @Browser_IE, @Browser_Firefox). 2 scenarios have been defined in CalculatorFeature.feature:

Scenario: Basepage is Calculator
	Given I navigated to /
	Then browser title is Calculator

Scenario Outline: Add Two Numbers
	Given I navigated to /
	And I have entered <SummandOne> into summandOne calculator
	And I have entered <SummandTwo> into summandTwo calculator
	When I press add
	Then the result should be <Result> on the screen

		| SummandOne | SummandTwo | Result |       
		| 50         | 70         | 120    | 
		| 1          | 10         | 11     |

Using targets in this way can significantly reduce the number of tests you need to write and maintain. You can reuse the same test and bindings for multiple browsers. Once you’ve set up your targets and web driver, all you need to do is tag your scenarios correctly. If you select “Traits” under Group By Project in the Test Explorer, the tests are split up by browser tag. You can easily run a test in a particular browser and identify which browser the tests failed in. The test report generated by SpecFlow+ Runner also splits up the test results by target/browser.

Remember that targets can be used for a lot more than executing the same test in multiple browsers with Selenium. Don’t forget to read the documentation on targets, as well as the sections on filters, target environments and deployment transformations.

Fit-for-purpose Gherkin

A common mistake I see is the tendency to write Gherkin specification as if they were a series of instructions describing the algorithm for how the test is executed: “enter X” or “open page Y”. This is understandable; after all, it’s how you might write a standard test script. It’s very easy to continue using the same approach with Gherkin.

But the power of Gherkin comes from its abstraction – the ability to formulate steps in plain English (or any of the other supported languages), rather than as an algorithm. Instead of describing how something is done, Gherkin should describe what is being done without worrying about the implementation details.

This is a powerful tool, separating the algorithmic steps (bindings) from the actual description of the test intended to be read by humans. The primary goal of Gherkin is to formulate your scenarios so that they are quick to read, clearly formulated and understandable to all stakeholders – including non-technical team members.

OK, but what does that mean in real practical terms? Let’s take a look at a very basic example where we have a web application with various user roles, including administrators. Any test interfacing with the UI that involves the administrator dashboard will require the following initial steps in order to execute the test:

  1. Open the login page by navigating to the appropriate URL.
  2. Enter the user name.
  3. Enter the password.
  4. Click on the Log In button
  5. Select Administrator Dashboard from the menu.

These steps are important– you will definitely need to log in to the system somehow and navigate to the appropriate area to perform to execute the test. But let’s take a step back from the nitty gritty details, and ask ourselves what our precondition is: we need to be logged in as administrator, and we need to be viewing the administrator dashboard.

So, as tempting as it is to translate each of the steps listed above into individual steps in Gherkin, it is much more advisable to write the following:

Given I am logged in with administrator privileges

And I am viewing the administrator dashboard

These statement clearly capture the current state of the system required in order to perform the test. It goes without saying that being logged in as administrator involves logging in, but how this is done is not where the focus of your discussion should be. These steps are implicit implementation details that add nothing to the discussion about what can be done on the administrator dashboard itself. Furthermore, you may interface directly with your API and not even necessarily perform these steps using the UI.

The only place you need to worry about the implementation itself – how you log in to the system – is in the binding code for the Given step. There is no reason to explicitly state these steps in Gherkin; in fact, doing so runs the risk of making the feature file bloated, and thus harder to understand and navigate. As a side effect, you will also end up with a lot of unnecessary bindings for the unnecessary steps that could easily be grouped together.

Now if you have been critically evaluating the previous Gherkin step, you might even argue that “Given I am viewing the administrator dashboard” already implies you are logged in as administrator. And to a certain extent I would agree with you. In this case, my preferred approach would be to move the first step (Given I am logged in with administrator privileges) to the scenario background – assuming the feature file I am working on covers a range of administrator functions, all of which require the user to be logged in as an admin. Using background statements is another great way to reduce the clutter in your Gherkin files.

Another thing to remember is that there are always implicit steps; the question is where to draw the boundary. It is probably unnecessary to specify “Given I am logged in” as the prerequisite for every test, just like it is unnecessary to add the step “Given I have started the application”. Of course, you will want to specify the login state for certain, specific tests: testing the login process presumably requires the user to not already be logged in, so this is a step that should be specified. Similarly, access to the administrator dashboard is limited to a certain user type – being logged in as an administrator is therefore an important criteria. In this case, it also includes the information that other user types cannot perform these actions.

So let’s come back to the actual implementation of the steps. We have reduced a number of explicit steps to a single Gherkin statement, but we still need to execute all of those steps. That’s easily handled by writing code for each of the necessary steps that is called from the binding:

public void GivenIAmLoggedInWithAdminPrivileges()

As mentioned above, this might be abstracted to simply interface with your API, rather than actually performing these steps via the UI. Of course, just like you can reduce multiple explicit steps to a single Gherkin step, you can also group different sub-functions within a parent function, which I would probably do here:

public void GivenIAmLoggedInWithAdminPrivileges()

So I have one single function to handle the login process, which can obviously be reused elsewhere. Furthermore, any changes to the login process only require a single change to the test automation code and do not require any changes to the Gherkin file.

This makes it easier to maintain your tests and reuse steps that represent logical units, rather than having multiple tests share a large number of mini-steps that are brittle. Changes to the login process or your API do not require the test to be rewritten; they only affect the bindings of the tests. Furthermore, the step Given I am logged in as with administrator privileges can be used by any test involving an administrator account in a single binding method.

Of course, these concepts are not new. You will at some point have taken a function that has grown over time, and split it up into smaller sub-functions. You should adopt a similar approach to Gherkin: it may help to try and think more in terms of general functions, rather than individual statements. This may require a paradigm shift on your part: instead of being as explicit as possible in your feature files, reduce the steps to the essentials without obscuring any key information. The less bloated and more concise your tests/specifications are, the better suited they are to encouraging meaningful discussion between stakeholders, and the more understandable they are to non-technical team members. At the end of the day, that is probably the biggest benefit to this approach – it enables everyone to understand what is (supposed to be) going on and why.

As with any code (yes, feature files are code files too), a certain amount of refactoring and maintenance will always be required as your project grows. However, ensuring that your Gherkin files are fit-for-purpose from the start will mean that this refactoring will be a lot easier and less frequent. It will also have far less of an impact on your binding code if you reduce the frequency with which you need to update your Gherkin steps.

Sending Execution Reports via Email

We’ve had a number of users ask how to send execution reports via email after the test run has been executed. SpecFlow+ Runner 1.6 (a pre-release version is available on NuGet) introduces the option to give reports static names, i.e. without a time stamp, meaning sending the report as an attachment is now relatively easy to automate.

To help you out, we’ve outlined the necessary steps in the documentation here. The example uses mailsend, but you can obviously use other command line tools (or use telnet if you are really hardcore).

The same principles of course apply to any file you might want to send, and sending files via email is obviously not restricted to environments using SpecFlow+ Runner.