SpecFlow 3 VSIX Considerations

As announced here, we need to update the SpecFlow Visual Studio extension (VSIX) for SpecFlow 3. The updates mean the new extension will not be compatible with older versions of SpecFlow ( <2.3.2). We plan to update the SpecFlow Visual Studio extension with the SpecFlow 3-compatible version when SpecFlow 3 is officially released. Note that a preview version of SpecFlow 3 is currently available, more details can be found here.

By default, the extension updates automatically, which will break older projects using versions of SpecFlow prior to 2.3.2. An alternative approach would be to release a dedicated SpecFlow 3 extension. There would then be two separate extensions for SpecFlow, which would avoid the automatic update from breaking old projects. However, we ended up discounting this approach for a number of reasons that are summarised below. Each approach has disadvantages and could potentially cause projects to stop working.

Approach 1: Release a New SpecFlow 3 Extension

In this case, we would release a new VSIX for SpecFlow 3, and leave the current extension online for users of earlier versions.

Pros

  • Users of earlier versions do not have to do anything (i.e. disable automatic updates) to prevent their projects from breaking.

Cons

  • Having two separate versions of a SpecFlow 3 extension in the gallery will cause confusion. Some users will invariably install the wrong version, increasing the number of support issues. This will persist for as long as both extensions are available, i.e. way past the transition period where most users will have migrated from 2 to 3.
  • There will be conflicts if both extensions are installed. There is no way to guarantee that only one of the extensions is installed.
  • Users suddenly need another extension for what appears to be “no good reason”.
  • Punishes users who consistently update to new versions, as these are the ones who need to jump through hoops to use the new release.

Approach 2: Upgrade the Existing Extension

In this case, we would update the existing VSIX for compatibility with SpecFlow 3.

Pros

  • While there may be some short-term confusion and support requests relating to projects suddenly not working (automatic update), these will die down quickly. After ~6 months, the dust will probably have settled.
  • All users will have their extension automatically upgraded to be compatible with the latest version. This is how it has always been.

Cons

  • Requires users not interested in Specflow 3 to make changes to their setup to avoid breaking changes. However, this is a one-off and involves clicking a check box.
  • There will invariably be some users who need to continue using SpecFlow <2.3.2 for now, and who will forget that automatic updates need to be enabled when they make the switch to version 3. Again, the number should decrease as time goes on.

With that in mind, we feel that upgrading the existing extension is better approach. It will cause some teething problems for a few months while the switch is made. After a while, the dust will settle. Conversely, we feel that providing two separate extensions runs the risk of causing confusion due to multiple versions of the extension, and the confusion will persist indefinitely. It will not simply die down after a few months if there are two mutually exclusive extensions available for download.

Your thoughts?

If you think we have overlooked some important aspects or have have alternative suggestions on how to improve the upgrade experience, please share them in the discussion here.

If you have reasons for using older versions of SpecFlow and do not want to regularly upgrade, let us know your motivation too. Both approaches will make it difficult to work with older SpecFlow versions and SpecFlow 3 on the same machine; we are not sure to what extent this could be an issue.

SpecFlow 3 with .NET Core Support: Public Preview now Available

The first public preview of SpecFlow with support for .NET Core is now available. If you want to try out the new version, please read the following information carefully. There are a number of steps that are necessary before you can use the public preview.

Please report any issues you experience here.

IMPORTANT! If you do not plan on updating to SpecFlow 3 soon, please read this announcement for all current users.

Requirements

.NET 4.7.1 is required. Please ensure you have installed the 4.7.1 framework.

Installing the Preview Visual Studio Extension

SpecFlow 3 requires an update to the Visual Studio extension for SpecFlow. During the preview period, you will need to connect to the following feed to download the preview version of the extension:

https://www.myget.org/F/specflow-vsix/vsix/

To do so:

  1. Select Tools | Options in Visual Studio.
  2. Locate Environment in the list, and select the Extensions and Updates sub-item.
  3. Click on Add to add a new feed.
  4. Give the new source a Name (e.g. “SpecFlow 3 preview”) and enter the source (“https://www.myget.org/F/specflow-vsix/vsix/”).
  5. Select Tools | Extensions and Updates. You should see an update listed on the left.
  6. Click on Updates to update the new extension.

Once a stable version of SpecFlow 3 has been released, the new version of the extension will go live for all users, and you can revert to the standard feed.

Supported Frameworks and Runners

The preview version supports the following test frameworks and runners:

  • SpecFlow+ Runner 3.0 or higher
  • NUnit 3.10 or higher
  • MSTest V2 1.3.2 or higher
  • xUnit 2.4.0 or higher

Installing NuGet Packages and Configuring a Unit Test Provider

The unit test provider is no longer configured in your app.config file. Instead, the unit test provider is now configured using plugins for the desired test frameworks. You will therefore need to add one of the following NuGet packages to your project to configure the unit test provider:

  • SpecRun.SpecFlow-3.0.0
  • SpecFlow.xUnit
  • SpecFlow.MsTest
  • SpecFlow.NUnit

Note: Make sure you do not add more than one of the unit test plugins to your project. If you do, an error message will be displayed.

  1. Right-click on your project, and select Manage NuGet Packages.
  2. Enable the Include prerelease option and search for “SpecFlow”.
  3. Install/update the following packages:
    • SpecFlow
    • SpecFlow.Tools.MsBuild.Generation
    • One of the unit test provider packages (see above).
  4. Install the test runner for your unit test provider (e.g. xunit.runner.visualstudio).

Configuration

Much of the configuration has been moved from the app.config file to other locations. When using .NET Core, you can no longer use the app.config file. You need to set general configuration options in the new specflow.json configuration file. This file is optional when using the Full Framework.

The structure of the .json configuration file reflects the structure of the old app.config. Some examples can be found here.

{
    "language":
    {
        "feature": "de-AT"
    }
}

In the above example, the feature file language is set to “de-AT”. This corresponds to the following entry in the app.config file:

<language feature="de-AT" />

If you are using the Full Framework, some settings are still available in the app.config file. However, much has changed, so please read the details below for information on how to configure SpecFlow 3.

Generating Code-behind Files

specflow.exe has been removed. Please use the SpecFlow.Tools.MsBuild.Generation NuGet package to generate the code-behind files, see https://specflow.org/documentation/Generate-Tests-from-MsBuild/

When using .NET Core or the new project format, it should be enough to simply add the package to your project.

Adding Feature Files

When adding a new feature file, several lines are added to your project file including references to the feature file. This is currently necessary for backwards compatibility with older versions. You will need to remove these lines from the project file manually whenever you add a new feature:

To remove these lines:

  1. Right-click on your project in the Solution Explorer and select Edit XYZ.csproj (where “XYZ” is the name of your project).
  2. Delete all elements with references to feature files (see screenshot above).

Sample Projects

Update .NET Core versions of the example projects are located here:

https://github.com/techtalk/SpecFlow-Examples/tree/feature/netcore-examples/NETCore%20Examples

No Reports Available

Reports have been removed from the main code base and reports are not available in SpecFlow 3.0. For more details, please refer to this GitHub issue.

SpecFlow+ Runner Restrictions

  • When using SpecFlow+ Runner, AppDomain separation is unavailable for .NET Core.
  • There is currently no support for the SpecFlow+ Runner server, meaning that adaptive test scheduling is unavailable.
  • Reports are currently unavailable for .NET Core. Let us know if this is a showstopper for you.

VS Integration Breaking Changes – Affects ALL users!

The upcoming SpecFlow 3 release will require an update to the Visual Studio extension for SpecFlow. Because the extension is normally updated automatically whenever a new version is released, this change has the potential to affect all users, not just those that upgrade to version 3! Please read the following information in detail.

What will break?

The new extension will not be compatible with versions of SpecFlow earlier than 2.3.2. If you are using an earlier version of SpecFlow, you should make sure that you have disabled automatic updates for the SpecFlow extension in Visual Studio. To do so:

  1. Select Tools | Extensions and Updates from the menu in Visual Studio.
  2. Enter “SpecFlow’ in the search field to restrict the entries in the list.
  3. Click on the “SpecFlow for Visual Studio” entry and disable the Automatically update this extension check box.
  4. This will prevent newer versions of the extension from being installed automatically. Once you are ready to upgrade to SpecFlow 3, you can enable this option again.

What limitations are there?

Because the Visual Studio extension can only be installed once per Visual Studio installation, you will not be able to mix SpecFlow 3 projects with projects that use a version of SpecFlow prior to 2.3.2.

How will the update be handled?

Our intention is to release a preview version of the Visual Studio extension that will not trigger automatic updates in Visual Studio for the duration of the preview period. If you want to try out the preview version of SpecFlow 3, you will need to add the feed to Visual Studio manually to install the new version of the extension. We will provide additional details on how to do this once the preview is available.

Once SpecFlow 3 has been officially released, we will update the live Visual Studio extension with the new version. This will cause your extension to automatically update if you have not disabled automatic updates (see above). From this point on, users of older versions of SpecFlow will need to download and install the compatible version of the Visual Studio extension manually and ensure that automatic updates are disabled.

I see a potential pitfall. What should I do?

If you see any potential pitfalls in this approach, please, let us know now! If you have suggestions for how to make this process easier, we would like to hear them! You can contact us at support@specflow.org.

Hacktoberfest – Contribute to Open Source Projects (including SpecFlow)!

Hacktoberfest is an annual event lasting from 1-31 October, aimed at encouraging contributions to open source projects. Hacktoberfest is open to everyone, from seasoned contributors to first-time contributors. During Hacktoberfest, pull requests you contribute to open source projects are counted. The first 50,000 contributors to reach 5 PRs will receive a T-shirt as a thank you. More details and a sign-up link can be found here.

Hacktoberfest’s key values are:

  • Everyone is welcome! Hacktoberfest participants are from around the world and of all skill levels. You are welcome to join in too!
  • Quantity is fun, quality is key! Meaningful contributions to open source both help the project, and help you develop your skills and engage with the community.
  • Short term action, long term impact! Open source projects are built upon the foundations of those who came before. Contributions have a lasting effect on both people and technology far beyond the month of October. Your contributions will be valued for many years to come.

If you would like to take part, we would be happy to see contributions to SpecFlow. The open SpecFlow issues can be found on GitHub here. If you are new to open source development, we would suggest that you take a look at the issues with labels such as “first-timers-only” and “up-for-grabs“. You can see a list of labels and navigate to a list of the issues with that label here.

Xray Support for SpecFlow in JIRA

We are pleased to announce that Xray, a test management solution for JIRA, now supports SpecFlow+ Runner. Many software development teams work with JIRA, but integrating SpecFlow with JIRA can be challenging. Xray helps out by allowing you to manage tests in JIRA, generate the corresponding feature files, and import your test results so they are visible in JIRA itself.

When using Xray, you can choose to edit and manage your scenarios in Jira, or continue working with feature files within your source repository. Both these methods are described here.

If you choose to manage features in JIRA, you can generate the necessary .feature files using the Export to Cucumber option in the user interface:

 

If you choose to manage your features files as part of your repository, there are a few steps you should take, including tagging tests with an ID to ensure they are always mapped to the same test in Xray. These steps are covered towards the end of the page.

In order to generate test results that can be viewed in JIRA, you will need to output your test results as a JSON report using SpecFlow+ Runnner. A suitable template is included with SpecFlow+ Runner (located under Templates in your project’s SpecRun.Runner packages folder). This report can then be imported into JIRA to view the tests’ status there.

If you are interested in finding out more about Xray, there is a short video overview available here. You can find more detailed information, including pricing, on the Atlassian marketplace. There are also regular webinars that walk you through the Xray’s features.

End of Visual Studio 2013 Support Imminent

We will remove support for Visual Studio 2013 with the upcoming release of SpecFlow 3 (due to be released soon). Downloads for the Visual Studio extension account for only a fraction of 1% of all downloads (a ratio of roughly 400:1), so we want to invest our time and resources elsewhere. For this reason, we will no longer support Visual Studio 2013 with SpecFlow 3, meaning there will be no more updates of the VS 2013 integration.

If you are still using Visual Studio 2013 and will want to upgrade to SpecFlow 3, we suggest that you start mapping out your upgrade path so you are prepared for the new release in time.

SpecFlow+ Runner 1.8 Released

SpecFlow+ Runner 1.8 has been released and is available for download from NuGet. The main changes in this version are:

  • Support for SpecFlow 2.4
  • Change to the behaviour of the test adapter for compatibility with Visual Studio 15.8

This version also includes a number of smaller bug fixes.

SpecFlow 2.4 Released

SpecFlow 2.4 has been released and is now available for download from NuGet. This is likely to be the last release of SpecFlow before we release SpecFlow 3.

SpecFlow 2.4 includes a number of new features:

  • Added ability to convert type to same type or one of the base types #1110
  • Added support for customization of dependency injection at feature level via a runtime plugin event to raise feature dependencies #1141
  • Allow marking steps as Obsolete and have a configurable behavior for it #1140
  • IGherkinParser and IGherkinParserFactory interfaces added to support Gherkin Parser pluggability #1143
  • Assist: remove accents from column headers and property names when comparing objects to a table #1096
  • Added NUnit current TestContext scenario container registration. See #936
  • Array & List support for strings and enums when instantiating class from Specflow table #1018
  • MSBuild: Experimental support for deleting code-behind files on clean and rebuild when MSBuild #1167, #1208
  • MSBuild: Experimental support for Net.SDK project system, only for targeting desktop framework, .net core is supported and won’t work #1223
  • Changed Parameter handling of specflow.exe #1112

There are also a number of bug fixes. The full list of changes can be found here.

 

Don’t forget to regenerate your features and restart Visual Studio after upgrading!

 

Discover BDD with Gaspar Nagy and Seb Rose

Gaspar Nagy – who has been involved with SpecFlow since its inception – and Seb Rose – a core member of the open source Cucumber project – have teamed up to publish Discovery – Explore behaviour using examples. This is the first book in a series on BDD. Gaspar had the following to say about the motivation behind writing the book, and the type of content you can expect.

 

When it comes to BDD, there are generally 3 types of people:

  1. Those who do not know what BDD stands for at all (how sad!)
  2. Those who know BDD, and claim to implement it with their team, but where it turns out that their focus is only on the tools
  3. Those for whom BDD is a holistic approach

The initial idea was for a SpecFlow book that would essentially be a port of the “Cucumber for Java” book: a collection of technical tips etc. In other words, yet another technical book. While working on the core, it quickly became apparent that a SpecFlow book would never be complete without mentioning the collaborative aspects of BDD. So we ended up diving deeper into the BDD realm. At some point we made a decision: let this book be about BDD, instead of SpecFlow. In fact, let there be a BDD book series!

Why? Because in our opinion the BDD process can be divided into 3 stages:

  1. Discovery: explore and discover details through examples
  2. Formulation: convert examples to scenarios
  3. Automation: write test automation code

My painful experience is that the earlier a team adopts BDD, the better results they get. So why not give teams a helping hand (or even a compass) right at the start, in the discovery phase? We decided that this would be the essence of the first book in the series.

It took several months of intensive work with Seb to complete the 5 chapters of Discovery. At about 100 pages, it is relatively short, and said to be an easy read, so you could read it on a plane. The book is written for everyone: developers and testers, as well as product owners and test managers. Discovery – Explore behaviour using examples is also tool-agnostic: the emphasis is not on the technical process but on the collaborative aspect. The book covers the following:

Chapter 1 provides an introduction to the nature of BDD, where the important elements are explained.

Chapter 2 is an important chapter about structured conversation, providing hints on how testers and business stakeholders can be better involved in meetings, and how a shared understanding of the requirements can be achieved. A number of techniques, in particular Example Mapping (introduced by Matt Wynne), are described here through imaginative use cases.

Chapter 3 is about examples, and how there are more than just Given/When/Then test cases.

Chapter 4 covers tips on how a BDD approach can fit your project, regardless of whether you use Scrum, Kanban or any other framework.

Chapter 5 is all about how to sell BDD to business stakeholders, and make them interested in BDD. It includes tips on how to demonstrate that BDD is more than a mere technique applicable only to the tasks of developers and testers. Illustrative examples ensure that business stakeholders can relate to the advantages and share a common understanding as to why BDD is good for the entire team, saving time and effort on bug hunting. In order to reap the biggest benefits of BDD, the business side should be involved in the entire process from the start to finish.

Get your copy and let the discovery begin! You can find it on Amazon/Leanpub through http://bddbooks.com!

 

SpecFlow and PackageReference

By default, the packages used in SpecFlow projects are stored in packages.config. However the new Visual Studio projects instead use the PackageReference element to reference packages. When using the PackageReference element, generator plugins will not be found, and you will see errors similar to the following:

Unable to find plugin in the plugin search path: SpecRun. Please check http://go.specflow.org/doc-plugins for details.
#error TechTalk.SpecFlow.Generator

This is due to SpecFlow not searching for the package in the correct location. To ensure that the plugin is found, you need to specify the path explicitly in your app.config file.

To do so, enter the path to the appropriate path in the plugins element of your app.config file in this case, e.g.:

<plugins>
      <add name="SpecRun" path="%USERPROFILE%\.nuget\packages\specrun.specflow.2-3-0\1.7.0\lib\net45" />
</plugins>

IMPORTANT! You will need to manually update this path when upgrading to a newer version of the package, as the path contains the version number (e.g. “1.7.0” in the above example).

You can find a sample project using the new project format here. The relevant line in the app.config file is here.

Note: This requires SpecFlow 2.3.2 of higher. This step should not be necessary once SpecFlow 3 is released.