Data Driven Testing or how to parameterize Selenium tests using MSTest

When automating E2E testing, it is often necessary to write many tests to test certain behavior, such as validating a numeric input. One way is to duplicate the method and change the parameter values, but with a large amount of test data, there can be a lot of duplicate code, and such tests are also difficult to maintain. But there is a simpler way, his idea is to create a parameterized test in such a way that one test method can be used to execute N tests with all the test data.

Introduction

When it comes to automated testing in C#, there are several test frameworks such as NUnit, xUnit.net, and MSTest. Each has its own advantages and disadvantages, but NUnit and xUnit.net still have a huge advantage over MSTest. Although MSTest is the standard testing framework that comes with Visual Studio, it was not the preferred test automation tool until the release of MSTest 2.

MSTest 2 is an open source version of MSTest. The previous version of MSTest did not have many of the features found in other test frameworks. However, the second version supports parallel test execution and data-driven testing, and is also extended with custom test attributes. Parameterized tests with MSTest became possible in version 2 of the framework using attributes/annotations.

Parameterization with the [DataRow] attribute

The attribute [DataRow]allows you to set test parameter values, and more than one attribute can be set [DataRow]. To convert a regular test into a parameterized one, it is enough to replace the attribute [TestMethod]with [DataTestMethod]and pass the necessary parameters for the test to [DataRow]the attribute.

In the example below, consider the abstract validation of a numeric input, passing the value itself as the first argument, and the corresponding error message in the tooltip as the second argument:

[DataTestMethod]
[DataRow(NumberValues.NotNumberValue, Tooltips.NotNumber)]
[DataRow(NumberValues.NotInAcceptableRangePositive, Tooltips.AllowedRange)]
[DataRow(NumberValues.NotInAcceptableRangeNegative, Tooltips.AllowedRange)]
[DataRow(NumberValues.ErrorInExponentialNumberNotation, Tooltips.ErrorInExponentialNumberNotation)]
[DataRow(NumberValues.OnlyOneSeparatorIsAllowed, Tooltips.OnlyOneSeparatorIsAllowed)]
public void NumberInputValidationTooltipShouldExist(string inputValue, string tooltipText)
{
    // Act
    SomeNumberInput
        .EnterValue(inputValue)
        .GetValidatorTooltip(out var currentTooltipText);

    // Assert
    Assert.AreEqual(tooltipText, currentTooltipText);
}

Note that MSTests 2 supports the keyword paramsso it can be used to define arrays:

[DataTestMethod]
[DataRow(NumberValues.NotNumberValue, new [] { "a", "b" })] // явное определение массива
[DataRow(NumberValues.NotNumberValue, "a", "b")]            // массив будет создан автоматически
public void NumberInputValidationTooltipShouldExist(string inputValue, params string[] b)
{
    // тело теста
}

Parameterization with the [DynamicData] attribute

The attribute [DynamicData]is used when non-const values ​​or complex objects are passed as parameters. This attribute allows you to get parameter values ​​from a method or property, with each row corresponding to a test value. The method or property must return IEnumerable<object[]>. This approach also avoids duplicate code with [DataRow]attributes if the same set is used in multiple tests.

In the example below, let’s consider testing a role model with a different set of permissions and getting parameters (a group of rights and their type) from a method and a property:

[DataTestMethod]
// Получение параметров из метода
[DynamicData(nameof(GetSystemRolePermissions), DynamicDataSourceType.Method)]
// Получение параметров из свойства
[DynamicData(nameof(SystemRolePermissions), DynamicDataSourceType.Property)]
public void IsSystemPermissionSetCorrectlyAfterCreate(string permissionGroupName, string permissionName)
{
    // Act
    RolesPage
        .OpenCreateSystemRoleModal()
        .SetRoleName(RoleName)
        .SetPermission(permissionGroupName, permissionName)
        .SaveRole()
        .SelectRoleByName(RoleName)
        .IsPermissionSet(permissionGroupName, permissionName, out var isPermissionSet);

    // Assert
    Assert.IsTrue(isPermissionSet);
}

public static IEnumerable<object[]> GetSystemRolePermissions()
{
    yield return new object[] { SystemPermissionGroupNames.ProjectCreation, PermissionType.Allowed };
    yield return new object[] { SystemPermissionGroupNames.Workflows, PermissionType.Edit };
    yield return new object[] { SystemPermissionGroupNames.Workflows, PermissionType.Read };
    yield return new object[] { SystemPermissionGroupNames.ObjectModel, PermissionType.Edit };
    // много входных данных с разным набором пермишенов
}

public static IEnumerable<object[]> SystemRolePermissions
{
    get 
    {
        yield return new object[] { SystemPermissionGroupNames.ProjectCreation, PermissionType.Allowed };
        yield return new object[] { SystemPermissionGroupNames.Workflows, PermissionType.Edit };
        yield return new object[] { SystemPermissionGroupNames.Workflows, PermissionType.Read };
        yield return new object[] { SystemPermissionGroupNames.ObjectModel, PermissionType.Edit };
        // много входных данных с разным набором пермишенов
    }
}

Parameterization with custom [DataSource] attribute

If the previous two attributes don’t work, you can create a custom attribute that must implement the ITestDataSource. This interface has two methods: GetDataand GetDisplayName. GetDatareturns rows of data. GetDisplayName returns the test name for the data row. This name will be available in the console or Test Explorer.

An example of a class with a custom attribute:

public class CustomDataSource : Attribute, ITestDataSource
{
    public IEnumerable<object[]> GetData(MethodInfo methodInfo)
    {
        yield return new object[] { SystemPermissionGroupNames.ProjectCreation, PermissionType.Allowed };
        yield return new object[] { SystemPermissionGroupNames.Workflows, PermissionType.Edit };
        yield return new object[] { SystemPermissionGroupNames.Workflows, PermissionType.Read };
        yield return new object[] { SystemPermissionGroupNames.ObjectModel, PermissionType.Edit };
    }

    public string GetDisplayName(MethodInfo methodInfo, object[] data)
    {
        return string.Format("DynamicDataTestMethod {0} with {1} parameters", methodInfo.Name, data.Length);
    }
}

An example of using an attribute in a test:

[DataTestMethod]
[CustomDataSource]
public void IsSystemPermissionSetCorrectlyAfterCreate(string permissionGroupName, string permissionName)
{
    // тело теста
}

Conclusion

Using [DataRow]and [DynamicData]attributes will allow you to create parameterized tests in MSTest 2 when developing data-driven tests. They should be sufficient for most cases. If the built-in attributes are not enough, they can be extended when developing your own  [DataSource] attributes to create a data source. However, if you are faced with the task of implementing parallel execution of Data Driven tests, then MSTest 2 is still inferior to other frameworks.