Acceptance Testing in Enterprise Environments

To be successful at acceptance testing enterprise systems and environments means becoming a master of automation. Test automation is about knowing which software testing tools and techniques to use, and which environments to use them in.

In this article, I’ll be sharing some details of a Java-based, end-to-end test framework that I’ve been working with, and listing a set of best practices that came out of this work.

First, let me explain how I see the landscape of acceptance testing in enterprise systems and environments, and share my strategies for navigating this landscape.

The landscape of functional test techniques

For me, there are two ways of seeing the landscape of functional testing.

1. Choose the right patterns

We can test a system at many levels of abstraction, and at many different points in the development phase. To illustrate, let me compare the pyramid testing pattern with its anti-pattern,  the ice-cream cone

With the ‘ice-cream cone’, we perform fewer unit tests, and focus testing effort on the manual, exploratory types of test, by testing the system’s user interface. With the ‘pyramid’, much of the testing effort focuses on unit and integration tests.

The benefits of the pyramid model pattern over the ice cream cone are:

• more can be automated;
• the tests are more compact, easier to maintain, and run faster;
• bugs are found sooner in the development life cycle.

2. Keep it balanced

A crucial part of any testing strategy is in deciding what tests to automate and by how much. Using the ‘pyramid’ test model is a good start, but it’s also important not to underestimate the value of other test techniques, both functional (e.g., exploratory) and non-functional (code analysis, performance, resilience, security, etc.).

A rule of thumb when automating functional tests is to keep user interface tests to a bare minimum, by, for example, only covering critical business paths. Also, whenever possible, prefer testing via APIs.

3. Choosing your tools

The choice of tooling for unit and integration tests relates strongly to the technology in which the component is being developed. Tools such as JUnit, Mockito, and Jest, help developers achieve test coverage early in the build stages. The higher levels of the test pyramid are where we can start to think of common testing frameworks.

Consumer Driven Contracts, supported by tools such as Spring Cloud Contract and PACT, are gaining popularity as a way of validating non-functional API contracts early in the development process. But, how do we validate critical business logic in integrated environments where multiple components and services are working together? How do we organise our test scenarios and make them executable with a reasonable amount of effort?

The answer is with an end-to-end testing framework.

An end-to-end Testing Framework

Despite there being many commercial frameworks available, my team decided to embrace open source, and use well-established solutions supported by large communities. We did this because:

• we like it when things have matured enough to just work;
• we want to benefit from the vast integration and customisation possibilities of open-source frameworks.

Our choice turns out to be correct, as we have now successfully addressed both internal and commercial needs for acceptance testing using the toolset I’ll now describe.

Overview of the end-to-end testing framework

The framework aims to support automated acceptance tests, and is designed to accommodate both API and GUI testing. It is implemented in JAVA, and it can be easily extended to connect to databases and messaging queues.

Test scenarios, created together with business owners and analysts, are written in Cucumber’s Gherkin syntax, and are grouped into features.

[link to BDD article when published]

Tests are executed with TestNG, CubumberJVM, andMaven, based on scenario tags to specify severity, components, etc. Tests can be run in parallel, and test data can be set up using TestNG or Cucumber hooks.

What’s under the hood?

• Cucumber (Gherkin) – scenarios and features
• Rest-assured – REST API step definitions
• JAX-WS – SOAP API step definitions
• Selenium / Selenide – GUI step definitions
• Zalenium / Selenium Grid – running GUI scenarios on different browsers, Video Recording Dashboard
• Allure – common format HTML tests reports
• TestNG and CucumberJVM, Maven – running the tests, before/after hooks
• Jenkins – CI/CD

Support Libraries:

• Selenide / The Waiter – Selenium wrappers
• Swagger Codegen – generating class model from Swagger
• Apache CXF – generating class model from WSDL
• Java Faker – fake data generator

Benefits and Features

1. Ready to deploy – The “boilerplate” is ready to be deployed as a Docker container, either on-premise or in the cloud. We prefer an out-of-the box Openshift or K8s project approach for the most efficient execution environment.
2. Executable test specifications—Gherkin provides a syntax for human-friendly test specifications which can be implemented and run as code. Features, Scenarios, and Steps make up the business-facing test specification, while Step Definitions together with support code carry out the testing tasks. Here are two examples of executable test scenarios.
   • API (basic Scenario):
   • GUI (using a Scenario Outline):
     please note that driving the scenario with data is as easy as adding new rows to the “Examples”  table!
3. Friendly reports—Allure delivers good looking HTML reports and out of the box integrations for RestAssured, Cucumber and Jenkins.
   
   

We’ve further customized our reports to provide:

• References to Video Recordings of GUI tests executions Dashboard

• Links to a ticket management system (e.g., JIRA). When needed, execution results can also be reported back to JIRA (through custom @after hooks).

Best Practices

Here are some best practices we recommend following when implementing and executing the scenarios.

1. Scenarios should be prepared together with business owners and analysts, and written in Cucumber (Gherkin), while taking advantage of features such as:
   • Scenario Outlines and Examples
   • Data tables
   • Tags (e.g. @gui, @api, @blocker, etc.)
2. The scenarios should be independent of each other. I.e., each of them should be able to be run separately (test data should not “leak” to other scenarios).
3. The scenarios should not contain unnecessary technical details, and they should be easy to understand.
4. Scenarios should be grouped in *.feature files according to the tested business domain (not necessarily per user stories).
5. When implementing GUI step definitions:
   • Page Object Pattern is recommended
   • “Sleeps” are forbidden
   • Id, css selectors are preferred over xpath
6. When implementing API step definitions:
   • Common properties should be reused with RestAssured RequestSpecifications
   • Filters should be used for logging
   • Whenever possible, Data Transfer Objects (DTO) should be created from API definitions (.wsdl, .swagger)
7. Tests should be run on dedicated ephemeral environments or with dedicated test data to avoid interference with manual tests.
8. Automatically generated data should be prefixed or suffixed so that it can be separated from manual test data.
9. Systems outside of the integration scope, which we don’t control in our environment, should be mocked.

Why not use Postman and Cypress.io instead?

Postman, together with Newman and Cypress.io, are great tools, but they are semi-commercial and somewhat limited. When working in large enterprise environments, the requirements for automated tests are sometimes simply more demanding.

Taking API tests as an example, we do use Postman as a development and test development aid, but RestAssured gives you more control, both over organizing your tests and executing them.

A good example of this is a test scenario we’ve recently had to cover, which required taking control over the SSL session. While Postman keeps the SSL session open between requests (in a non-configurable way), RestAssured lets you choose whether to keep the SSL connection alive or close it after subsequent requests (via its connection-config methods).

Cypress.io’s popularity is growing, but when full browser control (working with tabs and iframes) and cross browser testing are required, Selenium, despite its disadvantages, still has more to offer.

(UPDATE: Cypress now supports cross-browser testing.)

Keeping it together – more best practices

Here are some more tips on how to get the best out of your automated end-to-end tests.

1. Integrate with your CI/CD pipelines—Automated acceptance tests should be integrated into CI/CD pipelines, as a next step after automatically deploying your applications into dedicated test environments (created for example using OpenShift project templates). Failed acceptance tests mean failed deployments!
2. Make your automated tests part of the development process—Automated test scenarios should be written alongside product development. Writing tests in parallel with developing your features (as opposed to tests as an afterthought) will help you shorten the feedback loop and deliver a better product.
3. Work with stakeholders—Inviting stakeholders (business analysts, manual testers) to work on behaviour-driven development (BDD) test scenarios will let your teams communicate better, and might even help identify bugs or gaps before they make it into the code. Using a ubiquitous domain language helps avoid misunderstandings.
4. Don’t repeat yourself (and your tests)—Try to avoid duplicating business logic coverage across different test levels. End-to-end tests should only cover the most important user journeys in your applications.
5. Act on feedback—Bugs identified on any of the testing steps should be covered by tests at the lowest possible level of the test pyramid.
6. Watch the execution time—It’s a good idea to agree on a time budget regarding the end-to-end test suite execution, for example, 15 minutes. Remember that your automated tests are a “safety net”, and should not make the deployment process uncomfortably long.