Shifting left with end-to-end automation testing

The shift left paradigm in quality testing is a defining trend since agile methodologies have been adopted as the golden standard of software development. By shift left we mean checking for code quality earlier in the development cycle. In many organizations, the end-to-end test automation still follows a waterfall model, and often the automated tests follow the main code by a sprint or two. We now check code quality at unit and component level by providing early access to the end-to-end software application being developed, whether web services, web applications, or mobile applications. The quality engineers can start validating the final product much earlier.

Reasons Test Automation Strategy Fail

• Maintaining separate repositories:

In typical organizations, the above diagram is the common structure of test automation pipelines.

The development code for the Website, API or App, which we call test artifacts, reside in the main repository under source code control. Using different branching strategies, a dedicated release branch is built using a CI/CD tool like AzureDevOps, TeamCity, CircleCI, etc., and the test artifact is deployed to a test environment, which hosts the test data and configurations.

The test automation code resides in a separate repository, and on a schedule like overnight, the test automation jobs are triggered by Test Runner jobs, typically under the control of QA automation teams like in Jenkins to run the automated tests against the test artifact.

The following goes wrong in such setups:

• Outdated test automation code runs against newer versions of the test artifact giving wrong indication of the build quality.

• Updating test automation script will come down to individual responsibility of the test engineer, because potentially due to different repositories, the sanctity of test code repository is treated lower than the main development code repository.

• The test automation pipeline will blindly re-test the same artifact because there is no connection of build state change from the development pipeline to test automation pipeline, wasting computing resources.

• Over a period of time, with test automation code lagging behind development code and failures in test runs due to unfixed code issues, the level of confidence in the build quality provided by the automated tests goes down drastically.
  

Different tech stacks:
Another common pitfall companies take is to have different tech stack for development code and the test automation code, for example the website might be built using Angular JS but the test automation code will be Java 8 and Selenium-Java.

The following goes wrong in such setups:

• This can result in developers being unable to review or modify the test automation code if there is a skill gap.

• The test developers will not be able to make use of reusable components the development code used inside the application for testing. They have to develop redundant fixtures in the technology used to develop the test code.

• Configurations saved in different environments:
  When configurations like test data, third party integrations like Authentication/ Authorization, stubbed payment gateways, or server hardware configurations are saved on different test environments, the risk of replicating them in brand new test environments are compounded. Often, this will result in creation of golden gated test environments, whose uptime is the biggest factor determining the success or failure of test automation.

• CI/CD ambiguity:
  When test automation pipelines are on a separate tool all together, say development code is built using Azure DevOps and test automation scripts are triggered by Jenkins for example, the purpose of an incremental change in the development code getting tested by the test automation code is quite literally lost in connection since the systems are separate.

• Unmanageable test automation code:
  All of the above-mentioned problems get compounded when test automation code grows unmanageably. Without proper code quality guidelines, and reviews for the design, modularization of test code takes a hit. With rapidly changing features in development code, the test code lags behind to the point where it becomes useless and obsolete. You end up receiving a better return of investment when starting something new altogether.

• Delegation of concern:
  With separate repositories, build pipelines, and technologies comes an unintended sense of apathy towards test automation code, and the responsibility of maintaining them solely falls on test developers, even though quality of test code should be a responsibility shared by everyone in the team.
  

How Shift Left Can Help

• Check test automation code in the main repository:
  We propose checking in the end-to-end test automation code in the same repository as development, making sure that it adheres to the code quality guidelines of the repository. This eliminates the unknown factor of what the test code does for everyone on the team.
  
  It will help in making use of re-usable components in the development code for test automation like JSON Data Transfer Objects, endpoints, environment variables etc., without maintaining them in the separate end-to-end test repository.

• Use the same tech stack as development:
  When development code and test-automation reside within the same tech stack, it allows for efficient collaboration between developers and testers resulting in quality peer reviews and feedback loops. The end result is that the reusable code stored in the repository is modular and easy to maintain

• Configurations of environments:
  The configuration of environments, namely third-party integrations with access management, payment gateways, endpoint URLs etc., should be saved in the common repository, making them easier to deploy in any new environment. Sensitive information like passwords and access keys should be stored in safe locations for example Github secrets, last Pass vaults etc.
  
  When all the configurations are saved there, this eliminates the environment down risk if the machine goes down.
  
  Please note that this should not apply to test data stored in databases or other datastores, as their backup and restore should be managed separately.

• CI/CD clarity:
  With the same instance of CI/CD agent managing both development pipeline as well as the test automation pipeline, the two pipelines can be combined together based on various business use cases and costing decisions. The test runs can now be done on an on-demand and on-needed basis rather than ad hoc nightly test runs.

• Automate what’s critical and maintain those scripts:
  Having a huge number of automated end-to-end tests is of no value if the majority of them fail due to issues in the test automation code. So, it is imperative that an optimal number of tests are automated which will give a fair assessment of quality measure of the application.
  
  Another common mistake is to directly map a manual test to an automated test. Given the time and hardware constraints of running automated tests, it makes sense to cover maximum verifications in multiple manual tests during one navigation in an automated test.
  
  All the above points mentioned will serve as guidelines to keep the test scripts accurate to the latest versions of the software.

• Unification of concern:
  When all of the above points come together, the ownership of automated tests will be with the entire team, as the value offered by these tests can be utilized by the entire team to develop top class software applications.
  

Proposed Sample Workflow

Here is a sample workflow to demonstrate shift left in the test automation process.

The code can be organized in the same repository along with the configurations for various test environments in the image above.

Once we have the proposed code structure in place, using the configuration file for the test environment, do a local build and deployment of your application. For example, in your node setup:

• install your dependencies

• install software needed and set environment variables if any

• create a script in your package.json, say by name e2e-test, which will be used to run the end-to-end test

• bootup any Simulators if needed in case of Mobile apps

• do a local build

  • For react-native, issue

    • Android: react-native run-android

    • iOS: react-native run-ios

    • This would build and deploy the react-native mobile app into the connected Simulator. In case of iOS, a new Simulator would be booted up and app will be displayed.

• For Flutter, issue: flutter run

  • This would build and deploy the flutter mobile app into the connected Simulator. In case of iOS, a new Simulator would be booted up and app will be displayed.

• For Angular, issue: npm start

  • This would deploy the webpages to http://localhost:4200

  • Call the e2e-test script to run the end-to-end test against the local deployment

These steps can be scripted into a standalone script or CI/CD yaml file of CircleCI or Github actions. Last but not least, in order to accomplish this, team members need adequate hardware and tools like performant laptops and operating systems as well as the right training and skills.

Conclusion:
In summary, by co-locating development and end-to-end test automation code, making test automation code adhere to the coding standards of the repository, re-using components, sharing the responsibility of test results, and having a pre-check in end-to-end test locally on Simulators; the quality of the check in can be gated earlier, before any deployment to the test environments by finding defects faster, and resolving issues earlier.

Please note: we use node centric frameworks like React, Angular, React-native and Flutter, in the examples provided. For other technologies, we are happy to analyze, consult and build test frameworks for your organization, based on your tech stack.