Script Flakiness (Reasons & Resolution)

Author : Anisha Jain

What is Flakiness?

Flakiness is a term used for false failure in scripts. It is used to describe a script which is failing with no change & error in code. Flaky tests can be extremely painful to debug as they are hard to reproduce but at the same time require time & effort for fixing.

Why is it Important to reduce script flakiness?

False failures are never good. In case the team does not focus in reducing these occurrences, these numbers would increase exponentially with the growing automation suite. Script flakiness is something that cannot be avoided but we need to keep it in limit with a preventive approach.

Increased Flakiness can result in script failures when there is no issue which can lead to

1) Increased time in debugging the issue which is not cost effective

2) Repetitive False failures can lead to team members ignoring issues which can further result in actual bugs getting missed.

From both the cases described above we get to know that scripts are only useful when flakiness is very less & proper proactive measures are taken to make scripts reliable & stable.

Also one needs to ensure that reasons of new script flakiness are documented properly with their resolution. We need to ensure that these resolutions become core of our framework leading to a stable automation suite.

“Our Past learning can help us in developing a robust future”.

Reasons for Script Flakiness

There can be multiple reasons for Script Flakiness, I have mentioned few important one below

1) Infrastructure Challenge: Here scripts are failing because of slow network, DB slowness, and other environmental issues.

These type of Issues can be resolved by

a) Improved Infrastructure

b) Retry Implementation

2) Multithreading: Sometimes scripts get passed when executed separately but fails when scripts are executed in parallel using multithreading.

These types of Issue’s have to be resolved by framework enhancements & may require significant effort.

3) Hardcoded / Poor Test Data: Sometimes Automated scripts are executed on hardcoded test data. This data can be changed by other scripts or some action by other flows. This hardcoded data is one of the major reasons for flakiness.

We should use dynamic data while writing test scripts to handle and reduce this type of flakiness.

4) Dependent Test: Flakiness in some scripts is due to dependent test & dependencies not resolved properly. Some Scripts may have pre-requisite but if these are not handled properly, it may result in random script failures.

To resolve this all dependencies are to be addressed properly in automation.

5) Async Wait: In many cases failure’s happen when the script is unable to find some UI element on the page. Usually such are sometimes handled by poor sleep statements. A sleep statement for less than a second may not solve the issue & a sleep statement for more time can make whole automation suite processing slow. Thus we should use waitFor statements in such cases, with waitFor statement the test waits for a certain condition to become true.

Steps taken to Reduce Flakiness in Shiksha Automation suite.

In Shiksha we have taken multiple steps at different level to reduce flakiness which can be broadly characterized into following

1. Prevention Taken before Script Execution: We have taken many steps before actual script execution to check environment stability, this includes ensuring the new build is deployed before execution, environment required is up & running etc.

2. Execution Step Level Handling: There are multiple handlings done at individual step level to ensure that step level perquisite is present before actual step execution.

3. Retry Implementation at Multiple level to reduce Flakiness

· Retry at Test Case Level

· Retry at Script Level