Software is becoming an increasingly important factor in nearly every aspect of modern life. Software and SaaS companies like Salesforce depend on rapid innovation to address new markets and remain competitive. It’s also how nearly every industry outside of computer services is defining its products and services.
Consumers benefit from democratized information and global access to goods and services. This not only levels the playing field for small companies competing against enterprises where good ideas can prevail but it also creates new expectations of service. Consumers gravitate towards providers who are faster, more interesting or offer price advantages. They take these expectations into their roles at work and place similar expectations on their partners and suppliers.
As product and service expectations rise, the pressure to innovate through software swells. Fast iteration has become the key to better innovation — releasing quickly, assessing feedback and adjusting the path forward. But to iterate faster, all phases of the software development lifecycle need to move together. No one can afford to have testing be a bottleneck and slow the entire process.
Salesforce’s Community Cloud is massive, with over 500 million users, over 40 sites published daily, and around 50 thousand live customer sites. With so many users and organizations dependent on Community Cloud, if there is a bug, a functionality problem, or downtime, it is immediately visible and can impact customer satisfaction.
Automated software testing:
At Community Cloud we follow the Agile testing quadrants model that was first introduced by Brian Marick, and then refined and popularized by Lisa Crispin and Janet Gregory in their book Agile Testing: A Practical Guide for Testers and Agile Teams.
The agile testing quadrants model described in their book and depicted in Figure 1 divides testing into groups (quadrants) that can help teams plan the types of tests and resources needed to perform those tests. The numbering does not imply an order but rather a way to reference different types of tests and their purpose.
At Community Cloud, we had automated thousands of tests across Q1, Q2 and Q4, but were reliant on manual testing for Q3. Our desire was to extend automation of functional and E2E tests to cover key scenarios within Q3.
Challenges With Automating Q3 Testing
End-to-end (E2E) UI functional testing is where the rubber meets the road. E2E tests measure whether users will encounter errors when using your application. They are the last line of defense against bugs reaching production and impacting user experiences. They are also the most challenging tests to automate as they are the most comprehensive, need to cover frequently-changing applications, and inherently include real-world environment unknowns. In Q3 these are typically thought of as manual or exploratory tests, but we wanted to automate.
We have thousands of Selenium tests that run every day on our internally developed automation framework. However, they are shorter tests aimed at testing key functionality and require significant ongoing maintenance. There are around 40 scrum teams that work on the code tested by these tests, yet the results are often unpredictable, and getting a high-level view of critical use cases is difficult. It can feel like we work for the tests instead of the tests working for us.
Longer E2E tests that track critical user journeys were left to manual testers to validate and understand. Trying to automate these long complex journeys is hard, often resulting in flaky tests that break with minor code changes. Every failed test requires effort to troubleshoot and make fixes to the tests. Additional time is spent re-running the test, slowing the testing process, and extending the release cycle.
A key goal of testing is to find bugs before they reach production. Yet with all of the time spent maintaining these Selenium automated tests, one could question whether manual testing would be nearly as efficient.
Test Automation Criteria:
Our goal at Community Cloud was to automate long, complex E2E tests to give us headlights into whether critical user flows were functioning as intended. These E2E scenarios would help shift some of the Q3 tests from manual to automated. As software engineers do most of the test creation and maintenance, automating these journeys would free up time to do more exploratory testing or address the backlog.
Priorities:
- Test stability. E2E tests need to be an examination of the functionality of the software, not an exercise in keeping tests upright. We want to reduce work, not shift it from one task to another.
- Ease of use. At Community Cloud, we have different user types creating tests: QA Automation Engineers, Software Engineers, and sometimes non-technical users. We want the solution to be easy to set up, manage, and use.
- Fast authoring. Our development team moves quickly. We need test creation to be fast to keep up with our release schedule.
- Completeness of function. The solution needs to support the types of features in our application and the browsers we want to test. It also needs to integrate into our pipelines.
Comparison: Selenium vs. Testim
We had been running a Selenium test environment but we were facing some of the challenges mentioned above such as slow authoring, flaky tests, and high maintenance effort, so we wanted to look at other options.
We looked internally and at other commercial options before trying Testim.io, an AI-based test automation startup in San Francisco with an engineering office in Tel Aviv. Testim uses a Chrome extension to record functional and GUI end-to-end tests that can then be configured and edited in their application. They also use AI-based Smart Locators which uniquely identify each element in the DOM based on hundreds of attributes rather than a single selector. They claim that this makes their tests more stable and we wanted to test that out.
We ran a proof of concept comparing six extensive E2E tests in both the Selenium and Testim environments.
Selenium
- Multiple failures were observed, due to changes in the DOM and locators
- ~12 hour effort was required to identify, fix, and rerun the tests
- Automation / Software Engineer was needed to fix the tests
Testim
- There were no failures due to DOM or locator changes
- Troubleshooting was fast and only minimal time was needed to identify, fix, and rerun tests.
- Tests run on cloud/Sauce Labs and were proven unlikely for setup to go down.
- Any manual testing team contractor can rerun and fix the tests.
One of the key differences was the percentage of P1 bugs caught by each solution. P1 bugs are considered critical and require immediate resolution. With Testim’s solution, we found that nearly 50% of the bugs caught were P1s. With Selenium, only 15% of the bugs caught were P1s.
Solution and Benefits
We decided to go with Testim and have been rolling it out to different areas of Community Cloud applications. Salesforce is a company with a heavy focus on coded automation. Using a codeless automation tool like Testim was a bit of a paradigm shift that required convincing. Users quickly learned how to use the tool and create tests. After a few cycles, the increase in test stability proved the value.
Currently we have over 100 complex E2E tests running across multiple internal environments. These tests have proven to be easy to create, rarely fail, and much easier to troubleshoot and update when code changes.
Benefits:
- For the first time, we’ve automated complex E2E tests in Q3.
- Nearly 50% of the bugs caught by Testim are major P1 bugs.
- Quick and easy onboarding even of non-technical users.
- Engineers are spending less time fixing tests and more time doing exploratory testing.
- A single place to see the status of recent testing of critical paths.
- Significant reduction in the time spent planning for automation.
Conclusion:
End-to-end testing is hard, but it’s also a critical step in delivering quality software. At Community Cloud we are constantly trying to improve our processes to accelerate releases, increase quality, and reduce costs. Our journey to find a better way to automate end-to-end testing led us to Testim. In our side-by-side comparisons, Testim outperformed Selenium for us on key areas of test stability, troubleshooting, environment stability, and usability. We are now in the process of evaluating and rolling out Testim to additional cloud teams within Salesforce.
