At Salesforce, performance is a huge part of application development. From the beginning, we knew that in order to be successful and still maintain aggressive release cycles, we had to make performance analysis a core part of our development lifecycle.
As a senior performance engineer here, I am deeply familiar with the scale and performance considerations we make at every step in the feature development process. So, how does Salesforce maintain seamless performance while delivering multiple releases a year packed with new functionality? Read on.
We collaborate early and often
One way we ensure performance is considered throughout is by assigning a dedicated performance team to every app feature scrum team. During each release, a performance engineer works with their designated scrum team to capture and prioritize all the performance related tasks that need to be accomplished during the release. They provide early and actionable feedback to the developers on any performance issues such as reviewing architecture design, proof-of-concept tests on different implementation options, etc. This helps to catch and fix any bugs or regressions before the feature ever hits production.
By educating teams on performance best practices, and providing self-service tools to easily measure performance data, we have managed to greatly streamline the performance analysis process. With the daily transactions running into billions, our performance engineers worked diligently to optimize software code release over release. Along with our infrastructure/capacity increase, we have managed to keep our perf trust within the 300ms range.
To empower developers to find front-end performance issues as part of their development cycle, we developed powerful visualization (e.g. waterfall charts that shows dependencies, duration of an action, how much time is spent waiting to process JS) and integrated those visualizations with the chrome developer tool. This allows the developer to correlate the network timeline in the chrome developer tool with the aura action characteristics of a page. By enabling developers to easily measure the performance of the features that they are currently building, performance engineers get to solve for the more complex performance issues that require a deeper level of attention and analysis.
Automate our regression analysis using workloads
Workloads help simulate the functionality to be tested by generating and loading synthetic data shapes and volumes in a production-like test environment based on expected production traffic. Workloads are huge here at Salesforce. We use them for testing any regressions between the releases as well as measuring stability over time.
Stress and stability test for peak loads
At Salesforce, our customers — particularly our biggest enterprise customers — are constantly pushing our platform in new ways. It becomes absolutely critical for us to be able to scale and perform well at the highest levels of expected volumes. My team — Enterprise Scale — works with some of our top strategic customers to come up with a workload that simulates a day in the life of the customer. It takes into account their data shape, data volumes, peak traffic volumes, and skews. We then analyze the workload results for performance and stability, and work with individual feature teams to file any bugs to deliver any improvements that were uncovered. We also work with our product teams to come up with roadmap items related to any gaps in functionality with respect to scale or performance.
Monitor, analyze and react
Regardless of how much testing or analysis we do up front, there is almost always something that goes unnoticed. That’s where some of our monitoring tools come in handy. These uncover trends in our page load times and various other performance metrics, which we can analyze and use to predict performance bottlenecks. We also loop back things we’ve learned from customer production cases to enhance our performance and scalability test coverage.
For many major industries like finance, healthcare, or retail, every second matters. An increase in speed can directly help increase the revenue — for example, by increasing the number of healthcare claims processed or increasing the number of items sold and shipped. On the other hand, a frustrated customer experiencing bad performance will result in poor adoption. We know that performance is a core feature that our customers have come to expect out of our platform. Hence performance engineering at Salesforce involves collaboration, architecture design, test plans, analysis, visualization, workloads, monitoring and much more.
