Overcoming Complex Obstacles: Revealing Tableau’s Process for Creating New Features and Functionality

In our “Engineering Energizers” Q&A series, we examine the professional life experiences that have shaped Salesforce Engineering leaders. Meet Patrick Green, a Lead Member of the Technical Staff for Salesforce Engineering. Patrick supports the Project Sharing and Permissions (PSP) team at Tableau. Feature development plays a crucial role in their work, ensuring the smooth functioning of various Tableau products — including Tableau Cloud, Server, and Desktop — providing a structured approach to data management, and maintaining data security.

Read on to explore how Patrick’s team manages complex engineering challenges to develop and implement, and troubleshoot these features.

What is your team’s mission?

A permission rule establishes who is impacted (a group or user) and which capabilities they are allowed, denied, or unspecified.

Patrick shares his passion for Salesforce Engineering.

At a high level, how do you solve technical challenges that arise during the feature development process?

PSP employs a collaborative approach. We start by brainstorming ideas and creating high-level designs during meetings. To gain a deeper understanding of the challenge, we assign spikes to team members who conduct thorough investigations. These spikes involve delving into the technical aspects of the feature and assessing its integration with our existing codebase.

Timeboxing is an essential aspect of our spikes, ensuring that spikes are completed within a set timeframe. This helps us maintain momentum and keep the development process on track.

Once the spike is completed, we reconvene as a team to discuss the findings and gather feedback on the proposed designs or wireframes. Based on this feedback, we create work items and begin implementing the feature. However, if the challenge is straightforward, we can proceed directly without the need for extensive investigation.

What is the process for creating a new feature?

PSP follows a structured process for creating new features. First, we meet as a product organization to discuss high-priority features and understand customer needs — aligning our work with the company’s overall goals and objectives.

During that meeting, PSP also discusses enhancing previous features. In other words, some features move to the next release and we continue to make improvements on them over time.

Once we have identified the features for a new release, we estimate the effort required for each feature using story points, which factors in team size and velocity to estimate the time needed to complete work items and the amount of work we can fit into the release. We then announce that to the organization and begin building the features.

Patrick discusses how his team creates cutting-edge features.

What’s a specific technical challenge that your team faced and how did you overcome it?

We monitor various aspects such as function calls, public APIs, and critical endpoints. Based on performance metrics and other factors, we create alerts according to our organizational requirements. Alerts serve as indicators of potential issues or symptoms, such as high latency, and help us prioritize and investigate high-priority issues. For example, if the average latency for our API is expected to be 5 milliseconds, we would receive an alert in Slack if it exceeds 10 milliseconds.

Unfortunately, code evolution and changes in customer API usage patterns can impact the effectiveness of our alerts. Therefore, we continuously collaborate with other teams to fine-tune the alerts and address any noise or false positives, which ensures we receive accurate data on our backend services.

When facing performance issues like increases in high latency, the PSP team hunts down the source of these increases by collaborating with different teams and investigating backend services. This involves identifying any code changes, tracking memory leaks, crashes, and other factors. Each issue is unique, requiring a thorough investigation.

How do you approach prioritizing issues and troubleshooting after a feature releases?

The PSP team prioritizes authorization-related problems as they have a significant impact on user capabilities and content functionality. Authorization refers to the permissions code that determines what users and groups can do with Tableau content.

Image depicting how capabilities are evaluated through multiple levels of content.

To address these issues, the team remediates quickly and has a root cause analysis (RCA) process in place. The RCA process allows us to identify the underlying causes of incidents and improve our code to prevent similar issues in the future. Through internal monitoring and being on call, we can swiftly track down issues in production, often catching them in pre-production environments. PSP follows an on-call rotation, triages issues, and collaboratively resolves them. If needed, we reach out to other teams for support, making issue resolution a team effort.

Patrick explains what it’s like to work for Salesforce Engineering.

Can you share a complex challenge that required cross-team collaboration and how you worked together to solve it?

Recently, our internal API — which is used by the search team within Tableau — experienced high latency. To address this issue, we worked closely with the search team to identify the specific API causing the latency. PSP utilized tools like Grafana and Splunk to track down the various API calls and pinpointed the problem to high memory usage, resulting in timeouts in our backend.

Once PSP identified the source of the issue, the search team deployed a fix for the problem, which resolved the high latency they were experiencing. This improvement meant that customers no longer had to deal with slow loading times or extended spinners when using the search bar.

Together, all teams that were involved with the incident attended a RCA meeting. This meeting was very helpful for tracking down the root cause, which is invaluable for helping our team improve our features and operational skills. During the meeting, we analyzed time to fix, time to detect, and other metrics. We also utilized the Five Whys process to reach a root cause. Ultimately, by working together, we were able to identify and resolve the issue efficiently, ensuring a better experience for our customers.