Written by Viraj Jasani and Andrew Purtell
Data is the engine behind Salesforce operations, helping our customers make better decisions on a daily basis. The Big Data Storage (BDS) team, a key part of Salesforce’s engineering organization, deploys arguably one of the largest distributed database production footprints. This infrastructure is based on the combination of open-source HBase and Phoenix, designed to empower high scale, high throughput, low latency, real-time random read-write access to petabytes of data. This is a capability not achievable with off-the-shelf traditional relational databases.
In 2014, BDS described its motivations in detail, encapsulating how HBase and Phoenix solved many technical problems during the team’s ongoing 10+ year journey. An important part of this journey involved developing a strategy for upgrading infrastructure. At Salesforce’s scale, where customers demand non-stop, disruption-free 24/7 operation, significant challenges arose.
Read on to explore how BDS drove the upgrade from HBase 1 to HBase 2 with minimal impact to Salesforce customers.
Why upgrade to HBase 2?
For many years, running HBase 1 and Phoenix 4 presented minimal issues. Nonetheless, a major upgrade to HBase 2 and Phoenix 5 was pursued for several reasons such as:
- Availability. HBase, a strongly consistent (CP of CAP theorem) distributed database, lacked high availability during table region transitions (see our existing post on details of region transitions). HBase 2 significantly rearchitected region assignment workflows by eliminating its dependency on Zookeeper. This increased overall datastore availability and boosted infrastructure stability while maintaining customer trust.
- Need for speed. HBase 2 enhanced write request throughput with a design that writes to multiple replicas of the Write-Ahead-Log (WAL) files asynchronously — key for applications utilizing a database, delivering a high throughput.
- Low latency. Being a latency sensitive database, BDS encountered issues from long GC pauses at times. HBase 2 and Phoenix 5 provide compile and runtime support for Java 11. Using Java together with Shenandoah GC in production helped reduce latency.
- Less idle threads. During production, BDS regularly reported a higher number of idle threads that were intended to initiate Remote-Procedure Calls (or RPC). HBase 2 supports an asynchronous, event-driven, non-blocking socket-based Netty framework for reduced resource consumption and improved throughput during RPC calls.
- Quick security fixes. Staying close to the latest major open-source releases drives fast and effective resolution of third-party security vulnerabilities. HBase 1’s release line end-of-life-status further justified the move to HBase 2.
What challenges did BDS face?
Regardless of the upgrade, maximizing availability for customers is a key focus. Maintaining 99.999% of availability is critical, making any bugs and behavioral changes a priority for solving. Consequently, any major, minor, or patch upgrade of HBase/Phoenix meant that BDS had to maintain availability both during and after the completion of the rolling upgrade.
Salesforce has been engaged in a multi-year project to migrate from its own private datacenters into the public cloud infrastructure (more details here), and the flexibility of the public cloud infrastructure can greatly improve BDS team’s capability to perform a seamless migration by leveraging additional resources to build an exact replica of the given cluster, followed by a switchover of client traffic from the old version running cluster to the new cluster.
However, as of this moment, a significant number of clusters are still running on first-party data centers. Therefore, the upgrade from HBase 1 to HBase 2, along with Phoenix 4 to Phoenix 5, comes with a critical challenge: it had to be performed without any downtime regardless of the underlying infrastructure.
As an example of scale, production clusters with approximately 300 nodes take more than 5 hours to complete the rolling upgrade of all components. The significant re-architecture with HBase 2 makes performing the upgrade without downtime impossible. BDS has worked with the open-source community to find solutions for this as it’s a fairly unique challenge for organizations at Salesforce’s scale.
How did BDS prepare?
With any complex goal comes significant trade-offs. BDS methodically evaluated core functionalities in HBase/Phoenix, determining whether non-critical background activities could be temporarily disabled during the upgrade. This journey occurred over a two-year period, marked by unwavering dedication to delivering a seamless production rollout, which involved several key steps.
Stabilizing builds was an initial priority, which ensured stability for HBase and Phoenix downstream projects. This encompassed in-depth verification of code compilation and successfully testing the entire CI build.
The toughest challenge emerged when BDS needed to evaluate various tradeoffs and develop a strategy to execute a seamless rolling upgrade while eliminating downtime. This required significant time, developing an exact sequence of steps that achieved the goals while maintaining the service, while also working around compatibility issues in the intermediate states.
As the team encountered issues related to system table schema differences between the two versions, they recognized the possible unattainability of the task. However, BDS successfully resolved this by implementing a procedure that could update schemas without disturbing compatibility between old clients and new servers.
Additionally, as they encountered other hurdles, BDS contributed their fixes to the open-source codebase.
And as they used HBase and Phoenix, the team ensured that clients using JDBC connections for querying data could continue performing CRUD operations with minimal retries during the upgrade.
Finally, to manage the zero-downtime upgrade, it was quite crucial for BDS to emphasize building a procedure facilitating the transition from “Zookeeper based” to “Zookeeper-less” table region assignments — a critical prerequisite for their upgrade.
The team also performed rigorous performance testing throughout the lifecycle of the project. Being one of the active members of the open-source community also helped them get better traction on critical bug-fixes that they contributed.
A look at HBase 2 pre-requisite + upgrade procedural steps. (details here)
Learn more
- For a deeper dive, check out the in-depth technical details for this project here.
- Stay connected — join our Talent Community!
- Check out our Technology and Product teams to learn how you can get involved.
