This week, I sat down with James Taylor, the PMC Chair for the Apache Phoenix project (and long-time co-worker). If you’re not familiar with Phoenix, you can think of it as “SQL on HBase” (though as you’ll see below, it’s grown to be more than just that). You can get started here, with Phoenix In 15 Minutes Or Less.
Phoenix was the first top-level Apache Open Source project to come out of Salesforce Engineering. James and I talked about where Phoenix came from, and how it got where it is today.
Ian Varley: So, James: What is Apache Phoenix? How do you describe it to someone in an elevator?
James Taylor: My answer to that question has morphed several times over the life of the project.
IV: If I’m not mistaken, it used to be: “We put the SQL back in NoSQL!”
JT: Ha, yeah, that was one of our earlier one-liners. I always thought that was kind of funny. More seriously, though, for a while we described Phoenix as “a SQL skin on HBase”, and then later, as a “relational layer on HBase”. Those are accurate, but my current favorite one-liner is Phoenix is “online transaction processing and operational analytics for Hadoop”.
IV: That’s much broader; why the evolution?
JT: Well, over time it has grown beyond being just “fluff on HBase”. We certainly do use HBase as the backing store for Phoenix, and it abstracts the user away from all the details of using HBase. So, it’s not wrong to describe it as that. But, it’s also more than that.
Essentially, if you have some web application where you’re getting data from Hadoop, and you’re comfortable with HBase, or you’re already using HBase, then you can surface that data through Phoenix directly to run a web application. It supports building low-latency apps where data is stored in Hadoop. And because of the properties of HBase and Phoenix, that means it can include both low-latency OLTP usage as well as analytics usage.
IV: What’s the history of Apache Phoenix? How did you get involved?
JT: So, originally Phoenix started out as an internal project here at Salesforce. We were all figuring out what to do in our big data stack in the early days. HBase was being rolled out in prod, as described by Andrew and Lars in their post last week.
But on the engineering side at Salesforce, we had a conundrum: everyone at Salesforce knows SQL! It’s the data interface language people have been writing in here for, literally, decades. So we knew that if we were going to be successful long term, we needed something that would allow you to do SQL on top of HBase.
So, that’s how Phoenix was born–out of that desire. At first, the initial SQL support was pretty narrow, but it grew organically as we needed it for our use cases. As teams needed support for more constructs in SQL, we’d work with them to build that in, and within a short time it was actually pretty usable.
All of this was happening internally at Salesforce, of course. But, we realized quite quickly that this was a hole in the Hadoop ecosystem: something that would support low-latency applications using SQL over Hadoop, for mutating data, not just querying it. So we open sourced it on Github, originally on the forcedotcom org (since we hadn’t created the SalesforceEng org yet).
IV: And when you open sourced it, was there interest from the community?
JT: Oh, absolutely. We already had a pretty big presence in the HBase community, on the mailing lists and as contributors and committers. So people knew who we were, and paid attention when we said we had created this. A lot of folks jumped in quite quickly and started experimenting with it, using it.
IV: How long was it “just” a project on GitHub?
JT: It lived on GitHub for about nine months or so. It was at that point that we proposed it as an incubator project at the Apache Foundation. Along the way, we picked up several external company partners, including some early significant contributions from folks at Intel. It spent 6 months in incubator status, and then graduated as a top-level apache project–almost 2 years ago now.
IV: I remember! You made a cake, it was the first cake I ever saw with a footnote!
JT: That’s right! Yeah, we had a party during HBaseCon and a lot of the folks from that community came out in support. We roasted hotdogs.
IV: And has the community expanded since becoming an Apache project?
JT: Dramatically. Phoenix is part of the Hortonworks distribution of Hadoop, so it goes out with that. It’s also part of Cloudera labs, which is an add-on they offer. Our committer list includes someone from Cloudera, several folks from Hortonworks, a few people from Salesforce, Intel, NGData, and a variety of other external contributors.
IV: Why did you choose to release it as Open Source in the first place?
JT: Yeah, that’s an interesting question. I’ve always been personally interested in Open Source, as I think it’s a good way to grow a project. Like a lot of companies, Salesforce engineers always have more ideas they want to do than are actually possible, and each group is vying for resources. Open source became a really good way for us to grow Phoenix faster than we could do solely with folks in our own team. There’s a breadth and depth to that growth, because it’s so grassroots.
It’s also really interesting to see what happens when you do that: you see how other people want to use the software, in ways you didn’t envision originally. It grows in new and interesting ways. For example, now it has tie-ins with other projects in the Hadoop ecosystem: Spark integration, for example. We didn’t write that, it was a contributor in Canada at Interset, who needed two to work together; he did the integration, donated it, and now he’s part of the project PMC. It’s the same with Apache Pig, Hadoop map/reduce … we’re even doing Sqoop integration now.
IV: And was this integration the reason you picked the Apache foundation as the home of the project, as opposed to any of the other foundations?
JT: Yes, it’s important where it goes. Nearly all of the Hadoop ecosystem projects today are part of the Apache foundation.
Landing in a foundation is an important step in maturing, as a project. If you want external contribution, you should consider going to an organization that people at other companies are already familiar with and trust–one with a strong contributor focus, where someone who contributes has a clear path to becoming a committer, joining the PMC, etc. Being on the PMC gives you voting rights on projects, which means you help control their destiny and direction. That was our target–having that independence so contribution could come in from any source, and the direction would be a meritocracy.
IV: What are the gates to becoming a “top level” project at Apache?
JT: You have to show diversity on your project–that you’ve got committers across multiple organizations, and that you’ve got a community around this project in terms of the people using it, asking questions on mailing lists, etc. To get into incubator status, you just have to show that you’re interested in external contributors, but to graduate as a top level project, you have to actually have committers. We had that very early on, and that’s why our path to top level project was relatively short.
IV: What release are you on now, and what have been the major milestones?
JT: We’re currently on the 4.7 release, and 5.0 is on the horizon. We started at 1.2 when we open sourced it.
As we’ve grown, that’s happened along different axes. One is the depth of our SQL support. In the 1.x series, we were focussed on aggregate queries; that was our bread and butter. Then in 2.0, we introduced JOINs, and that was a big deal. It took our query planning requirements to a whole new level.
In 3.0, we pivoted on a different axis: supporting multiple versions of HBase (which was, and still is, the underlying storage and retrieval layer). In addition to support for HBase 0.94, Phoenix added support for HBase 0.98, 1.0, and 1.1. Now when we do a release, we actually do simultaneous releases across each HBase version that we support. On top of this, we firmed up our backward compatibility contract to ensure that minor upgrades can be done without taking any downtime and with complete backward compatibility (more on that here).
And now, we’ve introduced transactions, that’s the big new feature that’s in Phoenix 4.7 as a beta release.
IV: Can you explain a bit about what transactions are, and why they’re important?
JT: Sure, of course. In a nutshell, a database transaction is a mechanism that allows you to group related changes together into an atomic, isolated unit. So, the classic example of this is a bank transaction. If you want to transfer money from one account to another, there are two separate steps: decrease the balance of one account, and increase the balance of the other account by the same amount. If only one of these steps happens but not the other, you’re in big trouble–you’ve either created money or deleted it, depending on which part you completed. So it’s critically important that you be able to guarantee that either they both happen (which we call a “commit”), or that neither of them happen (which we call a “rollback”). And it’s also important that other users who are looking at the database don’t see the “in progress” results, because that could be just as bad.
IV: Got it. HBase is a database, but it doesn’t support transactions?
JT: It does, but only in a limited way. As a distributed database, supporting transactions is quite tricky actually. So out of the box, HBase only supports transactions within a single row of a database, not across rows or tables. The support we’re adding in Phoenix extends the transactional capabilities to the entire database.
IV: Why is this an important feature to your users?
JT: Beyond the ability to use explicit transactions, there’s actually another more immediate need: consistent secondary indexes. Let me back up and explain what that is.
Generally, in a database, you want to be able to access information in a variety of ways. You might want to look up a user in the user table by their email address, but you might also want to be able to look them up by name, or by an ID number. If you have a small amount of data, it’s no big deal, because the database can just scan over all the data to find your record. But when you have a huge, distributed database, with billions or trillions of records, doing that could take a very long time. So you need indexes–maps of how to use a different key (like “user name”) to find the actual record.
In HBase, the data store only keeps one index for your table: your primary key constraint (which is how the data is physically laid out on disk). If all your queries want to look records up by that primary key, you’re golden–your queries will be extremely fast. But invariably, you want to start doing queries outside of just the primary key, and that’s where secondary indexes come in.
In Phoenix, you can create a secondary index on any group of columns–in fact, we even support functional indexing on an expression. Then at runtime, the Phoenix query planner will automatically use either your primary data table or your secondary index, depending on what’s more efficient. And it will automatically keep that index in sync after every operation that changes data.
This is great, but where it starts to break down is in failure scenarios. The secondary index is actually implemented as a separate table, and as I said before, HBase has no consistency guarantees when writing to multiple tables. So that’s what transactions give you: when you commit a batch of rows, you can guarantee that it’s atomic, it will either succeed or fail completely.
So, if you put that together: now you have transactional secondary index, so that means the index won’t get out of sync with your data table if there’s a failure at any point in the operation.
IV: So, it provides the behavior that SQL programmers have come to expect from secondary indexes in relational databases.
JT: Exactly! And that makes programming against it easier. When you’re programming, the last thing you want to worry about is, “What do I do if my commit fails?”. That’s a very hard problem to reason about and deal with. Do you build some kind of offline process to fix things? If the data layer can deal with it, life is just easier.
IV: How is transaction support actually implemented in Phoenix?
JT: We’re actually leveraging a project called Tephra (which was actually just accepted as an Apache incubator project itself). Our work was in gluing that engine together with Phoenix. But it was pretty straightforward, that project did the heavy lifting on the transaction side.
IV: And what’s next for Phoenix?
JT: The 5.0 release is next. That’ll be a “generally available” release for transactions, and also introduce a new tie-in with Apache Calcite, which is a query parser / planner / optimizer framework used by other projects like Drill, Hive, Kylin, etc. It supports full SQL-92 compliance, which is important for us as a next step. It also has a pluggable cost-based query optimizer framework that we’ll leverage; you tie in your rules to this framework, and it has these nice layers where it generates a logical plan, uses relational algebra, etc. You can optimize over that, that’s where the rules come in, because there are always different ways to run the same query.
Calcite then spits out a fully optimized physical plan: where to get what, and how to do it. We’re plugging into that, which opens the door for a lot of interoperability with other projects that use Calcite. Things like Flink could plug in, or cube building in Kylin, or Drill for multi client node queries, and bigger analytic queries. So that’ll be the 5.0 series.
IV: What types of things are people using Phoenix for at Salesforce?
JT: We actually have 4 generally available products that are running on Phoenix today. A few of them were briefly described in Lars and Andrew’s post from last week: the Field Audit Trail work, which archives entity history; the Platform Event Monitoring, where customers can get information about how their app is running in production. The third is a component of the “Read Only Mode” that we can use for customers during certain relational database maintenance windows–Phoenix allows a subset of the SQL write traffic to proceed to a different destination and then be synced back into the relational database at the end of the window. And the fourth is an improvement to the relevancy of @mentions in Chatter (which is selfishly my favorite, since we use Chatter every day). When you type the “@” sign in a message, there are Phoenix queries running to find out who’s the most relevant person or group to mention.
There are a bunch of other uses that are in the development pipeline now, and we’ll talk about them more when they’re rolled out.
IV: James, thanks so much for taking the time to chat. Looking forward to seeing how Phoenix evolves over the next few years?
JT: My pleasure, thanks.
Do you use Apache Phoenix? Have you contributed to it? Give a shout in the comments.
