Introduction
Salesforce was one of the first major enterprises to adopt OpenJDK 11 at scale in production, starting our adoption journey shortly after its release in late 2018.
Cutting edge? Sure.
Safe? Absolutely.
You might not know this, but Salesforce has led the industry in integrating leading-edge, transformational technologies and providing them to our customers in a safe, secure, and seamless manner, without compromising our #1 core value: Trust. From gRPC to Kubernetes, Salesforce has a history of making early, bold advances into new tech.
In this case, upgrading the main Salesforce CRM application to OpenJDK 11 was a massive, cross-organizational undertaking. Getting it done right allowed us to harness several years’ worth of Java runtime innovations, provide a better experience to our customers, and contribute back to the open source community.
Background
In late 2018, OpenJDK 11 became available as the latest long-term support (LTS) Java release. This kicked off a long-awaited opportunity to move the Salesforce app forward and bring great new features and innovations to our internal developers.
Why did we think we could safely make the leap from OpenJDK 8 (the last prior LTS release) to OpenJDK 11?
Well, for one thing, we didn’t exactly make the leap in a single jump. While we only intended to use them as stepping stones, not in production, we upgraded to OpenJDK 9 and OpenJDK 10 as soon as these versions became available. As expected, the hardest part was the upgrade from OpenJDK 8 to OpenJDK 9, which required major changes to the Salesforce app. Upgrading from OpenJDK 9 to OpenJDK 10, and later from OpenJDK 10 to OpenJDK 11, required only relatively minor changes.
For another, Java’s backward-compatibility guarantee allows application code developed and compiled with an older version of Java to run on a newer version, and the power of this should not be underestimated. Backward compatibility is a huge help for migration because most of our code didn’t need to change.
One way that the Salesforce application takes advantage of Java’s backward compatibility is by separating the Java version used to build the Salesforce app from the Java version used to launch it. This allowed us to first focus on upgrading one side of the process from OpenJDK 8 to OpenJDK 11, while keeping the other unchanged at OpenJDK 8 and deferring its upgrade to OpenJDK 11 to a later time. Our goal for internal developers was to make switching between the OpenJDK 8 and OpenJDK 11 runtimes as simple and seamless as possible, by hiding all the differences and complexities inside the scripts used for initializing and launching the Salesforce app. As a result, for our developers, upgrading to an OpenJDK 11 runtime would become as simple as overriding a configuration property with an OpenJDK 11 version string.
That said, another challenge we faced was that our OpenJDK 11 migration work spanned multiple release cycles, and we had to make sure that none of our incremental changes to support OpenJDK 11 would break deployments to production, which were still based on OpenJDK 8, or have any negative impact on Customer Trust.
Platform Changes & Challenges
As we moved through OpenJDK versions one by one, we encountered many notable changes to the Java platform. The long path of our methodical progression meant that the changes brought up a lot of challenges, but we’ll dive into just a few of them here.
Class Paths and Modularity
One of the biggest changes that was introduced with the Java SE 9 platform was the Java Platform Module System (JPMS). JPMS divides the JDK into modules, where each module is a uniquely named, reusable group of related packages.
The good news is that Java 9 still supported the traditional class path, which works alongside the module path and is mapped to a special module called the unnamed module. As a result, all the JAR files that form the Salesforce app’s class path automatically participate in the module system, resulting in a mix of traditional class path and module path.
In fact, the entire class loader hierarchy of the Salesforce app is retained under Java 9 and beyond. It’s anchored by our web server and Servlet container, delegating to the OSGi class loader, which in turns delegates to the built-in class loaders of the Java runtime.
However, there was one significant change affecting class loading brought by Java 9 as part of the Jigsaw initiative. This was the removal of the Endorsed Standards Override Mechanism (used to support the loading of JAR files containing implementations of endorsed standards and standalone technologies) and the Extension Mechanism (used to support the loading of JAR files containing extensions or optional packages). Since the Salesforce app used to rely on both these mechanisms, all the affected JAR files had to be migrated to the Salesforce app’s module path, by using a combination of the —module-path, —upgrade-module-path, and —patch-module flags. None of these non-modular JAR files had to be converted to modules, though: they automatically became modular by being placed on the Salesforce app’s module path as dependencies. This feature is known as automatic modules and was created to ease the burden when transitioning existing applications to the new module system.
Another change affecting the Salesforce app was the removal of Java Enterprise Edition (“Java EE”) APIs, which the Salesforce app has depended on. Java 9 started separating these APIs into their own modules, which were annotated as deprecated for removal, indicating the intent to remove them in a future release. These modules were included in the runtime image, but not enabled by default. Instead, they had to be “activated” explicitly via the --add-modules flag.
Starting with Java 11, these modules were no longer included in the runtime (see JEP 320: Remove the Java EE and CORBA Modules). Instead, standalone versions of the Java EE and CORBA technologies were released as Maven artifacts and made available from third-party sites, such as Maven Central, from where we downloaded them and added them to the Salesforce app’s module path.
Backward Incompatibilities
While migrating the Java runtime of the Salesforce app to OpenJDK 11, we discovered a number of backward-incompatible changes. Most of these were “by design” and were covered in release notes, as is the case for those discussed below. (There was one true regression affecting the introspection of Boolean-typed bean properties; this was caused by a bug in the OpenJDK implementation itself, which we reported and was fixed.)
One example of a backward-incompatible change-by-design was very apparent, because it would cause the JVM to abort during startup with this error:Unrecognized VM option '<Option>'Error: Could not create the Java Virtual Machine.Error: A fatal exception has occurred. Program will exit.
The error was thrown because the Salesforce app had been using a number of garbage collection (GC) options that are no longer supported with Java 9 onward. Some of the affected GC options (e.g., UseParNewGC) had already been deprecated in JDK 8 (JEP 173) and were removed in Java 9 (JEP 214). Others, including PrintGCDateStamps and PrintGCTimeStamps, had become illegal as a result of how GC logging has been reimplemented in Java 9 (see JEP 271) to use the Unified JVM Logging framework introduced in JEP 158.
The challenge then became to continue to support these GC options for any Salesforce production instances that were still running OpenJDK 8, while suppressing them for any Salesforce production instances that had been upgraded to OpenJDK 11.
The scalable approach we adopted was to augment the ant target responsible for assembling the list of JVM arguments of the Salesforce app prior to launching it, such that it filtered out (using ant syntax) all the unsupported GC options in case the Java runtime was set to OpenJDK 11. This approach proved very flexible, allowing us to upgrade selected Salesforce production instances to OpenJDK 11 and roll back to OpenJDK 8 if needed. Once OpenJDK 11 had become the new default Java runtime and all production instances had been migrated successfully, the filter was removed from the ant target.
While the change affecting GC options broke the Salesforce app in an obvious way, other changes-by-design broke it in more subtle ways. One such change affected the context class loader of fork/join common pool threads, which no longer inherit the context class loader of the task-submitting thread, but are initialized with the system class loader. This change, whose effects manifested themselves across the Salesforce app in many different ways, is covered in the JDK 9 release notes, along with a workaround to restore the previous behavior.
Other changes-by-design affected Java language APIs from the core library, whose implementations had been changed to enforce their original API contracts more strictly. An example is java.util.HashMap.computeIfAbsent, whose implementation had been hardened to protect against reentrant use, where the mapping function passed to computeIfAbsent() modifies the very map on which computeIfAbsent() is called. Previously, this condition had gone unnoticed but could leave the map in an inconsistent state. However, starting with OpenJDK 9, it’s detected and flagged with a ConcurrentModificationException.
Third-Party Dependencies & Open Source Contributions
Along with the OpenJDK upgrade, we also needed to upgrade several of the Salesforce app’s underlying third-party dependencies. If you ignore the time that the team had to modify 2700+ Java test classes for the PowerMock upgrade, much of the work was fairly straightforward. That said, being an early adopter of OpenJDK 11, and given the complexity of the Salesforce app, it was expected that we’d have the chance to iron out some bugs along the way. This led to some great opportunities to contribute fixes back to the open source community.
OSGi
One opportunity presented itself with OSGi, where we ran into an issue with javax.annotation during startup. The javax.annotation package is one of the packages affected by JEP 320, which took effect with Java SE 11 and removed this and all other Java EE and CORBA packages from the JDK. As recommended by the same JEP, we had added all the JAR files that supply the missing packages, including javax.annotation-api.jar, to the Salesforce app’s module path, where they would be treated as automatic modules (see above). As per the JPMS specification, automatic modules are supposed to export all their packages – clearly something that was not happening in our case!
As it turned out, we had uncovered a bug (a violation of the JPMS specification) in the package resolution logic of the OSGi framework. We reported the issue to Eclipse Foundation, which manages the OSGi project, and submitted a fix for it. Our fix, which ensures that all packages of automatic modules are automatically added to the list of VM-supplied packages, was accepted, merged, and released to the OSGi community.
Procyon
As part of checking in Salesforce app code changes, developers submit their change list (CL) to pre-checkin, which, among other things, checks to make sure that their CL will not introduce any duplicate classes into the Salesforce app’s class path. Duplicate classes are classes with the same FQCN but different contents. Pre-checkin’s Duplicate-Class-Finder (DCF) relies on a Java decompiler from Procyon, which searches and decompiles classes with a given FQCN, searching through a list of JAR files on the class path.
DCF is integrated into the Salesforce app and inherits its Java runtime from the app. When executed against an OpenJDK 11 runtime, Procyon’s decompiler would fail. We reported the issue to Procyon and submitted a simplified, executable test case that would reproduce it. The Procyon developers reproduced the issue and fixed it, unblocking pre-checkin with the Salesforce app’s Java runtime set to OpenJDK 11.
Benefits from Day One
Multi-Release JAR Files
As mentioned earlier in this post, one way that the Salesforce application takes advantage of Java’s backward compatibility is by separating the Java version used to build the Salesforce app from the Java version used to launch it. This isolates the risk, so that even though the Salesforce app and its dependencies were still being built using OpenJDK 8, at runtime we could harness some of the new core Java APIs that had been added from Java 9 onwards (e.g., the new stack-walking API introduced by JEP 259), by taking advantage of multi-release JAR Files (JEP 238).
Multi-release JAR is a new feature introduced in Java 9: It extends the JAR file format by allowing multiple versions of the same Java class resource to co-exist within the same JAR file, where each version of the class may be implemented differently and is compiled against a different JDK version.
A multi-release-capable class loader automatically loads the appropriate classes (that is, those that match the JDK version of the Java runtime) from a multi-release JAR file. Both our Servlet container and OSGi class loaders support multi-release JAR files, and we expect an increasing number of third-party dependencies to be packaged using this format, as JDK 11 and above become more widely used.
Built-in Performance Improvements
One performance optimization that we got for free was provided by “compact strings” (JEP 254). This feature, first introduced in Java 9, provides a more memory-efficient internal representation of strings by moving from a char array to a more compact byte array (plus an encoding-flag field). This reduces the overall heap usage and related memory pressure of strings, which in turn has a positive impact on garbage collection and overall application performance.
Monitoring Improvements
Java Flight Recorder (JFR) is a profiling tool used to gather diagnostics and profiling data from a running Java application. JFR used to be available only as a commercial JDK add-on, but after having been open-sourced along with Java Mission Control, starting with OpenJDK 11. JFR can now be enabled on individual Salesforce application server instances to troubleshoot performance issues, which is a huge win.
Looking Forward
The upgrade to OpenJDK 11 was released to production without any major hiccups. Its rollout spanned roughly six months start to finish, following a well-vetted stagger strategy we usually follow to mitigate customer impact.
Soon after the rollout was complete, we shifted our focus to the Java version that’s used to build the Salesforce app. This was still set to OpenJDK 8, and we then upgraded it to OpenJDK 11 as well. By bumping the compile-time version of the app to OpenJDK 11, our developers have been empowered to use all the new Java language features that have been introduced since Java 9. These include the new stack-walking API, a new HTTP Client, improvements to try-with-resources statements, allowing private methods in interfaces, new methods in the Optional and Collectors classes, CompletableFuture improvements, the new var keyword for local-variable type inference, and many more.
We expect to see significant productivity and innovation gains from these new Java language features. And both the runtime and compile-time Java version upgrades to OpenJDK 11 have put us in a position to adopt future Java versions more quickly and seamlessly down the road.
Additional Resources
- JDK Release Notes
- Java Platform, Standard Edition Oracle JDK 9 Migration Guide
- JSR 376 (Java Platform Module System)
Thank you to Murthy Chintalapati, Rahul Shinde, and Larry Lopez for your additional contributions to this post.
