9 Migration and modularization strategies

When you’re done with this chapter, you not only understand the mechanisms behind migration challenges and modularization features—you also know how to best employ them in your efforts.

9.1  Migration strategies

With all the knowledge you’ve gathered in chapters 6 and 7, you’re prepared for every fight Java 9 might pick with you. Now it’s time to broaden our view and develop a larger strategy. How can you arrange the bits and pieces to make the migration as thorough and predictable as possible?

9.1.1  Preparatory updates

First of all, if you’re not on Java 8 yet, you should make that update first! Do yourself a favor and don’t jump two or more Java versions at once. Make an update, get all your tools and processes working, run it in production for a while, then tackle the next update. If you have any problems, you will really want to know which Java version or dependency update caused them.

Speaking of dependencies, another thing you can do without even looking at Java 9 is to start updating them as well as your tools. Besides the general benefit of being up to date, you might inadvertently update from a version that has problems with Java 9 to one that works fine with it. You won’t even notice you had a problem. If there is no Java 9 compatible version yet, being on the most recent release of your dependency or tool still makes it easier to update once a compatible version is published.

9.1.2  Estimating the effort

There are a few things you can do to get an impression of what lies ahead of you and we’ll look at those first. Next step is to evaluate and categorize the problems you found. I end this section with a small note on estimating concrete numbers.

Looking for trouble

These are the most obvious choices:

Carefully analyze the output, take note of new warnings and errors and try to link them to what previous chapters discussed. Look out for the removed command line options as described in section 6.5.3.

It is a good idea to apply some quick fixes like adding exports or Java EE modules. This allows you to see the tougher problems that may be hiding behind benign ones. In this phase, no fix is too quick or too dirty—anything that gets the build to throw a new error is a victory. If you get too many compile errors, you could compile with Java 8 and just run the tests on Java 9 (Maven: mvn surefire:test).

Then run JDeps on your project and your dependencies. Analyze dependencies on JDK-internal APIs (section 7.1.2) and take note of any Java EE modules (section 6.1). Also look for split packages between platform modules and application JARs (section 7.2.5).

Finally, search your code base for calls to AccessibleObject::setAccessible (section 7.1.4), casts to URLClassLoader (section 6.2), parsing of java.version system properties (section 6.5.1), or handcrafting resource URLs (section 6.3). Put everything you found on one big list—now it’s time to analyze it.

How bad is it?

The problems you’ve found should fall into the two categories "I’ve seen it in this book" and "What the &*!# is going on?". For the former, split it up further into "Has at least a temporary fix" and "Is a hard problem". Particularly hard problems are removed APIs and package splits between platform modules and JARs that do not implement an endorsed standard or a standalone technology.

It’s very important not to confuse prevalence with importance! You might get about a thousand errors because a Java EE module is missing, but fixing that is trivial. You’re in big trouble, on the other hand, if your core feature depends on that one cast of the application class loader to URLClassLoader. Or you might have a critical dependency on a removed API but because you’ve designed your system well, it just causes a few compile errors in one subproject.

A good approach is to ask yourself for each specific problem that you don’t know a solution for off the top of your head: "How bad would it be if I cut out the troublesome code and everything that depends on it?" How much would that hurt your project? In that vein, would it be possible to temporarily deactivate the troublesome code? Tests can be ignored, features toggled with flags. Get a sense for the how feasible it is to delay a fix and run the build and the application without it.

On estimating numbers

Chances are, somebody wants you to make an estimate that involves some hard numbers—maybe in hours, maybe in currency. That’s tough in general, but here it is particularly problematic.

A Java 9 migration makes you face the music of decisions long past. Your project might be tightly coupled to an outdated version of that web framework you wanted to update for years, or it might have accrued a lot of technical debt around that unmaintained library. And unfortunately both stop working on Java 9. What you have to do now is pay back some technical debt and everybody knows that the fees and interest can be hard to estimate. Finally, just like a good boss battle, the critical problem, the one that costs you the most to fix, could very well be hidden behind a few other troublemakers, so you can’t see it until you’re in too deep.

I’m not saying these scenarios are likely, just that they’re possible, so be careful with guessing how long it might take you to migrate to Java 9.

9.1.3  Continuously build on Java 9

Assuming you are continuously building your project, the next step is to set up a successful Java 9 build. There are many decisions to be made:

In the end, it is up to you to find answers that fit your project and continuous integration (CI) setup. Let me share some ideas that worked well in my migrations and then you can combine them any way you like.

Which branch to build?

You could be tempted to set up your own branch for the migration efforts and let your continuous integration server build that one with Java 9 while the others get built with Java 8 as before. But the migration can take time, so it is likely to result in a long-lived branch and I generally try not to have those for various reasons:

Which version to build?

Should the Java 9 build create separate version of your artifacts? Something like ...-JAVA-9-SNAPSHOT?

If you’ve decided to create a separate Java 9 branch, you are likely forced to create a separate version, too. Otherwise it is really easy to mix snapshot artifacts from different branches, which is bound to break the build the more the branches deviate. If you’ve decided to build from the main development branch, creating a separate version might not be that easy, but I never really tried because I found no good reason to do it.

Pretty much regardless of how you handle versions, when trying to get something to work on Java 9 it is likely that you’ll occasionally build the same subproject with the same version with Java 8. One thing I do again and again even though I resolve not to, is installing the artifacts I build with Java 9 in my local repository. You know, the knee-jerk mvn clean install? That’s really not a good idea because then you can’t use those artifacts in a Java 8 build because it does of course not support Java 9 bytecode.

So when building locally with Java 9, try to remember not to install the artifacts! I use mvn clean verify for that.

What to build with Java 9?

The end goal is to have the build tool run on Java 9 and build all projects across all phases/tasks. Depending on how many items you have on that list you created earlier, it is possible that you only need to change a few things to get you there. In that case, go for it—no reason to complicate things. On the other hand, if your list is a little more daunting, there are several ways to slice the Java 9 build:

Generally speaking, I’d prefer "by goal/task" approach for large, monolithic projects and the "by subproject" approach if the project is split into parts that are small enough to be tackled in one go.

If you go by subproject but one of them can’t be built on Java 9 for whatever reason, you face the problem that you can’t easily build the subprojects depending on it. I’ve been in that situation once and we decided to set up the Java 9 build in two runs:

Your build tool on Java 9

Until your project is fully migrated to Java 9, you might need to often switch between building it with 8 and 9. Have a look at how you can configure that for your build tool of choice without having to set the default Java version for your entire machine (Maven: ~/.mavenrc or the toolchain). Then consider automating the switch. I ended up writing a little script that would set $JAVA_HOME to either JDK 8 or JDK 9, so I could quickly pick the one I need.

Then, and this is a little meta, the build tool itself might not work properly on Java 9. Maybe the tool needs a Java EE module, maybe a plugin uses removed APIs. (I had that problem with a JAXB plugin for Maven, which needs java.xml.bind and relies on its internals.)

In that case you could consider running the build on Java 8 and only compile or test against Java 9, but that won’t work if the build does something with the created bytecode (which would be for Java 9) in its own process (which is Java 8). (I ran into that problem with the Java Remote Method Invocation Compiler rmic and it forced us to run the entire build on Java 9 even though we would’ve preferred not to.)

If you decide to run the build on Java 9 even though it doesn’t play nice with it, you’ll have to configure the build process itself with some of the new command line options. Doing this so that it’s easy on your fellow team members (nobody wants to add options manually) while keeping it working on Java 8 (which doesn’t know the new options) can be non-trivial (Maven: jvm.config). I found no way to make it work on both versions without requiring a file rename, so I ended up including that in my "Switch Java version" script.

How to configure the Java 9 build

So how do you keep a Java 8 and a Java 9 build running when you have to add version-specific configuration options to compiler, test runtime or other build tasks? Your build tool should help you out here. It likely has a feature that allows you to adapt the overall configuration to various circumstances (Maven: profiles). You should familiarize yourself with it as you might end up using it a lot.

While this allows you to use the same options for both Java 8 and Java 9, I recommend to not make it your first choice because it disables checks that help you find mistakes. Instead I prefer, separating the options by version if at all possible.

Fix first, solve later

Typically, most items on the list of Java 9 problems are straightforward to fix with a command line flag. Exporting an internal API, for example, is very easy. That does not solve the underlying problem, though. Sometimes the solution is easy as well, like replacing the internal sun.reflect.generics.reflectiveObjects.NotImplementedException with an UnsupportedOperationException (no kidding, had to do that more than once), but often it is not.

So should you aim for quick and dirty or for proper solutions that take a little longer? In the phase of trying to get a full build working, I recommend making the quick fix:

It is common to find new problems that were originally hidden behind an earlier build error. If that happens, make sure to add them to your list of Java 9 problems. Likewise scratch off those that you solved.

Keeping it green

Once you’ve set up a successful build, you should have a complete picture of all the Java 9 challenges you face. It is now time to properly solve them one by one.

Some of the issues may be tough or time-intensive to solve and you might even determine that they can’t be addressed until some later point in time—maybe once an important release was made or the budget has a little wiggle room. Don’t worry if it takes some time. With a build that every developer on the team can break and fix, you can never take a step in the wrong direction, so even if you have a lot of work ahead of you, you will eventually reach it in little steps.

9.1.4  Thoughts on command line options

With Java 9, you might end up applying more command line options than ever before—it sure has been like that for me. While doing so I had a few insights that I want to share with you:

Let’s go through them one by one.

Applying options with argument files

What’s new in Java 9 is that the JVM also recognizes argument files, so they can be shared between compilation and execution.

Relying on weak encapsulation

As section 7.1 explains in detail, the Java 9 runtime allows illegal access by default with nothing more than a warning. That’s great to run unprepared applications on Java 9, but I advise against relying on that during a proper build because it allows new illegal accesses to slip by unnoticed. Instead, I collect all the --add-exports and --add-opens I need and then activate strong encapsulation at run time with --illegal-access=deny.

The pitfalls of command line options

Using command line options has a few pitfalls:

All of these pitfalls make one thing very clear: Command line options are a fix, not a proper solution, and they have their own long-term costs. This is no accident—they were designed to make the undesired possible. Not easy, though, or there would be no incentive to solve the underlying problem.

So do your best to only rely on public and supported APIs, not to split packages, and to generally avoid the trouble this chapter describes. And, very importantly, reward libraries and frameworks that do the same! But the road to hell is paved with good intentions, so if everything else fails, use every command line flag at your disposal.

9.2  Modularization Strategies

In chapter 8 you’ve learned all about the unnamed module, automatic modules, and about mixing plain JARs, modular JARs, class path, and module path. But how do you put that into practice? What are the best strategies to incrementally modularize a code base? To answer these questions, imagine the entire Java ecosystem as a huge layered graph of artifacts (see figure 9.1).

At the bottom we have the JDK, which used to be a single node but thanks to the module system is now made up of about a hundred nodes with java.base as the foundation. Immediately on top sit those libraries that have no run-time dependencies outside the JDK (like SLF4J, Vavr, AssertJ), followed by those with just a few (for example Guava, JOOQ, JUnit 5). Somewhere in the middle we can see the frameworks with their deeper stacks (for example Spring, Hibernate) and on top of it all sit the applications.

Except for the JDK, all of these artifacts were plain JARs when Java 9 came out, and it will take a few years before most of them contain a module descriptor. But how will that come about? How can the ecosystem undergo such a massive change without breaking apart? The modularization strategies enabled by the unnamed module (section 8.2) and automatic modules (section 8.3) are the answer. They make it possible for the Java community to modularize the ecosystem almost independently of one another.

Looking at the ecosystem as a whole, your project’s place in it determines which strategy you must use for it. But as section 9.2.4 explains, these approaches also work within individual projects, in which case you can choose any of the three. Before we come to that, though, learning the strategies is easier when we assume that you modularize all your artifacts at once.

As I have often mentioned, a module has three basic properties: a name, a clearly defined API, and explicit dependencies. When creating a module, you obviously have to pick the name yourself. The exact exports can be quibbled over, but are mostly predetermined by which classes need to be accessible. The real challenge and where the rest of the ecosystem comes into play, are the dependencies. This section focuses on that aspect.

With all of that said, it is time to see how the three modularization strategies work.

9.2.1  Bottom-up modularization

This is the easiest case for turning your project’s JARs into modules: The assumption is that your code only depends on explicit modules (directly and indirectly). It doesn’t matter whether those are platform or application modules; you can go straight ahead:

You have now fully modularized your project—congratulations! If you’re maintaining a library or framework and your users place your JARs on the module path, they will become explicit modules and your users can start benefiting from the module system.

Almost as important but less obvious, thanks to the fact that all JARs on the class path end up in the unnamed module (see section 8.2), no one is forced to use it as a module. If someone sticks with the class path a while longer, your project will work just as if the module descriptor weren’t there.

9.2.2  Top-down modularization

If you’re an application developer and want to modularize any time soon, it is unlikely that all your dependencies already ship modular JARs. If they do, you’re lucky and can just take the bottom-up approach from section 9.2.1; otherwise you have to make use of automatic modules and start mixing module path and class path as follows:

Setting up the build tool or launch script, it might be easiest to place all remaining JARs on the module path as well and give it a go. While I don’t think that’s generally the best approach, it might work for you. In that case, go for it.

If you have problems with package splits or access to JDK-internal APIs, you might try placing those JARs on the class path. Since only automatic modules need them and they can read the unnamed module, that works fine.

In the future, once a formerly automatic module gets modularized, that setup may fail because it’s now a modular JAR on the module path and hence can’t access code from the class path. I consider that to be a good thing as it gives you a better insight into which of your dependencies are modules and which aren’t—it’s also a good opportunity to check out its module descriptor and learn about the project. To fix the problem, move that module’s dependencies onto the module path.

Note that you don’t have to worry about where automatic module names come from (see section 8.3.4). True, if they are based on the file name, you have to change some requires directives once they get an explicit module name, but since you control all module declarations, that’s not a big deal.

What about making sure modules that your non-modular dependencies require make it into the graph? An application could either require them in a module declaration, or use --add-modules to add them manually at compile and launch time. The latter is of course only an option if you have control over the launch command.

Your build tool may be able to make these decisions for you, but you still need to be aware of the options and how to configure them, so you can fix problems should they arise.

9.2.3  Inside-out modularization

Most libraries and particularly frameworks are neither at the bottom nor the top of the stack—what are they to do? They modularize inside-out. This has a little bit of bottom-up (section 9.2.1) in it because that you release modular JARs does not force your users to use them as modules. Other than that it works like top-down (section 9.2.2) with one important difference: You are planning to publish the modular JARs you built.

As I discussed at length in section 8.3.4, you should only ever publish modules with dependencies on automatic modules if those plain JARs define the Automatic-Module-Name entry in their manifest. Otherwise the risk of causing problems down the road when the module name changes is too high.

This might very well mean that you can not yet modularize your project. If you’re in this situation, please resist the temptation to do it anyways or you are likely to cause your users difficult problems.

I even want to take this one step further: Examine your direct and indirect dependencies and make sure none depends on an automatic module whose name is derived from the JAR file name. You’re looking for any dependency that is not a modular JAR and does not define the Automatic-Module-Name entry. I would not publish an artifact with a module descriptor that pulls in any such JAR—whether it’s my dependency or somebody else’s.

There’s also a subtle difference when it comes to platform modules that your non-modular dependencies need but you don’t. Where applications can easily use command line options, libraries or frameworks can’t. They can only document for their users that they need to add them, but that is bound to be overlooked by some. I hence advise to explicitly require all platform modules that your non-modular dependencies need.

9.2.4  Applying these strategies inside your project

Which of the three strategies you have to use is determined by your project’s place in that gigantic, ecosystem-wide dependency graph. But if your project is rather large, you might not be able to modularize it all at once and wonder how you can instead do that incrementally. Good news, you can apply very similar strategies in the small.

It’s often easiest to apply a bottom-up strategy to a project, first modularizing those subprojects that only depend on code outside your code base. This works particularly well if your dependencies are already modularized, but is not limited to that scenario. If they aren’t, you just need to apply some top-down logic to the lowest rung of your subprojects, making them use automatic modules to depend on plain JARs, and then build up from there.

Applied to a single project, the top down approach works exactly the same as applied to the ecosystem as a whole. Modularize an artifact at the top of the graph, place it on the module path, and turn its dependencies into automatic modules. Then slowly progress down the dependency tree.

You might even do it inside-out. Chapter 10 introduces services, a great way to use the module system to decouple dependencies—internal to your project but also across different projects. They are a good reason to start modularizing somewhere in the middle of your project’s dependency graph and move upwards or downwards from there.

While all of that is possible, you shouldn’t needlessly complicate matters. Once you have settled on an approach, you should to try to quickly and methodically modularize your project. Drawing this out and creating some modules here and there just means that you will have a hard time understanding your project’s dependency graph and that is precisely the antithesis of one of the module system’s important goals: reliable configuration.

9.3  Making JARs modular

All you need to do to turn a plain JAR into a modular JAR is add a module declaration to the source. Easy, right? Yes (wait for it), but (there we go!) there’s more to say about that step than immediately meets the eye:

This section tackles these topics one by one to make sure you’re getting the most bang for your buck.

9.3.1  Open modules as an intermediate step

A concept that can be very useful during incremental modularization of an application is that of open modules. Section 12.2.4 goes into details, but the gist is that an open module opts out of strong encapsulation: All its packages are exported and opened to reflection, which means all its public types are accessible during compilation and all other types and members are accessible via reflection. It is created by beginning its module declaration with open module.

Open modules come in handy when you aren’t happy with a JAR’s package layout. Maybe there are just lots of packages or maybe many packages contain public types that you’d rather not have accessible—in both cases a refactoring might take too much time to do it in the moment. Or maybe the module is used heavily under reflection and you don’t want to go through determining all the packages you need to open.

In such cases, opening the entire module is a good way to push those problems into the future. Caveats about technical debt apply—these modules opt out of strong encapsulation, which denies them the benefits that come with it.

9.3.2  Generating module declarations with JDeps

If you have a big project, you might have to create dozens or even hundreds of module declarations, which is a daunting task. Fortunately, you can use JDeps for most of it because large parts of that work are quite mechanical:

Beyond those basic properties, there may be some fine-tuning involved to make sure all dependencies are recorded and to configure the use of services (see chapter 10) or more detailed dependencies and APIs (see chapter 11), but everything up to that point can be generated by JDeps.

When JDeps generated the module-info.java files, it is up to to you to inspect and adapt them and move them into the correct source folders, so your next build can compile and package them.

Inspecting generated declarations

If you develop an application that has too many JARs to manually inspect all declarations and are fine with some hiccups, there is a more lenient option: You might get away with just trusting your tests, your continuous integration pipeline, and your fellow developers and testers with finding the little problems. In that case, make sure you have some time before the next release, so you can be confident you’ve fixed everything.

If you’re publishing your artifacts, though, you absolutely have to check declarations with great care! These are the most public parts of your API and changing them is often incompatible—work hard to prevent that from happening without good reason.

Missing dependencies

For JDeps to properly generate requires directives for a set of JARs, all of these JARs as well as all their direct dependencies must be present in the scanned directory. If dependencies are missing, JDeps will report them as follows:

To avoid erroneous module declarations, none are generated for the modules where not all dependencies are present.

Analyzing exports

Export directives are solely based on the analysis of which types are needed by other JARs. This almost guarantees that library JARs will see way too few exports. Keep this in mind when checking JDeps' output.

9.3.3  Hacking third-party JARs

It can sometimes be necessary to update third-party JARs. Maybe you need it to be an explicit module or at least an automatic module with a specific name. Maybe it already is a module but the module descriptor is faulty or causes problems with dependencies you’d prefer not to draw in. In such cases, the time has come to pull out the sharp tools and get to work. But be careful not to cut yourself.

With the caveats out of the way, let’s see how to manipulate third-party JARs if they don’t work well with your project. I show you how to

Adding and editing an automatic module name

A good reason to add an automatic module name to a JAR, besides the scenario where the JPMS can otherwise not derive a name, is if the project already defined one in newer versions but you can’t yet update to it for whatever reason. In that case, editing the JAR allows you to use a future-proof name in your module declarations.

The same approach can be used to edit an existing automatic module name. The jar tool will complain about Duplicate name in Manifest, but the new value nevertheless replaces the old one.

Adding and editing module descriptors

If turning a third-party JAR into a properly named automatic module is not enough for you or you have trouble with an explicit module, you can use jar --update to add or override a module descriptor. An important use case for the latter is to resolve the modular diamond of death described in section 8.3.4.

This adds the file module-info.class to ${jar}. Note that --update does not perform any checks. This makes it easy to, accidentally or on purpose, create JARs whose module descriptor and class files do not agree, for example on required dependencies. Use with care!

The more complicated task is to actually come up with a module descriptor. For the compiler to create one, you not only need a module declaration, but also all dependencies (their presence is checked as part of reliable configuration) and some resemblance of the JAR’s code (as sources or bytecode; otherwise the compiler complains of non-existent packages).

Your build tool should be able to help you with the dependencies (Maven: copy-dependencies). For the code, it is important that the compiler sees the entire module, not just the declaration. This can best be achieved by compiling the declaration while the module’s bytecode is added from its JAR with --patch-module. Section 7.2.4 introduces that option and listing 9.2 shows how to use it here.

Adding classes to modules

If you’re looking for a more lasting solution to your problem, you can once again use jar --update, in this case to add class files.

9.3.4  Publishing modular JARs for Java 8 and older

Whether you maintain an application, a library, or a framework, it is entirely possible that you target more than one Java version. Does that mean you have to skip the module system? Fortunately, not! There are two ways to deliver modular artifacts that work just fine on Java versions older than 9.

Whichever you chose, you first need to build your project for the target version. You can either use the compiler from the corresponding JDK or a newer one by setting -source and -target. If you pick the Java 9 compiler, check out the new flag --release in section 4.4. Finish this step by creating a JAR as you normally would. Note that this JAR runs perfectly fine on your desired Java release, but does not yet contain a module descriptor.

The next step is to compile the module declarations with Java 9. The best and most reliable way is to simply build the entire project with the Java 9 compiler.

Now you have two options for how to get the module descriptor into your JAR.

Just use jar --update

You can simply use jar --update as described in section 9.3.3 to add the module descriptor to the JAR. That works because JVMs before version 9 actually ignore the module descriptor. They only see other class files and because you build them for the correct version, everything just works.

While that is true for the JVM, it can not necessarily be said for all tools that process bytecode. Some trip over module-info.class and thus become useless for modular JARs. If you want to prevent that you have to create a multi-release JAR.

Create a multi-release JAR

From Java 9 on, jar allows the creation of so-called multi-release JARs (MR-JARs), which contain bytecode for different Java versions. Appendix E gives a thorough introduction to this new feature and to make the most out of this section you should give it a read. Here, I’m focusing on how to use MR-JARs, so that the module descriptor does not end up in the JAR’s root.

This feature goes far beyond adding module descriptors, so if you didn’t already, I really recommend you read appendix E.

9.4  Summary