The upcoming Jack & Jill compilers

The upcoming Jack & Jill compilers

 
Update: On the 14th of March 2017, Google announced that it is deprecating the Jack toolchain. You can continue to rely on DexGuard to shrink, optimize and obfuscate your Android applications.
 

First sightings

In the shadow of the release of Android 5.0 "Lollipop", the Android tools are seeing updates as well. This is good news for Android developers, who have to manage all these shiny new features. Useful improvements like resource shrinking and multidex have trickled out over the past few weeks. There are some signs of more goodness to come:

  • The Android SDK 21.1 contains two new build tools: jack.jar and jill.jar.
  • The Gradle plugin contains a new, undocumented flag: useJack.
  • The Android source repository contains a new toolchain named jack.

It seems like these additions haven't been announced or even noticed thus far, but they beg to be explored. I had a look, as an external observer. In short: Jack and Jill form a new compiler toolchain to transform java code into dalvik bytecode. They introduce a new intermediate representation called Jayce.

History

The Android platform has always had close ties with the Java platform. This approach leverages the existing expertise of developers and the wealth of supporting libraries, frameworks, IDEs, and other tools. This probably won't change anytime soon, but it does entail a dependency on the Java platform. For instance, java has evolved to Java 8, with extensive changes to support closures. The Android toolchain still doesn't support Java 8.

Based on its logs, the new compiler project was started many months ago by the Google development team from FlexyCore, a company that Google acquired in 2013. At the time, they offered a prototype for an ahead-of-time compiler for Android, based on the gcc toolchain. This concept made it into Android in the form of the Android RunTime, ART, now based on the LLVM toolchain. ART compiles code on the user's device, from Dalvik bytecode to native code, offering better and smoother performance.

The development team has now quietly been working on the compilation of java to dalvik bytecode. Developers go through this process many times a day, so they surely appreciate its importance.

The build process

The current Android build process transforms java code into dalvik bytecode:

A regular java compiler converts the application's source code into java bytecode. This is typically a compiler from the OpenJDK project, from Oracle, or from Eclipse.

The dex compiler takes the combined java bytecode and converts it to Android-specific dalvik bytecode. The build process may cache dexed versions of libraries, to speed up the conversion. The dalvik bytecode goes into the application.

If ProGuard is enabled, it combines the compiled bytecode and the library bytecode and optimizes it, before passing it to the dex compiler:

ProGuard is of course our own open-source optimizer and obfuscator for java bytecode. It produces code that is more compact (in a shrinking step), more efficient (in an optimization step), and more difficult to reverse engineer (in an obfuscation step). ProGuard reads a configuration file to steer its processing. It writes out a mapping file that allows to de-obfuscate any stack traces that the processed application might produce.

Conceptually, the jack compiler consolidates the functionality of javac, ProGuard, and dex in a single conversion step:

In actual builds, the jill compiler adds a new preprocessing and caching step:

Jill shields jack from java bytecode by converting it to intermediate jayce bytecode. Jack converts the java source code and the intermediate bytecode to dalvik bytecode. It includes ProGuard's functionality, reading ProGuard configurations, shrinking and obfuscating the code, and writing out a ProGuard mapping file.

In practice

You can already try the new compilers by enabling them in a Gradle build with the Android plugin 0.14.+:

defaultConfig {
  useJack true
}

You can see which commands are run with Gradle's "-i" option, and you can similarly run the compilers from the command-line. The new compilers are only activated when building application projects. Library projects are still built with javac and ProGuard.

The compilers already seem quite functional. Jack doesn't support some of the advanced features of ProGuard (yet), for instance to remove logging code:

    Proguard flag is not supported: -assumenosideeffects ...

In general, jack doesn't seem to perform any whole-program optimizations, which are a great performance-improving feature of ProGuard. I also encountered a compiler error in one of my quick experiments:

    Internal compiler error (version 1.0.001 (UJX30A 41a1...3ce9)).
    Warning: This may have produced partial or corrupted output.

The compilers will no doubt enter a long public testing phase when they are officially announced.

Internals

The jack compiler is built around the Eclipse java compiler. The back-end of course generates dalvik bytecode and jayce bytecode instead of java bytecode. It is an optimizing compiler, with optional shrinking and obfuscation. For this purpose, it reads ProGuard configuration files and it writes ProGuard mapping files.

Both compilers use the intermediate jayce format. Each java source file or class file can be compiled to a jayce file, which is then packaged and stored in a jar file. Intermediate representations are very common in optimizing compilers to speed up builds. Conceptually, the format could also be used to distribute pre-compiled libraries.

Java 8?

So does the compiler support Java 8? Unfortunately, not yet. It's even aware of its own limitations, for instance when you try to compile source code with closures:

    Lambda expressions are allowed only at source level 1.8 or above

The compiler supports source levels up to 1.7, but not 1.8...

In the Java world, the implementations of Java 8 include pervasive changes to the language, the bytecode format, the runtime libraries, and the virtual machine. These changes were by choice; support for closures could have been implemented at a compiler level. In the Android world, changing the dalvik bytecode and runtime presents more serious compatibility problems. Mobile devices are updated at a much slower pace, if at all. Having complete control over the compiler offers a possible way out.

Implications for external libraries and tools

Since the toolchain still supports both java source code and java bytecode, existing libraries can be used without problems. Even tools that generate or process bytecode during the build process, such as Dagger, should be fine, with the proper integration.

We will of course continue to make sure that DexGuard can be integrated nicely. Its advanced optimization and hardening features go far beyond the basic features, so it's definitely worth it!

Conclusion

The Jack and Jill compilers will streamline the Android build process and improve build performance. The toolchain strives for backward compatibility and opens up possibilities for future extension. We're looking forward to see it mature and grow.

(all opinions, predictions, and inaccuracies are my own)

Eric Lafortune