Improvements in Gradle for Android

Over the past months, performance improvements in Gradle core have highly benefitted Android developers, and more radical improvements are on the way. On the forefront of these are the modeling of variants, and a revolutionary new configuration model, which will further enhance the experience of developing Android applications with Android Studio beyond simply better performance. In this presentation from GOTO Conference CPH 2015, Etienne Studer takes a close look at these Gradle improvements and how they manifest in the context of Android and Android Studio.

---

Implicit Complexity of the Domain Multi-Language Support Resource/Code Generation Platform Diversity (02:50)

When you do Android development you are part of a team, possibly using multiple languages (Java, Android, etc). During the Android build, you’re looking at a lot:

• implicit complexity of the domain
• multi-language support
• resource/code generation
• platform diversity implicitly

You need a coordinated release, which is one of the strengths of Gradle: one big project, with sub-projects and it bonds it up. LinkedIn is a Gradle user: 2,000 developers, 2,000 components, 300,000 builds a week, and 1,000 release builds a day. If you have a very disconnected build, with artifacts, and you put it together, it is not going to scale.

The Android build system (06:35)

The Android build system uses: the Android Studio IDE (based on the IntelliJ platform), Android Gradle Plugin (based on the Gradle platform), and Android Tooling.

Have a single truth of build logic: Put all build logic into the build.
 Derive all information from the build model. Android Gradle Plugin and Gradle platform = treat them as one. The build is the only place that defines what your build does. You do configuration in your Eclipse. You define dependencies and in your build. The IDE will be configured based on your model.

In a unified build,
 Gradle is the single source of build logic. The build has to communicate with Gradle (to understand what it has to present/do) through the Tooling API. The tooling API lives in the Android Studio (IDE class path). Through interprocess communication, it talks to the Gradle build. The Gradle build (Gradle daemon) is a separate process. The interstudio will tell the tooling API “compile”, and the tooling API will talk to the daemon through interprocess communication. Daemon will compile that build with that task. With the Gradle daemon, you come in through the launcher. If you are in a continuous integration server, you also involve Gradle directly. All that works because Gradle is the single source of truth when it comes to build logic.

Tooling API (10:07)

There is also a mechanism to provide custom models. Android asks Gradle, please give me that model for Android. The model comes back and it contains all the information about the variants, and the flavors. IDE can consume that and configure the IDE accordingly. You can ask for that model through the tooling API. There is the client VM (the IDE). That is the little chore file (800 k’s or so). It asks Gradle, through the tooling API, give me that build model or invoke some tasks. Build models you ask for, to configure the IDE. Invoking build tasks you execute something in the build.

The advantages of that approach are:

• Backward compatibility.
• Runtime Isolation. If Android Studio had a memory leak, Gradle would not be affected because it is a different process.
• High level services (you can cancel a build). That goes to the tooling API; it has a continuous mode. You can also execute tests. If you have a specific API when you run a test, that the test is executed by Gradle (only Gradle knows what is the class path, etcetera, to run that test; IDE can only do an approximation).
• Build event model. As you run a build through the tooling API you get events, log output, progress. All this is consumable through the tooling API. When you are on the build, you see a tree that shows you all these events happening.

Goal, Performance, Approach: Minimize the Build Time While
 Using as Little Memory as Needed (13:35)

We all want performance to be lightning fast. Our goal is to minimize the build time, using as little memory as needed. We want quicker feedback about build figures. We want to do more builds/releases per day. And we want to do more rounds on continuous integration. When little changes in the build, little work should be done. Even if you have a big Android project, Gradle task, or Gradle help, it will take a while, and that time is proportional to the size of the build. And it should not be, because when you say Gradle help, you are not even executing anything in your build.

Two approaches: 1) You make little improvements, which may have a significant impact; 2) revolutionary changes to make the next step in terms of performance - you might only do local optimization, but not the global optimization.

Building with Gradle - today (16:21)

You have a two phase build: 1) configuration phase (build logic is executed; build model is built), 2) execution phase (build model is acted upon - the tasks are executed).

Execution phase (10:19)

class ConversionTask extends DefaultTask {
  @InputFiles

  def sourceFiles



  @OutputDirectory

  def targetDirectory



  @TaskAction

  def doSomeWork() {

      // consume the sourceFiles and write the result

      // to a file in the targetDirectory

  }
}
task foo(type: MyTask) {

  sourceFiles = files('input.txt');

  targetDirectory = file('build/result')

}

In terms of optimization during the execution phase, there are two features:

• Incremental build. Tasks have inputs and outputs in Gradle, which you can annotate or declare. With that information, Gradle can do incremental builds. If the inputs have not changed, and the outputs have not changed, there is no need to run that task again.

• Continuous build: Keep the session running between build runs gradlew test -t (-t stands for continuous mode). You can see it starts to build. If you now change something that is an input of the task invoked: it would automatically trigger the build. If you are using that continuous mode, you are making changes. If you are compiling, Gradle will spawn a compiler daemon. With continuous builds this daemon will stay alive. As long as the build session of a continuous build rounds, the compiler daemon will be there (much faster to use).

Configuration phase (24:44)

Imperative build logic (input), even when providing a declarative DSL. As soon as it gets imperative (in your code or in the plugin’s code), Gradle does not know what you are doing (it cannot optimize things). They use lazy evaluation/collection tricks: until you are done declaring your build flavors, you cannot redo anything. You end up with a declarative build model, but: 1) Gradle does not know how that model came together, 2) it is building the whole model.

For Android, at configuration time the whole model is built: that takes a long time. It also has to do the full dependency resolution and the processing at evaluation time, or configuration time. It does not know all the flavors, until very late.

To improve the configuration phase: 1) as new versions of Groovy to do the parsing of the DSL, it get’s faster; 2) Configure on demand. Only configure projects reached by those tasks (only working under certain conditions; if you violate these conditions, the output is not guaranteed).

Android Build - Pre-Dexing (29:22)

If you do a clean install, time (in this example) is used for the pre-dexing. All your libraries that you are consuming through dependencies, they have to be dexed. That takes a significant amount of time in this build. Once you run the build again, they are cached.

Pre-dexing optimizations (30:30)

• Clean, safe cache (but it is a generic cache). If you have dependencies and they need to be transformed before they are consumed, that should become part of Gradle, and then it can be used for the Android plugins.

• Parallelization. This pre-dexing could happen in parallel, but it is currently not happening in parallel. Something that Gradle could support, that this pre-dexing can happen in parallel.

• Distributed cache (already on the road map). The pre-dexing will first check the cache if it is already there. And if it is there, it will already be consumed. This reduces the time for pre-dexing on your local machine to zero. Before going to the distributed cache, it will check locally if something is already there. If it is not there, go to the distributed cache. If it is there, use it from there, download it, store it.

Android Build - Dexing (32:02)

Dexing takes a significant amount of time. You change one file, it has to dex everything again. All the other builds are affected as well. The bottleneck that we are focusing on now is the configuration phase.

Dexing optimizations (32:30)

• Faster Dexing
• Incremental Dexing

Building with Gradle - Experimental (33:00)

Everything is built into a build model that is then consumed (calling Gradle help). The new configuration model is revolutionary: something that we build up in parallel.

New Configuration Model (33:37)

• Apply the concepts already available in the Execution phase to the Configuration phase.
• Describe what the model should look like and Gradle will provide the implementation.

Modeling (34:29)

By managing the model, Gradle can be smart about it and what it does about it. It will be cleaner modeling, because now if you look at the tasks, you would see that there is execution and configuration logic. The task (or it will not be called a task anymore), you have your manage types in all they contain is data. There is nothing on there about execution. You can collaborate better. It will be more deterministic, and comprehensible.

By having a model (as for the execution), we have a task and a task graph. If there is a model for your build, and you know where things came from, you can do really interesting reports, and you know why things are happening (because you know who contributed what).

Managed types (36:02)

@Managed
interface Picture {

  String getName()
  void setName(String name)

  List<String> getTags()
}

Picture (managed type) is in an interface. Gradle will provide implementation (it knows if somebody sets the name). If it knows who sets the name, it is something it can report on. Another advantage of these managed types is that they can be persisted. As it goes and builds that model, if there are certain things it does not need, it can write them, gloat them again later. Or it can store them in the daemon, and when you run the next build, it can reuse them, it can cache things. Thus, they are externalizable.

Plugin (36:44)

Gradle plugin is very declarative. There is a model for creating a picture, and there is a way to mutate the picture. The picture is passed in, and you can set this argument. When you set them, because the implementation of picture is controlled by Gradle, again, it can track, and knows who did these changes.

class PicturesPlugin extends RuleSource {



  @Model

  void createPicture(Picture picture) {}



  @Mutate

  void configurePicture(Picture picture) {

    picture.name = 'mypic.jpg'

    picture.tags.addAll(['nature', 'night'])

  }


 }

DSL (37:30)

The last part is providing a DSL. Based on the example: here is my model, and a picture. I want to create a new picture instance, or picture node, and I want to set the name, and set the tags.

model {

  picture(Picture) {

    name = 'mypic.jpg'

    tags.addAll(['night', 'moon'])

  }

}

This looks very similar as in the Android plugin, but the implementation is controlled by Gradle.

Report (38:17)

In this report, createPicture: it can see what the type is, who created it, and the rules that were applied on that picture (the configure was called after the creation).

Android (Experimental) (38:51)

With all this rich information, it chips with two plugins. An old version and a new version. And the underlying logic of what to do during the build is the same, but the way it is configured is different. One is using the new model, the other one is using the old model. Every time a new Gradle release comes out, they make use of the latest features, and adjust their experimental plugins. At some point, this plugin will become the default plugin. Also, you have to use the dot notation; and you have to say create if it is a new one, you can use just the name if it is an existing one (That will go away).

model {

  android {

    compileSdkVersion = 21

  }

  android.buildTypes {

    debug {

    }

    create('qa')

  }

}

New Configuration Model (40:05)

The configuration will become parallelizable (like the task execution can be parallelizable). Manage times can be externalized. The model can be reused across invocations (because Gradle knows the inputs and the outputs for the configuration phase). That is really revolutionary change. We do not want to break what is there.

Roadmap (41:17)

Roadmap (for the next six to twelve months; no guarantees or commitments) will improve performance, and generify dependency management (not only between libraries, but higher level constructs).

Performance (41:57)

Apply. Fixing hotspots; caching and reusing (key aspect), working in parallel (run on the same machine, but also across machines). Distributed build will come as well. You have a build, you use different things, you need to configure things. That can happen in parallel. It can even happen across machines. And doing work in the background. The continuous mode that I showed is one example. You do your work, and while you do your work Gradle can already see if something has changed, and if it has changed it can already prepare it such that when you do want to run it, it is already there.

To Left-hand techniques (see side), can be applied to many things on the hand side: build configuration; dependency resolution (in parallel there too; caching); and task execution. If tasks were fully managed by Gradle (which will be the case with new configuration model): you could run these tasks in parallel.

If you use this new approach, based on these managed types, this is not an Android build, but still, even with a Gradle builds you can have variance. The key is: the less you do, or the less you want to do, the less Gradle has to do.

Dependency Management (45:02)

You can have your dependencies on the variance. When you consume external libraries, you want to consume the variant. Let’s say you create a library in variant ABC. Now we have another project that consumes this library and it also has variance ABC. You want to consume the variant, of that external dependency. Same for native libraries. You also want to consume the one. I you are on the debug build type, you want to use everything n debug. By storing this, this extra meta data with these variance. And then when it comes to resolution, it will try to pick the variant. If it finds one, it uses it. If it does not, it will blow up. Same will happen in the Java space. If I am building a library for JDK 8, and you are depending on another library, if that library says, I am only available or I am only, I am a variant only for JDK 9 and later, it cannot use it. This will be full variant of variant dependency management.

Gradle.com (46:23)

We are also working on Gradle.com. As you run your build, you can upload like a build receipt to Gradle.com and you share it. It will help you detect errors and propose solutions. You can attach a link to the build receipt and then somebody can go on it and say I see a problem here.