Revert "Revert "Checksum Marker in Dex File.""

This reverts commit e0606685c4daf8a5948cded9ffebf8d1ccca12c7.

PatchSet 1 is vanilla revert,
PatchSet 2 contains the performance fix.

ToT
FrameworkIncrementalNoDesugarDexAll(RunTime): 23649.723009 ms
FrameworkIncrementalNoDesugarDexGroupsOf1(RunTime): 1797.355014 ms
FrameworkIncrementalNoDesugarDexGroupsOf10(RunTime): 3820.357445 ms
FrameworkIncrementalNoDesugarDexGroupsOf100(RunTime): 11253.117639 ms
FrameworkIncrementalNoDesugarDexMerge(RunTime): 3616.097486 ms

PatchSet 1
FrameworkIncrementalNoDesugarDexAll(RunTime): 24173.855624 ms
FrameworkIncrementalNoDesugarDexGroupsOf1(RunTime): 1847.142406 ms
FrameworkIncrementalNoDesugarDexGroupsOf10(RunTime): 3946.710081 ms
FrameworkIncrementalNoDesugarDexGroupsOf100(RunTime): 11363.701225 ms
FrameworkIncrementalNoDesugarDexMerge(RunTime): 43893.294085 ms

PatchSet 2
FrameworkIncrementalNoDesugarDexAll(RunTime): 21882.13827 ms
FrameworkIncrementalNoDesugarDexGroupsOf1(RunTime): 2087.902606 ms
FrameworkIncrementalNoDesugarDexGroupsOf10(RunTime): 3621.403813 ms
FrameworkIncrementalNoDesugarDexGroupsOf100(RunTime): 10651.156906 ms
FrameworkIncrementalNoDesugarDexMerge(RunTime): 3582.435405 ms

If we are merging thousands of small dex, we create equal numbers of dex parser which means extractChecksum will re-walk all the parsed strings so far to look for the marker, making this somewhat of a log n*n operation.

Instead of relying on looking up the DexItemFactory, we can just look at the local strings within the dex file.

Since they are sorted, we only need to look in the reverse order and for the most part, don't need to look up more than 2 strings before we can conclude if the dex file has checksum marker.


Bug: 137109394
Change-Id: I497ed0e0089f15d0843cbc37f05b2d19d0a588c5
29 files changed
tree: e8b66bf7ea5f6f3efbe7e0d2c199f07a047d1b4c
  1. buildSrc/
  2. infra/
  3. library-licensing/
  4. scripts/
  5. src/
  6. tests/
  7. third_party/
  8. tools/
  9. .gitignore
  10. AUTHORS
  11. build.gradle
  12. codereview.settings
  13. compatibility-faq.md
  14. CONTRIBUTING.md
  15. copyAdditionalJctfCommonFiles.gradle
  16. LIBRARY-LICENSE
  17. LICENSE
  18. PRESUBMIT.py
  19. README.md
  20. settings.gradle
README.md

D8 dexer and R8 shrinker

The R8 repo contains two tools:

  • D8 is a dexer that converts java byte code to dex code.
  • R8 is a java program shrinking and minification tool that converts java byte code to optimized dex code.

D8 is a replacement for the DX dexer and R8 is a replacement for the Proguard shrinking and minification tool.

Downloading and building

The R8 project uses depot_tools from the chromium project to manage dependencies. Install depot_tools and add it to your path before proceeding.

The R8 project uses Java 8 language features and requires a Java 8 compiler and runtime system.

Typical steps to download and build:

$ git clone https://r8.googlesource.com/r8
$ cd r8
$ tools/gradle.py d8 r8

The tools/gradle.py script will bootstrap using depot_tools to download a version of gradle to use for building on the first run. This will produce two jar files: build/libs/d8.jar and build/libs/r8.jar.

Running D8

The D8 dexer has a simple command-line interface with only a few options.

The most important option is whether to build in debug or release mode. Debug is the default mode and includes debugging information in the resulting dex files. Debugging information contains information about local variables used when debugging dex code. This information is not useful when shipping final Android apps to users and therefore, final builds should use the --release flag to remove this debugging information to produce smaller dex files.

Typical invocations of D8 to produce dex file(s) in the out directoy:

Debug mode build:

$ java -jar build/libs/d8.jar --output out input.jar

Release mode build:

$ java -jar build/libs/d8.jar --release --output out input.jar

The full set of D8 options can be obtained by running the command line tool with the --help option.

Running R8

R8 is a Proguard replacement for whole-program optimization, shrinking and minification. R8 uses the Proguard keep rule format for specifying the entry points for an application.

Typical invocations of R8 to produce optimized dex file(s) in the out directory:

$ java -jar build/libs/r8.jar --release --output out --pg-conf proguard.cfg input.jar

The full set of R8 options can be obtained by running the command line tool with the --help option.

Testing

Typical steps to run tests:

$ tools/test.py --no_internal

The tools/test.py script will use depot_tools to download a lot of tests and test dependencies on the first run. This includes prebuilt version of the art runtime on which to validate the produced dex code.

Contributing

In order to contribute to D8/R8 you have to sign the Contributor License Agreement. If your contribution is owned by your employer you need the Corporate Contributor License Agreement.

Once the license agreement is in place, you can upload your patches using ‘git cl’ which is available in depot_tools. Once you have a change that you are happy with you should make sure that it passes all tests and then upload the change to our code review tool using:

$ git cl upload

On your first upload you will be asked to acquire credentials. Follow the instructions given by git cl upload.

On successful uploads a link to the code review is printed in the output of the upload command. In the code review tool you can assign reviewers and mark the change ready for review. At that point the code review tool will send emails to reviewers.

Getting help

For questions, reach out to us at r8-dev@googlegroups.com.

For D8, find known issues in the D8 issue tracker or file a new D8 bug report.

For R8, find known issues in the R8 issue tracker or file a new R8 bug report.