android compile tasks中dex过程源码情景分析

0x00 前言

MultiDex中出现的main dex capacity exceeded解决之道中我们知道main dex的class可以由maindexlist.txt指定，Android MultiDex机制杂谈中我们分析了google MultiDex机制中Secondary dex的install过程，那么，我们的app在android gradle build过程中，.dex文件是怎么创建的呢？ 再者，Secondary dex中的class是按什么顺序分配到不同dex中的呢？

---

0x01 android build system概述

为了解答上面的两个问题，本文将进一步分析android build system源码。
android build system是google提供的一组用来构建、运行、测试和打包我们app的工具集，包含了aapt、aidl、javac、dex、apkbuilder、Jarsigner、zipalign等工具。在我们构建app时，build进程会去按一定顺序调用上述工具来生成相应文件，而最终的输出将会是一个完整的可安装的.apk文件，构建流程如下：

构建系统先从product flavors, build types和dependencies中合并资源，如果不同目录下有重名资源，将按以下优先级进行覆盖：

dependencies > build types > product flavors > main source directory

1. aapt编译应用的资源文件（如AndroidManifest.xml），输出R.java文件
2. aidl把.aidl文件转换为对应的java interface文件
3. javac编译所有.java文件，输出.class文件
4. dex工具把上面生成的.class文件转换为.dex文件
5. apkbuilder把所有没编译的资源（如图片），编译过的资源和dex文件打包输出为.apk文件
6. 在release模式下，用zipalign工具对.apk进行对齐处理，以减少运行时内存占用

本文重点对第4步中.class经过dex到.dex过程源码进行分析。

---

0x02 android compile tasks分析

为了更好地分析.dex的产生过程，本文设定情景如下：

构建工具为gradle，采用android plugin 'com.android.application'，method数超过65535，需要进行multidex，并且指定了multiDexEnabled = true。

在shell终端cd到project根目录，输入：

gradle assemble

gradle进程会启动，在dex之前，进程控制流将进入VariantManager. createTasksForVariantData。添加完assemble task依赖后，会去调用taskManager.createTasksForVariantData(tasks, variantData)。由于android plugin为’com.android.application’，这里的taskManager是ApplicationTaskManager。

com/android/build/gradle/internal/VariantManager.java

/** * Create tasks for the specified variantData. */public void createTasksForVariantData(        final TaskFactory tasks,        final BaseVariantData<? extends BaseVariantOutputData> variantData) {    // Add dependency of assemble task on assemble build type task.    tasks.named("assemble", new Action<Task>() {        @Override        public void execute(Task task) {            BuildTypeData buildTypeData = buildTypes.get(                            variantData.getVariantConfiguration().getBuildType().getName());            task.dependsOn(buildTypeData.getAssembleTask());        }    });    ...        taskManager.createTasksForVariantData(tasks, variantData);    }}

ApplicationTaskManager.createTasksForVariantData()会通过ThreadRecorder.get().record()第二个callback参数的类型为Recorder.Block<Void>，在call回调中调用父类TaskManager.createPostCompilationTasks。ThreadRecorder可以记录该任务的在当前线程的执行时间，并且保证task之间是串行的。

/** * TaskManager for creating tasks in an Android application project. */public class ApplicationTaskManager extends TaskManager {    @Override    public void createTasksForVariantData(            @NonNull final TaskFactory tasks,            @NonNull final BaseVariantData<? extends BaseVariantOutputData> variantData) {            ...        // Add a compile task        ThreadRecorder.get().record(ExecutionType.APP_TASK_MANAGER_CREATE_COMPILE_TASK,                new Recorder.Block<Void>() {                    @Override                    public Void call() {                        AndroidTask<JavaCompile> javacTask = createJavacTask(tasks, variantScope);                        if (variantData.getVariantConfiguration().getUseJack()) {                            createJackTask(tasks, variantScope);                        } else {                            setJavaCompilerTask(javacTask, tasks, variantScope);                            createJarTask(tasks, variantScope);                            createPostCompilationTasks(tasks, variantScope);                        }                        return null;                    }                });                ...    }}

TaskManager.createPostCompilationTasks方法，这个方法比较长，我们分段来分析。

首先从config得到isMultiDexEnabled，isMultiDexEnabled，isLegacyMultiDexMode，由于已经假设当前为需要MultiDex的场景，因此isMultiDexEnabled为true。若isMinifyEnabled也为true，则说明输入jar包需要进行混淆，本场景先不考虑。

TaskManager.java

/** * Creates the post-compilation tasks for the given Variant. * * These tasks create the dex file from the .class files, plus optional intermediary steps like * proguard and jacoco * */public void createPostCompilationTasks(TaskFactory tasks, @NonNull final VariantScope variantScope) {    checkNotNull(variantScope.getJavacTask());    final ApkVariantData variantData = (ApkVariantData) variantScope.getVariantData();    final GradleVariantConfiguration config = variantData.getVariantConfiguration();    TransformManager transformManager = variantScope.getTransformManager();...    boolean isMinifyEnabled = config.isMinifyEnabled();    boolean isMultiDexEnabled = config.isMultiDexEnabled();    boolean isLegacyMultiDexMode = config.isLegacyMultiDexMode();    AndroidConfig extension = variantScope.getGlobalScope().getExtension();

在支持MultiDex的场景中，先创建manifestKeepListTask，将依赖设置为ManifestProcessorTask，这些android compile task由AndroidTask<TransformTask>类型来描述。

接着创建multiDexClassListTask，依赖manifestKeepListTask。这两个tasks用来输出maindexlist.txt，其中包含了MainDex中必须的class，可参见MultiDex中出现的main dex capacity exceeded解决之道。

// ----- Multi-Dex supportAndroidTask<TransformTask> multiDexClassListTask = null;// non Library test are running as native multi-dexif (isMultiDexEnabled && isLegacyMultiDexMode) {    if (AndroidGradleOptions.useNewShrinker(project)) {        throw new IllegalStateException("New shrinker + multidex not supported yet.");    }    // ----------    // create a transform to jar the inputs into a single jar.    if (!isMinifyEnabled) {        // merge the classes only, no need to package the resources since they are        // not used during the computation.        JarMergingTransform jarMergingTransform = new JarMergingTransform(                TransformManager.SCOPE_FULL_PROJECT);        transformManager.addTransform(tasks, variantScope, jarMergingTransform);    }        // ----------    // Create a task to collect the list of manifest entry points which are    // needed in the primary dex    AndroidTask<CreateManifestKeepList> manifestKeepListTask = androidTasks.create(tasks,            new CreateManifestKeepList.ConfigAction(variantScope));    manifestKeepListTask.dependsOn(tasks,            variantData.getOutputs().get(0).getScope().getManifestProcessorTask());    // ---------    // create the transform that's going to take the code and the proguard keep list    // from above and compute the main class list.    MultiDexTransform multiDexTransform = new MultiDexTransform(            variantScope.getManifestKeepListFile(),            variantScope,            null);    multiDexClassListTask = transformManager.addTransform(            tasks, variantScope, multiDexTransform);    multiDexClassListTask.dependsOn(tasks, manifestKeepListTask);}

最后创建dexTask，这个用来把.class文件转为.dex的task，它依赖multiDexClassListTask。

    // create dex transform    DexTransform dexTransform = new DexTransform(            extension.getDexOptions(),            config.getBuildType().isDebuggable(),            isMultiDexEnabled,            isMultiDexEnabled && isLegacyMultiDexMode ? variantScope.getMainDexListFile() : null,            variantScope.getPreDexOutputDir(),            variantScope.getGlobalScope().getAndroidBuilder(),            getLogger());    AndroidTask<TransformTask> dexTask = transformManager.addTransform(            tasks, variantScope, dexTransform);    // need to manually make dex task depend on MultiDexTransform since there's no stream    // consumption making this automatic    dexTask.optionalDependsOn(tasks, multiDexClassListTask);}

task执行时，gradle引擎会去调用含有@TaskAction注解的方法，TransformTask类拥有Transfrom类型字段，其transform方法被标记为@TaskAction。同样通过ThreadRecorder.get().record中回调call()，执行transform.transform()

TransformTask.java

/** * A task running a transform. */@ParallelizableTaskpublic class TransformTask extends StreamBasedTask implements Context {    private Transform transform;    ...    @TaskAction    void transform(final IncrementalTaskInputs incrementalTaskInputs)            throws IOException, TransformException, InterruptedException { ...        ThreadRecorder.get().record(ExecutionType.TASK_TRANSFORM,                new Recorder.Block<Void>() {                    @Override                    public Void call() throws Exception {                        transform.transform(                                TransformTask.this,                                consumedInputs.getValue(),                                referencedInputs.getValue(),                                outputStream != null ? outputStream.asOutput() : null,                                isIncremental.getValue());                        return null;                    }                },                new Recorder.Property("project", getProject().getName()),                new Recorder.Property("transform", transform.getName()),                new Recorder.Property("incremental", Boolean.toString(transform.isIncremental())));    }

上述android compile tasks关系可以用下图描述：

从gradle task角度上看，这些task都属于TransformTask（继承至DefaultTask），它们区别仅在于transform字段。DexTask是本文主要关心的task，下面分析这个task执行过程中都做了什么。

---

0x03 DexTask执行过程分析

android build system中dex过程发生在DexTask，DexTask关联的Transform是DexTransform。

当DexTransform.transfrom方法被调用时，会先创建并初始化main目录作为输出dex的目录，然后调用androidBuilder.convertByteCode方法进行.class到.dex的转换，此时jarInputs为classes.jar，directoryInputs长度为空，传递的boolean类型的multiDex参数来自build.gralde文件中在defaultConfig对multiDexEnabled = true的设置。

DexTransform.java

@Overridepublic void transform(        @NonNull Context context,        @NonNull Collection<TransformInput> inputs,        @NonNull Collection<TransformInput> referencedInputs,        @Nullable TransformOutputProvider outputProvider,        boolean isIncremental) throws TransformException, IOException, InterruptedException {        ...    // Gather a full list of all inputs.    List<JarInput> jarInputs = Lists.newArrayList();    List<DirectoryInput> directoryInputs = Lists.newArrayList();    for (TransformInput input : inputs) {        jarInputs.addAll(input.getJarInputs());        directoryInputs.addAll(input.getDirectoryInputs());    }            try {        // if only one scope or no per-scope dexing, just do a single pass that        // runs dx on everything.        if ((jarInputs.size() + directoryInputs.size()) == 1 || !dexOptions.getPreDexLibraries()) {            File outputDir = outputProvider.getContentLocation("main",                    getOutputTypes(), getScopes(),                    Format.DIRECTORY);            FileUtils.mkdirs(outputDir);            // first delete the output folder where the final dex file(s) will be.            FileUtils.emptyFolder(outputDir);            // gather the inputs. This mode is always non incremental, so just            // gather the top level folders/jars            final List<File> inputFiles = Lists.newArrayList();            for (JarInput jarInput : jarInputs) {                inputFiles.add(jarInput.getFile());            }            for (DirectoryInput directoryInput : directoryInputs) {                inputFiles.add(directoryInput.getFile());            }            androidBuilder.convertByteCode(                    inputFiles,                    outputDir,                    multiDex,                    mainDexListFile,                    dexOptions,                    null,                    false,                    true,                    new LoggedProcessOutputHandler(logger));        } else {

为了把输入的.class转换为.dex，AndroidBuilder.convertByteCode会另起进程去做dex，实际上是在新进程中exec dex工具，接下来我们进入dex源码，看看到底发生了什么。

 public void convertByteCode(         @NonNull Collection<File> inputs,         @NonNull File outDexFolder,                  boolean multidex,         @Nullable File mainDexList,         @NonNull DexOptions dexOptions,         @Nullable List<String> additionalParameters,         boolean incremental,         boolean optimize,         @NonNull ProcessOutputHandler processOutputHandler)         throws IOException, InterruptedException, ProcessException {...     BuildToolInfo buildToolInfo = mTargetInfo.getBuildTools();     DexProcessBuilder builder = new DexProcessBuilder(outDexFolder);     builder.setVerbose(mVerboseExec)             .setIncremental(incremental)             .setNoOptimize(!optimize)             .setMultiDex(multidex)             .setMainDexList(mainDexList)             .addInputs(verifiedInputs.build());     if (additionalParameters != null) {         builder.additionalParameters(additionalParameters);     }     JavaProcessInfo javaProcessInfo = builder.build(buildToolInfo, dexOptions);     ProcessResult result = mJavaProcessExecutor.execute(javaProcessInfo, processOutputHandler);     result.rethrowFailure().assertNormalExitValue(); }

---

0x04 dex过程分析

android 5.0中dex工具源码路径是dalvik/dx/src/com/android/dx，入口类是com.android.dx.command.Main，当解析到参数–dex时，转入com.android.dx.command.dexer.Main.main()

 public static void main(String[] args) {...     try {...             if (arg.equals("--dex")) {                 com.android.dx.command.dexer.Main.main(without(args, i));                 break;             } else if (arg.equals("--dump")) {                 com.android.dx.command.dump.Main.main(without(args, i));                 break;             }             ...         }

main会调用com.android.dx.command.dexer.Main.run()，此时args.multiDex为true，直接进入runMultiDex

com.android.dx.command.dexer.Main.java

public static int run(Arguments arguments) throws IOException { ...    try {        if (args.multiDex) {            return runMultiDex();        } else {            return runMonoDex();        }    } finally {        closeOutput(humanOutRaw);    }}

runMultiDex会调用processAllFiles，第一行代码调用createDexFile()

 private static boolean processAllFiles() {     createDexFile();...

createDexFile先检查outputDex(: DexFile)字段是否为空，不为空则调用writeDex()把该dex的byte[]添加到dexOutputArrays(: List<byte[]>)。

writeDex()具体是通过outputDex.toDex(humanOutWriter, args.verboseDump)得到dex的byte[]。java中数组的下标是int类型，长度为32bits，因此一个dex文件最大理论是4G，但实际由于method, field数等限制，正常最大也就10M左右。

然后还会为outputDex字段新建一个DexFile对象，表示当前dex文件已经处理完毕，可以开始处理新的dex文件了。这里假设进程第一次执行createDexFile，因此outputDex为null。

private static void createDexFile() {    if (outputDex != null) {        dexOutputArrays.add(writeDex());    }    outputDex = new DexFile(args.dexOptions);    if (args.dumpWidth != 0) {        outputDex.setDumpWidth(args.dumpWidth);    }}

随后processAllFiles会根据args中numThreads来决定是否需要创建线程池。

if (args.numThreads > 1) {    threadPool = Executors.newFixedThreadPool(args.numThreads);    parallelProcessorFutures = new ArrayList<Future<Void>>();}

接下来判断args.mainDexListFile，不为空说明指定了maindexlist.txt文件，这里假设不为空，filesNames数组是{‘path/way/to/classes.jar’}，长度为1。方法在for循环中调用processOne()

...   anyFilesProcessed = false;   String[] fileNames = args.fileNames;   ...   try {       if (args.mainDexListFile != null) {           // with --main-dex-list           FileNameFilter mainPassFilter = args.strictNameCheck ? new MainDexListFilter() :               new BestEffortMainDexListFilter();           // forced in main dex           for (int i = 0; i < fileNames.length; i++) {               processOne(fileNames[i], mainPassFilter);           }

processOne调用ClassPathOpener.process处理输入的classes.jar。ClassPathOpener会遍历classes.jar中的每个ZipEntry，读出byte[]，对每个ZipEntry在回调processFileBytes中调用Main.processFileBytes方法。

/** * Processes one pathname element. * * @param pathname {@code non-null;} the pathname to process. May * be the path of a class file, a jar file, or a directory * containing class files. * @param filter {@code non-null;} A filter for excluding files. */private static void processOne(String pathname, FileNameFilter filter) {    ClassPathOpener opener;    opener = new ClassPathOpener(pathname, false, filter,            new ClassPathOpener.Consumer() {        @Override        public boolean processFileBytes(String name, long lastModified, byte[] bytes) {            return Main.processFileBytes(name, lastModified, bytes);        }...   });    if (args.numThreads > 1) {        parallelProcessorFutures.add(threadPool.submit(new ParallelProcessor(opener)));    } else {        if (opener.process()) {            anyFilesProcessed = true;        }    }}

Main.processFileBytes把输入的bytes分为三类：

• .class文件
• .dex文件
• 资源文件

如果输入是.dex或资源文件，则把bytes分别写入libraryDexBuffers字段或outputResources字段，此时输入name(: String)为.class。当发现是class，则进一步调用processClass处理

 /**  * Processes one file, which may be either a class or a resource.  *  * @param name {@code non-null;} name of the file  * @param bytes {@code non-null;} contents of the file  * @return whether processing was successful  */ private static boolean processFileBytes(String name, long lastModified, byte[] bytes) {     boolean isClass = name.endsWith(".class");     boolean isClassesDex = name.equals(DexFormat.DEX_IN_JAR_NAME);     boolean keepResources = (outputResources != null);...      String fixedName = fixPath(name);     if (isClass) {         if (keepResources && args.keepClassesInJar) {             synchronized (outputResources) {                 outputResources.put(fixedName, bytes);             }         }         if (lastModified < minimumFileAge) {             return true;         }         return processClass(fixedName, bytes);     } else if (isClassesDex) {         synchronized (libraryDexBuffers) {             libraryDexBuffers.add(bytes);         }         return true;     } else {         synchronized (outputResources) {             outputResources.put(fixedName, bytes);         }         return true;     } }

processClass方法主要做了以下几件事：

1. 为传入的class创建DirectClassFile对象，对应.class字节码文件
2. 得到已经生成的dex的numMethodIds，numFieldIds
3. 得到新Class的constantPoolSize，计算maxMethodIdsInDex = numMethodIds + constantPoolSize + 新Class的方法数 + 2个预留method， 计算maxFieldIdsInDex = numFieldIds + constantPoolSize + 新Class的字段数 + 9个预留field
4. 一旦发现maxMethodIdsInDex > args.maxNumberOfIdxPerDex 或者 maxFieldIdsInDex > args.maxNumber OfIdxPerDex，说明当前dex已经满了，调用createDexFile创建新dex来容纳该Class
5. 否则，通过CfTranslator.translate方法将输入的DirectClassFile对象，得到ClassDefItem，添加到outputDex(: DexFile)

由此可以看出：

secondray dex中的class是根据classes.jar中ZipEntry的遍历顺序添加的。

/**  * Processes one classfile.  *  * @param name {@code non-null;} name of the file, clipped such that it  * <i>should</i> correspond to the name of the class it contains  * @param bytes {@code non-null;} contents of the file  * @return whether processing was successful  */ private static boolean processClass(String name, byte[] bytes) {     if (! args.coreLibrary) {         checkClassName(name);     }     DirectClassFile cf =         new DirectClassFile(bytes, name, args.cfOptions.strictNameCheck);     cf.setAttributeFactory(StdAttributeFactory.THE_ONE);     cf.getMagic();     int numMethodIds = outputDex.getMethodIds().items().size();     int numFieldIds = outputDex.getFieldIds().items().size();     int constantPoolSize = cf.getConstantPool().size();     int maxMethodIdsInDex = numMethodIds + constantPoolSize + cf.getMethods().size() +             MAX_METHOD_ADDED_DURING_DEX_CREATION;     int maxFieldIdsInDex = numFieldIds + constantPoolSize + cf.getFields().size() +             MAX_FIELD_ADDED_DURING_DEX_CREATION;     if (args.multiDex         // Never switch to the next dex if current dex is already empty         && (outputDex.getClassDefs().items().size() > 0)         && ((maxMethodIdsInDex > args.maxNumberOfIdxPerDex) ||             (maxFieldIdsInDex > args.maxNumberOfIdxPerDex))) {         DexFile completeDex = outputDex;         createDexFile();         assert  (completeDex.getMethodIds().items().size() <= numMethodIds +                 MAX_METHOD_ADDED_DURING_DEX_CREATION) &&                 (completeDex.getFieldIds().items().size() <= numFieldIds +                 MAX_FIELD_ADDED_DURING_DEX_CREATION);     }     try {         ClassDefItem clazz =             CfTranslator.translate(cf, bytes, args.cfOptions, args.dexOptions, outputDex);         synchronized (outputDex) {             outputDex.add(clazz);         }         return true;     } catch (ParseException ex) {         DxConsole.err.println("\ntrouble processing:");         if (args.debug) {             ex.printStackTrace(DxConsole.err);         } else {             ex.printContext(DxConsole.err);         }     }     errors.incrementAndGet();     return false; }

再回到processAllFiles，前面假设指定了maindexlist，如果minialMainDex也为true的话，会立即创建新的DexFile，保证这个main dex中只包含maindexlist里的类，如何指定可以参考MultiDex中出现的main dex capacity exceeded解决之道 0x05。前面没有过滤掉的class都会放入到secondary dex。

        if (dexOutputArrays.size() > 0) {            throw new DexException("Too many classes in " + Arguments.MAIN_DEX_LIST_OPTION                    + ", main dex capacity exceeded");        }        if (args.minimalMainDex) {            // start second pass directly in a secondary dex file.            createDexFile();        }        // remaining files        for (int i = 0; i < fileNames.length; i++) {            processOne(fileNames[i], new NotFilter(mainPassFilter));        }    } else {        // without --main-dex-list        for (int i = 0; i < fileNames.length; i++) {            processOne(fileNames[i], ClassPathOpener.acceptAll);        }    }} catch (StopProcessing ex) {    /*     * Ignore it and just let the error reporting do     * their things.     */}

在runMultiDex的最后，dex文件将以classes(..N).dex的形式输出在由args.outName指定的目录之下。

private static int runMultiDex() throws IOException {...        } else if (args.outName != null) {            File outDir = new File(args.outName);            assert outDir.isDirectory();            for (int i = 0; i < dexOutputArrays.size(); i++) {                OutputStream out = new FileOutputStream(new File(outDir, getDexFileName(i)));                try {                    out.write(dexOutputArrays.get(i));                } finally {                    closeOutput(out);                }            }        }

---

0x05 结论

通过对android build system中android plugin tasks和dx工具源码的分析，我们可以得出如下结论：

• .dex文件本质上是.class文件经过com.android.dx.dex.file.DexFile.toDex方法转换得到

• Secondary dex是在指定了multiDexEnabled = true且MainDex满足65535限制，或者指定multiDexEnabled = true和minimalMainDex = true的情况下，才会创建的dex，其包含的class是根据classes.jar中ZipEntry的遍历顺序添加的。

•