Facebook Buck和xctool：针对Android和iOS的开源构建工具

本文包含了最新发布的Buck和xctool的细节，这是两种Facebook内部用于本地Android和iOS应用程序的构建工具。

Buck

Facebook最近开源了他们用于构建本地Android应用程序的工具Buck。Buck受到了Google Blaze的启发，创建它是为了处理与多个Android库有复杂关联的应用程序，从而减少构建时间。引入Buck之后，Facebook开发的四种本地Android应用程序中使用了单一的代码树和构建工具，这让开发更简单、更流畅，错误更少。最初的38个库在四种应用程序之间共享了500个模块。使用Buck替换了最初基于Ant的系统之后，第一次针对代码树运行时，构建时间就从3分40秒降到1分30秒。

Buck是用Java编写的，而构建规则最初是用Python定义的，后来为了更快的规则评估而采用了Jython。工具现在被用于构建Android Java代码，但是团队正在努力使其支持构建纯粹的Java代码，而与Android无关，它可能会扩展到构建Python项目，可能还会支持其他类型的项目。

Facebook的Android代码是以树的形式组织的，其中独立的子树构成了有向无环图，支持与各种不同子树相关的构建规则的并行执行，从而减少了构建时间。包含规则的构建文件分布在代码树中，每个文件针对一个已存在的Android模块。只有模块中的文件有变化时，构建文件才会执行，而这些构建可以并行执行。据Buck的创造者Michael Bolin所说，把并行能力设置为八个线程，会把单元测试的时间从20分钟降低到4分钟。他在DevCon New York 2013的话题中详细描述了构建系统。

Buck知道如何处理文件的多个版本，正如Bolin在G+的一个话题中说明的：

在构建Android应用时，另一个困难的问题是生成R.java文件。如果不探究太多细节，你可能会构建多个Android应用，它们的Java文件依赖于同一个名为R的类。对于普通的Java类，你可能会编译一次，然后任意数量的其他Java类都可以依赖它。然而，在Android中，你需要编译的R的版本依赖于Android应用，其中编译的Java代码最终会被包含在内。为你构建的每种Android应用都编译Java代码的不同版本，效率会很低，所以Buck使用了一些聪明的技巧（为编译时而不是运行时创建不同的R.java文件），从而减少重复编译。

在不久的将来，团队准备增加一个后台程序，它会监控文件的编辑情况，一旦发现就会立即开始构建。

Buck运行在Mac OS和Linux平台上，团队正在考虑把它也移植到Windows上。

Gerrit的创建者Shawn Pearce建议使用Buck替换Maven，他提到了速度以及其他几种原因：

buck: clean: 0.452s, full 1m21.083s [*], no-op: 7.145s,
maven: clean: 4.596s, full 2m53.776s, no-op: 59.108s,

[*] 完整构建包括下载所有依赖，时间可能会因为远程服务器的性能差别而不同。

Pearce还指出Buck的一些缺点：

-不支持Windows
-没有本地的Maven Central支持（通过宏）
- 没有本地的GWT、Prolog和WAR支持（通过宏）
- Buck的引导程序需要Ant

xctool

xctool是Facebook最近开源的另一种构建工具，它用于构建iOS应用程序。xctool替换了xcodebuild，具有以下特性：

• 能够作为Xcode.app运行相同的测试
• 构建输出和测试结果都是JSON格式的，使得我们不需要解析输出
• xctool只有在发现错误的时候才打印消息，而xcodebuild对每个源文件都会打印。

我们想要知道为什么Facebook基于xcodebuild构建了另一种工具，所以采访了xctool的提交者Fred Potter，询问他为什么这个工具更好一些：

xctool的最大好处在于，它可以从命令行构建和运行单元测试，这和Xcode.app从图形化界面上达到一样的效果。如果你为iOS设置了持续集成系统，那么这就非常重要了。你想要能够自动化运行测试，那些测试与你的开发人员在本地计算机上运行的完全相同，而xcodebuild不会用和Xcode.app相同的方式来构建和运行测试。在Xcode 4中，苹果把单元测试集成到了Xcode中——与“构建”和“运行”一起，有一个新的“测试”动作；使用Xcode scheme，你可以选择启用或者禁用哪些单元测试；如果你依赖于iOS模拟器（也就是应用程序测试）来编写测试，那么Xcode会自动载入模拟器并运行测试。这些都是很大的改进，但看起来苹果并没有把这些改进融入到xcodebuild中，那使得自动化构建和测试非常困难。

另一个重大的问题是构建和测试失败的报告。使用xcodebuild，你会得到大量文本输出，其中包含编译命令、编译错误和警告以及OCUnit的测试输出。如果你想要自动确定哪个组件编译失败，或者哪个单元测试失败，那么你就需要编写自己的正则表达式解析器，那也是我们和其他iOS社区中的人一直在做的工作。那会有效果，但实在很麻烦。有了xctool，我们会让xcodebuild和OCUnit测试运行器把构建输出和测试结果作为JSON对象的结构化流输出。 这让我们可以很容易地以需要的形式来显示构建和测试结果。例如，我们创建了一个报表，以吸引人的、带有颜色的输出形式来显示结果（https://fpotter_public.s3.amazonaws.com/xctool-uicatalog.gif）。 还有人使用这来把测试结果输出为JUnit XML，那在流行的Jenkins构建系统中会显示得很好。

所以，我们最初创建xctool只是为了持续集成系统，但很多开发者最后都在本地计算机上使用它。如果你想要为运行测试拥有命令行的工作流，它会非常方便。

英文：http://www.infoq.com/news/2013/05/Facebook-buck-xctool-build

This article contains details on the recently open sourced Buck and xctool, the build tools used internally by Facebook for their native Android and iOS applications.

Buck

Facebook has recently open sourced Buck, their build tool used for native Android applications. Inspired by Google Blaze, Buck was created to deal with the complexity associated with multiple Android libraries and to reduce the build time. After introducing Buck, the four native Android applications developed by Facebook ended up using a single code tree and build tool, making development simpler, more streamlined and with fewer errors. The initial 38 libraries became in time about 500 modules shared among four applications. The building time dropped from 3m 40s to 1m 30s during the first run against the code tree, Buck replacing the initial system based on Ant.

Buck is written in Java with build rules defined initially in Python, later in Jython for a quicker rule evaluation. The tool is currently used for building Android Java code, but the team is in the process of enabling it to build pure Java code, unrelated to Android, and it may be extended to build Python projects and possibly others.

Facebook’s Android code is organized in a tree with independent subtrees forming a directed acyclic graph which enables parallel execution of build rules associated with various subtrees, leading to a shorter build time. Build files, containing the rules, are spread throughout the code tree, one for each Android module existing. A build file is executed only if there is a changed file within the module, and these builds can be executed in parallel. Turning on the parallel capability using 8 threads reduced the unit tests time from 20 minutes to 4, according to Michael Bolin, the creator of Buck. He detailed the build system during a DevCon New York 2013 session.

Buck knows how to deal with multiple versions of a file, as explained by Bolin in a G+ thread:

Another one of the hard problems in building Android apps is that of generating the R.java file. Without getting too deep into the details, you may be building multiple Android apps whose Java files depend on the same class named R. For an ordinary Java class, you would compile it once and any number of other Java classes could depend on it. However, in Android, the version of R that you need to compile against depends on the Android app in which the Java code being compiled will ultimately be included. It would be inefficient to compile a different version of your Java code for every Android app you are building, so Buck employs some clever tricks (creating different R.java files for compile-time as opposed to run-time) to reduce this duplicate compilation.

In the near future, the team intends to add a daemon which monitors for file edits, starting a build immediately when it encounters one.

Buck runs on Mac OS and Linux, the team considering a port to Windows.

Shawn Pearce, creator of Gerrit, proposed using Buck instead of Maven, mentioning speed among other reasons:

buck: clean: 0.452s, full 1m21.083s [*], no-op: 7.145s,
maven: clean: 4.596s, full 2m53.776s, no-op: 59.108s,

[*] full build includes downloading all dependencies, time can vary due to remote server performance.

Pearce also noted some of Buck’s drawbacks:

- No Windows support
- No native Maven Central support (added by macros)
- No native GWT, Prolog, or WAR support (added by macros)
- Bootstrap of Buck requires Ant

xctool

xctool is yet another build tool recently open sourced by Facebook, being used for building iOS applications. xctool replaces xcodebuild, featuring the following:

• Can run the same tests as Xcode.app
• The output of builds and test result is captured in JSON, making output parsing unnecessary
• xctool prints messages only when an error is found, and not for every source file as xcodebuild does

We wanted to know why Facebook has built another tool on top of xcodebuild, so we talked toFred Potter, committer on xctool, asking him why their tool is better:

The biggest benefit of xctool is that it can build and run unit tests from the command-line the same way that Xcode.app does from the GUI.  This is really important if you're setting up a continuous integration system for iOS.  You want to be able to run the same tests under automation that your developers are running on their local machines, and xcodebuild doesn't build and run tests the same way that Xcode.app does.  With Xcode 4, Apple really integrated unit testing into Xcode - there was a new 'Test' action along with 'Build' and 'Run'; with Xcode schemes you could select which of your unit tests were enabled or disabled; and, if you were writing tests that depended on the iOS Simulator (known as application tests), Xcode would automatically launch the simulator and run your tests.  These were great improvements, but it seems Apple didn't retrofit these onto xcodebuild and that made automated building and testing a lot more difficult.

Another big problem is reporting of build and test failures.  With xcodebuild, you get a giant stream of text output that contains compiler commands, compiler errors and warnings, and test output from OCUnit.  If you want to automatically determine which component failed to build or which unit test failed, you have to write your own regex-heavy parser which is what we and others in the iOS community had been doing.  It can work, but it's a bit cumbersome.  With xctool, we trick xcodebuild and the OCUnit test runners into outputting their build and test results as a structured stream of JSON objects.  This makes it easy to display build and test results in whatever format we need.  For example, we built a reporter that shows results with attractive, colored output (https://fpotter_public.s3.amazonaws.com/xctool-uicatalog.gif).  Others have used this to output test results as JUnit XML which displays nicely in the popular Jenkins build system.

So, we originally built xctool just for our continuous integration system, but many of our developers ended up using it on their local machines.  It's handy if you want a command-line workflow for running tests.