teuthology 执行流程(1)

teuthology 是一个用python写的用于Ceph功能性验证的自动化集成测试框架。teuthology 的工作方式主要通过控制远程主机（通常是虚拟机）验证Ceph的功能，测试用例包含在ceph-qa-suite中，大多是yaml风格的测试脚本。关于teuthology的介绍（here）和安装部署（here）在这篇blog中说明的很详细，本文重点关注teuthology的执行流程及部分代码分析。

Teuthology framework 的工作流程大致是这样的，首先通过scheduler节点生成需要测试suite的jobs，放到Beanstalk队列中，然后由worker节点从队列中取出任务，并在slave nodes上去执行。

scheduler生成jobs流程

teuthology 生成关于具体test suites的jobs是通过teuthology-suite命令执行的，一般在 ./teuthology/virtualenv/bin/ 目录下面，该目录下还包括其他跟teuthology相关的command：

• teuthology - Run individual jobs
• teuthology-coverage - Analyze code coverage via lcov
• teuthology-kill - Kill running jobs or entire runs
• teuthology-lock - Lock, unlock, and update status of machines
• teuthology-ls - List job results by examining an archive directory
• teuthology-openstack - Use OpenStack backend (wrapper around teuthology-suite)
• teuthology-nuke - Attempt to return a machine to a pristine state
• teuthology-queue - List, or delete, jobs in the queue
• teuthology-report - Submit test results to a web service (we use paddles)
• teuthology-results - Examing a finished run and email results
• teuthology-schedule - Schedule a single job
• teuthology-suite - Schedule a full run based on a suite (see suites in ceph-qa-suite)
• teuthology-updatekeys - Update SSH host keys for a machine
• teuthology-worker - Worker daemon to monitor the queue and execute jobs

ceph-qa-suite目录中包含的名字决定test suites的名字，目录中yaml文件通过组合或者单个产生有效的能够运行的test。运行一个suite，通过：

teuthology-suite -s <suite> [-c <ceph>] [-k <kernel>] [-e email] [-f flavor] [-t <teuth>] [-m <mtype>]

其中：

• suite: the name of the suite (the directory in ceph-qa-suite).
• ceph: ceph branch to be used.
• kernel: version of the kernel to be used.
• email: email address to send the results to.
• flavor: the kernel flavor to run against
• teuth: version of teuthology to run
• mtype: machine type of the run
• templates: template file used for further modifying the suite (optional)

-m 机器类型和 -s suite名字必须指定，一个更具体的例子：

teuthology-suite -s rbd -c wip-fix -k jewel -e [e-mail] -f basic -t jewel -m mira

上面的例子运行rbd suite使用ceph wip-fix分支，jewel内核，basic 内核flavor，teuthology的jewel分支。它会运行在mira的机器上，生成任务结束后会发送邮件到[e-mail]。

代码流程分析：

teuthology-suite命令执行入口程序 ./teuthology/suite/__init__.py 中的main()函数, 在该函数中，前面主要做了一些参数检查和配置文件赋值的工作

def main(args):

    conf = process_args(args)

    if conf.verbose:

        teuthology.log.setLevel(logging.DEBUG)

    if not conf.machine_type or conf.machine_type == 'None':

        schedule_fail("Must specify a machine_type")

    elif 'multi' in conf.machine_type:

        schedule_fail("'multi' is not a valid machine_type. " +

                      "Maybe you want 'plana,mira,burnupi' or similar")

    if conf.email:

        config.results_email = conf.email

    if conf.archive_upload:

        config.archive_upload = conf.archive_upload

        log.info('Will upload archives to ' + conf.archive_upload)

    if conf.rerun:

        rerun_filters = get_rerun_filters(conf.rerun, conf.rerun_statuses)

        if len(rerun_filters['descriptions']) == 0:

            log.warn(

                "No jobs matched the status filters: %s",

                conf.rerun_statuses,

            )

            return

        conf.filter_in.extend(rerun_filters['descriptions'])

        conf.suite = normalize_suite_name(rerun_filters['suite'])

    run = Run(conf)

    name = run.name

    run.prepare_and_schedule()

    if not conf.dry_run and conf.wait:

        return wait(name, config.max_job_time,

                    conf.archive_upload_url)

可以看出在main()函数中，最主要的还是红色的代码调用。先看Run类中，代码调用流程如下：

在create_initial_config()函数中：

• choose_kernel()

          kernel_hash = util.get_gitbuilder_hash() // 通过查询gitbuilder repo获得代表项目repository的head hash

                              ---> (arch, release, _os) = get_distro_defaults(distro, machine_type)

                                     

"""

    Given a distro (e.g. 'ubuntu') and machine type, return:

        (arch, release, pkg_type)

    This is used to default to:

        ('x86_64', 'trusty', 'deb') when passed 'ubuntu' and 'plana'

        ('armv7l', 'saucy', 'deb') when passed 'ubuntu' and 'saya'

        ('x86_64', 'wheezy', 'deb') when passed 'debian'

        ('x86_64', 'fedora20', 'rpm') when passed 'fedora'

    And ('x86_64', 'centos7', 'rpm') when passed anything else

    """

                                        bp = get_builder_project()

"""

    Depending on whether config.use_shaman is True or False, return

    GitbuilderProject or ShamanProject (the class, not an instance).

"""

                                        return bp.sha1

• choose_ceph_hash()

          """

        Get the ceph hash: if --sha1/-S is supplied, use it if it is valid, and

        just keep the ceph_branch around.  Otherwise use the current git branch

        tip.

        """

          ceph_hash = util.git_validate_sha1(repo_name, self.args.ceph_sha1) // Use http to validate that project contains sha1

• choose_ceph_version()

          ceph_version = util.package_version_for_hash(

                    ceph_hash, self.args.kernel_flavor, self.args.distro,

                    self.args.distro_version, self.args.machine_type,

                ) // 使用gitbuilder repos

在Run类中获得参数信息之后，开始准备和调度生成job，主要在prepare_and_schedule()函数中，将一些基本的参数放在一起为每个job执行teuthology-schedule， 然后把参数传给schedule_suite()， 最后调度一个“last_in_suite" job发送邮件给指定的地址（如果参数中传递了邮件地址）。

    def prepare_and_schedule(self):

        self.base_args = self.build_base_args()

        # Make sure the yaml paths are actually valid

        for yaml_path in self.base_yaml_paths:

            full_yaml_path = os.path.join(self.suite_repo_path, yaml_path)

            if not os.path.exists(full_yaml_path):

                raise IOError("File not found: " + full_yaml_path)

        num_jobs = self.schedule_suite()

        if self.base_config.email and num_jobs:

            arg = copy.deepcopy(self.base_args)

            arg.append('--last-in-suite')

            arg.extend(['--email', self.base_config.email])

            if self.args.timeout:

                arg.extend(['--timeout', self.args.timeout])

            util.teuthology_schedule(

                args=arg,

                dry_run=self.args.dry_run,

                verbose=self.args.verbose,

                log_prefix="Results email: ",

            )

            results_url = get_results_url(self.base_config.name)

            if results_url:

                log.info("Test results viewable at %s", results_url)

schedule_suite() 函数 调度suite-run，返回调度job的数量。

参考资料：

1. http://docs.ceph.com/teuthology/docs/index.html
2. http://blog.csdn.net/zstu_cc/article/details/49232187
3. http://blog.csdn.net/zstu_cc/article/details/49232225