Android中使用ioctl方式操作fence

    Android的hwc可以通过ioctl,把buffer data信息送入fb driver,由fb控制acquireFence和retireFence,以达到buffer的producer和consumer互相同步的机制,但是假如我们的hwc没有现成的ioctl可以用,又没有办法改到driver的code,我们还有一个选择:hwc可以打开/dev/sw_sync设备,通过一系列的ioctl来监控和控制fence.

    代码下载地址: https://android.googlesource.com/kernel/exynos.git
    文章中用到的code:
    exynos/include/linux/sync.h
    exynos/drivers/base/sync.c
    exynos/include/linux/sw_sync.h
    exynos/drivers/base/sw_sync.c

    在切入正文之前,我们先来了解下sync_timeline和sync_fence中出现的kref成员.在Android fb driver中的fence机制一文中,由于篇幅所限,没有对kref做过多的解释,在这一篇中我们来了解下kref的实现和工作原理.
关于kref

    sync_timeline和sync_fence中都有一个struct kref kref的成员,这东西是干什么用的呢?
sync_pt中有直接指向sync_timeline和sync_fence的指针,当sync_pt通过这些指针对sync_timeline或sync_fence做一些操作的时候,如果不能保证指向sync_time和sync_fence的指针仍然有效,我们就有大麻烦了.
    kref作为一个引用计数器可以我们帮助管理sync_timeline和sync_fence的生命周期,每当一个新的指针指向sync_time或者sync_fence时,通过kref_get增加计数;当指针不再需要的时候,通过kref_put减小计数;当kref引用为0时,自动调用对应struct的清理函数并释放资源.
那么kref是如何实现呢?其实非常简单,全部code都在单独的一个.h之中:include/linux/kref.h

struct kref {    atomic_t refcount;};

kref只有一个原子类型的refcount成员.所有对于refcount的操作都是原子的,不需要加锁.

static inline void kref_init(struct kref *kref){    atomic_set(&kref->refcount, 1); }

kref_init初始化refcount的值为1.

static inline void kref_get(struct kref *kref){    WARN_ON(!atomic_read(&kref->refcount));    atomic_inc(&kref->refcount);}

kref_get增加refcount的计数.

static inline int kref_sub(struct kref *kref, unsigned int count,         void (*release)(struct kref *kref)){    WARN_ON(release == NULL);    if (atomic_sub_and_test((int) count, &kref->refcount)) {        release(kref);        return 1;    }    return 0;}

kref_sub减小count个refcount的计数,如果recount归零,则调用传进来的release函数指针.
static inline int kref_put(struct kref *kref, void (*release)(struct kref *kref))
{
    return kref_sub(kref, 1, release);
}

kref_put是count为1的kref_sub特例.

我们来看下kref在sync_time,sync_pt之间是如何运用的.

struct sync_timeline *sync_timeline_create(const struct sync_timeline_ops *ops,   int size, const char *name){struct sync_timeline *obj;obj = kzalloc(size, GFP_KERNEL);...kref_init(&obj->kref);...return obj;}

sync_timeline_create的时候,对sync_timeline的kref成员做了初始化,kref->refcount == 1;

static void sync_timeline_free(struct kref *kref){struct sync_timeline *obj =container_of(kref, struct sync_timeline, kref);unsigned long flags;if (obj->ops->release_obj)obj->ops->release_obj(obj);spin_lock_irqsave(&sync_timeline_list_lock, flags);list_del(&obj->sync_timeline_list);spin_unlock_irqrestore(&sync_timeline_list_lock, flags);kfree(obj);}void sync_timeline_destroy(struct sync_timeline *obj){obj->destroyed = true;/* * If this is not the last reference, signal any children * that their parent is going away. */if (!kref_put(&obj->kref, sync_timeline_free))sync_timeline_signal(obj);}

sync_timeline_destroy的时候调用kref_put减小kref的refcount,如果refcount归零,则调用sync_timeline_free清理sync_timeline并释放资源.
为什么在sync_time_destroy的时候refcount可能会不为0呢?因为在sync_pt_create的时候会调用kref_get增加refcount计数.如果sync_timeline_destroy的时候,sync_timeline中的sync_pt还没有释放,sync_timeline的资源就不能释放,否则sync_pt中指向sync_timeline的的parent指针就会指向无效的地址.

struct sync_pt *sync_pt_create(struct sync_timeline *parent, int size){struct sync_pt *pt;if (size < sizeof(struct sync_pt))return NULL;pt = kzalloc(size, GFP_KERNEL);if (pt == NULL)return NULL;INIT_LIST_HEAD(&pt->active_list);//先增加parent的kref计数,再把自己加入到timeline中,保证在加入timeline的过程中timeline一直是有效的.kref_get(&parent->kref);sync_timeline_add_pt(parent, pt);return pt;}void sync_pt_free(struct sync_pt *pt){if (pt->parent->ops->free_pt)pt->parent->ops->free_pt(pt);sync_timeline_remove_pt(pt);//pt从timeline移除后,减小parent的kref计数,如果计数归零,说明timieline不再需要,可以清理了.kref_put(&pt->parent->kref, sync_timeline_free);kfree(pt);}

sync_fence中的kref的用法也是类似的,这里不再赘述.关于kref有兴趣的话还可以参考kernel中的说明文档:Documentation/kref.txt

ioctl方式操作sync_timeline和sync_fence

在上一篇文章中,我们讨论了fb driver中对于fence的处理,我们知道sw_sync_timeline"继承"了sync_time.阅读sw_sync.c可以发现,sw_sync_time还在/dev下生成了一个/dev/sw_sync的设备

static const struct file_operations sw_sync_fops = {.owner = THIS_MODULE,.open = sw_sync_open,.release = sw_sync_release,.unlocked_ioctl = sw_sync_ioctl,};static struct miscdevice sw_sync_dev = {.minor= MISC_DYNAMIC_MINOR,.name= "sw_sync",.fops= &sw_sync_fops,};int __init sw_sync_device_init(void){return misc_register(&sw_sync_dev);}void __exit sw_sync_device_remove(void){misc_deregister(&sw_sync_dev);}module_init(sw_sync_device_init);module_exit(sw_sync_device_remove);

我们可以看到对于/dev/sw_sync设备,sw_sync_timeline实现了三个方法

1. sw_sync_open

2. sw_sync_release

3. sw_sync_ioctl

我们分别来分析这三个函数

1. sw_sync_open

int sw_sync_open(struct inode *inode, struct file *file){struct sw_sync_timeline *obj;char task_comm[TASK_COMM_LEN];get_task_comm(task_comm, current);obj = sw_sync_timeline_create(task_comm);if (obj == NULL)return -ENOMEM;file->private_data = obj;return 0;}

sw_sync_timeline_create我们在上一篇文章Android fb driver中的fence机制中已经分析过,这里还是调用的同样的函数,只不过timeline直接使用了task_name作为name.之后sw_sync_timeline作为file的private_data保存起来,之后对该file操作的函数都可以通过private_data拿到sw_sync_timeline.

2. sw_sync_release

int sw_sync_release(struct inode *inode, struct file *file){struct sw_sync_timeline *obj = file->private_data;sync_timeline_destroy(&obj->obj);return 0;}

从preivate_data中拿出之前open的时候保存的sw_sync_timeline,调用sync_timeline_destroy来销毁timeline.

void sync_timeline_destroy(struct sync_timeline *obj){obj->destroyed = true;/* * If this is not the last reference, signal any children * that their parent is going away. */if (!kref_put(&obj->kref, sync_timeline_free))sync_timeline_signal(obj);}

调用sync_timeline_free,并通知调用sync_timeline_signal通知所有的pt.

3. sw_sync_ioctllong sw_sync_ioctl(struct file *file, unsigned int cmd, unsigned long arg){struct sw_sync_timeline *obj = file->private_data;switch (cmd) {case SW_SYNC_IOC_CREATE_FENCE:return sw_sync_ioctl_create_fence(obj, arg);case SW_SYNC_IOC_INC:return sw_sync_ioctl_inc(obj, arg);default:return -ENOTTY;}}

ioctl支持两种操作: SW_SYNC_IOC_CREATE_FENCE和SW_SYNC_IOC_INC,顾名思义,一个用来create fence,另一个用来增加sw_sync_timeline的value,进而signal sw_sync_pt和对应的sync_fence.其中sw_sync_ioctl_inc只是直接调用了sw_sync_timeline_inc,关于sw_sync_timeline_inc的详细内容请参考Android fb driver中的fence机制这一篇文章.我们来看下sw_sync_ioctl_create_fence.

struct sw_sync_create_fence_data {//value将赋值给sw_sync_pt->value__u32value;//sync_fence的namecharname[32];//return给user space的fence fd__s32fence; /* fd of new fence */};long sw_sync_ioctl_create_fence(struct sw_sync_timeline *obj, unsigned long arg){//获取一个可用的fdint fd = get_unused_fd();int err;struct sync_pt *pt;struct sync_fence *fence;struct sw_sync_create_fence_data data;if (fd < 0)return fd;if (copy_from_user(&data, (void __user *)arg, sizeof(data))) {err = -EFAULT;goto err;}//调用pt的create函数,将user space的data.value作为新pt的valuept = sw_sync_pt_create(obj, data.value);if (pt == NULL) {err = -ENOMEM;goto err;}data.name[sizeof(data.name) - 1] = '\0';//以pt为参数构建一个fence,data.name作为fence的名字fence = sync_fence_create(data.name, pt);if (fence == NULL) {sync_pt_free(pt);err = -ENOMEM;goto err;}//将fd写回user spacedata.fence = fd;if (copy_to_user((void __user *)arg, &data, sizeof(data))) {sync_fence_put(fence);err = -EFAULT;goto err;}//如果一切顺利,再把fence install到fd中,这样所有对fd的操作都是对fence的操作了.sync_fence_install(fence, fd);return 0;err:put_unused_fd(fd);return err;}

我们看到sw_sync提供的ioctl只能create timeline和inc timeline,也就是说只能singal fence.fence的用法一般是一方signal,另一方wait.那么user space要怎么做才能wait fence呢?我们注意到sw_sync_ioctl_create_fence把fence对应的fd送回了user space,既然我们拿到了fd,就可以对这个fd施展一些手段了.到sync.c中看下fence fd都支持哪些操作.

static const struct file_operations sync_fence_fops = {.release = sync_fence_release,.poll = sync_fence_poll,.unlocked_ioctl = sync_fence_ioctl,};static struct sync_fence *sync_fence_alloc(const char *name){...struct sync_fence *fence;unsigned long flags;fence = kzalloc(sizeof(struct sync_fence), GFP_KERNEL);if (fence == NULL)return NULL;fence->file = anon_inode_getfile("sync_fence", &sync_fence_fops, fence, 0);...}

从sync_fence_alloc函数中可以知道fence对应的file所支持的操作都列在sync_fence_fops中.


1.sync_fence_release

static int sync_fence_release(struct inode *inode, struct file *file){struct sync_fence *fence = file->private_data;unsigned long flags;/* * We need to remove all ways to access this fence before droping * our ref. * * start with its membership in the global fence list */spin_lock_irqsave(&sync_fence_list_lock, flags);list_del(&fence->sync_fence_list);spin_unlock_irqrestore(&sync_fence_list_lock, flags);/* * remove its pts from their parents so that sync_timeline_signal() * can't reference the fence. */sync_fence_detach_pts(fence);//减小fence的kref计数,如果计数归零,调用sync_fence_free释放资源.//sync_fence_free的时候会调用sync_fence_free_pts,继而调用sync_pt_free释放pt资源,都是简单的调用,不再展开kref_put(&fence->kref, sync_fence_free);return 0;}//将fence中pt_list_head中多有的pt从timeline中移除.static void sync_fence_detach_pts(struct sync_fence *fence){    struct list_head *pos, *n;    list_for_each_safe(pos, n, &fence->pt_list_head) {        struct sync_pt *pt = container_of(pos, struct sync_pt, pt_list);//从timeline的active_list和child_list中移除pt,code很简单这里不再展开.        sync_timeline_remove_pt(pt);    }    }

2.sync_fence_poll

static unsigned int sync_fence_poll(struct file *file, poll_table *wait){struct sync_fence *fence = file->private_data;//加入到poll_table的waitqueue是fence->wq,和我们在sync_fence_signal_pt和sync_fence_wait中的fence->wq是同一个wait_queue_head_t//具体请参考<a target=_blank href="http://blog.csdn.net/ear5cm/article/details/45093807">Android fb driver中的fence机制</a>一文poll_wait(file, &fence->wq, wait);/* * Make sure that reads to fence->status are ordered with the * wait queue event triggering */smp_rmb();//status取值//0 active//1 signal//-1 errorif (fence->status == 1)return POLLIN;else if (fence->status < 0)return POLLERR;elsereturn 0;}

3.sync_fence_ioctl

static long sync_fence_ioctl(struct file *file, unsigned int cmd,                 unsigned long arg){    struct sync_fence *fence = file->private_data;    switch (cmd) {    case SYNC_IOC_WAIT:        return sync_fence_ioctl_wait(fence, arg);    case SYNC_IOC_MERGE:        return sync_fence_ioctl_merge(fence, arg);    case SYNC_IOC_FENCE_INFO:        return sync_fence_ioctl_fence_info(fence, arg);    default:        return -ENOTTY;    }}

fence支持的3个ioctl中,
sync_fence_ioctl_wait我们在Android fb driver中的fence机制一文中已经分析过了.
sync_fence_ioctl_merge吃进的arg其实是sync_merge_data类型:

/** * struct sync_merge_data - data passed to merge ioctl * @fd2:file descriptor of second fence * @name:name of new fence * @fence:returns the fd of the new fence to userspace */struct sync_merge_data {__s32fd2; /* fd of second fence */charname[32]; /* name of new fence */__s32fence; /* fd on newly created fence */};

sync_fence_ioctl_merge的功能是将本身这个fence和fd2指向的fence中所有的pt合并到fence指向的新fence当中,新fence以name为名字.
sync_fence_ioctl_fence_info向user space写回fence和它所包含的pt的信息,送给user space的data类型其实是sync_fence_info_data.

/** * struct sync_fence_info_data - data returned from fence info ioctl * @len:ioctl caller writes the size of the buffer its passing in. *ioctl returns length of sync_fence_data reutnred to userspace *including pt_info. * @name:name of fence * @status:status of fence. 1: signaled 0:active <0:error * @pt_info:a sync_pt_info struct for every sync_pt in the fence */struct sync_fence_info_data {__u32len;charname[32];__s32status;__u8pt_info[0];};

其中pt_info是大小为0的数组,所以pt_info其实指向的是紧挨status后面的地址,pt_info的大小和fence中pt的个数和pt的具体类型有关,在Android fb driver中的fence机制中提到的sync_timeline->ops->fill_driver_data会负责填写每个pt的信息.


大致总结一下,在user space:
1. 通过open /dev/sw_sync打开sw_sync设备,open函数sw_sync_open会为我们create一个sw_sync_timeline
2. 通过ioctl SW_SYNC_IOC_CREATE_FENCE创造新的fence,kernel中的sw_sync_ioctl_create_fence函数会先create一个sw_sync_pt的,并把它装进一个fence,并把fence的fd返回给user space.
3. 通过ioctl SW_SYNC_IOC_INC call到sw_sync_ioctl_inc,增加sw_sync_timeline的value,以singal那些pt->value值小于等于timeline->value的pt,进而signal pt所属的fence.
4. 通过fence fd的ioctl SYNC_IOC_WAIT,可以在user space等待fence在一定时间段内被signal.

5. 当fence fd在kernel中所对应的fence file不再被引用时,sync_fence_release会释放fence资源.

PS:其实Android framework就是用的上述这个方法来实现fence机制.具体可以参考aosp中libsync的实现,code在aosp: system/core/libsync.

之后会单独开一篇来分析Android framework中的fence实现.