④NuPlayer播放框架之Renderer源码分析

[时间：2016-11] [状态：Open]
[关键词：android，nuplayer，开源播放器，播放框架，渲染器，render]

0 导读

之前我们分析了NuPlayer的实现代码，本文将重点聚焦于其中的一部分——渲染器（Renderer）。
从功能安排来说，Renderer的主要功能有：

• 音视频原始数据缓存操作
• 音频播放（到声卡）
• 视频显示（到显卡）
• 音视频同步
• 其他辅助播放器控制的操作
• 其他获取渲染状态/属性的接口

接下来主要从Renderer的对外接口和实现说明下其中的处理逻辑。

本文是我的NuPlayer播放框架的第四篇。

1 NuPlayer::Renderer对外接口及主要成员

// code frome ~/frameworks/av/media/libmediaplayerservice/nuplayer/NuPlayerRenderer.hstruct NuPlayer::Renderer : public AHandler {    Renderer(const sp<MediaPlayerBase::AudioSink> &sink,             const sp<AMessage> &notify, uint32_t flags = 0);    static size_t AudioSinkCallback(MediaPlayerBase::AudioSink *audioSink,            void *data, size_t size, void *me,            MediaPlayerBase::AudioSink::cb_event_t event);    // 缓冲音视频原始数据    void queueBuffer(bool audio,            const sp<ABuffer> &buffer, const sp<AMessage> &notifyConsumed);    void queueEOS(bool audio, status_t finalResult);    status_t setPlaybackSettings(const AudioPlaybackRate &rate /* sanitized */);    status_t getPlaybackSettings(AudioPlaybackRate *rate /* nonnull */);    status_t setSyncSettings(const AVSyncSettings &sync, float videoFpsHint);    status_t getSyncSettings(AVSyncSettings *sync /* nonnull */, float *videoFps /* nonnull */);    void flush(bool audio, bool notifyComplete);    void signalTimeDiscontinuity();    void signalAudioSinkChanged();    void signalDisableOffloadAudio();    void signalEnableOffloadAudio();    void pause();    void resume();    void setVideoFrameRate(float fps);    status_t getCurrentPosition(int64_t *mediaUs);    int64_t getVideoLateByUs();    status_t openAudioSink( const sp<AMessage> &format, bool offloadOnly,         bool hasVideo, uint32_t flags, bool *isOffloaded);    void closeAudioSink();private:    struct QueueEntry {        sp<ABuffer> mBuffer;        sp<AMessage> mNotifyConsumed;        size_t mOffset;        status_t mFinalResult;        int32_t mBufferOrdinal;    };    static const int64_t kMinPositionUpdateDelayUs;    sp<MediaPlayerBase::AudioSink> mAudioSink;    bool mUseVirtualAudioSink;    sp<AMessage> mNotify;    Mutex mLock;    uint32_t mFlags;    List<QueueEntry> mAudioQueue; // 音频缓冲    List<QueueEntry> mVideoQueue; // 视频缓冲    uint32_t mNumFramesWritten;    sp<VideoFrameScheduler> mVideoScheduler;    sp<MediaClock> mMediaClock;    float mPlaybackRate; // audio track rate}

首先看到的是Renderer本身是AHandler的子类。还记得之前的AHandler和ALooper配合使用的机制嘛？其中ALooper位于NuPlayer中，变量名为mRendererLooper。

2 NuPlayer中调用的Renderer接口

先回顾下NuPlayer源码解析中的调用接口。

• 构造/析构函数
• 设置播放控制参数——setPlaybackSettings/getPlaybackSettings/setVideoFrameRate/setSyncSettings/getSyncSettings
• AudioSink相关——openAudioSink/closeAudioSink
• 控制接口——pause/flush/resume/queueEOS
• 音频状态更新——signalEnableOffloadAudio/signalDisableOffloadAudio

• 音视频原始数据输入——queueBuffer
  NuPlayer中并未显示调用，而是将Renderer设置给ADecoder使用

  if (mVideoDecoder != NULL) {mVideoDecoder->setRenderer(mRenderer);}if (mAudioDecoder != NULL) {mAudioDecoder->setRenderer(mRenderer); }

3 Renderer具体接口分析

构造函数喝析构函数

构造函数最主要的是创建一个MediaClock，用于同步和计时。主要代码如下：

    mMediaClock = new MediaClock;    mPlaybackRate = mPlaybackSettings.mSpeed;    mMediaClock->setPlaybackRate(mPlaybackRate);

由于AHandler是智能指针，可以不考虑析构函数。不过可以看下代码中实现：

NuPlayer::Renderer::~Renderer() {    if (offloadingAudio()) {        mAudioSink->stop(); // 主要是针对AudioSink的处理        mAudioSink->flush();        mAudioSink->close();    }}

设置播放控制参数类接口

音频回放参数设置-setPlaybackSettings/getPlaybackSettings

主要接口定义及参数如下：

status_t setPlaybackSettings(const AudioPlaybackRate &rate /* sanitized */);status_t getPlaybackSettings(AudioPlaybackRate *rate /* nonnull */);struct AudioPlaybackRate {    float mSpeed; // 播放倍速    float mPitch; // 声调参数    enum AudioTimestretchStretchMode  mStretchMode; // 拉伸模式    enum AudioTimestretchFallbackMode mFallbackMode; // 备用模式};

从实际接口含义来看主要控制音频播放速率。最终设置函数将参数传递给mMediaClock->setPlaybackRate函数。

视频播放帧率参数-setVideoFrameRate

函数原型如下：void setVideoFrameRate(float fps);。只有一个参数视频播放帧率fps，最终实现函数将该参数设置给mVideoScheduler。实现如下：

void NuPlayer::Renderer::onSetVideoFrameRate(float fps) {    if (mVideoScheduler == NULL) {        mVideoScheduler = new VideoFrameScheduler();    }    mVideoScheduler->init(fps);}

音视频同步参数-setSyncSettings/getSyncSettings

接口声明及主要参数如下：

status_t setSyncSettings(const AVSyncSettings &sync, float videoFpsHint);status_t getSyncSettings(AVSyncSettings *sync /* nonnull */, float *videoFps /* nonnull */);// from ~/frameworks/av/include/media/AVSyncSettings.hstruct AVSyncSettings {    AVSyncSource mSource; // 同步基准    AVSyncAudioAdjustMode mAudioAdjustMode; // 音频调整方式    float mTolerance; // 最大容忍的调速时间    AVSyncSettings()        : mSource(AVSYNC_SOURCE_DEFAULT),          mAudioAdjustMode(AVSYNC_AUDIO_ADJUST_MODE_DEFAULT),          mTolerance(.044f) { }};

看代码实现就会发现，Renderer中并没有实现setSyncSettings，只是判断了必须使用必须使用默认的同步方式，判断逻辑如下：

status_t NuPlayer::Renderer::onConfigSync(const AVSyncSettings &sync, float videoFpsHint __unused) {    if (sync.mSource != AVSYNC_SOURCE_DEFAULT) {        return BAD_VALUE;    }    // TODO: support sync sources    return INVALID_OPERATION;}

至于这里涉及的MediaClock、AudioSink、VideoFrameSchedule后续有专门介绍。

AudioSink相关-openAudioSink/closeAudioSink

主要用于创建和关闭AudioSink，声明如下：

status_t openAudioSink(        const sp<AMessage> &format,        bool offloadOnly,        bool hasVideo,        uint32_t flags,        bool *isOffloaded);void closeAudioSink();

后续会解释两个接口。

控制接口-pause/flush/resume/queueEOS

pause/resume接口

暂停和恢复接口，实现类似，pause接口最终实现是在onPause中：

void NuPlayer::Renderer::onPause() {    if (mPaused) {        return;    }    {        Mutex::Autolock autoLock(mLock);        // we do not increment audio drain generation so that we fill audio buffer during pause.        ++mVideoDrainGeneration;        prepareForMediaRenderingStart_l();        mPaused = true;        mMediaClock->setPlaybackRate(0.0); // 设置成0.0，后面解释为什么    }    mDrainAudioQueuePending = false;    mDrainVideoQueuePending = false;    // Note: audio data may not have been decoded, and the AudioSink may not be opened.    mAudioSink->pause();    startAudioOffloadPauseTimeout();}

其最终通过mMediaClock->setPlaybackRate和mAudioSink->pause接口实现暂停功能。
resume接口最终实现是在onResume中，代码如下：

void NuPlayer::Renderer::onResume() {    if (!mPaused) {        return;    }    // Note: audio data may not have been decoded, and the AudioSink may not be opened.    cancelAudioOffloadPauseTimeout();    if (mAudioSink->ready()) {        status_t err = mAudioSink->start();        if (err != OK) {            ALOGE("cannot start AudioSink err %d", err);            notifyAudioTearDown(kDueToError);        }    }    {        Mutex::Autolock autoLock(mLock);        mPaused = false;        // rendering started message may have been delayed if we were paused.        if (mRenderingDataDelivered) {            notifyIfMediaRenderingStarted_l();        }        // configure audiosink as we did not do it when pausing        if (mAudioSink != NULL && mAudioSink->ready()) {            mAudioSink->setPlaybackRate(mPlaybackSettings);        }        mMediaClock->setPlaybackRate(mPlaybackRate);        if (!mAudioQueue.empty()) {            postDrainAudioQueue_l();        }    }    if (!mVideoQueue.empty()) {        postDrainVideoQueue();    }}

基本上是通过mAudioSink->start()和mMediaClock->setPlaybackRate实现，这过程中也有音视频队列清空的操作。

flush接口

主要分为针对音频的flush和针对视频的flush，具体实现时，音频主要是使用AudioSink的pause/flush/start接口，视频主要是使用清空缓冲队列和mVideoScheduler->restart实现。详细实现建议参考NuPlayer::Renderer::onFlush的代码。

queueEOS

添加流结束标志，最终实现是在onQueueEOS接口中，代码如下：

void NuPlayer::Renderer::onQueueEOS(const sp<AMessage> &msg) {    int32_t audio;    CHECK(msg->findInt32("audio", &audio));    if (dropBufferIfStale(audio, msg)) {        return;    }    int32_t finalResult;    CHECK(msg->findInt32("finalResult", &finalResult));    QueueEntry entry;    entry.mOffset = 0;    entry.mFinalResult = finalResult;    if (audio) { // 音频EOS        Mutex::Autolock autoLock(mLock);        if (mAudioQueue.empty() && mSyncQueues) {            syncQueuesDone_l();        }        mAudioQueue.push_back(entry);        postDrainAudioQueue_l();    } else { // 视频EOS        if (mVideoQueue.empty() && getSyncQueues()) {            Mutex::Autolock autoLock(mLock);            syncQueuesDone_l();        }        mVideoQueue.push_back(entry);        postDrainVideoQueue();    }}

音视频原始数据输入——queueBuffer

在NuPlayer中没看到这个函数调用，但总体来说这个应该由音视频解码器调用，主要将解码之后的音视频原始数据通知显示端并作缓存和同步。主要实现代码如下：（有删减）

void NuPlayer::Renderer::onQueueBuffer(const sp<AMessage> &msg) {    int32_t audio;    CHECK(msg->findInt32("audio", &audio));    if (dropBufferIfStale(audio, msg)) {        return;    }    sp<ABuffer> buffer;    CHECK(msg->findBuffer("buffer", &buffer)); // 传入的数据存储在这里    QueueEntry entry;    entry.mBuffer = buffer;    entry.mNotifyConsumed = notifyConsumed;    entry.mOffset = 0;    entry.mFinalResult = OK;    entry.mBufferOrdinal = ++mTotalBuffersQueued;    // 将数据放到音频或者视频缓冲队列中    if (audio) {        Mutex::Autolock autoLock(mLock);        mAudioQueue.push_back(entry);        postDrainAudioQueue_l();    } else {        mVideoQueue.push_back(entry);        postDrainVideoQueue();    }    // 后续代码是做同步的    Mutex::Autolock autoLock(mLock);    if (!mSyncQueues || mAudioQueue.empty() || mVideoQueue.empty()) {        return;    }    sp<ABuffer> firstAudioBuffer = (*mAudioQueue.begin()).mBuffer;    sp<ABuffer> firstVideoBuffer = (*mVideoQueue.begin()).mBuffer;    if (firstAudioBuffer == NULL || firstVideoBuffer == NULL) {        // EOS signalled on either queue.        syncQueuesDone_l();        return;    }    int64_t firstAudioTimeUs;    int64_t firstVideoTimeUs;    CHECK(firstAudioBuffer->meta()            ->findInt64("timeUs", &firstAudioTimeUs));    CHECK(firstVideoBuffer->meta()            ->findInt64("timeUs", &firstVideoTimeUs));    int64_t diff = firstVideoTimeUs - firstAudioTimeUs;    ALOGV("queueDiff = %.2f secs", diff / 1E6);    if (diff > 100000ll) { //         // Audio data starts More than 0.1 secs before video.        // Drop some audio.        (*mAudioQueue.begin()).mNotifyConsumed->post();        mAudioQueue.erase(mAudioQueue.begin());        return;    }    syncQueuesDone_l();}

4 MediaClock简介

看名字，MediaClock有点时钟同步的感觉，说白了就是一个多媒体时钟，是libstagefright提供的一个公共类。具体接口如下：

struct MediaClock : public RefBase {    MediaClock();    void setStartingTimeMedia(int64_t startingTimeMediaUs);    void clearAnchor();    void updateAnchor( int64_t anchorTimeMediaUs,            int64_t anchorTimeRealUs, int64_t maxTimeMediaUs = INT64_MAX);    void updateMaxTimeMedia(int64_t maxTimeMediaUs);    void setPlaybackRate(float rate);    float getPlaybackRate() const;    // 查询与实际时间|realUs|对应的多媒体时间，并将结果保存在|outMediaUs|中    status_t getMediaTime( int64_t realUs, int64_t *outMediaUs,            bool allowPastMaxTime = false) const;    // query real time corresponding to media time 查询与多媒体时间|targetMediaUs|对应的实际时间，结果保存在|outRealUs|中    status_t getRealTimeFor(int64_t targetMediaUs, int64_t *outRealUs) const;private:    int64_t mAnchorTimeMediaUs;    int64_t mAnchorTimeRealUs;    int64_t mMaxTimeMediaUs;    int64_t mStartingTimeMediaUs;    float mPlaybackRate;};

我把这个类的实现分为两部分，不需要逻辑判断的赋值或返回代码，需要额外计算的代码。先看简单的部分，函数功能主要是赋值和返回参数。

// code from ~/frameworks/av/media/libstagefright/MediaClock.cppMediaClock::MediaClock() : mAnchorTimeMediaUs(-1), mAnchorTimeRealUs(-1),      mMaxTimeMediaUs(INT64_MAX), mStartingTimeMediaUs(-1), mPlaybackRate(1.0) {}MediaClock::~MediaClock() {}void MediaClock::setStartingTimeMedia(int64_t startingTimeMediaUs) {    mStartingTimeMediaUs = startingTimeMediaUs;}void MediaClock::clearAnchor() {    mAnchorTimeMediaUs = -1;    mAnchorTimeRealUs = -1;}void MediaClock::updateMaxTimeMedia(int64_t maxTimeMediaUs) {    mMaxTimeMediaUs = maxTimeMediaUs;}float MediaClock::getPlaybackRate() const {    Mutex::Autolock autoLock(mLock);    return mPlaybackRate;}

这部分代码实现了时钟的主要功能，对多媒体时间和实际时间做了对应关系。（注意代码部分有删减，仅保留核心逻辑）

void MediaClock::updateAnchor(        int64_t anchorTimeMediaUs, // 锚点的播放时间戳        int64_t anchorTimeRealUs, // 锚点的实际时间        int64_t maxTimeMediaUs) {    int64_t nowUs = ALooper::GetNowUs(); // 当前系统时钟    int64_t nowMediaUs = anchorTimeMediaUs + (nowUs - anchorTimeRealUs) * (double)mPlaybackRate; // 转换为当前值，误差低    if (maxTimeMediaUs != -1) {        mMaxTimeMediaUs = maxTimeMediaUs;    }    mAnchorTimeRealUs = nowUs;    mAnchorTimeMediaUs = nowMediaUs;}void MediaClock::setPlaybackRate(float rate) {    CHECK_GE(rate, 0.0);    if (mAnchorTimeRealUs == -1) {        mPlaybackRate = rate;        return;    }    int64_t nowUs = ALooper::GetNowUs();    mAnchorTimeMediaUs += (nowUs - mAnchorTimeRealUs) * (double)mPlaybackRate;    mAnchorTimeRealUs = nowUs;    mPlaybackRate = rate;}// 以下两个函数完成MediaTime <-->realTime的映射，具体原理还是来自updateAnchorstatus_t MediaClock::getMediaTime(int64_t realUs, int64_t *outMediaUs, bool allowPastMaxTime) const {    return getMediaTime_l(realUs, outMediaUs, allowPastMaxTime);}status_t MediaClock::getMediaTime_l(int64_t realUs, int64_t *outMediaUs, bool allowPastMaxTime) const {    if (mAnchorTimeRealUs == -1) {        return NO_INIT;    }    int64_t mediaUs = mAnchorTimeMediaUs            + (realUs - mAnchorTimeRealUs) * (double)mPlaybackRate;    if (mediaUs > mMaxTimeMediaUs && !allowPastMaxTime) {        mediaUs = mMaxTimeMediaUs;    }    if (mediaUs < mStartingTimeMediaUs) {        mediaUs = mStartingTimeMediaUs;    }    if (mediaUs < 0) {        mediaUs = 0;    }    *outMediaUs = mediaUs;    return OK;}status_t MediaClock::getRealTimeFor(int64_t targetMediaUs, int64_t *outRealUs) const {    if (outRealUs == NULL) {        return BAD_VALUE;    }    if (mPlaybackRate == 0.0) {        return NO_INIT;    }    int64_t nowUs = ALooper::GetNowUs();    int64_t nowMediaUs;    status_t status =            getMediaTime_l(nowUs, &nowMediaUs, true /* allowPastMaxTime */);    if (status != OK) {        return status;    }    *outRealUs = (targetMediaUs - nowMediaUs) / (double)mPlaybackRate + nowUs;    return OK;}

还记得在前面解释Renderer::pause实现的时候把mPlaybackRate设置成0嘛，看到上面的计算代码基本上就可以明白了。
比较有意思的是针对mPlaybackRate的处理及Renderer调用的逻辑。下面是获得当前播放位置的函数实现

status_t NuPlayer::Renderer::getCurrentPosition(int64_t *mediaUs) {    // 注意是直接调用的MediaClock::getMediaTime()    status_t result = mMediaClock->getMediaTime(ALooper::GetNowUs(), mediaUs);    if (result == OK) {        return result;    }    // MediaClock未初始化，尝试初始化之    {        AudioTimestamp ts;// 另一种时钟计算方法        status_t res = mAudioSink->getTimestamp(ts);        if (res != OK) {            return result;        }        // AudioSink has rendered some frames.        int64_t nowUs = ALooper::GetNowUs();        int64_t nowMediaUs = mAudioSink->getPlayedOutDurationUs(nowUs)                + mAudioFirstAnchorTimeMediaUs;        mMediaClock->updateAnchor(nowMediaUs, nowUs, -1);    }    return mMediaClock->getMediaTime(ALooper::GetNowUs(), mediaUs);}

到这里基本解释清楚MediaClock是做什么的，但是疑问还在，音视频同步在哪里，怎么做到的？

5 AudioSink简介

以下资料来在Google group，内容如下：

AudioTrack is the hardware audio sink. AudioSink is used for in-memory
decode and potentially other applications where output doesn't go
straight to hardware.

翻译过来就是AudioTrack是一种特殊的AudioSink，与硬件对应；而AudioSink是用于内存解码的，所得数据不直接输出到音频设备上。
在之前文章MediaPlayer Interface&State中可以看到MediaPlayerBase里面有一个抽象类定义，AudioSink。下面是具体的接口：

class AudioSink : public RefBase {public:    enum cb_event_t {        CB_EVENT_FILL_BUFFER,   // Request to write more data to buffer.        CB_EVENT_STREAM_END,    // Sent after all the buffers queued in AF and HW are played                                // back (after stop is called)        CB_EVENT_TEAR_DOWN      // The AudioTrack was invalidated due to use case change:                                // Need to re-evaluate offloading options    };    // Callback returns the number of bytes actually written to the buffer.    typedef size_t (*AudioCallback)(            AudioSink *audioSink, void *buffer, size_t size, void *cookie, cb_event_t event);    virtual             ~AudioSink() {}    virtual bool        ready() const = 0; // audio output is open and ready    virtual ssize_t     bufferSize() const = 0;    virtual ssize_t     frameCount() const = 0;    virtual ssize_t     channelCount() const = 0;    virtual ssize_t     frameSize() const = 0;    virtual uint32_t    latency() const = 0;    virtual float       msecsPerFrame() const = 0;    virtual status_t    getPosition(uint32_t *position) const = 0;    virtual status_t    getTimestamp(AudioTimestamp &ts) const = 0;    virtual int64_t     getPlayedOutDurationUs(int64_t nowUs) const = 0;    virtual status_t    getFramesWritten(uint32_t *frameswritten) const = 0;    virtual audio_session_t getSessionId() const = 0;    virtual audio_stream_type_t getAudioStreamType() const = 0;    virtual uint32_t    getSampleRate() const = 0;    virtual int64_t     getBufferDurationInUs() const = 0;    // If no callback is specified, use the "write" API below to submit audio data.    virtual status_t    open(            uint32_t sampleRate, int channelCount, audio_channel_mask_t channelMask,            audio_format_t format=AUDIO_FORMAT_PCM_16_BIT,            int bufferCount=DEFAULT_AUDIOSINK_BUFFERCOUNT,            AudioCallback cb = NULL,            void *cookie = NULL,            audio_output_flags_t flags = AUDIO_OUTPUT_FLAG_NONE,            const audio_offload_info_t *offloadInfo = NULL,            bool doNotReconnect = false,            uint32_t suggestedFrameCount = 0) = 0;    virtual status_t    start() = 0;    /* Input parameter |size| is in byte units stored in |buffer|.     * Data is copied over and actual number of bytes written (>= 0)     * is returned, or no data is copied and a negative status code     * is returned (even when |blocking| is true).     * When |blocking| is false, AudioSink will immediately return after     * part of or full |buffer| is copied over.     * When |blocking| is true, AudioSink will wait to copy the entire     * buffer, unless an error occurs or the copy operation is     * prematurely stopped.     */    virtual ssize_t     write(const void* buffer, size_t size, bool blocking = true) = 0;    virtual void        stop() = 0;    virtual void        flush() = 0;    virtual void        pause() = 0;    virtual void        close() = 0;    virtual status_t    setPlaybackRate(const AudioPlaybackRate& rate) = 0;    virtual status_t    getPlaybackRate(AudioPlaybackRate* rate /* nonnull */) = 0;    virtual bool        needsTrailingPadding() { return true; }    virtual status_t    setParameters(const String8& /* keyValuePairs */) { return NO_ERROR; }    virtual String8     getParameters(const String8& /* keys */) { return String8::empty(); }};

在Renderer的构造函数中可以看到AudioSink是由NuPlayer传递过来的。明显的这仅仅是通过抽象实现了在Renderer中操作AudioSink及其子类的逻辑。当然在实际使用中，AudioSink也可以作为播放时间的参考，比如上面的getCurrentPosition的实现。这里面的open/close/start/stop/flush/pause/write接口均在Renderer中调用过，后续针对同步的解释会详细说明的。

6 VideoFrameScheduler简介

看名字，感觉这个功能跟MediaClock类似，只是专门针对视频帧的处理逻辑，这也是libstagefright提供的一个公共类，实际上是做视频渲染调整的，以保证视频渲染时间在VSYNC时间之后，防止出现画面撕裂的情况。其对外接口如下：

struct VideoFrameScheduler : public RefBase {    VideoFrameScheduler();    // (re)initialize scheduler 初始化，给定帧率    void init(float videoFps = -1);    // 仅在视频渲染时间不连续的情况下使用，比如seek    void restart();    // 通过renderTime计算视频帧的调整时间（单位纳秒）    nsecs_t schedule(nsecs_t renderTime);    // 返回主屏的垂直同步间隔    nsecs_t getVsyncPeriod();    // 返回帧率    float getFrameRate();    void release();}

内部实现我就不做解释了，基本意思还是从Renderer的调用中说起。Renderer中主要调用了VideoFrameScheduler的以下接口：

mVideoScheduler = new VideoFrameScheduler();mVideoScheduler->init(fps);mVideoScheduler->restart(); // 以下调用都在postDrainVideoQueue中realTimeUs = mVideoScheduler->schedule(realTimeUs * 1000) / 1000;int64_t twoVsyncsUs = 2 * (mVideoScheduler->getVsyncPeriod() / 1000);

7 音视频同步时如何实现的？

从Renderer接口层来看，没有任何关于同步处理的接口，仅有有限的几个控制接口flush/pause/resume，以及queueBuffer/queueEOS接口。同步问题的核心就在于ALooper-AHandler机制。其实真正的同步都是在消息循环的响应函数里实现的。先看音频。

Renderer中的音频同步机制

起始位置从音频PCM数据进入开始，处理在Renderer::queueBuffer()中，最终发送了kWhatQueueBuffer消息。这个消息的实际处理函数是Renderer::onQueueBuffer()。实际代码在“音视频原始数据输入——queueBuffer”中有，这里仅针对音频流程解释下。 基本逻辑很简单，保存传入的buffer参数，并通知输出下AudioQueue。

QueueEntry entry; Mutex::Autolock autoLock(mLock);mAudioQueue.push_back(entry);postDrainAudioQueue_l();

下面看看postDrainAudioQueue_l的实现，内部实现逻辑基本上就是边界判断加上发送kWhatDrainAudioQueue消息。

void NuPlayer::Renderer::postDrainAudioQueue_l(int64_t delayUs) {    if (mAudioQueue.empty()) return;    mDrainAudioQueuePending = true;    sp<AMessage> msg = new AMessage(kWhatDrainAudioQueue, this);    msg->setInt32("drainGeneration", mAudioDrainGeneration);    msg->post(delayUs);}

那就继续查看下这个消息如何处理的。

        case kWhatDrainAudioQueue:        {            mDrainAudioQueuePending = false;            if (onDrainAudioQueue()) {                uint32_t numFramesPlayed;                uint32_t numFramesPendingPlayout = mNumFramesWritten - numFramesPlayed;                // 这里是audio sink中缓存了多长的可用于播放的数据                int64_t delayUs = mAudioSink->msecsPerFrame() * numFramesPendingPlayout * 1000ll;                if (mPlaybackRate > 1.0f) {                    delayUs /= mPlaybackRate;                }                // 利用一半的延时来保证下次刷新时间（注意时间上有重叠）                delayUs /= 2;                // 参考buffer大小来估计最大的延时时间                const int64_t maxDrainDelayUs = std::max(                        mAudioSink->getBufferDurationInUs(), (int64_t)500000 /* half second */);                ALOGD_IF(delayUs > maxDrainDelayUs, "postDrainAudioQueue long delay: %lld > %lld",                        (long long)delayUs, (long long)maxDrainDelayUs);                Mutex::Autolock autoLock(mLock);                postDrainAudioQueue_l(delayUs); // 这里同一个消息重发了            }            break;        }

到这里，貌似还是没有同步的机制，不过我们已经知道这个音频播放消息的触发机制了，在queueBuffer和消息处理函数中都会触发，基本上就是定时器。还有最后一个函数onDrainAudioQueue()。下面是代码：

bool NuPlayer::Renderer::onDrainAudioQueue() {    uint32_t numFramesPlayed;    if (mAudioSink->getPosition(&numFramesPlayed) != OK) {              drainAudioQueueUntilLastEOS();        ALOGW("onDrainAudioQueue(): audio sink is not ready");        return false;    }    uint32_t prevFramesWritten = mNumFramesWritten;    while (!mAudioQueue.empty()) {        QueueEntry *entry = &*mAudioQueue.begin();        mLastAudioBufferDrained = entry->mBufferOrdinal;        if (entry->mBuffer == NULL) {            // 删除针对EOS的处理代码                    }        // ignore 0-sized buffer which could be EOS marker with no data        if (entry->mOffset == 0 && entry->mBuffer->size() > 0) {            int64_t mediaTimeUs;            CHECK(entry->mBuffer->meta()->findInt64("timeUs", &mediaTimeUs));            ALOGV("onDrainAudioQueue: rendering audio at media time %.2f secs",                    mediaTimeUs / 1E6);            onNewAudioMediaTime(mediaTimeUs);        }        size_t copy = entry->mBuffer->size() - entry->mOffset;        ssize_t written = mAudioSink->write(entry->mBuffer->data() + entry->mOffset,                                            copy, false /* blocking */);        if (written < 0) {/* ...忽略异常处理部分代码 */}        entry->mOffset += written;        size_t remainder = entry->mBuffer->size() - entry->mOffset;        if ((ssize_t)remainder < mAudioSink->frameSize()) {            if (remainder > 0) {// 这是直接凑成完整的一帧音频                ALOGW("Corrupted audio buffer has fractional frames, discarding %zu bytes.", remainder);                entry->mOffset += remainder;                copy -= remainder;            }            entry->mNotifyConsumed->post();            mAudioQueue.erase(mAudioQueue.begin());            entry = NULL;        }        size_t copiedFrames = written / mAudioSink->frameSize();        mNumFramesWritten += copiedFrames;        {            Mutex::Autolock autoLock(mLock);            int64_t maxTimeMedia;            maxTimeMedia = mAnchorTimeMediaUs +                        (int64_t)(max((long long)mNumFramesWritten - mAnchorNumFramesWritten, 0LL)                                * 1000LL * mAudioSink->msecsPerFrame());            mMediaClock->updateMaxTimeMedia(maxTimeMedia);            notifyIfMediaRenderingStarted_l();        }        if (written != (ssize_t)copy) {            // A short count was received from AudioSink::write()            //            // AudioSink write is called in non-blocking mode.            // It may return with a short count when:            //            // 1) Size to be copied is not a multiple of the frame size. Fractional frames are            //    discarded.            // 2) The data to be copied exceeds the available buffer in AudioSink.            // 3) An error occurs and data has been partially copied to the buffer in AudioSink.            // 4) AudioSink is an AudioCache for data retrieval, and the AudioCache is exceeded.            // (Case 1)            // Must be a multiple of the frame size.  If it is not a multiple of a frame size, it            // needs to fail, as we should not carry over fractional frames between calls.            CHECK_EQ(copy % mAudioSink->frameSize(), 0);            // (Case 2, 3, 4)            // Return early to the caller.            // Beware of calling immediately again as this may busy-loop if you are not careful.            ALOGV("AudioSink write short frame count %zd < %zu", written, copy);            break;        }    }    // calculate whether we need to reschedule another write.    bool reschedule = !mAudioQueue.empty()            && (!mPaused                || prevFramesWritten != mNumFramesWritten); // permit pause to fill buffers    //ALOGD("reschedule:%d  empty:%d  mPaused:%d  prevFramesWritten:%u  mNumFramesWritten:%u",    //        reschedule, mAudioQueue.empty(), mPaused, prevFramesWritten, mNumFramesWritten);    return reschedule;}

这里面比较主要的更新是onNewAudioMediaTime和mNumFramesWritten字段。
剩下的一部分代码是关于异常边界情况下的音视频处理逻辑：

    sp<ABuffer> firstAudioBuffer = (*mAudioQueue.begin()).mBuffer;    sp<ABuffer> firstVideoBuffer = (*mVideoQueue.begin()).mBuffer;    if (firstAudioBuffer == NULL || firstVideoBuffer == NULL) {        // 对于一个队列为空的情况，通知另个一队列EOS        syncQueuesDone_l();        return;    }    int64_t firstAudioTimeUs;    int64_t firstVideoTimeUs;    CHECK(firstAudioBuffer->meta()            ->findInt64("timeUs", &firstAudioTimeUs));    CHECK(firstVideoBuffer->meta()            ->findInt64("timeUs", &firstVideoTimeUs));    int64_t diff = firstVideoTimeUs - firstAudioTimeUs;    if (diff > 100000ll) {        // 音频数据时间戳比视频数据早0.1s，        (*mAudioQueue.begin()).mNotifyConsumed->post();        mAudioQueue.erase(mAudioQueue.begin());        return;    }    syncQueuesDone_l();

Renderer中的视频同步部分

和音频同步类似，入口在在Renderer::queueBuffer()，主要区分在Renderer::onQueueBuffer()中，代码如下：

// 如果是视频，则将数据存放到视频队列，然后安排刷新mVideoQueue.push_back(entry);postDrainVideoQueue();

下面按照之前的思路继续分析，接下来是postDrainVideoQueue实现，主要音视频同步逻辑位于这里。

void NuPlayer::Renderer::postDrainVideoQueue() {    if (mVideoQueue.empty()) {        return;    }    QueueEntry &entry = *mVideoQueue.begin();    sp<AMessage> msg = new AMessage(kWhatDrainVideoQueue, this); //这是实际处理视频缓冲区和显示的消息    msg->setInt32("drainGeneration", getDrainGeneration(false /* audio */));    if (entry.mBuffer == NULL) {        // EOS doesn't carry a timestamp.        msg->post();        mDrainVideoQueuePending = true;        return;    }    bool needRepostDrainVideoQueue = false;    int64_t delayUs;    int64_t nowUs = ALooper::GetNowUs();    int64_t realTimeUs;    int64_t mediaTimeUs;    CHECK(entry.mBuffer->meta()->findInt64("timeUs", &mediaTimeUs));    if (mFlags & FLAG_REAL_TIME) {                realTimeUs = mediaTimeUs;    } else {        {            Mutex::Autolock autoLock(mLock);            if (mAnchorTimeMediaUs < 0) { // 同步基准未设置的情况下，直接显示                mMediaClock->updateAnchor(mediaTimeUs, nowUs, mediaTimeUs);                mAnchorTimeMediaUs = mediaTimeUs;                realTimeUs = nowUs;            } else if (!mVideoSampleReceived) { // 第一帧未显示前，直接显示                // Always render the first video frame.                realTimeUs = nowUs;            } else if (mAudioFirstAnchorTimeMediaUs < 0 // 音频未播放之前，以视频为准                || mMediaClock->getRealTimeFor(mediaTimeUs, &realTimeUs) == OK) {                realTimeUs = getRealTimeUs(mediaTimeUs, nowUs);            } else if (mediaTimeUs - mAudioFirstAnchorTimeMediaUs >= 0) { // 视频超前的情况下，等待                needRepostDrainVideoQueue = true;                 realTimeUs = nowUs;            } else {                realTimeUs = nowUs;            }        }        // Heuristics to handle situation when media time changed without a        // discontinuity. If we have not drained an audio buffer that was        // received after this buffer, repost in 10 msec. Otherwise repost        // in 500 msec.        delayUs = realTimeUs - nowUs;        int64_t postDelayUs = -1;        if (delayUs > 500000) {            postDelayUs = 500000;            if (mHasAudio && (mLastAudioBufferDrained - entry.mBufferOrdinal) <= 0) {                postDelayUs = 10000;            }        } else if (needRepostDrainVideoQueue) {            // CHECK(mPlaybackRate > 0);            // CHECK(mAudioFirstAnchorTimeMediaUs >= 0);            // CHECK(mediaTimeUs - mAudioFirstAnchorTimeMediaUs >= 0);            postDelayUs = mediaTimeUs - mAudioFirstAnchorTimeMediaUs;            postDelayUs /= mPlaybackRate;        }        if (postDelayUs >= 0) {            msg->setWhat(kWhatPostDrainVideoQueue);            msg->post(postDelayUs);            mVideoScheduler->restart();            ALOGI("possible video time jump of %dms or uninitialized media clock, retrying in %dms",                    (int)(delayUs / 1000), (int)(postDelayUs / 1000));            mDrainVideoQueuePending = true;            return;        }    }    realTimeUs = mVideoScheduler->schedule(realTimeUs * 1000) / 1000;    int64_t twoVsyncsUs = 2 * (mVideoScheduler->getVsyncPeriod() / 1000);    delayUs = realTimeUs - nowUs;    // 上面代码的主要目的是计算这个延时    ALOGW_IF(delayUs > 500000, "unusually high delayUs: %" PRId64, delayUs);    // post 2 display refreshes before rendering is due    msg->post(delayUs > twoVsyncsUs ? delayUs - twoVsyncsUs : 0);    mDrainVideoQueuePending = true;}

这里主要的是发送了一个延时消息kWhatDrainVideoQueue，下面是如何处理的代码：

        case kWhatDrainVideoQueue:        {            int32_t generation;            CHECK(msg->findInt32("drainGeneration", &generation));            if (generation != getDrainGeneration(false /* audio */)) {                break;            }            mDrainVideoQueuePending = false;            onDrainVideoQueue();            postDrainVideoQueue(); // 注意这里相当于定时器的实现了            break;        }

直接调用onDrainVideoQueue函数，看看如何实现的：

void NuPlayer::Renderer::onDrainVideoQueue() {    if (mVideoQueue.empty()) {        return;    }    QueueEntry *entry = &*mVideoQueue.begin();    if (entry->mBuffer == NULL) {        // ...省略针对EOS 处理    }    int64_t nowUs = ALooper::GetNowUs();    int64_t realTimeUs;    int64_t mediaTimeUs = -1;    if (mFlags & FLAG_REAL_TIME) {        CHECK(entry->mBuffer->meta()->findInt64("timeUs", &realTimeUs));    } else {        CHECK(entry->mBuffer->meta()->findInt64("timeUs", &mediaTimeUs));        realTimeUs = getRealTimeUs(mediaTimeUs, nowUs);    }    bool tooLate = false;    if (!mPaused) {        setVideoLateByUs(nowUs - realTimeUs);        tooLate = (mVideoLateByUs > 40000);        if (tooLate) {            ALOGV("video late by %lld us (%.2f secs)",                 (long long)mVideoLateByUs, mVideoLateByUs / 1E6);        } else {            int64_t mediaUs = 0;            mMediaClock->getMediaTime(realTimeUs, &mediaUs);            ALOGV("rendering video at media time %.2f secs",                    (mFlags & FLAG_REAL_TIME ? realTimeUs :                    mediaUs) / 1E6);            if (!(mFlags & FLAG_REAL_TIME)                    && mLastAudioMediaTimeUs != -1                    && mediaTimeUs > mLastAudioMediaTimeUs) {                // If audio ends before video, video continues to drive media clock.                // Also smooth out videos >= 10fps.                mMediaClock->updateMaxTimeMedia(mediaTimeUs + 100000);            }        }    } else {        setVideoLateByUs(0);        if (!mVideoSampleReceived && !mHasAudio) {            // This will ensure that the first frame after a flush won't be used as anchor            // when renderer is in paused state, because resume can happen any time after seek.            Mutex::Autolock autoLock(mLock);            clearAnchorTime_l();        }    }    // Always render the first video frame while keeping stats on A/V sync.    if (!mVideoSampleReceived) {        realTimeUs = nowUs;        tooLate = false;    }    entry->mNotifyConsumed->setInt64("timestampNs", realTimeUs * 1000ll); // 上面所有计算的参数在这里使用了    entry->mNotifyConsumed->setInt32("render", !tooLate);    entry->mNotifyConsumed->post(); // 注意这里，实际是向解码器发送消息，用于显示    mVideoQueue.erase(mVideoQueue.begin());    entry = NULL;    mVideoSampleReceived = true;    if (!mPaused) { // 这里是通知NuPlayer层渲染开始        if (!mVideoRenderingStarted) {            mVideoRenderingStarted = true;            notifyVideoRenderingStart();        }        Mutex::Autolock autoLock(mLock);        notifyIfMediaRenderingStarted_l();    }}

到这里，小结下，读完这部分代码发现，NuPlayer::Renderer使用的以视频为基准的同步机制，音频晚了直接丢包，视频需要显示。同步主要位于视频缓冲区处理部分onDrainVideoQueue和音频缓冲区处理部分onDrainVideoQueue中。音视频的渲染都是采用类似定时器的机制，只不过视频显示需要依赖于实际解码器，音频播放需要依赖于AudioSink的接口。

8 总结

本文主要参考NuPlayer::Renderer的代码做的分析，持续时间比较长。我都怀疑自己具体写的对不对。
非常抱歉拖了这么久，文中代码比较多，如果诸位绝对不对胃口可以略过。
怎么说呢？ Renderer涉及部分比较多，包括NuPlayer、AudioSink、MediaClock、VideoScheduler等。细节还是有待分析，不过基本整理情况是什么了。
我到现在才认识到理解和整理出来的差距。还需要多历练下。