深入理解Android音视频同步机制（三）NuPlayer的avsync逻辑

• 深入理解Android音视频同步机制（一）概述
• 深入理解Android音视频同步机制（二）ExoPlayer的avsync逻辑
• 深入理解Android音视频同步机制（三）NuPlayer的avsync逻辑
• 深入理解Android音视频同步机制（四）MediaSync的使用与原理
• 深入理解Android音视频同步机制（五）如何从零开始写一个音视频同步的播放器

对于此前没有看过NuPlayer的朋友，我们在这里先用下面的时序图简单介绍一下NuPlayer在音视频同步这块的基本流程：

图中
NuPlayerDecoder拿到解码后的音视频数据后queueBuffer给NuPlayerRenderer，在NuPlayerRenderer中通过postDrainAudioQueue_l方法调用AudioTrack进行写入，并且获取“Audio当前播放的时间”，可以看到这里也调用了AudioTrack的getTimeStamp和getPosition方法，和ExoPlayer中类似，同时会利用MediaClock类记录一些锚点时间戳变量。NuPlayerRenderer中调用postDrainVideoQueue方法对video数据进行处理，包括计算实际送显时间，利用vsync信号调整送显时间等，这里的调整是利用VideoFrameScheduler类完成的。需要注意的是，实际上NuPlayerRenderer方法中只进行了avsync的调整，真正的播放还要通过onRendereBuffer调用到NuPlayerDecoder中，进而调用MediaCodec的release方法进行播放。

下面我会先简要的介绍NuPlayer avsync逻辑中的关键点，最后再进行详细的代码分析。

Video部分

1、利用pts和系统时间计算realTimeUs（视频帧应该在这个时间点显示）

 NuPlayer::Renderer::postDrainVideoQueue         int64_t nowUs = ALooper::GetNowUs();        BufferItem *bufferItem = &*mBufferItems.begin();        int64_t itemMediaUs = bufferItem->mTimestamp / 1000;       //这里就是调用MediaClock的getRealTimeFor方法，得到“视频帧应该显示的时间”        int64_t itemRealUs = getRealTime(itemMediaUs, nowUs);

realTimeUs = PTS - nowMediaUs + nowUs= PTS - (mAnchorTimeMediaUs + (nowUs - mAnchorTimeRealUs)) + nowUs 

mAnchorTimeMediaUs代表锚点媒体时间戳，可以理解为最开始播放的时候记录下来的第一个媒体时间戳。
mAnchorTimeRealUs代表锚点real系统时间戳。

nowUs - mAnchorTimeRealUs即为从开始播放到现在，系统时间经过了多久。
再加上mAnchorTimeMediaUs，即为“在当前系统时间下，对应的媒体时间戳”
用PTS减去这个时间，表示“还有多久该播放这一帧”
最后再加上一个系统时间，即为这一帧应该显示的时间。

2、利用vsync信号调整realTimeUs

 NuPlayer::Renderer::postDrainVideoQueue     realTimeUs = mVideoScheduler->schedule(realTimeUs * 1000) / 1000;

schedule方法非常复杂，我们难以完全理解，但也看到了计算ns精度的视频帧间隔的代码，这也与exoplayer的做法相同。

3、提前2倍vsync duration进行render

 NuPlayer::Renderer::postDrainVideoQueue     //2倍vsync duration    int64_t twoVsyncsUs = 2 * (mVideoScheduler->getVsyncPeriod() / 1000);    //利用调整后的realTimeUs再计算一次“还有多久该播放这一帧”    delayUs = realTimeUs - nowUs;    // post 2 display refreshes before rendering is due    //如果delayUs大于两倍vsync duration，则延迟到“距离显示时间两倍vsync duration之前的时间点”再发消息进入后面的流程，否则立即走后面的流程    msg->post(delayUs > twoVsyncsUs ? delayUs - twoVsyncsUs : 0);

4、丢帧与送显

 NuPlayer::Renderer::onDrainVideoQueue         //取出pts值        CHECK(entry->mBuffer->meta()->findInt64("timeUs", &mediaTimeUs));        nowUs = ALooper::GetNowUs();        //考虑到中间发消息等等会有耗时，所以这里重新利用pts计算一次realTimeUs        realTimeUs = getRealTimeUs(mediaTimeUs, nowUs);         //如果nowUs>realTimeUs，即代表视频帧来晚了         setVideoLateByUs(nowUs - realTimeUs);         //如果晚了40ms，即认为超过了门限值         tooLate = (mVideoLateByUs > 40000);         //把realTimeUs赋值给timestampNs，通过消息发出去          entry->mNotifyConsumed->setInt64("timestampNs", realTimeUs * 1000ll);

注意，这里我认为有个bug, 大家看这里重新计算了一次realTimeUs，却没有再用mVideoScheduler->schedule进行调整，相当于之前那次调整白白浪费了。

小结

1.计算video送显时间的核心公式如下
realTimeUs = PTS - nowMediaUs + nowUs
= PTS - (mAnchorTimeMediaUs + (nowUs - mAnchorTimeRealUs)) + nowUs

2.比较exoplayer和nuplayer，可以看到相同点包括：
a.都是比较系统时间与视频帧的送显时间来判断要不要丢帧，丢帧门限值固定为40ms；
b.都会在计算送显时间时考虑函数调用与消息传递的耗时；
c.计算送显在计算送显时间时，都利用到了vsync信号来对送显时间进行校准

在相同点之外，也存在着差异：
a.nuplayer会在最开始的时候就先确保音视频保持基本范围的同步
b.nuplayer中会有一个提前两倍vsync时间开始执行releaseOutputbuffer的逻辑，这一点与API注释中的描述一致

Audio部分

1、初始pts的纠正

NuPlayer::Renderer::onQueueBuffer     int64_t diff = firstVideoTimeUs - firstAudioTimeUs;...    if (diff > 100000ll) {        // Audio data starts More than 0.1 secs before video.        // Drop some audio.        // 这里是对音视频的第一个pts做一下纠正，保证一开始两者是同步的，但是这里只是考虑了audio提前的情况，而没有考虑video提前的情况        (*mAudioQueue.begin()).mNotifyConsumed->post();        mAudioQueue.erase(mAudioQueue.begin());        return;    }

2、利用pts更新几个锚点变量

NuPlayer::Renderer::onDrainAudioQueue//pts减去“还没播放的时间”，就是当前已经播放的时间，即playedDuration，将其设置为nowMediaUs    int64_t nowMediaUs = mediaTimeUs - getPendingAudioPlayoutDurationUs(nowUs);

//计算“还没播放的时间”//计算writtenFrames对应的duration   //writtenDuration = writtenFrames/sampleRate   int64_t writtenAudioDurationUs = getDurationUsIfPlayedAtSampleRate(mNumFramesWritten);//用wriitenDuration - playedDuration，即为“还没播出的时长pendingPlayDuration”    return writtenAudioDurationUs - getPlayedOutAudioDurationUs(nowUs);

//计算playedDuration – 使用getTimeStamp方法 status_t res = mAudioSink->getTimestamp(ts);//当前播放的framePosition        numFramesPlayed = ts.mPosition;       //framePosition对应的系统时间        numFramesPlayedAt =            ts.mTime.tv_sec * 1000000LL + ts.mTime.tv_nsec / 1000;int64_t durationUs = getDurationUsIfPlayedAtSampleRate(numFramesPlayed)            + nowUs - numFramesPlayedAt;

这里可以说是avsync的核心逻辑了
来简单说说这几个变量，numFramesPlayed代表“从底层获取到的已播放帧数”，需要注意的是，这个并不一定是当前系统时间下已经播放的实时帧数，而numFramesPlayedAt代表“numFramesPlayed对应的系统时间”，所以
durationUs = numFramesPlayed/sampleRate +nowUs - numFramesPlayedAt才是当前系统时间下已经播放的音频时长

//计算playedDuration – 使用getPosition方法         //与exoplayer中的逻辑一样，如果getTimestamp用不了，再走getposition流程        res = mAudioSink->getPosition(&numFramesPlayed);         numFramesPlayedAt = nowUs;       //当前系统时间加上latency才是真正playedOut的时间，这里取了latency/2，可以看做是一种平均，因为latency方法返回值可能并不准        numFramesPlayedAt += 1000LL * mAudioSink->latency() / 2; int64_t durationUs = getDurationUsIfPlayedAtSampleRate(numFramesPlayed)            + nowUs - numFramesPlayedAt;

//利用当前系统时间，当前播放的媒体时间戳，pts，更新锚点    mMediaClock->updateAnchor(nowMediaUs, nowUs, mediaTimeUs);

void MediaClock::updateAnchor(        int64_t anchorTimeMediaUs,        int64_t anchorTimeRealUs,        int64_t maxTimeMediaUs) {   …    int64_t nowUs = ALooper::GetNowUs();    int64_t nowMediaUs =        anchorTimeMediaUs + (nowUs - anchorTimeRealUs) * (double)mPlaybackRate;    ...    mAnchorTimeRealUs = nowUs;    mAnchorTimeMediaUs = nowMediaUs;}

小结

整个逻辑核心的公式就是如何计算已经播出的audio时长：
durationUs = numFramesPlayed/sampleRate +nowUs - numFramesPlayedAt
与exoplayer一样，可以通过getTimeStamp或者getPosition方法来获取

不同的地方有几点：首先是调用getTimeStamp的间隔不同，exoplayer中是500ms间隔，
而nuplayer中的间隔是pendingPlayedOutDuration/2，没有取定值；
其次是调用getPosition方法时，加上的是latency/2。

至于那些锚点变量的计算，看似复杂，其中心思想也大同小异。

NuPlayer avsync逻辑代码精读

先来看video部分

1.

在下面的方法中我们对outputbuffer进行处理

bool NuPlayer::Decoder::handleAnOutputBuffer(        size_t index,        size_t offset,        size_t size,        int64_t timeUs,//pts        int32_t flags) {sp<ABuffer> buffer;    mCodec->getOutputBuffer(index, &buffer);...    buffer->setRange(offset, size);    buffer->meta()->clear();    buffer->meta()->setInt64("timeUs", timeUs);…sp<AMessage> reply = new AMessage(kWhatRenderBuffer, this);...if (mRenderer != NULL) {        // send the buffer to renderer.        mRenderer->queueBuffer(mIsAudio, buffer, reply);        ...    }    return true;}

pts赋值给ABuffer，ABuffer作为queueBuffer方法的参数传给Renderer，kWhatRenderBuffer也作为消息传给Renderer

void NuPlayer::Renderer::onQueueBuffer(const sp<AMessage> &msg) {    ...    if (mHasVideo) {        if (mVideoScheduler == NULL) {            mVideoScheduler = new VideoFrameScheduler();            //获取vsync duration和vsynctime，可以看到，在NuPlayer的avsync中，也涉及到针对vsync时间的调整            mVideoScheduler->init();        }    }    sp<ABuffer> buffer;    CHECK(msg->findBuffer("buffer", &buffer));...    QueueEntry entry;    entry.mBuffer = buffer;...    if (audio) {        Mutex::Autolock autoLock(mLock);        mAudioQueue.push_back(entry);       //处理音频队列的地方        postDrainAudioQueue_l();    } else {        mVideoQueue.push_back(entry);       //处理视频队列的地方        postDrainVideoQueue();    }    Mutex::Autolock autoLock(mLock);...    sp<ABuffer> firstAudioBuffer = (*mAudioQueue.begin()).mBuffer;    sp<ABuffer> firstVideoBuffer = (*mVideoQueue.begin()).mBuffer;...    int64_t firstAudioTimeUs;    int64_t firstVideoTimeUs;    CHECK(firstAudioBuffer->meta()            ->findInt64("timeUs", &firstAudioTimeUs));    CHECK(firstVideoBuffer->meta()            ->findInt64("timeUs", &firstVideoTimeUs));    int64_t diff = firstVideoTimeUs - firstAudioTimeUs;...    if (diff > 100000ll) {        // Audio data starts More than 0.1 secs before video.        // Drop some audio.        // 这里是对音视频的第一个pts做一下纠正，保证一开始两者是同步的，但是这里只是考虑了audio提前的情况，而没有考虑video提前的情况        (*mAudioQueue.begin()).mNotifyConsumed->post();        mAudioQueue.erase(mAudioQueue.begin());        return;    }...}

在上面的方法中，获取了vsync时间和duration，这表示在后面的同步过程中会涉及到针对vsync信号的调整，同时还保证了音视频从一开始是同步的，下面进入正题

void NuPlayer::Renderer::postDrainVideoQueue() {    ...    QueueEntry &entry = *mVideoQueue.begin();    sp<AMessage> msg = new AMessage(kWhatDrainVideoQueue, this);    ...    int64_t delayUs;    int64_t nowUs = ALooper::GetNowUs();    int64_t realTimeUs;    if (mFlags & FLAG_REAL_TIME) {        ...    } else {        int64_t mediaTimeUs;        //取出视频帧的pts        CHECK(entry.mBuffer->meta()->findInt64("timeUs", &mediaTimeUs));        {            Mutex::Autolock autoLock(mLock);           //realTimeUs的含义是“视频帧应该在这个时间点显示”            if (mAnchorTimeMediaUs < 0) {                mMediaClock->updateAnchor(mediaTimeUs, nowUs, mediaTimeUs);                mAnchorTimeMediaUs = mediaTimeUs;                realTimeUs = nowUs;            } else {               //利用pts和系统时间计算realTimeUs,参见1.1                realTimeUs = getRealTimeUs(mediaTimeUs, nowUs);            }        }       ...        // Heuristics to handle situation when media time changed without a        // discontinuity. If we have not drained an audio buffer that was        // received after this buffer, repost in 10 msec. Otherwise repost        // in 500 msec.        // delayUs代表“还有多久该播放这一帧”        delayUs = realTimeUs - nowUs;        if (delayUs > 500000) {            //这一帧来的太早了，超过了500ms的门限值，则重新loop一次，和exoplayer中来的太早就再等一个10ms循环的机制相同            int64_t postDelayUs = 500000;            if (mHasAudio && (mLastAudioBufferDrained - entry.mBufferOrdinal) <= 0) {                postDelayUs = 10000;            }            msg->setWhat(kWhatPostDrainVideoQueue);            msg->post(postDelayUs);            mVideoScheduler->restart();            ALOGI("possible video time jump of %dms, retrying in %dms",                    (int)(delayUs / 1000), (int)(postDelayUs / 1000));            mDrainVideoQueuePending = true;            return;        }    }    //利用vsync信号调整realTimeUs，详细的调整方法参见1.2    realTimeUs = mVideoScheduler->schedule(realTimeUs * 1000) / 1000;    //2倍vsync duration    int64_t twoVsyncsUs = 2 * (mVideoScheduler->getVsyncPeriod() / 1000);    //利用调整后的realTimeUs再计算一次“还有多久该播放这一帧”    delayUs = realTimeUs - nowUs;    ALOGW_IF(delayUs > 500000, "unusually high delayUs: %" PRId64, delayUs);    // post 2 display refreshes before rendering is due    //如果delayUs大于两倍vsync duration，则延迟到“距离显示时间两倍vsync duration之前的时间点”再发消息进入后面的流程，否则立即走后面的流程    msg->post(delayUs > twoVsyncsUs ? delayUs - twoVsyncsUs : 0);    mDrainVideoQueuePending = true;}

1.1

下面的方法根据audio clock情况，利用视频帧pts和系统时间来计算“视频帧应该显示的时间”，实际上只利用了pts计算，如果计算失败，再返回传进来的系统时间，也就是立即显示的意思

int64_t NuPlayer::Renderer::getRealTimeUs(int64_t mediaTimeUs, int64_t nowUs) {    int64_t realUs;    //这里实际调用的是MediaClock中的方法，见下面分析    if (mMediaClock->getRealTimeFor(mediaTimeUs, &realUs) != OK) {        // If failed to get current position, e.g. due to audio clock is        // not ready, then just play out video immediately without delay.        return nowUs;    }    return realUs;}

总结这两个方法，可以得到如下的公式
realTimeUs = PTS - nowMediaUs + nowUs
= PTS - (mAnchorTimeMediaUs + (nowUs - mAnchorTimeRealUs)) + nowUs
mAnchorTimeMediaUs锚点媒体时间戳，可以理解为最开始播放的时候记录下来的第一个媒体时间戳
mAnchorTimeRealUs锚点real系统时间戳，
nowUs - mAnchorTimeRealUs即为从开始播放到现在，系统时间经过了多久。
这个时间再加上mAnchorTimeMediaUs，即为“在当前系统时间下，对应的媒体时间戳”，
用PTS减去这个时间，表示“还有多久该播放这一帧”。
最后再加上一个系统时间，即为这一帧应该显示的时间。
至于上面两个锚点时间的更新与计算，请见audio部分的分析，也颇为复杂

status_t MediaClock::getRealTimeFor(        int64_t targetMediaUs, int64_t *outRealUs) const {    ...    int64_t nowUs = ALooper::GetNowUs();    int64_t nowMediaUs;    status_t status =            getMediaTime_l(nowUs, &nowMediaUs, true /* allowPastMaxTime */);    ...    *outRealUs = (targetMediaUs - nowMediaUs) / (double)mPlaybackRate + nowUs;    return OK;}status_t MediaClock::getMediaTime_l(        int64_t realUs, int64_t *outMediaUs, bool allowPastMaxTime) const {    ...    int64_t mediaUs = mAnchorTimeMediaUs            + (realUs - mAnchorTimeRealUs) * (double)mPlaybackRate;    //下面是一系列的范围校验,关键的计算其实就是上面一行    if (mediaUs > mMaxTimeMediaUs && !allowPastMaxTime) {        mediaUs = mMaxTimeMediaUs;    }    if (mediaUs < mStartingTimeMediaUs) {        mediaUs = mStartingTimeMediaUs;    }    if (mediaUs < 0) {        mediaUs = 0;    }    *outMediaUs = mediaUs;    return OK;}

1.2

在下面的方法中，利用vsync信号调整视频帧应该显示的时间

nsecs_t VideoFrameScheduler::schedule(nsecs_t renderTime) {    …TODO: 非常复杂，还未理解透彻}

1.3

postDrainVideoQueue方法最后的msg post后走到下面的函数中

void NuPlayer::Renderer::onDrainVideoQueue() {    ...    QueueEntry *entry = &*mVideoQueue.begin();...    int64_t nowUs = -1;    int64_t realTimeUs;    if (mFlags & FLAG_REAL_TIME) {        ...    } else {        int64_t mediaTimeUs;       //取出pts值        CHECK(entry->mBuffer->meta()->findInt64("timeUs", &mediaTimeUs));        nowUs = ALooper::GetNowUs();        //考虑到中间发消息等等会有耗时，所以这里重新利用pts计算一次realTimeUs        realTimeUs = getRealTimeUs(mediaTimeUs, nowUs);    }    bool tooLate = false;    if (!mPaused) {        if (nowUs == -1) {            nowUs = ALooper::GetNowUs();        }        //如果nowUs>realTimeUs，即代表视频帧来晚了        setVideoLateByUs(nowUs - realTimeUs);       //如果晚了40ms，即认为超过了门限值        tooLate = (mVideoLateByUs > 40000);        if (tooLate) {            ALOGV("video late by %lld us (%.2f secs)",                 (long long)mVideoLateByUs, mVideoLateByUs / 1E6);        } else {            int64_t mediaUs = 0;            //realTimeUs是系统时间戳，转换为媒体时间戳，其实这里就是打印log用一下，这个mediaUs并没什么实际卵用            mMediaClock->getMediaTime(realTimeUs, &mediaUs);            ALOGV("rendering video at media time %.2f secs",                    (mFlags & FLAG_REAL_TIME ? realTimeUs :                    mediaUs) / 1E6);        }    } else {        ...    }    //把realTimeUs赋值给timestampNs，通过消息发出去   //TODO：这里为什么不再用videoFrameScheduler校正一下？？？不就相当于之前白校正了吗？    entry->mNotifyConsumed->setInt64("timestampNs", realTimeUs * 1000ll);    entry->mNotifyConsumed->setInt32("render", !tooLate);    entry->mNotifyConsumed->post();    mVideoQueue.erase(mVideoQueue.begin());    entry = NULL;    mVideoSampleReceived = true;    if (!mPaused) {        if (!mVideoRenderingStarted) {            mVideoRenderingStarted = true;            notifyVideoRenderingStart();        }        Mutex::Autolock autoLock(mLock);        notifyIfMediaRenderingStarted_l();    }}

回到Decoder中，收到kWhatRenderBuffer消息，调用下面的方法

void NuPlayer::Decoder::onRenderBuffer(const sp<AMessage> &msg) {    status_t err;    int32_t render;    size_t bufferIx;    int32_t eos;    CHECK(msg->findSize("buffer-ix", &bufferIx));    if (!mIsAudio) {        int64_t timeUs;        sp<ABuffer> buffer = mOutputBuffers[bufferIx];       //这个是pts        buffer->meta()->findInt64("timeUs", &timeUs);...    }    if (msg->findInt32("render", &render) && render) {        int64_t timestampNs;       //取出realTimeUs        CHECK(msg->findInt64("timestampNs", &timestampNs));       //下面对应的就是MediaCodec的releaseOutputBuffer方法        err = mCodec->renderOutputBufferAndRelease(bufferIx, timestampNs);    } else {        mNumOutputFramesDropped += !mIsAudio;        err = mCodec->releaseOutputBuffer(bufferIx);    }    ...}

下面来看nuplayer中audio部分的avsync逻辑，播放器不会调整audio的播放时间，我们关注的是如何获取当前播放的时间，即getCurrentPosition

2.

和video类似，处理audio数据的入口在postDrainAudioQueue_l
所不同的是，方法的入参有一个delayUs，说明audio处理的循环中间是有一定间隔的，并非不停地运转loop

void NuPlayer::Renderer::postDrainAudioQueue_l(int64_t delayUs) {    ...    mDrainAudioQueuePending = true;    sp<AMessage> msg = new AMessage(kWhatDrainAudioQueue, this);    msg->setInt32("drainGeneration", mAudioDrainGeneration);    msg->post(delayUs);}

这个delayUs是在下面传进来的

case kWhatDrainAudioQueue:        {            …            //onDrainAudioQueue方法中，如果最后发现audioQueue中还有数据，则会返回true，表示需要继续写入数据，详细的分析见下面            if (onDrainAudioQueue()) {               //通过getPosition方法获取已经播放的framePosition                uint32_t numFramesPlayed;                CHECK_EQ(mAudioSink->getPosition(&numFramesPlayed),                         (status_t)OK);              //写入的帧数writtenFrames减去已经播放的帧数即为还没有播放的帧数                uint32_t numFramesPendingPlayout =                    mNumFramesWritten - numFramesPlayed;                //将numFramesPendingPlayout转换为时间单位，设为delayUs                // This is how long the audio sink will have data to                // play back.                int64_t delayUs =                    mAudioSink->msecsPerFrame()                        * numFramesPendingPlayout * 1000ll;                if (mPlaybackRate > 1.0f) {                    delayUs /= mPlaybackRate;                }               //我们调用onDrainAudioQueue的间隔即为delayUs/2               //这样的话就和exoplayer类似了，exoplayer中写入audio数据的间隔是10ms，同时还有一个500ms的调用getTimeStamp间隔，在nuplayer中这个delayUs/2起到了相同的作用                // Let's give it more data after about half that time                // has elapsed.                Mutex::Autolock autoLock(mLock);                postDrainAudioQueue_l(delayUs / 2);            }            break;        }

对应onDrainVideoQueue的逻辑如下

bool NuPlayer::Renderer::onDrainAudioQueue() {    // TODO: This call to getPosition checks if AudioTrack has been created    // in AudioSink before draining audio. If AudioTrack doesn't exist, then    // CHECKs on getPosition will fail.    uint32_t numFramesPlayed;    if (mAudioSink->getPosition(&numFramesPlayed) != OK) {        ...        ALOGW("onDrainAudioQueue(): audio sink is not ready");        return false;    }    uint32_t prevFramesWritten = mNumFramesWritten;    //循环处理audioQueue中的音频数据    while (!mAudioQueue.empty()) {        QueueEntry *entry = &*mAudioQueue.begin();        mLastAudioBufferDrained = entry->mBufferOrdinal;        if (entry->mBuffer == NULL) {            // EOS            ...            return false;        }        // ignore 0-sized buffer which could be EOS marker with no data       //第一次drain audio        if (entry->mOffset == 0 && entry->mBuffer->size() > 0) {            int64_t mediaTimeUs;            //取pts            CHECK(entry->mBuffer->meta()->findInt64("timeUs", &mediaTimeUs));            ALOGV("onDrainAudioQueue: rendering audio at media time %.2f secs",                    mediaTimeUs / 1E6);            //利用pts初始化几个锚点变量，详见2.1            onNewAudioMediaTime(mediaTimeUs);        }        //copy表示buffer还剩多少音频数据        size_t copy = entry->mBuffer->size() - entry->mOffset;        //written返回的是实际写入的数据量，我们传入的目标量是copy值        ssize_t written = mAudioSink->write(entry->mBuffer->data() + entry->mOffset,                                            copy, false /* blocking */);        if (written < 0) {            // An error in AudioSink write. Perhaps the AudioSink was not properly opened.            ...            break;        }       //更新offset        entry->mOffset += written;        if (entry->mOffset == entry->mBuffer->size()) {            //当前buffer已经全部写入了，送给decoder的onRenderBuffer方法            entry->mNotifyConsumed->post();            mAudioQueue.erase(mAudioQueue.begin());            entry = NULL;        }        //更新mNumFramesWritten        size_t copiedFrames = written / mAudioSink->frameSize();        mNumFramesWritten += copiedFrames;...        if (written != (ssize_t)copy) {            // A short count was received from AudioSink::write()            //            // AudioSink write is called in non-blocking mode.            // It may return with a short count when:            //            // 1) Size to be copied is not a multiple of the frame size. We consider this fatal.            // 2) The data to be copied exceeds the available buffer in AudioSink.            // 3) An error occurs and data has been partially copied to the buffer in AudioSink.            // 4) AudioSink is an AudioCache for data retrieval, and the AudioCache is exceeded.            // (Case 1)            // Must be a multiple of the frame size.  If it is not a multiple of a frame size, it            // needs to fail, as we should not carry over fractional frames between calls.            CHECK_EQ(copy % mAudioSink->frameSize(), 0);            // (Case 2, 3, 4)            // Return early to the caller.            // Beware of calling immediately again as this may busy-loop if you are not careful.            ALOGV("AudioSink write short frame count %zd < %zu", written, copy);            break;        }    }    int64_t maxTimeMedia;    {        Mutex::Autolock autoLock(mLock);       //锚点媒体时间戳加上新写入帧数对应的时长，即为媒体时间戳最大值        maxTimeMedia =            mAnchorTimeMediaUs +                    (int64_t)(max((long long)mNumFramesWritten - mAnchorNumFramesWritten, 0LL)                            * 1000LL * mAudioSink->msecsPerFrame());    }   //更新MediaClock中的maxTimeMedia    mMediaClock->updateMaxTimeMedia(maxTimeMedia);    // calculate whether we need to reschedule another write.    //还有数据，继续循环准备下一次写入    bool reschedule = !mAudioQueue.empty()            && (!mPaused                || prevFramesWritten != mNumFramesWritten); // permit pause to fill buffers    //ALOGD("reschedule:%d  empty:%d  mPaused:%d  prevFramesWritten:%u  mNumFramesWritten:%u",    //        reschedule, mAudioQueue.empty(), mPaused, prevFramesWritten, mNumFramesWritten);    return reschedule;}

2.1

在下面的方法中对锚点变量进行初始化，传入参数为pts
回顾一下，mAnchorTimeMediaUs为锚点媒体时间戳，可以理解为最开始播放的时候记录下来的第一个媒体时间戳，mAnchorTimeRealUs为锚点real系统时间戳

void NuPlayer::Renderer::onNewAudioMediaTime(int64_t mediaTimeUs) {    Mutex::Autolock autoLock(mLock);    …    //用“初始audio pts”设置“初始锚点媒体时间戳”，参见2.1.1    setAudioFirstAnchorTimeIfNeeded_l(mediaTimeUs);    int64_t nowUs = ALooper::GetNowUs();    //pts减去还没播放的时间，其实就是当前已经播放的时间，即playedDuration，将其设置为nowMediaUs,关于如何计算”还没播放的时间”，参见2.1.2    int64_t nowMediaUs = mediaTimeUs - getPendingAudioPlayoutDurationUs(nowUs);    //利用当前系统时间，当前播放的媒体时间戳，pts，更新锚点，参见2.1.3    mMediaClock->updateAnchor(nowMediaUs, nowUs, mediaTimeUs);    //“锚点写入帧数量”初始化为0    mAnchorNumFramesWritten = mNumFramesWritten;   //将锚点媒体时间戳设置为初始audio pts    mAnchorTimeMediaUs = mediaTimeUs;}

2.1.1

在下面两个方法中利用初始audio pts设置初始锚点媒体时间戳

void NuPlayer::Renderer::setAudioFirstAnchorTimeIfNeeded_l(int64_t mediaUs) {    if (mAudioFirstAnchorTimeMediaUs == -1) {        mAudioFirstAnchorTimeMediaUs = mediaUs;        mMediaClock->setStartingTimeMedia(mediaUs);    }}void MediaClock::setStartingTimeMedia(int64_t startingTimeMediaUs) {    Mutex::Autolock autoLock(mLock);    mStartingTimeMediaUs = startingTimeMediaUs;}

2.1.2

在这里我们要先明确，音频存在着两个核心变量，一个是writtenFrames，代表写入audioTrack的帧数，一个是playedFramed，代表已经播出的帧数
在下面的方法中，计算的是音频“还没播出的时长”

// Calculate duration of pending samples if played at normal rate (i.e., 1.0).int64_t NuPlayer::Renderer::getPendingAudioPlayoutDurationUs(int64_t nowUs) {   //计算writtenFrames对应的duration   //writtenDuration = writtenFrames/sampleRate   int64_t writtenAudioDurationUs = getDurationUsIfPlayedAtSampleRate(mNumFramesWritten);//用wriitenDuration - playedDuration，即为“还没播出的时长pendingPlayDuration”，关于playedDuration的计算，参见2.1.2.1    return writtenAudioDurationUs - getPlayedOutAudioDurationUs(nowUs);}int64_t NuPlayer::Renderer::getDurationUsIfPlayedAtSampleRate(uint32_t numFrames) {    int32_t sampleRate = offloadingAudio() ?            mCurrentOffloadInfo.sample_rate : mCurrentPcmInfo.mSampleRate;    return (int64_t)((int32_t)numFrames * 1000000LL / sampleRate);}

2.1.2.1

在下面的方法中计算playedDuration，传入参数为系统时间

// Calculate duration of played samples if played at normal rate (i.e., 1.0).int64_t NuPlayer::Renderer::getPlayedOutAudioDurationUs(int64_t nowUs) {    uint32_t numFramesPlayed;    int64_t numFramesPlayedAt;    AudioTimestamp ts;    static const int64_t kStaleTimestamp100ms = 100000;    //对应java api中的AudioTrack.getTimeStamp    status_t res = mAudioSink->getTimestamp(ts);    if (res == OK) {                 // case 1: mixing audio tracks and offloaded tracks.        //当前播放的framePosition        numFramesPlayed = ts.mPosition;       //framePosition对应的系统时间        numFramesPlayedAt =            ts.mTime.tv_sec * 1000000LL + ts.mTime.tv_nsec / 1000;        //这个时间与系统时间不应该差的太多，阈值默认为100ms        const int64_t timestampAge = nowUs - numFramesPlayedAt;        if (timestampAge > kStaleTimestamp100ms) {            ALOGV("getTimestamp: returned stale timestamp nowUs(%lld) numFramesPlayedAt(%lld)",                    (long long)nowUs, (long long)numFramesPlayedAt);            numFramesPlayedAt = nowUs - kStaleTimestamp100ms;        }        //ALOGD("getTimestamp: OK %d %lld", numFramesPlayed, (long long)numFramesPlayedAt);    } else if (res == WOULD_BLOCK) { // case 2: transitory state on start of a new track        numFramesPlayed = 0;        numFramesPlayedAt = nowUs;        //ALOGD("getTimestamp: WOULD_BLOCK %d %lld",        //        numFramesPlayed, (long long)numFramesPlayedAt);    } else {                         // case 3: transitory at new track or audio fast tracks.       //与exoplayer中的逻辑一样，如果getTimestamp用不了，再走getposition流程        res = mAudioSink->getPosition(&numFramesPlayed);        CHECK_EQ(res, (status_t)OK);        numFramesPlayedAt = nowUs;       //当前系统时间加上latency才是真正playedOut的时间，这里取了latency/2，可以看做是一种平均，因为latency方法返回值可能并不准        numFramesPlayedAt += 1000LL * mAudioSink->latency() / 2; /* XXX */        //ALOGD("getPosition: %u %lld", numFramesPlayed, (long long)numFramesPlayedAt);    }    //来简单说说这几个变量，numFramesPlayed代表“从底层获取到的已播放帧数”，需要注意的是，这个并不一定是当前系统时间下已经播放的实时帧数，而numFramesPlayedAt代表“numFramesPlayed对应的系统时间”，所以durationUs = numFramesPlayed/sampleRate +nowUs - numFramesPlayedAt才是当前系统时间下已经播放的音频时长    int64_t durationUs = getDurationUsIfPlayedAtSampleRate(numFramesPlayed)            + nowUs - numFramesPlayedAt;    if (durationUs < 0) {        // Occurs when numFramesPlayed position is very small and the following:        // (1) In case 1, the time nowUs is computed before getTimestamp() is called and        //     numFramesPlayedAt is greater than nowUs by time more than numFramesPlayed.        // (2) In case 3, using getPosition and adding mAudioSink->latency() to        //     numFramesPlayedAt, by a time amount greater than numFramesPlayed.        //        // Both of these are transitory conditions.        ALOGV("getPlayedOutAudioDurationUs: negative duration %lld set to zero", (long long)durationUs);        durationUs = 0;    }    ALOGV("getPlayedOutAudioDurationUs(%lld) nowUs(%lld) frames(%u) framesAt(%lld)",            (long long)durationUs, (long long)nowUs, numFramesPlayed, (long long)numFramesPlayedAt);    return durationUs;}

2.1.3

在下面的方法中更新锚点,可以看到这里的逻辑和1.1中的完全一样，传进来的第三个参数是pts

void MediaClock::updateAnchor(        int64_t anchorTimeMediaUs,        int64_t anchorTimeRealUs,        int64_t maxTimeMediaUs) {   …    int64_t nowUs = ALooper::GetNowUs();    int64_t nowMediaUs =        anchorTimeMediaUs + (nowUs - anchorTimeRealUs) * (double)mPlaybackRate;    ...    mAnchorTimeRealUs = nowUs;    mAnchorTimeMediaUs = nowMediaUs;    mMaxTimeMediaUs = maxTimeMediaUs;}

2.2

在下面的方法中调用releaseOutputBuffer播放音频

void NuPlayer::Decoder::onRenderBuffer(const sp<AMessage> &msg) {    status_t err;    int32_t render;    size_t bufferIx;    int32_t eos;    CHECK(msg->findSize("buffer-ix", &bufferIx));   ...    if (msg->findInt32("render", &render) && render) {        //audio不会走这个分支    } else {        mNumOutputFramesDropped += !mIsAudio;       //对应mediacodec的releaseOutputBuffer方法        err = mCodec->releaseOutputBuffer(bufferIx);    }   ..}

各位看官，如果您觉得本人的博客对您有所帮助，可以扫描如下二维码进行打赏，打赏多少您随意~