Android N Audio播放五：如何选择Extractor

　　我们在Android N Audio播放三：prepare大揭秘介绍了在prepare的过程中会创建Extractor, Extractor的主要作用是从容器格式中把音频和视频剥离出来，为之后的解码提供音频流和视频流，要知道。音频和视频的解码是分离的， 所以Extractor这一步非常重要。

1. Extractor流程图
　　如惯例，我们还是先上流程图，对这个过程涉及到的类有个大概的了解。
　　

２. 流程详解
　　接下来我们就按照流程图中的步骤一步一步来细看。
　　
　　２.1 GenericSource
　　在prepare章节介绍过，在GenericSource的onPrepareAsync中会去创建Extractor.
　　./frameworks/av/media/libmediaplayerservice/nuplayer/GenericSource.cpp#onPrepareAsync

void NuPlayer::GenericSource::onPrepareAsync() {    // delayed data source creation    if (mDataSource == NULL) {        ALOGE("onPrepareAsync  mDataSource is null");        // set to false first, if the extractor        // comes back as secure, set it to true then.        mIsSecure = false;        .....    //JaychouNote1:创建Extractor    // init extractor from data source    status_t err = initFromDataSource();    if (err != OK) {        ALOGE("Failed to init from data source!");        notifyPreparedAndCleanup(err);        return;    }    ......}

01-02 10:52:29.113 E/GenericSource(  646): onPrepareAsync  mDataSource is null

　　./frameworks/av/media/libmediaplayerservice/nuplayer/GenericSource.cpp#initFromDataSource

status_t NuPlayer::GenericSource::initFromDataSource() {    ALOGE("initFromDataSource");    sp<IMediaExtractor> extractor;    String8 mimeType;    float confidence;    sp<AMessage> dummy;    bool isWidevineStreaming = false;    CHECK(mDataSource != NULL);    ......    //JaychouNote1: 我们这个例子既不是widewine也不是streaming        extractor = MediaExtractor::Create(mDataSource,                mimeType.isEmpty() ? NULL : mimeType.string());    }    if (extractor == NULL) {        return UNKNOWN_ERROR;    }    if (extractor->getDrmFlag()) {        checkDrmStatus(mDataSource);    }    mFileMeta = extractor->getMetaData();    if (mFileMeta != NULL) {        int64_t duration;        if (mFileMeta->findInt64(kKeyDuration, &duration)) {            mDurationUs = duration;        }       ......       //JaychouNote2: 这里的mAudioTrack和mVideoTrack就是剥离出来的音频和视频        // Do the string compare immediately with "mime",        // we can't assume "mime" would stay valid after another        // extractor operation, some extractors might modify meta        // during getTrack() and make it invalid.        if (!strncasecmp(mime, "audio/", 6)) {            if (mAudioTrack.mSource == NULL) {                mAudioTrack.mIndex = i;                mAudioTrack.mSource = track;                mAudioTrack.mPackets =                    new AnotherPacketSource(mAudioTrack.mSource->getFormat());                if (!strcasecmp(mime, MEDIA_MIMETYPE_AUDIO_VORBIS)) {                    mAudioIsVorbis = true;                } else {                    mAudioIsVorbis = false;                }                if (AVNuUtils::get()->isByteStreamModeEnabled(meta)) {                    mIsByteMode = true;                }            }        } else if (!strncasecmp(mime, "video/", 6)) {            if (mVideoTrack.mSource == NULL) {                mVideoTrack.mIndex = i;                mVideoTrack.mSource = track;                mVideoTrack.mPackets =                    new AnotherPacketSource(mVideoTrack.mSource->getFormat());                // check if the source requires secure buffers                int32_t secure;                if (meta->findInt32(kKeyRequiresSecureBuffers, &secure)                        && secure) {                    mIsSecure = true;                    if (mUIDValid) {                        extractor->setUID(mUID);                    }                }            }        }    ......    return OK;}

01-02 10:52:29.115 E/GenericSource(  646): initFromDataSource

这里我们看到做了两件比较重要的事情， 一是根据source去创建extractor, 二是根据创建的extractor拿到剥离出来的音频和视频，分别是mAudioTrack和mVideoTrack。

２.2 MediaExtractor
我们来看Create.
　　./frameworks/av/media/libstagefright/MediaExtractor#Create

sp<IMediaExtractor> MediaExtractor::Create(        const sp<DataSource> &source, const char *mime) {    ALOGV("MediaExtractor::Create %s", mime);        char value[PROPERTY_VALUE_MAX];    if (property_get("media.stagefright.extractremote", value, NULL)            && (!strcmp("0", value) || !strcasecmp("false", value))) {        // local extractor        ALOGW("creating media extractor in calling process");        return CreateFromService(source, mime);    } else {        // Check if it's WVM, since WVMExtractor needs to be created in the media server process,        // not the extractor process.        String8 mime8;        float confidence;        sp<AMessage> meta;        if (SniffWVM(source, &mime8, &confidence, &meta) &&                !strcasecmp(mime8, MEDIA_MIMETYPE_CONTAINER_WVM)) {                ALOGE("Create WVMExtractor");            return new WVMExtractor(source);        }        // Check if it's es-based DRM, since DRMExtractor needs to be created in the media server        // process, not the extractor process.        if (SniffDRM(source, &mime8, &confidence, &meta)) {            const char *drmMime = mime8.string();            ALOGV("Detected media content as '%s' with confidence %.2f", drmMime, confidence);            if (!strncmp(drmMime, "drm+es_based+", 13)) {                // DRMExtractor sets container metadata kKeyIsDRM to 1                return new DRMExtractor(source, drmMime + 14);            }        }    //JaychouNote1:这里既不是WVM也不是DRM，所以直接到这里。        // remote extractor        ALOGV("get service manager");        sp<IBinder> binder = defaultServiceManager()->getService(String16("media.extractor"));        if (binder != 0) {            sp<IMediaExtractorService> mediaExService(interface_cast<IMediaExtractorService>(binder));            sp<IMediaExtractor> ex = mediaExService->makeExtractor(RemoteDataSource::wrap(source), mime);            return ex;        } else {            ALOGE("extractor service not running");            return NULL;        }    }    return NULL;}

01-02 10:52:29.116 V/MediaExtractor(  646): MediaExtractor::Create (null)01-02 10:52:29.126 V/MediaExtractor(  646): get service manager

这里既不是WVM也不是DRM, 可以看到，是通过binder方式调用MediaExtractorService来创建Extractor.

２.3 MediaExtractorService
看看如何创建extractor.

./frameworks/av/services/mediaextractor/MediaExtractorService#makeExtractor

sp<IMediaExtractor> MediaExtractorService::makeExtractor(        const sp<IDataSource> &remoteSource, const char *mime) {    ALOGV("@@@ MediaExtractorService::makeExtractor for %s", mime);    sp<DataSource> localSource = DataSource::CreateFromIDataSource(remoteSource);    //JaychouNote1:又调回MediaExtractor    sp<IMediaExtractor> ret = MediaExtractor::CreateFromService(localSource, mime);    ALOGV("extractor service created %p (%s)",            ret.get(),            ret == NULL ? "" : ret->name());    ALOGV("extractor service  ret->name is %s", ret->name());    if (ret != NULL) {        registerMediaExtractor(ret, localSource, mime);    }    return ret;}

01-02 10:52:29.128 V/MediaExtractorService( 1777): @@@ MediaExtractorService::makeExtractor for (null)

这个…看不太懂为什么这么写，又回到了MediaExtractor，好吧，接着回去。

２.4 MediaExtractor
./frameworks/av/media/libstagefright/MediaExtractor#CreateFromService

sp<MediaExtractor> MediaExtractor::CreateFromService(        const sp<DataSource> &source, const char *mime) {    ALOGV("MediaExtractor::CreateFromService %s", mime);    DataSource::RegisterDefaultSniffers();    sp<AMessage> meta;    String8 tmp;    if (mime == NULL) {        float confidence;        //JaychouNote1:sniff函数算是选择Extractor的精华部分了。        if (!source->sniff(&tmp, &confidence, &meta)) {            ALOGV("FAILED to autodetect media content.");            return NULL;        }        mime = tmp.string();        ALOGV("Autodetected media content as '%s' with confidence %.2f",             mime, confidence);    }    bool isDrm = false;    // DRM MIME type syntax is "drm+type+original" where    // type is "es_based" or "container_based" and    // original is the content's cleartext MIME type    if (!strncmp(mime, "drm+", 4)) {        const char *originalMime = strchr(mime+4, '+');        if (originalMime == NULL) {            // second + not found            return NULL;        }        ++originalMime;        if (!strncmp(mime, "drm+es_based+", 13)) {            // DRMExtractor sets container metadata kKeyIsDRM to 1            return new DRMExtractor(source, originalMime);        } else if (!strncmp(mime, "drm+container_based+", 20)) {            mime = originalMime;            isDrm = true;        } else {            return NULL;        }    }    MediaExtractor *ret = NULL;    /*if ((ret = AVFactory::get()->createExtendedExtractor(source, mime, meta)) != NULL) {    } else*/ if (!strcasecmp(mime, MEDIA_MIMETYPE_CONTAINER_MPEG4)            || !strcasecmp(mime, "audio/mp4"）            || !strcasecmp(mime, "video/mpeg4")            || !strcasecmp(mime, "audio/x-m4a")) {        ret = new MPEG4Extractor(source);    } else if (!strcasecmp(mime, MEDIA_MIMETYPE_AUDIO_MPEG) || !strcasecmp(mime, MEDIA_MIMETYPE_AUDIO_MPEG3)) {        //JaychouNote2:根据探测到的MIME类型，创建相应的Extractor.        ret = new MP3Extractor(source, meta);    } else if (!strcasecmp(mime, MEDIA_MIMETYPE_AUDIO_AMR_NB)            || !strcasecmp(mime, MEDIA_MIMETYPE_AUDIO_AMR_WB)) {        ret = new AMRExtractor(source);    } else if (!strcasecmp(mime, MEDIA_MIMETYPE_AUDIO_FLAC)) {        ret = new FLACExtractor(source);    } else if (!strcasecmp(mime, MEDIA_MIMETYPE_CONTAINER_WAV)) {        ret = new WAVExtractor(source);    } else if (!strcasecmp(mime, MEDIA_MIMETYPE_CONTAINER_OGG)) {        ret = new OggExtractor(source);    } else if (!strcasecmp(mime, MEDIA_MIMETYPE_CONTAINER_MATROSKA)) {        ret = new MatroskaExtractor(source);    } else if (!strcasecmp(mime, MEDIA_MIMETYPE_CONTAINER_MPEG2TS)) {        ret = new MPEG2TSExtractor(source);    } else if (!strcasecmp(mime, MEDIA_MIMETYPE_CONTAINER_WVM) && getuid() == AID_MEDIA) {        // Return now.  WVExtractor should not have the DrmFlag set in the block below.        return new WVMExtractor(source);    } else if (!strcasecmp(mime, MEDIA_MIMETYPE_AUDIO_AAC_ADTS)) {        ret = new AACExtractor(source, meta);    } else if (!strcasecmp(mime, MEDIA_MIMETYPE_CONTAINER_MPEG2PS)) {        ret = new MPEG2PSExtractor(source);    } else if (!strcasecmp(mime, MEDIA_MIMETYPE_AUDIO_MIDI) || !strcasecmp(mime, "audio/sp-midi")|| !strcasecmp(mime, "audio/imelody")) {        ret = new MidiExtractor(source);    }    //ret = AVFactory::get()->updateExtractor(ret, source, mime, meta, flags);    //if (ret != NULL) {       if (isDrm) {           ret->setDrmFlag(true);       } else {           ret->setDrmFlag(false);       }    //}    return ret;}

这里做了两件非常重要的事情，一是通过Sniff函数去探测文件的类型，二是根据文件的类型来创建相应的Extractor.

01-02 10:52:29.130 V/MediaExtractor( 1777): MediaExtractor::CreateFromService (null)

２.5 DataSource
./frameworks/av/media/libstagefright/DataSource#sniff

bool DataSource::sniff(        String8 *mimeType, float *confidence, sp<AMessage> *meta) {    *mimeType = "";    *confidence = 0.0f;    meta->clear();    {        Mutex::Autolock autoLock(gSnifferMutex);        if (!gSniffersRegistered) {            return false;        }    }    for (List<SnifferFunc>::iterator it = gSniffers.begin();         it != gSniffers.end(); ++it) {        String8 newMimeType;        float newConfidence;        sp<AMessage> newMeta;        if ((*it)(this, &newMimeType, &newConfidence, &newMeta)) {            if (newConfidence > *confidence) {                *mimeType = newMimeType;                *confidence = newConfidence;                *meta = newMeta;            }        }    }    return *confidence > 0.0;}

sniff会遍历调用gSniffers里面注册的函数，gSniffers的赋值是在MediaExtractor=>CreateFromService的时候调用DataSource::RegisterDefaultSniffers();
./frameworks/av/media/libstagefright/DataSource#RegisterDefaultSniffers

void DataSource::RegisterDefaultSniffers() {    Mutex::Autolock autoLock(gSnifferMutex);    if (gSniffersRegistered) {        return;    }    RegisterSniffer_l(SniffMPEG4);    RegisterSniffer_l(SniffMatroska);    RegisterSniffer_l(SniffOgg);    RegisterSniffer_l(SniffWAV);    RegisterSniffer_l(SniffFLAC);    RegisterSniffer_l(SniffAMR);    RegisterSniffer_l(SniffMPEG2TS);    RegisterSniffer_l(SniffMP3);    RegisterSniffer_l(SniffAAC);    RegisterSniffer_l(SniffMPEG2PS);    if (getuid() == AID_MEDIA) {        // WVM only in the media server process        RegisterSniffer_l(SniffWVM);    }    RegisterSniffer_l(SniffMidi);    RegisterSniffer_l(AVUtils::get()->getExtendedSniffer());    char value[PROPERTY_VALUE_MAX];    if (property_get("drm.service.enabled", value, NULL)            && (!strcmp(value, "1") || !strcasecmp(value, "true"))) {        RegisterSniffer_l(SniffDRM);    }    gSniffersRegistered = true;}

而
./frameworks/av/media/libstagefright/DataSource#RegisterSniffer_l

void DataSource::RegisterSniffer_l(SnifferFunc func) {    for (List<SnifferFunc>::iterator it = gSniffers.begin();         it != gSniffers.end(); ++it) {        if (*it == func) {            return;        }    }    gSniffers.push_back(func);}

将每一个SniffXXX的函数指针加入到了gSniffers中。我们这里是MP3, 所以来看看SniffMP3.

２.6 MP3Extractor

./frameworks/av/media/libstagefright/MP3Extractor#SniffMP3

bool SniffMP3(        const sp<DataSource> &source, String8 *mimeType,        float *confidence, sp<AMessage> *meta) {    off64_t pos = 0;    off64_t post_id3_pos;    uint32_t header;    if (!Resync(source, 0, &pos, &post_id3_pos, &header)) {        return false;    }    *meta = new AMessage;    (*meta)->setInt64("offset", pos);    (*meta)->setInt32("header", header);    (*meta)->setInt64("post-id3-offset", post_id3_pos);    *mimeType = MEDIA_MIMETYPE_AUDIO_MPEG;    *confidence = 0.2f;    return true;}

主要是调用Resync
./frameworks/av/media/libstagefright/MP3Extractor#Resync

static bool Resync(        const sp<DataSource> &source, uint32_t match_header,        off64_t *inout_pos, off64_t *post_id3_pos, uint32_t *out_header) {    if (post_id3_pos != NULL) {        *post_id3_pos = 0;    }    if (*inout_pos == 0) {        // Skip an optional ID3 header if syncing at the very beginning        // of the datasource.        for (;;) {            uint8_t id3header[10];            //JaychouNote1: 调用FileSource的readAt读取header.            if (source->readAt(*inout_pos, id3header, sizeof(id3header))                    < (ssize_t)sizeof(id3header)) {                // If we can't even read these 10 bytes, we might as well bail                // out, even if there _were_ 10 bytes of valid mp3 audio data...                return false;            }            if (memcmp("ID3", id3header, 3)) {                break;            }            // Skip the ID3v2 header.            size_t len =                ((id3header[6] & 0x7f) << 21)                | ((id3header[7] & 0x7f) << 14)                | ((id3header[8] & 0x7f) << 7)                | (id3header[9] & 0x7f);            len += 10;            *inout_pos += len;            ALOGV("skipped ID3 tag, new starting offset is %lld (0x%016llx)",                    (long long)*inout_pos, (long long)*inout_pos);        }        if (post_id3_pos != NULL) {            *post_id3_pos = *inout_pos;        }    }       .......       //JaychouNote2:找到文件的第1帧.        size_t frame_size;        int sample_rate, num_channels, bitrate;        if (!GetMPEGAudioFrameSize(                    header, &frame_size,                    &sample_rate, &num_channels, &bitrate)) {            ++pos;            ++tmp;            --remainingBytes;            continue;        }        ALOGV("found possible 1st frame at %lld (header = 0x%08x)", (long long)pos, header);        // We found what looks like a valid frame,        // now find its successors.        off64_t test_pos = pos + frame_size;        valid = true;        for (int j = 0; j < 3; ++j) {            uint8_t tmp[4];            if (source->readAt(test_pos, tmp, 4) < 4) {                valid = false;                break;            }            uint32_t test_header = U32_AT(tmp);            ALOGV("subsequent header is %08x", test_header);            if ((test_header & kMask) != (header & kMask)) {                valid = false;                break;            }            //JaychouNote3:找到随后的帧，一般是随后的3帧。            size_t test_frame_size;            if (!GetMPEGAudioFrameSize(                        test_header, &test_frame_size)) {                valid = false;                break;            }            ALOGV("found subsequent frame #%d at %lld", j + 2, (long long)test_pos);            test_pos += test_frame_size;        }        if (valid) {            *inout_pos = pos;            if (out_header != NULL) {                *out_header = header;            }        } else {            ALOGV("no dice, no valid sequence of frames found.");        }        ++pos;        ++tmp;        --remainingBytes;    } while (!valid);    return valid;}

01-02 10:52:29.156 V/FileSource(  646): FileSource readAt mLength is 3661143, mOffset is 001-02 10:52:29.158 V/MP3Extractor( 1777): Resync sizeof id3header[10] is 1001-02 10:52:29.158 V/MP3Extractor( 1777): Resync id3header is ID301-02 10:52:29.159 V/MP3Extractor( 1777): skipped ID3 tag, new starting offset is 111 (0x000000000000006f)01-02 10:52:29.159 V/MP3Extractor( 1777): Resync sizeof id3header[10] is 1001-02 10:52:29.159 V/MP3Extractor( 1777): Resync id3header is \FF\FB\90d01-02 10:52:29.159 V/MP3Extractor( 1777): found possible 1st frame at 111 (header = 0xfffb9064)01-02 10:52:29.160 V/MP3Extractor( 1777): subsequent header is fffb926401-02 10:52:29.160 V/MP3Extractor( 1777): found subsequent frame #2 at 52801-02 10:52:29.160 V/MP3Extractor( 1777): subsequent header is fffb926401-02 10:52:29.161 V/MP3Extractor( 1777): found subsequent frame #3 at 94601-02 10:52:29.161 V/MP3Extractor( 1777): subsequent header is fffb926401-02 10:52:29.161 V/MP3Extractor( 1777): found subsequent frame #4 at 1364

在这个方法中，有三件重要的事情在做。
一是通过FileSource来读取文件的前10个字节来找ID3tag，如果找到，跳过去找第一帧．
二是通过GetMPEGAudioFrameSize试图找到第一帧。
三是通过GetMPEGAudioFrameSize找到随后的帧，一般是随后的3帧。

２.7 Avc_utils
来看看GetMPEGAudioFrameSize

./frameworks/av/media/libstagefright/Avc_utils#GetMPEGAudioFrameSize

bool GetMPEGAudioFrameSize(        uint32_t header, size_t *frame_size,        int *out_sampling_rate, int *out_channels,        int *out_bitrate, int *out_num_samples) {    *frame_size = 0;    if (out_sampling_rate) {        *out_sampling_rate = 0;    }    if (out_channels) {        *out_channels = 0;    }    if (out_bitrate) {        *out_bitrate = 0;    }    if (out_num_samples) {        *out_num_samples = 1152;    }    if ((header & 0xffe00000) != 0xffe00000) {        return false;    }    unsigned version = (header >> 19) & 3;    if (version == 0x01) {        return false;    }    unsigned layer = (header >> 17) & 3;    if (layer == 0x00) {        return false;    }    unsigned protection __unused = (header >> 16) & 1;    unsigned bitrate_index = (header >> 12) & 0x0f;    if (bitrate_index == 0 || bitrate_index == 0x0f) {        // Disallow "free" bitrate.        return false;    }    unsigned sampling_rate_index = (header >> 10) & 3;    if (sampling_rate_index == 3) {        return false;    }    static const int kSamplingRateV1[] = { 44100, 48000, 32000 };    int sampling_rate = kSamplingRateV1[sampling_rate_index];    if (version == 2 /* V2 */) {        sampling_rate /= 2;    } else if (version == 0 /* V2.5 */) {        sampling_rate /= 4;    }    unsigned padding = (header >> 9) & 1;    if (layer == 3) {        // layer I        static const int kBitrateV1[] = {            32, 64, 96, 128, 160, 192, 224, 256,            288, 320, 352, 384, 416, 448        };        static const int kBitrateV2[] = {            32, 48, 56, 64, 80, 96, 112, 128,            144, 160, 176, 192, 224, 256        };        int bitrate =            (version == 3 /* V1 */)                ? kBitrateV1[bitrate_index - 1]                : kBitrateV2[bitrate_index - 1];        if (out_bitrate) {            *out_bitrate = bitrate;        }        *frame_size = (12000 * bitrate / sampling_rate + padding) * 4;        if (out_num_samples) {            *out_num_samples = 384;        }    } else {        // layer II or III        static const int kBitrateV1L2[] = {            32, 48, 56, 64, 80, 96, 112, 128,            160, 192, 224, 256, 320, 384        };        static const int kBitrateV1L3[] = {            32, 40, 48, 56, 64, 80, 96, 112,            128, 160, 192, 224, 256, 320        };        static const int kBitrateV2[] = {            8, 16, 24, 32, 40, 48, 56, 64,            80, 96, 112, 128, 144, 160        };        int bitrate;        if (version == 3 /* V1 */) {            bitrate = (layer == 2 /* L2 */)                ? kBitrateV1L2[bitrate_index - 1]                : kBitrateV1L3[bitrate_index - 1];            if (out_num_samples) {                *out_num_samples = 1152;            }        } else {            // V2 (or 2.5)            bitrate = kBitrateV2[bitrate_index - 1];            if (out_num_samples) {                *out_num_samples = (layer == 1 /* L3 */) ? 576 : 1152;            }        }        if (out_bitrate) {            *out_bitrate = bitrate;        }        if (version == 3 /* V1 */) {            *frame_size = 144000 * bitrate / sampling_rate + padding;        } else {            // V2 or V2.5            size_t tmp = (layer == 1 /* L3 */) ? 72000 : 144000;            *frame_size = tmp * bitrate / sampling_rate + padding;        }    }    if (out_sampling_rate) {        *out_sampling_rate = sampling_rate;    }    if (out_channels) {        int channel_mode = (header >> 6) & 3;        *out_channels = (channel_mode == 3) ? 1 : 2;    }    return true;}

这个方法很复杂，也是探测文件类型最重要的步骤，主要是检测文件的前四帧， 并拿到采样率，声道数，比特率等等参数。

２.8 MediaExtractor
当Sniff函数探测到是MP3时，认为当前的MIME是MEDIA_MIMETYPE_AUDIO_MPEG，然后据此来创建MP3Extractor.

01-02 10:52:29.212 V/MediaExtractor( 1777): Autodetected media content as 'audio/mpeg' with confidence 0.80

./frameworks/av/media/libstagefright/MediaExtractor#CreateFromService

else if (!strcasecmp(mime, MEDIA_MIMETYPE_AUDIO_MPEG) || !strcasecmp(mime, MEDIA_MIMETYPE_AUDIO_MPEG3)) {        ret = new MP3Extractor(source, meta);}

创建了extractor以后，就可以根据它来剥离音频和视频了。为之后的解码做准备。

3. 总结
　　创建Extractor的过程并不繁琐，但具体的算法却比较复杂，没有非常专业的背景知识是很难理清的，需要理解MP3的文件结构．尤其是最后的Resync和GetMPEGAudioFrameSize， 不过没有关系，学习是一个渐进的过程，我相信通过不断的学习和知识的不断丰富，以后再来看这些实现会慢慢容易一些。