live555源码分析----H264的数据处理
来源:互联网 发布:大一c语言编程四则运算 编辑:程序博客网 时间:2024/05/22 00:35
现在来分析live555中关于H264的处理部分,主要包括从文件中读取数据进行并进行frame(NALU)的分割,然后对frame进行分片,这些工作都是在frame交给RTP sink之前完成的。- void MultiFramedRTPSink::packFrame() {
- if (fOutBuf->haveOverflowData()) {
- ...
- } else {
- ...
-
-
-
-
-
- fSource->getNextFrame(fOutBuf->curPtr(), fOutBuf->totalBytesAvailable(),
- afterGettingFrame, this, ourHandleClosure, this);
- }
- }
getNextFrame是定义在FramedSource中的非虚函数,从source中获取下一个frame,然后调用回调函数afterGettingFrame。afterGettingFrame被定义为静态函数,因为在C++中类成员函数是不能作为回调用函数的。不过这里为什么要用回调函数回? 注意,对于H264来说,上面的fSource并不是MPEGVideoStreamFramer类,因为在 H264VideoRTPSink::continuePlaying()函数中改变了fSource的值。 - Boolean H264VideoRTPSink::continuePlaying() {
-
-
- if (fOurFragmenter == NULL) {
-
-
- fOurFragmenter = new H264FUAFragmenter(envir(), fSource, OutPacketBuffer::maxSize,
- ourMaxPacketSize() - 12);
- fSource = fOurFragmenter;
- }
-
-
-
- return MultiFramedRTPSink::continuePlaying();
- }
fSource被指向了H264FUAFragmenter类,这个类主要实现了H264按照RFC3984进行RTP分包,不过这里的实现每个RTP中最多只包含一个NALU,没有实现组合封包的情形。这个类的继承关系如下:H264FUAFragmenter->FramedFilter->FramedSource。很明显,这是一个filter,包装了MPEGVideoStreamFramer类的对像。 先来看来看getNextFrame的实现- void FramedSource::getNextFrame(unsigned char* to, unsigned maxSize,
- afterGettingFunc* afterGettingFunc,
- void* afterGettingClientData,
- onCloseFunc* onCloseFunc,
- void* onCloseClientData) {
-
- if (fIsCurrentlyAwaitingData) {
- envir() << "FramedSource[" << this << "]::getNextFrame(): attempting to read more than once at the same time!\n";
- envir().internalError();
- }
-
-
- fTo = to;
- fMaxSize = maxSize;
- fNumTruncatedBytes = 0;
- fDurationInMicroseconds = 0;
- fAfterGettingFunc = afterGettingFunc;
- fAfterGettingClientData = afterGettingClientData;
- fOnCloseFunc = onCloseFunc;
- fOnCloseClientData = onCloseClientData;
- fIsCurrentlyAwaitingData = True;
-
-
- doGetNextFrame();
- }
上面的函数主要是进行一些成员变量的初始化,获取到的frame需要保存到fTo地址中,然后调用fAfterGettingFunc函数,若文件读取完毕,还需要调用fOnCloseFunc函数。重要的工作还是在doGetNextFrame函数中完成,不过它是定义在FramedSource类中的纯虚函数,需要在子类中重新实现。 现在来看H264FUAFragmenter中对doGetNextFrame的实现- void H264FUAFragmenter::doGetNextFrame() {
- if (fNumValidDataBytes == 1) {
-
-
- fInputSource->getNextFrame(&fInputBuffer[1], fInputBufferSize - 1,
- afterGettingFrame, this,
- FramedSource::handleClosure, this);
- } else {
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- if (fMaxSize < fMaxOutputPacketSize) {
- envir() << "H264FUAFragmenter::doGetNextFrame(): fMaxSize ("
- << fMaxSize << ") is smaller than expected\n";
- } else {
- fMaxSize = fMaxOutputPacketSize;
- }
-
-
- fLastFragmentCompletedNALUnit = True;
- if (fCurDataOffset == 1) {
- if (fNumValidDataBytes - 1 <= fMaxSize) {
-
-
-
- memmove(fTo, &fInputBuffer[1], fNumValidDataBytes - 1);
- fFrameSize = fNumValidDataBytes - 1;
- fCurDataOffset = fNumValidDataBytes;
- } else {
-
-
-
-
-
-
-
- fInputBuffer[0] = (fInputBuffer[1] & 0xE0) | 28;
- fInputBuffer[1] = 0x80 | (fInputBuffer[1] & 0x1F);
- memmove(fTo, fInputBuffer, fMaxSize);
- fFrameSize = fMaxSize;
- fCurDataOffset += fMaxSize - 1;
- fLastFragmentCompletedNALUnit = False;
- }
- } else {
-
-
-
-
-
-
-
-
-
-
- fInputBuffer[fCurDataOffset-2] = fInputBuffer[0];
- fInputBuffer[fCurDataOffset-1] = fInputBuffer[1]&~0x80;
- unsigned numBytesToSend = 2 + fNumValidDataBytes - fCurDataOffset;
- if (numBytesToSend > fMaxSize) {
-
- numBytesToSend = fMaxSize;
- fLastFragmentCompletedNALUnit = False;
- } else {
-
-
-
- fInputBuffer[fCurDataOffset-1] |= 0x40;
- fNumTruncatedBytes = fSaveNumTruncatedBytes;
- }
- memmove(fTo, &fInputBuffer[fCurDataOffset-2], numBytesToSend);
- fFrameSize = numBytesToSend;
- fCurDataOffset += numBytesToSend - 2;
- }
-
-
- if (fCurDataOffset >= fNumValidDataBytes) {
-
- fNumValidDataBytes = fCurDataOffset = 1;
- }
-
-
-
- FramedSource::afterGetting(this);
- }
- }
H264FUAFragmenter::doGetNextFrame函数第一次执行时,执行条件1,需要调用 MPEGVideoStreamFramer::doGetNextFrame读取一个新的frame,获取frame的具体过程稍后再分析。现在先看获取frame之后的工作,afterGettingFrame函数- void H264FUAFragmenter::afterGettingFrame(void* clientData, unsigned frameSize,
- unsigned numTruncatedBytes,
- struct timeval presentationTime,
- unsigned durationInMicroseconds) {
- H264FUAFragmenter* fragmenter = (H264FUAFragmenter*)clientData;
- fragmenter->afterGettingFrame1(frameSize, numTruncatedBytes, presentationTime,
- durationInMicroseconds);
- }
没什么好说的,再看afterGettingFrame1函数- void H264FUAFragmenter::afterGettingFrame1(unsigned frameSize,
- unsigned numTruncatedBytes,
- struct timeval presentationTime,
- unsigned durationInMicroseconds) {
- fNumValidDataBytes += frameSize;
- fSaveNumTruncatedBytes = numTruncatedBytes;
- fPresentationTime = presentationTime;
- fDurationInMicroseconds = durationInMicroseconds;
-
-
-
- doGetNextFrame();
- }
上面的代码首先记录几个数据到成员变量中,fNumValidDataBytes很重要,表示读取到的frame长度+1。然后又一次调用了H264FUAFragmenter::doGetNextFrame(),这里将进入H264FUAFragmenter::doGetNextFrame函数中第二个条件分支,这种循环调用很容易把人弄迷糊了。 H264FUAFragmenter::doGetNextFrame函数中第二个条件分支中,处理H264的RTP分片问题,这里是按照RFC3984进行RTP封装的。你应该注意到,在上篇文章"RTP的打包与发送"中,也出现了分片的代码(MultiFramedRTPSink::packFrame函数中),那里直接将frame按MTU的长度来拆分。那为什么H264还要自定义一套RTP打包的标准呢?暂时我也不清楚。
在H264FUAFragmenter::doGetNextFrame()最后调用了 FramedSource::afterGetting- void FramedSource::afterGetting(FramedSource* source) {
- source->fIsCurrentlyAwaitingData = False;
-
-
-
-
-
-
- if (source->fAfterGettingFunc != NULL) {
- (*(source->fAfterGettingFunc))(source->fAfterGettingClientData,
- source->fFrameSize, source->fNumTruncatedBytes,
- source->fPresentationTime,
- source->fDurationInMicroseconds);
- }
- }
上面的代码主要是调用了FramedSource::getNextFrame函数中传递下来的回调函数,这个回调函数就是MultiFramedRTPSink::afterGettingFrame,处理过程在上一篇文章"RTP的打包与发送"中已经分析过了。 现在来看MPEGVideoStreamFramer::doGetNextFrame获取Frame的过程。继承关系:H264VideoStreamFramer->MPEGVideoStreamFramer->FramedFilter->FramedSource。在继承路径中存在FrameFilter,这说明H264VideoStreamFramer包装了其它source(包装的是读取文件的字节流source)。doGetNextFrame函数首先在MPEGVideoStreamFramer中实现。- void MPEGVideoStreamFramer::doGetNextFrame() {
- fParser->registerReadInterest(fTo, fMaxSize);
- continueReadProcessing();
- }
这里的MPEGVideoStreamFramer::fParser,是一个MPEGVideoStreamParser类型指针,作为语法分析器。再来看continueReadProcessing函数- void MPEGVideoStreamFramer::continueReadProcessing() {
- unsigned acquiredFrameSize = fParser->parse();
- if (acquiredFrameSize > 0) {
-
-
- fFrameSize = acquiredFrameSize;
- fNumTruncatedBytes = fParser->numTruncatedBytes();
-
-
-
-
-
-
- fDurationInMicroseconds
- = (fFrameRate == 0.0 || ((int)fPictureCount) < 0) ? 0
- : (unsigned)((fPictureCount*1000000)/fFrameRate);
- #ifdef DEBUG
- fprintf(stderr, "%d bytes @%u.%06d, fDurationInMicroseconds: %d ((%d*1000000)/%f)\n", acquiredFrameSize, fPresentationTime.tv_sec, fPresentationTime.tv_usec, fDurationInMicroseconds, fPictureCount, fFrameRate);
- #endif
- fPictureCount = 0;
-
-
-
-
-
-
-
- afterGetting(this);
- } else {
-
-
-
- }
- }
函数中首先调用了MPEGVideoStreamParser::parse函数,将一个完整的Frame分析出来,并copy到了fTo(fTo就是OutPacketBuffer中的缓冲区)中,这其中肯定也实现了从文件中读取数据的过程。这里的fNumTruncatedBytes变量需要注意,fNumTruncatedBytes>0的话,表明Frame的实际长度大于fTo的最大长度,这将导致数据丢失,这时就要考虑增加缓冲区的长度了。成功获取一个Frame后,将调用afterGetting函数处理后续工作。 先来看parse函数,parse是定义在MPEGVideoStreamParser中的纯虚函数,在子类H264VideoStreamParser中实现。parse主要是从文件的字节流中,分离出一个个的Frame,对于H264而言其实就是对一个个的NALU。*.264文件的格式非常简单,每个NALU以 0x00000001 作为起始符号(中间的NALU也可以以0x000001作为起始符),顺序存放。- unsigned H264VideoStreamParser::parse() {
- try {
-
-
-
-
- if (!fHaveSeenFirstStartCode) {
-
- u_int32_t first4Bytes;
- while ((first4Bytes = test4Bytes()) != 0x00000001) {
- get1Byte(); setParseState();
- }
- skipBytes(4);
-
- setParseState();
- fHaveSeenFirstStartCode = True;
- }
-
-
-
- if (fOutputStartCodeSize > 0) {
-
- save4Bytes(0x00000001);
- }
-
-
-
-
-
-
- u_int8_t firstByte;
- if (haveSeenEOF()) {
-
-
-
-
-
- unsigned remainingDataSize = totNumValidBytes() - curOffset();
- if (remainingDataSize == 0) (void)get1Byte();
- #ifdef DEBUG
- fprintf(stderr, "This NAL unit (%d bytes) ends with EOF\n", remainingDataSize);
- #endif
- if (remainingDataSize == 0) return 0;
- firstByte = get1Byte();
- saveByte(firstByte);
-
- while (--remainingDataSize > 0) {
- saveByte(get1Byte());
- }
- } else {
- u_int32_t next4Bytes = test4Bytes();
- firstByte = next4Bytes>>24;
-
-
-
- while (next4Bytes != 0x00000001 && (next4Bytes&0xFFFFFF00) != 0x00000100) {
-
- if ((unsigned)(next4Bytes&0xFF) > 1) {
-
- save4Bytes(next4Bytes);
- skipBytes(4);
- } else {
-
- saveByte(next4Bytes>>24);
- skipBytes(1);
- }
- next4Bytes = test4Bytes();
- }
-
-
- if (next4Bytes == 0x00000001) {
- skipBytes(4);
- } else {
- skipBytes(3);
- }
- }
-
- u_int8_t nal_ref_idc = (firstByte&0x60)>>5;
- u_int8_t nal_unit_type = firstByte&0x1F;
- #ifdef DEBUG
- fprintf(stderr, "Parsed %d-byte NAL-unit (nal_ref_idc: %d, nal_unit_type: %d (\"%s\"))\n",
- curFrameSize()-fOutputStartCodeSize, nal_ref_idc, nal_unit_type, nal_unit_type_description[nal_unit_type]);
- #endif
-
-
-
-
-
- switch (nal_unit_type) {
- case 6: {
- analyze_sei_data();
-
- break;
- }
- case 7: {
-
-
-
-
- usingSource()->saveCopyOfSPS(fStartOfFrame + fOutputStartCodeSize, fTo - fStartOfFrame - fOutputStartCodeSize);
-
-
-
- unsigned num_units_in_tick, time_scale, fixed_frame_rate_flag;
- analyze_seq_parameter_set_data(num_units_in_tick, time_scale, fixed_frame_rate_flag);
- if (time_scale > 0 && num_units_in_tick > 0) {
- usingSource()->fFrameRate = time_scale/(2.0*num_units_in_tick);
- #ifdef DEBUG
- fprintf(stderr, "Set frame rate to %f fps\n", usingSource()->fFrameRate);
- if (fixed_frame_rate_flag == 0) {
- fprintf(stderr, "\tWARNING: \"fixed_frame_rate_flag\" was not set\n");
- }
- #endif
- } else {
- #ifdef DEBUG
- fprintf(stderr, "\tThis \"Sequence Parameter Set\" NAL unit contained no frame rate information, so we use a default frame rate of %f fps\n", usingSource()->fFrameRate);
- #endif
- }
- break;
- }
- case 8: {
-
- usingSource()->saveCopyOfPPS(fStartOfFrame + fOutputStartCodeSize, fTo - fStartOfFrame - fOutputStartCodeSize);
- }
- }
-
- usingSource()->setPresentationTime();
- #ifdef DEBUG
- unsigned long secs = (unsigned long)usingSource()->fPresentationTime.tv_sec;
- unsigned uSecs = (unsigned)usingSource()->fPresentationTime.tv_usec;
- fprintf(stderr, "\tPresentation time: %lu.%06u\n", secs, uSecs);
- #endif
-
-
-
-
-
-
-
-
-
- Boolean thisNALUnitEndsAccessUnit = False;
- if (haveSeenEOF()) {
-
- thisNALUnitEndsAccessUnit = True;
- } else {
- Boolean const isVCL = nal_unit_type <= 5 && nal_unit_type > 0;
- if (isVCL) {
- u_int32_t first4BytesOfNextNALUnit = test4Bytes();
- u_int8_t firstByteOfNextNALUnit = first4BytesOfNextNALUnit>>24;
- u_int8_t next_nal_ref_idc = (firstByteOfNextNALUnit&0x60)>>5;
- u_int8_t next_nal_unit_type = firstByteOfNextNALUnit&0x1F;
- if (next_nal_unit_type >= 6) {
-
-
- #ifdef DEBUG
- fprintf(stderr, "\t(The next NAL unit is not a VCL)\n");
- #endif
- thisNALUnitEndsAccessUnit = True;
- } else {
-
-
-
- Boolean IdrPicFlag = nal_unit_type == 5;
- Boolean next_IdrPicFlag = next_nal_unit_type == 5;
- if (next_IdrPicFlag != IdrPicFlag) {
-
- #ifdef DEBUG
- fprintf(stderr, "\t(IdrPicFlag differs in value)\n");
- #endif
- thisNALUnitEndsAccessUnit = True;
- } else if (next_nal_ref_idc != nal_ref_idc && next_nal_ref_idc*nal_ref_idc == 0) {
-
- #ifdef DEBUG
- fprintf(stderr, "\t(nal_ref_idc differs in value with one of the nal_ref_idc values being equal to 0)\n");
- #endif
- thisNALUnitEndsAccessUnit = True;
- } else if ((nal_unit_type == 1 || nal_unit_type == 2 || nal_unit_type == 5)
- && (next_nal_unit_type == 1 || next_nal_unit_type == 2 || next_nal_unit_type == 5)) {
-
-
-
-
- unsigned frame_num, pic_parameter_set_id, idr_pic_id;
- Boolean field_pic_flag, bottom_field_flag;
- analyze_slice_header(fStartOfFrame + fOutputStartCodeSize, fTo, nal_unit_type,
- frame_num, pic_parameter_set_id, idr_pic_id, field_pic_flag, bottom_field_flag);
-
-
- #ifdef DEBUG
- fprintf(stderr, " Next NAL unit's slice_header:\n");
- #endif
- u_int8_t next_slice_header[NUM_NEXT_SLICE_HEADER_BYTES_TO_ANALYZE];
- testBytes(next_slice_header, sizeof next_slice_header);
- unsigned next_frame_num, next_pic_parameter_set_id, next_idr_pic_id;
- Boolean next_field_pic_flag, next_bottom_field_flag;
- analyze_slice_header(next_slice_header, &next_slice_header[sizeof next_slice_header], next_nal_unit_type,
- next_frame_num, next_pic_parameter_set_id, next_idr_pic_id, next_field_pic_flag, next_bottom_field_flag);
-
- if (next_frame_num != frame_num) {
-
- #ifdef DEBUG
- fprintf(stderr, "\t(frame_num differs in value)\n");
- #endif
- thisNALUnitEndsAccessUnit = True;
- } else if (next_pic_parameter_set_id != pic_parameter_set_id) {
-
- #ifdef DEBUG
- fprintf(stderr, "\t(pic_parameter_set_id differs in value)\n");
- #endif
- thisNALUnitEndsAccessUnit = True;
- } else if (next_field_pic_flag != field_pic_flag) {
-
- #ifdef DEBUG
- fprintf(stderr, "\t(field_pic_flag differs in value)\n");
- #endif
- thisNALUnitEndsAccessUnit = True;
- } else if (next_bottom_field_flag != bottom_field_flag) {
-
- #ifdef DEBUG
- fprintf(stderr, "\t(bottom_field_flag differs in value)\n");
- #endif
- thisNALUnitEndsAccessUnit = True;
- } else if (next_IdrPicFlag == 1 && next_idr_pic_id != idr_pic_id) {
-
-
- #ifdef DEBUG
- fprintf(stderr, "\t(IdrPicFlag is equal to 1 for both and idr_pic_id differs in value)\n");
- #endif
- thisNALUnitEndsAccessUnit = True;
- }
- }
- }
- }
- }
-
- if (thisNALUnitEndsAccessUnit) {
- #ifdef DEBUG
- fprintf(stderr, "*****This NAL unit ends the current access unit*****\n");
- #endif
- usingSource()->fPictureEndMarker = True;
- ++usingSource()->fPictureCount;
-
-
-
-
-
-
- struct timeval& nextPT = usingSource()->fNextPresentationTime;
- nextPT = usingSource()->fPresentationTime;
- double nextFraction = nextPT.tv_usec/1000000.0 + 1/usingSource()->fFrameRate;
- unsigned nextSecsIncrement = (long)nextFraction;
- nextPT.tv_sec += (long)nextSecsIncrement;
- nextPT.tv_usec = (long)((nextFraction - nextSecsIncrement)*1000000);
- }
- setParseState();
-
-
- return curFrameSize();
- } catch (int ) {
- #ifdef DEBUG
- fprintf(stderr, "H264VideoStreamParser::parse() EXCEPTION (This is normal behavior - *not* an error)\n");
- #endif
- return 0;
- }
- }
H264VideoStreamParser::parse()函数除了取出Frame,还对NALU中的部分参数做了解释工作。对于PPS或者SPS类型的NALU,要保存到H264VideoStreamFramer中。"access unit",在这里可以理解为一副图像,一个"access unit"可以包含多个NALU,很显示这些NALU的时间戳应该是相同的。实际上,很多时候一个"access unit"单元只包含一个NALU,这里简单多了。分析过程是根据section 7.4.1.2.4 of the H.264 specification进行的。
H264数据处理完成后就可以进行RTP的打包与发送了,见下篇live555源码分析----RTP的打包与发送
原文链接:http://blog.csdn.net/gavinr/article/details/7042228
2 0