H.264 stream formats

There are two H.264 stream formats and they are sometimes called

1. Annex B
2. MP4

An H.264 stream is made of NALs (a unit of packaging)

1. Annex B: has start codes 0x00 0x00 0x00 0x01 NAL 0x00 0x00 0x00
   0x01 NAL etc
2. MP4: is size prefixed SIZE NAL SIZE NAL etc

The MP4 stream format doesn't contain any NALs of type SPS, PPS or AU delimter.

The Annex B format you'll find in MPEG-2 TS, RTP and some encoders default output.

The MP4 format you'll find in MP4 files. Both formats can be converted into each other.

Annex B -> MP4: remove start codes, insert length of NAL, filter out SPS, PPS and AU delimiter.

MP4 -> Annex B: remove length, insert start code, insert SPS for each I-frame, insert PPS for each frame, insert AU delimiter for each GOP.