Name explanation:
The GOP (image group) is mainly used as how many frames have been described in an I frame to the next I frame. Increasing the picture group (GOP) can effectively reduce the encoded video volume, but will also reduce the quality of the video. Each set of images (GOP, image groups) of the encoded video gives the sequence (PPS) and the image parameters of the frame (SPS) of the frame itself. SPS (Sequence Parameter Set) Sequence Parameter Set, PPS (Picture Parameter SET) image parameter set, containing various parameter information for image encoding, is the parameter information necessary for decoder initialization. IDR (Instantaneous Decoding Refresh) frame, that is, instant decoding refresh frame, intuitive means that the decoder refreshes the reference frame cache after receiving the IDR frame. The video frame before and after the IDR frame will not have any reference relationship, and the decoder can begin to decode any one IDR frame.
H.264 syntactics and semantics
In H.264, the syntax elements are organized into sequences, images, sheets, macroblocks, and sub-macroblocks. Each of these frames is NALU, understand NALU, understand the structure of H.264.
The syntax element is organized to have a hierarchical structure, which describes the information of each level, respectively.
Teminape structure of syntax elements in previous standards
In such a structure, the head of each layer and its data portion forms management and managed strong dependencies, the syntax element of the head is the core of the layer data, and once the head is lost, the information of the data portion is almost It is impossible to decode correctly. In particular, in the sequence layer and the image layer, due to the limit of the MTU (maximum transmission unit) size in the network, it is impossible to put all the syntax elements of the entire layer into the same packet. This time, if the packet where the head is lost, the layer is other The group cannot decode even if it can be properly received, resulting in waste of resources.
Templary structure of syntax elements in H.264
In H.264, the largest of the hierarchical structure is to cancel the sequence layer and the image layer, and freely from most of the syntax elements of the sequence and the image head to form a sequence and image two-stage parameter set, the rest of the part Put insert. The parameter set is an independent data unit that does not depend on other syntax elements outside the parameter collected.
Code current model
Simplified code stream structure model
The first image of a sequence is called an IDR image (immediately refresh image), and the IDR image is an image. H.264 Introducing the IDR image is for decoding synchronization, when the decoder decodes the IDR image, immediately clear the reference frame queue, output or discard the decoded data, re-find the parameter set, and start a new sequence. Thus, if major errors occur in the transmission of the previous sequence, such as severe packet loss, or other reasons, data misalignment, here you can get resynchronization. The image after the IDR image never references the data of the image before the IDR image to decode. The difference between the IDR image and the i image, the IDR image must be a I image, but the i image is not necessarily an IDR image. A sequence can have a lot of I images, and the image after the I image can reference the image between the I image to exercise reference.
H.264 Basic Flow NALU
The structure of the Elementary Stream, ES is divided into two layers, including video encoding layers (VCLs) and network adaptation layers (NAL). The video coding layer VCL is responsible for efficient video content, is a definition of the grammatical level of the core algorithm engine, block, macroblock, and the film, and he finally outputs the encoded data SODB. Network adaptation layer NAL is responsible for packaging and transmitting data as required by the network, packed SODB into RBSP and add NAL heads to form a NALU (NAL unit, NAL Unit).
The benefits of introducing NAL and splitting the VCL separation include two aspects:
Make signal processing and network transmission separation, VCL, and NAL can be implemented on different processing platforms; VCL and NAL separation design, enabling the gateway to reconstruct the VCL bitstream in different network environments due to network environments. Heavy encoding. Nouns Introduction SODB (STRING OF DATA BITS): The original data bitstream, the length is not necessplified, so it needs to be completed RBSP (Raw Byte Sequence PayLoad): Original Data Byteflow, SODB + RBSP Trailing Bits = RBSP, Add Plus Trailing Bits is to make an RBSP as a whole field number, byte alignment H.264's basic stream consists of a series of NETWORK ABSTRACTION LAYER Unit, and different NALU data is different, and each NALU may be IDR images, SPS, PPS, NON-IDR images, etc. The draft of H.264 pointed out that when the data stream is stored in the medium, start codes are added before each NALU: 0x000001 or 0x00000001 to indicate the start and termination position of a NALU. Under such a mechanism, the start code is detected in the code stream as a NALU to start identification, and when the next start code is detected, the current NALU ends. 3 to 4 bytes of each frame in each frame in the H.264 stream is the start_code (start code), 0x00000001 or 0x000001 of H.264. The 3-byte 0x000001 is only used, just a complete frame is compiled as a plurality of SLICE (tablets), starting from the second slice, including these SLICE NALUs use 3 bytes start code. That is, if the NALU corresponding to the SLICE is used in the beginning of a frame, the 0x00000001 is used, otherwise 0x000001. A video frame may contain multiple NALUs, which may be called a multi-slice video frame (one slice containing the video frame in a NALU)
NALU structure
NALU, NAL head + RBSP
NALU head structure
Length: 1byte, orbidden_bit (1bit) + nal_reference_bit (2bit) + NAL_UNIT_TYPE (5bit) f (forbidden_zero_bit): 1 bit, initial 0. When the network recognizes that this unit is a bit error, it can be set to 1 so that the recipient throws off the unit NRI (NAL_REF_IDC): 2 bits to indicate the importance level of the NALU. The greater the value, the more important it is currently the current NALU. When the specifier is more than 0, it is not clearly defined in Type (nal_unit_type): 5 bits, pointing in the type of NALU NALU decoding process nal_unit_type = 5: Indicates that the current NAL is a piece of the IDR image, in which case in the IDR image Nal_Unit_type of each piece should be equal to 5. Note that the IDR image cannot use partitions. NAL_UNIT_TYPE = 7 or 8: Each SPS or PPS only corresponds to a NALU.
RBSP structure
A typical RBSP unit sequence is shown in the figure. Each unit is transmitted in a stand-alone NAL unit. The header information of the NAL unit defines the type of the RBSP unit, and the rest of the NAL unit is RBSP data. Here is the RBSP sequence, requiring NAL packaging and NAL heads.
piece
Slice: One frame image can be encoded into one or more sheets, each tablet containing an integer macrob block, i.e., at least one macroblock per piece, a macroblock of the entire image. The purpose of fragmentation is to limit the diffusion and transmission of the error, keep the encoders independently. The syntax structure of the sheet specifies the type of the film, which image is belonging to which image, the reference image, etc., the data contains a series of encoded macroblocks, and or hops (not encoded) data. Each macroblock contains head units and residual data. When the type IDR image, Slice_Type is equal to 2, 4, 7, 9.
Macroblock
Macro Block: An encoded image is usually divided into several macroblocks, and a macroblock consists of a 16 x 16 luminance pixel and an additional 8 × 8 CB and an 8 × 8 CR color pixel block. In each image, several macroblocks are arranged in the form of a film. The I slice includes only I macroblocks, and the P sheets can include P and I macroblocks, while the B tablets can include B and I macroblocks. The I macroblock uses a pixel that has decoded from the current piece to perform intra prediction (the decoded pixels in other slices can be taken as a reference for intra prediction). The P macroblock is used as a reference image as a reference image, and an intra-coded macroblock can further make macroblock segmentation: 16 × 16, 16 × 8, 8 × 16 or 8 × 8 brightness Pixel block (and the included color pixel); if the 8 × 8 sub-macroblock is selected, it can be divided into various sub-macroblocks, with a size of 8 × 8, 8 × 4, 4 × 8 or 4 × 4 brightness pixel block (and the included color pixels). The B macroblock uses a two-way reference image (current and future encoded image frames) for intra prediction.
Summarize
The basic stream of H.264 is composed of a series of NETWORK ABSTRACTION Layer Unit, and different NALU data is different, and each NALU may be an IDR image, SPS, PPS, NON-IDR image, and the like.
In H.264, the syntax elements are organized into sequences, images, sheets, macroblocks, and sub-macroblocks. Each of these frames is NALU, understand NALU, understand the structure of H.264.
Reference: h.264.pdf in-depth, simple understanding of video encoding H264 Structure H.264 Format Analysis H.264 streaming media protocol format ANNEX B format and AVCC format depth resolution mobile hard solution critical process card X264 Source code parsing H264 Note H264 frame format Analysis FFMPEG Code H.264 frame type judgment
Our other product:
